2022-07-21T01:25:33.453 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-21T01:25:33.455 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359 --name yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi --description fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} --verbose -- /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/orig.config.yaml 2022-07-21T01:25:33.509 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-07-21T01:25:33.590 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359 branch: octopus-release description: fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} email: null first_in_suite: false job_id: '6941359' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: octopus-release ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - evicting unresponsive client - POOL_APP_NOT_ENABLED - has not responded to cap revoke by MDS for over - MDS_CLIENT_LATE_RELEASE - responding to mclientcaps sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: octopus-release sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - - client.1 sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 sleep_before_teardown: 0 suite: fs suite_branch: octopus-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 targets: smithi003.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDinwZIIfMdjuwjbCLOcoWbvzTlWjYXhwE3nvoFhlvF/iTjpSA7LVyBtgW4GvqT4FiRZkznnU7/A4tRfB4Szkl7hkZCMjrXC8P3Eah66sG1OfWE8EnQd4VbqZgj3BbOpAK4+U99qnwkISjnMOM9jQBAQRSUjlzN7Rv0EWxrb4AEXtl+lFNmUevGe8OtJs6s0KJF3N5ZgkzOzR6Tdi0yePraZFoHDyA5juvwSiO3YN1LUz0Ro9DtoMm8i0DATn7d3drfBMoDrHTdkWz44pCXInZVHBYQ3zSJQ0gvoRmOCpvAtSW/pam6BX/S9qQYPZfhtT9WnK29YnziXJbQ7zxdrd8GoGaLzufTqJIuCgFNheLOfBmOMPCbfGE+XfK3MOUfAUmvph9PbAjdpJX9N+1z4m8wUg30TRf8uD3atPjGKZPp7M+DKncEOFSx3FMcUGoSaIAzGa8EirtuqaweIlYCRPEObcU00eQHD+rj1FJJz9cBuovU0p1z6+++A22h5fOb7I8= smithi143.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDESHLkWB2ibBysFJ6hMDWnNs8GyPMOGDq9VFKWZSL5NtLe6TFdfQTc0H9l7PlL2IMQ97/bMZ3W3du8ZapyZpW67dEWOFbZqr1Ed2qetWMHCpsj4418kEP2XFLZQCO+ytBYw2yBVKGzVQoNuu0qbUke+BKO2o0Sxe0QTVmC5U0cVwHO39iOt/CZtRfSGZ+RUzLyjBI97i2NYtcB1TroAK174o2fe+f8leEczVwf4GBbTEJ+kA1OgbzkZaBFB+TKUEVrtghWN4nQuumVGIUs+P9Qlel7GhN1EkPaxgYiorP3bUo/7k8Tl+3CTVFuwnOug4Fl5t+aJ8cfPwHPV6be4PIpCIiiP9MTueNN1C6lBapRO926FP99vhA+VYYxruwAlplEUnaNdh7K6j58KRun0K/hbRR+9aERG6t0ly96WU2oE5UxjzRIQHV9q+a+cu1gdyOh8lKqMK2zwV3qZ4GRpOZJ12Z1MgDs2UeuYHE9diYFr8QtetIysFJxCWszDikfHYs= smithi171.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUDw9VoK2vcRLET5lYnBXRmv2lJkTKWQTmGSFDaibmQf5RfYJSQ2Od4HMUiqrEJwpLNXH5ZzD9hIZX7fxPGahS0SDeTJym62DMnpQIDl9bp7u6YP3H6I+kT/t1ND78jor51IOpOloIykT3kO2q6YE/OyRoNFFn+6TnP+3bP2tsB7j1aF9hQBUAAAWCB9E2vz+nplVprQRlgx4xzW0kMA3XJBOsQsL8S9uPw3rUXvFmNf2DLKW9O3q4lgp9hix3oP5p7q1lc1OhuFjnYMbXCjvU9KAKx52p+YlGy5mEiCSL76EH18t1x1IIgvxyXaGtBYhnaS+sg04XBLYceCupkBHZ/J3WbpKW6KKU3tKdM52HOIy02hppqWLx9KKdAt6xqDzX/P31H5CA4V/ZCrzyY5n9uSM0pzZE+HGoQIu3P7AE6IkfD0AHQWZi3gIrm+Wsj9WBYZQxhX9ARIj2R682MBXywck0IWa5aGdNSVQAt8NdS6EdQAHSmwlQklRlxb2NHE8= smithi174.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAJC1IbSPRKdIO9nqlOgWHwU4ohDj2qmW41hWsR5W1r4XC44vafIo9WIUXILqxmQY2dJRhrJ0iAIGf2aeNv5/rTtO+DtzfPi8z6GlxbmL33Sm22SphH5qKu6ddrxumT6jaan06s/GIgfZpt8/LbUPjbQfUc0q1JRsgTN1X9FdN6/Iz5j+pdNGDKRMt3d/DG2gDC2c812P/8WO/c1SsGJq7tMEE6djNpfJQrdyLpMSSA10YBSvOkhLirvyWLljCZLZ7Jk8WL6KlqKCTF3HJ1Le1iiRlhJ1ylR4Tem0Oy/BA+of3s+jEnnTHcBcEPchxMDhUC7cqRh5Uxpn/yYDu+yBPWctGajYt2HgQ3EK6RJ0JuvARD5zyj8cnI9si5M6sXmGkeNUVtbrJ6RmHtJCr/zZm4/IcZFTSS7vIcFVQFc6fH/kM1zBF9h8uQLvvVL60ZUrRDuAL6jEt/9xk+6THonxkrvJpceiWl9Q/sq9G/ZD/UzZVKkrCfBlAxQRKRiJIqgc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_misc teuthology_branch: main teuthology_sha1: 89d874d555f56b1c1f383df72487aecec0ee6cf9 timestamp: 2022-07-20_20:56:09 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-07-21T01:25:33.591 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa; will attempt to use it 2022-07-21T01:25:33.592 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks 2022-07-21T01:25:33.592 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-07-21T01:25:33.594 INFO:teuthology.task.internal:Checking packages... 2022-07-21T01:25:33.617 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '57544da114dbbe1c44f27b2ea18e3233e4e917e0' 2022-07-21T01:25:33.617 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-07-21T01:25:33.618 INFO:teuthology.packaging:ref: None 2022-07-21T01:25:33.618 INFO:teuthology.packaging:tag: None 2022-07-21T01:25:33.618 INFO:teuthology.packaging:branch: octopus-release 2022-07-21T01:25:33.619 INFO:teuthology.packaging:sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-21T01:25:33.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=octopus-release 2022-07-21T01:25:33.797 INFO:teuthology.task.internal:Found packages for ceph version 15.2.16-336-g57544da1-1focal 2022-07-21T01:25:33.797 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-07-21T01:25:33.804 INFO:teuthology.task.internal:no buildpackages task found 2022-07-21T01:25:33.804 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-07-21T01:25:33.818 INFO:teuthology.task.internal:Saving configuration 2022-07-21T01:25:33.835 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-07-21T01:25:33.843 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-21T01:25:33.869 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-21 01:20:02.342720', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMnO91yhnJhL4DZcnbrDBmX33TkUACb6xfSfNPXmMxTt'} 2022-07-21T01:25:33.893 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-21 01:20:02.339669', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBrvFCmVE/08b0geEkCwo130N81USzr2WKYQ+7lx2+BR'} 2022-07-21T01:25:33.917 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-21 01:20:02.337510', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL09f90wiVfw8uFEmK24COBagIEE/YkhirpzKqdCEcQ/'} 2022-07-21T01:25:33.941 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-21 01:20:02.341269', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEDHSj0rZgzeSRipayYskf1/AQsPTGAKzDkIRm41erGe'} 2022-07-21T01:25:33.942 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-07-21T01:25:33.949 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-07-21T01:25:33.949 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-07-21T01:25:33.950 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['client.0'] 2022-07-21T01:25:33.951 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['client.1'] 2022-07-21T01:25:33.951 INFO:teuthology.run_tasks:Running task console_log... 2022-07-21T01:25:34.091 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f65bf4a2d90>, signals=[15]) 2022-07-21T01:25:34.093 INFO:teuthology.run_tasks:Running task internal.connect... 2022-07-21T01:25:34.099 INFO:teuthology.task.internal:Opening connections... 2022-07-21T01:25:34.099 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T01:25:34.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T01:25:34.210 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T01:25:34.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T01:25:34.306 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2022-07-21T01:25:34.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T01:25:34.407 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2022-07-21T01:25:34.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T01:25:34.503 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-07-21T01:25:34.510 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2022-07-21T01:25:34.517 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2022-07-21T01:25:34.518 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2022-07-21T01:25:34.567 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="Ubuntu" 2022-07-21T01:25:34.567 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-21T01:25:34.568 INFO:teuthology.orchestra.run.smithi003.stdout:ID=ubuntu 2022-07-21T01:25:34.568 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE=debian 2022-07-21T01:25:34.568 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-21T01:25:34.568 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="20.04" 2022-07-21T01:25:34.569 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-21T01:25:34.569 INFO:teuthology.orchestra.run.smithi003.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-21T01:25:34.569 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-21T01:25:34.569 INFO:teuthology.orchestra.run.smithi003.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-21T01:25:34.569 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_CODENAME=focal 2022-07-21T01:25:34.570 INFO:teuthology.orchestra.run.smithi003.stdout:UBUNTU_CODENAME=focal 2022-07-21T01:25:34.571 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2022-07-21T01:25:34.591 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2022-07-21T01:25:34.599 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2022-07-21T01:25:34.599 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2022-07-21T01:25:34.648 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="Ubuntu" 2022-07-21T01:25:34.649 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-21T01:25:34.649 INFO:teuthology.orchestra.run.smithi143.stdout:ID=ubuntu 2022-07-21T01:25:34.649 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE=debian 2022-07-21T01:25:34.649 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-21T01:25:34.650 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="20.04" 2022-07-21T01:25:34.650 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-21T01:25:34.650 INFO:teuthology.orchestra.run.smithi143.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-21T01:25:34.651 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-21T01:25:34.651 INFO:teuthology.orchestra.run.smithi143.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-21T01:25:34.651 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_CODENAME=focal 2022-07-21T01:25:34.651 INFO:teuthology.orchestra.run.smithi143.stdout:UBUNTU_CODENAME=focal 2022-07-21T01:25:34.652 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2022-07-21T01:25:34.672 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2022-07-21T01:25:34.679 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2022-07-21T01:25:34.679 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2022-07-21T01:25:34.726 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="Ubuntu" 2022-07-21T01:25:34.727 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-21T01:25:34.727 INFO:teuthology.orchestra.run.smithi171.stdout:ID=ubuntu 2022-07-21T01:25:34.727 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE=debian 2022-07-21T01:25:34.728 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-21T01:25:34.728 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="20.04" 2022-07-21T01:25:34.728 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-21T01:25:34.729 INFO:teuthology.orchestra.run.smithi171.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-21T01:25:34.729 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-21T01:25:34.729 INFO:teuthology.orchestra.run.smithi171.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-21T01:25:34.730 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_CODENAME=focal 2022-07-21T01:25:34.730 INFO:teuthology.orchestra.run.smithi171.stdout:UBUNTU_CODENAME=focal 2022-07-21T01:25:34.731 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2022-07-21T01:25:34.752 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2022-07-21T01:25:34.759 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2022-07-21T01:25:34.759 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2022-07-21T01:25:34.809 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="Ubuntu" 2022-07-21T01:25:34.809 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-21T01:25:34.809 INFO:teuthology.orchestra.run.smithi174.stdout:ID=ubuntu 2022-07-21T01:25:34.810 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE=debian 2022-07-21T01:25:34.810 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-21T01:25:34.810 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="20.04" 2022-07-21T01:25:34.810 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-21T01:25:34.811 INFO:teuthology.orchestra.run.smithi174.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-21T01:25:34.811 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-21T01:25:34.811 INFO:teuthology.orchestra.run.smithi174.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-21T01:25:34.812 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_CODENAME=focal 2022-07-21T01:25:34.812 INFO:teuthology.orchestra.run.smithi174.stdout:UBUNTU_CODENAME=focal 2022-07-21T01:25:34.812 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2022-07-21T01:25:34.832 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-07-21T01:25:34.842 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-07-21T01:25:34.848 INFO:teuthology.task.internal:Checking for old test directory... 2022-07-21T01:25:34.848 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2022-07-21T01:25:34.851 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2022-07-21T01:25:34.854 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2022-07-21T01:25:34.857 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2022-07-21T01:25:34.863 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-07-21T01:25:34.869 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-07-21T01:25:34.870 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2022-07-21T01:25:34.872 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2022-07-21T01:25:34.904 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2022-07-21T01:25:34.907 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2022-07-21T01:25:34.925 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-07-21T01:25:35.027 INFO:teuthology.run_tasks:Running task kernel... 2022-07-21T01:25:35.041 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-07-21T01:25:35.042 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-07-21T01:25:35.042 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-21T01:25:35.044 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-21T01:25:35.044 DEBUG:teuthology.orchestra.run.smithi171:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-21T01:25:35.045 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-21T01:25:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:25:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:25:35.052 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2022-07-21T01:25:35.053 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2022-07-21T01:25:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:25:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:25:35.056 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2022-07-21T01:25:35.056 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2022-07-21T01:25:35.060 INFO:teuthology.orchestra.run.smithi171.stdout:5.4.0-122-generic 2022-07-21T01:25:35.061 INFO:teuthology.task.kernel:Running kernel on smithi171: 5.4.0-122-generic 2022-07-21T01:25:35.061 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-get clean 2022-07-21T01:25:35.100 INFO:teuthology.orchestra.run.smithi143.stdout:5.4.0-122-generic 2022-07-21T01:25:35.100 INFO:teuthology.task.kernel:Running kernel on smithi143: 5.4.0-122-generic 2022-07-21T01:25:35.101 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get clean 2022-07-21T01:25:35.103 INFO:teuthology.orchestra.run.smithi003.stdout:5.4.0-122-generic 2022-07-21T01:25:35.103 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.4.0-122-generic 2022-07-21T01:25:35.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get clean 2022-07-21T01:25:35.106 INFO:teuthology.orchestra.run.smithi174.stdout:5.4.0-122-generic 2022-07-21T01:25:35.106 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.4.0-122-generic 2022-07-21T01:25:35.107 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get clean 2022-07-21T01:25:35.234 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-get update 2022-07-21T01:25:35.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2022-07-21T01:25:35.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get update 2022-07-21T01:25:35.243 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2022-07-21T01:25:35.476 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T01:25:35.479 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T01:25:35.480 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-21T01:25:35.480 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-21T01:25:35.483 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-21T01:25:35.550 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-21T01:25:35.551 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-21T01:25:35.581 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-21T01:25:35.593 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T01:25:35.596 INFO:teuthology.orchestra.run.smithi171.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-21T01:25:35.611 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T01:25:35.612 INFO:teuthology.orchestra.run.smithi003.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-21T01:25:35.661 INFO:teuthology.orchestra.run.smithi171.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-21T01:25:35.662 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-21T01:25:35.861 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [696 kB] 2022-07-21T01:25:35.878 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,630 kB] 2022-07-21T01:25:35.946 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,990 kB] 2022-07-21T01:25:35.985 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-21T01:25:35.997 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-21T01:25:35.997 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-07-21T01:25:35.998 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,091 kB] 2022-07-21T01:25:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-21T01:25:36.003 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [155 kB] 2022-07-21T01:25:36.007 INFO:teuthology.orchestra.run.smithi171.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-21T01:25:36.007 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [711 kB] 2022-07-21T01:25:36.013 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-21T01:25:36.014 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,990 kB] 2022-07-21T01:25:36.018 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-21T01:25:36.018 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-21T01:25:36.019 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.7 kB] 2022-07-21T01:25:36.024 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-21T01:25:36.024 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-21T01:25:36.037 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-21T01:25:36.038 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-21T01:25:36.044 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-21T01:25:36.049 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-21T01:25:36.051 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-21T01:25:36.083 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,630 kB] 2022-07-21T01:25:36.086 INFO:teuthology.orchestra.run.smithi171.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,630 kB] 2022-07-21T01:25:36.108 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [696 kB] 2022-07-21T01:25:36.115 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-21T01:25:36.120 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-21T01:25:36.120 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-21T01:25:36.128 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-21T01:25:36.130 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-21T01:25:36.136 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-21T01:25:36.140 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-21T01:25:36.141 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-21T01:25:36.179 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-21T01:25:36.222 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,990 kB] 2022-07-21T01:25:36.226 INFO:teuthology.orchestra.run.smithi171.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,990 kB] 2022-07-21T01:25:36.459 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,630 kB] 2022-07-21T01:25:36.518 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-21T01:25:36.520 INFO:teuthology.orchestra.run.smithi171.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-21T01:25:36.533 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-21T01:25:36.535 INFO:teuthology.orchestra.run.smithi171.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-21T01:25:36.542 INFO:teuthology.orchestra.run.smithi003.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-07-21T01:25:36.543 INFO:teuthology.orchestra.run.smithi003.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,091 kB] 2022-07-21T01:25:36.544 INFO:teuthology.orchestra.run.smithi171.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-07-21T01:25:36.544 INFO:teuthology.orchestra.run.smithi171.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,091 kB] 2022-07-21T01:25:36.602 INFO:teuthology.orchestra.run.smithi003.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [155 kB] 2022-07-21T01:25:36.604 INFO:teuthology.orchestra.run.smithi171.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [155 kB] 2022-07-21T01:25:36.607 INFO:teuthology.orchestra.run.smithi003.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-21T01:25:36.609 INFO:teuthology.orchestra.run.smithi171.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [711 kB] 2022-07-21T01:25:36.626 INFO:teuthology.orchestra.run.smithi003.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [711 kB] 2022-07-21T01:25:36.633 INFO:teuthology.orchestra.run.smithi171.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-21T01:25:36.634 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-21T01:25:36.635 INFO:teuthology.orchestra.run.smithi003.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [696 kB] 2022-07-21T01:25:36.639 INFO:teuthology.orchestra.run.smithi171.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [696 kB] 2022-07-21T01:25:36.650 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-07-21T01:25:36.650 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,091 kB] 2022-07-21T01:25:36.652 INFO:teuthology.orchestra.run.smithi171.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-21T01:25:36.653 INFO:teuthology.orchestra.run.smithi003.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-21T01:25:36.653 INFO:teuthology.orchestra.run.smithi003.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-21T01:25:36.654 INFO:teuthology.orchestra.run.smithi003.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.7 kB] 2022-07-21T01:25:36.655 INFO:teuthology.orchestra.run.smithi171.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.7 kB] 2022-07-21T01:25:36.658 INFO:teuthology.orchestra.run.smithi171.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-21T01:25:36.660 INFO:teuthology.orchestra.run.smithi003.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-21T01:25:36.660 INFO:teuthology.orchestra.run.smithi003.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-21T01:25:36.670 INFO:teuthology.orchestra.run.smithi171.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-21T01:25:36.670 INFO:teuthology.orchestra.run.smithi171.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-21T01:25:36.699 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [155 kB] 2022-07-21T01:25:36.705 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [711 kB] 2022-07-21T01:25:36.716 INFO:teuthology.orchestra.run.smithi003.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-21T01:25:36.721 INFO:teuthology.orchestra.run.smithi003.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-21T01:25:36.723 INFO:teuthology.orchestra.run.smithi171.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-21T01:25:36.725 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-21T01:25:36.727 INFO:teuthology.orchestra.run.smithi171.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-21T01:25:36.739 INFO:teuthology.orchestra.run.smithi003.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-21T01:25:36.739 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-21T01:25:36.742 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.7 kB] 2022-07-21T01:25:36.763 INFO:teuthology.orchestra.run.smithi003.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-21T01:25:36.768 INFO:teuthology.orchestra.run.smithi003.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-21T01:25:36.788 INFO:teuthology.orchestra.run.smithi171.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-21T01:25:36.907 INFO:teuthology.orchestra.run.smithi171.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-21T01:25:36.927 INFO:teuthology.orchestra.run.smithi171.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-21T01:25:39.435 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 11.6 MB in 2s (6,072 kB/s) 2022-07-21T01:25:39.571 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 11.6 MB in 2s (6,067 kB/s) 2022-07-21T01:25:39.954 INFO:teuthology.orchestra.run.smithi171.stdout:Fetched 11.6 MB in 2s (4,835 kB/s) 2022-07-21T01:25:40.012 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 11.6 MB in 2s (4,734 kB/s) 2022-07-21T01:25:40.706 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2022-07-21T01:25:40.730 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-21T01:25:40.743 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2022-07-21T01:25:40.767 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-21T01:25:40.800 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2022-07-21T01:25:40.837 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2022-07-21T01:25:40.976 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2022-07-21T01:25:40.976 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2022-07-21T01:25:41.016 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2022-07-21T01:25:41.017 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2022-07-21T01:25:41.183 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2022-07-21T01:25:41.185 INFO:teuthology.orchestra.run.smithi143.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-21T01:25:41.185 INFO:teuthology.orchestra.run.smithi143.stdout:linux-image-generic set to manually installed. 2022-07-21T01:25:41.185 INFO:teuthology.orchestra.run.smithi143.stdout:0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:25:41.187 DEBUG:teuthology.orchestra.run.smithi143:> dpkg -s linux-image-generic 2022-07-21T01:25:41.206 DEBUG:teuthology.orchestra.run.smithi171:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-21T01:25:41.207 INFO:teuthology.orchestra.run.smithi143.stdout:Package: linux-image-generic 2022-07-21T01:25:41.208 INFO:teuthology.orchestra.run.smithi143.stdout:Status: install ok installed 2022-07-21T01:25:41.208 INFO:teuthology.orchestra.run.smithi143.stdout:Priority: optional 2022-07-21T01:25:41.208 INFO:teuthology.orchestra.run.smithi143.stdout:Section: kernel 2022-07-21T01:25:41.209 INFO:teuthology.orchestra.run.smithi143.stdout:Installed-Size: 19 2022-07-21T01:25:41.209 INFO:teuthology.orchestra.run.smithi143.stdout:Maintainer: Ubuntu Kernel Team 2022-07-21T01:25:41.209 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: amd64 2022-07-21T01:25:41.209 INFO:teuthology.orchestra.run.smithi143.stdout:Source: linux-meta 2022-07-21T01:25:41.210 INFO:teuthology.orchestra.run.smithi143.stdout:Version: 5.4.0.122.123 2022-07-21T01:25:41.210 INFO:teuthology.orchestra.run.smithi143.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-21T01:25:41.210 INFO:teuthology.orchestra.run.smithi143.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-21T01:25:41.210 INFO:teuthology.orchestra.run.smithi143.stdout:Recommends: thermald 2022-07-21T01:25:41.210 INFO:teuthology.orchestra.run.smithi143.stdout:Description: Generic Linux kernel image 2022-07-21T01:25:41.211 INFO:teuthology.orchestra.run.smithi143.stdout: This package will always depend on the latest generic kernel image 2022-07-21T01:25:41.211 INFO:teuthology.orchestra.run.smithi143.stdout: available. 2022-07-21T01:25:41.211 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-21T01:25:41.211 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2022-07-21T01:25:41.212 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-21T01:25:41.220 INFO:teuthology.orchestra.run.smithi174.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-21T01:25:41.220 INFO:teuthology.orchestra.run.smithi174.stdout:linux-image-generic set to manually installed. 2022-07-21T01:25:41.221 INFO:teuthology.orchestra.run.smithi174.stdout:0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:25:41.222 DEBUG:teuthology.orchestra.run.smithi174:> dpkg -s linux-image-generic 2022-07-21T01:25:41.237 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2022-07-21T01:25:41.237 INFO:teuthology.orchestra.run.smithi174.stdout:Package: linux-image-generic 2022-07-21T01:25:41.237 INFO:teuthology.orchestra.run.smithi174.stdout:Status: install ok installed 2022-07-21T01:25:41.238 INFO:teuthology.orchestra.run.smithi174.stdout:Priority: optional 2022-07-21T01:25:41.238 INFO:teuthology.orchestra.run.smithi174.stdout:Section: kernel 2022-07-21T01:25:41.238 INFO:teuthology.orchestra.run.smithi174.stdout:Installed-Size: 19 2022-07-21T01:25:41.239 INFO:teuthology.orchestra.run.smithi174.stdout:Maintainer: Ubuntu Kernel Team 2022-07-21T01:25:41.239 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: amd64 2022-07-21T01:25:41.239 INFO:teuthology.orchestra.run.smithi174.stdout:Source: linux-meta 2022-07-21T01:25:41.239 INFO:teuthology.orchestra.run.smithi174.stdout:Version: 5.4.0.122.123 2022-07-21T01:25:41.240 INFO:teuthology.orchestra.run.smithi174.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-21T01:25:41.240 INFO:teuthology.orchestra.run.smithi174.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-21T01:25:41.240 INFO:teuthology.orchestra.run.smithi174.stdout:Recommends: thermald 2022-07-21T01:25:41.240 INFO:teuthology.orchestra.run.smithi174.stdout:Description: Generic Linux kernel image 2022-07-21T01:25:41.240 INFO:teuthology.orchestra.run.smithi174.stdout: This package will always depend on the latest generic kernel image 2022-07-21T01:25:41.241 INFO:teuthology.orchestra.run.smithi174.stdout: available. 2022-07-21T01:25:41.242 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-21T01:25:41.242 INFO:teuthology.task.kernel:Enabling kdb on client.1... 2022-07-21T01:25:41.242 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-21T01:25:41.263 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2022-07-21T01:25:41.265 DEBUG:teuthology.parallel:result is None 2022-07-21T01:25:41.265 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2022-07-21T01:25:41.281 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2022-07-21T01:25:41.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-21T01:25:41.289 DEBUG:teuthology.parallel:result is None 2022-07-21T01:25:41.361 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2022-07-21T01:25:41.458 INFO:teuthology.orchestra.run.smithi171.stdout:Building dependency tree... 2022-07-21T01:25:41.458 INFO:teuthology.orchestra.run.smithi171.stdout:Reading state information... 2022-07-21T01:25:41.541 INFO:teuthology.orchestra.run.smithi003.stdout:Building dependency tree... 2022-07-21T01:25:41.541 INFO:teuthology.orchestra.run.smithi003.stdout:Reading state information... 2022-07-21T01:25:41.662 INFO:teuthology.orchestra.run.smithi171.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-21T01:25:41.662 INFO:teuthology.orchestra.run.smithi171.stdout:linux-image-generic set to manually installed. 2022-07-21T01:25:41.662 INFO:teuthology.orchestra.run.smithi171.stdout:0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:25:41.664 DEBUG:teuthology.orchestra.run.smithi171:> dpkg -s linux-image-generic 2022-07-21T01:25:41.681 INFO:teuthology.orchestra.run.smithi171.stdout:Package: linux-image-generic 2022-07-21T01:25:41.681 INFO:teuthology.orchestra.run.smithi171.stdout:Status: install ok installed 2022-07-21T01:25:41.681 INFO:teuthology.orchestra.run.smithi171.stdout:Priority: optional 2022-07-21T01:25:41.681 INFO:teuthology.orchestra.run.smithi171.stdout:Section: kernel 2022-07-21T01:25:41.682 INFO:teuthology.orchestra.run.smithi171.stdout:Installed-Size: 19 2022-07-21T01:25:41.682 INFO:teuthology.orchestra.run.smithi171.stdout:Maintainer: Ubuntu Kernel Team 2022-07-21T01:25:41.682 INFO:teuthology.orchestra.run.smithi171.stdout:Architecture: amd64 2022-07-21T01:25:41.682 INFO:teuthology.orchestra.run.smithi171.stdout:Source: linux-meta 2022-07-21T01:25:41.683 INFO:teuthology.orchestra.run.smithi171.stdout:Version: 5.4.0.122.123 2022-07-21T01:25:41.683 INFO:teuthology.orchestra.run.smithi171.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-21T01:25:41.683 INFO:teuthology.orchestra.run.smithi171.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-21T01:25:41.683 INFO:teuthology.orchestra.run.smithi171.stdout:Recommends: thermald 2022-07-21T01:25:41.684 INFO:teuthology.orchestra.run.smithi171.stdout:Description: Generic Linux kernel image 2022-07-21T01:25:41.684 INFO:teuthology.orchestra.run.smithi171.stdout: This package will always depend on the latest generic kernel image 2022-07-21T01:25:41.684 INFO:teuthology.orchestra.run.smithi171.stdout: available. 2022-07-21T01:25:41.685 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-21T01:25:41.685 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2022-07-21T01:25:41.685 DEBUG:teuthology.orchestra.run.smithi171:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-21T01:25:41.716 INFO:teuthology.orchestra.run.smithi171.stdout:ttyS1 2022-07-21T01:25:41.743 DEBUG:teuthology.parallel:result is None 2022-07-21T01:25:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-21T01:25:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:linux-image-generic set to manually installed. 2022-07-21T01:25:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:25:41.764 DEBUG:teuthology.orchestra.run.smithi003:> dpkg -s linux-image-generic 2022-07-21T01:25:41.785 INFO:teuthology.orchestra.run.smithi003.stdout:Package: linux-image-generic 2022-07-21T01:25:41.786 INFO:teuthology.orchestra.run.smithi003.stdout:Status: install ok installed 2022-07-21T01:25:41.786 INFO:teuthology.orchestra.run.smithi003.stdout:Priority: optional 2022-07-21T01:25:41.787 INFO:teuthology.orchestra.run.smithi003.stdout:Section: kernel 2022-07-21T01:25:41.787 INFO:teuthology.orchestra.run.smithi003.stdout:Installed-Size: 19 2022-07-21T01:25:41.788 INFO:teuthology.orchestra.run.smithi003.stdout:Maintainer: Ubuntu Kernel Team 2022-07-21T01:25:41.788 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: amd64 2022-07-21T01:25:41.789 INFO:teuthology.orchestra.run.smithi003.stdout:Source: linux-meta 2022-07-21T01:25:41.789 INFO:teuthology.orchestra.run.smithi003.stdout:Version: 5.4.0.122.123 2022-07-21T01:25:41.789 INFO:teuthology.orchestra.run.smithi003.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-21T01:25:41.790 INFO:teuthology.orchestra.run.smithi003.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-21T01:25:41.790 INFO:teuthology.orchestra.run.smithi003.stdout:Recommends: thermald 2022-07-21T01:25:41.791 INFO:teuthology.orchestra.run.smithi003.stdout:Description: Generic Linux kernel image 2022-07-21T01:25:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: This package will always depend on the latest generic kernel image 2022-07-21T01:25:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: available. 2022-07-21T01:25:41.792 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-21T01:25:41.793 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-07-21T01:25:41.793 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-21T01:25:41.814 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2022-07-21T01:25:41.836 DEBUG:teuthology.parallel:result is None 2022-07-21T01:25:41.836 INFO:teuthology.run_tasks:Running task internal.base... 2022-07-21T01:25:41.847 INFO:teuthology.task.internal:Creating test directory... 2022-07-21T01:25:41.848 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-21T01:25:41.879 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-21T01:25:41.882 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-21T01:25:41.884 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-21T01:25:41.891 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-07-21T01:25:41.898 INFO:teuthology.run_tasks:Running task internal.archive... 2022-07-21T01:25:41.906 INFO:teuthology.task.internal:Creating archive directory... 2022-07-21T01:25:41.907 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-21T01:25:41.909 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-21T01:25:41.932 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-21T01:25:41.934 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-21T01:25:41.947 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-07-21T01:25:41.954 INFO:teuthology.task.internal:Enabling coredump saving... 2022-07-21T01:25:41.954 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-21T01:25:41.957 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-21T01:25:41.959 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-21T01:25:41.961 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-21T01:25:41.980 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T01:25:41.981 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T01:25:41.983 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T01:25:41.987 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T01:25:41.990 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T01:25:41.995 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T01:25:41.998 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T01:25:42.000 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T01:25:42.001 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-07-21T01:25:42.014 INFO:teuthology.task.internal:Configuring sudo... 2022-07-21T01:25:42.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-21T01:25:42.039 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-21T01:25:42.044 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-21T01:25:42.047 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-21T01:25:42.063 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-07-21T01:25:42.072 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-07-21T01:25:42.073 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-21T01:25:42.075 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-21T01:25:42.077 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-21T01:25:42.080 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-21T01:25:42.113 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-21T01:25:42.128 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-21T01:25:42.135 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-07-21T01:25:42.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-21T01:25:42.154 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-21T01:25:42.162 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-21T01:25:42.169 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2022-07-21T01:25:42.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-21T01:25:42.202 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-21T01:25:42.210 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-21T01:25:42.217 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-07-21T01:25:42.218 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-21T01:25:42.240 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-21T01:25:42.248 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-21T01:25:42.255 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2022-07-21T01:25:42.255 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-21T01:25:42.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2022-07-21T01:25:42.322 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2022-07-21T01:25:42.325 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2022-07-21T01:25:42.328 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2022-07-21T01:25:42.386 INFO:teuthology.run_tasks:Running task internal.timer... 2022-07-21T01:25:42.394 INFO:teuthology.task.internal:Starting timer... 2022-07-21T01:25:42.395 INFO:teuthology.run_tasks:Running task pcp... 2022-07-21T01:25:42.405 INFO:teuthology.run_tasks:Running task selinux... 2022-07-21T01:25:42.413 DEBUG:teuthology.task.selinux:Excluding smithi003: OS 'ubuntu' does not support SELinux 2022-07-21T01:25:42.414 DEBUG:teuthology.task.selinux:Excluding smithi143: OS 'ubuntu' does not support SELinux 2022-07-21T01:25:42.414 DEBUG:teuthology.task.selinux:Excluding smithi171: OS 'ubuntu' does not support SELinux 2022-07-21T01:25:42.415 DEBUG:teuthology.task.selinux:Excluding smithi174: OS 'ubuntu' does not support SELinux 2022-07-21T01:25:42.415 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-07-21T01:25:42.415 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-07-21T01:25:42.415 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-07-21T01:25:42.416 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-07-21T01:25:42.425 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-07-21T01:25:42.432 INFO:teuthology.repo_utils:Fetching main from origin 2022-07-21T01:25:42.467 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-07-21T01:25:42.493 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-07-21T01:25:42.496 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi143.front.sepia.ceph.com,smithi171.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-07-21T01:30:17.548 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com'), Remote(name='ubuntu@smithi171.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2022-07-21T01:30:17.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2022-07-21T01:30:17.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T01:30:17.673 DEBUG:teuthology.orchestra.run.smithi003:> true 2022-07-21T01:30:17.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2022-07-21T01:30:17.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T01:30:17.889 DEBUG:teuthology.orchestra.run.smithi143:> true 2022-07-21T01:30:18.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2022-07-21T01:30:18.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T01:30:18.108 DEBUG:teuthology.orchestra.run.smithi171:> true 2022-07-21T01:30:18.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2022-07-21T01:30:18.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T01:30:18.330 DEBUG:teuthology.orchestra.run.smithi174:> true 2022-07-21T01:30:18.452 INFO:teuthology.run_tasks:Running task clock... 2022-07-21T01:30:18.464 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-07-21T01:30:18.465 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-21T01:30:18.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-21T01:30:18.470 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-21T01:30:18.470 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-21T01:30:18.473 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-21T01:30:18.473 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-21T01:30:18.477 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-21T01:30:18.478 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-21T01:30:18.505 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-21T01:30:18.505 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: Command line: ntpd -gq 2022-07-21T01:30:18.507 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: proto: precision = 0.126 usec (-23) 2022-07-21T01:30:18.507 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-21T01:30:18.508 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: Command line: ntpd -gq 2022-07-21T01:30:18.508 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: proto: precision = 0.120 usec (-23) 2022-07-21T01:30:18.509 INFO:teuthology.orchestra.run.smithi003.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-21T01:30:18.509 INFO:teuthology.orchestra.run.smithi143.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-21T01:30:18.510 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-21T01:30:18.511 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-21T01:30:18.511 INFO:teuthology.orchestra.run.smithi003.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-21T01:30:18.511 INFO:teuthology.orchestra.run.smithi143.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-21T01:30:18.514 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: restrict ::: KOD does nothing without LIMITED. 2022-07-21T01:30:18.514 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: Listen and drop on 0 v6wildcard [::]:123 2022-07-21T01:30:18.515 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-21T01:30:18.515 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: Listen normally on 2 lo 127.0.0.1:123 2022-07-21T01:30:18.515 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: Listen normally on 3 ens1f0 172.21.15.3:123 2022-07-21T01:30:18.516 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: Listen normally on 4 lo [::1]:123 2022-07-21T01:30:18.516 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15dc%4]:123 2022-07-21T01:30:18.516 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:18 ntpd[17801]: Listening on routing socket on fd #22 for interface updates 2022-07-21T01:30:18.517 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-21T01:30:18.517 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: Command line: ntpd -gq 2022-07-21T01:30:18.517 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: proto: precision = 0.120 usec (-23) 2022-07-21T01:30:18.518 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: restrict ::: KOD does nothing without LIMITED. 2022-07-21T01:30:18.518 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: Listen and drop on 0 v6wildcard [::]:123 2022-07-21T01:30:18.518 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-21T01:30:18.518 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: Listen normally on 2 lo 127.0.0.1:123 2022-07-21T01:30:18.519 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: Listen normally on 3 enp3s0f1 172.21.15.143:123 2022-07-21T01:30:18.519 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: Listen normally on 4 lo [::1]:123 2022-07-21T01:30:18.519 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7783%5]:123 2022-07-21T01:30:18.519 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:18 ntpd[18049]: Listening on routing socket on fd #22 for interface updates 2022-07-21T01:30:18.520 INFO:teuthology.orchestra.run.smithi171.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-21T01:30:18.520 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-21T01:30:18.521 INFO:teuthology.orchestra.run.smithi171.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-21T01:30:18.522 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: restrict ::: KOD does nothing without LIMITED. 2022-07-21T01:30:18.522 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: Listen and drop on 0 v6wildcard [::]:123 2022-07-21T01:30:18.523 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-21T01:30:18.523 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: Listen normally on 2 lo 127.0.0.1:123 2022-07-21T01:30:18.523 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: Listen normally on 3 enp3s0f1 172.21.15.171:123 2022-07-21T01:30:18.523 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: Listen normally on 4 lo [::1]:123 2022-07-21T01:30:18.524 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f6d%5]:123 2022-07-21T01:30:18.524 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:18 ntpd[18033]: Listening on routing socket on fd #22 for interface updates 2022-07-21T01:30:18.530 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-21T01:30:18.530 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: Command line: ntpd -gq 2022-07-21T01:30:18.531 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: proto: precision = 0.113 usec (-23) 2022-07-21T01:30:18.532 INFO:teuthology.orchestra.run.smithi174.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-21T01:30:18.532 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-21T01:30:18.533 INFO:teuthology.orchestra.run.smithi174.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-21T01:30:18.535 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: restrict ::: KOD does nothing without LIMITED. 2022-07-21T01:30:18.535 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: Listen and drop on 0 v6wildcard [::]:123 2022-07-21T01:30:18.535 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-21T01:30:18.536 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: Listen normally on 2 lo 127.0.0.1:123 2022-07-21T01:30:18.536 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: Listen normally on 3 enp3s0f1 172.21.15.174:123 2022-07-21T01:30:18.536 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: Listen normally on 4 lo [::1]:123 2022-07-21T01:30:18.536 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:771b%5]:123 2022-07-21T01:30:18.537 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:18 ntpd[18106]: Listening on routing socket on fd #22 for interface updates 2022-07-21T01:30:19.509 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:19.510 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:19.510 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:19.511 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:19.511 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:19.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:19.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:19.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:19.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:19.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:19.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:19.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:19.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:19.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:19 ntpd[18049]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:19.514 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:19 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:19.515 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:19 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:19.515 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:19 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:19.515 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:19 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:19.516 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:19 ntpd[18033]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:19.516 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:19 ntpd[18033]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:20.506 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:20 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:20.506 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:20 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:20.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:20 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:20.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:20 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:20.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:20 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:20.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:20 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:20.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:20 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:20.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:20 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:20.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:20 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:20.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:20 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:20.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:20 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:20.535 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:20 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:21.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:21 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:21 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:21 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:21 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:21 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:21 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:21 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:21.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:21 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:21.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.514 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:21.514 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:21.514 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.514 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.515 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:21.515 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:21.515 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.515 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.515 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:21.516 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:21 ntpd[18049]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:21.517 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:21.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:21.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.521 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:21.521 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:21.521 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.521 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.522 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:21.522 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:21 ntpd[18033]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:21.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:21.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:21.535 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:21.535 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:21.535 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:21.535 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:21 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:22.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:22.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:22.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:22.510 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:22.510 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:22.511 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:22.511 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:22.511 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:22.512 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:22.512 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:22.513 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:22.513 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:22 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:22.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:22 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:22.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:22 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:22.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:22 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:22.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:22 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:22.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:22 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:22.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:22 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:22.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:22 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:22.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:22 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:23.506 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:23 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.506 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:23 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:23 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:23 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:23 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:23.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:23 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:23.510 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.510 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.511 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.511 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.511 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:23.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:23.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:23.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:23.514 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.514 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.514 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:23.514 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:23 ntpd[18049]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:23.517 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.517 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.517 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.517 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:23.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:23.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:23.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:23.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:23.521 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:23 ntpd[18033]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:23.531 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:23.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:23.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:23.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:23.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:23.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:23 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:24.506 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:24.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:24.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:24.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:24.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:24.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:24.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:24.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:24.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:24.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:24.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:24.510 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:24 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:24.531 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:24 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:24.531 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:24 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:24.531 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:24 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:24.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:24 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:24.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:24 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:24.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:24 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:25.506 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:25 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:25 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:25 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:25 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:25 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:25.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:25 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:25.511 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.512 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.513 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:25.514 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:25.514 INFO:teuthology.orchestra.run.smithi143.stdout:21 Jul 01:30:25 ntpd[18049]: ntpd: time slew -0.001872 s 2022-07-21T01:30:25.514 INFO:teuthology.orchestra.run.smithi143.stdout:ntpd: time slew -0.001872s 2022-07-21T01:30:25.515 INFO:teuthology.orchestra.run.smithi143.stderr:21 Jul 01:30:25 ntpd[18049]: can't open /var/log/ntpstats/loopstats.20220721: Permission denied 2022-07-21T01:30:25.517 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.517 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.518 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:25.519 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:25.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:25.520 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:25.521 INFO:teuthology.orchestra.run.smithi171.stdout:21 Jul 01:30:25 ntpd[18033]: ntpd: time slew -0.001891 s 2022-07-21T01:30:25.521 INFO:teuthology.orchestra.run.smithi171.stdout:ntpd: time slew -0.001891s 2022-07-21T01:30:25.521 INFO:teuthology.orchestra.run.smithi171.stderr:21 Jul 01:30:25 ntpd[18033]: can't open /var/log/ntpstats/loopstats.20220721: Permission denied 2022-07-21T01:30:25.531 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:25.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:25.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:25.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:25.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:25.534 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:25 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:25.568 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2022-07-21T01:30:25.569 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2022-07-21T01:30:25.569 INFO:teuthology.orchestra.run.smithi143.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:25.569 INFO:teuthology.orchestra.run.smithi143.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:25.569 INFO:teuthology.orchestra.run.smithi143.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:25.570 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:25.577 INFO:teuthology.orchestra.run.smithi171.stdout: remote refid st t when poll reach delay offset jitter 2022-07-21T01:30:25.578 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================== 2022-07-21T01:30:25.578 INFO:teuthology.orchestra.run.smithi171.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:25.578 INFO:teuthology.orchestra.run.smithi171.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:25.578 INFO:teuthology.orchestra.run.smithi171.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:25.579 INFO:teuthology.orchestra.run.smithi171.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:26.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:26.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:26.507 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:26.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:26.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:26.508 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:26.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:26.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:26.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:26.509 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:26.510 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:26.510 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:26.511 INFO:teuthology.orchestra.run.smithi003.stdout:21 Jul 01:30:26 ntpd[17801]: ntpd: time slew -0.002007 s 2022-07-21T01:30:26.511 INFO:teuthology.orchestra.run.smithi003.stdout:ntpd: time slew -0.002007s 2022-07-21T01:30:26.511 INFO:teuthology.orchestra.run.smithi003.stderr:21 Jul 01:30:26 ntpd[17801]: can't open /var/log/ntpstats/loopstats.20220721: Permission denied 2022-07-21T01:30:26.531 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:26 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:26.531 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:26 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:26.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:26 ntpd[18106]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-21T01:30:26.532 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:26 ntpd[18106]: can't open /var/log/ntpstats/rawstats.20220721: Permission denied 2022-07-21T01:30:26.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:26 ntpd[18106]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-21T01:30:26.533 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:26 ntpd[18106]: can't open /var/log/ntpstats/peerstats.20220721: Permission denied 2022-07-21T01:30:26.534 INFO:teuthology.orchestra.run.smithi174.stdout:21 Jul 01:30:26 ntpd[18106]: ntpd: time slew +0.003853 s 2022-07-21T01:30:26.535 INFO:teuthology.orchestra.run.smithi174.stdout:ntpd: time slew +0.003853s 2022-07-21T01:30:26.536 INFO:teuthology.orchestra.run.smithi174.stderr:21 Jul 01:30:26 ntpd[18106]: can't open /var/log/ntpstats/loopstats.20220721: Permission denied 2022-07-21T01:30:26.564 INFO:teuthology.orchestra.run.smithi003.stdout: remote refid st t when poll reach delay offset jitter 2022-07-21T01:30:26.564 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================== 2022-07-21T01:30:26.565 INFO:teuthology.orchestra.run.smithi003.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:26.565 INFO:teuthology.orchestra.run.smithi003.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:26.565 INFO:teuthology.orchestra.run.smithi003.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:26.565 INFO:teuthology.orchestra.run.smithi003.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:26.598 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2022-07-21T01:30:26.598 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2022-07-21T01:30:26.599 INFO:teuthology.orchestra.run.smithi174.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:26.599 INFO:teuthology.orchestra.run.smithi174.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:26.599 INFO:teuthology.orchestra.run.smithi174.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:26.599 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-21T01:30:26.600 INFO:teuthology.run_tasks:Running task install... 2022-07-21T01:30:26.610 DEBUG:teuthology.task.install:project ceph 2022-07-21T01:30:26.611 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}} 2022-07-21T01:30:26.611 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'} 2022-07-21T01:30:26.612 INFO:teuthology.task.install:Using flavor: default 2022-07-21T01:30:26.618 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-07-21T01:30:26.619 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2022-07-21T01:30:26.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo lsb_release -is 2022-07-21T01:30:26.669 INFO:teuthology.orchestra.run.smithi003.stdout:Ubuntu 2022-07-21T01:30:26.674 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:30:26.675 DEBUG:teuthology.orchestra.run.smithi143:> sudo lsb_release -is 2022-07-21T01:30:26.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-key list | grep Ceph 2022-07-21T01:30:26.726 INFO:teuthology.orchestra.run.smithi143.stdout:Ubuntu 2022-07-21T01:30:26.731 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:30:26.732 DEBUG:teuthology.orchestra.run.smithi171:> sudo lsb_release -is 2022-07-21T01:30:26.733 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-key list | grep Ceph 2022-07-21T01:30:26.770 INFO:teuthology.orchestra.run.smithi003.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-21T01:30:26.783 INFO:teuthology.orchestra.run.smithi171.stdout:Ubuntu 2022-07-21T01:30:26.788 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:30:26.789 DEBUG:teuthology.orchestra.run.smithi174:> sudo lsb_release -is 2022-07-21T01:30:26.790 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-key list | grep Ceph 2022-07-21T01:30:26.830 INFO:teuthology.orchestra.run.smithi003.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-21T01:30:26.830 INFO:teuthology.orchestra.run.smithi003.stdout:uid [ unknown] Ceph.com (release key) 2022-07-21T01:30:26.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo lsb_release -is 2022-07-21T01:30:26.832 INFO:teuthology.orchestra.run.smithi143.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-21T01:30:26.839 INFO:teuthology.orchestra.run.smithi174.stdout:Ubuntu 2022-07-21T01:30:26.845 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:30:26.845 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-key list | grep Ceph 2022-07-21T01:30:26.880 INFO:teuthology.orchestra.run.smithi003.stdout:Ubuntu 2022-07-21T01:30:26.885 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:30:26.886 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-21T01:30:26.886 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-21T01:30:26.887 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-21T01:30:26.893 INFO:teuthology.orchestra.run.smithi171.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-21T01:30:26.894 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-21T01:30:26.894 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph.com (release key) 2022-07-21T01:30:26.894 DEBUG:teuthology.orchestra.run.smithi143:> sudo lsb_release -is 2022-07-21T01:30:26.940 INFO:teuthology.orchestra.run.smithi143.stdout:Ubuntu 2022-07-21T01:30:26.952 INFO:teuthology.orchestra.run.smithi174.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-21T01:30:26.953 INFO:teuthology.orchestra.run.smithi171.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-21T01:30:26.953 INFO:teuthology.orchestra.run.smithi171.stdout:uid [ unknown] Ceph.com (release key) 2022-07-21T01:30:26.954 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:30:26.954 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-21T01:30:26.955 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-21T01:30:26.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-21T01:30:26.957 DEBUG:teuthology.orchestra.run.smithi171:> sudo lsb_release -is 2022-07-21T01:30:27.003 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-21T01:30:27.003 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph.com (release key) 2022-07-21T01:30:27.004 DEBUG:teuthology.orchestra.run.smithi174:> sudo lsb_release -is 2022-07-21T01:30:27.008 INFO:teuthology.orchestra.run.smithi171.stdout:Ubuntu 2022-07-21T01:30:27.013 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:30:27.013 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-21T01:30:27.014 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-21T01:30:27.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-21T01:30:27.049 INFO:teuthology.orchestra.run.smithi174.stdout:Ubuntu 2022-07-21T01:30:27.055 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:30:27.055 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-21T01:30:27.056 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-21T01:30:27.056 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-21T01:30:27.198 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-21T01:30:27.199 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-21T01:30:27.240 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-21T01:30:27.241 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-21T01:30:27.286 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-21T01:30:27.286 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-21T01:30:27.360 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-21T01:30:27.361 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-21T01:30:27.412 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-07-21T01:30:27.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-21T01:30:27.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get update 2022-07-21T01:30:27.454 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2022-07-21T01:30:27.455 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-21T01:30:27.472 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2022-07-21T01:30:27.503 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-07-21T01:30:27.504 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-21T01:30:27.521 DEBUG:teuthology.orchestra.run.smithi171:> sudo apt-get update 2022-07-21T01:30:27.577 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2022-07-21T01:30:27.577 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-21T01:30:27.594 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2022-07-21T01:30:27.626 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-21T01:30:27.626 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-21T01:30:27.657 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-21T01:30:27.699 INFO:teuthology.orchestra.run.smithi003.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-21T01:30:27.701 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-21T01:30:27.725 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-21T01:30:27.727 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-21T01:30:27.729 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T01:30:27.733 INFO:teuthology.orchestra.run.smithi171.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-21T01:30:27.736 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-21T01:30:27.739 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T01:30:27.740 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T01:30:27.750 INFO:teuthology.orchestra.run.smithi003.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-21T01:30:27.751 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-21T01:30:27.753 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-21T01:30:27.757 INFO:teuthology.orchestra.run.smithi171.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-21T01:30:27.773 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-21T01:30:27.774 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T01:30:27.775 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-21T01:30:27.785 INFO:teuthology.orchestra.run.smithi171.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-21T01:30:27.796 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-21T01:30:27.811 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-21T01:30:27.814 INFO:teuthology.orchestra.run.smithi171.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-21T01:30:27.816 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-21T01:30:27.819 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-21T01:30:27.820 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-21T01:30:27.821 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-21T01:30:27.846 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-21T01:30:27.848 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-21T01:30:27.850 INFO:teuthology.orchestra.run.smithi171.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-21T01:30:27.853 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-21T01:30:27.875 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-21T01:30:27.899 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-21T01:30:27.901 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-21T01:30:27.910 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-21T01:30:27.945 INFO:teuthology.orchestra.run.smithi171.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-21T01:30:28.273 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 26.6 kB in 1s (41.5 kB/s) 2022-07-21T01:30:28.299 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 26.6 kB in 1s (37.8 kB/s) 2022-07-21T01:30:28.318 INFO:teuthology.orchestra.run.smithi171.stdout:Fetched 26.6 kB in 1s (41.8 kB/s) 2022-07-21T01:30:28.384 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 26.6 kB in 1s (42.2 kB/s) 2022-07-21T01:30:29.160 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2022-07-21T01:30:29.184 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-21T01:30:29.203 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2022-07-21T01:30:29.209 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2022-07-21T01:30:29.226 DEBUG:teuthology.orchestra.run.smithi171:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-21T01:30:29.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-21T01:30:29.254 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2022-07-21T01:30:29.265 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2022-07-21T01:30:29.288 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-21T01:30:29.291 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2022-07-21T01:30:29.301 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2022-07-21T01:30:29.355 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2022-07-21T01:30:29.417 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2022-07-21T01:30:29.418 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2022-07-21T01:30:29.468 INFO:teuthology.orchestra.run.smithi171.stdout:Building dependency tree... 2022-07-21T01:30:29.469 INFO:teuthology.orchestra.run.smithi171.stdout:Reading state information... 2022-07-21T01:30:29.482 INFO:teuthology.orchestra.run.smithi003.stdout:Building dependency tree... 2022-07-21T01:30:29.483 INFO:teuthology.orchestra.run.smithi003.stdout:Reading state information... 2022-07-21T01:30:29.532 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2022-07-21T01:30:29.532 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2022-07-21T01:30:29.559 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages were automatically installed and are no longer required: 2022-07-21T01:30:29.560 INFO:teuthology.orchestra.run.smithi143.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-21T01:30:29.560 INFO:teuthology.orchestra.run.smithi143.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-21T01:30:29.560 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2022-07-21T01:30:29.561 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-21T01:30:29.561 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-21T01:30:29.562 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-21T01:30:29.563 INFO:teuthology.orchestra.run.smithi143.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-21T01:30:29.563 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-21T01:30:29.563 INFO:teuthology.orchestra.run.smithi143.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-21T01:30:29.563 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-21T01:30:29.564 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-21T01:30:29.564 INFO:teuthology.orchestra.run.smithi143.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-21T01:30:29.564 INFO:teuthology.orchestra.run.smithi143.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-21T01:30:29.564 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-21T01:30:29.565 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-21T01:30:29.565 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-21T01:30:29.565 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-21T01:30:29.565 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-21T01:30:29.566 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-21T01:30:29.566 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-21T01:30:29.566 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-21T01:30:29.567 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-21T01:30:29.567 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet 2022-07-21T01:30:29.567 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2022-07-21T01:30:29.567 INFO:teuthology.orchestra.run.smithi143.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-21T01:30:29.567 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-21T01:30:29.568 INFO:teuthology.orchestra.run.smithi143.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-21T01:30:29.568 INFO:teuthology.orchestra.run.smithi143.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-21T01:30:29.568 INFO:teuthology.orchestra.run.smithi143.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-21T01:30:29.569 INFO:teuthology.orchestra.run.smithi143.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-21T01:30:29.569 INFO:teuthology.orchestra.run.smithi143.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-21T01:30:29.569 INFO:teuthology.orchestra.run.smithi143.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-21T01:30:29.569 INFO:teuthology.orchestra.run.smithi143.stdout:Recommended packages: 2022-07-21T01:30:29.570 INFO:teuthology.orchestra.run.smithi143.stdout: btrfs-tools 2022-07-21T01:30:29.626 INFO:teuthology.orchestra.run.smithi171.stdout:The following packages were automatically installed and are no longer required: 2022-07-21T01:30:29.626 INFO:teuthology.orchestra.run.smithi171.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-21T01:30:29.627 INFO:teuthology.orchestra.run.smithi171.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-21T01:30:29.627 INFO:teuthology.orchestra.run.smithi171.stdout:The following additional packages will be installed: 2022-07-21T01:30:29.627 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-21T01:30:29.627 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-21T01:30:29.629 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-21T01:30:29.630 INFO:teuthology.orchestra.run.smithi171.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-21T01:30:29.630 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-21T01:30:29.630 INFO:teuthology.orchestra.run.smithi171.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-21T01:30:29.630 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-21T01:30:29.631 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-21T01:30:29.631 INFO:teuthology.orchestra.run.smithi171.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-21T01:30:29.631 INFO:teuthology.orchestra.run.smithi171.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-21T01:30:29.631 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-21T01:30:29.632 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-21T01:30:29.632 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-21T01:30:29.632 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-21T01:30:29.632 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-21T01:30:29.632 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-21T01:30:29.633 INFO:teuthology.orchestra.run.smithi171.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-21T01:30:29.633 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-21T01:30:29.633 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-21T01:30:29.633 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet 2022-07-21T01:30:29.634 INFO:teuthology.orchestra.run.smithi171.stdout:Suggested packages: 2022-07-21T01:30:29.634 INFO:teuthology.orchestra.run.smithi171.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-21T01:30:29.634 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-21T01:30:29.634 INFO:teuthology.orchestra.run.smithi171.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-21T01:30:29.634 INFO:teuthology.orchestra.run.smithi171.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-21T01:30:29.635 INFO:teuthology.orchestra.run.smithi171.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-21T01:30:29.635 INFO:teuthology.orchestra.run.smithi171.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-21T01:30:29.635 INFO:teuthology.orchestra.run.smithi171.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-21T01:30:29.635 INFO:teuthology.orchestra.run.smithi171.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-21T01:30:29.636 INFO:teuthology.orchestra.run.smithi171.stdout:Recommended packages: 2022-07-21T01:30:29.636 INFO:teuthology.orchestra.run.smithi171.stdout: btrfs-tools 2022-07-21T01:30:29.647 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages were automatically installed and are no longer required: 2022-07-21T01:30:29.648 INFO:teuthology.orchestra.run.smithi003.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-21T01:30:29.649 INFO:teuthology.orchestra.run.smithi003.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-21T01:30:29.649 INFO:teuthology.orchestra.run.smithi003.stdout:The following additional packages will be installed: 2022-07-21T01:30:29.649 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-21T01:30:29.649 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-21T01:30:29.650 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-21T01:30:29.650 INFO:teuthology.orchestra.run.smithi003.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-21T01:30:29.651 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-21T01:30:29.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-21T01:30:29.651 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-21T01:30:29.652 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-21T01:30:29.652 INFO:teuthology.orchestra.run.smithi003.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-21T01:30:29.652 INFO:teuthology.orchestra.run.smithi003.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-21T01:30:29.652 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-21T01:30:29.653 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-21T01:30:29.653 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-21T01:30:29.653 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-21T01:30:29.653 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-21T01:30:29.653 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-21T01:30:29.654 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-21T01:30:29.654 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-21T01:30:29.654 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-21T01:30:29.654 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet 2022-07-21T01:30:29.655 INFO:teuthology.orchestra.run.smithi003.stdout:Suggested packages: 2022-07-21T01:30:29.655 INFO:teuthology.orchestra.run.smithi003.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-21T01:30:29.655 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-21T01:30:29.655 INFO:teuthology.orchestra.run.smithi003.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-21T01:30:29.656 INFO:teuthology.orchestra.run.smithi003.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-21T01:30:29.656 INFO:teuthology.orchestra.run.smithi003.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-21T01:30:29.656 INFO:teuthology.orchestra.run.smithi003.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-21T01:30:29.656 INFO:teuthology.orchestra.run.smithi003.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-21T01:30:29.656 INFO:teuthology.orchestra.run.smithi003.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-21T01:30:29.657 INFO:teuthology.orchestra.run.smithi003.stdout:Recommended packages: 2022-07-21T01:30:29.657 INFO:teuthology.orchestra.run.smithi003.stdout: btrfs-tools 2022-07-21T01:30:29.670 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2022-07-21T01:30:29.671 INFO:teuthology.orchestra.run.smithi143.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-21T01:30:29.671 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-21T01:30:29.671 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-21T01:30:29.672 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-21T01:30:29.672 INFO:teuthology.orchestra.run.smithi143.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-21T01:30:29.673 INFO:teuthology.orchestra.run.smithi143.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-21T01:30:29.673 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-21T01:30:29.673 INFO:teuthology.orchestra.run.smithi143.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-21T01:30:29.673 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-21T01:30:29.674 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-21T01:30:29.674 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-21T01:30:29.674 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-21T01:30:29.674 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-21T01:30:29.675 INFO:teuthology.orchestra.run.smithi143.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-21T01:30:29.675 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-21T01:30:29.675 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-21T01:30:29.675 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-21T01:30:29.675 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-21T01:30:29.676 INFO:teuthology.orchestra.run.smithi143.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-21T01:30:29.676 INFO:teuthology.orchestra.run.smithi143.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-21T01:30:29.676 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-21T01:30:29.676 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-21T01:30:29.678 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages will be upgraded: 2022-07-21T01:30:29.678 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 librbd1 2022-07-21T01:30:29.680 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2022-07-21T01:30:29.680 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-21T01:30:29.680 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-21T01:30:29.681 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2022-07-21T01:30:29.681 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-21T01:30:29.681 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-21T01:30:29.681 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-21T01:30:29.681 INFO:teuthology.orchestra.run.smithi174.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-21T01:30:29.682 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-21T01:30:29.682 INFO:teuthology.orchestra.run.smithi174.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-21T01:30:29.682 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-21T01:30:29.683 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-21T01:30:29.683 INFO:teuthology.orchestra.run.smithi174.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-21T01:30:29.683 INFO:teuthology.orchestra.run.smithi174.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-21T01:30:29.683 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-21T01:30:29.683 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-21T01:30:29.684 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-21T01:30:29.684 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-21T01:30:29.684 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-21T01:30:29.684 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-21T01:30:29.685 INFO:teuthology.orchestra.run.smithi174.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-21T01:30:29.685 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-21T01:30:29.685 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-21T01:30:29.685 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet 2022-07-21T01:30:29.686 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2022-07-21T01:30:29.686 INFO:teuthology.orchestra.run.smithi174.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-21T01:30:29.686 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-21T01:30:29.686 INFO:teuthology.orchestra.run.smithi174.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-21T01:30:29.686 INFO:teuthology.orchestra.run.smithi174.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-21T01:30:29.687 INFO:teuthology.orchestra.run.smithi174.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-21T01:30:29.687 INFO:teuthology.orchestra.run.smithi174.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-21T01:30:29.687 INFO:teuthology.orchestra.run.smithi174.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-21T01:30:29.687 INFO:teuthology.orchestra.run.smithi174.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-21T01:30:29.688 INFO:teuthology.orchestra.run.smithi174.stdout:Recommended packages: 2022-07-21T01:30:29.688 INFO:teuthology.orchestra.run.smithi174.stdout: btrfs-tools 2022-07-21T01:30:29.734 INFO:teuthology.orchestra.run.smithi143.stdout:2 upgraded, 101 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:30:29.735 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 153 MB of archives. 2022-07-21T01:30:29.735 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-21T01:30:29.735 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-21T01:30:29.737 INFO:teuthology.orchestra.run.smithi171.stdout:The following NEW packages will be installed: 2022-07-21T01:30:29.737 INFO:teuthology.orchestra.run.smithi171.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-21T01:30:29.738 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-21T01:30:29.738 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-21T01:30:29.740 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-21T01:30:29.741 INFO:teuthology.orchestra.run.smithi171.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-21T01:30:29.741 INFO:teuthology.orchestra.run.smithi171.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-21T01:30:29.741 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-21T01:30:29.741 INFO:teuthology.orchestra.run.smithi171.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-21T01:30:29.742 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-21T01:30:29.742 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-21T01:30:29.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-21T01:30:29.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-21T01:30:29.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-21T01:30:29.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-21T01:30:29.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-21T01:30:29.744 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-21T01:30:29.744 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-21T01:30:29.744 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-21T01:30:29.744 INFO:teuthology.orchestra.run.smithi171.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-21T01:30:29.745 INFO:teuthology.orchestra.run.smithi171.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-21T01:30:29.745 INFO:teuthology.orchestra.run.smithi171.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-21T01:30:29.745 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-21T01:30:29.745 INFO:teuthology.orchestra.run.smithi171.stdout:The following packages will be upgraded: 2022-07-21T01:30:29.746 INFO:teuthology.orchestra.run.smithi171.stdout: librados2 librbd1 2022-07-21T01:30:29.753 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-21T01:30:29.760 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-21T01:30:29.765 INFO:teuthology.orchestra.run.smithi003.stdout:The following NEW packages will be installed: 2022-07-21T01:30:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-21T01:30:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-21T01:30:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-21T01:30:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-21T01:30:29.767 INFO:teuthology.orchestra.run.smithi003.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-21T01:30:29.767 INFO:teuthology.orchestra.run.smithi003.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-21T01:30:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-21T01:30:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-21T01:30:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-21T01:30:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-21T01:30:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-21T01:30:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-21T01:30:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-21T01:30:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-21T01:30:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-21T01:30:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-21T01:30:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-21T01:30:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-21T01:30:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-21T01:30:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-21T01:30:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-21T01:30:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-21T01:30:29.776 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages will be upgraded: 2022-07-21T01:30:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 librbd1 2022-07-21T01:30:29.783 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2022-07-21T01:30:29.784 INFO:teuthology.orchestra.run.smithi174.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-21T01:30:29.784 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-21T01:30:29.784 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-21T01:30:29.786 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-21T01:30:29.786 INFO:teuthology.orchestra.run.smithi174.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-21T01:30:29.787 INFO:teuthology.orchestra.run.smithi174.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-21T01:30:29.787 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-21T01:30:29.787 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-21T01:30:29.787 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-21T01:30:29.788 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-21T01:30:29.788 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-21T01:30:29.788 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-21T01:30:29.788 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-21T01:30:29.789 INFO:teuthology.orchestra.run.smithi174.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-21T01:30:29.789 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-21T01:30:29.789 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-21T01:30:29.789 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-21T01:30:29.790 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-21T01:30:29.790 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-21T01:30:29.790 INFO:teuthology.orchestra.run.smithi174.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-21T01:30:29.790 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-21T01:30:29.791 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-21T01:30:29.791 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages will be upgraded: 2022-07-21T01:30:29.791 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 librbd1 2022-07-21T01:30:29.792 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-21T01:30:29.802 INFO:teuthology.orchestra.run.smithi171.stdout:2 upgraded, 101 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:30:29.803 INFO:teuthology.orchestra.run.smithi171.stdout:Need to get 153 MB of archives. 2022-07-21T01:30:29.803 INFO:teuthology.orchestra.run.smithi171.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-21T01:30:29.803 INFO:teuthology.orchestra.run.smithi171.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-21T01:30:29.817 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-21T01:30:29.819 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-21T01:30:29.821 INFO:teuthology.orchestra.run.smithi171.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-21T01:30:29.822 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-21T01:30:29.829 INFO:teuthology.orchestra.run.smithi003.stdout:2 upgraded, 101 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:30:29.829 INFO:teuthology.orchestra.run.smithi003.stdout:Need to get 153 MB of archives. 2022-07-21T01:30:29.829 INFO:teuthology.orchestra.run.smithi003.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-21T01:30:29.829 INFO:teuthology.orchestra.run.smithi003.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-21T01:30:29.830 INFO:teuthology.orchestra.run.smithi171.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-21T01:30:29.844 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-21T01:30:29.847 INFO:teuthology.orchestra.run.smithi003.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-21T01:30:29.851 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-21T01:30:29.854 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-21T01:30:29.855 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-21T01:30:29.855 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-21T01:30:29.857 INFO:teuthology.orchestra.run.smithi171.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-21T01:30:29.871 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-21T01:30:29.872 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-21T01:30:29.873 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-21T01:30:29.880 INFO:teuthology.orchestra.run.smithi003.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-21T01:30:29.884 INFO:teuthology.orchestra.run.smithi171.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-21T01:30:29.886 INFO:teuthology.orchestra.run.smithi171.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-21T01:30:29.887 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-21T01:30:29.887 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-21T01:30:29.888 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-21T01:30:29.888 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-21T01:30:29.889 INFO:teuthology.orchestra.run.smithi171.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-21T01:30:29.890 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-21T01:30:29.890 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-21T01:30:29.902 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-21T01:30:29.903 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-21T01:30:29.904 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-21T01:30:29.909 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-21T01:30:29.911 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-21T01:30:29.914 INFO:teuthology.orchestra.run.smithi171.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-21T01:30:29.915 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-21T01:30:29.919 INFO:teuthology.orchestra.run.smithi171.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-21T01:30:29.919 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-21T01:30:29.920 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-21T01:30:29.921 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-21T01:30:29.922 INFO:teuthology.orchestra.run.smithi171.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-21T01:30:29.923 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-21T01:30:29.935 INFO:teuthology.orchestra.run.smithi171.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-21T01:30:29.939 INFO:teuthology.orchestra.run.smithi171.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-21T01:30:29.940 INFO:teuthology.orchestra.run.smithi171.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-21T01:30:29.941 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-21T01:30:29.941 INFO:teuthology.orchestra.run.smithi003.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-21T01:30:29.942 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-21T01:30:29.943 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-21T01:30:29.943 INFO:teuthology.orchestra.run.smithi143.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-21T01:30:29.951 INFO:teuthology.orchestra.run.smithi003.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-21T01:30:29.952 INFO:teuthology.orchestra.run.smithi003.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-21T01:30:29.952 INFO:teuthology.orchestra.run.smithi171.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-21T01:30:29.955 INFO:teuthology.orchestra.run.smithi171.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-21T01:30:29.955 INFO:teuthology.orchestra.run.smithi171.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-21T01:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout:2 upgraded, 101 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 153 MB of archives. 2022-07-21T01:30:29.956 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-21T01:30:29.957 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-21T01:30:29.957 INFO:teuthology.orchestra.run.smithi171.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-21T01:30:29.957 INFO:teuthology.orchestra.run.smithi171.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-21T01:30:29.958 INFO:teuthology.orchestra.run.smithi143.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-21T01:30:29.959 INFO:teuthology.orchestra.run.smithi143.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-21T01:30:29.961 INFO:teuthology.orchestra.run.smithi143.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-21T01:30:29.961 INFO:teuthology.orchestra.run.smithi143.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-21T01:30:29.968 INFO:teuthology.orchestra.run.smithi003.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-21T01:30:29.969 INFO:teuthology.orchestra.run.smithi171.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-21T01:30:29.969 INFO:teuthology.orchestra.run.smithi171.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-21T01:30:29.970 INFO:teuthology.orchestra.run.smithi171.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-21T01:30:29.971 INFO:teuthology.orchestra.run.smithi003.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-21T01:30:29.971 INFO:teuthology.orchestra.run.smithi003.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-21T01:30:29.971 INFO:teuthology.orchestra.run.smithi171.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-21T01:30:29.973 INFO:teuthology.orchestra.run.smithi003.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-21T01:30:29.974 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-21T01:30:29.974 INFO:teuthology.orchestra.run.smithi003.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-21T01:30:29.975 INFO:teuthology.orchestra.run.smithi003.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-21T01:30:29.976 INFO:teuthology.orchestra.run.smithi003.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-21T01:30:29.976 INFO:teuthology.orchestra.run.smithi143.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-21T01:30:29.978 INFO:teuthology.orchestra.run.smithi143.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-21T01:30:29.986 INFO:teuthology.orchestra.run.smithi171.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-21T01:30:29.988 INFO:teuthology.orchestra.run.smithi003.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-21T01:30:29.988 INFO:teuthology.orchestra.run.smithi171.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-21T01:30:29.989 INFO:teuthology.orchestra.run.smithi003.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-21T01:30:29.990 INFO:teuthology.orchestra.run.smithi171.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-21T01:30:29.990 INFO:teuthology.orchestra.run.smithi171.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-21T01:30:29.991 INFO:teuthology.orchestra.run.smithi143.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-21T01:30:29.992 INFO:teuthology.orchestra.run.smithi143.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-21T01:30:29.992 INFO:teuthology.orchestra.run.smithi143.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-21T01:30:30.004 INFO:teuthology.orchestra.run.smithi003.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-21T01:30:30.006 INFO:teuthology.orchestra.run.smithi003.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-21T01:30:30.007 INFO:teuthology.orchestra.run.smithi003.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-21T01:30:30.009 INFO:teuthology.orchestra.run.smithi003.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-21T01:30:30.011 INFO:teuthology.orchestra.run.smithi003.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-21T01:30:30.012 INFO:teuthology.orchestra.run.smithi003.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-21T01:30:30.023 INFO:teuthology.orchestra.run.smithi171.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-21T01:30:30.024 INFO:teuthology.orchestra.run.smithi171.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-21T01:30:30.026 INFO:teuthology.orchestra.run.smithi171.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-21T01:30:30.027 INFO:teuthology.orchestra.run.smithi171.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-21T01:30:30.028 INFO:teuthology.orchestra.run.smithi171.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-21T01:30:30.041 INFO:teuthology.orchestra.run.smithi171.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-21T01:30:30.042 INFO:teuthology.orchestra.run.smithi003.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-21T01:30:30.052 INFO:teuthology.orchestra.run.smithi003.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-21T01:30:30.053 INFO:teuthology.orchestra.run.smithi003.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-21T01:30:30.053 INFO:teuthology.orchestra.run.smithi003.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-21T01:30:30.053 INFO:teuthology.orchestra.run.smithi003.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-21T01:30:30.054 INFO:teuthology.orchestra.run.smithi171.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-21T01:30:30.054 INFO:teuthology.orchestra.run.smithi171.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-21T01:30:30.056 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-21T01:30:30.058 INFO:teuthology.orchestra.run.smithi171.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-21T01:30:30.060 INFO:teuthology.orchestra.run.smithi171.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-21T01:30:30.061 INFO:teuthology.orchestra.run.smithi003.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-21T01:30:30.062 INFO:teuthology.orchestra.run.smithi171.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-21T01:30:30.063 INFO:teuthology.orchestra.run.smithi171.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-21T01:30:30.064 INFO:teuthology.orchestra.run.smithi171.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-21T01:30:30.064 INFO:teuthology.orchestra.run.smithi003.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-21T01:30:30.064 INFO:teuthology.orchestra.run.smithi003.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-21T01:30:30.077 INFO:teuthology.orchestra.run.smithi003.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-21T01:30:30.078 INFO:teuthology.orchestra.run.smithi003.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-21T01:30:30.079 INFO:teuthology.orchestra.run.smithi003.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-21T01:30:30.080 INFO:teuthology.orchestra.run.smithi003.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-21T01:30:30.081 INFO:teuthology.orchestra.run.smithi003.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-21T01:30:30.091 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-21T01:30:30.128 INFO:teuthology.orchestra.run.smithi171.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-21T01:30:30.175 INFO:teuthology.orchestra.run.smithi003.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-21T01:30:30.224 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-21T01:30:30.292 INFO:teuthology.orchestra.run.smithi171.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-21T01:30:30.304 INFO:teuthology.orchestra.run.smithi171.stdout:Get:42 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-21T01:30:30.316 INFO:teuthology.orchestra.run.smithi171.stdout:Get:43 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-21T01:30:30.317 INFO:teuthology.orchestra.run.smithi171.stdout:Get:44 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-21T01:30:30.349 INFO:teuthology.orchestra.run.smithi171.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-21T01:30:30.351 INFO:teuthology.orchestra.run.smithi171.stdout:Get:46 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-21T01:30:30.356 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-21T01:30:30.364 INFO:teuthology.orchestra.run.smithi171.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-21T01:30:30.365 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-21T01:30:30.380 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-21T01:30:30.434 INFO:teuthology.orchestra.run.smithi143.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-21T01:30:30.443 INFO:teuthology.orchestra.run.smithi171.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-21T01:30:30.449 INFO:teuthology.orchestra.run.smithi143.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-21T01:30:30.458 INFO:teuthology.orchestra.run.smithi171.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-21T01:30:30.460 INFO:teuthology.orchestra.run.smithi171.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-21T01:30:30.471 INFO:teuthology.orchestra.run.smithi171.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-21T01:30:30.483 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-21T01:30:30.505 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-21T01:30:30.510 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-21T01:30:30.529 INFO:teuthology.orchestra.run.smithi171.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-21T01:30:30.531 INFO:teuthology.orchestra.run.smithi171.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-21T01:30:30.531 INFO:teuthology.orchestra.run.smithi143.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-21T01:30:30.532 INFO:teuthology.orchestra.run.smithi171.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-21T01:30:30.533 INFO:teuthology.orchestra.run.smithi171.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-21T01:30:30.534 INFO:teuthology.orchestra.run.smithi171.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-21T01:30:30.534 INFO:teuthology.orchestra.run.smithi171.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-21T01:30:30.535 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-21T01:30:30.537 INFO:teuthology.orchestra.run.smithi171.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-21T01:30:30.538 INFO:teuthology.orchestra.run.smithi171.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-21T01:30:30.539 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-21T01:30:30.543 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-21T01:30:30.545 INFO:teuthology.orchestra.run.smithi171.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-21T01:30:30.546 INFO:teuthology.orchestra.run.smithi171.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-21T01:30:30.547 INFO:teuthology.orchestra.run.smithi171.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-21T01:30:30.551 INFO:teuthology.orchestra.run.smithi171.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-21T01:30:30.551 INFO:teuthology.orchestra.run.smithi143.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-21T01:30:30.552 INFO:teuthology.orchestra.run.smithi143.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-21T01:30:30.552 INFO:teuthology.orchestra.run.smithi143.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-21T01:30:30.553 INFO:teuthology.orchestra.run.smithi143.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-21T01:30:30.554 INFO:teuthology.orchestra.run.smithi143.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-21T01:30:30.554 INFO:teuthology.orchestra.run.smithi143.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-21T01:30:30.554 INFO:teuthology.orchestra.run.smithi143.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-21T01:30:30.555 INFO:teuthology.orchestra.run.smithi143.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-21T01:30:30.556 INFO:teuthology.orchestra.run.smithi143.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-21T01:30:30.556 INFO:teuthology.orchestra.run.smithi171.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-21T01:30:30.557 INFO:teuthology.orchestra.run.smithi171.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-21T01:30:30.558 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-21T01:30:30.559 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-21T01:30:30.560 INFO:teuthology.orchestra.run.smithi171.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-21T01:30:30.560 INFO:teuthology.orchestra.run.smithi171.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-21T01:30:30.561 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-21T01:30:30.569 INFO:teuthology.orchestra.run.smithi143.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-21T01:30:30.572 INFO:teuthology.orchestra.run.smithi143.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-21T01:30:30.580 INFO:teuthology.orchestra.run.smithi171.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-21T01:30:30.581 INFO:teuthology.orchestra.run.smithi171.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-21T01:30:30.586 INFO:teuthology.orchestra.run.smithi171.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-21T01:30:30.586 INFO:teuthology.orchestra.run.smithi171.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-21T01:30:30.587 INFO:teuthology.orchestra.run.smithi143.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-21T01:30:30.588 INFO:teuthology.orchestra.run.smithi143.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-21T01:30:30.589 INFO:teuthology.orchestra.run.smithi143.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-21T01:30:30.591 INFO:teuthology.orchestra.run.smithi143.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-21T01:30:30.592 INFO:teuthology.orchestra.run.smithi171.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-21T01:30:30.593 INFO:teuthology.orchestra.run.smithi171.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-21T01:30:30.593 INFO:teuthology.orchestra.run.smithi143.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-21T01:30:30.594 INFO:teuthology.orchestra.run.smithi171.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-21T01:30:30.594 INFO:teuthology.orchestra.run.smithi171.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-21T01:30:30.595 INFO:teuthology.orchestra.run.smithi143.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-21T01:30:30.595 INFO:teuthology.orchestra.run.smithi171.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-21T01:30:30.597 INFO:teuthology.orchestra.run.smithi143.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-21T01:30:30.598 INFO:teuthology.orchestra.run.smithi143.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-21T01:30:30.599 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-21T01:30:30.602 INFO:teuthology.orchestra.run.smithi143.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-21T01:30:30.602 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-21T01:30:30.603 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-21T01:30:30.605 INFO:teuthology.orchestra.run.smithi143.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-21T01:30:30.605 INFO:teuthology.orchestra.run.smithi143.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-21T01:30:30.609 INFO:teuthology.orchestra.run.smithi171.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-21T01:30:30.609 INFO:teuthology.orchestra.run.smithi171.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-21T01:30:30.616 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-21T01:30:30.617 INFO:teuthology.orchestra.run.smithi171.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-21T01:30:30.618 INFO:teuthology.orchestra.run.smithi171.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-21T01:30:30.619 INFO:teuthology.orchestra.run.smithi171.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-21T01:30:30.620 INFO:teuthology.orchestra.run.smithi171.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-21T01:30:30.621 INFO:teuthology.orchestra.run.smithi171.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-21T01:30:30.622 INFO:teuthology.orchestra.run.smithi143.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-21T01:30:30.622 INFO:teuthology.orchestra.run.smithi171.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-21T01:30:30.623 INFO:teuthology.orchestra.run.smithi143.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-21T01:30:30.627 INFO:teuthology.orchestra.run.smithi003.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-21T01:30:30.639 INFO:teuthology.orchestra.run.smithi143.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-21T01:30:30.640 INFO:teuthology.orchestra.run.smithi143.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-21T01:30:30.640 INFO:teuthology.orchestra.run.smithi143.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-21T01:30:30.641 INFO:teuthology.orchestra.run.smithi143.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-21T01:30:30.642 INFO:teuthology.orchestra.run.smithi143.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-21T01:30:30.642 INFO:teuthology.orchestra.run.smithi143.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-21T01:30:30.643 INFO:teuthology.orchestra.run.smithi143.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-21T01:30:30.644 INFO:teuthology.orchestra.run.smithi143.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-21T01:30:30.675 INFO:teuthology.orchestra.run.smithi003.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-21T01:30:30.679 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-21T01:30:30.681 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-21T01:30:30.683 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-21T01:30:30.686 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-21T01:30:30.690 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-21T01:30:30.690 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-21T01:30:30.700 INFO:teuthology.orchestra.run.smithi003.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-21T01:30:30.701 INFO:teuthology.orchestra.run.smithi003.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-21T01:30:30.708 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-21T01:30:30.713 INFO:teuthology.orchestra.run.smithi003.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-21T01:30:30.714 INFO:teuthology.orchestra.run.smithi003.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-21T01:30:30.714 INFO:teuthology.orchestra.run.smithi003.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-21T01:30:30.715 INFO:teuthology.orchestra.run.smithi003.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-21T01:30:30.728 INFO:teuthology.orchestra.run.smithi003.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-21T01:30:30.759 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-21T01:30:30.759 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-21T01:30:30.760 INFO:teuthology.orchestra.run.smithi003.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-21T01:30:30.761 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-21T01:30:30.761 INFO:teuthology.orchestra.run.smithi003.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-21T01:30:30.762 INFO:teuthology.orchestra.run.smithi003.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-21T01:30:30.768 INFO:teuthology.orchestra.run.smithi003.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-21T01:30:30.771 INFO:teuthology.orchestra.run.smithi003.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-21T01:30:30.777 INFO:teuthology.orchestra.run.smithi003.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-21T01:30:30.778 INFO:teuthology.orchestra.run.smithi003.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-21T01:30:30.779 INFO:teuthology.orchestra.run.smithi003.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-21T01:30:30.783 INFO:teuthology.orchestra.run.smithi003.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-21T01:30:30.785 INFO:teuthology.orchestra.run.smithi003.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-21T01:30:30.787 INFO:teuthology.orchestra.run.smithi003.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-21T01:30:30.794 INFO:teuthology.orchestra.run.smithi003.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-21T01:30:30.797 INFO:teuthology.orchestra.run.smithi003.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-21T01:30:30.798 INFO:teuthology.orchestra.run.smithi003.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-21T01:30:30.800 INFO:teuthology.orchestra.run.smithi003.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-21T01:30:30.811 INFO:teuthology.orchestra.run.smithi003.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-21T01:30:30.813 INFO:teuthology.orchestra.run.smithi003.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-21T01:30:30.815 INFO:teuthology.orchestra.run.smithi003.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-21T01:30:30.816 INFO:teuthology.orchestra.run.smithi003.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-21T01:30:30.817 INFO:teuthology.orchestra.run.smithi003.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-21T01:30:30.818 INFO:teuthology.orchestra.run.smithi003.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-21T01:30:30.819 INFO:teuthology.orchestra.run.smithi003.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-21T01:30:30.820 INFO:teuthology.orchestra.run.smithi003.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-21T01:30:30.833 INFO:teuthology.orchestra.run.smithi003.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-21T01:30:30.834 INFO:teuthology.orchestra.run.smithi003.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-21T01:30:30.835 INFO:teuthology.orchestra.run.smithi003.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-21T01:30:30.835 INFO:teuthology.orchestra.run.smithi003.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-21T01:30:30.839 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-21T01:30:30.845 INFO:teuthology.orchestra.run.smithi003.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-21T01:30:30.846 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-21T01:30:30.847 INFO:teuthology.orchestra.run.smithi003.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-21T01:30:30.848 INFO:teuthology.orchestra.run.smithi003.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-21T01:30:30.848 INFO:teuthology.orchestra.run.smithi003.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-21T01:30:30.849 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-21T01:30:30.850 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-21T01:30:30.853 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-21T01:30:30.856 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-21T01:30:30.860 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-21T01:30:30.863 INFO:teuthology.orchestra.run.smithi003.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-21T01:30:30.919 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-21T01:30:30.921 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-21T01:30:31.249 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-21T01:30:31.306 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-21T01:30:31.307 INFO:teuthology.orchestra.run.smithi174.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-21T01:30:31.334 INFO:teuthology.orchestra.run.smithi003.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-21T01:30:31.336 INFO:teuthology.orchestra.run.smithi003.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-21T01:30:31.347 INFO:teuthology.orchestra.run.smithi174.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-21T01:30:31.349 INFO:teuthology.orchestra.run.smithi174.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-21T01:30:31.349 INFO:teuthology.orchestra.run.smithi174.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-21T01:30:31.351 INFO:teuthology.orchestra.run.smithi174.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-21T01:30:31.351 INFO:teuthology.orchestra.run.smithi174.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-21T01:30:31.353 INFO:teuthology.orchestra.run.smithi174.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-21T01:30:31.354 INFO:teuthology.orchestra.run.smithi174.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-21T01:30:31.388 INFO:teuthology.orchestra.run.smithi003.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-21T01:30:31.410 INFO:teuthology.orchestra.run.smithi174.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-21T01:30:31.410 INFO:teuthology.orchestra.run.smithi174.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-21T01:30:31.491 INFO:teuthology.orchestra.run.smithi174.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-21T01:30:31.495 INFO:teuthology.orchestra.run.smithi174.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-21T01:30:31.498 INFO:teuthology.orchestra.run.smithi174.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-21T01:30:31.501 INFO:teuthology.orchestra.run.smithi174.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-21T01:30:31.503 INFO:teuthology.orchestra.run.smithi174.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-21T01:30:31.506 INFO:teuthology.orchestra.run.smithi174.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-21T01:30:31.508 INFO:teuthology.orchestra.run.smithi174.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-21T01:30:31.509 INFO:teuthology.orchestra.run.smithi174.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-21T01:30:31.571 INFO:teuthology.orchestra.run.smithi174.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-21T01:30:31.578 INFO:teuthology.orchestra.run.smithi174.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-21T01:30:31.651 INFO:teuthology.orchestra.run.smithi174.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-21T01:30:31.652 INFO:teuthology.orchestra.run.smithi174.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-21T01:30:31.653 INFO:teuthology.orchestra.run.smithi174.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-21T01:30:31.654 INFO:teuthology.orchestra.run.smithi174.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-21T01:30:31.672 INFO:teuthology.orchestra.run.smithi174.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-21T01:30:31.673 INFO:teuthology.orchestra.run.smithi174.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-21T01:30:31.674 INFO:teuthology.orchestra.run.smithi174.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-21T01:30:31.674 INFO:teuthology.orchestra.run.smithi174.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-21T01:30:31.684 INFO:teuthology.orchestra.run.smithi171.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-21T01:30:31.728 INFO:teuthology.orchestra.run.smithi174.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-21T01:30:31.731 INFO:teuthology.orchestra.run.smithi174.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-21T01:30:31.733 INFO:teuthology.orchestra.run.smithi174.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-21T01:30:31.810 INFO:teuthology.orchestra.run.smithi174.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-21T01:30:31.814 INFO:teuthology.orchestra.run.smithi174.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-21T01:30:31.884 INFO:teuthology.orchestra.run.smithi171.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-21T01:30:31.888 INFO:teuthology.orchestra.run.smithi171.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-21T01:30:31.913 INFO:teuthology.orchestra.run.smithi174.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-21T01:30:31.964 INFO:teuthology.orchestra.run.smithi171.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-21T01:30:32.011 INFO:teuthology.orchestra.run.smithi174.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-21T01:30:32.011 INFO:teuthology.orchestra.run.smithi174.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-21T01:30:32.080 INFO:teuthology.orchestra.run.smithi174.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-21T01:30:32.081 INFO:teuthology.orchestra.run.smithi174.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-21T01:30:32.167 INFO:teuthology.orchestra.run.smithi143.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-21T01:30:32.224 INFO:teuthology.orchestra.run.smithi174.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-21T01:30:32.289 INFO:teuthology.orchestra.run.smithi171.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-21T01:30:32.390 INFO:teuthology.orchestra.run.smithi143.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-21T01:30:32.605 INFO:teuthology.orchestra.run.smithi143.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-21T01:30:32.624 INFO:teuthology.orchestra.run.smithi143.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-21T01:30:32.715 INFO:teuthology.orchestra.run.smithi143.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-21T01:30:32.718 INFO:teuthology.orchestra.run.smithi143.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-21T01:30:32.836 INFO:teuthology.orchestra.run.smithi143.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-21T01:30:33.319 INFO:teuthology.orchestra.run.smithi174.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-21T01:30:33.354 INFO:teuthology.orchestra.run.smithi174.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-21T01:30:33.467 INFO:teuthology.orchestra.run.smithi174.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-21T01:30:33.728 INFO:teuthology.orchestra.run.smithi171.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-21T01:30:33.728 INFO:teuthology.orchestra.run.smithi171.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-21T01:30:33.743 INFO:teuthology.orchestra.run.smithi171.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-21T01:30:33.753 INFO:teuthology.orchestra.run.smithi171.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-21T01:30:33.754 INFO:teuthology.orchestra.run.smithi171.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-21T01:30:33.755 INFO:teuthology.orchestra.run.smithi171.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-21T01:30:33.917 INFO:teuthology.orchestra.run.smithi171.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-21T01:30:33.918 INFO:teuthology.orchestra.run.smithi171.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-21T01:30:34.276 INFO:teuthology.orchestra.run.smithi143.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-21T01:30:34.334 INFO:teuthology.orchestra.run.smithi143.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-21T01:30:34.361 INFO:teuthology.orchestra.run.smithi171.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-21T01:30:34.361 INFO:teuthology.orchestra.run.smithi171.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-21T01:30:34.400 INFO:teuthology.orchestra.run.smithi003.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-21T01:30:34.510 INFO:teuthology.orchestra.run.smithi143.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-21T01:30:35.134 INFO:teuthology.orchestra.run.smithi003.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-21T01:30:35.154 INFO:teuthology.orchestra.run.smithi003.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-21T01:30:35.371 INFO:teuthology.orchestra.run.smithi003.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-21T01:30:36.247 INFO:teuthology.orchestra.run.smithi003.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-21T01:30:36.720 INFO:teuthology.orchestra.run.smithi171.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-21T01:30:36.720 INFO:teuthology.orchestra.run.smithi171.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-21T01:30:36.721 INFO:teuthology.orchestra.run.smithi171.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-21T01:30:37.078 INFO:teuthology.orchestra.run.smithi171.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-21T01:30:37.668 INFO:teuthology.orchestra.run.smithi171.stdout:Fetched 153 MB in 7s (20.9 MB/s) 2022-07-21T01:30:37.830 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-21T01:30:37.871 INFO:teuthology.orchestra.run.smithi171.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-21T01:30:37.874 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-21T01:30:37.903 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-21T01:30:38.172 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-21T01:30:38.185 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-21T01:30:38.214 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-21T01:30:38.253 INFO:teuthology.orchestra.run.smithi174.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-21T01:30:38.320 INFO:teuthology.orchestra.run.smithi003.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-21T01:30:38.320 INFO:teuthology.orchestra.run.smithi003.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-21T01:30:38.331 INFO:teuthology.orchestra.run.smithi003.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-21T01:30:38.431 INFO:teuthology.orchestra.run.smithi003.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-21T01:30:38.432 INFO:teuthology.orchestra.run.smithi003.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-21T01:30:38.435 INFO:teuthology.orchestra.run.smithi003.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-21T01:30:38.483 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cmd2. 2022-07-21T01:30:38.497 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-21T01:30:38.541 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-21T01:30:38.660 INFO:teuthology.orchestra.run.smithi003.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-21T01:30:38.662 INFO:teuthology.orchestra.run.smithi003.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-21T01:30:38.830 INFO:teuthology.orchestra.run.smithi174.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-21T01:30:38.849 INFO:teuthology.orchestra.run.smithi174.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-21T01:30:38.911 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-21T01:30:38.923 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-21T01:30:39.020 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-21T01:30:39.024 INFO:teuthology.orchestra.run.smithi174.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-21T01:30:39.173 INFO:teuthology.orchestra.run.smithi003.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-21T01:30:39.174 INFO:teuthology.orchestra.run.smithi003.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-21T01:30:39.238 INFO:teuthology.orchestra.run.smithi143.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-21T01:30:39.355 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:39.498 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-21T01:30:39.744 INFO:teuthology.orchestra.run.smithi174.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-21T01:30:39.962 INFO:teuthology.orchestra.run.smithi143.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-21T01:30:39.997 INFO:teuthology.orchestra.run.smithi143.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-21T01:30:40.041 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:40.169 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-21T01:30:40.185 INFO:teuthology.orchestra.run.smithi143.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-21T01:30:40.840 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libcephfs2. 2022-07-21T01:30:40.852 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:40.889 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:41.024 INFO:teuthology.orchestra.run.smithi143.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-21T01:30:41.184 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-rados. 2022-07-21T01:30:41.196 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:41.225 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:41.587 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-21T01:30:41.601 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:41.645 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:41.972 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cephfs. 2022-07-21T01:30:41.986 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:42.022 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:42.152 INFO:teuthology.orchestra.run.smithi003.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-21T01:30:42.153 INFO:teuthology.orchestra.run.smithi003.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-21T01:30:42.153 INFO:teuthology.orchestra.run.smithi003.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-21T01:30:42.383 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-21T01:30:42.397 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:42.441 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:42.602 INFO:teuthology.orchestra.run.smithi174.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-21T01:30:42.603 INFO:teuthology.orchestra.run.smithi174.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-21T01:30:42.612 INFO:teuthology.orchestra.run.smithi174.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-21T01:30:42.658 INFO:teuthology.orchestra.run.smithi003.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-21T01:30:42.758 INFO:teuthology.orchestra.run.smithi174.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-21T01:30:42.759 INFO:teuthology.orchestra.run.smithi174.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-21T01:30:42.770 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-prettytable. 2022-07-21T01:30:42.772 INFO:teuthology.orchestra.run.smithi174.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-21T01:30:42.785 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-21T01:30:42.820 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-21T01:30:42.982 INFO:teuthology.orchestra.run.smithi174.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-21T01:30:42.983 INFO:teuthology.orchestra.run.smithi174.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-21T01:30:43.005 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-rbd. 2022-07-21T01:30:43.019 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:43.047 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:43.311 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 153 MB in 13s (11.9 MB/s) 2022-07-21T01:30:43.425 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-21T01:30:43.440 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-21T01:30:43.462 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-21T01:30:43.483 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-21T01:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-21T01:30:43.492 INFO:teuthology.orchestra.run.smithi174.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-21T01:30:43.504 INFO:teuthology.orchestra.run.smithi003.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-21T01:30:43.508 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-21T01:30:43.553 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-21T01:30:43.583 INFO:teuthology.orchestra.run.smithi143.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-21T01:30:43.584 INFO:teuthology.orchestra.run.smithi143.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-21T01:30:43.593 INFO:teuthology.orchestra.run.smithi143.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-21T01:30:43.780 INFO:teuthology.orchestra.run.smithi143.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-21T01:30:43.781 INFO:teuthology.orchestra.run.smithi143.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-21T01:30:43.784 INFO:teuthology.orchestra.run.smithi143.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-21T01:30:43.844 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-21T01:30:43.855 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-21T01:30:43.857 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-21T01:30:43.868 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-21T01:30:43.894 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-21T01:30:43.905 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-21T01:30:44.088 INFO:teuthology.orchestra.run.smithi143.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-21T01:30:44.089 INFO:teuthology.orchestra.run.smithi143.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-21T01:30:44.216 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cmd2. 2022-07-21T01:30:44.222 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package librgw2. 2022-07-21T01:30:44.231 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-21T01:30:44.235 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:44.264 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:44.274 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-21T01:30:44.586 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-21T01:30:44.601 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-21T01:30:44.669 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-21T01:30:44.670 INFO:teuthology.orchestra.run.smithi143.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-21T01:30:44.670 INFO:teuthology.orchestra.run.smithi143.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-21T01:30:44.758 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-rgw. 2022-07-21T01:30:44.772 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:44.841 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:45.003 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:45.203 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-21T01:30:45.215 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-21T01:30:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-21T01:30:45.373 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-21T01:30:45.748 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:45.780 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package liboath0. 2022-07-21T01:30:45.794 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-21T01:30:45.838 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-21T01:30:45.894 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-21T01:30:46.083 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libradosstriper1. 2022-07-21T01:30:46.095 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:46.124 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:46.458 INFO:teuthology.orchestra.run.smithi174.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-21T01:30:46.458 INFO:teuthology.orchestra.run.smithi174.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-21T01:30:46.459 INFO:teuthology.orchestra.run.smithi174.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-21T01:30:46.522 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libcephfs2. 2022-07-21T01:30:46.535 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:46.572 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:46.594 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-common. 2022-07-21T01:30:46.609 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:46.653 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:46.825 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rados. 2022-07-21T01:30:46.837 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:46.867 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:47.039 INFO:teuthology.orchestra.run.smithi174.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-21T01:30:47.236 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-21T01:30:47.252 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:47.294 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:47.385 INFO:teuthology.orchestra.run.smithi143.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-21T01:30:47.386 INFO:teuthology.orchestra.run.smithi143.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-21T01:30:47.387 INFO:teuthology.orchestra.run.smithi143.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-21T01:30:47.622 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cephfs. 2022-07-21T01:30:47.637 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:47.665 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 153 MB in 17s (8,859 kB/s) 2022-07-21T01:30:47.680 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:47.728 INFO:teuthology.orchestra.run.smithi143.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-21T01:30:47.810 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-21T01:30:47.851 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-21T01:30:47.854 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-21T01:30:47.892 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-21T01:30:47.966 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-21T01:30:47.979 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:48.007 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:48.219 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-21T01:30:48.234 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-21T01:30:48.277 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-prettytable. 2022-07-21T01:30:48.278 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-21T01:30:48.291 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-21T01:30:48.327 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-21T01:30:48.378 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 153 MB in 18s (8,469 kB/s) 2022-07-21T01:30:48.521 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-21T01:30:48.558 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cmd2. 2022-07-21T01:30:48.562 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-21T01:30:48.565 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-21T01:30:48.569 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-21T01:30:48.579 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rbd. 2022-07-21T01:30:48.594 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:48.605 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-21T01:30:48.622 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-21T01:30:48.638 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:48.708 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-base. 2022-07-21T01:30:48.722 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:48.793 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:48.897 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-21T01:30:48.912 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-21T01:30:48.933 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-21T01:30:48.946 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-21T01:30:48.947 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-21T01:30:49.025 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-21T01:30:49.041 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-21T01:30:49.055 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-21T01:30:49.099 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-21T01:30:49.209 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cmd2. 2022-07-21T01:30:49.223 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-21T01:30:49.277 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-21T01:30:49.367 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:49.377 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-21T01:30:49.391 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-21T01:30:49.452 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-21T01:30:49.512 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-21T01:30:49.612 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-21T01:30:49.625 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-21T01:30:49.713 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cryptography. 2022-07-21T01:30:49.726 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-21T01:30:49.728 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-21T01:30:49.755 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librgw2. 2022-07-21T01:30:49.763 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-21T01:30:49.767 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:49.804 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:50.038 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:50.078 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:50.141 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-21T01:30:50.183 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-openssl. 2022-07-21T01:30:50.197 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-21T01:30:50.217 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-21T01:30:50.241 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-21T01:30:50.349 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rgw. 2022-07-21T01:30:50.363 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:50.407 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:50.502 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-21T01:30:50.517 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:50.560 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:50.725 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:50.770 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs2. 2022-07-21T01:30:50.785 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:50.828 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:50.846 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-21T01:30:50.868 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-21T01:30:50.884 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-21T01:30:50.918 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-21T01:30:50.948 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-21T01:30:50.961 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-21T01:30:51.006 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-21T01:30:51.164 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rados. 2022-07-21T01:30:51.178 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:51.196 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liboath0. 2022-07-21T01:30:51.210 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-21T01:30:51.221 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:51.270 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-21T01:30:51.334 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-21T01:30:51.348 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-21T01:30:51.392 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-21T01:30:51.516 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs2. 2022-07-21T01:30:51.524 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-21T01:30:51.532 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:51.537 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:51.573 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libradosstriper1. 2022-07-21T01:30:51.574 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:51.575 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:51.587 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:51.632 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:51.754 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-jwt. 2022-07-21T01:30:51.770 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-21T01:30:51.812 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-21T01:30:51.886 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rados. 2022-07-21T01:30:51.902 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:51.903 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cephfs. 2022-07-21T01:30:51.918 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:51.935 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:51.962 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:52.098 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-logutils. 2022-07-21T01:30:52.110 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-common. 2022-07-21T01:30:52.111 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-21T01:30:52.123 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:52.168 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:52.174 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-21T01:30:52.255 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-21T01:30:52.269 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:52.289 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-21T01:30:52.303 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:52.313 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:52.348 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:52.409 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-21T01:30:52.424 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-21T01:30:52.459 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-21T01:30:52.633 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cephfs. 2022-07-21T01:30:52.649 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:52.684 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-prettytable. 2022-07-21T01:30:52.691 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:52.698 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-21T01:30:52.742 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-21T01:30:52.796 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-mako. 2022-07-21T01:30:52.809 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-21T01:30:52.854 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-21T01:30:52.953 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rbd. 2022-07-21T01:30:52.965 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:52.969 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-21T01:30:52.985 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:52.994 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:53.027 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:53.141 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-21T01:30:53.156 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-21T01:30:53.190 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-21T01:30:53.364 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-prettytable. 2022-07-21T01:30:53.379 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-21T01:30:53.380 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-21T01:30:53.395 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-21T01:30:53.422 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-21T01:30:53.439 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-21T01:30:53.502 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-21T01:30:53.518 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-21T01:30:53.560 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-21T01:30:53.675 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rbd. 2022-07-21T01:30:53.690 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:53.758 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:53.766 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-21T01:30:53.781 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-21T01:30:53.825 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-21T01:30:53.880 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-webob. 2022-07-21T01:30:53.893 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-21T01:30:53.938 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-21T01:30:54.070 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-21T01:30:54.082 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-21T01:30:54.111 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-21T01:30:54.144 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librgw2. 2022-07-21T01:30:54.159 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:54.194 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:54.245 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-base. 2022-07-21T01:30:54.255 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:54.282 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-21T01:30:54.298 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-21T01:30:54.308 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:54.332 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-21T01:30:54.448 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-21T01:30:54.461 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-21T01:30:54.498 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-21T01:30:54.593 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-bs4. 2022-07-21T01:30:54.607 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-21T01:30:54.679 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-21T01:30:54.730 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rgw. 2022-07-21T01:30:54.743 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:54.779 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:54.851 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librgw2. 2022-07-21T01:30:54.863 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:54.901 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:55.023 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-waitress. 2022-07-21T01:30:55.037 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-21T01:30:55.099 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-21T01:30:55.101 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-21T01:30:55.111 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-21T01:30:55.149 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-21T01:30:55.186 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cryptography. 2022-07-21T01:30:55.199 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-21T01:30:55.227 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-21T01:30:55.418 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-tempita. 2022-07-21T01:30:55.433 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-21T01:30:55.462 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rgw. 2022-07-21T01:30:55.474 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:55.501 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-21T01:30:55.512 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:55.527 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liboath0. 2022-07-21T01:30:55.542 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-21T01:30:55.581 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-openssl. 2022-07-21T01:30:55.585 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-21T01:30:55.597 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-21T01:30:55.631 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-21T01:30:55.795 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-paste. 2022-07-21T01:30:55.811 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-21T01:30:55.823 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-21T01:30:55.838 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-21T01:30:55.854 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-21T01:30:55.867 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-21T01:30:55.871 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libradosstriper1. 2022-07-21T01:30:55.881 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:30:55.882 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-21T01:30:55.884 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:55.925 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:55.929 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:56.174 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-21T01:30:56.176 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liboath0. 2022-07-21T01:30:56.188 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-21T01:30:56.189 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-21T01:30:56.219 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-21T01:30:56.232 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-21T01:30:56.271 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-21T01:30:56.285 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-21T01:30:56.299 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-common. 2022-07-21T01:30:56.315 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:56.321 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-21T01:30:56.349 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:56.529 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libradosstriper1. 2022-07-21T01:30:56.541 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:56.568 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-21T01:30:56.574 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-21T01:30:56.579 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:56.583 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-21T01:30:56.589 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-21T01:30:56.626 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-21T01:30:56.632 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-21T01:30:56.962 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-webtest. 2022-07-21T01:30:56.977 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-21T01:30:57.004 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jwt. 2022-07-21T01:30:57.017 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-21T01:30:57.021 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-21T01:30:57.041 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-common. 2022-07-21T01:30:57.054 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-21T01:30:57.057 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:57.099 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:57.298 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pecan. 2022-07-21T01:30:57.311 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-21T01:30:57.340 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-21T01:30:57.391 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-logutils. 2022-07-21T01:30:57.404 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-21T01:30:57.440 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-21T01:30:57.669 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-21T01:30:57.676 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-21T01:30:57.685 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-21T01:30:57.689 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-21T01:30:57.718 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-21T01:30:57.719 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-21T01:30:58.013 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-mako. 2022-07-21T01:30:58.027 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-21T01:30:58.056 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr. 2022-07-21T01:30:58.070 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:58.070 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-21T01:30:58.106 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:58.382 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-21T01:30:58.395 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-21T01:30:58.404 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-base. 2022-07-21T01:30:58.420 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:58.432 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-21T01:30:58.472 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:58.526 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mon. 2022-07-21T01:30:58.538 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:58.567 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:58.760 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-21T01:30:58.774 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-21T01:30:58.818 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-21T01:30:59.054 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-webob. 2022-07-21T01:30:59.067 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-21T01:30:59.096 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-21T01:30:59.130 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-base. 2022-07-21T01:30:59.144 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:59.190 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:59.383 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cryptography. 2022-07-21T01:30:59.398 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-21T01:30:59.441 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-21T01:30:59.515 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-21T01:30:59.530 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-21T01:30:59.573 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-21T01:30:59.626 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-osd. 2022-07-21T01:30:59.641 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:30:59.695 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:30:59.834 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-openssl. 2022-07-21T01:30:59.849 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-21T01:30:59.895 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-21T01:30:59.910 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-bs4. 2022-07-21T01:30:59.923 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-21T01:30:59.968 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-21T01:31:00.127 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cryptography. 2022-07-21T01:31:00.131 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-21T01:31:00.143 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-21T01:31:00.146 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:00.181 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:00.185 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-21T01:31:00.313 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-waitress. 2022-07-21T01:31:00.327 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-21T01:31:00.374 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-21T01:31:00.468 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-21T01:31:00.481 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-21T01:31:00.518 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-21T01:31:00.829 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-21T01:31:00.841 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-21T01:31:00.879 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-21T01:31:01.019 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-tempita. 2022-07-21T01:31:01.034 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-21T01:31:01.181 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-openssl. 2022-07-21T01:31:01.197 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-21T01:31:01.281 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-21T01:31:01.416 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-21T01:31:01.441 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jwt. 2022-07-21T01:31:01.457 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-21T01:31:01.500 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-21T01:31:01.514 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:01.550 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:01.727 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-paste. 2022-07-21T01:31:01.742 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-21T01:31:01.793 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-21T01:31:01.816 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-21T01:31:01.896 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-21T01:31:01.909 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-21T01:31:01.946 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-21T01:31:02.069 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-logutils. 2022-07-21T01:31:02.083 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-21T01:31:02.097 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-21T01:31:02.111 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-21T01:31:02.135 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-21T01:31:02.155 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-21T01:31:02.225 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-21T01:31:02.240 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-21T01:31:02.283 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-21T01:31:02.292 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph. 2022-07-21T01:31:02.309 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:02.342 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:02.421 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-21T01:31:02.437 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-21T01:31:02.458 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-21T01:31:02.473 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-21T01:31:02.479 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-21T01:31:02.516 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-21T01:31:02.561 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jwt. 2022-07-21T01:31:02.574 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-21T01:31:02.628 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-21T01:31:02.645 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-fuse. 2022-07-21T01:31:02.660 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:02.703 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:02.766 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-mako. 2022-07-21T01:31:02.782 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-21T01:31:02.802 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-webtest. 2022-07-21T01:31:02.818 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-21T01:31:02.840 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-21T01:31:02.885 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-21T01:31:02.915 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-logutils. 2022-07-21T01:31:02.930 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-21T01:31:02.973 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-21T01:31:03.106 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mds. 2022-07-21T01:31:03.121 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:03.160 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-21T01:31:03.163 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pecan. 2022-07-21T01:31:03.164 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:03.174 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-21T01:31:03.176 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-21T01:31:03.205 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-21T01:31:03.227 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-21T01:31:03.310 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-21T01:31:03.324 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-21T01:31:03.368 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-21T01:31:03.501 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-21T01:31:03.513 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-21T01:31:03.515 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-21T01:31:03.528 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-21T01:31:03.551 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-21T01:31:03.587 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-21T01:31:03.609 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package cephadm. 2022-07-21T01:31:03.622 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:03.671 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-mako. 2022-07-21T01:31:03.684 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-21T01:31:03.692 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:03.721 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-21T01:31:03.865 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webob. 2022-07-21T01:31:03.879 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-21T01:31:03.888 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr. 2022-07-21T01:31:03.902 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:03.923 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-21T01:31:03.937 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:03.987 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-jinja2. 2022-07-21T01:31:04.001 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-21T01:31:04.016 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-21T01:31:04.028 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-21T01:31:04.045 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-21T01:31:04.057 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-21T01:31:04.243 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-21T01:31:04.256 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-21T01:31:04.313 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-21T01:31:04.357 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mon. 2022-07-21T01:31:04.365 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-21T01:31:04.369 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-21T01:31:04.373 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:04.377 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:04.384 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-21T01:31:04.407 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:04.415 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:04.427 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-21T01:31:04.622 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-bs4. 2022-07-21T01:31:04.635 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-21T01:31:04.680 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-21T01:31:04.747 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webob. 2022-07-21T01:31:04.760 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-21T01:31:04.761 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-21T01:31:04.776 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-21T01:31:04.805 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-21T01:31:04.817 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-21T01:31:04.999 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-waitress. 2022-07-21T01:31:05.012 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-21T01:31:05.077 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-21T01:31:05.096 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libjs-underscore. 2022-07-21T01:31:05.109 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-21T01:31:05.134 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-21T01:31:05.148 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-21T01:31:05.153 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-21T01:31:05.208 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-21T01:31:05.376 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-tempita. 2022-07-21T01:31:05.391 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-21T01:31:05.434 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-21T01:31:05.452 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-osd. 2022-07-21T01:31:05.456 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-21T01:31:05.467 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:05.470 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bs4. 2022-07-21T01:31:05.470 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-21T01:31:05.485 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-21T01:31:05.510 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:05.514 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-21T01:31:05.519 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-21T01:31:05.738 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-paste. 2022-07-21T01:31:05.751 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-21T01:31:05.812 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-21T01:31:05.850 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-routes. 2022-07-21T01:31:05.865 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-21T01:31:05.873 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-waitress. 2022-07-21T01:31:05.885 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-21T01:31:05.908 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-21T01:31:05.925 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-21T01:31:06.123 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-21T01:31:06.139 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-21T01:31:06.182 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-21T01:31:06.262 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-21T01:31:06.268 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-tempita. 2022-07-21T01:31:06.276 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:06.282 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-21T01:31:06.319 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:06.326 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-21T01:31:06.484 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-21T01:31:06.498 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-21T01:31:06.551 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-21T01:31:06.629 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-paste. 2022-07-21T01:31:06.641 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-21T01:31:06.679 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-21T01:31:06.846 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webtest. 2022-07-21T01:31:06.861 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-21T01:31:06.904 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-21T01:31:07.016 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-21T01:31:07.031 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-21T01:31:07.074 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-21T01:31:07.182 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pecan. 2022-07-21T01:31:07.195 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-21T01:31:07.223 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-21T01:31:07.410 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-21T01:31:07.424 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-21T01:31:07.469 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-21T01:31:07.519 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-21T01:31:07.528 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-21T01:31:07.535 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:07.542 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-21T01:31:07.569 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:07.577 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-21T01:31:07.772 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webtest. 2022-07-21T01:31:07.785 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-21T01:31:07.822 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-21T01:31:07.838 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-decorator. 2022-07-21T01:31:07.854 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-21T01:31:07.882 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph. 2022-07-21T01:31:07.889 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr. 2022-07-21T01:31:07.897 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:07.901 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:07.922 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-21T01:31:07.932 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:07.939 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:08.075 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pecan. 2022-07-21T01:31:08.090 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-21T01:31:08.133 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-21T01:31:08.191 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-21T01:31:08.207 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-21T01:31:08.218 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-fuse. 2022-07-21T01:31:08.231 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:08.249 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-21T01:31:08.268 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:08.309 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mon. 2022-07-21T01:31:08.324 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:08.358 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:08.497 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-21T01:31:08.511 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-21T01:31:08.555 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-21T01:31:08.585 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-scipy. 2022-07-21T01:31:08.598 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-21T01:31:08.635 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-21T01:31:08.655 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mds. 2022-07-21T01:31:08.670 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:08.713 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:08.934 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr. 2022-07-21T01:31:08.948 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:08.992 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:09.182 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package cephadm. 2022-07-21T01:31:09.197 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:09.232 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:09.354 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-osd. 2022-07-21T01:31:09.367 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:09.411 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:09.479 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mon. 2022-07-21T01:31:09.492 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:09.527 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jinja2. 2022-07-21T01:31:09.536 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:09.540 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-21T01:31:09.576 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-21T01:31:09.905 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-21T01:31:09.919 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:09.952 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-21T01:31:09.963 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:09.966 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-21T01:31:10.035 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:10.425 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-21T01:31:10.439 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-21T01:31:10.483 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-21T01:31:10.499 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-osd. 2022-07-21T01:31:10.512 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:10.565 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:10.598 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-joblib. 2022-07-21T01:31:10.615 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-21T01:31:10.656 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-21T01:31:10.727 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libjs-underscore. 2022-07-21T01:31:10.740 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-21T01:31:10.794 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-21T01:31:11.043 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-sklearn. 2022-07-21T01:31:11.057 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-21T01:31:11.071 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-21T01:31:11.086 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-21T01:31:11.101 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:11.129 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-21T01:31:11.449 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-routes. 2022-07-21T01:31:11.463 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-21T01:31:11.507 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-21T01:31:11.842 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph. 2022-07-21T01:31:11.857 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:11.859 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-21T01:31:11.868 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-21T01:31:11.872 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:11.881 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:11.891 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:11.900 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:11.918 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:12.402 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-fuse. 2022-07-21T01:31:12.415 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:12.471 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:12.864 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mds. 2022-07-21T01:31:12.876 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-21T01:31:12.876 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:12.892 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:12.913 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:12.925 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:12.955 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph. 2022-07-21T01:31:12.970 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:13.004 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:13.053 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-cachetools. 2022-07-21T01:31:13.070 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-21T01:31:13.111 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-21T01:31:13.212 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-decorator. 2022-07-21T01:31:13.227 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-21T01:31:13.270 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-21T01:31:13.274 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-fuse. 2022-07-21T01:31:13.286 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:13.323 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:13.375 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package cephadm. 2022-07-21T01:31:13.388 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:13.416 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:13.448 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-21T01:31:13.461 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-21T01:31:13.506 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-21T01:31:13.581 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-21T01:31:13.597 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-21T01:31:13.640 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-21T01:31:13.703 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mds. 2022-07-21T01:31:13.711 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jinja2. 2022-07-21T01:31:13.716 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:13.724 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-21T01:31:13.760 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:13.761 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-21T01:31:13.851 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-21T01:31:13.865 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-21T01:31:13.909 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-21T01:31:13.942 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-scipy. 2022-07-21T01:31:13.956 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-21T01:31:14.026 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-21T01:31:14.080 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-21T01:31:14.095 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:14.138 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:14.228 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-rsa. 2022-07-21T01:31:14.230 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephadm. 2022-07-21T01:31:14.243 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:14.244 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-21T01:31:14.272 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:14.286 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-21T01:31:14.458 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-21T01:31:14.471 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-21T01:31:14.508 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-21T01:31:14.557 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-google-auth. 2022-07-21T01:31:14.570 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-21T01:31:14.609 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jinja2. 2022-07-21T01:31:14.623 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-21T01:31:14.631 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-21T01:31:14.667 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-21T01:31:14.778 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjs-underscore. 2022-07-21T01:31:14.790 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-21T01:31:14.827 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-21T01:31:14.934 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-blinker. 2022-07-21T01:31:14.947 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-21T01:31:14.984 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-21T01:31:14.995 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-21T01:31:15.010 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:15.053 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:15.097 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-21T01:31:15.112 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-21T01:31:15.146 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-21T01:31:15.221 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-21T01:31:15.237 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-21T01:31:15.276 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:15.287 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-21T01:31:15.303 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-21T01:31:15.345 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-21T01:31:15.373 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-21T01:31:15.389 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-21T01:31:15.448 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-21T01:31:15.449 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-routes. 2022-07-21T01:31:15.461 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-21T01:31:15.490 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-21T01:31:15.668 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-underscore. 2022-07-21T01:31:15.681 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-21T01:31:15.699 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-21T01:31:15.709 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-21T01:31:15.715 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-21T01:31:15.757 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-21T01:31:15.794 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-21T01:31:15.796 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-joblib. 2022-07-21T01:31:15.807 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:15.811 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-21T01:31:15.852 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:15.863 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-21T01:31:15.971 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-21T01:31:15.986 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-21T01:31:16.020 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-21T01:31:16.118 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-websocket. 2022-07-21T01:31:16.133 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-21T01:31:16.176 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-21T01:31:16.209 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-sklearn. 2022-07-21T01:31:16.221 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-21T01:31:16.249 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-routes. 2022-07-21T01:31:16.345 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-21T01:31:16.382 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-21T01:31:16.521 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-21T01:31:16.534 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-21T01:31:16.571 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-21T01:31:16.744 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-21T01:31:16.757 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:16.793 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:17.043 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-21T01:31:17.056 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:17.101 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:17.209 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-21T01:31:17.225 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:17.267 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:17.350 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-21T01:31:17.364 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:17.387 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-decorator. 2022-07-21T01:31:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:17.402 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-21T01:31:17.437 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-21T01:31:17.562 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-21T01:31:17.575 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-21T01:31:17.611 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-21T01:31:17.765 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-21T01:31:17.779 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-21T01:31:17.823 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-21T01:31:17.828 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-21T01:31:17.845 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:17.887 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:17.998 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-21T01:31:18.013 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-21T01:31:18.056 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:18.134 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-scipy. 2022-07-21T01:31:18.147 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-21T01:31:18.184 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-21T01:31:18.232 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-decorator. 2022-07-21T01:31:18.244 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-21T01:31:18.281 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-21T01:31:18.309 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package jq. 2022-07-21T01:31:18.324 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-21T01:31:18.392 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:18.503 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cachetools. 2022-07-21T01:31:18.516 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-21T01:31:18.553 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-21T01:31:18.593 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-21T01:31:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-21T01:31:18.628 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package socat. 2022-07-21T01:31:18.642 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-21T01:31:18.644 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-21T01:31:18.678 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-21T01:31:18.881 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-21T01:31:18.897 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-21T01:31:18.938 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-scipy. 2022-07-21T01:31:18.938 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-21T01:31:18.950 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-21T01:31:18.987 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-21T01:31:19.167 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-21T01:31:19.181 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-21T01:31:19.250 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-21T01:31:19.267 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-21T01:31:19.280 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-21T01:31:19.325 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-21T01:31:19.528 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package xmlstarlet. 2022-07-21T01:31:19.534 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-21T01:31:19.543 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-21T01:31:19.550 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-21T01:31:19.586 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-21T01:31:19.592 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:19.637 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rsa. 2022-07-21T01:31:19.651 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-21T01:31:19.686 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-21T01:31:19.949 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package ceph-test. 2022-07-21T01:31:19.956 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-google-auth. 2022-07-21T01:31:19.962 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:19.973 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-21T01:31:19.998 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:20.014 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-21T01:31:20.163 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-joblib. 2022-07-21T01:31:20.176 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-21T01:31:20.212 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-21T01:31:20.264 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-21T01:31:20.279 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-21T01:31:20.322 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:20.351 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-blinker. 2022-07-21T01:31:20.364 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-21T01:31:20.409 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-21T01:31:20.583 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn. 2022-07-21T01:31:20.596 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-21T01:31:20.670 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:20.745 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-21T01:31:20.759 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-21T01:31:20.803 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-21T01:31:20.868 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-joblib. 2022-07-21T01:31:20.883 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-21T01:31:20.917 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-21T01:31:21.132 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-21T01:31:21.145 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-21T01:31:21.182 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-21T01:31:21.272 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn. 2022-07-21T01:31:21.287 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-21T01:31:21.330 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:21.407 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-21T01:31:21.422 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:21.474 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:21.493 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-websocket. 2022-07-21T01:31:21.509 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-21T01:31:21.542 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-21T01:31:21.830 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-21T01:31:21.843 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-21T01:31:21.904 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-21T01:31:22.072 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-21T01:31:22.088 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:22.122 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:22.467 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-21T01:31:22.475 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:22.533 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:22.619 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cachetools. 2022-07-21T01:31:22.632 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-21T01:31:22.677 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-21T01:31:22.820 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-21T01:31:22.835 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-21T01:31:22.878 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-21T01:31:23.021 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-21T01:31:23.035 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-21T01:31:23.079 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-21T01:31:23.231 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-21T01:31:23.242 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cachetools. 2022-07-21T01:31:23.246 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-21T01:31:23.258 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-21T01:31:23.289 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:23.300 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-21T01:31:23.319 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-colorama. 2022-07-21T01:31:23.335 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-21T01:31:23.377 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-21T01:31:23.408 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-21T01:31:23.421 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-21T01:31:23.457 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-21T01:31:23.550 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package jq. 2022-07-21T01:31:23.563 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-21T01:31:23.603 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-21T01:31:23.608 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:23.619 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-21T01:31:23.661 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-21T01:31:23.672 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package cephfs-shell. 2022-07-21T01:31:23.685 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:23.730 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:23.794 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rsa. 2022-07-21T01:31:23.807 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-21T01:31:23.852 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-21T01:31:23.852 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package socat. 2022-07-21T01:31:23.865 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-21T01:31:23.894 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-21T01:31:24.007 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-21T01:31:24.015 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-21T01:31:24.021 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:24.028 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-21T01:31:24.065 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-21T01:31:24.066 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:24.164 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-google-auth. 2022-07-21T01:31:24.179 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-21T01:31:24.221 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-21T01:31:24.367 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-21T01:31:24.368 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rsa. 2022-07-21T01:31:24.380 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-21T01:31:24.381 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-21T01:31:24.402 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package nvme-cli. 2022-07-21T01:31:24.418 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-21T01:31:24.424 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-21T01:31:24.426 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-21T01:31:24.469 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-21T01:31:24.550 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-blinker. 2022-07-21T01:31:24.563 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-21T01:31:24.599 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-21T01:31:24.678 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package xmlstarlet. 2022-07-21T01:31:24.694 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-21T01:31:24.705 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-google-auth. 2022-07-21T01:31:24.719 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-21T01:31:24.727 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-21T01:31:24.780 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-21T01:31:24.832 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-21T01:31:24.846 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-21T01:31:24.877 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-21T01:31:24.882 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-21T01:31:24.891 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-21T01:31:24.935 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-21T01:31:25.073 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-test. 2022-07-21T01:31:25.086 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:25.100 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-blinker. 2022-07-21T01:31:25.115 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:25.115 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-21T01:31:25.158 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-21T01:31:25.201 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-attr. 2022-07-21T01:31:25.214 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-21T01:31:25.251 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-21T01:31:25.281 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-21T01:31:25.295 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-21T01:31:25.339 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-21T01:31:25.470 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-21T01:31:25.486 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-21T01:31:25.528 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-21T01:31:25.612 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-webencodings. 2022-07-21T01:31:25.626 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-21T01:31:25.633 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-websocket. 2022-07-21T01:31:25.649 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-21T01:31:25.670 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-21T01:31:25.691 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-21T01:31:25.882 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-21T01:31:25.896 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-21T01:31:25.940 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-21T01:31:25.966 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-html5lib. 2022-07-21T01:31:25.980 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-21T01:31:26.015 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-21T01:31:26.053 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-21T01:31:26.067 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-21T01:31:26.119 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-21T01:31:26.253 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-websocket. 2022-07-21T01:31:26.266 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-21T01:31:26.302 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-21T01:31:26.315 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-21T01:31:26.328 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-21T01:31:26.376 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-21T01:31:26.632 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-21T01:31:26.645 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-21T01:31:26.646 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-21T01:31:26.660 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:26.681 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-21T01:31:26.699 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:26.863 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-packaging. 2022-07-21T01:31:26.877 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-21T01:31:26.921 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-21T01:31:27.001 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-21T01:31:27.017 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-21T01:31:27.059 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-21T01:31:27.249 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pastescript. 2022-07-21T01:31:27.264 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-21T01:31:27.270 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-21T01:31:27.284 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:27.307 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-21T01:31:27.353 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:27.446 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-21T01:31:27.459 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-21T01:31:27.504 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:27.644 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pluggy. 2022-07-21T01:31:27.660 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-21T01:31:27.673 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-21T01:31:27.687 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-21T01:31:27.702 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-21T01:31:27.731 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-21T01:31:27.782 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package jq. 2022-07-21T01:31:27.795 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-21T01:31:27.840 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:28.014 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-psutil. 2022-07-21T01:31:28.030 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-21T01:31:28.063 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-21T01:31:28.068 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-21T01:31:28.084 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package socat. 2022-07-21T01:31:28.085 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-21T01:31:28.099 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-21T01:31:28.126 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:28.159 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-21T01:31:28.367 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-py. 2022-07-21T01:31:28.383 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-21T01:31:28.404 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package jq. 2022-07-21T01:31:28.417 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-21T01:31:28.441 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-21T01:31:28.454 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-colorama. 2022-07-21T01:31:28.551 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-21T01:31:28.618 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-21T01:31:28.623 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-21T01:31:28.637 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-21T01:31:28.672 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-21T01:31:28.707 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package socat. 2022-07-21T01:31:28.722 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-21T01:31:28.753 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-21T01:31:28.765 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-21T01:31:28.766 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-21T01:31:28.794 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-21T01:31:28.871 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package cephfs-shell. 2022-07-21T01:31:28.885 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:28.913 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:28.959 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package xmlstarlet. 2022-07-21T01:31:28.974 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-21T01:31:29.017 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-21T01:31:29.131 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-pytest. 2022-07-21T01:31:29.132 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-21T01:31:29.145 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-21T01:31:29.147 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:29.188 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-21T01:31:29.201 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:29.204 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-21T01:31:29.219 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-21T01:31:29.254 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-21T01:31:29.354 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-test. 2022-07-21T01:31:29.368 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:29.412 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:29.509 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package python3-simplejson. 2022-07-21T01:31:29.524 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-21T01:31:29.567 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-21T01:31:29.568 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package nvme-cli. 2022-07-21T01:31:29.583 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package xmlstarlet. 2022-07-21T01:31:29.583 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-21T01:31:29.596 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-21T01:31:29.641 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-21T01:31:29.644 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-21T01:31:29.987 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package radosgw. 2022-07-21T01:31:29.988 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-test. 2022-07-21T01:31:30.003 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:30.004 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:30.032 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-21T01:31:30.045 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:30.045 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:30.047 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-21T01:31:30.090 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-21T01:31:30.385 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-attr. 2022-07-21T01:31:30.399 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-21T01:31:30.459 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-21T01:31:30.788 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-webencodings. 2022-07-21T01:31:30.801 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-21T01:31:30.837 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-21T01:31:31.124 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-html5lib. 2022-07-21T01:31:31.139 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-21T01:31:31.148 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package rbd-fuse. 2022-07-21T01:31:31.162 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:31.173 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-21T01:31:31.206 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:31.485 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-21T01:31:31.499 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-21T01:31:31.529 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-21T01:31:31.543 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-21T01:31:31.853 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-21T01:31:31.971 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-packaging. 2022-07-21T01:31:31.978 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-21T01:31:31.984 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-21T01:31:32.013 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-21T01:31:32.105 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-21T01:31:32.290 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:32.349 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pastescript. 2022-07-21T01:31:32.363 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-21T01:31:32.407 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-21T01:31:32.752 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pluggy. 2022-07-21T01:31:32.767 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-21T01:31:32.800 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-colorama. 2022-07-21T01:31:32.810 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-21T01:31:32.815 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-21T01:31:32.849 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-21T01:31:32.893 INFO:teuthology.orchestra.run.smithi171.stdout:Adding system user cephadm....done 2022-07-21T01:31:33.067 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-21T01:31:33.086 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package cephfs-shell. 2022-07-21T01:31:33.099 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:33.143 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:33.155 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-psutil. 2022-07-21T01:31:33.172 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-21T01:31:33.213 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-21T01:31:33.407 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-21T01:31:33.416 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-colorama. 2022-07-21T01:31:33.422 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:33.422 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-21T01:31:33.431 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-21T01:31:33.464 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:33.499 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-21T01:31:33.550 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-py. 2022-07-21T01:31:33.566 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-21T01:31:33.608 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-21T01:31:33.715 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-21T01:31:33.769 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephfs-shell. 2022-07-21T01:31:33.776 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package nvme-cli. 2022-07-21T01:31:33.785 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-21T01:31:33.792 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-21T01:31:33.827 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:33.841 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-21T01:31:33.843 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-21T01:31:33.953 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-21T01:31:33.967 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-21T01:31:34.011 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-21T01:31:34.097 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-21T01:31:34.112 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:34.147 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:34.192 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-21T01:31:34.273 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-21T01:31:34.287 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-21T01:31:34.297 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pytest. 2022-07-21T01:31:34.313 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-21T01:31:34.331 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-21T01:31:34.347 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-21T01:31:34.484 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package nvme-cli. 2022-07-21T01:31:34.500 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-21T01:31:34.567 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-21T01:31:34.594 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-21T01:31:34.617 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-attr. 2022-07-21T01:31:34.633 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-21T01:31:34.675 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-21T01:31:34.675 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-simplejson. 2022-07-21T01:31:34.692 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-21T01:31:34.725 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-21T01:31:34.887 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-21T01:31:34.964 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-21T01:31:34.979 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-21T01:31:35.027 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webencodings. 2022-07-21T01:31:35.040 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-21T01:31:35.047 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-21T01:31:35.077 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-21T01:31:35.089 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package radosgw. 2022-07-21T01:31:35.103 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:35.147 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:35.180 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-21T01:31:35.300 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-attr. 2022-07-21T01:31:35.315 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-21T01:31:35.347 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-html5lib. 2022-07-21T01:31:35.350 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-21T01:31:35.362 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-21T01:31:35.405 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-21T01:31:35.473 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-21T01:31:35.695 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webencodings. 2022-07-21T01:31:35.709 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-21T01:31:35.726 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-21T01:31:35.741 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-21T01:31:35.746 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-21T01:31:35.775 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-21T01:31:35.783 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-21T01:31:36.067 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:36.074 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-html5lib. 2022-07-21T01:31:36.088 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-21T01:31:36.131 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-21T01:31:36.220 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-packaging. 2022-07-21T01:31:36.234 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-21T01:31:36.283 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-21T01:31:36.309 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package rbd-fuse. 2022-07-21T01:31:36.326 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:36.351 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:36.367 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:36.477 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-21T01:31:36.493 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-21T01:31:36.519 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-21T01:31:36.535 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-21T01:31:36.589 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastescript. 2022-07-21T01:31:36.603 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-21T01:31:36.647 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-21T01:31:36.668 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-21T01:31:36.837 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-21T01:31:36.980 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-21T01:31:36.997 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-packaging. 2022-07-21T01:31:37.011 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-21T01:31:37.020 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pluggy. 2022-07-21T01:31:37.037 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-21T01:31:37.079 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-21T01:31:37.083 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-21T01:31:37.131 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-21T01:31:37.148 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-21T01:31:37.257 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-21T01:31:37.325 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastescript. 2022-07-21T01:31:37.339 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-21T01:31:37.374 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-21T01:31:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-psutil. 2022-07-21T01:31:37.408 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-21T01:31:37.444 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-21T01:31:37.459 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:37.689 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-21T01:31:37.745 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pluggy. 2022-07-21T01:31:37.759 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-21T01:31:37.789 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-py. 2022-07-21T01:31:37.803 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-21T01:31:37.804 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-21T01:31:37.847 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-21T01:31:38.062 INFO:teuthology.orchestra.run.smithi003.stdout:Adding system user cephadm....done 2022-07-21T01:31:38.099 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-21T01:31:38.140 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-psutil. 2022-07-21T01:31:38.153 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-21T01:31:38.175 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-21T01:31:38.188 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-21T01:31:38.190 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-21T01:31:38.204 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-21T01:31:38.225 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-21T01:31:38.391 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-21T01:31:38.501 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-py. 2022-07-21T01:31:38.518 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-21T01:31:38.536 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pytest. 2022-07-21T01:31:38.551 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-21T01:31:38.559 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-21T01:31:38.574 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-21T01:31:38.586 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-21T01:31:38.701 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-21T01:31:38.876 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-21T01:31:38.915 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplejson. 2022-07-21T01:31:38.930 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-21T01:31:38.938 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-21T01:31:38.951 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-21T01:31:38.964 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-21T01:31:38.996 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-21T01:31:39.010 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-21T01:31:39.019 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-21T01:31:39.316 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pytest. 2022-07-21T01:31:39.329 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-21T01:31:39.345 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-21T01:31:39.351 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package radosgw. 2022-07-21T01:31:39.353 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-21T01:31:39.366 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-21T01:31:39.366 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:39.401 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:39.688 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-21T01:31:39.713 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-21T01:31:39.728 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplejson. 2022-07-21T01:31:39.743 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-21T01:31:39.795 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-21T01:31:39.989 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-21T01:31:40.006 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-21T01:31:40.115 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-21T01:31:40.199 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package radosgw. 2022-07-21T01:31:40.212 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:40.248 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:40.299 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-21T01:31:40.375 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-21T01:31:40.421 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package rbd-fuse. 2022-07-21T01:31:40.437 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:40.478 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:40.600 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-21T01:31:40.743 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-21T01:31:40.775 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-21T01:31:40.893 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-21T01:31:41.036 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-21T01:31:41.125 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-21T01:31:41.144 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:41.242 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-21T01:31:41.319 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package rbd-fuse. 2022-07-21T01:31:41.332 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-21T01:31:41.343 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-21T01:31:41.377 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:41.412 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-21T01:31:41.420 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:41.521 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-21T01:31:41.536 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:41.678 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-21T01:31:41.806 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-21T01:31:41.808 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-21T01:31:41.934 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:41.990 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-21T01:31:42.073 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-21T01:31:42.108 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-21T01:31:42.114 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user cephadm....done 2022-07-21T01:31:42.228 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-21T01:31:42.234 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-21T01:31:42.292 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-21T01:31:42.358 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-21T01:31:42.402 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:42.588 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-21T01:31:42.610 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-21T01:31:42.742 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-21T01:31:42.747 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-21T01:31:42.881 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-21T01:31:43.014 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user cephadm....done 2022-07-21T01:31:43.018 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-21T01:31:43.024 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-21T01:31:43.159 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-21T01:31:43.159 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-21T01:31:43.296 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-21T01:31:43.361 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-21T01:31:43.486 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-21T01:31:43.527 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-21T01:31:43.637 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-21T01:31:43.638 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-21T01:31:43.795 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-21T01:31:43.804 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-21T01:31:43.896 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-21T01:31:43.964 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-21T01:31:44.040 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-21T01:31:44.105 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-21T01:31:44.198 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-21T01:31:44.240 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-21T01:31:44.257 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-21T01:31:44.325 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-21T01:31:44.533 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-21T01:31:44.541 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-21T01:31:44.625 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-21T01:31:44.634 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-21T01:31:44.667 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-21T01:31:44.843 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-21T01:31:44.902 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-21T01:31:44.926 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-21T01:31:44.960 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-21T01:31:45.153 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-21T01:31:45.161 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-21T01:31:45.194 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-21T01:31:45.361 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-21T01:31:45.429 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:45.479 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-21T01:31:45.498 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-21T01:31:45.646 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-21T01:31:45.722 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:45.738 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-21T01:31:45.764 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-21T01:31:45.891 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-21T01:31:45.898 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-21T01:31:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-21T01:31:46.048 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:46.182 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-21T01:31:46.201 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-21T01:31:46.340 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:46.358 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-21T01:31:46.466 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-21T01:31:46.467 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-21T01:31:46.483 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:46.561 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-21T01:31:46.687 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-21T01:31:46.743 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-21T01:31:46.759 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-21T01:31:46.793 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-21T01:31:46.813 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-21T01:31:47.052 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-21T01:31:47.110 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-21T01:31:47.128 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-21T01:31:47.173 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-21T01:31:47.229 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-21T01:31:47.345 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-21T01:31:47.378 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-21T01:31:47.450 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:47.505 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-21T01:31:47.568 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-21T01:31:47.696 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-21T01:31:47.913 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-21T01:31:47.980 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-21T01:31:47.989 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-21T01:31:48.181 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-21T01:31:48.408 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-21T01:31:48.457 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-21T01:31:48.524 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-21T01:31:48.709 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-21T01:31:48.766 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-21T01:31:48.808 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-21T01:31:48.985 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-21T01:31:49.100 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-21T01:31:49.111 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-21T01:31:49.168 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-21T01:31:49.192 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:49.218 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-21T01:31:49.301 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:49.402 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-21T01:31:49.403 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-21T01:31:49.470 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-21T01:31:49.494 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-21T01:31:49.688 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-21T01:31:49.746 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-21T01:31:49.788 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-21T01:31:49.847 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-21T01:31:49.887 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-21T01:31:49.964 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-21T01:31:50.123 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-21T01:31:50.131 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-21T01:31:50.180 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-21T01:31:50.257 INFO:teuthology.orchestra.run.smithi003.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-21T01:31:50.408 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-21T01:31:50.541 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-21T01:31:50.550 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-21T01:31:50.581 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-21T01:31:50.676 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-21T01:31:50.776 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:50.834 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-21T01:31:50.902 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-21T01:31:50.950 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-21T01:31:50.994 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-21T01:31:51.100 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:51.213 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:51.227 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-21T01:31:51.337 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-21T01:31:51.339 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-21T01:31:51.410 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-21T01:31:51.446 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-21T01:31:51.547 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-21T01:31:51.596 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-21T01:31:51.682 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:51.706 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:51.770 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-21T01:31:51.865 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-21T01:31:51.871 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-21T01:31:52.008 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-21T01:31:52.150 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:52.164 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-21T01:31:52.276 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-21T01:31:52.289 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-21T01:31:52.350 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-21T01:31:52.501 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-21T01:31:52.624 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-21T01:31:52.722 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:52.803 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-21T01:31:52.890 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-21T01:31:52.904 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-21T01:31:52.917 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-21T01:31:53.017 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-21T01:31:53.134 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-21T01:31:53.231 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-21T01:31:53.234 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-21T01:31:53.335 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-21T01:31:53.453 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:53.515 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-21T01:31:53.554 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-21T01:31:53.586 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-21T01:31:53.808 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-21T01:31:53.831 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:53.871 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-21T01:31:53.880 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-21T01:31:53.934 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-21T01:31:53.957 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:54.083 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-21T01:31:54.182 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:54.214 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-21T01:31:54.235 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-21T01:31:54.308 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:54.502 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-21T01:31:54.524 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-21T01:31:54.528 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-21T01:31:54.534 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-21T01:31:54.534 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-21T01:31:54.774 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-21T01:31:54.811 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-21T01:31:54.812 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-21T01:31:54.900 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-21T01:31:54.985 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:55.086 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-21T01:31:55.090 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-21T01:31:55.104 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-21T01:31:55.235 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-21T01:31:55.355 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-21T01:31:55.357 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-21T01:31:55.422 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:55.507 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-21T01:31:55.523 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-21T01:31:55.561 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-21T01:31:55.662 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-21T01:31:55.664 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-21T01:31:55.771 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-21T01:31:55.818 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-21T01:31:55.858 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:56.017 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-21T01:31:56.043 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:56.106 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-21T01:31:56.178 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-21T01:31:56.194 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:56.279 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-21T01:31:56.303 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:56.352 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-21T01:31:56.380 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-21T01:31:56.413 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-21T01:31:56.415 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:56.541 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-21T01:31:56.664 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-21T01:31:56.825 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-21T01:31:56.941 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:57.092 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-21T01:31:57.135 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:57.253 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:57.449 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:57.521 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:57.575 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-21T01:31:57.639 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-21T01:31:57.701 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-21T01:31:57.852 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-21T01:31:58.005 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:58.131 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:58.153 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:58.257 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-21T01:31:58.271 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-21T01:31:58.548 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:58.551 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:58.581 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-21T01:31:58.642 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-21T01:31:58.700 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:31:58.825 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-21T01:31:58.873 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:58.942 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-21T01:31:59.000 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:59.148 INFO:teuthology.orchestra.run.smithi171.stdout:Adding group ceph....done 2022-07-21T01:31:59.202 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-21T01:31:59.244 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-21T01:31:59.253 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-21T01:31:59.253 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-21T01:31:59.512 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-21T01:31:59.549 INFO:teuthology.orchestra.run.smithi171.stdout:Adding system user ceph....done 2022-07-21T01:31:59.562 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:59.662 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-21T01:31:59.718 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:31:59.726 INFO:teuthology.orchestra.run.smithi171.stdout:Setting system user ceph properties....done 2022-07-21T01:31:59.742 INFO:teuthology.orchestra.run.smithi171.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-21T01:31:59.821 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-21T01:31:59.835 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-21T01:31:59.894 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-21T01:31:59.981 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:00.090 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-21T01:32:00.120 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-21T01:32:00.164 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:00.224 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-21T01:32:00.282 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-21T01:32:00.408 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-21T01:32:00.447 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-21T01:32:00.592 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:00.681 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:00.743 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-21T01:32:00.799 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:00.824 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:00.950 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:01.062 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-21T01:32:01.076 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:01.091 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-21T01:32:01.092 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-21T01:32:01.201 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-21T01:32:01.455 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:01.571 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-21T01:32:01.581 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-21T01:32:01.723 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-21T01:32:01.727 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:01.833 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-21T01:32:01.864 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:01.973 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:02.107 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-21T01:32:02.134 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:02.142 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:32:02.268 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-21T01:32:02.285 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-21T01:32:02.308 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:02.420 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-21T01:32:02.434 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-21T01:32:02.546 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-21T01:32:02.569 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-21T01:32:02.579 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:02.832 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-21T01:32:02.833 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-21T01:32:02.864 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-21T01:32:02.870 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:02.991 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-21T01:32:03.005 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:03.237 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-21T01:32:03.237 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-21T01:32:03.259 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-21T01:32:03.387 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:03.458 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:03.542 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:03.668 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:03.673 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:03.689 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-21T01:32:03.690 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-21T01:32:03.799 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-21T01:32:03.912 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-21T01:32:03.912 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-21T01:32:03.940 INFO:teuthology.orchestra.run.smithi003.stdout:Adding group ceph....done 2022-07-21T01:32:04.083 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-21T01:32:04.143 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:04.278 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:04.316 INFO:teuthology.orchestra.run.smithi003.stdout:Adding system user ceph....done 2022-07-21T01:32:04.387 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:04.401 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-21T01:32:04.510 INFO:teuthology.orchestra.run.smithi003.stdout:Setting system user ceph properties....done 2022-07-21T01:32:04.513 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-21T01:32:04.526 INFO:teuthology.orchestra.run.smithi003.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-21T01:32:04.687 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-21T01:32:04.752 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-21T01:32:04.753 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-21T01:32:04.770 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:04.789 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-21T01:32:04.896 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.007 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-21T01:32:05.013 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.082 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-21T01:32:05.140 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-21T01:32:05.196 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.323 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.410 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.480 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.499 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.509 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-21T01:32:05.536 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.606 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.670 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.725 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-21T01:32:05.726 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-21T01:32:05.813 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-21T01:32:05.827 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-21T01:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-21T01:32:05.952 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:06.186 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:06.191 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-21T01:32:06.254 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:06.336 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:06.380 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-21T01:32:06.462 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:06.501 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:06.550 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:06.624 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-21T01:32:06.626 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:06.893 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-21T01:32:06.899 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-21T01:32:06.920 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:07.046 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-21T01:32:07.307 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:07.335 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:07.547 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-21T01:32:07.547 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-21T01:32:07.851 INFO:teuthology.orchestra.run.smithi174.stdout:Adding group ceph....done 2022-07-21T01:32:07.974 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:07.986 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:08.186 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user ceph....done 2022-07-21T01:32:08.218 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-21T01:32:08.218 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-21T01:32:08.354 INFO:teuthology.orchestra.run.smithi174.stdout:Setting system user ceph properties....done 2022-07-21T01:32:08.368 INFO:teuthology.orchestra.run.smithi174.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-21T01:32:08.509 INFO:teuthology.orchestra.run.smithi143.stdout:Adding group ceph....done 2022-07-21T01:32:08.512 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-21T01:32:08.663 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:08.824 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-21T01:32:08.833 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:08.887 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user ceph....done 2022-07-21T01:32:09.055 INFO:teuthology.orchestra.run.smithi143.stdout:Setting system user ceph properties....done 2022-07-21T01:32:09.070 INFO:teuthology.orchestra.run.smithi143.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-21T01:32:09.214 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-21T01:32:09.275 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:09.313 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-21T01:32:09.313 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-21T01:32:09.401 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:09.528 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-21T01:32:09.586 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-21T01:32:09.697 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-21T01:32:09.697 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-21T01:32:09.774 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:09.900 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:09.943 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:10.013 INFO:teuthology.orchestra.run.smithi171.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-21T01:32:10.015 DEBUG:teuthology.orchestra.run.smithi171:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-21T01:32:10.060 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:10.067 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:10.082 INFO:teuthology.orchestra.run.smithi171.stdout:Reading package lists... 2022-07-21T01:32:10.260 INFO:teuthology.orchestra.run.smithi171.stdout:Building dependency tree... 2022-07-21T01:32:10.261 INFO:teuthology.orchestra.run.smithi171.stdout:Reading state information... 2022-07-21T01:32:10.297 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-21T01:32:10.298 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-21T01:32:10.325 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-21T01:32:10.326 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-21T01:32:10.336 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:10.419 INFO:teuthology.orchestra.run.smithi171.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-21T01:32:10.420 INFO:teuthology.orchestra.run.smithi171.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-21T01:32:10.420 INFO:teuthology.orchestra.run.smithi171.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-21T01:32:10.421 INFO:teuthology.orchestra.run.smithi171.stdout:The following packages were automatically installed and are no longer required: 2022-07-21T01:32:10.421 INFO:teuthology.orchestra.run.smithi171.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-21T01:32:10.421 INFO:teuthology.orchestra.run.smithi171.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-21T01:32:10.422 INFO:teuthology.orchestra.run.smithi171.stdout:Suggested packages: 2022-07-21T01:32:10.423 INFO:teuthology.orchestra.run.smithi171.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-21T01:32:10.454 INFO:teuthology.orchestra.run.smithi171.stdout:The following NEW packages will be installed: 2022-07-21T01:32:10.455 INFO:teuthology.orchestra.run.smithi171.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-21T01:32:10.455 INFO:teuthology.orchestra.run.smithi171.stdout: libssl-dev 2022-07-21T01:32:10.636 INFO:teuthology.orchestra.run.smithi171.stdout:0 upgraded, 10 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:32:10.637 INFO:teuthology.orchestra.run.smithi171.stdout:Need to get 2,914 kB of archives. 2022-07-21T01:32:10.637 INFO:teuthology.orchestra.run.smithi171.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-21T01:32:10.637 INFO:teuthology.orchestra.run.smithi171.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-21T01:32:10.678 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-21T01:32:10.744 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:10.862 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:10.954 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:10.962 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:11.106 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-21T01:32:11.109 INFO:teuthology.orchestra.run.smithi171.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-21T01:32:11.155 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:11.247 INFO:teuthology.orchestra.run.smithi171.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-21T01:32:11.255 INFO:teuthology.orchestra.run.smithi171.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-21T01:32:11.272 INFO:teuthology.orchestra.run.smithi171.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-21T01:32:11.280 INFO:teuthology.orchestra.run.smithi171.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-21T01:32:11.280 INFO:teuthology.orchestra.run.smithi171.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-21T01:32:11.286 INFO:teuthology.orchestra.run.smithi171.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-21T01:32:11.286 INFO:teuthology.orchestra.run.smithi171.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-21T01:32:11.287 INFO:teuthology.orchestra.run.smithi171.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-21T01:32:11.295 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-21T01:32:11.375 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-21T01:32:11.376 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-21T01:32:11.399 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-21T01:32:11.715 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:11.744 INFO:teuthology.orchestra.run.smithi171.stdout:Fetched 2,914 kB in 1s (3,146 kB/s) 2022-07-21T01:32:11.839 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:11.884 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package flex. 2022-07-21T01:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-21T01:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:11.940 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-21T01:32:11.940 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-21T01:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-21T01:32:12.062 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-21T01:32:12.063 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-21T01:32:12.450 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:12.480 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package bison. 2022-07-21T01:32:12.495 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-21T01:32:12.516 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:12.534 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-21T01:32:12.650 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:12.698 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-21T01:32:12.699 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-21T01:32:12.935 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-21T01:32:12.950 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-21T01:32:13.036 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-21T01:32:13.152 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-21T01:32:13.152 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-21T01:32:13.294 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:13.398 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package dump. 2022-07-21T01:32:13.412 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-21T01:32:13.454 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:13.455 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking dump (0.4b46-6) ... 2022-07-21T01:32:13.610 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:13.735 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:13.880 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package indent. 2022-07-21T01:32:13.897 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-21T01:32:13.926 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-21T01:32:13.926 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-21T01:32:13.936 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking indent (2.2.12-1) ... 2022-07-21T01:32:13.984 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:14.118 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-21T01:32:14.215 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-21T01:32:14.230 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-21T01:32:14.273 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-21T01:32:14.364 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:14.515 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:14.519 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-21T01:32:14.523 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-21T01:32:14.523 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-21T01:32:14.534 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-21T01:32:14.535 INFO:teuthology.orchestra.run.smithi003.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-21T01:32:14.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-21T01:32:14.606 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2022-07-21T01:32:14.610 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-21T01:32:14.658 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:14.788 INFO:teuthology.orchestra.run.smithi003.stdout:Building dependency tree... 2022-07-21T01:32:14.789 INFO:teuthology.orchestra.run.smithi003.stdout:Reading state information... 2022-07-21T01:32:14.886 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-21T01:32:14.886 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-21T01:32:14.939 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-21T01:32:14.953 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:14.954 INFO:teuthology.orchestra.run.smithi003.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-21T01:32:14.954 INFO:teuthology.orchestra.run.smithi003.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-21T01:32:14.954 INFO:teuthology.orchestra.run.smithi003.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-21T01:32:14.954 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages were automatically installed and are no longer required: 2022-07-21T01:32:14.955 INFO:teuthology.orchestra.run.smithi003.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-21T01:32:14.955 INFO:teuthology.orchestra.run.smithi003.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-21T01:32:14.956 INFO:teuthology.orchestra.run.smithi003.stdout:Suggested packages: 2022-07-21T01:32:14.956 INFO:teuthology.orchestra.run.smithi003.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-21T01:32:14.963 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:14.987 INFO:teuthology.orchestra.run.smithi003.stdout:The following NEW packages will be installed: 2022-07-21T01:32:14.988 INFO:teuthology.orchestra.run.smithi003.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-21T01:32:14.989 INFO:teuthology.orchestra.run.smithi003.stdout: libssl-dev 2022-07-21T01:32:15.016 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:15.044 INFO:teuthology.orchestra.run.smithi003.stdout:0 upgraded, 10 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:32:15.044 INFO:teuthology.orchestra.run.smithi003.stdout:Need to get 2,914 kB of archives. 2022-07-21T01:32:15.045 INFO:teuthology.orchestra.run.smithi003.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-21T01:32:15.045 INFO:teuthology.orchestra.run.smithi003.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-21T01:32:15.081 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:15.178 INFO:teuthology.orchestra.run.smithi003.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-21T01:32:15.182 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:15.268 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-21T01:32:15.281 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:15.286 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-21T01:32:15.289 INFO:teuthology.orchestra.run.smithi003.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-21T01:32:15.309 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-21T01:32:15.324 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-21T01:32:15.326 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:15.326 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:15.328 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-21T01:32:15.329 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-21T01:32:15.346 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-21T01:32:15.347 INFO:teuthology.orchestra.run.smithi003.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-21T01:32:15.347 INFO:teuthology.orchestra.run.smithi003.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-21T01:32:15.485 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:15.588 INFO:teuthology.orchestra.run.smithi171.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-21T01:32:15.602 INFO:teuthology.orchestra.run.smithi171.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-21T01:32:15.612 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-21T01:32:15.626 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-21T01:32:15.637 INFO:teuthology.orchestra.run.smithi171.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-21T01:32:15.755 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-21T01:32:15.950 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 2,914 kB in 1s (4,871 kB/s) 2022-07-21T01:32:16.048 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-21T01:32:16.107 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package flex. 2022-07-21T01:32:16.138 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-21T01:32:16.152 INFO:teuthology.orchestra.run.smithi003.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-21T01:32:16.155 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:16.197 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-21T01:32:16.260 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-21T01:32:16.387 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-21T01:32:16.488 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-21T01:32:16.614 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-21T01:32:16.702 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package bison. 2022-07-21T01:32:16.718 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-21T01:32:16.764 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-21T01:32:16.774 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:16.900 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up dump (0.4b46-6) ... 2022-07-21T01:32:16.996 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-21T01:32:17.044 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-21T01:32:17.133 INFO:teuthology.orchestra.run.smithi171.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-21T01:32:17.164 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-21T01:32:17.179 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-21T01:32:17.196 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up indent (2.2.12-1) ... 2022-07-21T01:32:17.281 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-21T01:32:17.314 INFO:teuthology.orchestra.run.smithi171.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:17.449 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-21T01:32:17.618 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package dump. 2022-07-21T01:32:17.632 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-21T01:32:17.634 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-21T01:32:17.667 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking dump (0.4b46-6) ... 2022-07-21T01:32:17.979 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package indent. 2022-07-21T01:32:17.996 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-21T01:32:18.046 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking indent (2.2.12-1) ... 2022-07-21T01:32:18.312 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-21T01:32:18.315 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-21T01:32:18.331 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-21T01:32:18.374 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-21T01:32:18.694 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-21T01:32:18.709 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-21T01:32:18.732 INFO:teuthology.orchestra.run.smithi174.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-21T01:32:18.735 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-21T01:32:18.743 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-21T01:32:18.807 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2022-07-21T01:32:18.926 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-21T01:32:18.986 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2022-07-21T01:32:18.986 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2022-07-21T01:32:19.142 INFO:teuthology.orchestra.run.smithi174.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-21T01:32:19.143 INFO:teuthology.orchestra.run.smithi174.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-21T01:32:19.143 INFO:teuthology.orchestra.run.smithi174.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-21T01:32:19.143 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2022-07-21T01:32:19.144 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-21T01:32:19.144 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-21T01:32:19.145 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2022-07-21T01:32:19.145 INFO:teuthology.orchestra.run.smithi174.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-21T01:32:19.177 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2022-07-21T01:32:19.178 INFO:teuthology.orchestra.run.smithi174.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-21T01:32:19.179 INFO:teuthology.orchestra.run.smithi174.stdout: libssl-dev 2022-07-21T01:32:19.360 INFO:teuthology.orchestra.run.smithi174.stdout:0 upgraded, 10 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:32:19.361 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 2,914 kB of archives. 2022-07-21T01:32:19.361 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-21T01:32:19.361 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-21T01:32:19.375 INFO:teuthology.orchestra.run.smithi143.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-21T01:32:19.378 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-21T01:32:19.447 INFO:teuthology.orchestra.run.smithi171.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-21T01:32:19.448 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2022-07-21T01:32:19.531 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-21T01:32:19.546 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:19.589 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:19.626 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2022-07-21T01:32:19.627 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2022-07-21T01:32:19.781 INFO:teuthology.orchestra.run.smithi143.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-21T01:32:19.781 INFO:teuthology.orchestra.run.smithi143.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-21T01:32:19.781 INFO:teuthology.orchestra.run.smithi143.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-21T01:32:19.781 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages were automatically installed and are no longer required: 2022-07-21T01:32:19.783 INFO:teuthology.orchestra.run.smithi143.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-21T01:32:19.783 INFO:teuthology.orchestra.run.smithi143.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-21T01:32:19.784 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2022-07-21T01:32:19.784 INFO:teuthology.orchestra.run.smithi143.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-21T01:32:19.813 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2022-07-21T01:32:19.814 INFO:teuthology.orchestra.run.smithi143.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-21T01:32:19.815 INFO:teuthology.orchestra.run.smithi143.stdout: libssl-dev 2022-07-21T01:32:19.840 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-21T01:32:19.866 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-21T01:32:19.868 INFO:teuthology.orchestra.run.smithi143.stdout:0 upgraded, 10 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T01:32:19.868 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 2,914 kB of archives. 2022-07-21T01:32:19.869 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-21T01:32:19.869 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-21T01:32:19.882 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:19.925 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:19.978 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-21T01:32:19.987 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-21T01:32:20.000 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-21T01:32:20.004 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-21T01:32:20.011 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-21T01:32:20.012 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-21T01:32:20.016 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-21T01:32:20.018 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-21T01:32:20.018 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-21T01:32:20.111 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-21T01:32:20.114 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-21T01:32:20.135 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-21T01:32:20.137 INFO:teuthology.orchestra.run.smithi171.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-21T01:32:20.140 DEBUG:teuthology.parallel:result is None 2022-07-21T01:32:20.153 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-21T01:32:20.154 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-21T01:32:20.169 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-21T01:32:20.171 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-21T01:32:20.172 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-21T01:32:20.172 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-21T01:32:20.183 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-21T01:32:20.227 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-21T01:32:20.475 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 2,914 kB in 1s (3,120 kB/s) 2022-07-21T01:32:20.601 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package flex. 2022-07-21T01:32:20.648 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-21T01:32:20.652 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:20.684 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-21T01:32:20.707 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-21T01:32:20.769 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 2,914 kB in 1s (4,887 kB/s) 2022-07-21T01:32:20.807 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-21T01:32:20.899 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package flex. 2022-07-21T01:32:20.943 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-21T01:32:20.946 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:20.958 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-21T01:32:20.980 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-21T01:32:21.067 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-21T01:32:21.168 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-21T01:32:21.195 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package bison. 2022-07-21T01:32:21.210 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-21T01:32:21.269 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:21.274 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-21T01:32:21.378 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up dump (0.4b46-6) ... 2022-07-21T01:32:21.468 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package bison. 2022-07-21T01:32:21.474 INFO:teuthology.orchestra.run.smithi003.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-21T01:32:21.482 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-21T01:32:21.531 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-21T01:32:21.555 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-21T01:32:21.645 INFO:teuthology.orchestra.run.smithi003.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-21T01:32:21.699 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-21T01:32:21.714 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-21T01:32:21.715 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up indent (2.2.12-1) ... 2022-07-21T01:32:21.800 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-21T01:32:21.841 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:21.956 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-21T01:32:21.968 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-21T01:32:21.970 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-21T01:32:22.057 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-21T01:32:22.086 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-21T01:32:22.119 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package dump. 2022-07-21T01:32:22.134 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-21T01:32:22.169 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking dump (0.4b46-6) ... 2022-07-21T01:32:22.418 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package dump. 2022-07-21T01:32:22.435 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-21T01:32:22.464 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package indent. 2022-07-21T01:32:22.477 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking dump (0.4b46-6) ... 2022-07-21T01:32:22.478 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-21T01:32:22.522 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking indent (2.2.12-1) ... 2022-07-21T01:32:22.782 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package indent. 2022-07-21T01:32:22.799 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-21T01:32:22.808 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-21T01:32:22.822 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-21T01:32:22.840 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking indent (2.2.12-1) ... 2022-07-21T01:32:22.866 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-21T01:32:23.069 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-21T01:32:23.076 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-21T01:32:23.084 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-21T01:32:23.092 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-21T01:32:23.119 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-21T01:32:23.138 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-21T01:32:23.412 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-21T01:32:23.426 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-21T01:32:23.462 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-21T01:32:23.481 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-21T01:32:23.494 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:23.539 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:23.791 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-21T01:32:23.804 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:23.841 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:23.866 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-21T01:32:23.882 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:23.924 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:24.005 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-21T01:32:24.077 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-21T01:32:24.091 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-21T01:32:24.135 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-21T01:32:24.201 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-21T01:32:24.216 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-21T01:32:24.260 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:24.512 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-21T01:32:24.526 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-21T01:32:24.570 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-21T01:32:24.600 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-21T01:32:24.816 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-21T01:32:24.915 INFO:teuthology.orchestra.run.smithi003.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-21T01:32:24.919 DEBUG:teuthology.parallel:result is None 2022-07-21T01:32:24.967 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-21T01:32:25.038 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-21T01:32:25.092 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-21T01:32:25.167 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-21T01:32:25.219 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-21T01:32:25.318 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-21T01:32:25.336 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:25.437 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up dump (0.4b46-6) ... 2022-07-21T01:32:25.443 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-21T01:32:25.534 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-21T01:32:25.544 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-21T01:32:25.599 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-21T01:32:25.645 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:25.713 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-21T01:32:25.776 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up indent (2.2.12-1) ... 2022-07-21T01:32:25.779 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up dump (0.4b46-6) ... 2022-07-21T01:32:25.866 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-21T01:32:25.901 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:25.931 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-21T01:32:26.012 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-21T01:32:26.020 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-21T01:32:26.091 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up indent (2.2.12-1) ... 2022-07-21T01:32:26.163 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-21T01:32:26.241 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-21T01:32:26.368 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-21T01:32:26.511 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-21T01:32:28.007 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-21T01:32:28.355 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-21T01:32:28.680 INFO:teuthology.orchestra.run.smithi174.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-21T01:32:28.682 DEBUG:teuthology.parallel:result is None 2022-07-21T01:32:29.137 INFO:teuthology.orchestra.run.smithi143.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-21T01:32:29.140 DEBUG:teuthology.parallel:result is None 2022-07-21T01:32:29.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo lsb_release -is 2022-07-21T01:32:29.192 INFO:teuthology.orchestra.run.smithi003.stdout:Ubuntu 2022-07-21T01:32:29.198 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:32:29.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-21T01:32:29.375 DEBUG:teuthology.orchestra.run.smithi003:> dpkg-query -W -f '${Version}' ceph 2022-07-21T01:32:29.398 INFO:teuthology.orchestra.run.smithi003.stdout:15.2.16-336-g57544da1-1focal 2022-07-21T01:32:29.399 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-21T01:32:29.400 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-21T01:32:29.400 DEBUG:teuthology.orchestra.run.smithi143:> sudo lsb_release -is 2022-07-21T01:32:29.453 INFO:teuthology.orchestra.run.smithi143.stdout:Ubuntu 2022-07-21T01:32:29.459 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:32:29.459 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-21T01:32:29.609 DEBUG:teuthology.orchestra.run.smithi143:> dpkg-query -W -f '${Version}' ceph 2022-07-21T01:32:29.633 INFO:teuthology.orchestra.run.smithi143.stdout:15.2.16-336-g57544da1-1focal 2022-07-21T01:32:29.634 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-21T01:32:29.634 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-21T01:32:29.635 DEBUG:teuthology.orchestra.run.smithi171:> sudo lsb_release -is 2022-07-21T01:32:29.688 INFO:teuthology.orchestra.run.smithi171.stdout:Ubuntu 2022-07-21T01:32:29.693 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:32:29.694 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-21T01:32:29.853 DEBUG:teuthology.orchestra.run.smithi171:> dpkg-query -W -f '${Version}' ceph 2022-07-21T01:32:29.875 INFO:teuthology.orchestra.run.smithi171.stdout:15.2.16-336-g57544da1-1focal 2022-07-21T01:32:29.876 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-21T01:32:29.876 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-21T01:32:29.877 DEBUG:teuthology.orchestra.run.smithi174:> sudo lsb_release -is 2022-07-21T01:32:29.930 INFO:teuthology.orchestra.run.smithi174.stdout:Ubuntu 2022-07-21T01:32:29.935 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-21T01:32:29.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-21T01:32:30.104 DEBUG:teuthology.orchestra.run.smithi174:> dpkg-query -W -f '${Version}' ceph 2022-07-21T01:32:30.127 INFO:teuthology.orchestra.run.smithi174.stdout:15.2.16-336-g57544da1-1focal 2022-07-21T01:32:30.128 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-21T01:32:30.129 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-21T01:32:30.130 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-07-21T01:32:30.130 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-07-21T01:32:30.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-21T01:32:30.149 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2022-07-21T01:32:30.149 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-21T01:32:30.167 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-07-21T01:32:30.168 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-21T01:32:30.186 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2022-07-21T01:32:30.187 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-21T01:32:30.202 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-07-21T01:32:30.203 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-07-21T01:32:30.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2022-07-21T01:32:30.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-21T01:32:30.236 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2022-07-21T01:32:30.237 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2022-07-21T01:32:30.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-21T01:32:30.263 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-07-21T01:32:30.264 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2022-07-21T01:32:30.281 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-21T01:32:30.293 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2022-07-21T01:32:30.293 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2022-07-21T01:32:30.310 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-21T01:32:30.324 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-07-21T01:32:30.325 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-07-21T01:32:30.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-21T01:32:30.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-21T01:32:30.401 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2022-07-21T01:32:30.402 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-21T01:32:30.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-21T01:32:30.432 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-07-21T01:32:30.433 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-21T01:32:30.449 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-21T01:32:30.507 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2022-07-21T01:32:30.507 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-21T01:32:30.523 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-21T01:32:30.579 INFO:teuthology.run_tasks:Running task ceph... 2022-07-21T01:32:30.646 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-07-21T01:32:30.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2022-07-21T01:32:30.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /var/log/ceph 2022-07-21T01:32:30.653 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /var/log/ceph 2022-07-21T01:32:30.655 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /var/log/ceph 2022-07-21T01:32:30.670 INFO:tasks.ceph:Disabling ceph logrotate... 2022-07-21T01:32:30.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-21T01:32:30.707 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-21T01:32:30.710 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-21T01:32:30.713 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-21T01:32:30.728 INFO:tasks.ceph:Creating extra log directories... 2022-07-21T01:32:30.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-21T01:32:30.731 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-21T01:32:30.767 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-21T01:32:30.772 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-21T01:32:30.795 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2022-07-21T01:32:30.796 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2022-07-21T01:32:30.796 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2022-07-21T01:32:30.797 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-07-21T01:32:30.797 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-21T01:32:30.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-21T01:32:30.884 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2022-07-21T01:32:30.885 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-21T01:32:30.892 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-21T01:32:30.929 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-07-21T01:32:30.930 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-21T01:32:30.937 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-21T01:32:30.977 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2022-07-21T01:32:30.978 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-21T01:32:30.985 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-21T01:32:31.030 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-07-21T01:32:31.030 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-07-21T01:32:31.031 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359', 'branch': 'octopus-release', 'description': 'fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}', 'email': None, 'first_in_suite': False, 'job_id': '6941359', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'octopus-release'}, 'ceph': {'cephfs_ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'octopus-release', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'], ['client.0'], ['client.1']], 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'octopus-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0', 'targets': {'smithi003.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDinwZIIfMdjuwjbCLOcoWbvzTlWjYXhwE3nvoFhlvF/iTjpSA7LVyBtgW4GvqT4FiRZkznnU7/A4tRfB4Szkl7hkZCMjrXC8P3Eah66sG1OfWE8EnQd4VbqZgj3BbOpAK4+U99qnwkISjnMOM9jQBAQRSUjlzN7Rv0EWxrb4AEXtl+lFNmUevGe8OtJs6s0KJF3N5ZgkzOzR6Tdi0yePraZFoHDyA5juvwSiO3YN1LUz0Ro9DtoMm8i0DATn7d3drfBMoDrHTdkWz44pCXInZVHBYQ3zSJQ0gvoRmOCpvAtSW/pam6BX/S9qQYPZfhtT9WnK29YnziXJbQ7zxdrd8GoGaLzufTqJIuCgFNheLOfBmOMPCbfGE+XfK3MOUfAUmvph9PbAjdpJX9N+1z4m8wUg30TRf8uD3atPjGKZPp7M+DKncEOFSx3FMcUGoSaIAzGa8EirtuqaweIlYCRPEObcU00eQHD+rj1FJJz9cBuovU0p1z6+++A22h5fOb7I8=', 'smithi143.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDESHLkWB2ibBysFJ6hMDWnNs8GyPMOGDq9VFKWZSL5NtLe6TFdfQTc0H9l7PlL2IMQ97/bMZ3W3du8ZapyZpW67dEWOFbZqr1Ed2qetWMHCpsj4418kEP2XFLZQCO+ytBYw2yBVKGzVQoNuu0qbUke+BKO2o0Sxe0QTVmC5U0cVwHO39iOt/CZtRfSGZ+RUzLyjBI97i2NYtcB1TroAK174o2fe+f8leEczVwf4GBbTEJ+kA1OgbzkZaBFB+TKUEVrtghWN4nQuumVGIUs+P9Qlel7GhN1EkPaxgYiorP3bUo/7k8Tl+3CTVFuwnOug4Fl5t+aJ8cfPwHPV6be4PIpCIiiP9MTueNN1C6lBapRO926FP99vhA+VYYxruwAlplEUnaNdh7K6j58KRun0K/hbRR+9aERG6t0ly96WU2oE5UxjzRIQHV9q+a+cu1gdyOh8lKqMK2zwV3qZ4GRpOZJ12Z1MgDs2UeuYHE9diYFr8QtetIysFJxCWszDikfHYs=', 'smithi171.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUDw9VoK2vcRLET5lYnBXRmv2lJkTKWQTmGSFDaibmQf5RfYJSQ2Od4HMUiqrEJwpLNXH5ZzD9hIZX7fxPGahS0SDeTJym62DMnpQIDl9bp7u6YP3H6I+kT/t1ND78jor51IOpOloIykT3kO2q6YE/OyRoNFFn+6TnP+3bP2tsB7j1aF9hQBUAAAWCB9E2vz+nplVprQRlgx4xzW0kMA3XJBOsQsL8S9uPw3rUXvFmNf2DLKW9O3q4lgp9hix3oP5p7q1lc1OhuFjnYMbXCjvU9KAKx52p+YlGy5mEiCSL76EH18t1x1IIgvxyXaGtBYhnaS+sg04XBLYceCupkBHZ/J3WbpKW6KKU3tKdM52HOIy02hppqWLx9KKdAt6xqDzX/P31H5CA4V/ZCrzyY5n9uSM0pzZE+HGoQIu3P7AE6IkfD0AHQWZi3gIrm+Wsj9WBYZQxhX9ARIj2R682MBXywck0IWa5aGdNSVQAt8NdS6EdQAHSmwlQklRlxb2NHE8=', 'smithi174.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAJC1IbSPRKdIO9nqlOgWHwU4ohDj2qmW41hWsR5W1r4XC44vafIo9WIUXILqxmQY2dJRhrJ0iAIGf2aeNv5/rTtO+DtzfPi8z6GlxbmL33Sm22SphH5qKu6ddrxumT6jaan06s/GIgfZpt8/LbUPjbQfUc0q1JRsgTN1X9FdN6/Iz5j+pdNGDKRMt3d/DG2gDC2c812P/8WO/c1SsGJq7tMEE6djNpfJQrdyLpMSSA10YBSvOkhLirvyWLljCZLZ7Jk8WL6KlqKCTF3HJ1Le1iiRlhJ1ylR4Tem0Oy/BA+of3s+jEnnTHcBcEPchxMDhUC7cqRh5Uxpn/yYDu+yBPWctGajYt2HgQ3EK6RJ0JuvARD5zyj8cnI9si5M6sXmGkeNUVtbrJ6RmHtJCr/zZm4/IcZFTSS7vIcFVQFc6fH/kM1zBF9h8uQLvvVL60ZUrRDuAL6jEt/9xk+6THonxkrvJpceiWl9Q/sq9G/ZD/UzZVKkrCfBlAxQRKRiJIqgc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_misc']}}], 'teuthology_branch': 'main', 'teuthology_sha1': '89d874d555f56b1c1f383df72487aecec0ee6cf9', 'timestamp': '2022-07-20_20:56:09', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-07-21T01:32:31.032 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-21T01:32:31.034 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-21T01:32:31.037 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-21T01:32:31.040 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-21T01:32:31.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2022-07-21T01:32:31.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/run/ceph 2022-07-21T01:32:31.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/run/ceph 2022-07-21T01:32:31.088 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/run/ceph 2022-07-21T01:32:31.103 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-07-21T01:32:31.103 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2022-07-21T01:32:31.109 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-21T01:32:31.110 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-21T01:32:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-21T01:32:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-21T01:32:31.117 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 360 Links: 1 2022-07-21T01:32:31.117 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-21T01:32:31.117 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2022-07-21 01:29:31.276420004 +0000 2022-07-21T01:32:31.117 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2022-07-21 01:29:30.940419997 +0000 2022-07-21T01:32:31.118 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2022-07-21 01:29:30.940419997 +0000 2022-07-21T01:32:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2022-07-21T01:32:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2022-07-21T01:32:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2022-07-21T01:32:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000387847 s, 1.3 MB/s 2022-07-21T01:32:31.130 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-21T01:32:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-21T01:32:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-21T01:32:31.185 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 367 Links: 1 2022-07-21T01:32:31.186 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-21T01:32:31.186 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2022-07-21 01:29:31.780420015 +0000 2022-07-21T01:32:31.186 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2022-07-21 01:29:31.480420008 +0000 2022-07-21T01:32:31.186 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2022-07-21 01:29:31.480420008 +0000 2022-07-21T01:32:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2022-07-21T01:32:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2022-07-21T01:32:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2022-07-21T01:32:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000388374 s, 1.3 MB/s 2022-07-21T01:32:31.200 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-21T01:32:31.206 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-21T01:32:31.206 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-21T01:32:31.207 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 372 Links: 1 2022-07-21T01:32:31.207 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-21T01:32:31.208 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2022-07-21 01:29:32.364420027 +0000 2022-07-21T01:32:31.208 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2022-07-21 01:29:32.012420020 +0000 2022-07-21T01:32:31.209 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2022-07-21 01:29:32.012420020 +0000 2022-07-21T01:32:31.209 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2022-07-21T01:32:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2022-07-21T01:32:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2022-07-21T01:32:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000382955 s, 1.3 MB/s 2022-07-21T01:32:31.220 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-21T01:32:31.226 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-21T01:32:31.226 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-21T01:32:31.226 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 6h/6d Inode: 379 Links: 1 2022-07-21T01:32:31.227 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-21T01:32:31.227 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2022-07-21 01:29:32.884420038 +0000 2022-07-21T01:32:31.227 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2022-07-21 01:29:32.584420032 +0000 2022-07-21T01:32:31.228 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2022-07-21 01:29:32.584420032 +0000 2022-07-21T01:32:31.228 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2022-07-21T01:32:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2022-07-21T01:32:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2022-07-21T01:32:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000356911 s, 1.4 MB/s 2022-07-21T01:32:31.241 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:31.241 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2022-07-21T01:32:31.242 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2022-07-21T01:32:31.249 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-21T01:32:31.250 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-21T01:32:31.257 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-21T01:32:31.258 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-21T01:32:31.258 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 360 Links: 1 2022-07-21T01:32:31.258 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-21T01:32:31.258 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2022-07-21 01:29:31.620759597 +0000 2022-07-21T01:32:31.259 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2022-07-21 01:29:31.292759606 +0000 2022-07-21T01:32:31.259 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2022-07-21 01:29:31.292759606 +0000 2022-07-21T01:32:31.259 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2022-07-21T01:32:31.268 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2022-07-21T01:32:31.268 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2022-07-21T01:32:31.269 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000427334 s, 1.2 MB/s 2022-07-21T01:32:31.273 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-21T01:32:31.280 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-21T01:32:31.281 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-21T01:32:31.281 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 366 Links: 1 2022-07-21T01:32:31.281 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-21T01:32:31.281 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2022-07-21 01:29:32.204759581 +0000 2022-07-21T01:32:31.282 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2022-07-21 01:29:31.832759591 +0000 2022-07-21T01:32:31.282 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2022-07-21 01:29:31.832759591 +0000 2022-07-21T01:32:31.282 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2022-07-21T01:32:31.290 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2022-07-21T01:32:31.290 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2022-07-21T01:32:31.290 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000368866 s, 1.4 MB/s 2022-07-21T01:32:31.294 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-21T01:32:31.343 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-21T01:32:31.344 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-21T01:32:31.344 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 372 Links: 1 2022-07-21T01:32:31.344 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-21T01:32:31.345 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2022-07-21 01:29:32.792759564 +0000 2022-07-21T01:32:31.345 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2022-07-21 01:29:32.408759575 +0000 2022-07-21T01:32:31.345 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2022-07-21 01:29:32.408759575 +0000 2022-07-21T01:32:31.345 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2022-07-21T01:32:31.354 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2022-07-21T01:32:31.354 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2022-07-21T01:32:31.355 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000397527 s, 1.3 MB/s 2022-07-21T01:32:31.359 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-21T01:32:31.366 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-21T01:32:31.366 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-21T01:32:31.366 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 377 Links: 1 2022-07-21T01:32:31.367 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-21T01:32:31.367 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2022-07-21 01:29:33.340759549 +0000 2022-07-21T01:32:31.367 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2022-07-21 01:29:33.016759558 +0000 2022-07-21T01:32:31.367 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2022-07-21 01:29:33.016759558 +0000 2022-07-21T01:32:31.368 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2022-07-21T01:32:31.373 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2022-07-21T01:32:31.374 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2022-07-21T01:32:31.374 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000423571 s, 1.2 MB/s 2022-07-21T01:32:31.378 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:31.379 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-07-21T01:32:31.379 INFO:tasks.ceph:Generating config... 2022-07-21T01:32:31.381 INFO:tasks.ceph:[client] client mount timeout = 600 2022-07-21T01:32:31.381 INFO:tasks.ceph:[client] debug client = 20 2022-07-21T01:32:31.381 INFO:tasks.ceph:[client] debug ms = 1 2022-07-21T01:32:31.382 INFO:tasks.ceph:[client] rados mon op timeout = 15m 2022-07-21T01:32:31.382 INFO:tasks.ceph:[client] rados osd op timeout = 15m 2022-07-21T01:32:31.383 INFO:tasks.ceph:[mds] debug mds = 20 2022-07-21T01:32:31.383 INFO:tasks.ceph:[mds] debug ms = 1 2022-07-21T01:32:31.383 INFO:tasks.ceph:[mds] mds bal frag = True 2022-07-21T01:32:31.384 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2022-07-21T01:32:31.384 INFO:tasks.ceph:[mds] mds bal merge size = 5 2022-07-21T01:32:31.384 INFO:tasks.ceph:[mds] mds bal split bits = 3 2022-07-21T01:32:31.385 INFO:tasks.ceph:[mds] mds bal split size = 100 2022-07-21T01:32:31.385 INFO:tasks.ceph:[mds] mds debug frag = True 2022-07-21T01:32:31.385 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2022-07-21T01:32:31.386 INFO:tasks.ceph:[mds] mds op complaint time = 180 2022-07-21T01:32:31.386 INFO:tasks.ceph:[mds] mds verify scatter = True 2022-07-21T01:32:31.386 INFO:tasks.ceph:[mds] osd op complaint time = 180 2022-07-21T01:32:31.387 INFO:tasks.ceph:[mds] rados mon op timeout = 15m 2022-07-21T01:32:31.387 INFO:tasks.ceph:[mds] rados osd op timeout = 15m 2022-07-21T01:32:31.387 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-07-21T01:32:31.388 INFO:tasks.ceph:[mgr] debug ms = 1 2022-07-21T01:32:31.388 INFO:tasks.ceph:[mon] debug mon = 20 2022-07-21T01:32:31.388 INFO:tasks.ceph:[mon] debug ms = 1 2022-07-21T01:32:31.389 INFO:tasks.ceph:[mon] debug paxos = 20 2022-07-21T01:32:31.389 INFO:tasks.ceph:[mon] mon op complaint time = 120 2022-07-21T01:32:31.389 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2022-07-21T01:32:31.390 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2022-07-21T01:32:31.390 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2022-07-21T01:32:31.390 INFO:tasks.ceph:[osd] debug bluefs = 20 2022-07-21T01:32:31.391 INFO:tasks.ceph:[osd] debug bluestore = 20 2022-07-21T01:32:31.391 INFO:tasks.ceph:[osd] debug ms = 1 2022-07-21T01:32:31.391 INFO:tasks.ceph:[osd] debug osd = 20 2022-07-21T01:32:31.392 INFO:tasks.ceph:[osd] debug rocksdb = 10 2022-07-21T01:32:31.392 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2022-07-21T01:32:31.392 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2022-07-21T01:32:31.393 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2022-07-21T01:32:31.393 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2022-07-21T01:32:31.393 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2022-07-21T01:32:31.394 INFO:tasks.ceph:[osd] osd op complaint time = 180 2022-07-21T01:32:31.394 INFO:tasks.ceph:Setting up mon.a... 2022-07-21T01:32:31.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-07-21T01:32:31.450 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.keyring 2022-07-21T01:32:31.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-07-21T01:32:31.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-21T01:32:31.503 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.3'), ('mon.b', '172.21.15.143'), ('mon.c', '[v2:172.21.15.143:3301,v1:172.21.15.143:6790]')] 2022-07-21T01:32:31.504 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.3 --add b 172.21.15.143 --addv c '[v2:172.21.15.143:3301,v1:172.21.15.143:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:31.539 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:31.539 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: generated fsid 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:32:31.539 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 0 2022-07-21T01:32:31.540 INFO:teuthology.orchestra.run.smithi003.stdout:fsid 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:32:31.540 INFO:teuthology.orchestra.run.smithi003.stdout:last_changed 2022-07-21T01:32:31.546453+0000 2022-07-21T01:32:31.540 INFO:teuthology.orchestra.run.smithi003.stdout:created 2022-07-21T01:32:31.546453+0000 2022-07-21T01:32:31.540 INFO:teuthology.orchestra.run.smithi003.stdout:min_mon_release 0 (unknown) 2022-07-21T01:32:31.541 INFO:teuthology.orchestra.run.smithi003.stdout:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2022-07-21T01:32:31.541 INFO:teuthology.orchestra.run.smithi003.stdout:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2022-07-21T01:32:31.541 INFO:teuthology.orchestra.run.smithi003.stdout:2: [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] mon.c 2022-07-21T01:32:31.541 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2022-07-21T01:32:31.543 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 033d20cc-eff0-4d91-a7d1-56acf083b1d2... 2022-07-21T01:32:31.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:32:31.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:32:31.549 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:32:31.552 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:32:31.575 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2022-07-21T01:32:31.576 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2022-07-21T01:32:31.576 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:32:31.576 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2022-07-21T01:32:31.577 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.577 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2022-07-21T01:32:31.577 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.577 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2022-07-21T01:32:31.578 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.578 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2022-07-21T01:32:31.578 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2022-07-21T01:32:31.578 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.579 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2022-07-21T01:32:31.579 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2022-07-21T01:32:31.579 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.579 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:32:31.580 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2022-07-21T01:32:31.580 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.580 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:32:31.580 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:32:31.581 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.581 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2022-07-21T01:32:31.581 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.581 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2022-07-21T01:32:31.582 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2022-07-21T01:32:31.582 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:32:31.582 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:32:31.582 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2022-07-21T01:32:31.583 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:32:31.583 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2022-07-21T01:32:31.583 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:32:31.583 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:32:31.583 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.584 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:32:31.584 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.584 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2022-07-21T01:32:31.584 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.585 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2022-07-21T01:32:31.585 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.585 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2022-07-21T01:32:31.585 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2022-07-21T01:32:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2022-07-21T01:32:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:32:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:32:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.586 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2022-07-21T01:32:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2022-07-21T01:32:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2022-07-21T01:32:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2022-07-21T01:32:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2022-07-21T01:32:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2022-07-21T01:32:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2022-07-21T01:32:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2022-07-21T01:32:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2022-07-21T01:32:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2022-07-21T01:32:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2022-07-21T01:32:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:32:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2022-07-21T01:32:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2022-07-21T01:32:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:32:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2022-07-21T01:32:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2022-07-21T01:32:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2022-07-21T01:32:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2022-07-21T01:32:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression mode = aggressive 2022-07-21T01:32:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2022-07-21T01:32:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 20 2022-07-21T01:32:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 20 2022-07-21T01:32:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:32:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2022-07-21T01:32:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 10 2022-07-21T01:32:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:32:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2022-07-21T01:32:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:32:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:32:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2022-07-21T01:32:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2022-07-21T01:32:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.596 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2022-07-21T01:32:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:32:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2022-07-21T01:32:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2022-07-21T01:32:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2022-07-21T01:32:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:32:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:32:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2022-07-21T01:32:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.598 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2022-07-21T01:32:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:32:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2022-07-21T01:32:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2022-07-21T01:32:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2022-07-21T01:32:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2022-07-21T01:32:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2022-07-21T01:32:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:32:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2022-07-21T01:32:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2022-07-21T01:32:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:32:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:32:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:32:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:32:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:32:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:32:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:32:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: mon op complaint time = 120 2022-07-21T01:32:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:32:31.603 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2022-07-21T01:32:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2022-07-21T01:32:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2022-07-21T01:32:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2022-07-21T01:32:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:32:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:32:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: client mount timeout = 600 2022-07-21T01:32:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: debug client = 20 2022-07-21T01:32:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:32:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 15m 2022-07-21T01:32:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 15m 2022-07-21T01:32:31.605 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2022-07-21T01:32:31.606 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2022-07-21T01:32:31.606 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2022-07-21T01:32:31.606 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2022-07-21T01:32:31.606 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2022-07-21T01:32:31.606 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2022-07-21T01:32:31.607 INFO:teuthology.orchestra.run.smithi003.stdout:[mds] 2022-07-21T01:32:31.607 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds = 20 2022-07-21T01:32:31.607 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:32:31.607 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal frag = True 2022-07-21T01:32:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal fragment size max = 10000 2022-07-21T01:32:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal merge size = 5 2022-07-21T01:32:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal split bits = 3 2022-07-21T01:32:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal split size = 100 2022-07-21T01:32:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug frag = True 2022-07-21T01:32:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug scatterstat = True 2022-07-21T01:32:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: mds op complaint time = 180 2022-07-21T01:32:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: mds verify scatter = True 2022-07-21T01:32:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2022-07-21T01:32:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 15m 2022-07-21T01:32:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 15m 2022-07-21T01:32:31.613 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2022-07-21T01:32:31.613 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2022-07-21T01:32:31.613 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:32:31.614 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = cephx 2022-07-21T01:32:31.614 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.614 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2022-07-21T01:32:31.614 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.615 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2022-07-21T01:32:31.615 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.615 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2022-07-21T01:32:31.615 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2022-07-21T01:32:31.615 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.616 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2022-07-21T01:32:31.616 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2022-07-21T01:32:31.616 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.616 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:32:31.617 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2022-07-21T01:32:31.617 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.617 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:32:31.617 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:32:31.617 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.618 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2022-07-21T01:32:31.618 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.618 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2022-07-21T01:32:31.618 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2022-07-21T01:32:31.619 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:32:31.619 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:32:31.619 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2022-07-21T01:32:31.619 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:32:31.620 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2022-07-21T01:32:31.620 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:32:31.620 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:32:31.620 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.621 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:32:31.621 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.621 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2022-07-21T01:32:31.621 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.621 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2022-07-21T01:32:31.622 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.622 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2022-07-21T01:32:31.622 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2022-07-21T01:32:31.622 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2022-07-21T01:32:31.623 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:32:31.623 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:32:31.623 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.623 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2022-07-21T01:32:31.624 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2022-07-21T01:32:31.624 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.624 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2022-07-21T01:32:31.624 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 600 2022-07-21T01:32:31.624 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.625 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2022-07-21T01:32:31.625 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2022-07-21T01:32:31.625 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.625 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2022-07-21T01:32:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2022-07-21T01:32:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2022-07-21T01:32:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2022-07-21T01:32:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2022-07-21T01:32:31.627 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.627 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:32:31.627 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.627 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2022-07-21T01:32:31.628 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.628 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2022-07-21T01:32:31.628 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.628 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:32:31.628 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2022-07-21T01:32:31.629 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.629 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2022-07-21T01:32:31.629 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2022-07-21T01:32:31.629 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore block size = 96636764160 2022-07-21T01:32:31.630 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore compression mode = aggressive 2022-07-21T01:32:31.630 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore fsck on mount = True 2022-07-21T01:32:31.630 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluefs = 20 2022-07-21T01:32:31.630 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluestore = 20 2022-07-21T01:32:31.631 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:32:31.631 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 20 2022-07-21T01:32:31.631 INFO:teuthology.orchestra.run.smithi174.stdout: debug rocksdb = 10 2022-07-21T01:32:31.631 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:32:31.631 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd full ratio = 0.9 2022-07-21T01:32:31.632 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:32:31.632 INFO:teuthology.orchestra.run.smithi174.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:32:31.632 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = bluestore 2022-07-21T01:32:31.632 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2022-07-21T01:32:31.633 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.633 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2022-07-21T01:32:31.633 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:32:31.633 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2022-07-21T01:32:31.633 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2022-07-21T01:32:31.634 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2022-07-21T01:32:31.634 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:32:31.634 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:32:31.634 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2022-07-21T01:32:31.635 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.635 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2022-07-21T01:32:31.635 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:32:31.635 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2022-07-21T01:32:31.636 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 20 2022-07-21T01:32:31.636 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2022-07-21T01:32:31.636 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2022-07-21T01:32:31.636 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2022-07-21T01:32:31.636 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:32:31.637 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2022-07-21T01:32:31.637 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2022-07-21T01:32:31.637 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:32:31.637 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.638 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:32:31.638 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:32:31.638 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:32:31.638 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.639 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:32:31.639 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:32:31.639 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:32:31.639 INFO:teuthology.orchestra.run.smithi174.stdout: mon op complaint time = 120 2022-07-21T01:32:31.640 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:32:31.640 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2022-07-21T01:32:31.640 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2022-07-21T01:32:31.640 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2022-07-21T01:32:31.640 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2022-07-21T01:32:31.641 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:32:31.641 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:32:31.641 INFO:teuthology.orchestra.run.smithi174.stdout: client mount timeout = 600 2022-07-21T01:32:31.641 INFO:teuthology.orchestra.run.smithi174.stdout: debug client = 20 2022-07-21T01:32:31.642 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:32:31.642 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 15m 2022-07-21T01:32:31.642 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 15m 2022-07-21T01:32:31.642 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2022-07-21T01:32:31.642 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.b] 2022-07-21T01:32:31.643 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.c] 2022-07-21T01:32:31.643 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.a] 2022-07-21T01:32:31.643 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.c] 2022-07-21T01:32:31.643 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.b] 2022-07-21T01:32:31.643 INFO:teuthology.orchestra.run.smithi174.stdout:[mds] 2022-07-21T01:32:31.644 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds = 20 2022-07-21T01:32:31.644 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:32:31.644 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal frag = True 2022-07-21T01:32:31.644 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal fragment size max = 10000 2022-07-21T01:32:31.645 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal merge size = 5 2022-07-21T01:32:31.645 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split bits = 3 2022-07-21T01:32:31.645 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split size = 100 2022-07-21T01:32:31.645 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug frag = True 2022-07-21T01:32:31.645 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug scatterstat = True 2022-07-21T01:32:31.646 INFO:teuthology.orchestra.run.smithi174.stdout: mds op complaint time = 180 2022-07-21T01:32:31.646 INFO:teuthology.orchestra.run.smithi174.stdout: mds verify scatter = True 2022-07-21T01:32:31.646 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2022-07-21T01:32:31.646 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 15m 2022-07-21T01:32:31.647 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 15m 2022-07-21T01:32:31.647 INFO:teuthology.orchestra.run.smithi171.stdout:[global] 2022-07-21T01:32:31.647 INFO:teuthology.orchestra.run.smithi171.stdout: chdir = "" 2022-07-21T01:32:31.647 INFO:teuthology.orchestra.run.smithi171.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:32:31.648 INFO:teuthology.orchestra.run.smithi171.stdout: auth supported = cephx 2022-07-21T01:32:31.648 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.648 INFO:teuthology.orchestra.run.smithi171.stdout: filestore xattr use omap = true 2022-07-21T01:32:31.648 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.649 INFO:teuthology.orchestra.run.smithi171.stdout: mon clock drift allowed = 1.000 2022-07-21T01:32:31.649 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.649 INFO:teuthology.orchestra.run.smithi171.stdout: osd crush chooseleaf type = 0 2022-07-21T01:32:31.649 INFO:teuthology.orchestra.run.smithi171.stdout: auth debug = true 2022-07-21T01:32:31.650 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.650 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on old message = true 2022-07-21T01:32:31.650 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on bug = true 2022-07-21T01:32:31.650 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.651 INFO:teuthology.orchestra.run.smithi171.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:32:31.651 INFO:teuthology.orchestra.run.smithi171.stdout: mon pg warn max object skew = 0 2022-07-21T01:32:31.651 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.651 INFO:teuthology.orchestra.run.smithi171.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:32:31.651 INFO:teuthology.orchestra.run.smithi171.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:32:31.652 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.652 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default size = 2 2022-07-21T01:32:31.652 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.652 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow primary affinity = true 2022-07-21T01:32:31.653 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow pg remap = true 2022-07-21T01:32:31.653 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:32:31.653 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:32:31.653 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on no sortbitwise = false 2022-07-21T01:32:31.654 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:32:31.654 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on too few osds = false 2022-07-21T01:32:31.654 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:32:31.654 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:32:31.654 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.655 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:32:31.655 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.655 INFO:teuthology.orchestra.run.smithi171.stdout: osd default data pool replay window = 5 2022-07-21T01:32:31.655 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.656 INFO:teuthology.orchestra.run.smithi171.stdout: mon allow pool delete = true 2022-07-21T01:32:31.656 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.656 INFO:teuthology.orchestra.run.smithi171.stdout: mon cluster log file level = debug 2022-07-21T01:32:31.656 INFO:teuthology.orchestra.run.smithi171.stdout: debug asserts on shutdown = true 2022-07-21T01:32:31.657 INFO:teuthology.orchestra.run.smithi171.stdout: mon health detail to clog = false 2022-07-21T01:32:31.657 INFO:teuthology.orchestra.run.smithi171.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:32:31.657 INFO:teuthology.orchestra.run.smithi171.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:32:31.657 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.658 INFO:teuthology.orchestra.run.smithi171.stdout:[osd] 2022-07-21T01:32:31.658 INFO:teuthology.orchestra.run.smithi171.stdout: osd journal size = 100 2022-07-21T01:32:31.658 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.658 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub load threshold = 5.0 2022-07-21T01:32:31.658 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub max interval = 600 2022-07-21T01:32:31.659 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.659 INFO:teuthology.orchestra.run.smithi171.stdout: osd recover clone overlap = true 2022-07-21T01:32:31.659 INFO:teuthology.orchestra.run.smithi171.stdout: osd recovery max chunk = 1048576 2022-07-21T01:32:31.659 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.660 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug shutdown = true 2022-07-21T01:32:31.660 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug op order = true 2022-07-21T01:32:31.660 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify stray on activate = true 2022-07-21T01:32:31.660 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.661 INFO:teuthology.orchestra.run.smithi171.stdout: osd open classes on start = true 2022-07-21T01:32:31.661 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug pg log writeout = true 2022-07-21T01:32:31.661 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.661 INFO:teuthology.orchestra.run.smithi171.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:32:31.661 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.662 INFO:teuthology.orchestra.run.smithi171.stdout: osd map max advance = 10 2022-07-21T01:32:31.662 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.662 INFO:teuthology.orchestra.run.smithi171.stdout: journal zero on create = true 2022-07-21T01:32:31.662 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.663 INFO:teuthology.orchestra.run.smithi171.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:32:31.663 INFO:teuthology.orchestra.run.smithi171.stdout: filestore apply finisher threads = 3 2022-07-21T01:32:31.663 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.663 INFO:teuthology.orchestra.run.smithi171.stdout: bdev debug aio = true 2022-07-21T01:32:31.663 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug misdirected ops = true 2022-07-21T01:32:31.664 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore block size = 96636764160 2022-07-21T01:32:31.664 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore compression mode = aggressive 2022-07-21T01:32:31.664 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore fsck on mount = True 2022-07-21T01:32:31.664 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluefs = 20 2022-07-21T01:32:31.665 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluestore = 20 2022-07-21T01:32:31.665 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:32:31.665 INFO:teuthology.orchestra.run.smithi171.stdout: debug osd = 20 2022-07-21T01:32:31.665 INFO:teuthology.orchestra.run.smithi171.stdout: debug rocksdb = 10 2022-07-21T01:32:31.666 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:32:31.666 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd full ratio = 0.9 2022-07-21T01:32:31.666 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:32:31.666 INFO:teuthology.orchestra.run.smithi171.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:32:31.667 INFO:teuthology.orchestra.run.smithi171.stdout: osd objectstore = bluestore 2022-07-21T01:32:31.667 INFO:teuthology.orchestra.run.smithi171.stdout: osd op complaint time = 180 2022-07-21T01:32:31.667 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.667 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr] 2022-07-21T01:32:31.667 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:32:31.668 INFO:teuthology.orchestra.run.smithi171.stdout: debug mgr = 20 2022-07-21T01:32:31.668 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2022-07-21T01:32:31.668 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2022-07-21T01:32:31.668 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:32:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:32:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: mgr/telemetry/nag = false 2022-07-21T01:32:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.669 INFO:teuthology.orchestra.run.smithi171.stdout:[mon] 2022-07-21T01:32:31.670 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:32:31.670 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2022-07-21T01:32:31.670 INFO:teuthology.orchestra.run.smithi171.stdout: debug paxos = 20 2022-07-21T01:32:31.670 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2022-07-21T01:32:31.671 INFO:teuthology.orchestra.run.smithi171.stdout: mon data avail warn = 5 2022-07-21T01:32:31.671 INFO:teuthology.orchestra.run.smithi171.stdout: mon mgr mkfs grace = 240 2022-07-21T01:32:31.671 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:32:31.671 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd reporter subtree level = osd 2022-07-21T01:32:31.672 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd prime pg temp = true 2022-07-21T01:32:31.672 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:32:31.672 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.672 INFO:teuthology.orchestra.run.smithi171.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:32:31.673 INFO:teuthology.orchestra.run.smithi171.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:32:31.673 INFO:teuthology.orchestra.run.smithi171.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:32:31.673 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.673 INFO:teuthology.orchestra.run.smithi171.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:32:31.673 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:32:31.674 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:32:31.674 INFO:teuthology.orchestra.run.smithi171.stdout: mon op complaint time = 120 2022-07-21T01:32:31.674 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:32:31.674 INFO:teuthology.orchestra.run.smithi171.stdout:[client] 2022-07-21T01:32:31.675 INFO:teuthology.orchestra.run.smithi171.stdout: rgw cache enabled = true 2022-07-21T01:32:31.675 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable ops log = true 2022-07-21T01:32:31.675 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable usage log = true 2022-07-21T01:32:31.675 INFO:teuthology.orchestra.run.smithi171.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:32:31.676 INFO:teuthology.orchestra.run.smithi171.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:32:31.676 INFO:teuthology.orchestra.run.smithi171.stdout: client mount timeout = 600 2022-07-21T01:32:31.676 INFO:teuthology.orchestra.run.smithi171.stdout: debug client = 20 2022-07-21T01:32:31.676 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:32:31.676 INFO:teuthology.orchestra.run.smithi171.stdout: rados mon op timeout = 15m 2022-07-21T01:32:31.677 INFO:teuthology.orchestra.run.smithi171.stdout: rados osd op timeout = 15m 2022-07-21T01:32:31.677 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.a] 2022-07-21T01:32:31.677 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.b] 2022-07-21T01:32:31.677 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.c] 2022-07-21T01:32:31.678 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.a] 2022-07-21T01:32:31.678 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.c] 2022-07-21T01:32:31.678 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.b] 2022-07-21T01:32:31.678 INFO:teuthology.orchestra.run.smithi171.stdout:[mds] 2022-07-21T01:32:31.679 INFO:teuthology.orchestra.run.smithi171.stdout: debug mds = 20 2022-07-21T01:32:31.679 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:32:31.679 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal frag = True 2022-07-21T01:32:31.679 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal fragment size max = 10000 2022-07-21T01:32:31.680 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal merge size = 5 2022-07-21T01:32:31.680 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal split bits = 3 2022-07-21T01:32:31.680 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal split size = 100 2022-07-21T01:32:31.680 INFO:teuthology.orchestra.run.smithi171.stdout: mds debug frag = True 2022-07-21T01:32:31.681 INFO:teuthology.orchestra.run.smithi171.stdout: mds debug scatterstat = True 2022-07-21T01:32:31.681 INFO:teuthology.orchestra.run.smithi171.stdout: mds op complaint time = 180 2022-07-21T01:32:31.681 INFO:teuthology.orchestra.run.smithi171.stdout: mds verify scatter = True 2022-07-21T01:32:31.681 INFO:teuthology.orchestra.run.smithi171.stdout: osd op complaint time = 180 2022-07-21T01:32:31.681 INFO:teuthology.orchestra.run.smithi171.stdout: rados mon op timeout = 15m 2022-07-21T01:32:31.682 INFO:teuthology.orchestra.run.smithi171.stdout: rados osd op timeout = 15m 2022-07-21T01:32:31.682 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2022-07-21T01:32:31.682 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2022-07-21T01:32:31.683 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:32:31.683 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2022-07-21T01:32:31.683 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.683 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2022-07-21T01:32:31.683 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.684 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2022-07-21T01:32:31.684 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.684 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2022-07-21T01:32:31.684 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2022-07-21T01:32:31.685 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.685 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2022-07-21T01:32:31.685 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2022-07-21T01:32:31.686 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.686 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:32:31.686 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2022-07-21T01:32:31.687 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.687 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:32:31.687 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:32:31.687 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.688 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2022-07-21T01:32:31.688 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.688 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2022-07-21T01:32:31.688 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2022-07-21T01:32:31.689 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:32:31.689 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:32:31.689 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2022-07-21T01:32:31.689 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:32:31.690 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2022-07-21T01:32:31.690 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:32:31.690 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:32:31.690 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.691 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:32:31.691 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.691 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2022-07-21T01:32:31.691 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.691 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2022-07-21T01:32:31.692 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.692 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2022-07-21T01:32:31.692 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2022-07-21T01:32:31.692 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2022-07-21T01:32:31.693 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:32:31.693 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:32:31.693 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.693 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2022-07-21T01:32:31.693 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2022-07-21T01:32:31.694 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.694 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2022-07-21T01:32:31.694 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2022-07-21T01:32:31.694 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.695 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2022-07-21T01:32:31.695 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2022-07-21T01:32:31.695 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.695 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2022-07-21T01:32:31.695 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2022-07-21T01:32:31.696 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2022-07-21T01:32:31.696 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.696 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2022-07-21T01:32:31.696 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2022-07-21T01:32:31.697 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.697 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:32:31.697 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.697 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2022-07-21T01:32:31.697 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.698 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2022-07-21T01:32:31.698 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.698 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:32:31.698 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2022-07-21T01:32:31.699 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.699 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2022-07-21T01:32:31.699 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2022-07-21T01:32:31.699 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2022-07-21T01:32:31.700 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore compression mode = aggressive 2022-07-21T01:32:31.700 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2022-07-21T01:32:31.700 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 20 2022-07-21T01:32:31.700 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 20 2022-07-21T01:32:31.700 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:32:31.701 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2022-07-21T01:32:31.701 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 10 2022-07-21T01:32:31.701 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:32:31.701 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2022-07-21T01:32:31.702 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:32:31.702 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:32:31.702 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2022-07-21T01:32:31.702 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2022-07-21T01:32:31.702 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.703 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2022-07-21T01:32:31.703 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:32:31.703 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2022-07-21T01:32:31.703 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2022-07-21T01:32:31.704 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2022-07-21T01:32:31.704 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:32:31.704 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:32:31.704 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2022-07-21T01:32:31.704 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.705 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2022-07-21T01:32:31.705 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:32:31.705 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2022-07-21T01:32:31.706 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2022-07-21T01:32:31.706 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2022-07-21T01:32:31.706 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2022-07-21T01:32:31.706 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2022-07-21T01:32:31.706 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:32:31.707 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2022-07-21T01:32:31.707 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2022-07-21T01:32:31.707 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:32:31.707 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.708 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:32:31.708 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:32:31.708 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:32:31.708 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.708 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:32:31.709 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:32:31.709 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:32:31.709 INFO:teuthology.orchestra.run.smithi143.stdout: mon op complaint time = 120 2022-07-21T01:32:31.709 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:32:31.710 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2022-07-21T01:32:31.710 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2022-07-21T01:32:31.710 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2022-07-21T01:32:31.710 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2022-07-21T01:32:31.711 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:32:31.711 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:32:31.711 INFO:teuthology.orchestra.run.smithi143.stdout: client mount timeout = 600 2022-07-21T01:32:31.711 INFO:teuthology.orchestra.run.smithi143.stdout: debug client = 20 2022-07-21T01:32:31.711 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:32:31.712 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 15m 2022-07-21T01:32:31.712 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 15m 2022-07-21T01:32:31.712 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2022-07-21T01:32:31.712 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2022-07-21T01:32:31.713 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2022-07-21T01:32:31.713 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.a] 2022-07-21T01:32:31.713 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.c] 2022-07-21T01:32:31.713 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.b] 2022-07-21T01:32:31.713 INFO:teuthology.orchestra.run.smithi143.stdout:[mds] 2022-07-21T01:32:31.714 INFO:teuthology.orchestra.run.smithi143.stdout: debug mds = 20 2022-07-21T01:32:31.714 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:32:31.714 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal frag = True 2022-07-21T01:32:31.714 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal fragment size max = 10000 2022-07-21T01:32:31.715 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal merge size = 5 2022-07-21T01:32:31.715 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal split bits = 3 2022-07-21T01:32:31.715 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal split size = 100 2022-07-21T01:32:31.715 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug frag = True 2022-07-21T01:32:31.716 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug scatterstat = True 2022-07-21T01:32:31.716 INFO:teuthology.orchestra.run.smithi143.stdout: mds op complaint time = 180 2022-07-21T01:32:31.716 INFO:teuthology.orchestra.run.smithi143.stdout: mds verify scatter = True 2022-07-21T01:32:31.716 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2022-07-21T01:32:31.716 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 15m 2022-07-21T01:32:31.717 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 15m 2022-07-21T01:32:31.717 INFO:tasks.ceph:Creating admin key on mon.a... 2022-07-21T01:32:31.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-07-21T01:32:31.766 INFO:tasks.ceph:Copying monmap to all nodes... 2022-07-21T01:32:31.775 DEBUG:teuthology.orchestra.remote:smithi003:/etc/ceph/ceph.keyring is 206B 2022-07-21T01:32:31.841 DEBUG:teuthology.orchestra.remote:smithi003:/home/ubuntu/cephtest/ceph.monmap is 379B 2022-07-21T01:32:31.856 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T01:32:31.857 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-07-21T01:32:31.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-21T01:32:31.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-21T01:32:31.886 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2022-07-21T01:32:31.886 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:31.936 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T01:32:31.936 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2022-07-21T01:32:31.937 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-21T01:32:31.937 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-21T01:32:31.964 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2022-07-21T01:32:31.965 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:31.970 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi171.front.sepia.ceph.com 2022-07-21T01:32:31.970 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-07-21T01:32:31.971 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-21T01:32:31.971 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-21T01:32:31.999 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2022-07-21T01:32:32.000 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:32.006 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi174.front.sepia.ceph.com 2022-07-21T01:32:32.007 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2022-07-21T01:32:32.007 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-21T01:32:32.008 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-21T01:32:32.033 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2022-07-21T01:32:32.034 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:32.040 INFO:tasks.ceph:Setting up mon nodes... 2022-07-21T01:32:32.040 INFO:tasks.ceph:Setting up mgr nodes... 2022-07-21T01:32:32.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2022-07-21T01:32:32.086 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2022-07-21T01:32:32.088 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2022-07-21T01:32:32.144 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2022-07-21T01:32:32.147 INFO:tasks.ceph:Setting up mds nodes... 2022-07-21T01:32:32.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2022-07-21T01:32:32.192 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2022-07-21T01:32:32.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2022-07-21T01:32:32.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2022-07-21T01:32:32.294 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2022-07-21T01:32:32.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2022-07-21T01:32:32.311 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2022-07-21T01:32:32.355 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2022-07-21T01:32:32.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2022-07-21T01:32:32.369 INFO:tasks.ceph_client:Setting up client nodes... 2022-07-21T01:32:32.370 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-07-21T01:32:32.416 INFO:teuthology.orchestra.run.smithi171.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-07-21T01:32:32.431 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2022-07-21T01:32:32.478 INFO:teuthology.orchestra.run.smithi174.stdout:creating /etc/ceph/ceph.client.1.keyring 2022-07-21T01:32:32.491 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-07-21T01:32:32.491 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-07-21T01:32:32.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-07-21T01:32:32.509 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:32.509 INFO:tasks.ceph:role: osd.0 2022-07-21T01:32:32.510 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T01:32:32.510 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-21T01:32:33.050 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-21T01:32:33.050 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-21T01:32:33.050 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-21T01:32:33.051 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 2022-07-21T01:32:33.051 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-21T01:32:33.051 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2022-07-21T01:32:33.052 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-21T01:32:33.052 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-21T01:32:33.052 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-21T01:32:33.052 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-21T01:32:33.053 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2022-07-21T01:32:33.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-07-21T01:32:33.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-07-21T01:32:33.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2022-07-21T01:32:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:32:33.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-07-21T01:32:33.218 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:33.219 INFO:tasks.ceph:role: osd.1 2022-07-21T01:32:33.219 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T01:32:33.220 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-21T01:32:33.740 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-21T01:32:33.741 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-21T01:32:33.741 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-21T01:32:33.741 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 2022-07-21T01:32:33.741 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-21T01:32:33.742 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2022-07-21T01:32:33.742 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-21T01:32:33.742 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-21T01:32:33.742 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-21T01:32:33.743 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-21T01:32:33.744 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2022-07-21T01:32:33.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-07-21T01:32:33.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-07-21T01:32:33.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2022-07-21T01:32:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:32:33.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-07-21T01:32:33.853 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:33.853 INFO:tasks.ceph:role: osd.2 2022-07-21T01:32:33.854 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T01:32:33.854 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-21T01:32:34.314 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-21T01:32:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-21T01:32:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-21T01:32:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 2022-07-21T01:32:34.315 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-21T01:32:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2022-07-21T01:32:34.315 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-21T01:32:34.315 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-21T01:32:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-21T01:32:34.316 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-21T01:32:34.317 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2022-07-21T01:32:34.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-07-21T01:32:34.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-07-21T01:32:34.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2022-07-21T01:32:34.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:32:34.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2022-07-21T01:32:34.404 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:34.404 INFO:tasks.ceph:role: osd.3 2022-07-21T01:32:34.405 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T01:32:34.405 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-07-21T01:32:34.900 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-07-21T01:32:34.900 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-21T01:32:34.901 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-21T01:32:34.901 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 2022-07-21T01:32:34.901 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-21T01:32:34.901 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2022-07-21T01:32:34.902 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-21T01:32:34.902 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-21T01:32:34.902 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-21T01:32:34.902 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-21T01:32:34.904 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2022-07-21T01:32:34.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2022-07-21T01:32:34.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2022-07-21T01:32:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:32:34.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2022-07-21T01:32:34.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:35.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:35.090+0000 7f0d49234d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2022-07-21T01:32:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:35.094+0000 7f0d49234d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2022-07-21T01:32:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:35.094+0000 7f0d49234d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-07-21T01:32:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:35.094+0000 7f0d49234d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-07-21T01:32:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:35.094+0000 7f0d49234d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-07-21T01:32:35.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:35.098+0000 7f0d49234d80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2022-07-21T01:32:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:35.122+0000 7f0d49234d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-21T01:32:37.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-07-21T01:32:37.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:37.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:37.618+0000 7f148cfe8d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2022-07-21T01:32:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:37.618+0000 7f148cfe8d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2022-07-21T01:32:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:37.618+0000 7f148cfe8d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-07-21T01:32:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:37.618+0000 7f148cfe8d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-07-21T01:32:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:37.618+0000 7f148cfe8d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-07-21T01:32:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:37.618+0000 7f148cfe8d80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2022-07-21T01:32:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:37.638+0000 7f148cfe8d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-21T01:32:40.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-07-21T01:32:40.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:40.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:40.182+0000 7fd5f9716d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2022-07-21T01:32:40.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:40.182+0000 7fd5f9716d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2022-07-21T01:32:40.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:40.182+0000 7fd5f9716d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-07-21T01:32:40.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:40.182+0000 7fd5f9716d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-07-21T01:32:40.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:40.182+0000 7fd5f9716d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-07-21T01:32:40.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:40.186+0000 7fd5f9716d80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2022-07-21T01:32:40.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:40.202+0000 7fd5f9716d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-21T01:32:42.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-07-21T01:32:42.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:42.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:42.758+0000 7fd511da1d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2022-07-21T01:32:42.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:42.758+0000 7fd511da1d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2022-07-21T01:32:42.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:42.758+0000 7fd511da1d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-07-21T01:32:42.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:42.758+0000 7fd511da1d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-07-21T01:32:42.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:42.758+0000 7fd511da1d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-07-21T01:32:42.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:42.758+0000 7fd511da1d80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2022-07-21T01:32:42.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:32:42.778+0000 7fd511da1d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-21T01:32:45.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-07-21T01:32:45.220 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2022-07-21T01:32:45.238 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:45.238 INFO:tasks.ceph:role: osd.4 2022-07-21T01:32:45.239 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T01:32:45.239 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-21T01:32:45.764 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-21T01:32:45.765 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-21T01:32:45.765 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-21T01:32:45.765 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2022-07-21T01:32:45.765 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-21T01:32:45.766 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2022-07-21T01:32:45.766 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-21T01:32:45.766 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-21T01:32:45.766 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-21T01:32:45.767 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-21T01:32:45.768 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2022-07-21T01:32:45.768 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2022-07-21T01:32:45.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2022-07-21T01:32:45.876 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2022-07-21T01:32:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:32:45.878 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2022-07-21T01:32:45.937 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:45.937 INFO:tasks.ceph:role: osd.5 2022-07-21T01:32:45.938 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T01:32:45.938 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-21T01:32:46.444 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-21T01:32:46.444 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-21T01:32:46.445 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-21T01:32:46.445 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2022-07-21T01:32:46.445 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-21T01:32:46.445 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2022-07-21T01:32:46.445 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-21T01:32:46.446 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-21T01:32:46.446 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-21T01:32:46.446 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-21T01:32:46.447 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2022-07-21T01:32:46.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2022-07-21T01:32:46.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2022-07-21T01:32:46.532 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2022-07-21T01:32:46.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:32:46.535 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2022-07-21T01:32:46.545 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:46.546 INFO:tasks.ceph:role: osd.6 2022-07-21T01:32:46.546 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T01:32:46.547 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-21T01:32:47.020 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-21T01:32:47.020 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-21T01:32:47.021 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-21T01:32:47.021 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2022-07-21T01:32:47.021 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-21T01:32:47.021 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2022-07-21T01:32:47.022 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-21T01:32:47.022 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-21T01:32:47.022 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-21T01:32:47.022 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-21T01:32:47.023 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2022-07-21T01:32:47.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2022-07-21T01:32:47.097 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2022-07-21T01:32:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:32:47.109 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2022-07-21T01:32:47.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2022-07-21T01:32:47.168 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-21T01:32:47.168 INFO:tasks.ceph:role: osd.7 2022-07-21T01:32:47.169 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T01:32:47.169 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-07-21T01:32:47.667 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-07-21T01:32:47.667 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-21T01:32:47.668 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-21T01:32:47.668 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2022-07-21T01:32:47.668 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-21T01:32:47.668 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2022-07-21T01:32:47.669 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-21T01:32:47.669 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-21T01:32:47.669 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-21T01:32:47.669 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-21T01:32:47.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2022-07-21T01:32:47.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2022-07-21T01:32:47.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2022-07-21T01:32:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:32:47.755 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2022-07-21T01:32:47.756 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:47.857 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:47.854+0000 7fdbb0e86d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2022-07-21T01:32:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:47.858+0000 7fdbb0e86d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2022-07-21T01:32:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:47.858+0000 7fdbb0e86d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-07-21T01:32:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:47.858+0000 7fdbb0e86d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-07-21T01:32:47.859 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:47.858+0000 7fdbb0e86d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-07-21T01:32:47.866 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:47.866+0000 7fdbb0e86d80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2022-07-21T01:32:47.894 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:47.894+0000 7fdbb0e86d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-21T01:32:50.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-07-21T01:32:50.349 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:50.387 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:50.386+0000 7f6eb73edd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2022-07-21T01:32:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:50.386+0000 7f6eb73edd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2022-07-21T01:32:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:50.386+0000 7f6eb73edd80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-07-21T01:32:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:50.386+0000 7f6eb73edd80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-07-21T01:32:50.388 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:50.386+0000 7f6eb73edd80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-07-21T01:32:50.389 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:50.386+0000 7f6eb73edd80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2022-07-21T01:32:50.408 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:50.406+0000 7f6eb73edd80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-21T01:32:52.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-07-21T01:32:52.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:52.940 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:52.938+0000 7f90d20ded80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2022-07-21T01:32:52.941 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:52.938+0000 7f90d20ded80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2022-07-21T01:32:52.941 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:52.938+0000 7f90d20ded80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-07-21T01:32:52.941 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:52.938+0000 7f90d20ded80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-07-21T01:32:52.942 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:52.938+0000 7f90d20ded80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-07-21T01:32:52.942 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:52.942+0000 7f90d20ded80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2022-07-21T01:32:52.960 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:52.958+0000 7f90d20ded80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-21T01:32:55.403 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-07-21T01:32:55.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:32:55.457 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:55.458+0000 7f704d071d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2022-07-21T01:32:55.458 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:55.458+0000 7f704d071d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2022-07-21T01:32:55.458 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:55.458+0000 7f704d071d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-07-21T01:32:55.458 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:55.458+0000 7f704d071d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-07-21T01:32:55.459 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:55.458+0000 7f704d071d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-07-21T01:32:55.459 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:55.458+0000 7f704d071d80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2022-07-21T01:32:55.477 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:32:55.478+0000 7f704d071d80 -1 freelist read_size_meta_from_db missing size meta in DB 2022-07-21T01:32:57.894 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-07-21T01:32:57.952 INFO:tasks.ceph:Reading keys from all nodes... 2022-07-21T01:32:57.953 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:32:57.960 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.GLguKy0q50 2022-07-21T01:32:57.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.GLguKy0q50 2022-07-21T01:32:58.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.GLguKy0q50 2022-07-21T01:32:58.038 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.GLguKy0q50 is 56B 2022-07-21T01:32:58.054 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.GLguKy0q50 2022-07-21T01:32:58.060 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:32:58.107 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.crURAPyAuY 2022-07-21T01:32:58.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.crURAPyAuY 2022-07-21T01:32:58.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.crURAPyAuY 2022-07-21T01:32:58.193 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.crURAPyAuY is 56B 2022-07-21T01:32:58.209 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.crURAPyAuY 2022-07-21T01:32:58.216 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:32:58.263 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.0twh6y0v8V 2022-07-21T01:32:58.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/mds/ceph-c/keyring /tmp/tmp.0twh6y0v8V 2022-07-21T01:32:58.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.0twh6y0v8V 2022-07-21T01:32:58.348 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.0twh6y0v8V is 56B 2022-07-21T01:32:58.363 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.0twh6y0v8V 2022-07-21T01:32:58.370 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:32:58.423 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.CqPg1LcUZp 2022-07-21T01:32:58.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.CqPg1LcUZp 2022-07-21T01:32:58.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.CqPg1LcUZp 2022-07-21T01:32:58.505 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.CqPg1LcUZp is 56B 2022-07-21T01:32:58.522 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.CqPg1LcUZp 2022-07-21T01:32:58.530 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:32:58.579 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.5a5nYXGVtm 2022-07-21T01:32:58.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.5a5nYXGVtm 2022-07-21T01:32:58.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.5a5nYXGVtm 2022-07-21T01:32:58.660 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.5a5nYXGVtm is 56B 2022-07-21T01:32:58.675 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.5a5nYXGVtm 2022-07-21T01:32:58.682 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:32:58.731 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.qs6vZMdXdW 2022-07-21T01:32:58.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.qs6vZMdXdW 2022-07-21T01:32:58.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.qs6vZMdXdW 2022-07-21T01:32:58.812 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.qs6vZMdXdW is 56B 2022-07-21T01:32:58.827 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.qs6vZMdXdW 2022-07-21T01:32:58.833 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:32:58.883 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.wM8uFwFMw7 2022-07-21T01:32:58.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.wM8uFwFMw7 2022-07-21T01:32:58.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.wM8uFwFMw7 2022-07-21T01:32:58.972 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.wM8uFwFMw7 is 56B 2022-07-21T01:32:58.987 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.wM8uFwFMw7 2022-07-21T01:32:58.994 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:32:59.000 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.zDtc1BvjAs 2022-07-21T01:32:59.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.zDtc1BvjAs 2022-07-21T01:32:59.060 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.zDtc1BvjAs 2022-07-21T01:32:59.080 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.zDtc1BvjAs is 56B 2022-07-21T01:32:59.096 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.zDtc1BvjAs 2022-07-21T01:32:59.102 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:32:59.151 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.sKGUHCf9fz 2022-07-21T01:32:59.152 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.sKGUHCf9fz 2022-07-21T01:32:59.167 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.sKGUHCf9fz 2022-07-21T01:32:59.187 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.sKGUHCf9fz is 56B 2022-07-21T01:32:59.203 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.sKGUHCf9fz 2022-07-21T01:32:59.210 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:32:59.258 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.8pKhcH00U6 2022-07-21T01:32:59.259 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.8pKhcH00U6 2022-07-21T01:32:59.272 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.8pKhcH00U6 2022-07-21T01:32:59.336 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.8pKhcH00U6 is 56B 2022-07-21T01:32:59.351 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.8pKhcH00U6 2022-07-21T01:32:59.356 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:32:59.407 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.tbLNvtw1fU 2022-07-21T01:32:59.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.tbLNvtw1fU 2022-07-21T01:32:59.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.tbLNvtw1fU 2022-07-21T01:32:59.488 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.tbLNvtw1fU is 56B 2022-07-21T01:32:59.504 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.tbLNvtw1fU 2022-07-21T01:32:59.510 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:32:59.559 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.KHIm078Ecj 2022-07-21T01:32:59.559 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.KHIm078Ecj 2022-07-21T01:32:59.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.KHIm078Ecj 2022-07-21T01:32:59.598 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.KHIm078Ecj is 56B 2022-07-21T01:32:59.651 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.KHIm078Ecj 2022-07-21T01:32:59.658 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:32:59.665 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.59y9qmZ6Ck 2022-07-21T01:32:59.665 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.59y9qmZ6Ck 2022-07-21T01:32:59.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.59y9qmZ6Ck 2022-07-21T01:32:59.709 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.59y9qmZ6Ck is 56B 2022-07-21T01:32:59.763 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.59y9qmZ6Ck 2022-07-21T01:32:59.786 DEBUG:teuthology.orchestra.remote:smithi171:/etc/ceph/ceph.client.0.keyring is 59B 2022-07-21T01:32:59.816 DEBUG:teuthology.orchestra.remote:smithi174:/etc/ceph/ceph.client.1.keyring is 59B 2022-07-21T01:32:59.833 INFO:tasks.ceph:Adding keys to all mons... 2022-07-21T01:32:59.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2022-07-21T01:32:59.836 DEBUG:teuthology.orchestra.run.smithi143:> sudo tee -a /etc/ceph/ceph.keyring 2022-07-21T01:32:59.849 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.y] 2022-07-21T01:32:59.849 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw== 2022-07-21T01:32:59.850 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2022-07-21T01:32:59.850 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw== 2022-07-21T01:32:59.850 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2022-07-21T01:32:59.851 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg== 2022-07-21T01:32:59.851 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2022-07-21T01:32:59.852 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ== 2022-07-21T01:32:59.852 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2022-07-21T01:32:59.853 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g== 2022-07-21T01:32:59.853 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2022-07-21T01:32:59.853 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg== 2022-07-21T01:32:59.853 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.3] 2022-07-21T01:32:59.854 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag== 2022-07-21T01:32:59.854 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.x] 2022-07-21T01:32:59.854 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw== 2022-07-21T01:32:59.855 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2022-07-21T01:32:59.855 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ== 2022-07-21T01:32:59.855 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.4] 2022-07-21T01:32:59.855 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w== 2022-07-21T01:32:59.856 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.5] 2022-07-21T01:32:59.856 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw== 2022-07-21T01:32:59.856 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.6] 2022-07-21T01:32:59.856 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg== 2022-07-21T01:32:59.857 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.7] 2022-07-21T01:32:59.857 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A== 2022-07-21T01:32:59.857 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2022-07-21T01:32:59.857 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q== 2022-07-21T01:32:59.858 INFO:teuthology.orchestra.run.smithi003.stdout:[client.1] 2022-07-21T01:32:59.858 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw== 2022-07-21T01:32:59.860 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.y] 2022-07-21T01:32:59.860 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw== 2022-07-21T01:32:59.860 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.a] 2022-07-21T01:32:59.861 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw== 2022-07-21T01:32:59.861 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.c] 2022-07-21T01:32:59.861 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg== 2022-07-21T01:32:59.861 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.0] 2022-07-21T01:32:59.862 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ== 2022-07-21T01:32:59.862 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.1] 2022-07-21T01:32:59.862 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g== 2022-07-21T01:32:59.862 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.2] 2022-07-21T01:32:59.863 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg== 2022-07-21T01:32:59.863 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.3] 2022-07-21T01:32:59.863 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag== 2022-07-21T01:32:59.863 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.x] 2022-07-21T01:32:59.863 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw== 2022-07-21T01:32:59.864 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.b] 2022-07-21T01:32:59.864 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ== 2022-07-21T01:32:59.864 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.4] 2022-07-21T01:32:59.864 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w== 2022-07-21T01:32:59.865 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.5] 2022-07-21T01:32:59.865 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw== 2022-07-21T01:32:59.865 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.6] 2022-07-21T01:32:59.865 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg== 2022-07-21T01:32:59.865 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.7] 2022-07-21T01:32:59.866 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A== 2022-07-21T01:32:59.866 INFO:teuthology.orchestra.run.smithi143.stdout:[client.0] 2022-07-21T01:32:59.866 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q== 2022-07-21T01:32:59.866 INFO:teuthology.orchestra.run.smithi143.stdout:[client.1] 2022-07-21T01:32:59.867 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw== 2022-07-21T01:32:59.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-21T01:32:59.870 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-21T01:32:59.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-21T01:32:59.955 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-21T01:32:59.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-21T01:33:00.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-21T01:33:00.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.084 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.161 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.243 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-21T01:33:00.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-21T01:33:00.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-21T01:33:00.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-21T01:33:00.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.631 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-21T01:33:00.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-21T01:33:00.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-21T01:33:00.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-21T01:33:00.812 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-21T01:33:00.856 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-07-21T01:33:00.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-07-21T01:33:00.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-21T01:33:00.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-07-21T01:33:00.990 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2022-07-21T01:33:01.008 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-21T01:33:01.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:33:01.036 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:33:01.055 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:33:01.058 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:33:01.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2022-07-21T01:33:01.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2022-07-21T01:33:01.179 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-21T01:33:01.255 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2022-07-21T01:33:01.271 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:33:01.274 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-07-21T01:33:01.280 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-07-21T01:33:01.281 INFO:tasks.ceph.mon.a:Restarting daemon 2022-07-21T01:33:01.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-07-21T01:33:01.284 INFO:tasks.ceph.mon.a:Started 2022-07-21T01:33:01.284 INFO:tasks.ceph.mon.b:Restarting daemon 2022-07-21T01:33:01.284 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2022-07-21T01:33:01.287 INFO:tasks.ceph.mon.b:Started 2022-07-21T01:33:01.287 INFO:tasks.ceph.mon.c:Restarting daemon 2022-07-21T01:33:01.288 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2022-07-21T01:33:01.290 INFO:tasks.ceph.mon.c:Started 2022-07-21T01:33:01.290 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-07-21T01:33:01.291 INFO:tasks.ceph.mgr.y:Restarting daemon 2022-07-21T01:33:01.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2022-07-21T01:33:01.293 INFO:tasks.ceph.mgr.y:Started 2022-07-21T01:33:01.294 INFO:tasks.ceph.mgr.x:Restarting daemon 2022-07-21T01:33:01.295 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2022-07-21T01:33:01.297 INFO:tasks.ceph.mgr.x:Started 2022-07-21T01:33:01.298 INFO:tasks.ceph:Setting crush tunables to default 2022-07-21T01:33:01.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd crush tunables default 2022-07-21T01:33:01.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.406+0000 7f0074217700 1 Processor -- start 2022-07-21T01:33:01.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.406+0000 7f0074217700 1 -- start start 2022-07-21T01:33:01.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.406+0000 7f0074217700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:01.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.406+0000 7f0074217700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.406+0000 7f0074217700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006c0af440 con 0x7f006c0abec0 2022-07-21T01:33:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.406+0000 7f0074217700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c141060 con 0x7f006c0ad030 2022-07-21T01:33:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.406+0000 7f0074217700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c0ad4a0 con 0x7f006c060af0 2022-07-21T01:33:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f00717b2700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f006c0abec0 0x7f006c143620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50246/0 (socket says 172.21.15.3:50246) 2022-07-21T01:33:01.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f00717b2700 1 -- 172.21.15.3:0/713883691 learned_addr learned my addr 172.21.15.3:0/713883691 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:01.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:01.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:01.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f00727b4700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-21T01:33:01.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f00727b4700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:01.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:01.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-07-21T01:33:01.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 msgr2=0x7f006c0acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-21T01:33:01.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 msgr2=0x7f006c0acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:01.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:01.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.410+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-07-21T01:33:01.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:01.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:01.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f00727b4700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-21T01:33:01.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f00727b4700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:01.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:01.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2022-07-21T01:33:01.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 msgr2=0x7f006c0acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-21T01:33:01.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 msgr2=0x7f006c0acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:01.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:01.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:01.610+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2022-07-21T01:33:02.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:02.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:02.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f00727b4700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-21T01:33:02.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f00727b4700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:02.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:02.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2022-07-21T01:33:02.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 msgr2=0x7f006c0acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-21T01:33:02.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 msgr2=0x7f006c0acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:02.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:02.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.010+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2022-07-21T01:33:02.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:02.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:02.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f00727b4700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-21T01:33:02.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f00727b4700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:02.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2022-07-21T01:33:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 msgr2=0x7f006c0acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-21T01:33:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 msgr2=0x7f006c0acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:02.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:02.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:02.814+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2022-07-21T01:33:04.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.406+0000 7f005b7fe700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 msgr2=0x7f006c0acb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-21T01:33:04.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.406+0000 7f005b7fe700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c060af0 0x7f006c0acb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:04.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.406+0000 7f005b7fe700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-21T01:33:04.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.406+0000 7f005b7fe700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.143:3300/0 conn(0x7f006c0ad030 0x7f006c143d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:04.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.406+0000 7f005b7fe700 1 -- 172.21.15.3:0/713883691 >> v1:172.21.15.3:6789/0 conn(0x7f006c0abec0 legacy=0x7f006c143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:04.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.410+0000 7f005b7fe700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:04.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.410+0000 7f005b7fe700 1 -- 172.21.15.3:0/713883691 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006c0b13e0 con 0x7f004c005e30 2022-07-21T01:33:04.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.410+0000 7f005b7fe700 1 -- 172.21.15.3:0/713883691 --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c0ad4a0 con 0x7f004c0033f0 2022-07-21T01:33:04.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.410+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:04.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.410+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 msgr2=0x7f004c005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-21T01:33:04.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.410+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 msgr2=0x7f004c005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:04.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.410+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:04.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.410+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-07-21T01:33:04.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.614+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:04.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.614+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 msgr2=0x7f004c005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-21T01:33:04.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.614+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 msgr2=0x7f004c005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:04.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.614+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:04.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:04.614+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2022-07-21T01:33:05.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.014+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:05.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.014+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 msgr2=0x7f004c005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-21T01:33:05.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.014+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 msgr2=0x7f004c005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:05.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.014+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:05.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.014+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2022-07-21T01:33:05.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.814+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:05.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.814+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 msgr2=0x7f004c005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-21T01:33:05.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.814+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 msgr2=0x7f004c005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-21T01:33:05.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.814+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-21T01:33:05.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:05.814+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2022-07-21T01:33:06.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.674+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460392794 0 0) 0x7f006c0b13e0 con 0x7f004c005e30 2022-07-21T01:33:06.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.674+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0054003260 con 0x7f004c005e30 2022-07-21T01:33:06.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939597472 0 0) 0x7f0054003260 con 0x7f004c005e30 2022-07-21T01:33:06.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 msgr2=0x7f004c005830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-21T01:33:06.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 --2- 172.21.15.3:0/713883691 >> v2:172.21.15.3:3300/0 conn(0x7f004c0033f0 0x7f004c005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c00ac10 con 0x7f004c005e30 2022-07-21T01:33:06.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f0068001720 con 0x7f004c005e30 2022-07-21T01:33:06.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 >> v1:172.21.15.143:6789/0 conn(0x7f004c005e30 legacy=0x7f004c0082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:06.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f00727b4700 1 -- 172.21.15.3:0/713883691 reap_dead start 2022-07-21T01:33:06.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f0054003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:06.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 0x7f0054004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:06.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f0054003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:06.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 0x7f005400d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:06.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c0ad4a0 con 0x7f0054003750 2022-07-21T01:33:06.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f00717b2700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 0x7f0054004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:06.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f00727b4700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 0x7f005400d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:06.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005400e040 con 0x7f0054004ec0 2022-07-21T01:33:06.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005400e200 con 0x7f0054004500 2022-07-21T01:33:06.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 msgr2=0x7f0054004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:06.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 0x7f0054004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 msgr2=0x7f005400d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:06.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 0x7f005400d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0071fb3700 1 -- 172.21.15.3:0/713883691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0054003260 con 0x7f0054003750 2022-07-21T01:33:06.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.678+0000 7f0071fb3700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f0054003b50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f005c00a000 tx=0x7f005c008040).ready entity=mon.0 client_cookie=d2e18f35e17f7069 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:06.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f005c01e070 con 0x7f0054003750 2022-07-21T01:33:06.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c019260 con 0x7f0054003750 2022-07-21T01:33:06.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0070fb1700 1 -- 172.21.15.3:0/713883691 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f005c017a70 con 0x7f0054003750 2022-07-21T01:33:06.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0074217700 1 -- 172.21.15.3:0/713883691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 msgr2=0x7f0054003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:06.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0074217700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f0054003b50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f005c00a000 tx=0x7f005c008040).stop 2022-07-21T01:33:06.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0074217700 1 -- 172.21.15.3:0/713883691 shutdown_connections 2022-07-21T01:33:06.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0074217700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 0x7f005400d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0074217700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 0x7f0054004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0074217700 1 --2- 172.21.15.3:0/713883691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f0054003b50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0074217700 1 -- 172.21.15.3:0/713883691 >> 172.21.15.3:0/713883691 conn(0x7f006c062a30 msgr2=0x7f006c062e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:06.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0074217700 1 -- 172.21.15.3:0/713883691 shutdown_connections 2022-07-21T01:33:06.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.686+0000 7f0074217700 1 -- 172.21.15.3:0/713883691 wait complete. 2022-07-21T01:33:06.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0074217700 1 Processor -- start 2022-07-21T01:33:06.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0074217700 1 -- start start 2022-07-21T01:33:06.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0074217700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f006c1425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:06.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0074217700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 0x7f006c151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:06.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0074217700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 0x7f006c13bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:06.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0074217700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1522b0 con 0x7f0054003750 2022-07-21T01:33:06.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0074217700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c14ec50 con 0x7f0054004ec0 2022-07-21T01:33:06.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0074217700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1513e0 con 0x7f0054004500 2022-07-21T01:33:06.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0071fb3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f006c1425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:06.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0071fb3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f006c1425d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42036/0 (socket says 172.21.15.3:42036) 2022-07-21T01:33:06.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0071fb3700 1 -- 172.21.15.3:0/1700337444 learned_addr learned my addr 172.21.15.3:0/1700337444 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:06.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f00717b2700 1 --2- 172.21.15.3:0/1700337444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 0x7f006c151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:06.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f00727b4700 1 --2- 172.21.15.3:0/1700337444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 0x7f006c13bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0071fb3700 1 -- 172.21.15.3:0/1700337444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 msgr2=0x7f006c151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0071fb3700 1 --2- 172.21.15.3:0/1700337444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 0x7f006c151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0071fb3700 1 -- 172.21.15.3:0/1700337444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 msgr2=0x7f006c13bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:06.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0071fb3700 1 --2- 172.21.15.3:0/1700337444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 0x7f006c13bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.690+0000 7f0071fb3700 1 -- 172.21.15.3:0/1700337444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c151680 con 0x7f0054003750 2022-07-21T01:33:06.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.694+0000 7f0071fb3700 1 --2- 172.21.15.3:0/1700337444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f006c1425d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f005c00a000 tx=0x7f005c024000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:06.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.694+0000 7f005affd700 1 -- 172.21.15.3:0/1700337444 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f005c006940 con 0x7f0054003750 2022-07-21T01:33:06.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.694+0000 7f005affd700 1 -- 172.21.15.3:0/1700337444 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c017e60 con 0x7f0054003750 2022-07-21T01:33:06.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.694+0000 7f0074217700 1 -- 172.21.15.3:0/1700337444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006c151920 con 0x7f0054003750 2022-07-21T01:33:06.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.694+0000 7f005affd700 1 -- 172.21.15.3:0/1700337444 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f005c018dc0 con 0x7f0054003750 2022-07-21T01:33:06.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.694+0000 7f0074217700 1 -- 172.21.15.3:0/1700337444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006c14e8a0 con 0x7f0054003750 2022-07-21T01:33:06.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.694+0000 7f005affd700 1 -- 172.21.15.3:0/1700337444 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f005c036070 con 0x7f0054003750 2022-07-21T01:33:06.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.694+0000 7f005affd700 1 -- 172.21.15.3:0/1700337444 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f005c028030 con 0x7f0054003750 2022-07-21T01:33:06.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.694+0000 7f0074217700 1 -- 172.21.15.3:0/1700337444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006c0b0ea0 con 0x7f0054003750 2022-07-21T01:33:06.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.698+0000 7f005affd700 1 -- 172.21.15.3:0/1700337444 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f005c034060 con 0x7f0054003750 2022-07-21T01:33:06.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.762+0000 7f0074217700 1 -- 172.21.15.3:0/1700337444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f006c0574e0 con 0x7f0054003750 2022-07-21T01:33:06.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.766+0000 7f005affd700 1 -- 172.21.15.3:0/1700337444 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f005c019600 con 0x7f0054003750 2022-07-21T01:33:06.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.770+0000 7f0074217700 1 -- 172.21.15.3:0/1700337444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 msgr2=0x7f006c1425d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:06.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.770+0000 7f0074217700 1 --2- 172.21.15.3:0/1700337444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f006c1425d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f005c00a000 tx=0x7f005c024000).stop 2022-07-21T01:33:06.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.770+0000 7f0074217700 1 -- 172.21.15.3:0/1700337444 shutdown_connections 2022-07-21T01:33:06.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.770+0000 7f0074217700 1 --2- 172.21.15.3:0/1700337444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0054004ec0 0x7f006c13bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.770+0000 7f0074217700 1 --2- 172.21.15.3:0/1700337444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0054004500 0x7f006c151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.770+0000 7f0074217700 1 --2- 172.21.15.3:0/1700337444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0054003750 0x7f006c1425d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:06.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.770+0000 7f0074217700 1 -- 172.21.15.3:0/1700337444 >> 172.21.15.3:0/1700337444 conn(0x7f006c062a30 msgr2=0x7f006c0b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:06.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.770+0000 7f0074217700 1 -- 172.21.15.3:0/1700337444 shutdown_connections 2022-07-21T01:33:06.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:06.770+0000 7f0074217700 1 -- 172.21.15.3:0/1700337444 wait complete. 2022-07-21T01:33:06.769 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2022-07-21T01:33:06.779 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-07-21T01:33:06.780 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:33:06.787 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.KPw0Zx5sIl 2022-07-21T01:33:06.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.KPw0Zx5sIl 2022-07-21T01:33:06.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.KPw0Zx5sIl 2022-07-21T01:33:06.865 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.KPw0Zx5sIl is 37B 2022-07-21T01:33:06.881 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.KPw0Zx5sIl 2022-07-21T01:33:06.888 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:33:06.935 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.No0kkjbxXP 2022-07-21T01:33:06.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.No0kkjbxXP 2022-07-21T01:33:06.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.No0kkjbxXP 2022-07-21T01:33:06.974 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.No0kkjbxXP is 37B 2022-07-21T01:33:06.987 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.No0kkjbxXP 2022-07-21T01:33:06.993 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:33:07.039 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.yZidVewVFv 2022-07-21T01:33:07.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.yZidVewVFv 2022-07-21T01:33:07.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.yZidVewVFv 2022-07-21T01:33:07.116 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.yZidVewVFv is 37B 2022-07-21T01:33:07.130 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.yZidVewVFv 2022-07-21T01:33:07.137 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T01:33:07.183 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.jGrHG4krWU 2022-07-21T01:33:07.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.jGrHG4krWU 2022-07-21T01:33:07.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.jGrHG4krWU 2022-07-21T01:33:07.226 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.jGrHG4krWU is 37B 2022-07-21T01:33:07.239 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.jGrHG4krWU 2022-07-21T01:33:07.245 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:33:07.252 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.Qbp3JIQVFC 2022-07-21T01:33:07.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.Qbp3JIQVFC 2022-07-21T01:33:07.322 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.Qbp3JIQVFC 2022-07-21T01:33:07.349 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.Qbp3JIQVFC is 37B 2022-07-21T01:33:07.403 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.Qbp3JIQVFC 2022-07-21T01:33:07.411 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:33:07.458 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.Bk34BVME6b 2022-07-21T01:33:07.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.Bk34BVME6b 2022-07-21T01:33:07.473 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.Bk34BVME6b 2022-07-21T01:33:07.498 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.Bk34BVME6b is 37B 2022-07-21T01:33:07.549 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.Bk34BVME6b 2022-07-21T01:33:07.557 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:33:07.563 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.Nu6ZMO5KNA 2022-07-21T01:33:07.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.Nu6ZMO5KNA 2022-07-21T01:33:07.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.Nu6ZMO5KNA 2022-07-21T01:33:07.646 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.Nu6ZMO5KNA is 37B 2022-07-21T01:33:07.697 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.Nu6ZMO5KNA 2022-07-21T01:33:07.704 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T01:33:07.754 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.FZE0eG5Ikl 2022-07-21T01:33:07.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.FZE0eG5Ikl 2022-07-21T01:33:07.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.FZE0eG5Ikl 2022-07-21T01:33:07.840 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.FZE0eG5Ikl is 37B 2022-07-21T01:33:07.854 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.FZE0eG5Ikl 2022-07-21T01:33:07.860 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 098a3930-1b74-4dbc-980c-5137ac329c44 0 2022-07-21T01:33:08.005 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 Processor -- start 2022-07-21T01:33:08.005 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 -- start start 2022-07-21T01:33:08.007 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe70c0b2930 0x7fe70c147600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.008 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe70c0b1ea0 0x7fe70c14b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.008 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c05c620 0x7fe70c0a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.008 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70c145340 con 0x7fe70c0b1ea0 2022-07-21T01:33:08.009 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70c0b2d10 con 0x7fe70c0b2930 2022-07-21T01:33:08.009 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c14ac60 con 0x7fe70c05c620 2022-07-21T01:33:08.010 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe710a57700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe70c0b2930 0x7fe70c147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.010 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe710a57700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe70c0b2930 0x7fe70c147600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49570/0 (socket says 172.21.15.143:49570) 2022-07-21T01:33:08.011 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe710a57700 1 -- 172.21.15.143:0/2402509577 learned_addr learned my addr 172.21.15.143:0/2402509577 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:08.011 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711258700 1 --2- 172.21.15.143:0/2402509577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c05c620 0x7fe70c0a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.012 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe70bfff700 1 --2- 172.21.15.143:0/2402509577 >> v2:172.21.15.3:3300/0 conn(0x7fe70c0b1ea0 0x7fe70c14b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.012 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711258700 1 -- 172.21.15.143:0/2402509577 >> v2:172.21.15.143:3300/0 conn(0x7fe70c0b2930 msgr2=0x7fe70c147600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:08.012 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711258700 1 --2- 172.21.15.143:0/2402509577 >> v2:172.21.15.143:3300/0 conn(0x7fe70c0b2930 0x7fe70c147600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.013 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711258700 1 -- 172.21.15.143:0/2402509577 >> v2:172.21.15.3:3300/0 conn(0x7fe70c0b1ea0 msgr2=0x7fe70c14b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:08.013 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711258700 1 --2- 172.21.15.143:0/2402509577 >> v2:172.21.15.3:3300/0 conn(0x7fe70c0b1ea0 0x7fe70c14b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.014 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711258700 1 -- 172.21.15.143:0/2402509577 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c0b1050 con 0x7fe70c05c620 2022-07-21T01:33:08.014 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe710a57700 1 --2- 172.21.15.143:0/2402509577 >> v2:172.21.15.143:3300/0 conn(0x7fe70c0b2930 0x7fe70c147600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:08.015 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711258700 1 --2- 172.21.15.143:0/2402509577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c05c620 0x7fe70c0a8d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe7040032f0 tx=0x7fe7040091d0).ready entity=mon.2 client_cookie=afa349c753f72b85 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.015 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe70b7fe700 1 -- 172.21.15.143:0/2402509577 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe70400b680 con 0x7fe70c05c620 2022-07-21T01:33:08.016 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe70b7fe700 1 -- 172.21.15.143:0/2402509577 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70400b7e0 con 0x7fe70c05c620 2022-07-21T01:33:08.016 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe70b7fe700 1 -- 172.21.15.143:0/2402509577 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe70400b950 con 0x7fe70c05c620 2022-07-21T01:33:08.022 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 -- 172.21.15.143:0/2402509577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c05c620 msgr2=0x7fe70c0a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.022 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 --2- 172.21.15.143:0/2402509577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c05c620 0x7fe70c0a8d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe7040032f0 tx=0x7fe7040091d0).stop 2022-07-21T01:33:08.022 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 -- 172.21.15.143:0/2402509577 shutdown_connections 2022-07-21T01:33:08.023 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 --2- 172.21.15.143:0/2402509577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c05c620 0x7fe70c0a8d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.023 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 --2- 172.21.15.143:0/2402509577 >> v2:172.21.15.3:3300/0 conn(0x7fe70c0b1ea0 0x7fe70c14b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.023 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 --2- 172.21.15.143:0/2402509577 >> v2:172.21.15.143:3300/0 conn(0x7fe70c0b2930 0x7fe70c147600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.023 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 -- 172.21.15.143:0/2402509577 >> 172.21.15.143:0/2402509577 conn(0x7fe70c0612d0 msgr2=0x7fe70c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.024 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 -- 172.21.15.143:0/2402509577 shutdown_connections 2022-07-21T01:33:08.024 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.006+0000 7fe711a59700 1 -- 172.21.15.143:0/2402509577 wait complete. 2022-07-21T01:33:08.024 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 Processor -- start 2022-07-21T01:33:08.024 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 -- start start 2022-07-21T01:33:08.025 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe70c05c620 0x7fe70c146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.025 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe70c0b1ea0 0x7fe70c0ac470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.025 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c0b2930 0x7fe70c156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.025 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c0abda0 con 0x7fe70c0b1ea0 2022-07-21T01:33:08.025 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c1498c0 con 0x7fe70c05c620 2022-07-21T01:33:08.026 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c149b60 con 0x7fe70c0b2930 2022-07-21T01:33:08.026 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711258700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c0b2930 0x7fe70c156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.026 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe710a57700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe70c05c620 0x7fe70c146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.026 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711258700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c0b2930 0x7fe70c156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51560/0 (socket says 172.21.15.143:51560) 2022-07-21T01:33:08.027 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe710a57700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe70c05c620 0x7fe70c146fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49576/0 (socket says 172.21.15.143:49576) 2022-07-21T01:33:08.027 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711258700 1 -- 172.21.15.143:0/3161323699 learned_addr learned my addr 172.21.15.143:0/3161323699 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:08.027 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe70bfff700 1 --2- 172.21.15.143:0/3161323699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe70c0b1ea0 0x7fe70c0ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.027 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe710a57700 1 -- 172.21.15.143:0/3161323699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c0b2930 msgr2=0x7fe70c156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.028 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe710a57700 1 --2- 172.21.15.143:0/3161323699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c0b2930 0x7fe70c156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.028 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe710a57700 1 -- 172.21.15.143:0/3161323699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe70c0b1ea0 msgr2=0x7fe70c0ac470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.028 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe710a57700 1 --2- 172.21.15.143:0/3161323699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe70c0b1ea0 0x7fe70c0ac470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.028 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe710a57700 1 -- 172.21.15.143:0/3161323699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc00b1e0 con 0x7fe70c05c620 2022-07-21T01:33:08.029 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe710a57700 1 --2- 172.21.15.143:0/3161323699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe70c05c620 0x7fe70c146fc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe6fc00c000 tx=0x7fe6fc006eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.029 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe709ffb700 1 -- 172.21.15.143:0/3161323699 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6fc012600 con 0x7fe70c05c620 2022-07-21T01:33:08.029 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe709ffb700 1 -- 172.21.15.143:0/3161323699 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc012760 con 0x7fe70c05c620 2022-07-21T01:33:08.029 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 -- 172.21.15.143:0/3161323699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe704009470 con 0x7fe70c05c620 2022-07-21T01:33:08.030 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe709ffb700 1 -- 172.21.15.143:0/3161323699 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6fc012ac0 con 0x7fe70c05c620 2022-07-21T01:33:08.030 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 -- 172.21.15.143:0/3161323699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70c0ac960 con 0x7fe70c05c620 2022-07-21T01:33:08.030 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe709ffb700 1 -- 172.21.15.143:0/3161323699 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fe6fc022070 con 0x7fe70c05c620 2022-07-21T01:33:08.030 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe709ffb700 1 -- 172.21.15.143:0/3161323699 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe6fc01b9f0 con 0x7fe70c05c620 2022-07-21T01:33:08.031 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.010+0000 7fe711a59700 1 -- 172.21.15.143:0/3161323699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70c0574e0 con 0x7fe70c05c620 2022-07-21T01:33:08.031 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.014+0000 7fe709ffb700 1 -- 172.21.15.143:0/3161323699 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fe6fc016020 con 0x7fe70c05c620 2022-07-21T01:33:08.083 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.082+0000 7fe711a59700 1 -- 172.21.15.143:0/3161323699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", "id": 0} v 0) v1 -- 0x7fe70c0ac1d0 con 0x7fe70c05c620 2022-07-21T01:33:08.092 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.090+0000 7fe709ffb700 1 -- 172.21.15.143:0/3161323699 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe6fc01bbb0 con 0x7fe70c05c620 2022-07-21T01:33:08.092 INFO:teuthology.orchestra.run.smithi143.stdout:0 2022-07-21T01:33:08.093 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.094+0000 7fe711a59700 1 -- 172.21.15.143:0/3161323699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe70c05c620 msgr2=0x7fe70c146fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.094 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.094+0000 7fe711a59700 1 --2- 172.21.15.143:0/3161323699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe70c05c620 0x7fe70c146fc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe6fc00c000 tx=0x7fe6fc006eb0).stop 2022-07-21T01:33:08.094 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.094+0000 7fe711a59700 1 -- 172.21.15.143:0/3161323699 shutdown_connections 2022-07-21T01:33:08.094 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.094+0000 7fe711a59700 1 --2- 172.21.15.143:0/3161323699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe70c0b2930 0x7fe70c156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.095 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.094+0000 7fe711a59700 1 --2- 172.21.15.143:0/3161323699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe70c0b1ea0 0x7fe70c0ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.095 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.094+0000 7fe711a59700 1 --2- 172.21.15.143:0/3161323699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe70c05c620 0x7fe70c146fc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.096 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.094+0000 7fe711a59700 1 -- 172.21.15.143:0/3161323699 >> 172.21.15.143:0/3161323699 conn(0x7fe70c0612d0 msgr2=0x7fe70c0b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.096 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.094+0000 7fe711a59700 1 -- 172.21.15.143:0/3161323699 shutdown_connections 2022-07-21T01:33:08.096 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.094+0000 7fe711a59700 1 -- 172.21.15.143:0/3161323699 wait complete. 2022-07-21T01:33:08.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new ec4680af-1feb-47f3-949c-315962389518 1 2022-07-21T01:33:08.206 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406cefc700 1 Processor -- start 2022-07-21T01:33:08.206 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406cefc700 1 -- start start 2022-07-21T01:33:08.210 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406cefc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40680b0740 0x7f40680b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.210 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406cefc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f40680a9da0 0x7f40680b1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.210 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406cefc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4068143ab0 con 0x7f40680b38a0 2022-07-21T01:33:08.210 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406cefc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4068143d50 con 0x7f40680a9da0 2022-07-21T01:33:08.211 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406cefc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4068143ff0 con 0x7f40680b0740 2022-07-21T01:33:08.211 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406659c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40680b0740 0x7f40680b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.211 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406659c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40680b0740 0x7f40680b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49582/0 (socket says 172.21.15.143:49582) 2022-07-21T01:33:08.211 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f4065d9b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f40680b38a0 0x7f406813d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:55616/0 (socket says 172.21.15.143:55616) 2022-07-21T01:33:08.212 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406659c700 1 -- 172.21.15.143:0/3634705450 learned_addr learned my addr 172.21.15.143:0/3634705450 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:08.212 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f4066d9d700 1 --2- 172.21.15.143:0/3634705450 >> v2:172.21.15.3:3300/0 conn(0x7f40680a9da0 0x7f40680b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.212 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406659c700 1 -- 172.21.15.143:0/3634705450 >> v2:172.21.15.3:3300/0 conn(0x7f40680a9da0 msgr2=0x7f40680b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:08.213 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406659c700 1 --2- 172.21.15.143:0/3634705450 >> v2:172.21.15.3:3300/0 conn(0x7f40680a9da0 0x7f40680b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.213 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406659c700 1 -- 172.21.15.143:0/3634705450 >> v1:172.21.15.143:6789/0 conn(0x7f40680b38a0 legacy=0x7f406813d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.213 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406659c700 1 -- 172.21.15.143:0/3634705450 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4068144290 con 0x7f40680b0740 2022-07-21T01:33:08.213 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f4066d9d700 1 --2- 172.21.15.143:0/3634705450 >> v2:172.21.15.3:3300/0 conn(0x7f40680a9da0 0x7f40680b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:08.213 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.206+0000 7f406659c700 1 --2- 172.21.15.143:0/3634705450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b0740 0x7f40680b0b20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f40580032f0 tx=0x7f4058009290).ready entity=mon.1 client_cookie=85d162e0d421cb33 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.214 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406559a700 1 -- 172.21.15.143:0/3634705450 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405800b870 con 0x7f40680b0740 2022-07-21T01:33:08.214 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406559a700 1 -- 172.21.15.143:0/3634705450 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405800b9d0 con 0x7f40680b0740 2022-07-21T01:33:08.214 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406559a700 1 -- 172.21.15.143:0/3634705450 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405800bb40 con 0x7f40680b0740 2022-07-21T01:33:08.214 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 -- 172.21.15.143:0/3634705450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b0740 msgr2=0x7f40680b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.215 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 --2- 172.21.15.143:0/3634705450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b0740 0x7f40680b0b20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f40580032f0 tx=0x7f4058009290).stop 2022-07-21T01:33:08.215 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 -- 172.21.15.143:0/3634705450 shutdown_connections 2022-07-21T01:33:08.215 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 --2- 172.21.15.143:0/3634705450 >> v2:172.21.15.3:3300/0 conn(0x7f40680a9da0 0x7f40680b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.216 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 --2- 172.21.15.143:0/3634705450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b0740 0x7f40680b0b20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.216 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 -- 172.21.15.143:0/3634705450 >> 172.21.15.143:0/3634705450 conn(0x7f4068060950 msgr2=0x7f4068060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.216 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 -- 172.21.15.143:0/3634705450 shutdown_connections 2022-07-21T01:33:08.216 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 -- 172.21.15.143:0/3634705450 wait complete. 2022-07-21T01:33:08.220 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 Processor -- start 2022-07-21T01:33:08.220 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 -- start start 2022-07-21T01:33:08.220 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.210+0000 7f406cefc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40680a9da0 0x7f406805fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.220 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406cefc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40680b0740 0x7f4068138af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.221 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406cefc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b38a0 0x7f4068138fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.221 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406cefc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40681504d0 con 0x7f40680b0740 2022-07-21T01:33:08.221 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406cefc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40680b5f10 con 0x7f40680b38a0 2022-07-21T01:33:08.221 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406cefc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40680ac8e0 con 0x7f40680a9da0 2022-07-21T01:33:08.221 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40680a9da0 0x7f406805fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.222 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f4066d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b38a0 0x7f4068138fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.222 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40680a9da0 0x7f406805fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51568/0 (socket says 172.21.15.143:51568) 2022-07-21T01:33:08.222 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406659c700 1 -- 172.21.15.143:0/161576705 learned_addr learned my addr 172.21.15.143:0/161576705 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:08.222 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f4065d9b700 1 --2- 172.21.15.143:0/161576705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40680b0740 0x7f4068138af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.223 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f4066d9d700 1 -- 172.21.15.143:0/161576705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40680a9da0 msgr2=0x7f406805fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.223 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f4066d9d700 1 --2- 172.21.15.143:0/161576705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40680a9da0 0x7f406805fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.223 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f4066d9d700 1 -- 172.21.15.143:0/161576705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40680b0740 msgr2=0x7f4068138af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.223 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f4066d9d700 1 --2- 172.21.15.143:0/161576705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40680b0740 0x7f4068138af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.224 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f4066d9d700 1 -- 172.21.15.143:0/161576705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4058009530 con 0x7f40680b38a0 2022-07-21T01:33:08.224 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406659c700 1 --2- 172.21.15.143:0/161576705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40680a9da0 0x7f406805fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:08.224 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f4066d9d700 1 --2- 172.21.15.143:0/161576705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b38a0 0x7f4068138fe0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f406000a000 tx=0x7f4060008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.224 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f405f7fe700 1 -- 172.21.15.143:0/161576705 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4060011780 con 0x7f40680b38a0 2022-07-21T01:33:08.225 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f405f7fe700 1 -- 172.21.15.143:0/161576705 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406000de90 con 0x7f40680b38a0 2022-07-21T01:33:08.225 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406cefc700 1 -- 172.21.15.143:0/161576705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40680b5c70 con 0x7f40680b38a0 2022-07-21T01:33:08.225 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f405f7fe700 1 -- 172.21.15.143:0/161576705 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4060011ad0 con 0x7f40680b38a0 2022-07-21T01:33:08.225 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406cefc700 1 -- 172.21.15.143:0/161576705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40681394d0 con 0x7f40680b38a0 2022-07-21T01:33:08.226 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f405f7fe700 1 -- 172.21.15.143:0/161576705 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f4060006020 con 0x7f40680b38a0 2022-07-21T01:33:08.226 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f405f7fe700 1 -- 172.21.15.143:0/161576705 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7f4060011e40 con 0x7f40680b38a0 2022-07-21T01:33:08.226 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.214+0000 7f406cefc700 1 -- 172.21.15.143:0/161576705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406813ddf0 con 0x7f40680b38a0 2022-07-21T01:33:08.226 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.218+0000 7f405f7fe700 1 -- 172.21.15.143:0/161576705 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f4060005ad0 con 0x7f40680b38a0 2022-07-21T01:33:08.290 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.290+0000 7f406cefc700 1 -- 172.21.15.143:0/161576705 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ec4680af-1feb-47f3-949c-315962389518", "id": 1} v 0) v1 -- 0x7f40680574e0 con 0x7f40680b38a0 2022-07-21T01:33:08.301 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f405f7fe700 1 -- 172.21.15.143:0/161576705 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ec4680af-1feb-47f3-949c-315962389518", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f40600290b0 con 0x7f40680b38a0 2022-07-21T01:33:08.302 INFO:teuthology.orchestra.run.smithi143.stdout:1 2022-07-21T01:33:08.303 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f406cefc700 1 -- 172.21.15.143:0/161576705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b38a0 msgr2=0x7f4068138fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.303 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f406cefc700 1 --2- 172.21.15.143:0/161576705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b38a0 0x7f4068138fe0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f406000a000 tx=0x7f4060008040).stop 2022-07-21T01:33:08.303 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f406cefc700 1 -- 172.21.15.143:0/161576705 shutdown_connections 2022-07-21T01:33:08.304 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f406cefc700 1 --2- 172.21.15.143:0/161576705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40680b38a0 0x7f4068138fe0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.304 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f406cefc700 1 --2- 172.21.15.143:0/161576705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40680b0740 0x7f4068138af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.304 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f406cefc700 1 --2- 172.21.15.143:0/161576705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40680a9da0 0x7f406805fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.304 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f406cefc700 1 -- 172.21.15.143:0/161576705 >> 172.21.15.143:0/161576705 conn(0x7f4068060950 msgr2=0x7f406813baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.305 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f406cefc700 1 -- 172.21.15.143:0/161576705 shutdown_connections 2022-07-21T01:33:08.305 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.302+0000 7f406cefc700 1 -- 172.21.15.143:0/161576705 wait complete. 2022-07-21T01:33:08.315 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new e321c85f-57d4-4b3d-92c6-18843b5f43aa 2 2022-07-21T01:33:08.409 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 Processor -- start 2022-07-21T01:33:08.409 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 -- start start 2022-07-21T01:33:08.412 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe89c060960 0x7fe89c140230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.412 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c140720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.412 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c060010 con 0x7fe89c05fc30 2022-07-21T01:33:08.413 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c1433d0 con 0x7fe89c060960 2022-07-21T01:33:08.413 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0acdf0 con 0x7fe89c0ac690 2022-07-21T01:33:08.413 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a28d9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.413 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a28d9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c140720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51578/0 (socket says 172.21.15.143:51578) 2022-07-21T01:33:08.414 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a28d9700 1 -- 172.21.15.143:0/710802456 learned_addr learned my addr 172.21.15.143:0/710802456 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:08.414 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a18d7700 1 --2- 172.21.15.143:0/710802456 >> v2:172.21.15.3:3300/0 conn(0x7fe89c060960 0x7fe89c140230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.414 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a28d9700 1 -- 172.21.15.143:0/710802456 >> v2:172.21.15.3:3300/0 conn(0x7fe89c060960 msgr2=0x7fe89c140230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:08.414 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a28d9700 1 --2- 172.21.15.143:0/710802456 >> v2:172.21.15.3:3300/0 conn(0x7fe89c060960 0x7fe89c140230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.415 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a28d9700 1 -- 172.21.15.143:0/710802456 >> v1:172.21.15.3:6789/0 conn(0x7fe89c05fc30 legacy=0x7fe89c143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.415 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a28d9700 1 -- 172.21.15.143:0/710802456 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0ad090 con 0x7fe89c0ac690 2022-07-21T01:33:08.415 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a18d7700 1 --2- 172.21.15.143:0/710802456 >> v2:172.21.15.3:3300/0 conn(0x7fe89c060960 0x7fe89c140230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:08.416 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a28d9700 1 --2- 172.21.15.143:0/710802456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c140720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe8940032f0 tx=0x7fe894009290).ready entity=mon.2 client_cookie=b1ed867ffa8131f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.416 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a10d6700 1 -- 172.21.15.143:0/710802456 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe89400b8a0 con 0x7fe89c0ac690 2022-07-21T01:33:08.416 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a10d6700 1 -- 172.21.15.143:0/710802456 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89400ba00 con 0x7fe89c0ac690 2022-07-21T01:33:08.416 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a10d6700 1 -- 172.21.15.143:0/710802456 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe89400bb70 con 0x7fe89c0ac690 2022-07-21T01:33:08.417 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 -- 172.21.15.143:0/710802456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 msgr2=0x7fe89c140720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.417 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 --2- 172.21.15.143:0/710802456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c140720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe8940032f0 tx=0x7fe894009290).stop 2022-07-21T01:33:08.417 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 -- 172.21.15.143:0/710802456 shutdown_connections 2022-07-21T01:33:08.417 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 --2- 172.21.15.143:0/710802456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c140720 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.418 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 --2- 172.21.15.143:0/710802456 >> v2:172.21.15.3:3300/0 conn(0x7fe89c060960 0x7fe89c140230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.418 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 -- 172.21.15.143:0/710802456 >> 172.21.15.143:0/710802456 conn(0x7fe89c0b2b00 msgr2=0x7fe89c0b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.418 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 -- 172.21.15.143:0/710802456 shutdown_connections 2022-07-21T01:33:08.418 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.410+0000 7fe8a433c700 1 -- 172.21.15.143:0/710802456 wait complete. 2022-07-21T01:33:08.421 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 Processor -- start 2022-07-21T01:33:08.422 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 -- start start 2022-07-21T01:33:08.422 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe89c05fc30 0x7fe89c0b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.422 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe89c060960 0x7fe89c0ad630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.423 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c0ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.423 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0b5680 con 0x7fe89c060960 2022-07-21T01:33:08.423 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0adb20 con 0x7fe89c05fc30 2022-07-21T01:33:08.423 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0b47b0 con 0x7fe89c0ac690 2022-07-21T01:33:08.424 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a28d9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c0ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.424 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a20d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe89c05fc30 0x7fe89c0b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.424 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a28d9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c0ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51584/0 (socket says 172.21.15.143:51584) 2022-07-21T01:33:08.424 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a28d9700 1 -- 172.21.15.143:0/1011985320 learned_addr learned my addr 172.21.15.143:0/1011985320 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:08.425 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a18d7700 1 --2- 172.21.15.143:0/1011985320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe89c060960 0x7fe89c0ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.425 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a20d8700 1 -- 172.21.15.143:0/1011985320 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 msgr2=0x7fe89c0ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.425 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a20d8700 1 --2- 172.21.15.143:0/1011985320 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c0ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.425 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a20d8700 1 -- 172.21.15.143:0/1011985320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe89c060960 msgr2=0x7fe89c0ad630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.425 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a20d8700 1 --2- 172.21.15.143:0/1011985320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe89c060960 0x7fe89c0ad630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.426 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a20d8700 1 -- 172.21.15.143:0/1011985320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894009530 con 0x7fe89c05fc30 2022-07-21T01:33:08.426 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a28d9700 1 --2- 172.21.15.143:0/1011985320 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c0ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:08.426 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a20d8700 1 --2- 172.21.15.143:0/1011985320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe89c05fc30 0x7fe89c0b5190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe89c0b7080 tx=0x7fe898007b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.427 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe88b7fe700 1 -- 172.21.15.143:0/1011985320 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe89800cdb0 con 0x7fe89c05fc30 2022-07-21T01:33:08.427 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe88b7fe700 1 -- 172.21.15.143:0/1011985320 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe898007030 con 0x7fe89c05fc30 2022-07-21T01:33:08.427 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 -- 172.21.15.143:0/1011985320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c0b4a50 con 0x7fe89c05fc30 2022-07-21T01:33:08.427 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe88b7fe700 1 -- 172.21.15.143:0/1011985320 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe898007390 con 0x7fe89c05fc30 2022-07-21T01:33:08.428 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe8a433c700 1 -- 172.21.15.143:0/1011985320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c0ae6d0 con 0x7fe89c05fc30 2022-07-21T01:33:08.428 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe88b7fe700 1 -- 172.21.15.143:0/1011985320 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fe8980066e0 con 0x7fe89c05fc30 2022-07-21T01:33:08.428 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.414+0000 7fe88b7fe700 1 -- 172.21.15.143:0/1011985320 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7fe898006970 con 0x7fe89c05fc30 2022-07-21T01:33:08.428 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.418+0000 7fe8a433c700 1 -- 172.21.15.143:0/1011985320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c0b45f0 con 0x7fe89c05fc30 2022-07-21T01:33:08.428 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.418+0000 7fe88b7fe700 1 -- 172.21.15.143:0/1011985320 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fe898006dd0 con 0x7fe89c05fc30 2022-07-21T01:33:08.480 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.482+0000 7fe8a433c700 1 -- 172.21.15.143:0/1011985320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", "id": 2} v 0) v1 -- 0x7fe89c0574e0 con 0x7fe89c05fc30 2022-07-21T01:33:08.490 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe88b7fe700 1 -- 172.21.15.143:0/1011985320 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe898014410 con 0x7fe89c05fc30 2022-07-21T01:33:08.491 INFO:teuthology.orchestra.run.smithi143.stdout:2 2022-07-21T01:33:08.493 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe8a433c700 1 -- 172.21.15.143:0/1011985320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe89c05fc30 msgr2=0x7fe89c0b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.493 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe8a433c700 1 --2- 172.21.15.143:0/1011985320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe89c05fc30 0x7fe89c0b5190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe89c0b7080 tx=0x7fe898007b50).stop 2022-07-21T01:33:08.494 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe8a433c700 1 -- 172.21.15.143:0/1011985320 shutdown_connections 2022-07-21T01:33:08.494 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe8a433c700 1 --2- 172.21.15.143:0/1011985320 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe89c0ac690 0x7fe89c0ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.495 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe8a433c700 1 --2- 172.21.15.143:0/1011985320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe89c060960 0x7fe89c0ad630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.495 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe8a433c700 1 --2- 172.21.15.143:0/1011985320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe89c05fc30 0x7fe89c0b5190 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.495 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe8a433c700 1 -- 172.21.15.143:0/1011985320 >> 172.21.15.143:0/1011985320 conn(0x7fe89c0b2b00 msgr2=0x7fe89c0b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.496 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe8a433c700 1 -- 172.21.15.143:0/1011985320 shutdown_connections 2022-07-21T01:33:08.496 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.490+0000 7fe8a433c700 1 -- 172.21.15.143:0/1011985320 wait complete. 2022-07-21T01:33:08.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 7b3e9c1c-2503-46ca-871b-61bd90000981 3 2022-07-21T01:33:08.604 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.602+0000 7fbf93048700 1 Processor -- start 2022-07-21T01:33:08.604 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.602+0000 7fbf93048700 1 -- start start 2022-07-21T01:33:08.618 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbf8c0b0740 0x7fbf8c0b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.619 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8c143ab0 con 0x7fbf8c0a9da0 2022-07-21T01:33:08.620 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8c143d50 con 0x7fbf8c0b38a0 2022-07-21T01:33:08.620 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8c143ff0 con 0x7fbf8c0b0740 2022-07-21T01:33:08.621 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8bfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fbf8c0b38a0 0x7fbf8c13d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:55640/0 (socket says 172.21.15.143:55640) 2022-07-21T01:33:08.621 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8bfff700 1 -- 172.21.15.143:0/2388164246 learned_addr learned my addr 172.21.15.143:0/2388164246 (peer_addr_for_me v1:172.21.15.143:0/0) 2022-07-21T01:33:08.622 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf90de4700 1 --2- 172.21.15.143:0/2388164246 >> v2:172.21.15.3:3300/0 conn(0x7fbf8c0b0740 0x7fbf8c0b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.622 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661750241 0 0) 0x7fbf8c143d50 con 0x7fbf8c0b38a0 2022-07-21T01:33:08.623 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf70003260 con 0x7fbf8c0b38a0 2022-07-21T01:33:08.623 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780439186 0 0) 0x7fbf8c143ab0 con 0x7fbf8c0a9da0 2022-07-21T01:33:08.624 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf8c143d50 con 0x7fbf8c0a9da0 2022-07-21T01:33:08.624 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1093030360 0 0) 0x7fbf70003260 con 0x7fbf8c0b38a0 2022-07-21T01:33:08.625 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 >> v2:172.21.15.3:3300/0 conn(0x7fbf8c0b0740 msgr2=0x7fbf8c0b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:08.625 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 --2- 172.21.15.143:0/2388164246 >> v2:172.21.15.3:3300/0 conn(0x7fbf8c0b0740 0x7fbf8c0b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.626 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 >> v1:172.21.15.3:6789/0 conn(0x7fbf8c0a9da0 legacy=0x7fbf8c0b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.626 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c144290 con 0x7fbf8c0b38a0 2022-07-21T01:33:08.627 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fbf78001660 con 0x7fbf8c0b38a0 2022-07-21T01:33:08.627 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 >> v1:172.21.15.143:6789/0 conn(0x7fbf8c0b38a0 legacy=0x7fbf8c13d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.627 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf700038e0 0x7fbf70003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.627 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf70004500 0x7fbf70004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.628 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf70004e70 0x7fbf7000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.628 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c143ff0 con 0x7fbf70004e70 2022-07-21T01:33:08.628 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7000bdd0 con 0x7fbf70004500 2022-07-21T01:33:08.628 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7000bf40 con 0x7fbf700038e0 2022-07-21T01:33:08.629 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8bfff700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf70004500 0x7fbf70004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.629 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf915e5700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf70004e70 0x7fbf7000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.629 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf90de4700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf700038e0 0x7fbf70003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.629 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8bfff700 1 -- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf700038e0 msgr2=0x7fbf70003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:08.630 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8bfff700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf700038e0 0x7fbf70003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.630 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8bfff700 1 -- 172.21.15.143:0/2388164246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf70004e70 msgr2=0x7fbf7000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:08.631 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8bfff700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf70004e70 0x7fbf7000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.631 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8bfff700 1 -- 172.21.15.143:0/2388164246 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf70003260 con 0x7fbf70004500 2022-07-21T01:33:08.632 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf915e5700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf70004e70 0x7fbf7000b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:08.632 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf915e5700 1 -- 172.21.15.143:0/2388164246 reap_dead start 2022-07-21T01:33:08.633 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8bfff700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf70004500 0x7fbf70004980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fbf78001430 tx=0x7fbf8c143ee0).ready entity=mon.1 client_cookie=b7ce83d3cf2a6103 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.633 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf780071a0 con 0x7fbf70004500 2022-07-21T01:33:08.634 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf78007300 con 0x7fbf70004500 2022-07-21T01:33:08.634 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf8b7fe700 1 -- 172.21.15.143:0/2388164246 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf7800c8f0 con 0x7fbf70004500 2022-07-21T01:33:08.635 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 -- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf70004500 msgr2=0x7fbf70004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.635 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf70004500 0x7fbf70004980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fbf78001430 tx=0x7fbf8c143ee0).stop 2022-07-21T01:33:08.635 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 -- 172.21.15.143:0/2388164246 shutdown_connections 2022-07-21T01:33:08.636 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 --2- 172.21.15.143:0/2388164246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf70004500 0x7fbf70004980 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.636 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 -- 172.21.15.143:0/2388164246 >> 172.21.15.143:0/2388164246 conn(0x7fbf8c060950 msgr2=0x7fbf8c060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.636 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 -- 172.21.15.143:0/2388164246 shutdown_connections 2022-07-21T01:33:08.637 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.606+0000 7fbf93048700 1 -- 172.21.15.143:0/2388164246 wait complete. 2022-07-21T01:33:08.637 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 Processor -- start 2022-07-21T01:33:08.638 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 -- start start 2022-07-21T01:33:08.638 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf70004500 0x7fbf8c05fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.639 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf8c063a60 0x7fbf8c13cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.639 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf8c13d3e0 0x7fbf8c0aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.640 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c067190 con 0x7fbf8c063a60 2022-07-21T01:33:08.640 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c154380 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.640 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c154620 con 0x7fbf70004500 2022-07-21T01:33:08.641 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf90de4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf70004500 0x7fbf8c05fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.641 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf90de4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf70004500 0x7fbf8c05fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51598/0 (socket says 172.21.15.143:51598) 2022-07-21T01:33:08.642 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf90de4700 1 -- 172.21.15.143:0/90791206 learned_addr learned my addr 172.21.15.143:0/90791206 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:08.642 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf915e5700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf8c13d3e0 0x7fbf8c0aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.643 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf8bfff700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf8c063a60 0x7fbf8c13cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.644 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf915e5700 1 -- 172.21.15.143:0/90791206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf70004500 msgr2=0x7fbf8c05fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.644 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf915e5700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf70004500 0x7fbf8c05fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.645 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf915e5700 1 -- 172.21.15.143:0/90791206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf8c063a60 msgr2=0x7fbf8c13cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.645 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf915e5700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf8c063a60 0x7fbf8c13cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.646 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf915e5700 1 -- 172.21.15.143:0/90791206 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c0673f0 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.646 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf90de4700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf70004500 0x7fbf8c05fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf915e5700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf8c13d3e0 0x7fbf8c0aea20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fbf84001550 tx=0x7fbf84009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf89ffb700 1 -- 172.21.15.143:0/90791206 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf84015070 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.647 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf89ffb700 1 -- 172.21.15.143:0/90791206 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf84003220 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.648 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 -- 172.21.15.143:0/90791206 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8c13d860 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.648 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf89ffb700 1 -- 172.21.15.143:0/90791206 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf84003580 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.649 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.610+0000 7fbf93048700 1 -- 172.21.15.143:0/90791206 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8c067890 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.649 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.614+0000 7fbf89ffb700 1 -- 172.21.15.143:0/90791206 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fbf84022070 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.650 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.614+0000 7fbf89ffb700 1 -- 172.21.15.143:0/90791206 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7fbf8401a8d0 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.650 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.614+0000 7fbf93048700 1 -- 172.21.15.143:0/90791206 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8c067580 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.651 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.618+0000 7fbf89ffb700 1 -- 172.21.15.143:0/90791206 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fbf84020060 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.682 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.682+0000 7fbf93048700 1 -- 172.21.15.143:0/90791206 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", "id": 3} v 0) v1 -- 0x7fbf8c143730 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.692 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.690+0000 7fbf89ffb700 1 -- 172.21.15.143:0/90791206 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbf84029020 con 0x7fbf8c13d3e0 2022-07-21T01:33:08.692 INFO:teuthology.orchestra.run.smithi143.stdout:3 2022-07-21T01:33:08.694 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.694+0000 7fbf93048700 1 -- 172.21.15.143:0/90791206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf8c13d3e0 msgr2=0x7fbf8c0aea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.694 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.694+0000 7fbf93048700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf8c13d3e0 0x7fbf8c0aea20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fbf84001550 tx=0x7fbf84009040).stop 2022-07-21T01:33:08.694 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.694+0000 7fbf93048700 1 -- 172.21.15.143:0/90791206 shutdown_connections 2022-07-21T01:33:08.694 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.694+0000 7fbf93048700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf8c13d3e0 0x7fbf8c0aea20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.695 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.694+0000 7fbf93048700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf8c063a60 0x7fbf8c13cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.695 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.694+0000 7fbf93048700 1 --2- 172.21.15.143:0/90791206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf70004500 0x7fbf8c05fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.695 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.694+0000 7fbf93048700 1 -- 172.21.15.143:0/90791206 >> 172.21.15.143:0/90791206 conn(0x7fbf8c060950 msgr2=0x7fbf8c144840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.695 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.694+0000 7fbf93048700 1 -- 172.21.15.143:0/90791206 shutdown_connections 2022-07-21T01:33:08.696 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.694+0000 7fbf93048700 1 -- 172.21.15.143:0/90791206 wait complete. 2022-07-21T01:33:08.705 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 86ac80f1-2bd0-43c8-8335-a06fe0ac9606 4 2022-07-21T01:33:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.806+0000 7f7fdff5b700 1 Processor -- start 2022-07-21T01:33:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.806+0000 7f7fdff5b700 1 -- start start 2022-07-21T01:33:08.811 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.806+0000 7f7fdff5b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8061860 0x7f7fd80b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.811 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.806+0000 7f7fdff5b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd8140220 con 0x7f7fd80b1b10 2022-07-21T01:33:08.811 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.806+0000 7f7fdff5b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd81404c0 con 0x7f7fd8060d00 2022-07-21T01:33:08.812 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.806+0000 7f7fdff5b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8061150 con 0x7f7fd8061860 2022-07-21T01:33:08.812 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fde4f8700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f7fd8060d00 0x7f7fd805f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:55660/0 (socket says 172.21.15.143:55660) 2022-07-21T01:33:08.812 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fde4f8700 1 -- 172.21.15.143:0/4006898425 learned_addr learned my addr 172.21.15.143:0/4006898425 (peer_addr_for_me v1:172.21.15.143:0/0) 2022-07-21T01:33:08.812 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdd4f6700 1 --2- 172.21.15.143:0/4006898425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8061860 0x7f7fd80b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.813 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdccf5700 1 -- 172.21.15.143:0/4006898425 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511513928 0 0) 0x7f7fd81404c0 con 0x7f7fd8060d00 2022-07-21T01:33:08.813 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdccf5700 1 -- 172.21.15.143:0/4006898425 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc0003260 con 0x7f7fd8060d00 2022-07-21T01:33:08.814 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdd4f6700 1 -- 172.21.15.143:0/4006898425 >> v1:172.21.15.143:6789/0 conn(0x7f7fd8060d00 legacy=0x7f7fd805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.814 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdd4f6700 1 -- 172.21.15.143:0/4006898425 >> v1:172.21.15.3:6789/0 conn(0x7f7fd80b1b10 legacy=0x7f7fd8139e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.815 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdd4f6700 1 -- 172.21.15.143:0/4006898425 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd8140760 con 0x7f7fd8061860 2022-07-21T01:33:08.815 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdd4f6700 1 --2- 172.21.15.143:0/4006898425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8061860 0x7f7fd80b1ef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7fc8002f70 tx=0x7f7fc8006df0).ready entity=mon.2 client_cookie=f1b1785d2de82bef server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.815 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdccf5700 1 -- 172.21.15.143:0/4006898425 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fc800d570 con 0x7f7fd8061860 2022-07-21T01:33:08.816 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdccf5700 1 -- 172.21.15.143:0/4006898425 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc8000e90 con 0x7f7fd8061860 2022-07-21T01:33:08.817 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdccf5700 1 -- 172.21.15.143:0/4006898425 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fc800d8c0 con 0x7f7fd8061860 2022-07-21T01:33:08.817 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 -- 172.21.15.143:0/4006898425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8061860 msgr2=0x7f7fd80b1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.817 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 --2- 172.21.15.143:0/4006898425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8061860 0x7f7fd80b1ef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7fc8002f70 tx=0x7f7fc8006df0).stop 2022-07-21T01:33:08.817 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 -- 172.21.15.143:0/4006898425 shutdown_connections 2022-07-21T01:33:08.818 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 --2- 172.21.15.143:0/4006898425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8061860 0x7f7fd80b1ef0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.818 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 -- 172.21.15.143:0/4006898425 >> 172.21.15.143:0/4006898425 conn(0x7f7fd80b2b00 msgr2=0x7f7fd80b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.818 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 -- 172.21.15.143:0/4006898425 shutdown_connections 2022-07-21T01:33:08.819 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 -- 172.21.15.143:0/4006898425 wait complete. 2022-07-21T01:33:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 Processor -- start 2022-07-21T01:33:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 -- start start 2022-07-21T01:33:08.825 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd8060d00 0x7f7fd813cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.825 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd8061860 0x7f7fd8065ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd80b1b10 0x7f7fd80663e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80670d0 con 0x7f7fd8060d00 2022-07-21T01:33:08.826 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd813c5e0 con 0x7f7fd8061860 2022-07-21T01:33:08.828 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fdff5b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8143560 con 0x7f7fd80b1b10 2022-07-21T01:33:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fde4f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd80b1b10 0x7f7fd80663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.829 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fde4f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd80b1b10 0x7f7fd80663e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51614/0 (socket says 172.21.15.143:51614) 2022-07-21T01:33:08.830 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.810+0000 7f7fde4f8700 1 -- 172.21.15.143:0/2390659634 learned_addr learned my addr 172.21.15.143:0/2390659634 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:08.830 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fdd4f6700 1 --2- 172.21.15.143:0/2390659634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd8061860 0x7f7fd8065ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.831 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fddcf7700 1 --2- 172.21.15.143:0/2390659634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd8060d00 0x7f7fd813cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:08.831 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fde4f8700 1 -- 172.21.15.143:0/2390659634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd8061860 msgr2=0x7f7fd8065ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.831 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fde4f8700 1 --2- 172.21.15.143:0/2390659634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd8061860 0x7f7fd8065ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.831 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fde4f8700 1 -- 172.21.15.143:0/2390659634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd8060d00 msgr2=0x7f7fd813cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.832 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fde4f8700 1 --2- 172.21.15.143:0/2390659634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd8060d00 0x7f7fd813cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.832 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fde4f8700 1 -- 172.21.15.143:0/2390659634 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc800b180 con 0x7f7fd80b1b10 2022-07-21T01:33:08.832 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fde4f8700 1 --2- 172.21.15.143:0/2390659634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd80b1b10 0x7f7fd80663e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7fd0001490 tx=0x7f7fd0007e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:08.832 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fc6ffd700 1 -- 172.21.15.143:0/2390659634 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd0005d90 con 0x7f7fd80b1b10 2022-07-21T01:33:08.833 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fc6ffd700 1 -- 172.21.15.143:0/2390659634 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd00070d0 con 0x7f7fd80b1b10 2022-07-21T01:33:08.833 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fdff5b700 1 -- 172.21.15.143:0/2390659634 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd8143800 con 0x7f7fd80b1b10 2022-07-21T01:33:08.833 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fc6ffd700 1 -- 172.21.15.143:0/2390659634 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd0007430 con 0x7f7fd80b1b10 2022-07-21T01:33:08.834 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fdff5b700 1 -- 172.21.15.143:0/2390659634 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd80668d0 con 0x7f7fd80b1b10 2022-07-21T01:33:08.834 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fc6ffd700 1 -- 172.21.15.143:0/2390659634 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f7fd0021070 con 0x7f7fd80b1b10 2022-07-21T01:33:08.835 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fc6ffd700 1 -- 172.21.15.143:0/2390659634 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f7fd0011c00 con 0x7f7fd80b1b10 2022-07-21T01:33:08.836 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.814+0000 7f7fdff5b700 1 -- 172.21.15.143:0/2390659634 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd813c3a0 con 0x7f7fd80b1b10 2022-07-21T01:33:08.836 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.818+0000 7f7fc6ffd700 1 -- 172.21.15.143:0/2390659634 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f7fd0038870 con 0x7f7fd80b1b10 2022-07-21T01:33:08.891 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.886+0000 7f7fdff5b700 1 -- 172.21.15.143:0/2390659634 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", "id": 4} v 0) v1 -- 0x7f7fd8143990 con 0x7f7fd80b1b10 2022-07-21T01:33:08.899 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.894+0000 7f7fc6ffd700 1 -- 172.21.15.143:0/2390659634 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7fd0028030 con 0x7f7fd80b1b10 2022-07-21T01:33:08.899 INFO:teuthology.orchestra.run.smithi143.stdout:4 2022-07-21T01:33:08.901 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.894+0000 7f7fdff5b700 1 -- 172.21.15.143:0/2390659634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd80b1b10 msgr2=0x7f7fd80663e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:08.901 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.894+0000 7f7fdff5b700 1 --2- 172.21.15.143:0/2390659634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd80b1b10 0x7f7fd80663e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7fd0001490 tx=0x7f7fd0007e00).stop 2022-07-21T01:33:08.902 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.898+0000 7f7fdff5b700 1 -- 172.21.15.143:0/2390659634 shutdown_connections 2022-07-21T01:33:08.902 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.898+0000 7f7fdff5b700 1 --2- 172.21.15.143:0/2390659634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd80b1b10 0x7f7fd80663e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.902 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.898+0000 7f7fdff5b700 1 --2- 172.21.15.143:0/2390659634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd8061860 0x7f7fd8065ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.903 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.898+0000 7f7fdff5b700 1 --2- 172.21.15.143:0/2390659634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd8060d00 0x7f7fd813cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:08.903 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.898+0000 7f7fdff5b700 1 -- 172.21.15.143:0/2390659634 >> 172.21.15.143:0/2390659634 conn(0x7f7fd80b2b00 msgr2=0x7f7fd80b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:08.903 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.898+0000 7f7fdff5b700 1 -- 172.21.15.143:0/2390659634 shutdown_connections 2022-07-21T01:33:08.903 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:08.898+0000 7f7fdff5b700 1 -- 172.21.15.143:0/2390659634 wait complete. 2022-07-21T01:33:08.910 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 25f35213-a67d-47e3-ac21-0ec2654b7e56 5 2022-07-21T01:33:09.034 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.022+0000 7f878333c700 1 Processor -- start 2022-07-21T01:33:09.035 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.022+0000 7f878333c700 1 -- start start 2022-07-21T01:33:09.035 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f877c0b2730 0x7f877c0666b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.035 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877c0654c0 con 0x7f877c0b3190 2022-07-21T01:33:09.035 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877c0b3570 con 0x7f877c066ba0 2022-07-21T01:33:09.036 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f877c062190 con 0x7f877c0b2730 2022-07-21T01:33:09.036 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781b39700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f877c0b2730 0x7f877c0666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.036 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8782b3b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f877c066ba0 0x7f877c14dab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:55672/0 (socket says 172.21.15.143:55672) 2022-07-21T01:33:09.037 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8782b3b700 1 -- 172.21.15.143:0/2488345881 learned_addr learned my addr 172.21.15.143:0/2488345881 (peer_addr_for_me v1:172.21.15.143:0/0) 2022-07-21T01:33:09.037 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781338700 1 -- 172.21.15.143:0/2488345881 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202455459 0 0) 0x7f877c0b3570 con 0x7f877c066ba0 2022-07-21T01:33:09.038 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781338700 1 -- 172.21.15.143:0/2488345881 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8764003260 con 0x7f877c066ba0 2022-07-21T01:33:09.038 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781b39700 1 -- 172.21.15.143:0/2488345881 >> v1:172.21.15.143:6789/0 conn(0x7f877c066ba0 legacy=0x7f877c14dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.038 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781b39700 1 -- 172.21.15.143:0/2488345881 >> v1:172.21.15.3:6789/0 conn(0x7f877c0b3190 legacy=0x7f877c060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.039 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781b39700 1 -- 172.21.15.143:0/2488345881 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c05fbf0 con 0x7f877c0b2730 2022-07-21T01:33:09.039 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781b39700 1 --2- 172.21.15.143:0/2488345881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 0x7f877c0666b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f876c002f70 tx=0x7f876c006df0).ready entity=mon.1 client_cookie=f5c606256066c594 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:09.039 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781338700 1 -- 172.21.15.143:0/2488345881 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f876c00d610 con 0x7f877c0b2730 2022-07-21T01:33:09.040 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781338700 1 -- 172.21.15.143:0/2488345881 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876c000e90 con 0x7f877c0b2730 2022-07-21T01:33:09.040 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f8781338700 1 -- 172.21.15.143:0/2488345881 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f876c00d960 con 0x7f877c0b2730 2022-07-21T01:33:09.040 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 -- 172.21.15.143:0/2488345881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 msgr2=0x7f877c0666b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.041 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 --2- 172.21.15.143:0/2488345881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 0x7f877c0666b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f876c002f70 tx=0x7f876c006df0).stop 2022-07-21T01:33:09.041 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 -- 172.21.15.143:0/2488345881 shutdown_connections 2022-07-21T01:33:09.042 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 --2- 172.21.15.143:0/2488345881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 0x7f877c0666b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.042 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 -- 172.21.15.143:0/2488345881 >> 172.21.15.143:0/2488345881 conn(0x7f877c063870 msgr2=0x7f877c063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:09.042 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 -- 172.21.15.143:0/2488345881 shutdown_connections 2022-07-21T01:33:09.042 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 -- 172.21.15.143:0/2488345881 wait complete. 2022-07-21T01:33:09.043 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 Processor -- start 2022-07-21T01:33:09.043 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.026+0000 7f878333c700 1 -- start start 2022-07-21T01:33:09.043 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878333c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c066ba0 0x7f877c0b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.044 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878333c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 0x7f877c152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.044 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878333c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f877c0b3190 0x7f877c152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.044 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878333c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c152650 con 0x7f877c066ba0 2022-07-21T01:33:09.045 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878333c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c0ac020 con 0x7f877c0b2730 2022-07-21T01:33:09.045 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878333c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877c153490 con 0x7f877c0b3190 2022-07-21T01:33:09.045 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8781b39700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 0x7f877c152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.045 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8782b3b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f877c0b3190 0x7f877c152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.046 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8781b39700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 0x7f877c152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49644/0 (socket says 172.21.15.143:49644) 2022-07-21T01:33:09.046 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8781b39700 1 -- 172.21.15.143:0/899039751 learned_addr learned my addr 172.21.15.143:0/899039751 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:09.047 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8782b3b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f877c0b3190 0x7f877c152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51626/0 (socket says 172.21.15.143:51626) 2022-07-21T01:33:09.047 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878233a700 1 --2- 172.21.15.143:0/899039751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c066ba0 0x7f877c0b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.047 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8782b3b700 1 -- 172.21.15.143:0/899039751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 msgr2=0x7f877c152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.048 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8782b3b700 1 --2- 172.21.15.143:0/899039751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 0x7f877c152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.048 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8782b3b700 1 -- 172.21.15.143:0/899039751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c066ba0 msgr2=0x7f877c0b66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.048 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8782b3b700 1 --2- 172.21.15.143:0/899039751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c066ba0 0x7f877c0b66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.049 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8782b3b700 1 -- 172.21.15.143:0/899039751 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c00b180 con 0x7f877c0b3190 2022-07-21T01:33:09.049 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8781b39700 1 --2- 172.21.15.143:0/899039751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 0x7f877c152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:09.049 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f8782b3b700 1 --2- 172.21.15.143:0/899039751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f877c0b3190 0x7f877c152fa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f877c0ac1b0 tx=0x7f8774007d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f876b7fe700 1 -- 172.21.15.143:0/899039751 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8774005ba0 con 0x7f877c0b3190 2022-07-21T01:33:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f876b7fe700 1 -- 172.21.15.143:0/899039751 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8774005d00 con 0x7f877c0b3190 2022-07-21T01:33:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878333c700 1 -- 172.21.15.143:0/899039751 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c0abae0 con 0x7f877c0b3190 2022-07-21T01:33:09.051 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f876b7fe700 1 -- 172.21.15.143:0/899039751 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87740072c0 con 0x7f877c0b3190 2022-07-21T01:33:09.051 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878333c700 1 -- 172.21.15.143:0/899039751 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c0aa350 con 0x7f877c0b3190 2022-07-21T01:33:09.051 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f876b7fe700 1 -- 172.21.15.143:0/899039751 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f8774010070 con 0x7f877c0b3190 2022-07-21T01:33:09.052 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f876b7fe700 1 -- 172.21.15.143:0/899039751 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f8774013a00 con 0x7f877c0b3190 2022-07-21T01:33:09.052 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.030+0000 7f878333c700 1 -- 172.21.15.143:0/899039751 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c1527f0 con 0x7f877c0b3190 2022-07-21T01:33:09.052 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.034+0000 7f876b7fe700 1 -- 172.21.15.143:0/899039751 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f8774013bc0 con 0x7f877c0b3190 2022-07-21T01:33:09.103 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.102+0000 7f878333c700 1 -- 172.21.15.143:0/899039751 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", "id": 5} v 0) v1 -- 0x7f877c05fe90 con 0x7f877c0b3190 2022-07-21T01:33:09.112 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.110+0000 7f876b7fe700 1 -- 172.21.15.143:0/899039751 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8774013d60 con 0x7f877c0b3190 2022-07-21T01:33:09.113 INFO:teuthology.orchestra.run.smithi143.stdout:5 2022-07-21T01:33:09.116 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.114+0000 7f878333c700 1 -- 172.21.15.143:0/899039751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f877c0b3190 msgr2=0x7f877c152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.117 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.114+0000 7f878333c700 1 --2- 172.21.15.143:0/899039751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f877c0b3190 0x7f877c152fa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f877c0ac1b0 tx=0x7f8774007d40).stop 2022-07-21T01:33:09.117 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.114+0000 7f878333c700 1 -- 172.21.15.143:0/899039751 shutdown_connections 2022-07-21T01:33:09.118 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.114+0000 7f878333c700 1 --2- 172.21.15.143:0/899039751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f877c0b3190 0x7f877c152fa0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.119 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.114+0000 7f878333c700 1 --2- 172.21.15.143:0/899039751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877c0b2730 0x7f877c152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.120 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.114+0000 7f878333c700 1 --2- 172.21.15.143:0/899039751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c066ba0 0x7f877c0b66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.120 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.114+0000 7f878333c700 1 -- 172.21.15.143:0/899039751 >> 172.21.15.143:0/899039751 conn(0x7f877c063870 msgr2=0x7f877c1437a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:09.120 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.114+0000 7f878333c700 1 -- 172.21.15.143:0/899039751 shutdown_connections 2022-07-21T01:33:09.121 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.114+0000 7f878333c700 1 -- 172.21.15.143:0/899039751 wait complete. 2022-07-21T01:33:09.126 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2 6 2022-07-21T01:33:09.235 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.230+0000 7fd560dee700 1 Processor -- start 2022-07-21T01:33:09.236 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.230+0000 7fd560dee700 1 -- start start 2022-07-21T01:33:09.237 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.230+0000 7fd560dee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c066b60 0x7fd55c13e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.237 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.230+0000 7fd560dee700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55c05f8e0 con 0x7fd55c066780 2022-07-21T01:33:09.238 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.230+0000 7fd560dee700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55c0b1630 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.239 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.230+0000 7fd560dee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c05fb10 con 0x7fd55c066b60 2022-07-21T01:33:09.240 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.230+0000 7fd55bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c066b60 0x7fd55c13e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.241 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.230+0000 7fd55affd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd55c0b1bc0 0x7fd55c140e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:55682/0 (socket says 172.21.15.143:55682) 2022-07-21T01:33:09.241 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.230+0000 7fd55affd700 1 -- 172.21.15.143:0/1489983803 learned_addr learned my addr 172.21.15.143:0/1489983803 (peer_addr_for_me v1:172.21.15.143:0/0) 2022-07-21T01:33:09.242 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd55bfff700 1 -- 172.21.15.143:0/1489983803 >> v1:172.21.15.143:6789/0 conn(0x7fd55c0b1bc0 legacy=0x7fd55c140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.242 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd55bfff700 1 -- 172.21.15.143:0/1489983803 >> v1:172.21.15.3:6789/0 conn(0x7fd55c066780 legacy=0x7fd55c05ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.243 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd55bfff700 1 -- 172.21.15.143:0/1489983803 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c0b18d0 con 0x7fd55c066b60 2022-07-21T01:33:09.243 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd55bfff700 1 --2- 172.21.15.143:0/1489983803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c066b60 0x7fd55c13e2b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fd5540032f0 tx=0x7fd5540091d0).ready entity=mon.2 client_cookie=cdb8c67d04febaa4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:09.244 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd55a7fc700 1 -- 172.21.15.143:0/1489983803 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd55400b8a0 con 0x7fd55c066b60 2022-07-21T01:33:09.245 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd55a7fc700 1 -- 172.21.15.143:0/1489983803 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55400ba00 con 0x7fd55c066b60 2022-07-21T01:33:09.245 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd55a7fc700 1 -- 172.21.15.143:0/1489983803 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd55400bb70 con 0x7fd55c066b60 2022-07-21T01:33:09.246 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd560dee700 1 -- 172.21.15.143:0/1489983803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c066b60 msgr2=0x7fd55c13e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.247 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd560dee700 1 --2- 172.21.15.143:0/1489983803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c066b60 0x7fd55c13e2b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fd5540032f0 tx=0x7fd5540091d0).stop 2022-07-21T01:33:09.248 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd560dee700 1 -- 172.21.15.143:0/1489983803 shutdown_connections 2022-07-21T01:33:09.249 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd560dee700 1 --2- 172.21.15.143:0/1489983803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c066b60 0x7fd55c13e2b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.249 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd560dee700 1 -- 172.21.15.143:0/1489983803 >> 172.21.15.143:0/1489983803 conn(0x7fd55c063870 msgr2=0x7fd55c063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:09.250 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd560dee700 1 -- 172.21.15.143:0/1489983803 shutdown_connections 2022-07-21T01:33:09.251 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd560dee700 1 -- 172.21.15.143:0/1489983803 wait complete. 2022-07-21T01:33:09.252 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd560dee700 1 Processor -- start 2022-07-21T01:33:09.253 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.234+0000 7fd560dee700 1 -- start start 2022-07-21T01:33:09.261 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd560dee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd55c066780 0x7fd55c0ad710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.262 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd560dee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c066b60 0x7fd55c14d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.263 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd560dee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c0b1bc0 0x7fd55c14da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.263 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd560dee700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c150eb0 con 0x7fd55c066780 2022-07-21T01:33:09.264 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd560dee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c151150 con 0x7fd55c066b60 2022-07-21T01:33:09.264 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd560dee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c1422a0 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.265 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c066b60 0x7fd55c14d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.266 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c0b1bc0 0x7fd55c14da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.266 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c066b60 0x7fd55c14d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49656/0 (socket says 172.21.15.143:49656) 2022-07-21T01:33:09.267 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55affd700 1 -- 172.21.15.143:0/423303567 learned_addr learned my addr 172.21.15.143:0/423303567 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:09.268 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55b7fe700 1 --2- 172.21.15.143:0/423303567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd55c066780 0x7fd55c0ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.268 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55bfff700 1 -- 172.21.15.143:0/423303567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c066b60 msgr2=0x7fd55c14d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.269 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55bfff700 1 --2- 172.21.15.143:0/423303567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c066b60 0x7fd55c14d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.269 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55bfff700 1 -- 172.21.15.143:0/423303567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd55c066780 msgr2=0x7fd55c0ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.270 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55bfff700 1 --2- 172.21.15.143:0/423303567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd55c066780 0x7fd55c0ad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.270 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55bfff700 1 -- 172.21.15.143:0/423303567 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd554009470 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.271 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55affd700 1 --2- 172.21.15.143:0/423303567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c066b60 0x7fd55c14d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:09.272 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd55bfff700 1 --2- 172.21.15.143:0/423303567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c0b1bc0 0x7fd55c14da00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd5540032f0 tx=0x7fd554016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:09.272 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd558ff9700 1 -- 172.21.15.143:0/423303567 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd55400bde0 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.273 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd558ff9700 1 -- 172.21.15.143:0/423303567 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55401f3f0 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.273 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd560dee700 1 -- 172.21.15.143:0/423303567 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55c144b50 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.274 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd558ff9700 1 -- 172.21.15.143:0/423303567 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd55401f750 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.275 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd560dee700 1 -- 172.21.15.143:0/423303567 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55c0adc00 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.275 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd558ff9700 1 -- 172.21.15.143:0/423303567 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fd554026030 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.275 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd558ff9700 1 -- 172.21.15.143:0/423303567 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7fd5540289d0 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.276 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.238+0000 7fd560dee700 1 -- 172.21.15.143:0/423303567 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55c05f440 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.276 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.242+0000 7fd558ff9700 1 -- 172.21.15.143:0/423303567 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fd554024070 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.334 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.334+0000 7fd560dee700 1 -- 172.21.15.143:0/423303567 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", "id": 6} v 0) v1 -- 0x7fd55c0b12b0 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.343 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.342+0000 7fd558ff9700 1 -- 172.21.15.143:0/423303567 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd55401b040 con 0x7fd55c0b1bc0 2022-07-21T01:33:09.344 INFO:teuthology.orchestra.run.smithi143.stdout:6 2022-07-21T01:33:09.348 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.342+0000 7fd560dee700 1 -- 172.21.15.143:0/423303567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c0b1bc0 msgr2=0x7fd55c14da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.349 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.342+0000 7fd560dee700 1 --2- 172.21.15.143:0/423303567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c0b1bc0 0x7fd55c14da00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd5540032f0 tx=0x7fd554016000).stop 2022-07-21T01:33:09.349 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.342+0000 7fd560dee700 1 -- 172.21.15.143:0/423303567 shutdown_connections 2022-07-21T01:33:09.349 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.342+0000 7fd560dee700 1 --2- 172.21.15.143:0/423303567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c0b1bc0 0x7fd55c14da00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.349 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.342+0000 7fd560dee700 1 --2- 172.21.15.143:0/423303567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c066b60 0x7fd55c14d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.350 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.342+0000 7fd560dee700 1 --2- 172.21.15.143:0/423303567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd55c066780 0x7fd55c0ad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.350 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.342+0000 7fd560dee700 1 -- 172.21.15.143:0/423303567 >> 172.21.15.143:0/423303567 conn(0x7fd55c063870 msgr2=0x7fd55c13f790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:09.350 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.346+0000 7fd560dee700 1 -- 172.21.15.143:0/423303567 shutdown_connections 2022-07-21T01:33:09.351 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.346+0000 7fd560dee700 1 -- 172.21.15.143:0/423303567 wait complete. 2022-07-21T01:33:09.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new f0260c49-f670-47ef-b037-b5d9ace42688 7 2022-07-21T01:33:09.463 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3461999700 1 Processor -- start 2022-07-21T01:33:09.463 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3461999700 1 -- start start 2022-07-21T01:33:09.471 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3461999700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0a9320 0x7f345c1466e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.471 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3461999700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f345c0b0d10 0x7f345c146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.472 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3461999700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c0b12d0 con 0x7f345c0b0560 2022-07-21T01:33:09.472 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3461999700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345c0a9700 con 0x7f345c0b0d10 2022-07-21T01:33:09.472 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3461999700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0b1500 con 0x7f345c0a9320 2022-07-21T01:33:09.473 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3460997700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0a9320 0x7f345c1466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.473 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f345bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f345c0b0d10 0x7f345c146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.474 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3460997700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0a9320 0x7f345c1466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:51640/0 (socket says 172.21.15.143:51640) 2022-07-21T01:33:09.474 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3460997700 1 -- 172.21.15.143:0/1852518661 learned_addr learned my addr 172.21.15.143:0/1852518661 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:09.474 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f345bfff700 1 -- 172.21.15.143:0/1852518661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0a9320 msgr2=0x7f345c1466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:09.475 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f345bfff700 1 --2- 172.21.15.143:0/1852518661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0a9320 0x7f345c1466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.475 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f345bfff700 1 -- 172.21.15.143:0/1852518661 >> v1:172.21.15.3:6789/0 conn(0x7f345c0b0560 legacy=0x7f345c140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:09.475 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f345bfff700 1 -- 172.21.15.143:0/1852518661 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c0b17a0 con 0x7f345c0b0d10 2022-07-21T01:33:09.476 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.462+0000 7f3460997700 1 --2- 172.21.15.143:0/1852518661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0a9320 0x7f345c1466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:09.476 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f345bfff700 1 --2- 172.21.15.143:0/1852518661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0b0d10 0x7f345c146bd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f345c0b0420 tx=0x7f3448006df0).ready entity=mon.1 client_cookie=6f66791ca39a5240 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:09.476 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f345b7fe700 1 -- 172.21.15.143:0/1852518661 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f344800d570 con 0x7f345c0b0d10 2022-07-21T01:33:09.477 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f345b7fe700 1 -- 172.21.15.143:0/1852518661 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3448000e90 con 0x7f345c0b0d10 2022-07-21T01:33:09.477 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f345b7fe700 1 -- 172.21.15.143:0/1852518661 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f344800d8c0 con 0x7f345c0b0d10 2022-07-21T01:33:09.477 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 -- 172.21.15.143:0/1852518661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0b0d10 msgr2=0x7f345c146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.478 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 --2- 172.21.15.143:0/1852518661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0b0d10 0x7f345c146bd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f345c0b0420 tx=0x7f3448006df0).stop 2022-07-21T01:33:09.478 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 -- 172.21.15.143:0/1852518661 shutdown_connections 2022-07-21T01:33:09.478 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 --2- 172.21.15.143:0/1852518661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0b0d10 0x7f345c146bd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.479 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 --2- 172.21.15.143:0/1852518661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0a9320 0x7f345c1466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.479 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 -- 172.21.15.143:0/1852518661 >> 172.21.15.143:0/1852518661 conn(0x7f345c0612d0 msgr2=0x7f345c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:09.479 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 -- 172.21.15.143:0/1852518661 shutdown_connections 2022-07-21T01:33:09.479 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 -- 172.21.15.143:0/1852518661 wait complete. 2022-07-21T01:33:09.480 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 Processor -- start 2022-07-21T01:33:09.480 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 -- start start 2022-07-21T01:33:09.480 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0a9320 0x7f345c1564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.481 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0b0560 0x7f345c13d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.481 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f345c0b0d10 0x7f345c152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:09.481 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0b3f80 con 0x7f345c0b0d10 2022-07-21T01:33:09.482 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0ac960 con 0x7f345c0a9320 2022-07-21T01:33:09.482 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3461999700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c155ae0 con 0x7f345c0b0560 2022-07-21T01:33:09.482 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3460997700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0a9320 0x7f345c1564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.483 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3460997700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0a9320 0x7f345c1564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49666/0 (socket says 172.21.15.143:49666) 2022-07-21T01:33:09.483 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.466+0000 7f3460997700 1 -- 172.21.15.143:0/2445429886 learned_addr learned my addr 172.21.15.143:0/2445429886 (peer_addr_for_me v2:172.21.15.143:0/0) 2022-07-21T01:33:09.483 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f345bfff700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0b0560 0x7f345c13d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.484 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3461198700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f345c0b0d10 0x7f345c152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:09.484 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3460997700 1 -- 172.21.15.143:0/2445429886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0b0560 msgr2=0x7f345c13d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.484 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3460997700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0b0560 0x7f345c13d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.484 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3460997700 1 -- 172.21.15.143:0/2445429886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f345c0b0d10 msgr2=0x7f345c152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.485 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3460997700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f345c0b0d10 0x7f345c152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.485 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3460997700 1 -- 172.21.15.143:0/2445429886 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345000b1e0 con 0x7f345c0a9320 2022-07-21T01:33:09.486 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3461198700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f345c0b0d10 0x7f345c152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:09.486 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3460997700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0a9320 0x7f345c1564c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f345000c000 tx=0x7f3450006eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:09.486 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3459ffb700 1 -- 172.21.15.143:0/2445429886 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f345000b4c0 con 0x7f345c0a9320 2022-07-21T01:33:09.487 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3459ffb700 1 -- 172.21.15.143:0/2445429886 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3450012600 con 0x7f345c0a9320 2022-07-21T01:33:09.487 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3461999700 1 -- 172.21.15.143:0/2445429886 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344800b180 con 0x7f345c0a9320 2022-07-21T01:33:09.487 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3459ffb700 1 -- 172.21.15.143:0/2445429886 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3450012960 con 0x7f345c0a9320 2022-07-21T01:33:09.487 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3461999700 1 -- 172.21.15.143:0/2445429886 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345c13d700 con 0x7f345c0a9320 2022-07-21T01:33:09.488 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3459ffb700 1 -- 172.21.15.143:0/2445429886 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f3450022070 con 0x7f345c0a9320 2022-07-21T01:33:09.488 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3459ffb700 1 -- 172.21.15.143:0/2445429886 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7f345001bae0 con 0x7f345c0a9320 2022-07-21T01:33:09.488 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.470+0000 7f3461999700 1 -- 172.21.15.143:0/2445429886 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345c0574e0 con 0x7f345c0a9320 2022-07-21T01:33:09.489 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.474+0000 7f3459ffb700 1 -- 172.21.15.143:0/2445429886 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f3450039870 con 0x7f345c0a9320 2022-07-21T01:33:09.536 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.534+0000 7f3459ffb700 1 -- 172.21.15.143:0/2445429886 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 2) v1 ==== 32894+0+0 (secure 0 0 0) 0x7f345001b450 con 0x7f345c0a9320 2022-07-21T01:33:09.538 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.538+0000 7f3461999700 1 -- 172.21.15.143:0/2445429886 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", "id": 7} v 0) v1 -- 0x7f345c1475c0 con 0x7f345c0a9320 2022-07-21T01:33:09.547 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.546+0000 7f3459ffb700 1 -- 172.21.15.143:0/2445429886 <== mon.1 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3450029020 con 0x7f345c0a9320 2022-07-21T01:33:09.548 INFO:teuthology.orchestra.run.smithi143.stdout:7 2022-07-21T01:33:09.549 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.550+0000 7f3461999700 1 -- 172.21.15.143:0/2445429886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0a9320 msgr2=0x7f345c1564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:09.549 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.550+0000 7f3461999700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0a9320 0x7f345c1564c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f345000c000 tx=0x7f3450006eb0).stop 2022-07-21T01:33:09.550 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.550+0000 7f3461999700 1 -- 172.21.15.143:0/2445429886 shutdown_connections 2022-07-21T01:33:09.550 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.550+0000 7f3461999700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f345c0b0d10 0x7f345c152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.550 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.550+0000 7f3461999700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f345c0b0560 0x7f345c13d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.550 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.550+0000 7f3461999700 1 --2- 172.21.15.143:0/2445429886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f345c0a9320 0x7f345c1564c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:09.551 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.550+0000 7f3461999700 1 -- 172.21.15.143:0/2445429886 >> 172.21.15.143:0/2445429886 conn(0x7f345c0612d0 msgr2=0x7f345c0b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:09.551 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.550+0000 7f3461999700 1 -- 172.21.15.143:0/2445429886 shutdown_connections 2022-07-21T01:33:09.551 INFO:teuthology.orchestra.run.smithi143.stderr:2022-07-21T01:33:09.550+0000 7f3461999700 1 -- 172.21.15.143:0/2445429886 wait complete. 2022-07-21T01:33:09.560 INFO:tasks.ceph.osd.0:Restarting daemon 2022-07-21T01:33:09.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-07-21T01:33:09.562 INFO:tasks.ceph.osd.0:Started 2022-07-21T01:33:09.563 INFO:tasks.ceph.osd.1:Restarting daemon 2022-07-21T01:33:09.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-07-21T01:33:09.565 INFO:tasks.ceph.osd.1:Started 2022-07-21T01:33:09.566 INFO:tasks.ceph.osd.2:Restarting daemon 2022-07-21T01:33:09.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-07-21T01:33:09.568 INFO:tasks.ceph.osd.2:Started 2022-07-21T01:33:09.569 INFO:tasks.ceph.osd.3:Restarting daemon 2022-07-21T01:33:09.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-07-21T01:33:09.572 INFO:tasks.ceph.osd.3:Started 2022-07-21T01:33:09.573 INFO:tasks.ceph.osd.4:Restarting daemon 2022-07-21T01:33:09.573 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2022-07-21T01:33:09.575 INFO:tasks.ceph.osd.4:Started 2022-07-21T01:33:09.576 INFO:tasks.ceph.osd.5:Restarting daemon 2022-07-21T01:33:09.576 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2022-07-21T01:33:09.578 INFO:tasks.ceph.osd.5:Started 2022-07-21T01:33:09.579 INFO:tasks.ceph.osd.6:Restarting daemon 2022-07-21T01:33:09.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2022-07-21T01:33:09.581 INFO:tasks.ceph.osd.6:Started 2022-07-21T01:33:09.581 INFO:tasks.ceph.osd.7:Restarting daemon 2022-07-21T01:33:09.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2022-07-21T01:33:09.584 INFO:tasks.ceph.osd.7:Started 2022-07-21T01:33:09.584 INFO:tasks.ceph:Waiting for OSDs to come up 2022-07-21T01:33:09.970 INFO:tasks.ceph.osd.5.smithi143.stderr:2022-07-21T01:33:09.970+0000 7fdb61472d80 -1 Falling back to public interface 2022-07-21T01:33:10.179 INFO:tasks.ceph.osd.2.smithi003.stderr:2022-07-21T01:33:10.182+0000 7f8d23512d80 -1 Falling back to public interface 2022-07-21T01:33:10.187 INFO:tasks.ceph.osd.1.smithi003.stderr:2022-07-21T01:33:10.190+0000 7fe5068d4d80 -1 Falling back to public interface 2022-07-21T01:33:10.188 INFO:tasks.ceph.osd.3.smithi003.stderr:2022-07-21T01:33:10.190+0000 7f3b6be9ed80 -1 Falling back to public interface 2022-07-21T01:33:10.189 INFO:tasks.ceph.osd.0.smithi003.stderr:2022-07-21T01:33:10.190+0000 7fe6081edd80 -1 Falling back to public interface 2022-07-21T01:33:10.190 INFO:tasks.ceph.osd.4.smithi143.stderr:2022-07-21T01:33:10.190+0000 7fdcd23cdd80 -1 Falling back to public interface 2022-07-21T01:33:10.202 INFO:tasks.ceph.osd.7.smithi143.stderr:2022-07-21T01:33:10.202+0000 7f5ed3c50d80 -1 Falling back to public interface 2022-07-21T01:33:10.214 INFO:tasks.ceph.osd.6.smithi143.stderr:2022-07-21T01:33:10.214+0000 7fa042dacd80 -1 Falling back to public interface 2022-07-21T01:33:10.391 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:10.488 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb377c700 1 Processor -- start 2022-07-21T01:33:10.488 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb377c700 1 -- start start 2022-07-21T01:33:10.490 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb377c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcfac0b6c40 0x7fcfac0b7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:10.490 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb377c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcfac147390 0x7fcfac061740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:10.490 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb377c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac146710 con 0x7fcfac061c30 2022-07-21T01:33:10.491 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb377c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfac1469b0 con 0x7fcfac147390 2022-07-21T01:33:10.491 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb377c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfac146c50 con 0x7fcfac0b6c40 2022-07-21T01:33:10.491 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb0d17700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcfac147390 0x7fcfac061740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:10.491 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb0d17700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcfac147390 0x7fcfac061740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42096/0 (socket says 172.21.15.3:42096) 2022-07-21T01:33:10.492 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.490+0000 7fcfb0d17700 1 -- 172.21.15.3:0/3662858354 learned_addr learned my addr 172.21.15.3:0/3662858354 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:10.492 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb1518700 1 --2- 172.21.15.3:0/3662858354 >> v2:172.21.15.143:3300/0 conn(0x7fcfac0b6c40 0x7fcfac0b7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:10.492 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb0d17700 1 -- 172.21.15.3:0/3662858354 >> v2:172.21.15.143:3300/0 conn(0x7fcfac0b6c40 msgr2=0x7fcfac0b7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:10.492 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb0d17700 1 --2- 172.21.15.3:0/3662858354 >> v2:172.21.15.143:3300/0 conn(0x7fcfac0b6c40 0x7fcfac0b7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:10.493 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb0d17700 1 -- 172.21.15.3:0/3662858354 >> v1:172.21.15.143:6789/0 conn(0x7fcfac061c30 legacy=0x7fcfac13cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:10.493 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb0d17700 1 -- 172.21.15.3:0/3662858354 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac146ef0 con 0x7fcfac147390 2022-07-21T01:33:10.493 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb0d17700 1 --2- 172.21.15.3:0/3662858354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 0x7fcfac061740 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fcfac0b21a0 tx=0x7fcf9c006eb0).ready entity=mon.0 client_cookie=864d74bcf522caf0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:10.497 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcf9bfff700 1 -- 172.21.15.3:0/3662858354 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf9c00d610 con 0x7fcfac147390 2022-07-21T01:33:10.498 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcf9bfff700 1 -- 172.21.15.3:0/3662858354 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c000e90 con 0x7fcfac147390 2022-07-21T01:33:10.498 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcf9bfff700 1 -- 172.21.15.3:0/3662858354 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf9c00d9a0 con 0x7fcfac147390 2022-07-21T01:33:10.498 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb377c700 1 -- 172.21.15.3:0/3662858354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 msgr2=0x7fcfac061740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:10.498 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb377c700 1 --2- 172.21.15.3:0/3662858354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 0x7fcfac061740 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fcfac0b21a0 tx=0x7fcf9c006eb0).stop 2022-07-21T01:33:10.499 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb377c700 1 -- 172.21.15.3:0/3662858354 shutdown_connections 2022-07-21T01:33:10.499 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb377c700 1 --2- 172.21.15.3:0/3662858354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 0x7fcfac061740 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:10.499 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb377c700 1 --2- 172.21.15.3:0/3662858354 >> v2:172.21.15.143:3300/0 conn(0x7fcfac0b6c40 0x7fcfac0b7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:10.499 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb377c700 1 -- 172.21.15.3:0/3662858354 >> 172.21.15.3:0/3662858354 conn(0x7fcfac05fb50 msgr2=0x7fcfac05c650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:10.499 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb377c700 1 -- 172.21.15.3:0/3662858354 shutdown_connections 2022-07-21T01:33:10.500 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.494+0000 7fcfb377c700 1 -- 172.21.15.3:0/3662858354 wait complete. 2022-07-21T01:33:10.500 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 Processor -- start 2022-07-21T01:33:10.500 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 -- start start 2022-07-21T01:33:10.500 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac061c30 0x7fcfac0adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:10.501 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfac0b6c40 0x7fcfac152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:10.501 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 0x7fcfac153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:10.502 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac13e300 con 0x7fcfac147390 2022-07-21T01:33:10.502 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0af880 con 0x7fcfac061c30 2022-07-21T01:33:10.502 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac1411b0 con 0x7fcfac0b6c40 2022-07-21T01:33:10.502 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb1d19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 0x7fcfac153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:10.503 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb1518700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac061c30 0x7fcfac0adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:10.503 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb1518700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac061c30 0x7fcfac0adbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:47550/0 (socket says 172.21.15.3:47550) 2022-07-21T01:33:10.503 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb1d19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 0x7fcfac153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42106/0 (socket says 172.21.15.3:42106) 2022-07-21T01:33:10.503 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb1518700 1 -- 172.21.15.3:0/3357149838 learned_addr learned my addr 172.21.15.3:0/3357149838 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:10.504 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb0d17700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfac0b6c40 0x7fcfac152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:10.504 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb0d17700 1 -- 172.21.15.3:0/3357149838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac061c30 msgr2=0x7fcfac0adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:10.504 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb0d17700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac061c30 0x7fcfac0adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:10.504 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb0d17700 1 -- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 msgr2=0x7fcfac153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:10.504 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb0d17700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 0x7fcfac153160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:10.505 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb0d17700 1 -- 172.21.15.3:0/3357149838 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c00b1e0 con 0x7fcfac0b6c40 2022-07-21T01:33:10.505 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb1d19700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 0x7fcfac153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:10.505 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb0d17700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfac0b6c40 0x7fcfac152c70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fcfac140b20 tx=0x7fcf9c015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:10.505 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcf9a7fc700 1 -- 172.21.15.3:0/3357149838 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf9c0117e0 con 0x7fcfac0b6c40 2022-07-21T01:33:10.506 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcf9a7fc700 1 -- 172.21.15.3:0/3357149838 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c011940 con 0x7fcfac0b6c40 2022-07-21T01:33:10.506 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcf9a7fc700 1 -- 172.21.15.3:0/3357149838 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf9c011ca0 con 0x7fcfac0b6c40 2022-07-21T01:33:10.506 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac141450 con 0x7fcfac0b6c40 2022-07-21T01:33:10.506 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac0b0570 con 0x7fcfac0b6c40 2022-07-21T01:33:10.506 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.498+0000 7fcf9a7fc700 1 -- 172.21.15.3:0/3357149838 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 2) v1 ==== 32894+0+0 (secure 0 0 0) 0x7fcf9c01a040 con 0x7fcfac0b6c40 2022-07-21T01:33:10.507 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.502+0000 7fcf9a7fc700 1 -- 172.21.15.3:0/3357149838 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7fcf9c03b730 con 0x7fcfac0b6c40 2022-07-21T01:33:10.507 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.502+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfac0afa80 con 0x7fcfac0b6c40 2022-07-21T01:33:10.507 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.506+0000 7fcf9a7fc700 1 -- 172.21.15.3:0/3357149838 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fcf9c025350 con 0x7fcfac0b6c40 2022-07-21T01:33:10.554 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.558+0000 7fcf9a7fc700 1 -- 172.21.15.3:0/3357149838 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7fcf9c00f070 con 0x7fcfac0b6c40 2022-07-21T01:33:10.554 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.558+0000 7fcf9a7fc700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf8c054480 0x7fcf8c056890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:10.557 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.558+0000 7fcfb1518700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf8c054480 0x7fcf8c056890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:10.558 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.562+0000 7fcfb1518700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf8c054480 0x7fcf8c056890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fcfa800b000 tx=0x7fcfa8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:10.619 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.622+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfac1415e0 con 0x7fcfac0b6c40 2022-07-21T01:33:10.621 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcf9a7fc700 1 -- 172.21.15.3:0/3357149838 <== mon.2 v2:172.21.15.143:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+8254 (secure 0 0 0) 0x7fcf9c036540 con 0x7fcfac0b6c40 2022-07-21T01:33:10.622 INFO:teuthology.misc.health.smithi003.stdout: 2022-07-21T01:33:10.622 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:10.553176+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:10.623 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf8c054480 msgr2=0x7fcf8c056890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:10.624 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf8c054480 0x7fcf8c056890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fcfa800b000 tx=0x7fcfa8006df0).stop 2022-07-21T01:33:10.624 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfac0b6c40 msgr2=0x7fcfac152c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:10.624 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfac0b6c40 0x7fcfac152c70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fcfac140b20 tx=0x7fcf9c015000).stop 2022-07-21T01:33:10.625 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 shutdown_connections 2022-07-21T01:33:10.625 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfac147390 0x7fcfac153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:10.625 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfac0b6c40 0x7fcfac152c70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:10.626 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf8c054480 0x7fcf8c056890 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:10.626 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 --2- 172.21.15.3:0/3357149838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac061c30 0x7fcfac0adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:10.626 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 >> 172.21.15.3:0/3357149838 conn(0x7fcfac05fb50 msgr2=0x7fcfac0b98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:10.626 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 shutdown_connections 2022-07-21T01:33:10.627 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:10.626+0000 7fcfb377c700 1 -- 172.21.15.3:0/3357149838 wait complete. 2022-07-21T01:33:10.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-07-21T01:33:11.152 INFO:tasks.ceph.osd.5.smithi143.stderr:2022-07-21T01:33:11.150+0000 7fdb61472d80 -1 osd.5 0 log_to_monitors {default=true} 2022-07-21T01:33:11.381 INFO:tasks.ceph.osd.1.smithi003.stderr:2022-07-21T01:33:11.382+0000 7fe5068d4d80 -1 osd.1 0 log_to_monitors {default=true} 2022-07-21T01:33:11.386 INFO:tasks.ceph.osd.4.smithi143.stderr:2022-07-21T01:33:11.386+0000 7fdcd23cdd80 -1 osd.4 0 log_to_monitors {default=true} 2022-07-21T01:33:11.390 INFO:tasks.ceph.osd.7.smithi143.stderr:2022-07-21T01:33:11.390+0000 7f5ed3c50d80 -1 osd.7 0 log_to_monitors {default=true} 2022-07-21T01:33:11.397 INFO:tasks.ceph.osd.6.smithi143.stderr:2022-07-21T01:33:11.398+0000 7fa042dacd80 -1 osd.6 0 log_to_monitors {default=true} 2022-07-21T01:33:11.604 INFO:tasks.ceph.osd.2.smithi003.stderr:2022-07-21T01:33:11.606+0000 7f8d23512d80 -1 osd.2 0 log_to_monitors {default=true} 2022-07-21T01:33:11.627 INFO:tasks.ceph.osd.0.smithi003.stderr:2022-07-21T01:33:11.630+0000 7fe6081edd80 -1 osd.0 0 log_to_monitors {default=true} 2022-07-21T01:33:11.638 INFO:tasks.ceph.osd.3.smithi003.stderr:2022-07-21T01:33:11.642+0000 7f3b6be9ed80 -1 osd.3 0 log_to_monitors {default=true} 2022-07-21T01:33:12.579 INFO:tasks.ceph.osd.5.smithi143.stderr:2022-07-21T01:33:12.578+0000 7fdb5351b700 -1 osd.5 0 waiting for initial osdmap 2022-07-21T01:33:12.580 INFO:tasks.ceph.osd.7.smithi143.stderr:2022-07-21T01:33:12.582+0000 7f5ec5cf9700 -1 osd.7 0 waiting for initial osdmap 2022-07-21T01:33:12.581 INFO:tasks.ceph.osd.1.smithi003.stderr:2022-07-21T01:33:12.586+0000 7fe4f897d700 -1 osd.1 0 waiting for initial osdmap 2022-07-21T01:33:12.583 INFO:tasks.ceph.osd.4.smithi143.stderr:2022-07-21T01:33:12.582+0000 7fdcc4476700 -1 osd.4 0 waiting for initial osdmap 2022-07-21T01:33:12.584 INFO:tasks.ceph.osd.6.smithi143.stderr:2022-07-21T01:33:12.582+0000 7fa034e55700 -1 osd.6 0 waiting for initial osdmap 2022-07-21T01:33:12.587 INFO:tasks.ceph.osd.7.smithi143.stderr:2022-07-21T01:33:12.586+0000 7f5ecd174700 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-21T01:33:12.589 INFO:tasks.ceph.osd.5.smithi143.stderr:2022-07-21T01:33:12.590+0000 7fdb5a996700 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-21T01:33:12.592 INFO:tasks.ceph.osd.4.smithi143.stderr:2022-07-21T01:33:12.594+0000 7fdccb8f1700 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-21T01:33:12.593 INFO:tasks.ceph.osd.6.smithi143.stderr:2022-07-21T01:33:12.594+0000 7fa03c2d0700 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-21T01:33:12.593 INFO:tasks.ceph.osd.1.smithi003.stderr:2022-07-21T01:33:12.598+0000 7fe4ffdf8700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-21T01:33:13.584 INFO:tasks.ceph.osd.0.smithi003.stderr:2022-07-21T01:33:13.586+0000 7fe5fa296700 -1 osd.0 0 waiting for initial osdmap 2022-07-21T01:33:13.585 INFO:tasks.ceph.osd.3.smithi003.stderr:2022-07-21T01:33:13.586+0000 7f3b5df47700 -1 osd.3 0 waiting for initial osdmap 2022-07-21T01:33:13.585 INFO:tasks.ceph.osd.2.smithi003.stderr:2022-07-21T01:33:13.586+0000 7f8d155bb700 -1 osd.2 0 waiting for initial osdmap 2022-07-21T01:33:13.592 INFO:tasks.ceph.osd.0.smithi003.stderr:2022-07-21T01:33:13.594+0000 7fe601711700 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-21T01:33:13.593 INFO:tasks.ceph.osd.3.smithi003.stderr:2022-07-21T01:33:13.598+0000 7f3b653c2700 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-21T01:33:13.595 INFO:tasks.ceph.osd.2.smithi003.stderr:2022-07-21T01:33:13.598+0000 7f8d1ca36700 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-21T01:33:17.455 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:17.563 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c3a9ee700 1 Processor -- start 2022-07-21T01:33:17.564 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c3a9ee700 1 -- start start 2022-07-21T01:33:17.565 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c3a9ee700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2c340b1e60 0x7f2c3413cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.565 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c3a9ee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c3413d110 0x7f2c340b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.566 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c3a9ee700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c340b4130 con 0x7f2c340b06e0 2022-07-21T01:33:17.566 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c3a9ee700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c34143030 con 0x7f2c340b1e60 2022-07-21T01:33:17.567 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c3a9ee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c341467c0 con 0x7f2c3413d110 2022-07-21T01:33:17.568 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c399ec700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2c340b1e60 0x7f2c3413cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.568 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c399ec700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2c340b1e60 0x7f2c3413cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42352/0 (socket says 172.21.15.3:42352) 2022-07-21T01:33:17.569 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c399ec700 1 -- 172.21.15.3:0/439623471 learned_addr learned my addr 172.21.15.3:0/439623471 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:17.569 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c3a1ed700 1 --1- 172.21.15.3:0/439623471 >> v1:172.21.15.143:6789/0 conn(0x7f2c340b06e0 0x7f2c34143490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:52750/0 (socket says 172.21.15.3:52750) 2022-07-21T01:33:17.569 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c391eb700 1 --2- 172.21.15.3:0/439623471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c3413d110 0x7f2c340b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.570 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c399ec700 1 -- 172.21.15.3:0/439623471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c3413d110 msgr2=0x7f2c340b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:17.570 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c399ec700 1 --2- 172.21.15.3:0/439623471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c3413d110 0x7f2c340b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.571 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c399ec700 1 -- 172.21.15.3:0/439623471 >> v1:172.21.15.143:6789/0 conn(0x7f2c340b06e0 legacy=0x7f2c34143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.572 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c399ec700 1 -- 172.21.15.3:0/439623471 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c340b4360 con 0x7f2c340b1e60 2022-07-21T01:33:17.572 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c399ec700 1 --2- 172.21.15.3:0/439623471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b1e60 0x7f2c3413cc20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f2c24002f70 tx=0x7f2c24006eb0).ready entity=mon.0 client_cookie=ce375dbb70fb6eb7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:17.572 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c389ea700 1 -- 172.21.15.3:0/439623471 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c2400d570 con 0x7f2c340b1e60 2022-07-21T01:33:17.573 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c389ea700 1 -- 172.21.15.3:0/439623471 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c24000e90 con 0x7f2c340b1e60 2022-07-21T01:33:17.573 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.558+0000 7f2c389ea700 1 -- 172.21.15.3:0/439623471 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c2400d900 con 0x7f2c340b1e60 2022-07-21T01:33:17.573 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/439623471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b1e60 msgr2=0x7f2c3413cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.574 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 --2- 172.21.15.3:0/439623471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b1e60 0x7f2c3413cc20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f2c24002f70 tx=0x7f2c24006eb0).stop 2022-07-21T01:33:17.574 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/439623471 shutdown_connections 2022-07-21T01:33:17.574 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 --2- 172.21.15.3:0/439623471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c3413d110 0x7f2c340b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.575 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 --2- 172.21.15.3:0/439623471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b1e60 0x7f2c3413cc20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.575 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/439623471 >> 172.21.15.3:0/439623471 conn(0x7f2c34066eb0 msgr2=0x7f2c340672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:17.575 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/439623471 shutdown_connections 2022-07-21T01:33:17.576 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/439623471 wait complete. 2022-07-21T01:33:17.576 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 Processor -- start 2022-07-21T01:33:17.576 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- start start 2022-07-21T01:33:17.577 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b06e0 0x7f2c340ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.577 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c340b1e60 0x7f2c340b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.577 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c340b0300 0x7f2c340b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.578 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340b5d60 con 0x7f2c340b06e0 2022-07-21T01:33:17.578 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340b5820 con 0x7f2c340b0300 2022-07-21T01:33:17.578 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340b5ac0 con 0x7f2c340b1e60 2022-07-21T01:33:17.578 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c399ec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b06e0 0x7f2c340ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.579 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c391eb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c340b1e60 0x7f2c340b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.579 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c399ec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b06e0 0x7f2c340ad7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42356/0 (socket says 172.21.15.3:42356) 2022-07-21T01:33:17.579 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c399ec700 1 -- 172.21.15.3:0/1413148298 learned_addr learned my addr 172.21.15.3:0/1413148298 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:17.579 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a1ed700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c340b0300 0x7f2c340b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.580 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c399ec700 1 -- 172.21.15.3:0/1413148298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c340b1e60 msgr2=0x7f2c340b66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.580 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c399ec700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c340b1e60 0x7f2c340b66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.580 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c399ec700 1 -- 172.21.15.3:0/1413148298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c340b0300 msgr2=0x7f2c340b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.581 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c399ec700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c340b0300 0x7f2c340b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.581 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c399ec700 1 -- 172.21.15.3:0/1413148298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2400b1e0 con 0x7f2c340b06e0 2022-07-21T01:33:17.581 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c399ec700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b06e0 0x7f2c340ad7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2c24002f70 tx=0x7f2c24015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:17.582 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c32ffd700 1 -- 172.21.15.3:0/1413148298 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c2400de30 con 0x7f2c340b06e0 2022-07-21T01:33:17.582 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c32ffd700 1 -- 172.21.15.3:0/1413148298 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c24011b80 con 0x7f2c340b06e0 2022-07-21T01:33:17.583 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c340b50a0 con 0x7f2c340b06e0 2022-07-21T01:33:17.583 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c32ffd700 1 -- 172.21.15.3:0/1413148298 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c2401e460 con 0x7f2c340b06e0 2022-07-21T01:33:17.583 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.562+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c340afed0 con 0x7f2c340b06e0 2022-07-21T01:33:17.584 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.566+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c34146520 con 0x7f2c340b06e0 2022-07-21T01:33:17.584 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.566+0000 7f2c32ffd700 1 -- 172.21.15.3:0/1413148298 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2c24025030 con 0x7f2c340b06e0 2022-07-21T01:33:17.585 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.566+0000 7f2c32ffd700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c14053a70 0x7f2c14055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.585 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.566+0000 7f2c391eb700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c14053a70 0x7f2c14055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.586 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.566+0000 7f2c32ffd700 1 -- 172.21.15.3:0/1413148298 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4900+0+0 (secure 0 0 0) 0x7f2c2406cc60 con 0x7f2c340b06e0 2022-07-21T01:33:17.586 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.566+0000 7f2c391eb700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c14053a70 0x7f2c14055f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2c2800b000 tx=0x7f2c280091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:17.586 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.574+0000 7f2c32ffd700 1 -- 172.21.15.3:0/1413148298 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2c24043f00 con 0x7f2c340b06e0 2022-07-21T01:33:17.696 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.698+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c340574e0 con 0x7f2c340b06e0 2022-07-21T01:33:17.697 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c32ffd700 1 -- 172.21.15.3:0/1413148298 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+11805 (secure 0 0 0) 0x7f2c24023020 con 0x7f2c340b06e0 2022-07-21T01:33:17.697 INFO:teuthology.misc.health.smithi003.stdout: 2022-07-21T01:33:17.698 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":17,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:16.624132+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:17.700 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c14053a70 msgr2=0x7f2c14055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.700 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c14053a70 0x7f2c14055f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2c2800b000 tx=0x7f2c280091d0).stop 2022-07-21T01:33:17.700 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b06e0 msgr2=0x7f2c340ad7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.700 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b06e0 0x7f2c340ad7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2c24002f70 tx=0x7f2c24015000).stop 2022-07-21T01:33:17.701 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 shutdown_connections 2022-07-21T01:33:17.701 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c14053a70 0x7f2c14055f30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.701 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c340b0300 0x7f2c340b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.701 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c340b1e60 0x7f2c340b66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.701 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 --2- 172.21.15.3:0/1413148298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c340b06e0 0x7f2c340ad7e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.702 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 >> 172.21.15.3:0/1413148298 conn(0x7f2c34066eb0 msgr2=0x7f2c3413b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:17.702 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 shutdown_connections 2022-07-21T01:33:17.702 INFO:teuthology.misc.health.smithi003.stderr:2022-07-21T01:33:17.702+0000 7f2c3a9ee700 1 -- 172.21.15.3:0/1413148298 wait complete. 2022-07-21T01:33:17.710 DEBUG:teuthology.misc:8 of 8 OSDs are up 2022-07-21T01:33:17.710 INFO:tasks.ceph:Creating RBD pool 2022-07-21T01:33:17.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool create rbd 8 2022-07-21T01:33:17.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.818+0000 7fae46e61700 1 Processor -- start 2022-07-21T01:33:17.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.818+0000 7fae46e61700 1 -- start start 2022-07-21T01:33:17.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae400a92c0 0x7fae40140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fae4005c620 0x7fae400af220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae400b0f80 con 0x7fae400b0a70 2022-07-21T01:33:17.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae401413e0 con 0x7fae4005c620 2022-07-21T01:33:17.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae400b3500 con 0x7fae400a92c0 2022-07-21T01:33:17.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46660700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fae4005c620 0x7fae400af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46660700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fae4005c620 0x7fae400af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42366/0 (socket says 172.21.15.3:42366) 2022-07-21T01:33:17.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae4565e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fae400b0a70 0x7fae400b2df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50644/0 (socket says 172.21.15.3:50644) 2022-07-21T01:33:17.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46660700 1 -- 172.21.15.3:0/3876996813 learned_addr learned my addr 172.21.15.3:0/3876996813 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:17.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae45e5f700 1 --2- 172.21.15.3:0/3876996813 >> v2:172.21.15.143:3300/0 conn(0x7fae400a92c0 0x7fae40140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae44e5d700 1 -- 172.21.15.3:0/3876996813 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827464032 0 0) 0x7fae400b0f80 con 0x7fae400b0a70 2022-07-21T01:33:17.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae44e5d700 1 -- 172.21.15.3:0/3876996813 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae28003260 con 0x7fae400b0a70 2022-07-21T01:33:17.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46660700 1 -- 172.21.15.3:0/3876996813 >> v2:172.21.15.143:3300/0 conn(0x7fae400a92c0 msgr2=0x7fae40140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:17.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46660700 1 --2- 172.21.15.3:0/3876996813 >> v2:172.21.15.143:3300/0 conn(0x7fae400a92c0 0x7fae40140ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46660700 1 -- 172.21.15.3:0/3876996813 >> v1:172.21.15.3:6789/0 conn(0x7fae400b0a70 legacy=0x7fae400b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46660700 1 -- 172.21.15.3:0/3876996813 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae400b26b0 con 0x7fae4005c620 2022-07-21T01:33:17.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46660700 1 --2- 172.21.15.3:0/3876996813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae4005c620 0x7fae400af220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fae380032f0 tx=0x7fae38009290).ready entity=mon.0 client_cookie=1abb2ff103a48bbc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:17.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae44e5d700 1 -- 172.21.15.3:0/3876996813 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae3800b870 con 0x7fae4005c620 2022-07-21T01:33:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae44e5d700 1 -- 172.21.15.3:0/3876996813 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3800b9d0 con 0x7fae4005c620 2022-07-21T01:33:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae44e5d700 1 -- 172.21.15.3:0/3876996813 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae3800bb40 con 0x7fae4005c620 2022-07-21T01:33:17.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 -- 172.21.15.3:0/3876996813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae4005c620 msgr2=0x7fae400af220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 --2- 172.21.15.3:0/3876996813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae4005c620 0x7fae400af220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fae380032f0 tx=0x7fae38009290).stop 2022-07-21T01:33:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 -- 172.21.15.3:0/3876996813 shutdown_connections 2022-07-21T01:33:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 --2- 172.21.15.3:0/3876996813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae4005c620 0x7fae400af220 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 --2- 172.21.15.3:0/3876996813 >> v2:172.21.15.143:3300/0 conn(0x7fae400a92c0 0x7fae40140ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 -- 172.21.15.3:0/3876996813 >> 172.21.15.3:0/3876996813 conn(0x7fae40061360 msgr2=0x7fae40062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 -- 172.21.15.3:0/3876996813 shutdown_connections 2022-07-21T01:33:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.822+0000 7fae46e61700 1 -- 172.21.15.3:0/3876996813 wait complete. 2022-07-21T01:33:17.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 Processor -- start 2022-07-21T01:33:17.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 -- start start 2022-07-21T01:33:17.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4005c620 0x7fae400b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae400a92c0 0x7fae400ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae400b0a70 0x7fae400adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae400b5f70 con 0x7fae400a92c0 2022-07-21T01:33:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae400b4340 con 0x7fae400b0a70 2022-07-21T01:33:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae400b6160 con 0x7fae4005c620 2022-07-21T01:33:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae4565e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae400a92c0 0x7fae400ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae4565e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae400a92c0 0x7fae400ad9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42372/0 (socket says 172.21.15.3:42372) 2022-07-21T01:33:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae4565e700 1 -- 172.21.15.3:0/1507323399 learned_addr learned my addr 172.21.15.3:0/1507323399 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46660700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae400b0a70 0x7fae400adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae45e5f700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4005c620 0x7fae400b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae4565e700 1 -- 172.21.15.3:0/1507323399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4005c620 msgr2=0x7fae400b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae4565e700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4005c620 0x7fae400b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae4565e700 1 -- 172.21.15.3:0/1507323399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae400b0a70 msgr2=0x7fae400adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:17.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae4565e700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae400b0a70 0x7fae400adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:17.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae4565e700 1 -- 172.21.15.3:0/1507323399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38009530 con 0x7fae400a92c0 2022-07-21T01:33:17.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae45e5f700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4005c620 0x7fae400b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:17.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae4565e700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae400a92c0 0x7fae400ad9f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fae30001110 tx=0x7fae30007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:17.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae2effd700 1 -- 172.21.15.3:0/1507323399 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae3000ce00 con 0x7fae400a92c0 2022-07-21T01:33:17.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae2effd700 1 -- 172.21.15.3:0/1507323399 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae300072e0 con 0x7fae400a92c0 2022-07-21T01:33:17.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae400b37d0 con 0x7fae400a92c0 2022-07-21T01:33:17.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae2effd700 1 -- 172.21.15.3:0/1507323399 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae30005740 con 0x7fae400a92c0 2022-07-21T01:33:17.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.826+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae400b4770 con 0x7fae400a92c0 2022-07-21T01:33:17.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.830+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae400b2950 con 0x7fae400a92c0 2022-07-21T01:33:17.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.830+0000 7fae2effd700 1 -- 172.21.15.3:0/1507323399 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fae30010070 con 0x7fae400a92c0 2022-07-21T01:33:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.830+0000 7fae2effd700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae20053a70 0x7fae20055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.830+0000 7fae45e5f700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae20053a70 0x7fae20055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:17.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.830+0000 7fae2effd700 1 -- 172.21.15.3:0/1507323399 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4900+0+0 (secure 0 0 0) 0x7fae3006b200 con 0x7fae400a92c0 2022-07-21T01:33:17.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.830+0000 7fae45e5f700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae20053a70 0x7fae20055f30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fae3c008000 tx=0x7fae3c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:17.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.834+0000 7fae2effd700 1 -- 172.21.15.3:0/1507323399 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fae300424a0 con 0x7fae400a92c0 2022-07-21T01:33:17.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:17.958+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fae400574e0 con 0x7fae400a92c0 2022-07-21T01:33:18.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.874+0000 7fae2effd700 1 -- 172.21.15.3:0/1507323399 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v18) v1 ==== 109+0+0 (secure 0 0 0) 0x7fae30045ac0 con 0x7fae400a92c0 2022-07-21T01:33:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae20053a70 msgr2=0x7fae20055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae20053a70 0x7fae20055f30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fae3c008000 tx=0x7fae3c008040).stop 2022-07-21T01:33:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae400a92c0 msgr2=0x7fae400ad9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae400a92c0 0x7fae400ad9f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fae30001110 tx=0x7fae30007a90).stop 2022-07-21T01:33:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 shutdown_connections 2022-07-21T01:33:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae20053a70 0x7fae20055f30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae400b0a70 0x7fae400adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:18.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae400a92c0 0x7fae400ad9f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:18.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 --2- 172.21.15.3:0/1507323399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4005c620 0x7fae400b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.878+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 >> 172.21.15.3:0/1507323399 conn(0x7fae40061360 msgr2=0x7fae4005fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.882+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 shutdown_connections 2022-07-21T01:33:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:18.882+0000 7fae46e61700 1 -- 172.21.15.3:0/1507323399 wait complete. 2022-07-21T01:33:18.883 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'rbd' created 2022-07-21T01:33:18.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-07-21T01:33:19.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.006+0000 7f68c4ec0700 1 Processor -- start 2022-07-21T01:33:19.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.006+0000 7f68c4ec0700 1 -- start start 2022-07-21T01:33:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68c4ec0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f68c012ba00 0x7f68c00b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68c4ec0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c0061e00 con 0x7f68c0061a20 2022-07-21T01:33:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68c4ec0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c012c090 con 0x7f68c00b7080 2022-07-21T01:33:19.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68c4ec0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c00b2e40 con 0x7f68c012ba00 2022-07-21T01:33:19.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68be59c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f68c0061a20 0x7f68c012b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50660/0 (socket says 172.21.15.3:50660) 2022-07-21T01:33:19.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68be59c700 1 -- 172.21.15.3:0/3413631463 learned_addr learned my addr 172.21.15.3:0/3413631463 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bed9d700 1 --2- 172.21.15.3:0/3413631463 >> v2:172.21.15.143:3300/0 conn(0x7f68c012ba00 0x7f68c00b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bdd9b700 1 -- 172.21.15.3:0/3413631463 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105467052 0 0) 0x7f68c0061e00 con 0x7f68c0061a20 2022-07-21T01:33:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bdd9b700 1 -- 172.21.15.3:0/3413631463 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a0003260 con 0x7f68c0061a20 2022-07-21T01:33:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bed9d700 1 -- 172.21.15.3:0/3413631463 >> v1:172.21.15.143:6789/0 conn(0x7f68c00b7080 legacy=0x7f68c0146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bed9d700 1 -- 172.21.15.3:0/3413631463 >> v1:172.21.15.3:6789/0 conn(0x7f68c0061a20 legacy=0x7f68c012b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bed9d700 1 -- 172.21.15.3:0/3413631463 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c012c330 con 0x7f68c012ba00 2022-07-21T01:33:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bed9d700 1 --2- 172.21.15.3:0/3413631463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68c012ba00 0x7f68c00b4300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f68b80032f0 tx=0x7f68b80091d0).ready entity=mon.1 client_cookie=b8825541874ff3e2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bdd9b700 1 -- 172.21.15.3:0/3413631463 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68b800b6c0 con 0x7f68c012ba00 2022-07-21T01:33:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bdd9b700 1 -- 172.21.15.3:0/3413631463 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b800b820 con 0x7f68c012ba00 2022-07-21T01:33:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68bdd9b700 1 -- 172.21.15.3:0/3413631463 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68b800b990 con 0x7f68c012ba00 2022-07-21T01:33:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3413631463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68c012ba00 msgr2=0x7f68c00b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.010+0000 7f68c4ec0700 1 --2- 172.21.15.3:0/3413631463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68c012ba00 0x7f68c00b4300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f68b80032f0 tx=0x7f68b80091d0).stop 2022-07-21T01:33:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.014+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3413631463 shutdown_connections 2022-07-21T01:33:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.014+0000 7f68c4ec0700 1 --2- 172.21.15.3:0/3413631463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68c012ba00 0x7f68c00b4300 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.014+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3413631463 >> 172.21.15.3:0/3413631463 conn(0x7f68c0066f70 msgr2=0x7f68c0067380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.014+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3413631463 shutdown_connections 2022-07-21T01:33:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.014+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3413631463 wait complete. 2022-07-21T01:33:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.014+0000 7f68c4ec0700 1 Processor -- start 2022-07-21T01:33:19.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.014+0000 7f68c4ec0700 1 -- start start 2022-07-21T01:33:19.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68c4ec0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68c0061a20 0x7f68c0155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:19.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68c4ec0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c00b7080 0x7f68c00b9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:19.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68c4ec0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68c012ba00 0x7f68c0141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:19.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68c4ec0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c00b7560 con 0x7f68c00b7080 2022-07-21T01:33:19.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68c4ec0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c0143500 con 0x7f68c012ba00 2022-07-21T01:33:19.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68c4ec0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c0153170 con 0x7f68c0061a20 2022-07-21T01:33:19.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b5d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c00b7080 0x7f68c00b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:19.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b5d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c00b7080 0x7f68c00b9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42386/0 (socket says 172.21.15.3:42386) 2022-07-21T01:33:19.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b5d9b700 1 -- 172.21.15.3:0/3494020809 learned_addr learned my addr 172.21.15.3:0/3494020809 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:19.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68be59c700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68c0061a20 0x7f68c0155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:19.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68bed9d700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68c012ba00 0x7f68c0141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:19.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b5d9b700 1 -- 172.21.15.3:0/3494020809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68c0061a20 msgr2=0x7f68c0155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:19.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b5d9b700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68c0061a20 0x7f68c0155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:19.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b5d9b700 1 -- 172.21.15.3:0/3494020809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68c012ba00 msgr2=0x7f68c0141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:19.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b5d9b700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68c012ba00 0x7f68c0141a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:19.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b5d9b700 1 -- 172.21.15.3:0/3494020809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8009470 con 0x7f68c00b7080 2022-07-21T01:33:19.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b5d9b700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c00b7080 0x7f68c00b9770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f68c00b21f0 tx=0x7f68a8007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:19.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b7fff700 1 -- 172.21.15.3:0/3494020809 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a8007030 con 0x7f68c00b7080 2022-07-21T01:33:19.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b7fff700 1 -- 172.21.15.3:0/3494020809 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a8007190 con 0x7f68c00b7080 2022-07-21T01:33:19.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c0153410 con 0x7f68c00b7080 2022-07-21T01:33:19.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c00ba080 con 0x7f68c00b7080 2022-07-21T01:33:19.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68b7fff700 1 -- 172.21.15.3:0/3494020809 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a8006510 con 0x7f68c00b7080 2022-07-21T01:33:19.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.018+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c01535a0 con 0x7f68c00b7080 2022-07-21T01:33:19.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.022+0000 7f68b7fff700 1 -- 172.21.15.3:0/3494020809 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f68a8010030 con 0x7f68c00b7080 2022-07-21T01:33:19.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.022+0000 7f68b7fff700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68a4053a10 0x7f68a4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:19.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.022+0000 7f68b7fff700 1 -- 172.21.15.3:0/3494020809 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f68a806b580 con 0x7f68c00b7080 2022-07-21T01:33:19.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.022+0000 7f68be59c700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68a4053a10 0x7f68a4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:19.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.022+0000 7f68be59c700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68a4053a10 0x7f68a4055ed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f68b000d000 tx=0x7f68b0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.030+0000 7f68b7fff700 1 -- 172.21.15.3:0/3494020809 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f68a8042650 con 0x7f68c00b7080 2022-07-21T01:33:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.154+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f68c012c5d0 con 0x7f68c00b7080 2022-07-21T01:33:19.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.890+0000 7f68b7fff700 1 -- 172.21.15.3:0/3494020809 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v19) v1 ==== 173+0+0 (secure 0 0 0) 0x7f68a8045c70 con 0x7f68c00b7080 2022-07-21T01:33:19.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68a4053a10 msgr2=0x7f68a4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:19.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68a4053a10 0x7f68a4055ed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f68b000d000 tx=0x7f68b0009040).stop 2022-07-21T01:33:19.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c00b7080 msgr2=0x7f68c00b9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:19.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c00b7080 0x7f68c00b9770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f68c00b21f0 tx=0x7f68a8007c10).stop 2022-07-21T01:33:19.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 shutdown_connections 2022-07-21T01:33:19.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68a4053a10 0x7f68a4055ed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:19.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68c012ba00 0x7f68c0141a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:19.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c00b7080 0x7f68c00b9770 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:19.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 --2- 172.21.15.3:0/3494020809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68c0061a20 0x7f68c0155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:19.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 >> 172.21.15.3:0/3494020809 conn(0x7f68c0066f70 msgr2=0x7f68c005ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:19.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 shutdown_connections 2022-07-21T01:33:19.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:19.894+0000 7f68c4ec0700 1 -- 172.21.15.3:0/3494020809 wait complete. 2022-07-21T01:33:19.895 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rbd' on pool 'rbd' 2022-07-21T01:33:19.903 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-07-21T01:33:19.904 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:33:19.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:33:19.907 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:33:19.907 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:33:19.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:33:19.911 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:33:19.912 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:33:19.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:33:19.915 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:33:19.916 INFO:tasks.ceph:Setting up CephFS filesystem... 2022-07-21T01:33:19.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:19.977 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:33:19.984 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:33:19.985 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:33:20.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb8a653700 1 Processor -- start 2022-07-21T01:33:20.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb8a653700 1 -- start start 2022-07-21T01:33:20.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb8a653700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 0x7fcb84061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb8a653700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb841405d0 0x7fcb84140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb8a653700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb840aee80 con 0x7fcb840ad750 2022-07-21T01:33:20.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb8a653700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb840b70b0 con 0x7fcb841405d0 2022-07-21T01:33:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb8a653700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb840af0b0 con 0x7fcb840614b0 2022-07-21T01:33:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb89e52700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb841405d0 0x7fcb84140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb88e50700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcb840ad750 0x7fcb840b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50702/0 (socket says 172.21.15.3:50702) 2022-07-21T01:33:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb88e50700 1 -- 172.21.15.3:0/2110752213 learned_addr learned my addr 172.21.15.3:0/2110752213 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.030+0000 7fcb89651700 1 --2- 172.21.15.3:0/2110752213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 0x7fcb84061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb77fff700 1 -- 172.21.15.3:0/2110752213 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942316490 0 0) 0x7fcb840aee80 con 0x7fcb840ad750 2022-07-21T01:33:20.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb77fff700 1 -- 172.21.15.3:0/2110752213 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb68003260 con 0x7fcb840ad750 2022-07-21T01:33:20.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb89e52700 1 -- 172.21.15.3:0/2110752213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 msgr2=0x7fcb84061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:20.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb89e52700 1 --2- 172.21.15.3:0/2110752213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 0x7fcb84061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb89e52700 1 -- 172.21.15.3:0/2110752213 >> v1:172.21.15.3:6789/0 conn(0x7fcb840ad750 legacy=0x7fcb840b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb89e52700 1 -- 172.21.15.3:0/2110752213 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb840af350 con 0x7fcb841405d0 2022-07-21T01:33:20.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb89e52700 1 --2- 172.21.15.3:0/2110752213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb841405d0 0x7fcb84140a60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcb7c0032f0 tx=0x7fcb7c0091d0).ready entity=mon.0 client_cookie=5164f7ba31f523da server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb77fff700 1 -- 172.21.15.3:0/2110752213 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7c00b600 con 0x7fcb841405d0 2022-07-21T01:33:20.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb77fff700 1 -- 172.21.15.3:0/2110752213 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c00b760 con 0x7fcb841405d0 2022-07-21T01:33:20.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb77fff700 1 -- 172.21.15.3:0/2110752213 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7c00b8d0 con 0x7fcb841405d0 2022-07-21T01:33:20.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 -- 172.21.15.3:0/2110752213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb841405d0 msgr2=0x7fcb84140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 --2- 172.21.15.3:0/2110752213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb841405d0 0x7fcb84140a60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcb7c0032f0 tx=0x7fcb7c0091d0).stop 2022-07-21T01:33:20.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 -- 172.21.15.3:0/2110752213 shutdown_connections 2022-07-21T01:33:20.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 --2- 172.21.15.3:0/2110752213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb841405d0 0x7fcb84140a60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 --2- 172.21.15.3:0/2110752213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 0x7fcb84061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 -- 172.21.15.3:0/2110752213 >> 172.21.15.3:0/2110752213 conn(0x7fcb84066f70 msgr2=0x7fcb84067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:20.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 -- 172.21.15.3:0/2110752213 shutdown_connections 2022-07-21T01:33:20.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 -- 172.21.15.3:0/2110752213 wait complete. 2022-07-21T01:33:20.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 Processor -- start 2022-07-21T01:33:20.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.034+0000 7fcb8a653700 1 -- start start 2022-07-21T01:33:20.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb8a653700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 0x7fcb840b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb8a653700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb840ad750 0x7fcb840ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb8a653700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb841405d0 0x7fcb840abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb8a653700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84145fd0 con 0x7fcb840ad750 2022-07-21T01:33:20.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb8a653700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb840b1320 con 0x7fcb841405d0 2022-07-21T01:33:20.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb8a653700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb840aaa10 con 0x7fcb840614b0 2022-07-21T01:33:20.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb88e50700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb840ad750 0x7fcb840ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb88e50700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb840ad750 0x7fcb840ab9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42426/0 (socket says 172.21.15.3:42426) 2022-07-21T01:33:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb89651700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 0x7fcb840b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb88e50700 1 -- 172.21.15.3:0/857425577 learned_addr learned my addr 172.21.15.3:0/857425577 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb89e52700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb841405d0 0x7fcb840abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb88e50700 1 -- 172.21.15.3:0/857425577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 msgr2=0x7fcb840b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb88e50700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 0x7fcb840b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb88e50700 1 -- 172.21.15.3:0/857425577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb841405d0 msgr2=0x7fcb840abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb88e50700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb841405d0 0x7fcb840abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb88e50700 1 -- 172.21.15.3:0/857425577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c009470 con 0x7fcb840ad750 2022-07-21T01:33:20.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb89e52700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb841405d0 0x7fcb840abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:20.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb88e50700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb840ad750 0x7fcb840ab9b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fcb78001110 tx=0x7fcb78007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb767fc700 1 -- 172.21.15.3:0/857425577 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7800ce20 con 0x7fcb840ad750 2022-07-21T01:33:20.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb767fc700 1 -- 172.21.15.3:0/857425577 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb780072e0 con 0x7fcb840ad750 2022-07-21T01:33:20.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb840acca0 con 0x7fcb840ad750 2022-07-21T01:33:20.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb767fc700 1 -- 172.21.15.3:0/857425577 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb78005800 con 0x7fcb840ad750 2022-07-21T01:33:20.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.038+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb840ab350 con 0x7fcb840ad750 2022-07-21T01:33:20.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.042+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb840b10f0 con 0x7fcb840ad750 2022-07-21T01:33:20.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.042+0000 7fcb767fc700 1 -- 172.21.15.3:0/857425577 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb78010070 con 0x7fcb840ad750 2022-07-21T01:33:20.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.042+0000 7fcb767fc700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb6c053950 0x7fcb6c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.042+0000 7fcb767fc700 1 -- 172.21.15.3:0/857425577 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fcb7806b360 con 0x7fcb840ad750 2022-07-21T01:33:20.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.042+0000 7fcb89651700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb6c053950 0x7fcb6c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.042+0000 7fcb89651700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb6c053950 0x7fcb6c055e10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fcb8000b000 tx=0x7fcb80006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.046+0000 7fcb767fc700 1 -- 172.21.15.3:0/857425577 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb780424b0 con 0x7fcb840ad750 2022-07-21T01:33:20.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.178+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb840aabd0 con 0x7fcb840ad750 2022-07-21T01:33:20.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.178+0000 7fcb767fc700 1 -- 172.21.15.3:0/857425577 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+13191 (secure 0 0 0) 0x7fcb78018020 con 0x7fcb840ad750 2022-07-21T01:33:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:20.177 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":19,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:19.877711+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:20.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb6c053950 msgr2=0x7fcb6c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb6c053950 0x7fcb6c055e10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fcb8000b000 tx=0x7fcb80006df0).stop 2022-07-21T01:33:20.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb840ad750 msgr2=0x7fcb840ab9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb840ad750 0x7fcb840ab9b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fcb78001110 tx=0x7fcb78007ad0).stop 2022-07-21T01:33:20.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 shutdown_connections 2022-07-21T01:33:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb6c053950 0x7fcb6c055e10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb841405d0 0x7fcb840abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb840ad750 0x7fcb840ab9b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 --2- 172.21.15.3:0/857425577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb840614b0 0x7fcb840b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 >> 172.21.15.3:0/857425577 conn(0x7fcb84066f70 msgr2=0x7fcb840b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:20.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 shutdown_connections 2022-07-21T01:33:20.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.182+0000 7fcb8a653700 1 -- 172.21.15.3:0/857425577 wait complete. 2022-07-21T01:33:20.190 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:33:20.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:33:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 Processor -- start 2022-07-21T01:33:20.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 -- start start 2022-07-21T01:33:20.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f707014ad10 0x7f707005f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70700acb40 0x7f707014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70700614f0 con 0x7f70700b00a0 2022-07-21T01:33:20.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707014b770 con 0x7f707014ad10 2022-07-21T01:33:20.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707014ba10 con 0x7f70700acb40 2022-07-21T01:33:20.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f707014ad10 0x7f707005f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f70700b00a0 0x7f70700af350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50714/0 (socket says 172.21.15.3:50714) 2022-07-21T01:33:20.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706f7fe700 1 -- 172.21.15.3:0/2284135254 learned_addr learned my addr 172.21.15.3:0/2284135254 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:20.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f7074b2a700 1 --2- 172.21.15.3:0/2284135254 >> v2:172.21.15.143:3300/0 conn(0x7f70700acb40 0x7f707014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706ffff700 1 -- 172.21.15.3:0/2284135254 >> v2:172.21.15.143:3300/0 conn(0x7f70700acb40 msgr2=0x7f707014b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:20.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706ffff700 1 --2- 172.21.15.3:0/2284135254 >> v2:172.21.15.143:3300/0 conn(0x7f70700acb40 0x7f707014b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706ffff700 1 -- 172.21.15.3:0/2284135254 >> v1:172.21.15.3:6789/0 conn(0x7f70700b00a0 legacy=0x7f70700af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706ffff700 1 -- 172.21.15.3:0/2284135254 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70700aeeb0 con 0x7f707014ad10 2022-07-21T01:33:20.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706ffff700 1 --2- 172.21.15.3:0/2284135254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f707014ad10 0x7f707005f840 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7060002f70 tx=0x7f7060006df0).ready entity=mon.0 client_cookie=dff266e3c6bbe01c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706effd700 1 -- 172.21.15.3:0/2284135254 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f706000d570 con 0x7f707014ad10 2022-07-21T01:33:20.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706effd700 1 -- 172.21.15.3:0/2284135254 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7060000e90 con 0x7f707014ad10 2022-07-21T01:33:20.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f706effd700 1 -- 172.21.15.3:0/2284135254 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f706000d8c0 con 0x7f707014ad10 2022-07-21T01:33:20.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 -- 172.21.15.3:0/2284135254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f707014ad10 msgr2=0x7f707005f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 --2- 172.21.15.3:0/2284135254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f707014ad10 0x7f707005f840 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7060002f70 tx=0x7f7060006df0).stop 2022-07-21T01:33:20.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 -- 172.21.15.3:0/2284135254 shutdown_connections 2022-07-21T01:33:20.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 --2- 172.21.15.3:0/2284135254 >> v2:172.21.15.143:3300/0 conn(0x7f70700acb40 0x7f707014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 --2- 172.21.15.3:0/2284135254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f707014ad10 0x7f707005f840 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 -- 172.21.15.3:0/2284135254 >> 172.21.15.3:0/2284135254 conn(0x7f70700b5700 msgr2=0x7f70700b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:20.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 -- 172.21.15.3:0/2284135254 shutdown_connections 2022-07-21T01:33:20.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.306+0000 7f707658d700 1 -- 172.21.15.3:0/2284135254 wait complete. 2022-07-21T01:33:20.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 Processor -- start 2022-07-21T01:33:20.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 -- start start 2022-07-21T01:33:20.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70700acb40 0x7f70700b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70700b00a0 0x7f707013b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f707014ad10 0x7f707013bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7070146ee0 con 0x7f70700b00a0 2022-07-21T01:33:20.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701460f0 con 0x7f70700acb40 2022-07-21T01:33:20.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7070146390 con 0x7f707014ad10 2022-07-21T01:33:20.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70700b00a0 0x7f707013b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70700b00a0 0x7f707013b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42440/0 (socket says 172.21.15.3:42440) 2022-07-21T01:33:20.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706f7fe700 1 -- 172.21.15.3:0/432984735 learned_addr learned my addr 172.21.15.3:0/432984735 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:20.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f7074b2a700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f707014ad10 0x7f707013bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706ffff700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70700acb40 0x7f70700b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706f7fe700 1 -- 172.21.15.3:0/432984735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f707014ad10 msgr2=0x7f707013bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706f7fe700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f707014ad10 0x7f707013bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706f7fe700 1 -- 172.21.15.3:0/432984735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70700acb40 msgr2=0x7f70700b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706f7fe700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70700acb40 0x7f70700b0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706f7fe700 1 -- 172.21.15.3:0/432984735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706000b180 con 0x7f70700b00a0 2022-07-21T01:33:20.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706f7fe700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70700b00a0 0x7f707013b900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f70700621f0 tx=0x7f7064007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706d7fa700 1 -- 172.21.15.3:0/432984735 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f706400ce30 con 0x7f70700b00a0 2022-07-21T01:33:20.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706d7fa700 1 -- 172.21.15.3:0/432984735 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70640072e0 con 0x7f70700b00a0 2022-07-21T01:33:20.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f706d7fa700 1 -- 172.21.15.3:0/432984735 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7064005800 con 0x7f70700b00a0 2022-07-21T01:33:20.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707013c2e0 con 0x7f70700b00a0 2022-07-21T01:33:20.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.310+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070141ca0 con 0x7f70700b00a0 2022-07-21T01:33:20.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.314+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7070146710 con 0x7f70700b00a0 2022-07-21T01:33:20.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.314+0000 7f706d7fa700 1 -- 172.21.15.3:0/432984735 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7064010070 con 0x7f70700b00a0 2022-07-21T01:33:20.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.314+0000 7f706d7fa700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7050053b30 0x7f7050055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.314+0000 7f706d7fa700 1 -- 172.21.15.3:0/432984735 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f706406b1f0 con 0x7f70700b00a0 2022-07-21T01:33:20.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.314+0000 7f706ffff700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7050053b30 0x7f7050055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.314+0000 7f706ffff700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7050053b30 0x7f7050055ff0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f7060018000 tx=0x7f7060014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.318+0000 7f706d7fa700 1 -- 172.21.15.3:0/432984735 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7064042440 con 0x7f70700b00a0 2022-07-21T01:33:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f70700afa60 con 0x7f70700b00a0 2022-07-21T01:33:20.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f706d7fa700 1 -- 172.21.15.3:0/432984735 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v19) v1 ==== 110+0+10 (secure 0 0 0) 0x7f7064018070 con 0x7f70700b00a0 2022-07-21T01:33:20.444 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:33:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7050053b30 msgr2=0x7f7050055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7050053b30 0x7f7050055ff0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f7060018000 tx=0x7f7060014000).stop 2022-07-21T01:33:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70700b00a0 msgr2=0x7f707013b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70700b00a0 0x7f707013b900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f70700621f0 tx=0x7f7064007c40).stop 2022-07-21T01:33:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 shutdown_connections 2022-07-21T01:33:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f707014ad10 0x7f707013bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70700b00a0 0x7f707013b900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7050053b30 0x7f7050055ff0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.446+0000 7f707658d700 1 --2- 172.21.15.3:0/432984735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70700acb40 0x7f70700b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.450+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 >> 172.21.15.3:0/432984735 conn(0x7f70700b5700 msgr2=0x7f7070148f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.450+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 shutdown_connections 2022-07-21T01:33:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.450+0000 7f707658d700 1 -- 172.21.15.3:0/432984735 wait complete. 2022-07-21T01:33:20.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:33:20.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.570+0000 7f585dbd3700 1 Processor -- start 2022-07-21T01:33:20.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.570+0000 7f585dbd3700 1 -- start start 2022-07-21T01:33:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.570+0000 7f585dbd3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f58580b41f0 0x7f5858061d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.570+0000 7f585dbd3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5858142ff0 0x7f58581434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.570+0000 7f585dbd3700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58580b60e0 con 0x7f58580b6420 2022-07-21T01:33:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.570+0000 7f585dbd3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58580b2100 con 0x7f58580b41f0 2022-07-21T01:33:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.570+0000 7f585dbd3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58580b23a0 con 0x7f5858142ff0 2022-07-21T01:33:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585cbd1700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f58580b6420 0x7f5858061600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50726/0 (socket says 172.21.15.3:50726) 2022-07-21T01:33:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585cbd1700 1 -- 172.21.15.3:0/276843489 learned_addr learned my addr 172.21.15.3:0/276843489 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585d3d2700 1 --2- 172.21.15.3:0/276843489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5858142ff0 0x7f58581434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f5857fff700 1 --2- 172.21.15.3:0/276843489 >> v2:172.21.15.143:3300/0 conn(0x7f58580b41f0 0x7f5858061d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f58577fe700 1 -- 172.21.15.3:0/276843489 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352922049 0 0) 0x7f5848001110 con 0x7f58580b6420 2022-07-21T01:33:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f58577fe700 1 -- 172.21.15.3:0/276843489 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5838003260 con 0x7f58580b6420 2022-07-21T01:33:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585d3d2700 1 -- 172.21.15.3:0/276843489 >> v2:172.21.15.143:3300/0 conn(0x7f58580b41f0 msgr2=0x7f5858061d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585d3d2700 1 --2- 172.21.15.3:0/276843489 >> v2:172.21.15.143:3300/0 conn(0x7f58580b41f0 0x7f5858061d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585d3d2700 1 -- 172.21.15.3:0/276843489 >> v1:172.21.15.3:6789/0 conn(0x7f58580b6420 legacy=0x7f5858061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585d3d2700 1 -- 172.21.15.3:0/276843489 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58580b2640 con 0x7f5858142ff0 2022-07-21T01:33:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585d3d2700 1 --2- 172.21.15.3:0/276843489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5858142ff0 0x7f58581434a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f58500032f0 tx=0x7f58500091d0).ready entity=mon.2 client_cookie=4b6085d6ae64c76b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f58577fe700 1 -- 172.21.15.3:0/276843489 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f585000b600 con 0x7f5858142ff0 2022-07-21T01:33:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f58577fe700 1 -- 172.21.15.3:0/276843489 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585000b760 con 0x7f5858142ff0 2022-07-21T01:33:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f58577fe700 1 -- 172.21.15.3:0/276843489 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f585000b8d0 con 0x7f5858142ff0 2022-07-21T01:33:20.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585dbd3700 1 -- 172.21.15.3:0/276843489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5858142ff0 msgr2=0x7f58581434a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585dbd3700 1 --2- 172.21.15.3:0/276843489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5858142ff0 0x7f58581434a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f58500032f0 tx=0x7f58500091d0).stop 2022-07-21T01:33:20.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585dbd3700 1 -- 172.21.15.3:0/276843489 shutdown_connections 2022-07-21T01:33:20.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585dbd3700 1 --2- 172.21.15.3:0/276843489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5858142ff0 0x7f58581434a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585dbd3700 1 --2- 172.21.15.3:0/276843489 >> v2:172.21.15.143:3300/0 conn(0x7f58580b41f0 0x7f5858061d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585dbd3700 1 -- 172.21.15.3:0/276843489 >> 172.21.15.3:0/276843489 conn(0x7f585805c620 msgr2=0x7f585805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585dbd3700 1 -- 172.21.15.3:0/276843489 shutdown_connections 2022-07-21T01:33:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.574+0000 7f585dbd3700 1 -- 172.21.15.3:0/276843489 wait complete. 2022-07-21T01:33:20.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585dbd3700 1 Processor -- start 2022-07-21T01:33:20.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585dbd3700 1 -- start start 2022-07-21T01:33:20.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585dbd3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58580b41f0 0x7f58580b9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585dbd3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58580b6420 0x7f5858155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585dbd3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5858142ff0 0x7f58581560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585dbd3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585813fc20 con 0x7f58580b41f0 2022-07-21T01:33:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585dbd3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58580b0080 con 0x7f5858142ff0 2022-07-21T01:33:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585dbd3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58580b0320 con 0x7f58580b6420 2022-07-21T01:33:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585cbd1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58580b41f0 0x7f58580b9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585cbd1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58580b41f0 0x7f58580b9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42452/0 (socket says 172.21.15.3:42452) 2022-07-21T01:33:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585cbd1700 1 -- 172.21.15.3:0/2415849584 learned_addr learned my addr 172.21.15.3:0/2415849584 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f5857fff700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58580b6420 0x7f5858155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585d3d2700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5858142ff0 0x7f58581560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585cbd1700 1 -- 172.21.15.3:0/2415849584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58580b6420 msgr2=0x7f5858155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585cbd1700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58580b6420 0x7f5858155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585cbd1700 1 -- 172.21.15.3:0/2415849584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5858142ff0 msgr2=0x7f58581560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585cbd1700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5858142ff0 0x7f58581560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.578+0000 7f585cbd1700 1 -- 172.21.15.3:0/2415849584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5850009470 con 0x7f58580b41f0 2022-07-21T01:33:20.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f585cbd1700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58580b41f0 0x7f58580b9b60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f58480012a0 tx=0x7f5848007900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f5855ffb700 1 -- 172.21.15.3:0/2415849584 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58480070d0 con 0x7f58580b41f0 2022-07-21T01:33:20.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f5855ffb700 1 -- 172.21.15.3:0/2415849584 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5848007230 con 0x7f58580b41f0 2022-07-21T01:33:20.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58580b05c0 con 0x7f58580b41f0 2022-07-21T01:33:20.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f5855ffb700 1 -- 172.21.15.3:0/2415849584 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5848005610 con 0x7f58580b41f0 2022-07-21T01:33:20.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58581565c0 con 0x7f58580b41f0 2022-07-21T01:33:20.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58580b0750 con 0x7f58580b41f0 2022-07-21T01:33:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f5855ffb700 1 -- 172.21.15.3:0/2415849584 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f584801a070 con 0x7f58580b41f0 2022-07-21T01:33:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f5855ffb700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f583c0539e0 0x7f583c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f5855ffb700 1 -- 172.21.15.3:0/2415849584 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f584806afe0 con 0x7f58580b41f0 2022-07-21T01:33:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f5857fff700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f583c0539e0 0x7f583c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.582+0000 7f5857fff700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f583c0539e0 0x7f583c055ea0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f584000b000 tx=0x7f5840006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.590+0000 7f5855ffb700 1 -- 172.21.15.3:0/2415849584 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5848042180 con 0x7f58580b41f0 2022-07-21T01:33:20.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.722+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f58580b60e0 con 0x7f58580b41f0 2022-07-21T01:33:20.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.722+0000 7f5855ffb700 1 -- 172.21.15.3:0/2415849584 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f58580b60e0 con 0x7f58580b41f0 2022-07-21T01:33:20.720 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:20.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f583c0539e0 msgr2=0x7f583c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f583c0539e0 0x7f583c055ea0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f584000b000 tx=0x7f5840006df0).stop 2022-07-21T01:33:20.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58580b41f0 msgr2=0x7f58580b9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58580b41f0 0x7f58580b9b60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f58480012a0 tx=0x7f5848007900).stop 2022-07-21T01:33:20.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 shutdown_connections 2022-07-21T01:33:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f583c0539e0 0x7f583c055ea0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5858142ff0 0x7f58581560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58580b6420 0x7f5858155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 --2- 172.21.15.3:0/2415849584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58580b41f0 0x7f58580b9b60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 >> 172.21.15.3:0/2415849584 conn(0x7f585805c620 msgr2=0x7f58580b8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 shutdown_connections 2022-07-21T01:33:20.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.726+0000 7f585dbd3700 1 -- 172.21.15.3:0/2415849584 wait complete. 2022-07-21T01:33:20.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:33:20.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.834+0000 7ff67c397700 1 Processor -- start 2022-07-21T01:33:20.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.834+0000 7ff67c397700 1 -- start start 2022-07-21T01:33:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff6740b5d80 0x7ff6740b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff674144240 con 0x7ff6740b6710 2022-07-21T01:33:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6740b6fe0 con 0x7ff6740b4550 2022-07-21T01:33:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff674143d00 con 0x7ff6740b5d80 2022-07-21T01:33:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a133700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff6740b6710 0x7ff674061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50740/0 (socket says 172.21.15.3:50740) 2022-07-21T01:33:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a133700 1 -- 172.21.15.3:0/1607112851 learned_addr learned my addr 172.21.15.3:0/1607112851 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a934700 1 --2- 172.21.15.3:0/1607112851 >> v2:172.21.15.3:3300/0 conn(0x7ff6740b5d80 0x7ff6740b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268933989 0 0) 0x7ff674144240 con 0x7ff6740b6710 2022-07-21T01:33:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c003260 con 0x7ff6740b6710 2022-07-21T01:33:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2322391556 0 0) 0x7ff65c003260 con 0x7ff6740b6710 2022-07-21T01:33:20.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 >> v2:172.21.15.3:3300/0 conn(0x7ff6740b5d80 msgr2=0x7ff6740b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:20.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 --2- 172.21.15.3:0/1607112851 >> v2:172.21.15.3:3300/0 conn(0x7ff6740b5d80 0x7ff6740b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 >> v1:172.21.15.143:6789/0 conn(0x7ff6740b4550 legacy=0x7ff67412bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff674143fa0 con 0x7ff6740b6710 2022-07-21T01:33:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7ff670001660 con 0x7ff6740b6710 2022-07-21T01:33:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 >> v1:172.21.15.3:6789/0 conn(0x7ff6740b6710 legacy=0x7ff674061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff65c003750 0x7ff65c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c004500 0x7ff65c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff65c004e70 0x7ff65c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff674143d00 con 0x7ff65c003750 2022-07-21T01:33:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c00bd20 con 0x7ff65c004500 2022-07-21T01:33:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c00bee0 con 0x7ff65c004e70 2022-07-21T01:33:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a133700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff65c003750 0x7ff65c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679932700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c004500 0x7ff65c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a133700 1 -- 172.21.15.3:0/1607112851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff65c004e70 msgr2=0x7ff65c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a133700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff65c004e70 0x7ff65c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:33:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a133700 1 -- 172.21.15.3:0/1607112851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c004500 msgr2=0x7ff65c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a133700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c004500 0x7ff65c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a133700 1 -- 172.21.15.3:0/1607112851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c003260 con 0x7ff65c003750 2022-07-21T01:33:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a934700 1 -- 172.21.15.3:0/1607112851 reap_dead start 2022-07-21T01:33:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679932700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c004500 0x7ff65c004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67a133700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff65c003750 0x7ff65c003b70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7ff670001370 tx=0x7ff674144130).ready entity=mon.0 client_cookie=e9b91af1c84f202f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6700070d0 con 0x7ff65c003750 2022-07-21T01:33:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff670007230 con 0x7ff65c003750 2022-07-21T01:33:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff679131700 1 -- 172.21.15.3:0/1607112851 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff67000c8b0 con 0x7ff65c003750 2022-07-21T01:33:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 -- 172.21.15.3:0/1607112851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff65c003750 msgr2=0x7ff65c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff65c003750 0x7ff65c003b70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7ff670001370 tx=0x7ff674144130).stop 2022-07-21T01:33:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 -- 172.21.15.3:0/1607112851 shutdown_connections 2022-07-21T01:33:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 --2- 172.21.15.3:0/1607112851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff65c003750 0x7ff65c003b70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 -- 172.21.15.3:0/1607112851 >> 172.21.15.3:0/1607112851 conn(0x7ff67405c620 msgr2=0x7ff67405ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 -- 172.21.15.3:0/1607112851 shutdown_connections 2022-07-21T01:33:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.838+0000 7ff67c397700 1 -- 172.21.15.3:0/1607112851 wait complete. 2022-07-21T01:33:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 Processor -- start 2022-07-21T01:33:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 -- start start 2022-07-21T01:33:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c003750 0x7ff67412a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff674141080 0x7ff674152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6740af8c0 0x7ff6740afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67412b150 con 0x7ff6740af8c0 2022-07-21T01:33:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6740ae540 con 0x7ff65c003750 2022-07-21T01:33:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67412a2f0 con 0x7ff674141080 2022-07-21T01:33:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a934700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6740af8c0 0x7ff6740afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a133700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c003750 0x7ff67412a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a934700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6740af8c0 0x7ff6740afd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42476/0 (socket says 172.21.15.3:42476) 2022-07-21T01:33:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a934700 1 -- 172.21.15.3:0/2827843970 learned_addr learned my addr 172.21.15.3:0/2827843970 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff679932700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff674141080 0x7ff674152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a934700 1 -- 172.21.15.3:0/2827843970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff674141080 msgr2=0x7ff674152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a934700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff674141080 0x7ff674152460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a934700 1 -- 172.21.15.3:0/2827843970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c003750 msgr2=0x7ff67412a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a934700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c003750 0x7ff67412a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a934700 1 -- 172.21.15.3:0/2827843970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67412b380 con 0x7ff6740af8c0 2022-07-21T01:33:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff679932700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff674141080 0x7ff674152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67a934700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6740af8c0 0x7ff6740afd40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7ff66c001210 tx=0x7ff66c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff6637fe700 1 -- 172.21.15.3:0/2827843970 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff66c0127f0 con 0x7ff6740af8c0 2022-07-21T01:33:20.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff6637fe700 1 -- 172.21.15.3:0/2827843970 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66c012950 con 0x7ff6740af8c0 2022-07-21T01:33:20.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff6637fe700 1 -- 172.21.15.3:0/2827843970 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff66c012cb0 con 0x7ff6740af8c0 2022-07-21T01:33:20.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6740b3be0 con 0x7ff6740af8c0 2022-07-21T01:33:20.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.842+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6740ae180 con 0x7ff6740af8c0 2022-07-21T01:33:20.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.846+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67412a460 con 0x7ff6740af8c0 2022-07-21T01:33:20.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.846+0000 7ff6637fe700 1 -- 172.21.15.3:0/2827843970 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff66c01a3b0 con 0x7ff6740af8c0 2022-07-21T01:33:20.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.846+0000 7ff6637fe700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff654053a10 0x7ff654055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:20.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.846+0000 7ff6637fe700 1 -- 172.21.15.3:0/2827843970 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff66c06bb60 con 0x7ff6740af8c0 2022-07-21T01:33:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.846+0000 7ff67a133700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff654053a10 0x7ff654055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.846+0000 7ff67a133700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff654053a10 0x7ff654055ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff670019000 tx=0x7ff670015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.850+0000 7ff6637fe700 1 -- 172.21.15.3:0/2827843970 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff67412a460 con 0x7ff6740af8c0 2022-07-21T01:33:20.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.978+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff67412a6d0 con 0x7ff6740af8c0 2022-07-21T01:33:20.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff6637fe700 1 -- 172.21.15.3:0/2827843970 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v20) v1 ==== 84+0+143 (secure 0 0 0) 0x7ff66c016020 con 0x7ff6740af8c0 2022-07-21T01:33:20.978 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:20.978 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:33:20.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:20.978 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:33:20.979 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:33:20.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:20.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:20.979 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:33:20.980 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:33:20.980 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:20.980 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:33:20.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff654053a10 msgr2=0x7ff654055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff654053a10 0x7ff654055ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff670019000 tx=0x7ff670015000).stop 2022-07-21T01:33:20.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6740af8c0 msgr2=0x7ff6740afd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:20.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6740af8c0 0x7ff6740afd40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7ff66c001210 tx=0x7ff66c0091d0).stop 2022-07-21T01:33:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 shutdown_connections 2022-07-21T01:33:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6740af8c0 0x7ff6740afd40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff674141080 0x7ff674152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff654053a10 0x7ff654055ed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 --2- 172.21.15.3:0/2827843970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff65c003750 0x7ff67412a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:20.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 >> 172.21.15.3:0/2827843970 conn(0x7ff67405c620 msgr2=0x7ff674128cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:20.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 shutdown_connections 2022-07-21T01:33:20.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:20.982+0000 7ff67c397700 1 -- 172.21.15.3:0/2827843970 wait complete. 2022-07-21T01:33:20.990 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:33:20.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:33:21.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.098+0000 7f90b91ec700 1 Processor -- start 2022-07-21T01:33:21.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.098+0000 7f90b91ec700 1 -- start start 2022-07-21T01:33:21.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f90b40b2340 0x7f90b405f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:21.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90b4143e20 0x7f90b412c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:21.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b40b3dd0 con 0x7f90b4060eb0 2022-07-21T01:33:21.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b4063f80 con 0x7f90b40b2340 2022-07-21T01:33:21.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b40b2720 con 0x7f90b4143e20 2022-07-21T01:33:21.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f90b40b2340 0x7f90b405f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:21.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f90b40b2340 0x7f90b405f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42482/0 (socket says 172.21.15.3:42482) 2022-07-21T01:33:21.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b3fff700 1 -- 172.21.15.3:0/3284692674 learned_addr learned my addr 172.21.15.3:0/3284692674 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:21.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b89eb700 1 --2- 172.21.15.3:0/3284692674 >> v2:172.21.15.143:3300/0 conn(0x7f90b4143e20 0x7f90b412c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:21.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b3fff700 1 -- 172.21.15.3:0/3284692674 >> v2:172.21.15.143:3300/0 conn(0x7f90b4143e20 msgr2=0x7f90b412c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:21.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b3fff700 1 --2- 172.21.15.3:0/3284692674 >> v2:172.21.15.143:3300/0 conn(0x7f90b4143e20 0x7f90b412c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:21.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b3fff700 1 -- 172.21.15.3:0/3284692674 >> v1:172.21.15.143:6789/0 conn(0x7f90b4060eb0 legacy=0x7f90b40b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:21.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b3fff700 1 -- 172.21.15.3:0/3284692674 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b40b2fc0 con 0x7f90b40b2340 2022-07-21T01:33:21.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b89eb700 1 --2- 172.21.15.3:0/3284692674 >> v2:172.21.15.143:3300/0 conn(0x7f90b4143e20 0x7f90b412c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:21.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b3fff700 1 --2- 172.21.15.3:0/3284692674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b40b2340 0x7f90b405f840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f90a4002f70 tx=0x7f90a4006eb0).ready entity=mon.0 client_cookie=f2e0475b795c3e37 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:21.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b37fe700 1 -- 172.21.15.3:0/3284692674 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90a400d5c0 con 0x7f90b40b2340 2022-07-21T01:33:21.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b37fe700 1 -- 172.21.15.3:0/3284692674 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4000e90 con 0x7f90b40b2340 2022-07-21T01:33:21.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b37fe700 1 -- 172.21.15.3:0/3284692674 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90a400d950 con 0x7f90b40b2340 2022-07-21T01:33:21.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 -- 172.21.15.3:0/3284692674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b40b2340 msgr2=0x7f90b405f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:21.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 --2- 172.21.15.3:0/3284692674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b40b2340 0x7f90b405f840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f90a4002f70 tx=0x7f90a4006eb0).stop 2022-07-21T01:33:21.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 -- 172.21.15.3:0/3284692674 shutdown_connections 2022-07-21T01:33:21.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 --2- 172.21.15.3:0/3284692674 >> v2:172.21.15.143:3300/0 conn(0x7f90b4143e20 0x7f90b412c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:21.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 --2- 172.21.15.3:0/3284692674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b40b2340 0x7f90b405f840 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:21.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 -- 172.21.15.3:0/3284692674 >> 172.21.15.3:0/3284692674 conn(0x7f90b405c620 msgr2=0x7f90b405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:21.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 -- 172.21.15.3:0/3284692674 shutdown_connections 2022-07-21T01:33:21.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.102+0000 7f90b91ec700 1 -- 172.21.15.3:0/3284692674 wait complete. 2022-07-21T01:33:21.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 Processor -- start 2022-07-21T01:33:21.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 -- start start 2022-07-21T01:33:21.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b4060eb0 0x7f90b4145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:21.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b4143e20 0x7f90b4146e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:21.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90b4147330 0x7f90b41560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:21.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40af3a0 con 0x7f90b4143e20 2022-07-21T01:33:21.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40af640 con 0x7f90b4060eb0 2022-07-21T01:33:21.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40af8e0 con 0x7f90b4147330 2022-07-21T01:33:21.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90ab7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b4143e20 0x7f90b4146e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:21.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90ab7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b4143e20 0x7f90b4146e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42488/0 (socket says 172.21.15.3:42488) 2022-07-21T01:33:21.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90ab7fe700 1 -- 172.21.15.3:0/4257642485 learned_addr learned my addr 172.21.15.3:0/4257642485 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:21.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b3fff700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b4060eb0 0x7f90b4145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:21.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b89eb700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90b4147330 0x7f90b41560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:21.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90ab7fe700 1 -- 172.21.15.3:0/4257642485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90b4147330 msgr2=0x7f90b41560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:21.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90ab7fe700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90b4147330 0x7f90b41560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:21.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90ab7fe700 1 -- 172.21.15.3:0/4257642485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b4060eb0 msgr2=0x7f90b4145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:21.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90ab7fe700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b4060eb0 0x7f90b4145ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:21.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90ab7fe700 1 -- 172.21.15.3:0/4257642485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac009470 con 0x7f90b4143e20 2022-07-21T01:33:21.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90ab7fe700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b4143e20 0x7f90b4146e40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f90b40b6450 tx=0x7f909c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:21.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b1ffb700 1 -- 172.21.15.3:0/4257642485 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f909c012400 con 0x7f90b4143e20 2022-07-21T01:33:21.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b1ffb700 1 -- 172.21.15.3:0/4257642485 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909c012560 con 0x7f90b4143e20 2022-07-21T01:33:21.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b1ffb700 1 -- 172.21.15.3:0/4257642485 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f909c0128c0 con 0x7f90b4143e20 2022-07-21T01:33:21.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a400b1e0 con 0x7f90b4143e20 2022-07-21T01:33:21.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.106+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b4067180 con 0x7f90b4143e20 2022-07-21T01:33:21.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.110+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b4061d60 con 0x7f90b4143e20 2022-07-21T01:33:21.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.110+0000 7f90b1ffb700 1 -- 172.21.15.3:0/4257642485 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f909c019030 con 0x7f90b4143e20 2022-07-21T01:33:21.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.110+0000 7f90b1ffb700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f908c053a10 0x7f908c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:21.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.110+0000 7f90b1ffb700 1 -- 172.21.15.3:0/4257642485 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f909c06af40 con 0x7f90b4143e20 2022-07-21T01:33:21.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.110+0000 7f90b3fff700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f908c053a10 0x7f908c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:21.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.110+0000 7f90b3fff700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f908c053a10 0x7f908c055ed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f90a401c000 tx=0x7f90a4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:21.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.114+0000 7f90b1ffb700 1 -- 172.21.15.3:0/4257642485 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f909c042190 con 0x7f90b4143e20 2022-07-21T01:33:21.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:21.242+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f90b40afaa0 con 0x7f90b4143e20 2022-07-21T01:33:22.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.010+0000 7f90b1ffb700 1 -- 172.21.15.3:0/4257642485 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v21) v1 ==== 133+0+0 (secure 0 0 0) 0x7f909c0457b0 con 0x7f90b4143e20 2022-07-21T01:33:22.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f908c053a10 msgr2=0x7f908c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:22.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f908c053a10 0x7f908c055ed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f90a401c000 tx=0x7f90a4015000).stop 2022-07-21T01:33:22.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b4143e20 msgr2=0x7f90b4146e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:22.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b4143e20 0x7f90b4146e40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f90b40b6450 tx=0x7f909c007c50).stop 2022-07-21T01:33:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 shutdown_connections 2022-07-21T01:33:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90b4147330 0x7f90b41560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b4143e20 0x7f90b4146e40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f908c053a10 0x7f908c055ed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 --2- 172.21.15.3:0/4257642485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b4060eb0 0x7f90b4145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 >> 172.21.15.3:0/4257642485 conn(0x7f90b405c620 msgr2=0x7f90b40b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:22.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 shutdown_connections 2022-07-21T01:33:22.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.014+0000 7f90b91ec700 1 -- 172.21.15.3:0/4257642485 wait complete. 2022-07-21T01:33:22.014 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:33:22.026 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2022-07-21T01:33:22.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data m=2 k=2 crush-failure-domain=osd 2022-07-21T01:33:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.138+0000 7efbfefa6700 1 Processor -- start 2022-07-21T01:33:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.138+0000 7efbfefa6700 1 -- start start 2022-07-21T01:33:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.138+0000 7efbfefa6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efbf80b5d80 0x7efbf80b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf8144240 con 0x7efbf80b6710 2022-07-21T01:33:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80b6fe0 con 0x7efbf80b4550 2022-07-21T01:33:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf8143d00 con 0x7efbf80b5d80 2022-07-21T01:33:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfcd42700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efbf80b6710 0x7efbf8061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50778/0 (socket says 172.21.15.3:50778) 2022-07-21T01:33:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfcd42700 1 -- 172.21.15.3:0/2929180527 learned_addr learned my addr 172.21.15.3:0/2929180527 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfd543700 1 --2- 172.21.15.3:0/2929180527 >> v2:172.21.15.143:3300/0 conn(0x7efbf80b5d80 0x7efbf80b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456530238 0 0) 0x7efbf8144240 con 0x7efbf80b6710 2022-07-21T01:33:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe4003260 con 0x7efbf80b6710 2022-07-21T01:33:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383553051 0 0) 0x7efbf80b6fe0 con 0x7efbf80b4550 2022-07-21T01:33:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf8144240 con 0x7efbf80b4550 2022-07-21T01:33:22.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1255557548 0 0) 0x7efbe4003260 con 0x7efbf80b6710 2022-07-21T01:33:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 >> v2:172.21.15.143:3300/0 conn(0x7efbf80b5d80 msgr2=0x7efbf80b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 --2- 172.21.15.3:0/2929180527 >> v2:172.21.15.143:3300/0 conn(0x7efbf80b5d80 0x7efbf80b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 >> v1:172.21.15.143:6789/0 conn(0x7efbf80b4550 legacy=0x7efbf812bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8143fa0 con 0x7efbf80b6710 2022-07-21T01:33:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7efbe8001660 con 0x7efbf80b6710 2022-07-21T01:33:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 >> v1:172.21.15.3:6789/0 conn(0x7efbf80b6710 legacy=0x7efbf8061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbe40038e0 0x7efbe4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbe4004500 0x7efbe4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbe4004e70 0x7efbe400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf8143d00 con 0x7efbe40038e0 2022-07-21T01:33:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbe400bcf0 con 0x7efbe4004e70 2022-07-21T01:33:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbe400beb0 con 0x7efbe4004500 2022-07-21T01:33:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfcd42700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbe40038e0 0x7efbe4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfd543700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbe4004e70 0x7efbe400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf7fff700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbe4004500 0x7efbe4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfcd42700 1 -- 172.21.15.3:0/2929180527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbe4004500 msgr2=0x7efbe4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfcd42700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbe4004500 0x7efbe4004980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfcd42700 1 -- 172.21.15.3:0/2929180527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbe4004e70 msgr2=0x7efbe400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfcd42700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbe4004e70 0x7efbe400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfcd42700 1 -- 172.21.15.3:0/2929180527 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe4003260 con 0x7efbe40038e0 2022-07-21T01:33:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfd543700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbe4004e70 0x7efbe400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfd543700 1 -- 172.21.15.3:0/2929180527 reap_dead start 2022-07-21T01:33:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfcd42700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbe40038e0 0x7efbe4003ce0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7efbe8001430 tx=0x7efbf8144130).ready entity=mon.0 client_cookie=d855103f8fea9f3f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbe80070d0 con 0x7efbe40038e0 2022-07-21T01:33:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe8007230 con 0x7efbe40038e0 2022-07-21T01:33:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbf77fe700 1 -- 172.21.15.3:0/2929180527 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbe800c8f0 con 0x7efbe40038e0 2022-07-21T01:33:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 -- 172.21.15.3:0/2929180527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbe40038e0 msgr2=0x7efbe4003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbe40038e0 0x7efbe4003ce0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7efbe8001430 tx=0x7efbf8144130).stop 2022-07-21T01:33:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 -- 172.21.15.3:0/2929180527 shutdown_connections 2022-07-21T01:33:22.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 --2- 172.21.15.3:0/2929180527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbe40038e0 0x7efbe4003ce0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 -- 172.21.15.3:0/2929180527 >> 172.21.15.3:0/2929180527 conn(0x7efbf805c620 msgr2=0x7efbf805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:22.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 -- 172.21.15.3:0/2929180527 shutdown_connections 2022-07-21T01:33:22.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.142+0000 7efbfefa6700 1 -- 172.21.15.3:0/2929180527 wait complete. 2022-07-21T01:33:22.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfefa6700 1 Processor -- start 2022-07-21T01:33:22.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfefa6700 1 -- start start 2022-07-21T01:33:22.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfefa6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf8142540 0x7efbf8142920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:22.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfefa6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf80b2fe0 0x7efbf8127500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:22.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfefa6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf80aff10 0x7efbf80b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:22.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfefa6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80b0880 con 0x7efbf80aff10 2022-07-21T01:33:22.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfefa6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80b2220 con 0x7efbf80b2fe0 2022-07-21T01:33:22.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfefa6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80b3530 con 0x7efbf8142540 2022-07-21T01:33:22.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfd543700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf80aff10 0x7efbf80b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfd543700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf80aff10 0x7efbf80b0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42510/0 (socket says 172.21.15.3:42510) 2022-07-21T01:33:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfd543700 1 -- 172.21.15.3:0/3349201937 learned_addr learned my addr 172.21.15.3:0/3349201937 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbf7fff700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf80b2fe0 0x7efbf8127500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfcd42700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf8142540 0x7efbf8142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfd543700 1 -- 172.21.15.3:0/3349201937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf8142540 msgr2=0x7efbf8142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfd543700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf8142540 0x7efbf8142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfd543700 1 -- 172.21.15.3:0/3349201937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf80b2fe0 msgr2=0x7efbf8127500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfd543700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf80b2fe0 0x7efbf8127500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfd543700 1 -- 172.21.15.3:0/3349201937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf80b37d0 con 0x7efbf80aff10 2022-07-21T01:33:22.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbf7fff700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf80b2fe0 0x7efbf8127500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:22.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfd543700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf80aff10 0x7efbf80b0390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7efbf81526b0 tx=0x7efbf0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:22.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbf5ffb700 1 -- 172.21.15.3:0/3349201937 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbf0013550 con 0x7efbf80aff10 2022-07-21T01:33:22.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbf5ffb700 1 -- 172.21.15.3:0/3349201937 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf00136b0 con 0x7efbf80aff10 2022-07-21T01:33:22.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf80b3a70 con 0x7efbf80aff10 2022-07-21T01:33:22.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.146+0000 7efbf5ffb700 1 -- 172.21.15.3:0/3349201937 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efbf0013a10 con 0x7efbf80aff10 2022-07-21T01:33:22.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.150+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf8155e10 con 0x7efbf80aff10 2022-07-21T01:33:22.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.150+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf8061780 con 0x7efbf80aff10 2022-07-21T01:33:22.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.150+0000 7efbf5ffb700 1 -- 172.21.15.3:0/3349201937 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efbf001d030 con 0x7efbf80aff10 2022-07-21T01:33:22.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.150+0000 7efbf5ffb700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efbd8053aa0 0x7efbd8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:22.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.150+0000 7efbf5ffb700 1 -- 172.21.15.3:0/3349201937 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5607+0+0 (secure 0 0 0) 0x7efbf006bf00 con 0x7efbf80aff10 2022-07-21T01:33:22.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.150+0000 7efbfcd42700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efbd8053aa0 0x7efbd8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:22.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.150+0000 7efbfcd42700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efbd8053aa0 0x7efbd8055f60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7efbe8016000 tx=0x7efbe8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:22.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.154+0000 7efbf5ffb700 1 -- 172.21.15.3:0/3349201937 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efbf0042e80 con 0x7efbf80aff10 2022-07-21T01:33:22.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:22.290+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7efbf80574e0 con 0x7efbf80aff10 2022-07-21T01:33:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.014+0000 7efbf5ffb700 1 -- 172.21.15.3:0/3349201937 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v22) v1 ==== 154+0+0 (secure 0 0 0) 0x7efbf001b020 con 0x7efbf80aff10 2022-07-21T01:33:23.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efbd8053aa0 msgr2=0x7efbd8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:23.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efbd8053aa0 0x7efbd8055f60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7efbe8016000 tx=0x7efbe8015000).stop 2022-07-21T01:33:23.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf80aff10 msgr2=0x7efbf80b0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:23.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf80aff10 0x7efbf80b0390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7efbf81526b0 tx=0x7efbf0008040).stop 2022-07-21T01:33:23.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 shutdown_connections 2022-07-21T01:33:23.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf80aff10 0x7efbf80b0390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:23.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efbd8053aa0 0x7efbd8055f60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:23.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf80b2fe0 0x7efbf8127500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:23.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 --2- 172.21.15.3:0/3349201937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf8142540 0x7efbf8142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:23.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 >> 172.21.15.3:0/3349201937 conn(0x7efbf805c620 msgr2=0x7efbf8128b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:23.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.018+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 shutdown_connections 2022-07-21T01:33:23.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.022+0000 7efbfefa6700 1 -- 172.21.15.3:0/3349201937 wait complete. 2022-07-21T01:33:23.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 erasure cephfs_data 2022-07-21T01:33:23.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.154+0000 7f6585d27700 1 Processor -- start 2022-07-21T01:33:23.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.154+0000 7f6585d27700 1 -- start start 2022-07-21T01:33:23.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.154+0000 7f6585d27700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6580066e70 0x7f658014eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:23.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6580060670 con 0x7f6580060290 2022-07-21T01:33:23.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6580060e00 con 0x7f65800617e0 2022-07-21T01:33:23.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65800672c0 con 0x7f6580066e70 2022-07-21T01:33:23.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6580066e70 0x7f658014eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:23.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6580060290 0x7f6580065b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50794/0 (socket says 172.21.15.3:50794) 2022-07-21T01:33:23.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657f7fe700 1 -- 172.21.15.3:0/408584498 learned_addr learned my addr 172.21.15.3:0/408584498 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:23.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657e7fc700 1 -- 172.21.15.3:0/408584498 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010133735 0 0) 0x7f6580060670 con 0x7f6580060290 2022-07-21T01:33:23.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657e7fc700 1 -- 172.21.15.3:0/408584498 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6560003260 con 0x7f6580060290 2022-07-21T01:33:23.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657ffff700 1 -- 172.21.15.3:0/408584498 >> v1:172.21.15.143:6789/0 conn(0x7f65800617e0 legacy=0x7f6580066760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:23.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657e7fc700 1 -- 172.21.15.3:0/408584498 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288245037 0 0) 0x7f6580060e00 con 0x7f65800617e0 2022-07-21T01:33:23.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657ffff700 1 -- 172.21.15.3:0/408584498 >> v1:172.21.15.3:6789/0 conn(0x7f6580060290 legacy=0x7f6580065b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:23.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657ffff700 1 -- 172.21.15.3:0/408584498 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65800610a0 con 0x7f6580066e70 2022-07-21T01:33:23.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657ffff700 1 --2- 172.21.15.3:0/408584498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6580066e70 0x7f658014eb70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f6568001800 tx=0x7f65780091d0).ready entity=mon.0 client_cookie=e1f9a2891058d088 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:23.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657e7fc700 1 -- 172.21.15.3:0/408584498 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6568001660 con 0x7f6580066e70 2022-07-21T01:33:23.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657e7fc700 1 -- 172.21.15.3:0/408584498 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657800b580 con 0x7f6580066e70 2022-07-21T01:33:23.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f657e7fc700 1 -- 172.21.15.3:0/408584498 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f657800b6f0 con 0x7f6580066e70 2022-07-21T01:33:23.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 -- 172.21.15.3:0/408584498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6580066e70 msgr2=0x7f658014eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:23.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 --2- 172.21.15.3:0/408584498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6580066e70 0x7f658014eb70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f6568001800 tx=0x7f65780091d0).stop 2022-07-21T01:33:23.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 -- 172.21.15.3:0/408584498 shutdown_connections 2022-07-21T01:33:23.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 --2- 172.21.15.3:0/408584498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6580066e70 0x7f658014eb70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:23.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 -- 172.21.15.3:0/408584498 >> 172.21.15.3:0/408584498 conn(0x7f658005d0c0 msgr2=0x7f658005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:23.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 -- 172.21.15.3:0/408584498 shutdown_connections 2022-07-21T01:33:23.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.158+0000 7f6585d27700 1 -- 172.21.15.3:0/408584498 wait complete. 2022-07-21T01:33:23.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 Processor -- start 2022-07-21T01:33:23.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 -- start start 2022-07-21T01:33:23.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6580060290 0x7f6580142760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:23.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65800617e0 0x7f65800a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:23.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6580066e70 0x7f65800a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:23.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65801472a0 con 0x7f65800617e0 2022-07-21T01:33:23.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6580145b80 con 0x7f6580060290 2022-07-21T01:33:23.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6580147490 con 0x7f6580066e70 2022-07-21T01:33:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65800617e0 0x7f65800a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65800617e0 0x7f65800a2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42522/0 (socket says 172.21.15.3:42522) 2022-07-21T01:33:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657effd700 1 -- 172.21.15.3:0/1549525366 learned_addr learned my addr 172.21.15.3:0/1549525366 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657f7fe700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6580060290 0x7f6580142760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657ffff700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6580066e70 0x7f65800a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657effd700 1 -- 172.21.15.3:0/1549525366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6580066e70 msgr2=0x7f65800a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657effd700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6580066e70 0x7f65800a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657effd700 1 -- 172.21.15.3:0/1549525366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6580060290 msgr2=0x7f6580142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657effd700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6580060290 0x7f6580142760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:23.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657effd700 1 -- 172.21.15.3:0/1549525366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6578009470 con 0x7f65800617e0 2022-07-21T01:33:23.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657ffff700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6580066e70 0x7f65800a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:23.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657effd700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65800617e0 0x7f65800a2fd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f65800a4720 tx=0x7f6570007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:23.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657cff9700 1 -- 172.21.15.3:0/1549525366 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6570012400 con 0x7f65800617e0 2022-07-21T01:33:23.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657cff9700 1 -- 172.21.15.3:0/1549525366 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6570012560 con 0x7f65800617e0 2022-07-21T01:33:23.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6580145e10 con 0x7f65800617e0 2022-07-21T01:33:23.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f657cff9700 1 -- 172.21.15.3:0/1549525366 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65700128c0 con 0x7f65800617e0 2022-07-21T01:33:23.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.162+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6580142c50 con 0x7f65800617e0 2022-07-21T01:33:23.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.166+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6580061340 con 0x7f65800617e0 2022-07-21T01:33:23.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.166+0000 7f657cff9700 1 -- 172.21.15.3:0/1549525366 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6570019030 con 0x7f65800617e0 2022-07-21T01:33:23.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.166+0000 7f657cff9700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6564053a70 0x7f6564055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:23.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.166+0000 7f657cff9700 1 -- 172.21.15.3:0/1549525366 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5863+0+0 (secure 0 0 0) 0x7f657006af70 con 0x7f65800617e0 2022-07-21T01:33:23.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.166+0000 7f657f7fe700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6564053a70 0x7f6564055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:23.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.166+0000 7f657f7fe700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6564053a70 0x7f6564055f30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f656800d000 tx=0x7f6568000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:23.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.174+0000 7f657cff9700 1 -- 172.21.15.3:0/1549525366 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6570041fb0 con 0x7f65800617e0 2022-07-21T01:33:23.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:23.294+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8, "pool_type": "erasure", "erasure_code_profile": "cephfs_data"} v 0) v1 -- 0x7f65800574e0 con 0x7f65800617e0 2022-07-21T01:33:25.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.038+0000 7f657cff9700 1 -- 172.21.15.3:0/1549525366 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8, "pool_type": "erasure", "erasure_code_profile": "cephfs_data"}]=0 pool 'cephfs_data' created v24) v1 ==== 188+0+0 (secure 0 0 0) 0x7f65700455d0 con 0x7f65800617e0 2022-07-21T01:33:25.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6564053a70 msgr2=0x7f6564055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:25.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6564053a70 0x7f6564055f30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f656800d000 tx=0x7f6568000bc0).stop 2022-07-21T01:33:25.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65800617e0 msgr2=0x7f65800a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:25.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65800617e0 0x7f65800a2fd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f65800a4720 tx=0x7f6570007ac0).stop 2022-07-21T01:33:25.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 shutdown_connections 2022-07-21T01:33:25.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6580066e70 0x7f65800a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:25.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65800617e0 0x7f65800a2fd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:25.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6564053a70 0x7f6564055f30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:25.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 --2- 172.21.15.3:0/1549525366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6580060290 0x7f6580142760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:25.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 >> 172.21.15.3:0/1549525366 conn(0x7f658005d0c0 msgr2=0x7f65800b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:25.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 shutdown_connections 2022-07-21T01:33:25.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.042+0000 7f6585d27700 1 -- 172.21.15.3:0/1549525366 wait complete. 2022-07-21T01:33:25.045 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:33:25.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true 2022-07-21T01:33:25.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa48ffff700 1 Processor -- start 2022-07-21T01:33:25.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa48ffff700 1 -- start start 2022-07-21T01:33:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa48ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 0x7fa490142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa48ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa4900ad000 0x7fa49013fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa48ffff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa490143ab0 con 0x7fa4900b3260 2022-07-21T01:33:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa48ffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4900b0c80 con 0x7fa4900ad000 2022-07-21T01:33:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa48ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4900ae810 con 0x7fa4900b08a0 2022-07-21T01:33:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa4867fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa4900b3260 0x7fa4901433a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50812/0 (socket says 172.21.15.3:50812) 2022-07-21T01:33:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa4867fc700 1 -- 172.21.15.3:0/661160219 learned_addr learned my addr 172.21.15.3:0/661160219 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa48f7fe700 1 --2- 172.21.15.3:0/661160219 >> v2:172.21.15.143:3300/0 conn(0x7fa4900ad000 0x7fa49013fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.166+0000 7fa48effd700 1 --2- 172.21.15.3:0/661160219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 0x7fa490142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48e7fc700 1 -- 172.21.15.3:0/661160219 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060919027 0 0) 0x7fa490143ab0 con 0x7fa4900b3260 2022-07-21T01:33:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48e7fc700 1 -- 172.21.15.3:0/661160219 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa470003260 con 0x7fa4900b3260 2022-07-21T01:33:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48f7fe700 1 -- 172.21.15.3:0/661160219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 msgr2=0x7fa490142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:25.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48f7fe700 1 --2- 172.21.15.3:0/661160219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 0x7fa490142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:25.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48f7fe700 1 -- 172.21.15.3:0/661160219 >> v1:172.21.15.3:6789/0 conn(0x7fa4900b3260 legacy=0x7fa4901433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48f7fe700 1 -- 172.21.15.3:0/661160219 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4900aeab0 con 0x7fa4900ad000 2022-07-21T01:33:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48effd700 1 --2- 172.21.15.3:0/661160219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 0x7fa490142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48f7fe700 1 --2- 172.21.15.3:0/661160219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 0x7fa49013fa30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa4880032f0 tx=0x7fa4880091d0).ready entity=mon.1 client_cookie=a82d1393ae0edcd5 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48e7fc700 1 -- 172.21.15.3:0/661160219 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa48800b600 con 0x7fa4900ad000 2022-07-21T01:33:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48e7fc700 1 -- 172.21.15.3:0/661160219 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48800b760 con 0x7fa4900ad000 2022-07-21T01:33:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48e7fc700 1 -- 172.21.15.3:0/661160219 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa48800b8d0 con 0x7fa4900ad000 2022-07-21T01:33:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48ffff700 1 -- 172.21.15.3:0/661160219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 msgr2=0x7fa49013fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48ffff700 1 --2- 172.21.15.3:0/661160219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 0x7fa49013fa30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa4880032f0 tx=0x7fa4880091d0).stop 2022-07-21T01:33:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48ffff700 1 -- 172.21.15.3:0/661160219 shutdown_connections 2022-07-21T01:33:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48ffff700 1 --2- 172.21.15.3:0/661160219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 0x7fa49013fa30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48ffff700 1 --2- 172.21.15.3:0/661160219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 0x7fa490142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48ffff700 1 -- 172.21.15.3:0/661160219 >> 172.21.15.3:0/661160219 conn(0x7fa4900612d0 msgr2=0x7fa490062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48ffff700 1 -- 172.21.15.3:0/661160219 shutdown_connections 2022-07-21T01:33:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.170+0000 7fa48ffff700 1 -- 172.21.15.3:0/661160219 wait complete. 2022-07-21T01:33:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 Processor -- start 2022-07-21T01:33:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 -- start start 2022-07-21T01:33:25.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 0x7fa4900b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 0x7fa4901500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:25.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4900b3260 0x7fa490150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa490154550 con 0x7fa4900b3260 2022-07-21T01:33:25.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa490066f90 con 0x7fa4900ad000 2022-07-21T01:33:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa490067230 con 0x7fa4900b08a0 2022-07-21T01:33:25.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4900b3260 0x7fa490150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:25.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 0x7fa4900b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:25.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa4867fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 0x7fa4901500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:25.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 0x7fa4900b6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:47984/0 (socket says 172.21.15.3:47984) 2022-07-21T01:33:25.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48effd700 1 -- 172.21.15.3:0/3172186698 learned_addr learned my addr 172.21.15.3:0/3172186698 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:25.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4900b3260 0x7fa490150590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42540/0 (socket says 172.21.15.3:42540) 2022-07-21T01:33:25.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48effd700 1 -- 172.21.15.3:0/3172186698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 msgr2=0x7fa4901500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:25.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48effd700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 0x7fa4901500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48effd700 1 -- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4900b3260 msgr2=0x7fa490150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:25.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48effd700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4900b3260 0x7fa490150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48effd700 1 -- 172.21.15.3:0/3172186698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48000c310 con 0x7fa4900ad000 2022-07-21T01:33:25.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48f7fe700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4900b3260 0x7fa490150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:25.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48effd700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 0x7fa4900b6240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fa48000d000 tx=0x7fa480008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:25.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48cff9700 1 -- 172.21.15.3:0/3172186698 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa48000aa20 con 0x7fa4900ad000 2022-07-21T01:33:25.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48cff9700 1 -- 172.21.15.3:0/3172186698 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48000ab80 con 0x7fa4900ad000 2022-07-21T01:33:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa488009470 con 0x7fa4900ad000 2022-07-21T01:33:25.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48cff9700 1 -- 172.21.15.3:0/3172186698 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4800113f0 con 0x7fa4900ad000 2022-07-21T01:33:25.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.174+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4900b5260 con 0x7fa4900ad000 2022-07-21T01:33:25.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.178+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4900574e0 con 0x7fa4900ad000 2022-07-21T01:33:25.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.178+0000 7fa48cff9700 1 -- 172.21.15.3:0/3172186698 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa480017030 con 0x7fa4900ad000 2022-07-21T01:33:25.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.178+0000 7fa48cff9700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa474053b00 0x7fa474055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.178+0000 7fa48cff9700 1 -- 172.21.15.3:0/3172186698 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6315+0+0 (secure 0 0 0) 0x7fa48006c170 con 0x7fa4900ad000 2022-07-21T01:33:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.178+0000 7fa4867fc700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa474053b00 0x7fa474055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.178+0000 7fa4867fc700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa474053b00 0x7fa474055fc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa47800d000 tx=0x7fa478009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.190+0000 7fa48cff9700 1 -- 172.21.15.3:0/3172186698 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa480042e90 con 0x7fa4900ad000 2022-07-21T01:33:25.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:25.318+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fa4900b2e00 con 0x7fa4900ad000 2022-07-21T01:33:26.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.042+0000 7fa48cff9700 1 -- 172.21.15.3:0/3172186698 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v25) v1 ==== 166+0+0 (secure 0 0 0) 0x7fa4800464b0 con 0x7fa4900ad000 2022-07-21T01:33:26.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa474053b00 msgr2=0x7fa474055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa474053b00 0x7fa474055fc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa47800d000 tx=0x7fa478009040).stop 2022-07-21T01:33:26.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 msgr2=0x7fa4900b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 0x7fa4900b6240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fa48000d000 tx=0x7fa480008040).stop 2022-07-21T01:33:26.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 shutdown_connections 2022-07-21T01:33:26.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4900b3260 0x7fa490150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4900b08a0 0x7fa4901500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa474053b00 0x7fa474055fc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 --2- 172.21.15.3:0/3172186698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4900ad000 0x7fa4900b6240 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 >> 172.21.15.3:0/3172186698 conn(0x7fa4900612d0 msgr2=0x7fa4900a8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:26.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 shutdown_connections 2022-07-21T01:33:26.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.050+0000 7fa48ffff700 1 -- 172.21.15.3:0/3172186698 wait complete. 2022-07-21T01:33:26.050 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 4 allow_ec_overwrites to true 2022-07-21T01:33:26.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:33:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d5dd2700 1 Processor -- start 2022-07-21T01:33:26.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d5dd2700 1 -- start start 2022-07-21T01:33:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d5dd2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f76d00b41f0 0x7f76d0061d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d5dd2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f76d0142ff0 0x7f76d0143480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d5dd2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d00b60e0 con 0x7f76d00b6420 2022-07-21T01:33:26.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d5dd2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d00b2100 con 0x7f76d0142ff0 2022-07-21T01:33:26.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d5dd2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d00b23a0 con 0x7f76d00b41f0 2022-07-21T01:33:26.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d55d1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f76d0142ff0 0x7f76d0143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d55d1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f76d0142ff0 0x7f76d0143480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42548/0 (socket says 172.21.15.3:42548) 2022-07-21T01:33:26.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.174+0000 7f76d55d1700 1 -- 172.21.15.3:0/3685132522 learned_addr learned my addr 172.21.15.3:0/3685132522 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:26.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cffff700 1 --2- 172.21.15.3:0/3685132522 >> v2:172.21.15.143:3300/0 conn(0x7f76d00b41f0 0x7f76d0061d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cffff700 1 -- 172.21.15.3:0/3685132522 >> v2:172.21.15.3:3300/0 conn(0x7f76d0142ff0 msgr2=0x7f76d0143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cffff700 1 --2- 172.21.15.3:0/3685132522 >> v2:172.21.15.3:3300/0 conn(0x7f76d0142ff0 0x7f76d0143480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cffff700 1 -- 172.21.15.3:0/3685132522 >> v1:172.21.15.143:6789/0 conn(0x7f76d00b6420 legacy=0x7f76d0061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cf7fe700 1 -- 172.21.15.3:0/3685132522 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072727774 0 0) 0x7f76c0001110 con 0x7f76d00b6420 2022-07-21T01:33:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cffff700 1 -- 172.21.15.3:0/3685132522 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d00b2640 con 0x7f76d00b41f0 2022-07-21T01:33:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76d55d1700 1 --2- 172.21.15.3:0/3685132522 >> v2:172.21.15.3:3300/0 conn(0x7f76d0142ff0 0x7f76d0143480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cffff700 1 --2- 172.21.15.3:0/3685132522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d00b41f0 0x7f76d0061d10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f76c4002f70 tx=0x7f76c4006df0).ready entity=mon.1 client_cookie=ddf625509e42b095 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:26.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cf7fe700 1 -- 172.21.15.3:0/3685132522 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76c400d570 con 0x7f76d00b41f0 2022-07-21T01:33:26.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cf7fe700 1 -- 172.21.15.3:0/3685132522 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c4000e90 con 0x7f76d00b41f0 2022-07-21T01:33:26.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76cf7fe700 1 -- 172.21.15.3:0/3685132522 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76c400d8c0 con 0x7f76d00b41f0 2022-07-21T01:33:26.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76d5dd2700 1 -- 172.21.15.3:0/3685132522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d00b41f0 msgr2=0x7f76d0061d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76d5dd2700 1 --2- 172.21.15.3:0/3685132522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d00b41f0 0x7f76d0061d10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f76c4002f70 tx=0x7f76c4006df0).stop 2022-07-21T01:33:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76d5dd2700 1 -- 172.21.15.3:0/3685132522 shutdown_connections 2022-07-21T01:33:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76d5dd2700 1 --2- 172.21.15.3:0/3685132522 >> v2:172.21.15.3:3300/0 conn(0x7f76d0142ff0 0x7f76d0143480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76d5dd2700 1 --2- 172.21.15.3:0/3685132522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d00b41f0 0x7f76d0061d10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76d5dd2700 1 -- 172.21.15.3:0/3685132522 >> 172.21.15.3:0/3685132522 conn(0x7f76d005c620 msgr2=0x7f76d005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:26.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76d5dd2700 1 -- 172.21.15.3:0/3685132522 shutdown_connections 2022-07-21T01:33:26.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.178+0000 7f76d5dd2700 1 -- 172.21.15.3:0/3685132522 wait complete. 2022-07-21T01:33:26.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d5dd2700 1 Processor -- start 2022-07-21T01:33:26.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d5dd2700 1 -- start start 2022-07-21T01:33:26.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d5dd2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d00b41f0 0x7f76d013ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d5dd2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d00b6420 0x7f76d00b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d5dd2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d0142ff0 0x7f76d00afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d5dd2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d013e970 con 0x7f76d00b41f0 2022-07-21T01:33:26.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d5dd2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00ae4d0 con 0x7f76d0142ff0 2022-07-21T01:33:26.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d5dd2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d013f5b0 con 0x7f76d00b6420 2022-07-21T01:33:26.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d4dd0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d00b41f0 0x7f76d013ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d4dd0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d00b41f0 0x7f76d013ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42550/0 (socket says 172.21.15.3:42550) 2022-07-21T01:33:26.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d4dd0700 1 -- 172.21.15.3:0/656156832 learned_addr learned my addr 172.21.15.3:0/656156832 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:26.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76cffff700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d00b6420 0x7f76d00b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.182+0000 7f76d55d1700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d0142ff0 0x7f76d00afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76d4dd0700 1 -- 172.21.15.3:0/656156832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d00b6420 msgr2=0x7f76d00b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76d4dd0700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d00b6420 0x7f76d00b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76d4dd0700 1 -- 172.21.15.3:0/656156832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d0142ff0 msgr2=0x7f76d00afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76d4dd0700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d0142ff0 0x7f76d00afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76d4dd0700 1 -- 172.21.15.3:0/656156832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8009530 con 0x7f76d00b41f0 2022-07-21T01:33:26.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76d4dd0700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d00b41f0 0x7f76d013ffc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f76c0000ed0 tx=0x7f76c0001830).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:26.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76cdffb700 1 -- 172.21.15.3:0/656156832 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76c0013450 con 0x7f76d00b41f0 2022-07-21T01:33:26.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76cdffb700 1 -- 172.21.15.3:0/656156832 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c00135b0 con 0x7f76d00b41f0 2022-07-21T01:33:26.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c400b180 con 0x7f76d00b41f0 2022-07-21T01:33:26.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76cdffb700 1 -- 172.21.15.3:0/656156832 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76c0013910 con 0x7f76d00b41f0 2022-07-21T01:33:26.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d013fa80 con 0x7f76d00b41f0 2022-07-21T01:33:26.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d013fd20 con 0x7f76d00b41f0 2022-07-21T01:33:26.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76cdffb700 1 -- 172.21.15.3:0/656156832 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f76c001a060 con 0x7f76d00b41f0 2022-07-21T01:33:26.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.186+0000 7f76cdffb700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f76b0053b00 0x7f76b0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.190+0000 7f76cdffb700 1 -- 172.21.15.3:0/656156832 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6315+0+0 (secure 0 0 0) 0x7f76c006b5c0 con 0x7f76d00b41f0 2022-07-21T01:33:26.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.190+0000 7f76cffff700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f76b0053b00 0x7f76b0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.190+0000 7f76cffff700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f76b0053b00 0x7f76b0055fc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f76c401a000 tx=0x7f76c4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:26.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.198+0000 7f76cdffb700 1 -- 172.21.15.3:0/656156832 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f76d013fd20 con 0x7f76d00b41f0 2022-07-21T01:33:26.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.366+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f76d00b60e0 con 0x7f76d00b41f0 2022-07-21T01:33:26.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.374+0000 7f76cdffb700 1 -- 172.21.15.3:0/656156832 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 3 and data pool 4 v5) v1 ==== 187+0+0 (secure 0 0 0) 0x7f76d00b60e0 con 0x7f76d00b41f0 2022-07-21T01:33:26.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f76b0053b00 msgr2=0x7f76b0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f76b0053b00 0x7f76b0055fc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f76c401a000 tx=0x7f76c4014000).stop 2022-07-21T01:33:26.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d00b41f0 msgr2=0x7f76d013ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d00b41f0 0x7f76d013ffc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f76c0000ed0 tx=0x7f76c0001830).stop 2022-07-21T01:33:26.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 shutdown_connections 2022-07-21T01:33:26.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f76b0053b00 0x7f76b0055fc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d0142ff0 0x7f76d00afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d00b6420 0x7f76d00b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 --2- 172.21.15.3:0/656156832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d00b41f0 0x7f76d013ffc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 >> 172.21.15.3:0/656156832 conn(0x7f76d005c620 msgr2=0x7f76d00b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:26.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 shutdown_connections 2022-07-21T01:33:26.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.382+0000 7f76d5dd2700 1 -- 172.21.15.3:0/656156832 wait complete. 2022-07-21T01:33:26.384 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 3 and data pool 4 2022-07-21T01:33:26.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:26.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe35f0fd700 1 Processor -- start 2022-07-21T01:33:26.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe35f0fd700 1 -- start start 2022-07-21T01:33:26.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe35f0fd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe3580b0a10 0x7fe35814adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe35f0fd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580b2c80 con 0x7fe3580aa2b0 2022-07-21T01:33:26.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe35f0fd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580b2900 con 0x7fe3580b0630 2022-07-21T01:33:26.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe35f0fd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3580b2eb0 con 0x7fe3580b0a10 2022-07-21T01:33:26.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe357fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe3580aa2b0 0x7fe3580ae5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50848/0 (socket says 172.21.15.3:50848) 2022-07-21T01:33:26.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe357fff700 1 -- 172.21.15.3:0/23403591 learned_addr learned my addr 172.21.15.3:0/23403591 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:26.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe35d69a700 1 --2- 172.21.15.3:0/23403591 >> v2:172.21.15.3:3300/0 conn(0x7fe3580b0a10 0x7fe35814adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.494+0000 7fe3577fe700 1 -- 172.21.15.3:0/23403591 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254036428 0 0) 0x7fe3580b2c80 con 0x7fe3580aa2b0 2022-07-21T01:33:26.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe3577fe700 1 -- 172.21.15.3:0/23403591 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe33c003260 con 0x7fe3580aa2b0 2022-07-21T01:33:26.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35d69a700 1 -- 172.21.15.3:0/23403591 >> v1:172.21.15.143:6789/0 conn(0x7fe3580b0630 legacy=0x7fe3580adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35d69a700 1 -- 172.21.15.3:0/23403591 >> v1:172.21.15.3:6789/0 conn(0x7fe3580aa2b0 legacy=0x7fe3580ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35d69a700 1 -- 172.21.15.3:0/23403591 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3580a9d00 con 0x7fe3580b0a10 2022-07-21T01:33:26.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35d69a700 1 --2- 172.21.15.3:0/23403591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 0x7fe35814adf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe3580ad560 tx=0x7fe3500091d0).ready entity=mon.0 client_cookie=5dbc69fb82b3bbf2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:26.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe3577fe700 1 -- 172.21.15.3:0/23403591 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe35000b680 con 0x7fe3580b0a10 2022-07-21T01:33:26.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe3577fe700 1 -- 172.21.15.3:0/23403591 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35000b7e0 con 0x7fe3580b0a10 2022-07-21T01:33:26.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe3577fe700 1 -- 172.21.15.3:0/23403591 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe35000b950 con 0x7fe3580b0a10 2022-07-21T01:33:26.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35f0fd700 1 -- 172.21.15.3:0/23403591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 msgr2=0x7fe35814adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35f0fd700 1 --2- 172.21.15.3:0/23403591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 0x7fe35814adf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe3580ad560 tx=0x7fe3500091d0).stop 2022-07-21T01:33:26.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35f0fd700 1 -- 172.21.15.3:0/23403591 shutdown_connections 2022-07-21T01:33:26.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35f0fd700 1 --2- 172.21.15.3:0/23403591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 0x7fe35814adf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35f0fd700 1 -- 172.21.15.3:0/23403591 >> 172.21.15.3:0/23403591 conn(0x7fe358060950 msgr2=0x7fe358060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:26.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35f0fd700 1 -- 172.21.15.3:0/23403591 shutdown_connections 2022-07-21T01:33:26.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35f0fd700 1 -- 172.21.15.3:0/23403591 wait complete. 2022-07-21T01:33:26.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.498+0000 7fe35f0fd700 1 Processor -- start 2022-07-21T01:33:26.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35f0fd700 1 -- start start 2022-07-21T01:33:26.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35f0fd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3580aa2b0 0x7fe3580abf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35f0fd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3580b0630 0x7fe3580aa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35f0fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 0x7fe3580aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35f0fd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35813c7e0 con 0x7fe3580b0a10 2022-07-21T01:33:26.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35f0fd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35813c440 con 0x7fe3580b0630 2022-07-21T01:33:26.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35f0fd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe358144e50 con 0x7fe3580aa2b0 2022-07-21T01:33:26.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35d69a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 0x7fe3580aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35d69a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 0x7fe3580aae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42578/0 (socket says 172.21.15.3:42578) 2022-07-21T01:33:26.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35d69a700 1 -- 172.21.15.3:0/873736239 learned_addr learned my addr 172.21.15.3:0/873736239 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:26.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35ce99700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3580aa2b0 0x7fe3580abf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe357fff700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3580b0630 0x7fe3580aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35d69a700 1 -- 172.21.15.3:0/873736239 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3580aa2b0 msgr2=0x7fe3580abf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35d69a700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3580aa2b0 0x7fe3580abf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35d69a700 1 -- 172.21.15.3:0/873736239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3580b0630 msgr2=0x7fe3580aa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35d69a700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3580b0630 0x7fe3580aa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35d69a700 1 -- 172.21.15.3:0/873736239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe350009470 con 0x7fe3580b0a10 2022-07-21T01:33:26.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35d69a700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 0x7fe3580aae20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fe358143f20 tx=0x7fe350016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:26.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe355ffb700 1 -- 172.21.15.3:0/873736239 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe35000b680 con 0x7fe3580b0a10 2022-07-21T01:33:26.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe355ffb700 1 -- 172.21.15.3:0/873736239 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe350021630 con 0x7fe3580b0a10 2022-07-21T01:33:26.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3581450f0 con 0x7fe3580b0a10 2022-07-21T01:33:26.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe355ffb700 1 -- 172.21.15.3:0/873736239 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe350021990 con 0x7fe3580b0a10 2022-07-21T01:33:26.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.502+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe358145630 con 0x7fe3580b0a10 2022-07-21T01:33:26.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.506+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3580574e0 con 0x7fe3580b0a10 2022-07-21T01:33:26.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.506+0000 7fe355ffb700 1 -- 172.21.15.3:0/873736239 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe350027020 con 0x7fe3580b0a10 2022-07-21T01:33:26.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.506+0000 7fe355ffb700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe338053a10 0x7fe338055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.506+0000 7fe355ffb700 1 -- 172.21.15.3:0/873736239 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe35006d410 con 0x7fe3580b0a10 2022-07-21T01:33:26.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.506+0000 7fe35ce99700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe338053a10 0x7fe338055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.506+0000 7fe35ce99700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe338053a10 0x7fe338055ed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe34c00d000 tx=0x7fe34c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:26.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.510+0000 7fe355ffb700 1 -- 172.21.15.3:0/873736239 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe350044100 con 0x7fe3580b0a10 2022-07-21T01:33:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.634+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3580447e0 con 0x7fe3580b0a10 2022-07-21T01:33:26.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe355ffb700 1 -- 172.21.15.3:0/873736239 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7fe350015090 con 0x7fe3580b0a10 2022-07-21T01:33:26.635 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:26.635 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:26.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe338053a10 msgr2=0x7fe338055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe35f0fd700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe338053a10 0x7fe338055ed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe34c00d000 tx=0x7fe34c007b90).stop 2022-07-21T01:33:26.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 msgr2=0x7fe3580aae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe35f0fd700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 0x7fe3580aae20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fe358143f20 tx=0x7fe350016000).stop 2022-07-21T01:33:26.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 shutdown_connections 2022-07-21T01:33:26.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe35f0fd700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3580b0a10 0x7fe3580aae20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe35f0fd700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe338053a10 0x7fe338055ed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe35f0fd700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3580b0630 0x7fe3580aa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.638+0000 7fe35f0fd700 1 --2- 172.21.15.3:0/873736239 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3580aa2b0 0x7fe3580abf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.642+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 >> 172.21.15.3:0/873736239 conn(0x7fe358060950 msgr2=0x7fe3580b4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:26.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.642+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 shutdown_connections 2022-07-21T01:33:26.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.642+0000 7fe35f0fd700 1 -- 172.21.15.3:0/873736239 wait complete. 2022-07-21T01:33:26.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:26.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd26d024700 1 Processor -- start 2022-07-21T01:33:26.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd26d024700 1 -- start start 2022-07-21T01:33:26.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd26d024700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd268066a50 0x7fd2680b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd26d024700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2680b1700 con 0x7fd26805fe20 2022-07-21T01:33:26.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd26d024700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd268066e30 con 0x7fd2680b0e60 2022-07-21T01:33:26.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd26d024700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2680b1500 con 0x7fd268066a50 2022-07-21T01:33:26.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd267fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd268066a50 0x7fd2680b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd26c823700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd26805fe20 0x7fd2680b6a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50866/0 (socket says 172.21.15.3:50866) 2022-07-21T01:33:26.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd26c823700 1 -- 172.21.15.3:0/1335654737 learned_addr learned my addr 172.21.15.3:0/1335654737 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:26.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd267fff700 1 -- 172.21.15.3:0/1335654737 >> v1:172.21.15.143:6789/0 conn(0x7fd2680b0e60 legacy=0x7fd2680b6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd267fff700 1 -- 172.21.15.3:0/1335654737 >> v1:172.21.15.3:6789/0 conn(0x7fd26805fe20 legacy=0x7fd2680b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd266ffd700 1 -- 172.21.15.3:0/1335654737 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560823181 0 0) 0x7fd2680b1700 con 0x7fd26805fe20 2022-07-21T01:33:26.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd267fff700 1 -- 172.21.15.3:0/1335654737 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26805f9c0 con 0x7fd268066a50 2022-07-21T01:33:26.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.754+0000 7fd267fff700 1 --2- 172.21.15.3:0/1335654737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 0x7fd2680b2330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fd250002f70 tx=0x7fd250006df0).ready entity=mon.0 client_cookie=fad7a39adb116f3e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:26.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd266ffd700 1 -- 172.21.15.3:0/1335654737 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd25000d610 con 0x7fd268066a50 2022-07-21T01:33:26.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd266ffd700 1 -- 172.21.15.3:0/1335654737 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd250000e90 con 0x7fd268066a50 2022-07-21T01:33:26.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd266ffd700 1 -- 172.21.15.3:0/1335654737 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd25000d960 con 0x7fd268066a50 2022-07-21T01:33:26.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 -- 172.21.15.3:0/1335654737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 msgr2=0x7fd2680b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 --2- 172.21.15.3:0/1335654737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 0x7fd2680b2330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fd250002f70 tx=0x7fd250006df0).stop 2022-07-21T01:33:26.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 -- 172.21.15.3:0/1335654737 shutdown_connections 2022-07-21T01:33:26.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 --2- 172.21.15.3:0/1335654737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 0x7fd2680b2330 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 -- 172.21.15.3:0/1335654737 >> 172.21.15.3:0/1335654737 conn(0x7fd268063800 msgr2=0x7fd268063be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:26.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 -- 172.21.15.3:0/1335654737 shutdown_connections 2022-07-21T01:33:26.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 -- 172.21.15.3:0/1335654737 wait complete. 2022-07-21T01:33:26.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 Processor -- start 2022-07-21T01:33:26.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 -- start start 2022-07-21T01:33:26.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd26805fe20 0x7fd2680aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 0x7fd26813cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2680b0e60 0x7fd2680ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680ac010 con 0x7fd268066a50 2022-07-21T01:33:26.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd268144530 con 0x7fd2680b0e60 2022-07-21T01:33:26.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd26d024700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd268144790 con 0x7fd26805fe20 2022-07-21T01:33:26.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.758+0000 7fd2677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 0x7fd26813cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 0x7fd26813cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42590/0 (socket says 172.21.15.3:42590) 2022-07-21T01:33:26.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2677fe700 1 -- 172.21.15.3:0/148902615 learned_addr learned my addr 172.21.15.3:0/148902615 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:26.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd267fff700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd26805fe20 0x7fd2680aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd26c823700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2680b0e60 0x7fd2680ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2677fe700 1 -- 172.21.15.3:0/148902615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd26805fe20 msgr2=0x7fd2680aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2677fe700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd26805fe20 0x7fd2680aaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2677fe700 1 -- 172.21.15.3:0/148902615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2680b0e60 msgr2=0x7fd2680ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2677fe700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2680b0e60 0x7fd2680ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2677fe700 1 -- 172.21.15.3:0/148902615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25000b180 con 0x7fd268066a50 2022-07-21T01:33:26.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd267fff700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd26805fe20 0x7fd2680aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:26.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2677fe700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 0x7fd26813cda0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fd268063430 tx=0x7fd258007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:26.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2657fa700 1 -- 172.21.15.3:0/148902615 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd25800cd80 con 0x7fd268066a50 2022-07-21T01:33:26.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2657fa700 1 -- 172.21.15.3:0/148902615 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd258012400 con 0x7fd268066a50 2022-07-21T01:33:26.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2657fa700 1 -- 172.21.15.3:0/148902615 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd258012760 con 0x7fd268066a50 2022-07-21T01:33:26.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd268144a30 con 0x7fd268066a50 2022-07-21T01:33:26.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd268143f60 con 0x7fd268066a50 2022-07-21T01:33:26.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd268145220 con 0x7fd268066a50 2022-07-21T01:33:26.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.762+0000 7fd2657fa700 1 -- 172.21.15.3:0/148902615 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd258019030 con 0x7fd268066a50 2022-07-21T01:33:26.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.766+0000 7fd2657fa700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd24c053a70 0x7fd24c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:26.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.766+0000 7fd2657fa700 1 -- 172.21.15.3:0/148902615 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd25806b450 con 0x7fd268066a50 2022-07-21T01:33:26.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.766+0000 7fd267fff700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd24c053a70 0x7fd24c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:26.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.766+0000 7fd267fff700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd24c053a70 0x7fd24c055f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fd25001a000 tx=0x7fd250015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:26.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.766+0000 7fd2657fa700 1 -- 172.21.15.3:0/148902615 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd268145220 con 0x7fd268066a50 2022-07-21T01:33:26.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.894+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd268145220 con 0x7fd268066a50 2022-07-21T01:33:26.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.898+0000 7fd2657fa700 1 -- 172.21.15.3:0/148902615 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7fd268145220 con 0x7fd268066a50 2022-07-21T01:33:26.895 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:26.896 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:26.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.898+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd24c053a70 msgr2=0x7fd24c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.898+0000 7fd26d024700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd24c053a70 0x7fd24c055f30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fd25001a000 tx=0x7fd250015000).stop 2022-07-21T01:33:26.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.898+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 msgr2=0x7fd26813cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:26.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.898+0000 7fd26d024700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 0x7fd26813cda0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fd268063430 tx=0x7fd258007c20).stop 2022-07-21T01:33:26.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.902+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 shutdown_connections 2022-07-21T01:33:26.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.902+0000 7fd26d024700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd24c053a70 0x7fd24c055f30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.902+0000 7fd26d024700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2680b0e60 0x7fd2680ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.902+0000 7fd26d024700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd268066a50 0x7fd26813cda0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.902+0000 7fd26d024700 1 --2- 172.21.15.3:0/148902615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd26805fe20 0x7fd2680aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:26.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.902+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 >> 172.21.15.3:0/148902615 conn(0x7fd268063800 msgr2=0x7fd268139af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:26.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.902+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 shutdown_connections 2022-07-21T01:33:26.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:26.902+0000 7fd26d024700 1 -- 172.21.15.3:0/148902615 wait complete. 2022-07-21T01:33:26.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:33:27.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.026+0000 7f1caffff700 1 Processor -- start 2022-07-21T01:33:27.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.026+0000 7f1caffff700 1 -- start start 2022-07-21T01:33:27.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.026+0000 7f1caffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00aaf10 0x7f1cb0067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb0065af0 con 0x7f1cb00ab2f0 2022-07-21T01:33:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb00650a0 con 0x7f1cb00b4ee0 2022-07-21T01:33:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb00b3e70 con 0x7f1cb00aaf10 2022-07-21T01:33:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cae7fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1cb00ab2f0 0x7f1cb0064990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50878/0 (socket says 172.21.15.3:50878) 2022-07-21T01:33:27.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cae7fc700 1 -- 172.21.15.3:0/1434925156 learned_addr learned my addr 172.21.15.3:0/1434925156 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:27.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caeffd700 1 --2- 172.21.15.3:0/1434925156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00aaf10 0x7f1cb0067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cadffb700 1 -- 172.21.15.3:0/1434925156 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762559366 0 0) 0x7f1cb0065af0 con 0x7f1cb00ab2f0 2022-07-21T01:33:27.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cadffb700 1 -- 172.21.15.3:0/1434925156 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c94003260 con 0x7f1cb00ab2f0 2022-07-21T01:33:27.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cadffb700 1 -- 172.21.15.3:0/1434925156 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672207360 0 0) 0x7f1cb00650a0 con 0x7f1cb00b4ee0 2022-07-21T01:33:27.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cadffb700 1 -- 172.21.15.3:0/1434925156 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb0065af0 con 0x7f1cb00b4ee0 2022-07-21T01:33:27.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cadffb700 1 -- 172.21.15.3:0/1434925156 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1571096233 0 0) 0x7f1c94003260 con 0x7f1cb00ab2f0 2022-07-21T01:33:27.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caeffd700 1 -- 172.21.15.3:0/1434925156 >> v1:172.21.15.143:6789/0 conn(0x7f1cb00b4ee0 legacy=0x7f1cb00b76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caeffd700 1 -- 172.21.15.3:0/1434925156 >> v1:172.21.15.3:6789/0 conn(0x7f1cb00ab2f0 legacy=0x7f1cb0064990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caeffd700 1 -- 172.21.15.3:0/1434925156 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb0066ff0 con 0x7f1cb00aaf10 2022-07-21T01:33:27.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caeffd700 1 --2- 172.21.15.3:0/1434925156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00aaf10 0x7f1cb0067490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f1ca4001290 tx=0x7f1ca0006df0).ready entity=mon.2 client_cookie=b60a6ecdf5ba3977 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:27.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cadffb700 1 -- 172.21.15.3:0/1434925156 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ca4001800 con 0x7f1cb00aaf10 2022-07-21T01:33:27.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cadffb700 1 -- 172.21.15.3:0/1434925156 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca0000e90 con 0x7f1cb00aaf10 2022-07-21T01:33:27.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1cadffb700 1 -- 172.21.15.3:0/1434925156 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ca0012460 con 0x7f1cb00aaf10 2022-07-21T01:33:27.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 -- 172.21.15.3:0/1434925156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00aaf10 msgr2=0x7f1cb0067490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 --2- 172.21.15.3:0/1434925156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00aaf10 0x7f1cb0067490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f1ca4001290 tx=0x7f1ca0006df0).stop 2022-07-21T01:33:27.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 -- 172.21.15.3:0/1434925156 shutdown_connections 2022-07-21T01:33:27.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 --2- 172.21.15.3:0/1434925156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00aaf10 0x7f1cb0067490 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 -- 172.21.15.3:0/1434925156 >> 172.21.15.3:0/1434925156 conn(0x7f1cb0062d70 msgr2=0x7f1cb005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:27.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 -- 172.21.15.3:0/1434925156 shutdown_connections 2022-07-21T01:33:27.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.030+0000 7f1caffff700 1 -- 172.21.15.3:0/1434925156 wait complete. 2022-07-21T01:33:27.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caffff700 1 Processor -- start 2022-07-21T01:33:27.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caffff700 1 -- start start 2022-07-21T01:33:27.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cb00aaf10 0x7f1cb0141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb00ab2f0 0x7f1cb00a92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00b4ee0 0x7f1cb00a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb00a9cd0 con 0x7f1cb00ab2f0 2022-07-21T01:33:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb00adbe0 con 0x7f1cb00aaf10 2022-07-21T01:33:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb00ade80 con 0x7f1cb00b4ee0 2022-07-21T01:33:27.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1cae7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb00ab2f0 0x7f1cb00a92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cb00aaf10 0x7f1cb0141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caf7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00b4ee0 0x7f1cb00a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cb00aaf10 0x7f1cb0141020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:48048/0 (socket says 172.21.15.3:48048) 2022-07-21T01:33:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1cae7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb00ab2f0 0x7f1cb00a92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42604/0 (socket says 172.21.15.3:42604) 2022-07-21T01:33:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caeffd700 1 -- 172.21.15.3:0/674154542 learned_addr learned my addr 172.21.15.3:0/674154542 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caeffd700 1 -- 172.21.15.3:0/674154542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00b4ee0 msgr2=0x7f1cb00a97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.034+0000 7f1caeffd700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00b4ee0 0x7f1cb00a97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1caeffd700 1 -- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb00ab2f0 msgr2=0x7f1cb00a92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1caeffd700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb00ab2f0 0x7f1cb00a92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1caeffd700 1 -- 172.21.15.3:0/674154542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca000b180 con 0x7f1cb00aaf10 2022-07-21T01:33:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1cae7fc700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb00ab2f0 0x7f1cb00a92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:27.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1caeffd700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cb00aaf10 0x7f1cb0141020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f1ca4001290 tx=0x7f1ca0014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:27.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1c9ffff700 1 -- 172.21.15.3:0/674154542 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ca000d920 con 0x7f1cb00aaf10 2022-07-21T01:33:27.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1c9ffff700 1 -- 172.21.15.3:0/674154542 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca000da80 con 0x7f1cb00aaf10 2022-07-21T01:33:27.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb00ae120 con 0x7f1cb00aaf10 2022-07-21T01:33:27.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1c9ffff700 1 -- 172.21.15.3:0/674154542 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ca000dde0 con 0x7f1cb00aaf10 2022-07-21T01:33:27.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb00a4cb0 con 0x7f1cb00aaf10 2022-07-21T01:33:27.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb00574e0 con 0x7f1cb00aaf10 2022-07-21T01:33:27.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1c9ffff700 1 -- 172.21.15.3:0/674154542 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ca0025030 con 0x7f1cb00aaf10 2022-07-21T01:33:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1c9ffff700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c8c0539e0 0x7f1c8c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1c9ffff700 1 -- 172.21.15.3:0/674154542 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1ca006cd80 con 0x7f1cb00aaf10 2022-07-21T01:33:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.038+0000 7f1cae7fc700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c8c0539e0 0x7f1c8c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.042+0000 7f1cae7fc700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c8c0539e0 0x7f1c8c055ea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1ca400d000 tx=0x7f1ca4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.050+0000 7f1c9ffff700 1 -- 172.21.15.3:0/674154542 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ca0070030 con 0x7f1cb00aaf10 2022-07-21T01:33:27.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.214+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1cb00ae5c0 con 0x7f1cb00aaf10 2022-07-21T01:33:27.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.390+0000 7f1c9ffff700 1 -- 172.21.15.3:0/674154542 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1ca0023020 con 0x7f1cb00aaf10 2022-07-21T01:33:27.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c8c0539e0 msgr2=0x7f1c8c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c8c0539e0 0x7f1c8c055ea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1ca400d000 tx=0x7f1ca4009040).stop 2022-07-21T01:33:27.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cb00aaf10 msgr2=0x7f1cb0141020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cb00aaf10 0x7f1cb0141020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f1ca4001290 tx=0x7f1ca0014000).stop 2022-07-21T01:33:27.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 shutdown_connections 2022-07-21T01:33:27.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cb00b4ee0 0x7f1cb00a97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb00ab2f0 0x7f1cb00a92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c8c0539e0 0x7f1c8c055ea0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 --2- 172.21.15.3:0/674154542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cb00aaf10 0x7f1cb0141020 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 >> 172.21.15.3:0/674154542 conn(0x7f1cb0062d70 msgr2=0x7f1cb00aa600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 shutdown_connections 2022-07-21T01:33:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.394+0000 7f1caffff700 1 -- 172.21.15.3:0/674154542 wait complete. 2022-07-21T01:33:27.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:33:27.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.522+0000 7fbb905a6700 1 Processor -- start 2022-07-21T01:33:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.522+0000 7fbb905a6700 1 -- start start 2022-07-21T01:33:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbb880b1f40 0x7fbb880b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb88148530 0x7fbb880b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb880b04e0 con 0x7fbb880a7250 2022-07-21T01:33:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb880a9250 con 0x7fbb88148530 2022-07-21T01:33:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb880b0780 con 0x7fbb880b1f40 2022-07-21T01:33:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8db41700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb88148530 0x7fbb880b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8db41700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb88148530 0x7fbb880b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42608/0 (socket says 172.21.15.3:42608) 2022-07-21T01:33:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8db41700 1 -- 172.21.15.3:0/3472672673 learned_addr learned my addr 172.21.15.3:0/3472672673 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8e342700 1 --2- 172.21.15.3:0/3472672673 >> v2:172.21.15.143:3300/0 conn(0x7fbb880b1f40 0x7fbb880b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8db41700 1 -- 172.21.15.3:0/3472672673 >> v2:172.21.15.143:3300/0 conn(0x7fbb880b1f40 msgr2=0x7fbb880b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8db41700 1 --2- 172.21.15.3:0/3472672673 >> v2:172.21.15.143:3300/0 conn(0x7fbb880b1f40 0x7fbb880b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8db41700 1 -- 172.21.15.3:0/3472672673 >> v1:172.21.15.143:6789/0 conn(0x7fbb880a7250 legacy=0x7fbb881448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8db41700 1 -- 172.21.15.3:0/3472672673 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb880b0a20 con 0x7fbb88148530 2022-07-21T01:33:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8db41700 1 --2- 172.21.15.3:0/3472672673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88148530 0x7fbb880b2320 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbb881478c0 tx=0x7fbb84006eb0).ready entity=mon.0 client_cookie=ab6364546714faeb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8d340700 1 -- 172.21.15.3:0/3472672673 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb8400d610 con 0x7fbb88148530 2022-07-21T01:33:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8d340700 1 -- 172.21.15.3:0/3472672673 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb84000e90 con 0x7fbb88148530 2022-07-21T01:33:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb8d340700 1 -- 172.21.15.3:0/3472672673 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb8400d960 con 0x7fbb88148530 2022-07-21T01:33:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 -- 172.21.15.3:0/3472672673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88148530 msgr2=0x7fbb880b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 --2- 172.21.15.3:0/3472672673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88148530 0x7fbb880b2320 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbb881478c0 tx=0x7fbb84006eb0).stop 2022-07-21T01:33:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 -- 172.21.15.3:0/3472672673 shutdown_connections 2022-07-21T01:33:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 --2- 172.21.15.3:0/3472672673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88148530 0x7fbb880b2320 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 --2- 172.21.15.3:0/3472672673 >> v2:172.21.15.143:3300/0 conn(0x7fbb880b1f40 0x7fbb880b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 -- 172.21.15.3:0/3472672673 >> 172.21.15.3:0/3472672673 conn(0x7fbb8805f840 msgr2=0x7fbb8805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 -- 172.21.15.3:0/3472672673 shutdown_connections 2022-07-21T01:33:27.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 -- 172.21.15.3:0/3472672673 wait complete. 2022-07-21T01:33:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 Processor -- start 2022-07-21T01:33:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.526+0000 7fbb905a6700 1 -- start start 2022-07-21T01:33:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb905a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb880a7250 0x7fbb8814a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb905a6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b1f40 0x7fbb881439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb905a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88148530 0x7fbb88143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb905a6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880a99d0 con 0x7fbb880b1f40 2022-07-21T01:33:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb905a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8814aa00 con 0x7fbb880a7250 2022-07-21T01:33:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb905a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8814ac30 con 0x7fbb88148530 2022-07-21T01:33:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8db41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b1f40 0x7fbb881439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8db41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b1f40 0x7fbb881439d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42614/0 (socket says 172.21.15.3:42614) 2022-07-21T01:33:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8db41700 1 -- 172.21.15.3:0/2013547198 learned_addr learned my addr 172.21.15.3:0/2013547198 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8e342700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb880a7250 0x7fbb8814a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8eb43700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88148530 0x7fbb88143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8db41700 1 -- 172.21.15.3:0/2013547198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88148530 msgr2=0x7fbb88143ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8db41700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88148530 0x7fbb88143ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8db41700 1 -- 172.21.15.3:0/2013547198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb880a7250 msgr2=0x7fbb8814a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8db41700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb880a7250 0x7fbb8814a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8db41700 1 -- 172.21.15.3:0/2013547198 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8400b1e0 con 0x7fbb880b1f40 2022-07-21T01:33:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb8db41700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b1f40 0x7fbb881439d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbb881478c0 tx=0x7fbb84015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:27.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb777fe700 1 -- 172.21.15.3:0/2013547198 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb84002f50 con 0x7fbb880b1f40 2022-07-21T01:33:27.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb777fe700 1 -- 172.21.15.3:0/2013547198 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8400d990 con 0x7fbb880b1f40 2022-07-21T01:33:27.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb880aa370 con 0x7fbb880b1f40 2022-07-21T01:33:27.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb777fe700 1 -- 172.21.15.3:0/2013547198 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb84011d30 con 0x7fbb880b1f40 2022-07-21T01:33:27.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.530+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8814f1d0 con 0x7fbb880b1f40 2022-07-21T01:33:27.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.534+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb880b0cc0 con 0x7fbb880b1f40 2022-07-21T01:33:27.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.534+0000 7fbb777fe700 1 -- 172.21.15.3:0/2013547198 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbb8401a040 con 0x7fbb880b1f40 2022-07-21T01:33:27.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.534+0000 7fbb777fe700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb68053a70 0x7fbb68055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.534+0000 7fbb777fe700 1 -- 172.21.15.3:0/2013547198 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbb8406cea0 con 0x7fbb880b1f40 2022-07-21T01:33:27.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.534+0000 7fbb8e342700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb68053a70 0x7fbb68055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.534+0000 7fbb8e342700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb68053a70 0x7fbb68055f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fbb7800c000 tx=0x7fbb78008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:27.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.542+0000 7fbb777fe700 1 -- 172.21.15.3:0/2013547198 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb84043b90 con 0x7fbb880b1f40 2022-07-21T01:33:27.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.702+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb880574e0 con 0x7fbb880b1f40 2022-07-21T01:33:27.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.702+0000 7fbb777fe700 1 -- 172.21.15.3:0/2013547198 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2474 (secure 0 0 0) 0x7fbb84023070 con 0x7fbb880b1f40 2022-07-21T01:33:27.703 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:27.703 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/22093665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":22093665},{"type":"v1","addr":"172.21.15.143:6833","nonce":22093665}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/140525254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140525254},{"type":"v1","addr":"172.21.15.3:6835","nonce":140525254}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:33:26.371191+0000","modified":"2022-07-21T01:33:27.386296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/4030643578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":4030643578},{"type":"v1","addr":"172.21.15.3:6837","nonce":4030643578}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-21T01:33:27.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb68053a70 msgr2=0x7fbb68055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb68053a70 0x7fbb68055f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fbb7800c000 tx=0x7fbb78008040).stop 2022-07-21T01:33:27.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b1f40 msgr2=0x7fbb881439d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b1f40 0x7fbb881439d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbb881478c0 tx=0x7fbb84015000).stop 2022-07-21T01:33:27.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 shutdown_connections 2022-07-21T01:33:27.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88148530 0x7fbb88143ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b1f40 0x7fbb881439d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb68053a70 0x7fbb68055f30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 --2- 172.21.15.3:0/2013547198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb880a7250 0x7fbb8814a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 >> 172.21.15.3:0/2013547198 conn(0x7fbb8805f840 msgr2=0x7fbb88144930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:27.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 shutdown_connections 2022-07-21T01:33:27.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.706+0000 7fbb905a6700 1 -- 172.21.15.3:0/2013547198 wait complete. 2022-07-21T01:33:27.708 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 6 2022-07-21T01:33:27.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:27.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 Processor -- start 2022-07-21T01:33:27.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 -- start start 2022-07-21T01:33:27.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54580aa040 0x7f54580b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f54580b32c0 0x7f54580af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54580aa420 con 0x7f545814b6f0 2022-07-21T01:33:27.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54580ae490 con 0x7f54580aa040 2022-07-21T01:33:27.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54580af6f0 con 0x7f54580b32c0 2022-07-21T01:33:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f5456d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f545814b6f0 0x7f545805f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50906/0 (socket says 172.21.15.3:50906) 2022-07-21T01:33:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f5456d9d700 1 -- 172.21.15.3:0/3573658338 learned_addr learned my addr 172.21.15.3:0/3573658338 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545659c700 1 --2- 172.21.15.3:0/3573658338 >> v2:172.21.15.3:3300/0 conn(0x7f54580aa040 0x7f54580b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f544ffff700 1 --2- 172.21.15.3:0/3573658338 >> v2:172.21.15.143:3300/0 conn(0x7f54580b32c0 0x7f54580af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f5455d9b700 1 -- 172.21.15.3:0/3573658338 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227817623 0 0) 0x7f54580aa420 con 0x7f545814b6f0 2022-07-21T01:33:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f5455d9b700 1 -- 172.21.15.3:0/3573658338 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c003260 con 0x7f545814b6f0 2022-07-21T01:33:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545659c700 1 -- 172.21.15.3:0/3573658338 >> v2:172.21.15.143:3300/0 conn(0x7f54580b32c0 msgr2=0x7f54580af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545659c700 1 --2- 172.21.15.3:0/3573658338 >> v2:172.21.15.143:3300/0 conn(0x7f54580b32c0 0x7f54580af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545659c700 1 -- 172.21.15.3:0/3573658338 >> v1:172.21.15.3:6789/0 conn(0x7f545814b6f0 legacy=0x7f545805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545659c700 1 -- 172.21.15.3:0/3573658338 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5458148300 con 0x7f54580aa040 2022-07-21T01:33:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545659c700 1 --2- 172.21.15.3:0/3573658338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580aa040 0x7f54580b3870 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5448002f70 tx=0x7f5448006df0).ready entity=mon.0 client_cookie=ec524ac5ec10b3c7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:27.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f5455d9b700 1 -- 172.21.15.3:0/3573658338 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f544800d4f0 con 0x7f54580aa040 2022-07-21T01:33:27.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f5455d9b700 1 -- 172.21.15.3:0/3573658338 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5448000e90 con 0x7f54580aa040 2022-07-21T01:33:27.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f5455d9b700 1 -- 172.21.15.3:0/3573658338 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f544800d840 con 0x7f54580aa040 2022-07-21T01:33:27.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 -- 172.21.15.3:0/3573658338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580aa040 msgr2=0x7f54580b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 --2- 172.21.15.3:0/3573658338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580aa040 0x7f54580b3870 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5448002f70 tx=0x7f5448006df0).stop 2022-07-21T01:33:27.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 -- 172.21.15.3:0/3573658338 shutdown_connections 2022-07-21T01:33:27.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 --2- 172.21.15.3:0/3573658338 >> v2:172.21.15.143:3300/0 conn(0x7f54580b32c0 0x7f54580af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 --2- 172.21.15.3:0/3573658338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580aa040 0x7f54580b3870 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 -- 172.21.15.3:0/3573658338 >> 172.21.15.3:0/3573658338 conn(0x7f5458060950 msgr2=0x7f5458060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:27.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 -- 172.21.15.3:0/3573658338 shutdown_connections 2022-07-21T01:33:27.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.822+0000 7f545ca62700 1 -- 172.21.15.3:0/3573658338 wait complete. 2022-07-21T01:33:27.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 Processor -- start 2022-07-21T01:33:27.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 -- start start 2022-07-21T01:33:27.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54580aa040 0x7f545814c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580b32c0 0x7f5458148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545814b6f0 0x7f54580b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545814ac30 con 0x7f54580b32c0 2022-07-21T01:33:27.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5458149150 con 0x7f54580aa040 2022-07-21T01:33:27.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54580b1210 con 0x7f545814b6f0 2022-07-21T01:33:27.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580b32c0 0x7f5458148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580b32c0 0x7f5458148c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42630/0 (socket says 172.21.15.3:42630) 2022-07-21T01:33:27.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544ffff700 1 -- 172.21.15.3:0/737453906 learned_addr learned my addr 172.21.15.3:0/737453906 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:27.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f5456d9d700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545814b6f0 0x7f54580b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545659c700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54580aa040 0x7f545814c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544ffff700 1 -- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545814b6f0 msgr2=0x7f54580b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544ffff700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545814b6f0 0x7f54580b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544ffff700 1 -- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54580aa040 msgr2=0x7f545814c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544ffff700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54580aa040 0x7f545814c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544ffff700 1 -- 172.21.15.3:0/737453906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544800b180 con 0x7f54580b32c0 2022-07-21T01:33:27.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f5456d9d700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545814b6f0 0x7f54580b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:27.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545659c700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54580aa040 0x7f545814c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:27.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544ffff700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580b32c0 0x7f5458148c60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5458149290 tx=0x7f5440006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:27.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544f7fe700 1 -- 172.21.15.3:0/737453906 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f544000c830 con 0x7f54580b32c0 2022-07-21T01:33:27.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544f7fe700 1 -- 172.21.15.3:0/737453906 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544000c990 con 0x7f54580b32c0 2022-07-21T01:33:27.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54580b14b0 con 0x7f54580b32c0 2022-07-21T01:33:27.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f544f7fe700 1 -- 172.21.15.3:0/737453906 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f544000ccf0 con 0x7f54580b32c0 2022-07-21T01:33:27.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.826+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54580b1de0 con 0x7f54580b32c0 2022-07-21T01:33:27.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.830+0000 7f544f7fe700 1 -- 172.21.15.3:0/737453906 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f544001c460 con 0x7f54580b32c0 2022-07-21T01:33:27.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.830+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54581485a0 con 0x7f54580b32c0 2022-07-21T01:33:27.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.830+0000 7f544f7fe700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5430053b30 0x7f5430055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:27.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.830+0000 7f544f7fe700 1 -- 172.21.15.3:0/737453906 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f544006c610 con 0x7f54580b32c0 2022-07-21T01:33:27.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.830+0000 7f545659c700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5430053b30 0x7f5430055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:27.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.830+0000 7f545659c700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5430053b30 0x7f5430055ff0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f5448018000 tx=0x7f5448014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:27.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.834+0000 7f544f7fe700 1 -- 172.21.15.3:0/737453906 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5440043260 con 0x7f54580b32c0 2022-07-21T01:33:27.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.962+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54580574e0 con 0x7f54580b32c0 2022-07-21T01:33:27.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.962+0000 7f544f7fe700 1 -- 172.21.15.3:0/737453906 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7f5440046880 con 0x7f54580b32c0 2022-07-21T01:33:27.961 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:27.961 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:27.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5430053b30 msgr2=0x7f5430055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5430053b30 0x7f5430055ff0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f5448018000 tx=0x7f5448014000).stop 2022-07-21T01:33:27.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580b32c0 msgr2=0x7f5458148c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580b32c0 0x7f5458148c60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5458149290 tx=0x7f5440006df0).stop 2022-07-21T01:33:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 shutdown_connections 2022-07-21T01:33:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545814b6f0 0x7f54580b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54580b32c0 0x7f5458148c60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5430053b30 0x7f5430055ff0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 --2- 172.21.15.3:0/737453906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54580aa040 0x7f545814c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:27.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 >> 172.21.15.3:0/737453906 conn(0x7f5458060950 msgr2=0x7f545814d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:27.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 shutdown_connections 2022-07-21T01:33:27.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:27.966+0000 7f545ca62700 1 -- 172.21.15.3:0/737453906 wait complete. 2022-07-21T01:33:27.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:27.978 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-07-21T01:33:28.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8668a34700 1 Processor -- start 2022-07-21T01:33:28.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8668a34700 1 -- start start 2022-07-21T01:33:28.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8668a34700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640b2710 0x7f866414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8668a34700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86640a8c50 0x7f866414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8668a34700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86640a7250 con 0x7f86640b01a0 2022-07-21T01:33:28.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8668a34700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86640a7480 con 0x7f86640a8c50 2022-07-21T01:33:28.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8668a34700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86640a7720 con 0x7f86640b2710 2022-07-21T01:33:28.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8662ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f86640b01a0 0x7f86640a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50918/0 (socket says 172.21.15.3:50918) 2022-07-21T01:33:28.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8662ffd700 1 -- 172.21.15.3:0/1346380629 learned_addr learned my addr 172.21.15.3:0/1346380629 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:28.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f86637fe700 1 --2- 172.21.15.3:0/1346380629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640b2710 0x7f866414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f8663fff700 1 --2- 172.21.15.3:0/1346380629 >> v2:172.21.15.143:3300/0 conn(0x7f86640a8c50 0x7f866414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f86637fe700 1 -- 172.21.15.3:0/1346380629 >> v2:172.21.15.143:3300/0 conn(0x7f86640a8c50 msgr2=0x7f866414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:28.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f86637fe700 1 --2- 172.21.15.3:0/1346380629 >> v2:172.21.15.143:3300/0 conn(0x7f86640a8c50 0x7f866414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f86637fe700 1 -- 172.21.15.3:0/1346380629 >> v1:172.21.15.3:6789/0 conn(0x7f86640b01a0 legacy=0x7f86640a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f86627fc700 1 -- 172.21.15.3:0/1346380629 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300406109 0 0) 0x7f86640a7250 con 0x7f86640b01a0 2022-07-21T01:33:28.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f86637fe700 1 -- 172.21.15.3:0/1346380629 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86640a79c0 con 0x7f86640b2710 2022-07-21T01:33:28.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.082+0000 7f86637fe700 1 --2- 172.21.15.3:0/1346380629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640b2710 0x7f866414afd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f865c0032f0 tx=0x7f865c0091d0).ready entity=mon.2 client_cookie=7641eb356a6c1f83 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f86627fc700 1 -- 172.21.15.3:0/1346380629 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f865c00b6c0 con 0x7f86640b2710 2022-07-21T01:33:28.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f86627fc700 1 -- 172.21.15.3:0/1346380629 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c00b820 con 0x7f86640b2710 2022-07-21T01:33:28.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f86627fc700 1 -- 172.21.15.3:0/1346380629 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f865c00b990 con 0x7f86640b2710 2022-07-21T01:33:28.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 -- 172.21.15.3:0/1346380629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640b2710 msgr2=0x7f866414afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 --2- 172.21.15.3:0/1346380629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640b2710 0x7f866414afd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f865c0032f0 tx=0x7f865c0091d0).stop 2022-07-21T01:33:28.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 -- 172.21.15.3:0/1346380629 shutdown_connections 2022-07-21T01:33:28.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 --2- 172.21.15.3:0/1346380629 >> v2:172.21.15.143:3300/0 conn(0x7f86640a8c50 0x7f866414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 --2- 172.21.15.3:0/1346380629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640b2710 0x7f866414afd0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 -- 172.21.15.3:0/1346380629 >> 172.21.15.3:0/1346380629 conn(0x7f8664060310 msgr2=0x7f86640b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:28.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 -- 172.21.15.3:0/1346380629 shutdown_connections 2022-07-21T01:33:28.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 -- 172.21.15.3:0/1346380629 wait complete. 2022-07-21T01:33:28.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 Processor -- start 2022-07-21T01:33:28.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.086+0000 7f8668a34700 1 -- start start 2022-07-21T01:33:28.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8668a34700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640a8c50 0x7f866414d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8668a34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86640b01a0 0x7f86640aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8668a34700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86640b2710 0x7f866414b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8668a34700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8664156740 con 0x7f86640b2710 2022-07-21T01:33:28.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8668a34700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86641569e0 con 0x7f86640b01a0 2022-07-21T01:33:28.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8668a34700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8664156c80 con 0x7f86640a8c50 2022-07-21T01:33:28.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8663fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86640b2710 0x7f866414b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8663fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86640b2710 0x7f866414b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42646/0 (socket says 172.21.15.3:42646) 2022-07-21T01:33:28.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8663fff700 1 -- 172.21.15.3:0/3090367626 learned_addr learned my addr 172.21.15.3:0/3090367626 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:28.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f86637fe700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640a8c50 0x7f866414d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8662ffd700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86640b01a0 0x7f86640aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8663fff700 1 -- 172.21.15.3:0/3090367626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640a8c50 msgr2=0x7f866414d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8663fff700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640a8c50 0x7f866414d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8663fff700 1 -- 172.21.15.3:0/3090367626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86640b01a0 msgr2=0x7f86640aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8663fff700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86640b01a0 0x7f86640aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8663fff700 1 -- 172.21.15.3:0/3090367626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c009470 con 0x7f86640b2710 2022-07-21T01:33:28.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8663fff700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86640b2710 0x7f866414b5c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f865400c000 tx=0x7f8654006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8660ff9700 1 -- 172.21.15.3:0/3090367626 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8654011750 con 0x7f86640b2710 2022-07-21T01:33:28.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8660ff9700 1 -- 172.21.15.3:0/3090367626 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86540118b0 con 0x7f86640b2710 2022-07-21T01:33:28.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8660ff9700 1 -- 172.21.15.3:0/3090367626 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8654011c10 con 0x7f86640b2710 2022-07-21T01:33:28.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8664156f20 con 0x7f86640b2710 2022-07-21T01:33:28.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.090+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8664157660 con 0x7f86640b2710 2022-07-21T01:33:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.094+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86640574e0 con 0x7f86640b2710 2022-07-21T01:33:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.094+0000 7f8660ff9700 1 -- 172.21.15.3:0/3090367626 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8654011d70 con 0x7f86640b2710 2022-07-21T01:33:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.094+0000 7f8660ff9700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8644053b30 0x7f8644055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.094+0000 7f8660ff9700 1 -- 172.21.15.3:0/3090367626 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f865406c490 con 0x7f86640b2710 2022-07-21T01:33:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.094+0000 7f86637fe700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8644053b30 0x7f8644055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.094+0000 7f86637fe700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8644053b30 0x7f8644055ff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f865c01b000 tx=0x7f865c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.102+0000 7f8660ff9700 1 -- 172.21.15.3:0/3090367626 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8654043140 con 0x7f86640b2710 2022-07-21T01:33:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.222+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f866406a5e0 con 0x7f86640b2710 2022-07-21T01:33:28.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8660ff9700 1 -- 172.21.15.3:0/3090367626 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7f8654046760 con 0x7f86640b2710 2022-07-21T01:33:28.224 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:28.224 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:28.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8644053b30 msgr2=0x7f8644055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8644053b30 0x7f8644055ff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f865c01b000 tx=0x7f865c016000).stop 2022-07-21T01:33:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86640b2710 msgr2=0x7f866414b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86640b2710 0x7f866414b5c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f865400c000 tx=0x7f8654006df0).stop 2022-07-21T01:33:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 shutdown_connections 2022-07-21T01:33:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86640b2710 0x7f866414b5c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8644053b30 0x7f8644055ff0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86640b01a0 0x7f86640aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 --2- 172.21.15.3:0/3090367626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86640a8c50 0x7f866414d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.226+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 >> 172.21.15.3:0/3090367626 conn(0x7f8664060310 msgr2=0x7f8664146030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:28.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.230+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 shutdown_connections 2022-07-21T01:33:28.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.230+0000 7f8668a34700 1 -- 172.21.15.3:0/3090367626 wait complete. 2022-07-21T01:33:28.237 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:33:21.299936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:33:24.075599+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:33:28.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:33:28.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.358+0000 7fdfa5b76700 1 Processor -- start 2022-07-21T01:33:28.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.358+0000 7fdfa5b76700 1 -- start start 2022-07-21T01:33:28.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdfa00aaf10 0x7fdfa0067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdfa00b4ee0 0x7fdfa00b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa0065af0 con 0x7fdfa00ab2f0 2022-07-21T01:33:28.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa00650a0 con 0x7fdfa00b4ee0 2022-07-21T01:33:28.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa00b3e70 con 0x7fdfa00aaf10 2022-07-21T01:33:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5375700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdfa00b4ee0 0x7fdfa00b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5375700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdfa00b4ee0 0x7fdfa00b76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42654/0 (socket says 172.21.15.3:42654) 2022-07-21T01:33:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5375700 1 -- 172.21.15.3:0/724905169 learned_addr learned my addr 172.21.15.3:0/724905169 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa4b74700 1 --2- 172.21.15.3:0/724905169 >> v2:172.21.15.143:3300/0 conn(0x7fdfa00aaf10 0x7fdfa0067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5375700 1 -- 172.21.15.3:0/724905169 >> v2:172.21.15.143:3300/0 conn(0x7fdfa00aaf10 msgr2=0x7fdfa0067490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5375700 1 --2- 172.21.15.3:0/724905169 >> v2:172.21.15.143:3300/0 conn(0x7fdfa00aaf10 0x7fdfa0067490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5375700 1 -- 172.21.15.3:0/724905169 >> v1:172.21.15.143:6789/0 conn(0x7fdfa00ab2f0 legacy=0x7fdfa0064990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5375700 1 -- 172.21.15.3:0/724905169 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa0066ff0 con 0x7fdfa00b4ee0 2022-07-21T01:33:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5375700 1 --2- 172.21.15.3:0/724905169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 0x7fdfa00b76b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fdf980032f0 tx=0x7fdf98009290).ready entity=mon.0 client_cookie=9e434c16af663188 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdf9f7fe700 1 -- 172.21.15.3:0/724905169 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf9800b870 con 0x7fdfa00b4ee0 2022-07-21T01:33:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdf9f7fe700 1 -- 172.21.15.3:0/724905169 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9800b9d0 con 0x7fdfa00b4ee0 2022-07-21T01:33:28.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdf9f7fe700 1 -- 172.21.15.3:0/724905169 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf9800bb40 con 0x7fdfa00b4ee0 2022-07-21T01:33:28.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 -- 172.21.15.3:0/724905169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 msgr2=0x7fdfa00b76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 --2- 172.21.15.3:0/724905169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 0x7fdfa00b76b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fdf980032f0 tx=0x7fdf98009290).stop 2022-07-21T01:33:28.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 -- 172.21.15.3:0/724905169 shutdown_connections 2022-07-21T01:33:28.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 --2- 172.21.15.3:0/724905169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 0x7fdfa00b76b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 --2- 172.21.15.3:0/724905169 >> v2:172.21.15.143:3300/0 conn(0x7fdfa00aaf10 0x7fdfa0067490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 -- 172.21.15.3:0/724905169 >> 172.21.15.3:0/724905169 conn(0x7fdfa0062d70 msgr2=0x7fdfa005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:28.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.362+0000 7fdfa5b76700 1 -- 172.21.15.3:0/724905169 shutdown_connections 2022-07-21T01:33:28.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5b76700 1 -- 172.21.15.3:0/724905169 wait complete. 2022-07-21T01:33:28.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5b76700 1 Processor -- start 2022-07-21T01:33:28.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5b76700 1 -- start start 2022-07-21T01:33:28.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5b76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00aaf10 0x7fdfa0150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5b76700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00ab2f0 0x7fdfa00aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5b76700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 0x7fdfa0142c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5b76700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa0142390 con 0x7fdfa00b4ee0 2022-07-21T01:33:28.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5b76700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa0150d20 con 0x7fdfa00ab2f0 2022-07-21T01:33:28.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5b76700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa01427c0 con 0x7fdfa00aaf10 2022-07-21T01:33:28.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5375700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 0x7fdfa0142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdf9ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00ab2f0 0x7fdfa00aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa4b74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00aaf10 0x7fdfa0150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5375700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 0x7fdfa0142c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42660/0 (socket says 172.21.15.3:42660) 2022-07-21T01:33:28.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdfa5375700 1 -- 172.21.15.3:0/2586892094 learned_addr learned my addr 172.21.15.3:0/2586892094 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:28.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.366+0000 7fdf9ffff700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00ab2f0 0x7fdfa00aee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:48106/0 (socket says 172.21.15.3:48106) 2022-07-21T01:33:28.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdfa4b74700 1 -- 172.21.15.3:0/2586892094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00ab2f0 msgr2=0x7fdfa00aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdfa4b74700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00ab2f0 0x7fdfa00aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdfa4b74700 1 -- 172.21.15.3:0/2586892094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 msgr2=0x7fdfa0142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdfa4b74700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 0x7fdfa0142c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdfa4b74700 1 -- 172.21.15.3:0/2586892094 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf98009530 con 0x7fdfa00aaf10 2022-07-21T01:33:28.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdfa4b74700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00aaf10 0x7fdfa0150700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fdfa0145010 tx=0x7fdf90006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdf9dffb700 1 -- 172.21.15.3:0/2586892094 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf90010780 con 0x7fdfa00aaf10 2022-07-21T01:33:28.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdf9dffb700 1 -- 172.21.15.3:0/2586892094 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf900108e0 con 0x7fdfa00aaf10 2022-07-21T01:33:28.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa014dd60 con 0x7fdfa00aaf10 2022-07-21T01:33:28.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdf9dffb700 1 -- 172.21.15.3:0/2586892094 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf90010c40 con 0x7fdfa00aaf10 2022-07-21T01:33:28.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa01450c0 con 0x7fdfa00aaf10 2022-07-21T01:33:28.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa00ae0b0 con 0x7fdfa00aaf10 2022-07-21T01:33:28.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdf9dffb700 1 -- 172.21.15.3:0/2586892094 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdf9001c460 con 0x7fdfa00aaf10 2022-07-21T01:33:28.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.370+0000 7fdf9dffb700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdf840539e0 0x7fdf84055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.374+0000 7fdf9ffff700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdf840539e0 0x7fdf84055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.374+0000 7fdf9dffb700 1 -- 172.21.15.3:0/2586892094 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fdf9006c330 con 0x7fdfa00aaf10 2022-07-21T01:33:28.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.374+0000 7fdf9ffff700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdf840539e0 0x7fdf84055ea0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdf8800d000 tx=0x7fdf88007b60).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.382+0000 7fdf9dffb700 1 -- 172.21.15.3:0/2586892094 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdf90042f90 con 0x7fdfa00aaf10 2022-07-21T01:33:28.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fdfa00574e0 con 0x7fdfa00aaf10 2022-07-21T01:33:28.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdf9dffb700 1 -- 172.21.15.3:0/2586892094 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v26) v1 ==== 110+0+10 (secure 0 0 0) 0x7fdf900465b0 con 0x7fdfa00aaf10 2022-07-21T01:33:28.507 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:33:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdf840539e0 msgr2=0x7fdf84055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdf840539e0 0x7fdf84055ea0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdf8800d000 tx=0x7fdf88007b60).stop 2022-07-21T01:33:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00aaf10 msgr2=0x7fdfa0150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00aaf10 0x7fdfa0150700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fdfa0145010 tx=0x7fdf90006df0).stop 2022-07-21T01:33:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 shutdown_connections 2022-07-21T01:33:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdfa00b4ee0 0x7fdfa0142c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdf840539e0 0x7fdf84055ea0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00ab2f0 0x7fdfa00aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 --2- 172.21.15.3:0/2586892094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00aaf10 0x7fdfa0150700 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 >> 172.21.15.3:0/2586892094 conn(0x7fdfa0062d70 msgr2=0x7fdfa01415b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.510+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 shutdown_connections 2022-07-21T01:33:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.514+0000 7fdfa5b76700 1 -- 172.21.15.3:0/2586892094 wait complete. 2022-07-21T01:33:28.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:33:28.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.634+0000 7fb76b454700 1 Processor -- start 2022-07-21T01:33:28.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.634+0000 7fb76b454700 1 -- start start 2022-07-21T01:33:28.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb7640a9da0 0x7fb764060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb7641402d0 0x7fb764140720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb764067030 con 0x7fb7640b31d0 2022-07-21T01:33:28.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb764064a70 con 0x7fb7641402d0 2022-07-21T01:33:28.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb764061f60 con 0x7fb7640a9da0 2022-07-21T01:33:28.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76ac53700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb7641402d0 0x7fb764140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76ac53700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb7641402d0 0x7fb764140720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42668/0 (socket says 172.21.15.3:42668) 2022-07-21T01:33:28.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76ac53700 1 -- 172.21.15.3:0/3180355145 learned_addr learned my addr 172.21.15.3:0/3180355145 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:28.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76a452700 1 --1- 172.21.15.3:0/3180355145 >> v1:172.21.15.3:6789/0 conn(0x7fb7640b31d0 0x7fb764066020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50946/0 (socket says 172.21.15.3:50946) 2022-07-21T01:33:28.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb769c51700 1 --2- 172.21.15.3:0/3180355145 >> v2:172.21.15.143:3300/0 conn(0x7fb7640a9da0 0x7fb764060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76ac53700 1 -- 172.21.15.3:0/3180355145 >> v2:172.21.15.143:3300/0 conn(0x7fb7640a9da0 msgr2=0x7fb764060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:28.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76ac53700 1 --2- 172.21.15.3:0/3180355145 >> v2:172.21.15.143:3300/0 conn(0x7fb7640a9da0 0x7fb764060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76ac53700 1 -- 172.21.15.3:0/3180355145 >> v1:172.21.15.3:6789/0 conn(0x7fb7640b31d0 legacy=0x7fb764066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76ac53700 1 -- 172.21.15.3:0/3180355145 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7640b35b0 con 0x7fb7641402d0 2022-07-21T01:33:28.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb769c51700 1 --2- 172.21.15.3:0/3180355145 >> v2:172.21.15.143:3300/0 conn(0x7fb7640a9da0 0x7fb764060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:28.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76ac53700 1 --2- 172.21.15.3:0/3180355145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7641402d0 0x7fb764140720 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb75c0032f0 tx=0x7fb75c00b530).ready entity=mon.0 client_cookie=63aecb9a42b85702 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb769450700 1 -- 172.21.15.3:0/3180355145 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75c00be70 con 0x7fb7641402d0 2022-07-21T01:33:28.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb769450700 1 -- 172.21.15.3:0/3180355145 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c010470 con 0x7fb7641402d0 2022-07-21T01:33:28.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb769450700 1 -- 172.21.15.3:0/3180355145 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75c0107d0 con 0x7fb7641402d0 2022-07-21T01:33:28.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 -- 172.21.15.3:0/3180355145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7641402d0 msgr2=0x7fb764140720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 --2- 172.21.15.3:0/3180355145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7641402d0 0x7fb764140720 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb75c0032f0 tx=0x7fb75c00b530).stop 2022-07-21T01:33:28.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 -- 172.21.15.3:0/3180355145 shutdown_connections 2022-07-21T01:33:28.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 --2- 172.21.15.3:0/3180355145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7641402d0 0x7fb764140720 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 --2- 172.21.15.3:0/3180355145 >> v2:172.21.15.143:3300/0 conn(0x7fb7640a9da0 0x7fb764060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 -- 172.21.15.3:0/3180355145 >> 172.21.15.3:0/3180355145 conn(0x7fb764063870 msgr2=0x7fb764063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:28.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 -- 172.21.15.3:0/3180355145 shutdown_connections 2022-07-21T01:33:28.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.638+0000 7fb76b454700 1 -- 172.21.15.3:0/3180355145 wait complete. 2022-07-21T01:33:28.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76b454700 1 Processor -- start 2022-07-21T01:33:28.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76b454700 1 -- start start 2022-07-21T01:33:28.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76b454700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7640a9da0 0x7fb76413d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76b454700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7640b31d0 0x7fb7640abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76b454700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7641402d0 0x7fb7640ac280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76b454700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640aeb10 con 0x7fb7640a9da0 2022-07-21T01:33:28.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76b454700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640ade80 con 0x7fb7640b31d0 2022-07-21T01:33:28.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76b454700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640af7f0 con 0x7fb7641402d0 2022-07-21T01:33:28.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76a452700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7640a9da0 0x7fb76413d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76a452700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7640a9da0 0x7fb76413d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42670/0 (socket says 172.21.15.3:42670) 2022-07-21T01:33:28.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76a452700 1 -- 172.21.15.3:0/3170337407 learned_addr learned my addr 172.21.15.3:0/3170337407 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:28.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb769c51700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7640b31d0 0x7fb7640abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76ac53700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7641402d0 0x7fb7640ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76a452700 1 -- 172.21.15.3:0/3170337407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7641402d0 msgr2=0x7fb7640ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76a452700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7641402d0 0x7fb7640ac280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76a452700 1 -- 172.21.15.3:0/3170337407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7640b31d0 msgr2=0x7fb7640abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76a452700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7640b31d0 0x7fb7640abd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.642+0000 7fb76a452700 1 -- 172.21.15.3:0/3170337407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c00b7d0 con 0x7fb7640a9da0 2022-07-21T01:33:28.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb76a452700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7640a9da0 0x7fb76413d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb7640adfc0 tx=0x7fb760007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb7537fe700 1 -- 172.21.15.3:0/3170337407 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7600070d0 con 0x7fb7640a9da0 2022-07-21T01:33:28.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb7537fe700 1 -- 172.21.15.3:0/3170337407 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb760007230 con 0x7fb7640a9da0 2022-07-21T01:33:28.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7640afa90 con 0x7fb7640a9da0 2022-07-21T01:33:28.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb7537fe700 1 -- 172.21.15.3:0/3170337407 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb760005740 con 0x7fb7640a9da0 2022-07-21T01:33:28.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7640ae670 con 0x7fb7640a9da0 2022-07-21T01:33:28.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7640aee80 con 0x7fb7640a9da0 2022-07-21T01:33:28.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb7537fe700 1 -- 172.21.15.3:0/3170337407 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb760010070 con 0x7fb7640a9da0 2022-07-21T01:33:28.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb7537fe700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7440539e0 0x7fb744055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb769c51700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7440539e0 0x7fb744055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb7537fe700 1 -- 172.21.15.3:0/3170337407 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb76006b5f0 con 0x7fb7640a9da0 2022-07-21T01:33:28.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.646+0000 7fb769c51700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7440539e0 0x7fb744055ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb75400c000 tx=0x7fb754008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.654+0000 7fb7537fe700 1 -- 172.21.15.3:0/3170337407 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb760042330 con 0x7fb7640a9da0 2022-07-21T01:33:28.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb764066af0 con 0x7fb7640a9da0 2022-07-21T01:33:28.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb7537fe700 1 -- 172.21.15.3:0/3170337407 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb760045950 con 0x7fb7640a9da0 2022-07-21T01:33:28.775 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:28.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7440539e0 msgr2=0x7fb744055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7440539e0 0x7fb744055ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb75400c000 tx=0x7fb754008040).stop 2022-07-21T01:33:28.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7640a9da0 msgr2=0x7fb76413d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7640a9da0 0x7fb76413d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb7640adfc0 tx=0x7fb760007c00).stop 2022-07-21T01:33:28.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 shutdown_connections 2022-07-21T01:33:28.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7641402d0 0x7fb7640ac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7440539e0 0x7fb744055ea0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7640b31d0 0x7fb7640abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 --2- 172.21.15.3:0/3170337407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7640a9da0 0x7fb76413d390 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 >> 172.21.15.3:0/3170337407 conn(0x7fb764063870 msgr2=0x7fb7640b18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:28.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 shutdown_connections 2022-07-21T01:33:28.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.778+0000 7fb76b454700 1 -- 172.21.15.3:0/3170337407 wait complete. 2022-07-21T01:33:28.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:33:28.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e2856700 1 Processor -- start 2022-07-21T01:33:28.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e2856700 1 -- start start 2022-07-21T01:33:28.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e2856700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61dc05fd90 0x7f61dc0b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e2856700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 0x7f61dc0b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e2856700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61dc1440f0 con 0x7f61dc0b66f0 2022-07-21T01:33:28.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e2856700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61dc0ba840 con 0x7f61dc05fd90 2022-07-21T01:33:28.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e2856700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc060170 con 0x7f61dc0b5b20 2022-07-21T01:33:28.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61db7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f61dc0b66f0 0x7f61dc0ba130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50960/0 (socket says 172.21.15.3:50960) 2022-07-21T01:33:28.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61db7fe700 1 -- 172.21.15.3:0/1873127784 learned_addr learned my addr 172.21.15.3:0/1873127784 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:28.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61dbfff700 1 --2- 172.21.15.3:0/1873127784 >> v2:172.21.15.143:3300/0 conn(0x7f61dc05fd90 0x7f61dc0b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e0df3700 1 --2- 172.21.15.3:0/1873127784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 0x7f61dc0b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61daffd700 1 -- 172.21.15.3:0/1873127784 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11798742 0 0) 0x7f61dc1440f0 con 0x7f61dc0b66f0 2022-07-21T01:33:28.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61daffd700 1 -- 172.21.15.3:0/1873127784 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61bc003260 con 0x7f61dc0b66f0 2022-07-21T01:33:28.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e0df3700 1 -- 172.21.15.3:0/1873127784 >> v2:172.21.15.143:3300/0 conn(0x7f61dc05fd90 msgr2=0x7f61dc0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:28.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e0df3700 1 --2- 172.21.15.3:0/1873127784 >> v2:172.21.15.143:3300/0 conn(0x7f61dc05fd90 0x7f61dc0b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e0df3700 1 -- 172.21.15.3:0/1873127784 >> v1:172.21.15.3:6789/0 conn(0x7f61dc0b66f0 legacy=0x7f61dc0ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e0df3700 1 -- 172.21.15.3:0/1873127784 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61dc143be0 con 0x7f61dc0b5b20 2022-07-21T01:33:28.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61e0df3700 1 --2- 172.21.15.3:0/1873127784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 0x7f61dc0b4b90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f61d4002f70 tx=0x7f61d4006df0).ready entity=mon.2 client_cookie=84570dbecd34c7db server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.898+0000 7f61daffd700 1 -- 172.21.15.3:0/1873127784 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61d400d4a0 con 0x7f61dc0b5b20 2022-07-21T01:33:28.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61daffd700 1 -- 172.21.15.3:0/1873127784 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d4000e90 con 0x7f61dc0b5b20 2022-07-21T01:33:28.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61daffd700 1 -- 172.21.15.3:0/1873127784 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61d400d7f0 con 0x7f61dc0b5b20 2022-07-21T01:33:28.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 -- 172.21.15.3:0/1873127784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 msgr2=0x7f61dc0b4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 --2- 172.21.15.3:0/1873127784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 0x7f61dc0b4b90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f61d4002f70 tx=0x7f61d4006df0).stop 2022-07-21T01:33:28.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 -- 172.21.15.3:0/1873127784 shutdown_connections 2022-07-21T01:33:28.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 --2- 172.21.15.3:0/1873127784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 0x7f61dc0b4b90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 --2- 172.21.15.3:0/1873127784 >> v2:172.21.15.143:3300/0 conn(0x7f61dc05fd90 0x7f61dc0b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 -- 172.21.15.3:0/1873127784 >> 172.21.15.3:0/1873127784 conn(0x7f61dc062530 msgr2=0x7f61dc066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:28.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 -- 172.21.15.3:0/1873127784 shutdown_connections 2022-07-21T01:33:28.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 -- 172.21.15.3:0/1873127784 wait complete. 2022-07-21T01:33:28.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 Processor -- start 2022-07-21T01:33:28.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 -- start start 2022-07-21T01:33:28.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc05fd90 0x7f61dc1448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 0x7f61dc146440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61dc0b66f0 0x7f61dc146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc0b3a30 con 0x7f61dc0b66f0 2022-07-21T01:33:28.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc0b04a0 con 0x7f61dc05fd90 2022-07-21T01:33:28.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e2856700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc0b0740 con 0x7f61dc0b5b20 2022-07-21T01:33:28.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.902+0000 7f61e0df3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61dc0b66f0 0x7f61dc146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e0df3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61dc0b66f0 0x7f61dc146930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42688/0 (socket says 172.21.15.3:42688) 2022-07-21T01:33:28.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e0df3700 1 -- 172.21.15.3:0/398863699 learned_addr learned my addr 172.21.15.3:0/398863699 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:28.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61dbfff700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc05fd90 0x7f61dc1448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61db7fe700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 0x7f61dc146440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e0df3700 1 -- 172.21.15.3:0/398863699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 msgr2=0x7f61dc146440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e0df3700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 0x7f61dc146440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e0df3700 1 -- 172.21.15.3:0/398863699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc05fd90 msgr2=0x7f61dc1448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:28.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e0df3700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc05fd90 0x7f61dc1448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:28.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e0df3700 1 -- 172.21.15.3:0/398863699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d400b180 con 0x7f61dc0b66f0 2022-07-21T01:33:28.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e0df3700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61dc0b66f0 0x7f61dc146930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f61d4002f70 tx=0x7f61d4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61d97fa700 1 -- 172.21.15.3:0/398863699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61d4016070 con 0x7f61dc0b66f0 2022-07-21T01:33:28.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61d97fa700 1 -- 172.21.15.3:0/398863699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d4013850 con 0x7f61dc0b66f0 2022-07-21T01:33:28.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61d97fa700 1 -- 172.21.15.3:0/398863699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61d4013bb0 con 0x7f61dc0b66f0 2022-07-21T01:33:28.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61dc0af740 con 0x7f61dc0b66f0 2022-07-21T01:33:28.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61dc0afe80 con 0x7f61dc0b66f0 2022-07-21T01:33:28.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.906+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61dc143e50 con 0x7f61dc0b66f0 2022-07-21T01:33:28.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.910+0000 7f61d97fa700 1 -- 172.21.15.3:0/398863699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f61d4025030 con 0x7f61dc0b66f0 2022-07-21T01:33:28.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.910+0000 7f61d97fa700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c0053a70 0x7f61c0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:28.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.910+0000 7f61d97fa700 1 -- 172.21.15.3:0/398863699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f61d406d170 con 0x7f61dc0b66f0 2022-07-21T01:33:28.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.910+0000 7f61dbfff700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c0053a70 0x7f61c0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:28.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.910+0000 7f61dbfff700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c0053a70 0x7f61c0055f30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f61cc00b000 tx=0x7f61cc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:28.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:28.918+0000 7f61d97fa700 1 -- 172.21.15.3:0/398863699 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f61d4043da0 con 0x7f61dc0b66f0 2022-07-21T01:33:29.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.042+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f61dc0574e0 con 0x7f61dc0b66f0 2022-07-21T01:33:29.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.042+0000 7f61d97fa700 1 -- 172.21.15.3:0/398863699 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+10 (secure 0 0 0) 0x7f61d40230a0 con 0x7f61dc0b66f0 2022-07-21T01:33:29.039 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:29.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c0053a70 msgr2=0x7f61c0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c0053a70 0x7f61c0055f30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f61cc00b000 tx=0x7f61cc006df0).stop 2022-07-21T01:33:29.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61dc0b66f0 msgr2=0x7f61dc146930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61dc0b66f0 0x7f61dc146930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f61d4002f70 tx=0x7f61d4014000).stop 2022-07-21T01:33:29.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 shutdown_connections 2022-07-21T01:33:29.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61dc0b66f0 0x7f61dc146930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61dc0b5b20 0x7f61dc146440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c0053a70 0x7f61c0055f30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 --2- 172.21.15.3:0/398863699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc05fd90 0x7f61dc1448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 >> 172.21.15.3:0/398863699 conn(0x7f61dc062530 msgr2=0x7f61dc0b8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 shutdown_connections 2022-07-21T01:33:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.046+0000 7f61e2856700 1 -- 172.21.15.3:0/398863699 wait complete. 2022-07-21T01:33:29.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:33:29.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f49164ce700 1 Processor -- start 2022-07-21T01:33:29.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f49164ce700 1 -- start start 2022-07-21T01:33:29.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f49164ce700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100b38a0 0x7f49100b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f49164ce700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f49100a9da0 0x7f49100b1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f49164ce700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4910143ab0 con 0x7f49100b0740 2022-07-21T01:33:29.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f49164ce700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4910143d50 con 0x7f49100a9da0 2022-07-21T01:33:29.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f49164ce700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4910143ff0 con 0x7f49100b38a0 2022-07-21T01:33:29.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f49100b0740 0x7f491013d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50972/0 (socket says 172.21.15.3:50972) 2022-07-21T01:33:29.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490ffff700 1 -- 172.21.15.3:0/316829005 learned_addr learned my addr 172.21.15.3:0/316829005 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:29.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f4914a6b700 1 --2- 172.21.15.3:0/316829005 >> v2:172.21.15.143:3300/0 conn(0x7f49100a9da0 0x7f49100b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490f7fe700 1 --2- 172.21.15.3:0/316829005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100b38a0 0x7f49100b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490effd700 1 -- 172.21.15.3:0/316829005 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564275341 0 0) 0x7f4910143ab0 con 0x7f49100b0740 2022-07-21T01:33:29.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490effd700 1 -- 172.21.15.3:0/316829005 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48fc003260 con 0x7f49100b0740 2022-07-21T01:33:29.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490f7fe700 1 -- 172.21.15.3:0/316829005 >> v2:172.21.15.143:3300/0 conn(0x7f49100a9da0 msgr2=0x7f49100b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:29.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490f7fe700 1 --2- 172.21.15.3:0/316829005 >> v2:172.21.15.143:3300/0 conn(0x7f49100a9da0 0x7f49100b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490effd700 1 -- 172.21.15.3:0/316829005 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845208690 0 0) 0x7f48fc003260 con 0x7f49100b0740 2022-07-21T01:33:29.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490f7fe700 1 -- 172.21.15.3:0/316829005 >> v1:172.21.15.3:6789/0 conn(0x7f49100b0740 legacy=0x7f491013d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490f7fe700 1 -- 172.21.15.3:0/316829005 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4910144290 con 0x7f49100b38a0 2022-07-21T01:33:29.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f4914a6b700 1 --2- 172.21.15.3:0/316829005 >> v2:172.21.15.143:3300/0 conn(0x7f49100a9da0 0x7f49100b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:29.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.170+0000 7f490f7fe700 1 --2- 172.21.15.3:0/316829005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100b38a0 0x7f49100b0b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4904002f70 tx=0x7f4904006df0).ready entity=mon.2 client_cookie=fd473829a8579014 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:29.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f490effd700 1 -- 172.21.15.3:0/316829005 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490400d4a0 con 0x7f49100b38a0 2022-07-21T01:33:29.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f490effd700 1 -- 172.21.15.3:0/316829005 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4904000e90 con 0x7f49100b38a0 2022-07-21T01:33:29.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f490effd700 1 -- 172.21.15.3:0/316829005 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490400d7f0 con 0x7f49100b38a0 2022-07-21T01:33:29.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 -- 172.21.15.3:0/316829005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100b38a0 msgr2=0x7f49100b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 --2- 172.21.15.3:0/316829005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100b38a0 0x7f49100b0b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4904002f70 tx=0x7f4904006df0).stop 2022-07-21T01:33:29.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 -- 172.21.15.3:0/316829005 shutdown_connections 2022-07-21T01:33:29.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 --2- 172.21.15.3:0/316829005 >> v2:172.21.15.143:3300/0 conn(0x7f49100a9da0 0x7f49100b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 --2- 172.21.15.3:0/316829005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100b38a0 0x7f49100b0b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 -- 172.21.15.3:0/316829005 >> 172.21.15.3:0/316829005 conn(0x7f4910060950 msgr2=0x7f4910060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:29.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 -- 172.21.15.3:0/316829005 shutdown_connections 2022-07-21T01:33:29.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 -- 172.21.15.3:0/316829005 wait complete. 2022-07-21T01:33:29.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 Processor -- start 2022-07-21T01:33:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 -- start start 2022-07-21T01:33:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100a9da0 0x7f491013ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49100b0740 0x7f4910141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49100b38a0 0x7f4910152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910142a70 con 0x7f49100b0740 2022-07-21T01:33:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491014fb90 con 0x7f49100b38a0 2022-07-21T01:33:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f49164ce700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491014fdb0 con 0x7f49100a9da0 2022-07-21T01:33:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.174+0000 7f490f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49100b0740 0x7f4910141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49100b0740 0x7f4910141fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42700/0 (socket says 172.21.15.3:42700) 2022-07-21T01:33:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490f7fe700 1 -- 172.21.15.3:0/4064211922 learned_addr learned my addr 172.21.15.3:0/4064211922 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f4914a6b700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49100b38a0 0x7f4910152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490ffff700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100a9da0 0x7f491013ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490f7fe700 1 -- 172.21.15.3:0/4064211922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100a9da0 msgr2=0x7f491013ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490f7fe700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100a9da0 0x7f491013ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490f7fe700 1 -- 172.21.15.3:0/4064211922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49100b38a0 msgr2=0x7f4910152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490f7fe700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49100b38a0 0x7f4910152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490f7fe700 1 -- 172.21.15.3:0/4064211922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4908009470 con 0x7f49100b0740 2022-07-21T01:33:29.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490ffff700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100a9da0 0x7f491013ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:29.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490f7fe700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49100b0740 0x7f4910141fd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f491014f250 tx=0x7f4904014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:29.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490d7fa700 1 -- 172.21.15.3:0/4064211922 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4904016070 con 0x7f49100b0740 2022-07-21T01:33:29.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490d7fa700 1 -- 172.21.15.3:0/4064211922 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4904013780 con 0x7f49100b0740 2022-07-21T01:33:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490400b180 con 0x7f49100b0740 2022-07-21T01:33:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490d7fa700 1 -- 172.21.15.3:0/4064211922 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4904013b00 con 0x7f49100b0740 2022-07-21T01:33:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491013e400 con 0x7f49100b0740 2022-07-21T01:33:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49100af050 con 0x7f49100b0740 2022-07-21T01:33:29.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.178+0000 7f490d7fa700 1 -- 172.21.15.3:0/4064211922 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f490400f030 con 0x7f49100b0740 2022-07-21T01:33:29.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.182+0000 7f490d7fa700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48f0053980 0x7f48f0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.182+0000 7f490d7fa700 1 -- 172.21.15.3:0/4064211922 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f490406cf70 con 0x7f49100b0740 2022-07-21T01:33:29.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.182+0000 7f490ffff700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48f0053980 0x7f48f0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.182+0000 7f490ffff700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48f0053980 0x7f48f0055e40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f490000b000 tx=0x7f4900001660).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:29.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.186+0000 7f490d7fa700 1 -- 172.21.15.3:0/4064211922 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4904043d40 con 0x7f49100b0740 2022-07-21T01:33:29.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.314+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f491013ddf0 con 0x7f49100b0740 2022-07-21T01:33:29.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.314+0000 7f490d7fa700 1 -- 172.21.15.3:0/4064211922 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4904023020 con 0x7f49100b0740 2022-07-21T01:33:29.403 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:29.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.314+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48f0053980 msgr2=0x7f48f0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.314+0000 7f49164ce700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48f0053980 0x7f48f0055e40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f490000b000 tx=0x7f4900001660).stop 2022-07-21T01:33:29.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.314+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49100b0740 msgr2=0x7f4910141fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.314+0000 7f49164ce700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49100b0740 0x7f4910141fd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f491014f250 tx=0x7f4904014000).stop 2022-07-21T01:33:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.318+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 shutdown_connections 2022-07-21T01:33:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.318+0000 7f49164ce700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48f0053980 0x7f48f0055e40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.318+0000 7f49164ce700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49100b38a0 0x7f4910152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.318+0000 7f49164ce700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49100b0740 0x7f4910141fd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.318+0000 7f49164ce700 1 --2- 172.21.15.3:0/4064211922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49100a9da0 0x7f491013ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.318+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 >> 172.21.15.3:0/4064211922 conn(0x7f4910060950 msgr2=0x7f491013b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:29.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.318+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 shutdown_connections 2022-07-21T01:33:29.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.318+0000 7f49164ce700 1 -- 172.21.15.3:0/4064211922 wait complete. 2022-07-21T01:33:29.409 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-07-21T01:33:29.410 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2022-07-21T01:33:29.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2022-07-21T01:33:29.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 Processor -- start 2022-07-21T01:33:29.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 -- start start 2022-07-21T01:33:29.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b6710 0x7f3cb80b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3cb80b6af0 0x7f3cb812bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb8144240 con 0x7f3cb80b4550 2022-07-21T01:33:29.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb8143d00 con 0x7f3cb80b6af0 2022-07-21T01:33:29.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8143fa0 con 0x7f3cb80b6710 2022-07-21T01:33:29.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbdcc5700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3cb80b4550 0x7f3cb8061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50986/0 (socket says 172.21.15.3:50986) 2022-07-21T01:33:29.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbdcc5700 1 -- 172.21.15.3:0/400786569 learned_addr learned my addr 172.21.15.3:0/400786569 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:29.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbecc7700 1 --2- 172.21.15.3:0/400786569 >> v2:172.21.15.143:3300/0 conn(0x7f3cb80b6af0 0x7f3cb812bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbe4c6700 1 --2- 172.21.15.3:0/400786569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b6710 0x7f3cb80b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbd4c4700 1 -- 172.21.15.3:0/400786569 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388228518 0 0) 0x7f3cb8144240 con 0x7f3cb80b4550 2022-07-21T01:33:29.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbd4c4700 1 -- 172.21.15.3:0/400786569 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca0003260 con 0x7f3cb80b4550 2022-07-21T01:33:29.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbecc7700 1 -- 172.21.15.3:0/400786569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b6710 msgr2=0x7f3cb80b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:29.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbecc7700 1 --2- 172.21.15.3:0/400786569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b6710 0x7f3cb80b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbecc7700 1 -- 172.21.15.3:0/400786569 >> v1:172.21.15.3:6789/0 conn(0x7f3cb80b4550 legacy=0x7f3cb8061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbecc7700 1 -- 172.21.15.3:0/400786569 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8061780 con 0x7f3cb80b6af0 2022-07-21T01:33:29.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbecc7700 1 --2- 172.21.15.3:0/400786569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb80b6af0 0x7f3cb812bb90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f3cb00032f0 tx=0x7f3cb00091d0).ready entity=mon.1 client_cookie=5e77d12e9eaeec16 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:29.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbd4c4700 1 -- 172.21.15.3:0/400786569 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cb000b600 con 0x7f3cb80b6af0 2022-07-21T01:33:29.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbd4c4700 1 -- 172.21.15.3:0/400786569 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb000b760 con 0x7f3cb80b6af0 2022-07-21T01:33:29.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cbd4c4700 1 -- 172.21.15.3:0/400786569 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cb000b8d0 con 0x7f3cb80b6af0 2022-07-21T01:33:29.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 -- 172.21.15.3:0/400786569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb80b6af0 msgr2=0x7f3cb812bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 --2- 172.21.15.3:0/400786569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb80b6af0 0x7f3cb812bb90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f3cb00032f0 tx=0x7f3cb00091d0).stop 2022-07-21T01:33:29.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 -- 172.21.15.3:0/400786569 shutdown_connections 2022-07-21T01:33:29.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 --2- 172.21.15.3:0/400786569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb80b6af0 0x7f3cb812bb90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 --2- 172.21.15.3:0/400786569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b6710 0x7f3cb80b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 -- 172.21.15.3:0/400786569 >> 172.21.15.3:0/400786569 conn(0x7f3cb805c620 msgr2=0x7f3cb805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:29.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 -- 172.21.15.3:0/400786569 shutdown_connections 2022-07-21T01:33:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.534+0000 7f3cc072a700 1 -- 172.21.15.3:0/400786569 wait complete. 2022-07-21T01:33:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 Processor -- start 2022-07-21T01:33:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 -- start start 2022-07-21T01:33:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b4550 0x7f3cb8123800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb80b6710 0x7f3cb81404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cb80b6af0 0x7f3cb81409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80b2400 con 0x7f3cb80b6af0 2022-07-21T01:33:29.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8120470 con 0x7f3cb80b6710 2022-07-21T01:33:29.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb81201d0 con 0x7f3cb80b4550 2022-07-21T01:33:29.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbecc7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cb80b6af0 0x7f3cb81409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbecc7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cb80b6af0 0x7f3cb81409e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42716/0 (socket says 172.21.15.3:42716) 2022-07-21T01:33:29.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbecc7700 1 -- 172.21.15.3:0/1670350860 learned_addr learned my addr 172.21.15.3:0/1670350860 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:29.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbdcc5700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb80b6710 0x7f3cb81404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbe4c6700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b4550 0x7f3cb8123800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbecc7700 1 -- 172.21.15.3:0/1670350860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b4550 msgr2=0x7f3cb8123800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbecc7700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b4550 0x7f3cb8123800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbecc7700 1 -- 172.21.15.3:0/1670350860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb80b6710 msgr2=0x7f3cb81404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbecc7700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb80b6710 0x7f3cb81404f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbecc7700 1 -- 172.21.15.3:0/1670350860 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb0009470 con 0x7f3cb80b6af0 2022-07-21T01:33:29.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbe4c6700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b4550 0x7f3cb8123800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:29.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cbecc7700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cb80b6af0 0x7f3cb81409e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f3cb00032f0 tx=0x7f3cb0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:29.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3ca77fe700 1 -- 172.21.15.3:0/1670350860 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cb000b8d0 con 0x7f3cb80b6af0 2022-07-21T01:33:29.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3ca77fe700 1 -- 172.21.15.3:0/1670350860 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb00205f0 con 0x7f3cb80b6af0 2022-07-21T01:33:29.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb80b2660 con 0x7f3cb80b6af0 2022-07-21T01:33:29.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3ca77fe700 1 -- 172.21.15.3:0/1670350860 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cb000b600 con 0x7f3cb80b6af0 2022-07-21T01:33:29.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.538+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb811de50 con 0x7f3cb80b6af0 2022-07-21T01:33:29.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.542+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb80b2820 con 0x7f3cb80b6af0 2022-07-21T01:33:29.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.542+0000 7f3ca77fe700 1 -- 172.21.15.3:0/1670350860 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3cb0027030 con 0x7f3cb80b6af0 2022-07-21T01:33:29.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.542+0000 7f3ca77fe700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c98053a70 0x7f3c98055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.542+0000 7f3cbe4c6700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c98053a70 0x7f3c98055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.542+0000 7f3ca77fe700 1 -- 172.21.15.3:0/1670350860 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3cb006d230 con 0x7f3cb80b6af0 2022-07-21T01:33:29.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.542+0000 7f3cbe4c6700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c98053a70 0x7f3c98055f30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f3cb400b000 tx=0x7f3cb4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:29.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.550+0000 7f3ca77fe700 1 -- 172.21.15.3:0/1670350860 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3cb0043ff0 con 0x7f3cb80b6af0 2022-07-21T01:33:29.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.702+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f3cb8061a20 con 0x7f3cb80b6af0 2022-07-21T01:33:29.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.710+0000 7f3ca77fe700 1 -- 172.21.15.3:0/1670350860 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+131768 (secure 0 0 0) 0x7f3cb0025020 con 0x7f3cb80b6af0 2022-07-21T01:33:29.708 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:29.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c98053a70 msgr2=0x7f3c98055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c98053a70 0x7f3c98055f30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f3cb400b000 tx=0x7f3cb4008040).stop 2022-07-21T01:33:29.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cb80b6af0 msgr2=0x7f3cb81409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cb80b6af0 0x7f3cb81409e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f3cb00032f0 tx=0x7f3cb0016000).stop 2022-07-21T01:33:29.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 shutdown_connections 2022-07-21T01:33:29.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cb80b6af0 0x7f3cb81409e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c98053a70 0x7f3c98055f30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb80b6710 0x7f3cb81404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 --2- 172.21.15.3:0/1670350860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb80b4550 0x7f3cb8123800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 >> 172.21.15.3:0/1670350860 conn(0x7f3cb805c620 msgr2=0x7f3cb81432f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:29.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 shutdown_connections 2022-07-21T01:33:29.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.714+0000 7f3cc072a700 1 -- 172.21.15.3:0/1670350860 wait complete. 2022-07-21T01:33:29.723 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"active_gid":4103,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":22700},{"type":"v1","addr":"172.21.15.3:6801","nonce":22700}]},"active_addr":"172.21.15.3:6801/22700","active_change":"2022-07-21T01:33:09.536236+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540138292840890367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":297615216}]},{"addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":138889149}]},{"addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2124746737}]}]}} 2022-07-21T01:33:29.730 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2022-07-21T01:33:29.731 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2022-07-21T01:33:29.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:29.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8d566700 1 Processor -- start 2022-07-21T01:33:29.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8d566700 1 -- start start 2022-07-21T01:33:29.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8d566700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b41f0 0x7fbb88061d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8d566700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb88142ff0 0x7fbb881434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8d566700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb880b60e0 con 0x7fbb880b6420 2022-07-21T01:33:29.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8d566700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb880b2100 con 0x7fbb88142ff0 2022-07-21T01:33:29.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8d566700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880b23a0 con 0x7fbb880b41f0 2022-07-21T01:33:29.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb87fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbb880b6420 0x7fbb88061600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51000/0 (socket says 172.21.15.3:51000) 2022-07-21T01:33:29.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb87fff700 1 -- 172.21.15.3:0/3481107896 learned_addr learned my addr 172.21.15.3:0/3481107896 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:29.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8cd65700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb88142ff0 0x7fbb881434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb7f7fe700 1 --2- 172.21.15.3:0/3481107896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b41f0 0x7fbb88061d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb877fe700 1 -- 172.21.15.3:0/3481107896 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435458858 0 0) 0x7fbb78001110 con 0x7fbb880b6420 2022-07-21T01:33:29.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb877fe700 1 -- 172.21.15.3:0/3481107896 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb68003260 con 0x7fbb880b6420 2022-07-21T01:33:29.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8cd65700 1 -- 172.21.15.3:0/3481107896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b41f0 msgr2=0x7fbb88061d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:29.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8cd65700 1 --2- 172.21.15.3:0/3481107896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b41f0 0x7fbb88061d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8cd65700 1 -- 172.21.15.3:0/3481107896 >> v1:172.21.15.3:6789/0 conn(0x7fbb880b6420 legacy=0x7fbb88061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8cd65700 1 -- 172.21.15.3:0/3481107896 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb880b2640 con 0x7fbb88142ff0 2022-07-21T01:33:29.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8cd65700 1 --2- 172.21.15.3:0/3481107896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88142ff0 0x7fbb881434a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fbb800032f0 tx=0x7fbb800091d0).ready entity=mon.0 client_cookie=8f913f74537753fc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:29.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb877fe700 1 -- 172.21.15.3:0/3481107896 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb8000b600 con 0x7fbb88142ff0 2022-07-21T01:33:29.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb877fe700 1 -- 172.21.15.3:0/3481107896 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8000b760 con 0x7fbb88142ff0 2022-07-21T01:33:29.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb877fe700 1 -- 172.21.15.3:0/3481107896 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb8000b8d0 con 0x7fbb88142ff0 2022-07-21T01:33:29.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8d566700 1 -- 172.21.15.3:0/3481107896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88142ff0 msgr2=0x7fbb881434a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.846+0000 7fbb8d566700 1 --2- 172.21.15.3:0/3481107896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88142ff0 0x7fbb881434a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fbb800032f0 tx=0x7fbb800091d0).stop 2022-07-21T01:33:29.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 -- 172.21.15.3:0/3481107896 shutdown_connections 2022-07-21T01:33:29.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 --2- 172.21.15.3:0/3481107896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88142ff0 0x7fbb881434a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 --2- 172.21.15.3:0/3481107896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b41f0 0x7fbb88061d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 -- 172.21.15.3:0/3481107896 >> 172.21.15.3:0/3481107896 conn(0x7fbb8805c620 msgr2=0x7fbb8805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:29.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 -- 172.21.15.3:0/3481107896 shutdown_connections 2022-07-21T01:33:29.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 -- 172.21.15.3:0/3481107896 wait complete. 2022-07-21T01:33:29.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 Processor -- start 2022-07-21T01:33:29.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 -- start start 2022-07-21T01:33:29.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b41f0 0x7fbb880b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b6420 0x7fbb88152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb88142ff0 0x7fbb88152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880afe20 con 0x7fbb880b41f0 2022-07-21T01:33:29.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880b00c0 con 0x7fbb88142ff0 2022-07-21T01:33:29.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880b0360 con 0x7fbb880b6420 2022-07-21T01:33:29.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb87fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b41f0 0x7fbb880b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb87fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b41f0 0x7fbb880b50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42726/0 (socket says 172.21.15.3:42726) 2022-07-21T01:33:29.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb87fff700 1 -- 172.21.15.3:0/2235735948 learned_addr learned my addr 172.21.15.3:0/2235735948 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:29.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb7f7fe700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b6420 0x7fbb88152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8cd65700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb88142ff0 0x7fbb88152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb87fff700 1 -- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b6420 msgr2=0x7fbb88152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb87fff700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b6420 0x7fbb88152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb87fff700 1 -- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb88142ff0 msgr2=0x7fbb88152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb87fff700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb88142ff0 0x7fbb88152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb87fff700 1 -- 172.21.15.3:0/2235735948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb80009470 con 0x7fbb880b41f0 2022-07-21T01:33:29.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb7f7fe700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b6420 0x7fbb88152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:29.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8cd65700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb88142ff0 0x7fbb88152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:29.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb87fff700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b41f0 0x7fbb880b50b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbb780012a0 tx=0x7fbb78007900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:29.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb85ffb700 1 -- 172.21.15.3:0/2235735948 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb7800ce30 con 0x7fbb880b41f0 2022-07-21T01:33:29.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb85ffb700 1 -- 172.21.15.3:0/2235735948 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb780072e0 con 0x7fbb880b41f0 2022-07-21T01:33:29.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8813e9d0 con 0x7fbb880b41f0 2022-07-21T01:33:29.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.850+0000 7fbb85ffb700 1 -- 172.21.15.3:0/2235735948 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb78005610 con 0x7fbb880b41f0 2022-07-21T01:33:29.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.854+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb88140210 con 0x7fbb880b41f0 2022-07-21T01:33:29.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.854+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb880b0520 con 0x7fbb880b41f0 2022-07-21T01:33:29.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.854+0000 7fbb85ffb700 1 -- 172.21.15.3:0/2235735948 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbb78010070 con 0x7fbb880b41f0 2022-07-21T01:33:29.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.854+0000 7fbb85ffb700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb6c053b00 0x7fbb6c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:29.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.854+0000 7fbb85ffb700 1 -- 172.21.15.3:0/2235735948 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbb7806b510 con 0x7fbb880b41f0 2022-07-21T01:33:29.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.854+0000 7fbb7f7fe700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb6c053b00 0x7fbb6c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:29.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.854+0000 7fbb7f7fe700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb6c053b00 0x7fbb6c055fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fbb7000b000 tx=0x7fbb70008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:29.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.858+0000 7fbb85ffb700 1 -- 172.21.15.3:0/2235735948 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb780422d0 con 0x7fbb880b41f0 2022-07-21T01:33:29.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.986+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb880b60e0 con 0x7fbb880b41f0 2022-07-21T01:33:29.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb85ffb700 1 -- 172.21.15.3:0/2235735948 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7fbb880b60e0 con 0x7fbb880b41f0 2022-07-21T01:33:29.987 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:29.987 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:29.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb6c053b00 msgr2=0x7fbb6c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb6c053b00 0x7fbb6c055fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fbb7000b000 tx=0x7fbb70008040).stop 2022-07-21T01:33:29.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b41f0 msgr2=0x7fbb880b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:29.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b41f0 0x7fbb880b50b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbb780012a0 tx=0x7fbb78007900).stop 2022-07-21T01:33:29.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 shutdown_connections 2022-07-21T01:33:29.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb6c053b00 0x7fbb6c055fc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb88142ff0 0x7fbb88152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb880b6420 0x7fbb88152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 --2- 172.21.15.3:0/2235735948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb880b41f0 0x7fbb880b50b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:29.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 >> 172.21.15.3:0/2235735948 conn(0x7fbb8805c620 msgr2=0x7fbb880b8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:29.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 shutdown_connections 2022-07-21T01:33:29.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:29.990+0000 7fbb8d566700 1 -- 172.21.15.3:0/2235735948 wait complete. 2022-07-21T01:33:30.001 INFO:tasks.ceph.ceph_manager.ceph:all up! 2022-07-21T01:33:30.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:30.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.118+0000 7efed8d22700 1 Processor -- start 2022-07-21T01:33:30.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.118+0000 7efed8d22700 1 -- start start 2022-07-21T01:33:30.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.118+0000 7efed8d22700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efed40ad670 0x7efed40b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.118+0000 7efed8d22700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efed40af120 0x7efed40b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.118+0000 7efed8d22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 0x7efed405fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.118+0000 7efed8d22700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed40b0c40 con 0x7efed40af120 2022-07-21T01:33:30.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.118+0000 7efed8d22700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed40b0ee0 con 0x7efed40ad670 2022-07-21T01:33:30.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.118+0000 7efed8d22700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed40ae9e0 con 0x7efed405f840 2022-07-21T01:33:30.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.118+0000 7efed2ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efed40af120 0x7efed40b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed37fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efed40ad670 0x7efed40b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed2ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efed40af120 0x7efed40b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42738/0 (socket says 172.21.15.3:42738) 2022-07-21T01:33:30.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed2ffd700 1 -- 172.21.15.3:0/407730562 learned_addr learned my addr 172.21.15.3:0/407730562 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:30.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed3fff700 1 --2- 172.21.15.3:0/407730562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 0x7efed405fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed37fe700 1 -- 172.21.15.3:0/407730562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 msgr2=0x7efed405fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:30.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed37fe700 1 --2- 172.21.15.3:0/407730562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 0x7efed405fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed37fe700 1 -- 172.21.15.3:0/407730562 >> v2:172.21.15.3:3300/0 conn(0x7efed40af120 msgr2=0x7efed40b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:30.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed37fe700 1 --2- 172.21.15.3:0/407730562 >> v2:172.21.15.3:3300/0 conn(0x7efed40af120 0x7efed40b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed37fe700 1 -- 172.21.15.3:0/407730562 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed40aec80 con 0x7efed40ad670 2022-07-21T01:33:30.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed37fe700 1 --2- 172.21.15.3:0/407730562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40ad670 0x7efed40b0570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7efecc0032f0 tx=0x7efecc0091d0).ready entity=mon.1 client_cookie=689a081d0eaee2ec server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed27fc700 1 -- 172.21.15.3:0/407730562 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efecc00b600 con 0x7efed40ad670 2022-07-21T01:33:30.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed27fc700 1 -- 172.21.15.3:0/407730562 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc00b760 con 0x7efed40ad670 2022-07-21T01:33:30.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed27fc700 1 -- 172.21.15.3:0/407730562 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efecc00b8d0 con 0x7efed40ad670 2022-07-21T01:33:30.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 -- 172.21.15.3:0/407730562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40ad670 msgr2=0x7efed40b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 --2- 172.21.15.3:0/407730562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40ad670 0x7efed40b0570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7efecc0032f0 tx=0x7efecc0091d0).stop 2022-07-21T01:33:30.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 -- 172.21.15.3:0/407730562 shutdown_connections 2022-07-21T01:33:30.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 --2- 172.21.15.3:0/407730562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 0x7efed405fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 --2- 172.21.15.3:0/407730562 >> v2:172.21.15.3:3300/0 conn(0x7efed40af120 0x7efed40b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 --2- 172.21.15.3:0/407730562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40ad670 0x7efed40b0570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 -- 172.21.15.3:0/407730562 >> 172.21.15.3:0/407730562 conn(0x7efed4061360 msgr2=0x7efed4062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 -- 172.21.15.3:0/407730562 shutdown_connections 2022-07-21T01:33:30.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 -- 172.21.15.3:0/407730562 wait complete. 2022-07-21T01:33:30.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 Processor -- start 2022-07-21T01:33:30.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.122+0000 7efed8d22700 1 -- start start 2022-07-21T01:33:30.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed8d22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 0x7efed4064190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed8d22700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efed40ad670 0x7efed4152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed8d22700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40af120 0x7efed4152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed8d22700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed4152130 con 0x7efed40ad670 2022-07-21T01:33:30.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed8d22700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed40ac620 con 0x7efed40af120 2022-07-21T01:33:30.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed8d22700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed40ac880 con 0x7efed405f840 2022-07-21T01:33:30.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efed40ad670 0x7efed4152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efed40ad670 0x7efed4152780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42740/0 (socket says 172.21.15.3:42740) 2022-07-21T01:33:30.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed2ffd700 1 -- 172.21.15.3:0/725335328 learned_addr learned my addr 172.21.15.3:0/725335328 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:30.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed3fff700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40af120 0x7efed4152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed37fe700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 0x7efed4064190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed2ffd700 1 -- 172.21.15.3:0/725335328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 msgr2=0x7efed4064190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed2ffd700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 0x7efed4064190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed2ffd700 1 -- 172.21.15.3:0/725335328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40af120 msgr2=0x7efed4152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed2ffd700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40af120 0x7efed4152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed2ffd700 1 -- 172.21.15.3:0/725335328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc009470 con 0x7efed40ad670 2022-07-21T01:33:30.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed3fff700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40af120 0x7efed4152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:30.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed2ffd700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efed40ad670 0x7efed4152780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7efec800c000 tx=0x7efec8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed0ff9700 1 -- 172.21.15.3:0/725335328 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efec80136e0 con 0x7efed40ad670 2022-07-21T01:33:30.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed0ff9700 1 -- 172.21.15.3:0/725335328 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec8013840 con 0x7efed40ad670 2022-07-21T01:33:30.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed40acab0 con 0x7efed40ad670 2022-07-21T01:33:30.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed0ff9700 1 -- 172.21.15.3:0/725335328 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efec8013ba0 con 0x7efed40ad670 2022-07-21T01:33:30.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed40ac380 con 0x7efed40ad670 2022-07-21T01:33:30.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed41524c0 con 0x7efed40ad670 2022-07-21T01:33:30.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.126+0000 7efed0ff9700 1 -- 172.21.15.3:0/725335328 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efec8023070 con 0x7efed40ad670 2022-07-21T01:33:30.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.130+0000 7efed0ff9700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efeb40539e0 0x7efeb4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.130+0000 7efed37fe700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efeb40539e0 0x7efeb4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.130+0000 7efed0ff9700 1 -- 172.21.15.3:0/725335328 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efec806c400 con 0x7efed40ad670 2022-07-21T01:33:30.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.130+0000 7efed37fe700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efeb40539e0 0x7efeb4055ea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7efecc019000 tx=0x7efecc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.134+0000 7efed0ff9700 1 -- 172.21.15.3:0/725335328 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efec8043160 con 0x7efed40ad670 2022-07-21T01:33:30.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.258+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efed40af590 con 0x7efed40ad670 2022-07-21T01:33:30.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed0ff9700 1 -- 172.21.15.3:0/725335328 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7efec8046780 con 0x7efed40ad670 2022-07-21T01:33:30.259 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:30.259 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:30.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efeb40539e0 msgr2=0x7efeb4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efeb40539e0 0x7efeb4055ea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7efecc019000 tx=0x7efecc016000).stop 2022-07-21T01:33:30.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efed40ad670 msgr2=0x7efed4152780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efed40ad670 0x7efed4152780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7efec800c000 tx=0x7efec8008040).stop 2022-07-21T01:33:30.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 shutdown_connections 2022-07-21T01:33:30.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efeb40539e0 0x7efeb4055ea0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efed40af120 0x7efed4152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efed40ad670 0x7efed4152780 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 --2- 172.21.15.3:0/725335328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efed405f840 0x7efed4064190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 >> 172.21.15.3:0/725335328 conn(0x7efed4061360 msgr2=0x7efed413e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 shutdown_connections 2022-07-21T01:33:30.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.262+0000 7efed8d22700 1 -- 172.21.15.3:0/725335328 wait complete. 2022-07-21T01:33:30.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-07-21T01:33:30.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.378+0000 7f51dbfff700 1 Processor -- start 2022-07-21T01:33:30.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51dbfff700 1 -- start start 2022-07-21T01:33:30.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51dbfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f51dc0af550 0x7f51dc0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51dbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc060330 0x7f51dc1445e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51dbfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51dc0aa630 con 0x7f51dc144ad0 2022-07-21T01:33:30.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51dbfff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51dc0b15e0 con 0x7f51dc0af550 2022-07-21T01:33:30.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51dbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc0af930 con 0x7f51dc060330 2022-07-21T01:33:30.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51db7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f51dc144ad0 0x7f51dc0a8d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51032/0 (socket says 172.21.15.3:51032) 2022-07-21T01:33:30.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51db7fe700 1 -- 172.21.15.3:0/1196816469 learned_addr learned my addr 172.21.15.3:0/1196816469 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:30.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51daffd700 1 --2- 172.21.15.3:0/1196816469 >> v2:172.21.15.143:3300/0 conn(0x7f51dc0af550 0x7f51dc0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51da7fc700 1 --2- 172.21.15.3:0/1196816469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc060330 0x7f51dc1445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51d9ffb700 1 -- 172.21.15.3:0/1196816469 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285710489 0 0) 0x7f51dc0aa630 con 0x7f51dc144ad0 2022-07-21T01:33:30.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51d9ffb700 1 -- 172.21.15.3:0/1196816469 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c0003260 con 0x7f51dc144ad0 2022-07-21T01:33:30.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51daffd700 1 -- 172.21.15.3:0/1196816469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc060330 msgr2=0x7f51dc1445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:30.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51daffd700 1 --2- 172.21.15.3:0/1196816469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc060330 0x7f51dc1445e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51daffd700 1 -- 172.21.15.3:0/1196816469 >> v1:172.21.15.3:6789/0 conn(0x7f51dc144ad0 legacy=0x7f51dc0a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51daffd700 1 -- 172.21.15.3:0/1196816469 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc05fd10 con 0x7f51dc0af550 2022-07-21T01:33:30.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51da7fc700 1 --2- 172.21.15.3:0/1196816469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc060330 0x7f51dc1445e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:30.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.382+0000 7f51daffd700 1 --2- 172.21.15.3:0/1196816469 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51dc0af550 0x7f51dc0b0e40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f51cc002f70 tx=0x7f51cc006df0).ready entity=mon.1 client_cookie=4314873965228ce1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51d9ffb700 1 -- 172.21.15.3:0/1196816469 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51cc00d4a0 con 0x7f51dc0af550 2022-07-21T01:33:30.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51d9ffb700 1 -- 172.21.15.3:0/1196816469 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc000e90 con 0x7f51dc0af550 2022-07-21T01:33:30.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51d9ffb700 1 -- 172.21.15.3:0/1196816469 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51cc00d7f0 con 0x7f51dc0af550 2022-07-21T01:33:30.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51dbfff700 1 -- 172.21.15.3:0/1196816469 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51dc0af550 msgr2=0x7f51dc0b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51dbfff700 1 --2- 172.21.15.3:0/1196816469 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51dc0af550 0x7f51dc0b0e40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f51cc002f70 tx=0x7f51cc006df0).stop 2022-07-21T01:33:30.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51dbfff700 1 -- 172.21.15.3:0/1196816469 shutdown_connections 2022-07-21T01:33:30.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51dbfff700 1 --2- 172.21.15.3:0/1196816469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc060330 0x7f51dc1445e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51dbfff700 1 --2- 172.21.15.3:0/1196816469 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51dc0af550 0x7f51dc0b0e40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51dbfff700 1 -- 172.21.15.3:0/1196816469 >> 172.21.15.3:0/1196816469 conn(0x7f51dc0612d0 msgr2=0x7f51dc062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51dbfff700 1 -- 172.21.15.3:0/1196816469 shutdown_connections 2022-07-21T01:33:30.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51dbfff700 1 -- 172.21.15.3:0/1196816469 wait complete. 2022-07-21T01:33:30.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.386+0000 7f51dbfff700 1 Processor -- start 2022-07-21T01:33:30.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 -- start start 2022-07-21T01:33:30.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51dc060330 0x7f51dc0aace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51dc0af550 0x7f51dc063fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc144ad0 0x7f51dc0644d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc0aec90 con 0x7f51dc0af550 2022-07-21T01:33:30.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc140f20 con 0x7f51dc060330 2022-07-21T01:33:30.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc0aeec0 con 0x7f51dc144ad0 2022-07-21T01:33:30.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51da7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51dc0af550 0x7f51dc063fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51da7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51dc0af550 0x7f51dc063fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42754/0 (socket says 172.21.15.3:42754) 2022-07-21T01:33:30.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51da7fc700 1 -- 172.21.15.3:0/238400276 learned_addr learned my addr 172.21.15.3:0/238400276 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:30.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51db7fe700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc144ad0 0x7f51dc0644d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51daffd700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51dc060330 0x7f51dc0aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51da7fc700 1 -- 172.21.15.3:0/238400276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc144ad0 msgr2=0x7f51dc0644d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51da7fc700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51dc144ad0 0x7f51dc0644d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51da7fc700 1 -- 172.21.15.3:0/238400276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51dc060330 msgr2=0x7f51dc0aace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51da7fc700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51dc060330 0x7f51dc0aace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51da7fc700 1 -- 172.21.15.3:0/238400276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d000b180 con 0x7f51dc0af550 2022-07-21T01:33:30.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51da7fc700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51dc0af550 0x7f51dc063fe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f51d000c000 tx=0x7f51d0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51cbfff700 1 -- 172.21.15.3:0/238400276 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51d000c7e0 con 0x7f51dc0af550 2022-07-21T01:33:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51cbfff700 1 -- 172.21.15.3:0/238400276 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d000c940 con 0x7f51dc0af550 2022-07-21T01:33:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc00b180 con 0x7f51dc0af550 2022-07-21T01:33:30.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51cbfff700 1 -- 172.21.15.3:0/238400276 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51d000cca0 con 0x7f51dc0af550 2022-07-21T01:33:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51dc05f840 con 0x7f51dc0af550 2022-07-21T01:33:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51cbfff700 1 -- 172.21.15.3:0/238400276 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f51d001c460 con 0x7f51dc0af550 2022-07-21T01:33:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51dc140d80 con 0x7f51dc0af550 2022-07-21T01:33:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.390+0000 7f51cbfff700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51b8053b00 0x7f51b8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.394+0000 7f51cbfff700 1 -- 172.21.15.3:0/238400276 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f51d006c600 con 0x7f51dc0af550 2022-07-21T01:33:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.394+0000 7f51cbfff700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f51b805b400 0x7f51b805d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.394+0000 7f51daffd700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51b8053b00 0x7f51b8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.394+0000 7f51cbfff700 1 -- 172.21.15.3:0/238400276 --> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51b805de80 con 0x7f51b805b400 2022-07-21T01:33:30.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.394+0000 7f51cbfff700 1 -- 172.21.15.3:0/238400276 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51d0043360 con 0x7f51dc0af550 2022-07-21T01:33:30.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.394+0000 7f51db7fe700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f51b805b400 0x7f51b805d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.394+0000 7f51db7fe700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f51b805b400 0x7f51b805d8a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.394+0000 7f51daffd700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51b8053b00 0x7f51b8055fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f51cc01a000 tx=0x7f51cc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.394+0000 7f51cbfff700 1 -- 172.21.15.3:0/238400276 <== osd.0 v2:172.21.15.3:6813/22896 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f51b805de80 con 0x7f51b805b400 2022-07-21T01:33:30.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.410+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 --> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f51dc140bb0 con 0x7f51b805b400 2022-07-21T01:33:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51cbfff700 1 -- 172.21.15.3:0/238400276 <== osd.0 v2:172.21.15.3:6813/22896 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f51dc140bb0 con 0x7f51b805b400 2022-07-21T01:33:30.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f51b805b400 msgr2=0x7f51b805d8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f51b805b400 0x7f51b805d8a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51b8053b00 msgr2=0x7f51b8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51b8053b00 0x7f51b8055fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f51cc01a000 tx=0x7f51cc014000).stop 2022-07-21T01:33:30.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51dc0af550 msgr2=0x7f51dc063fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 --2- 172.21.15.3:0/238400276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51dc0af550 0x7f51dc063fe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f51d000c000 tx=0x7f51d0006df0).stop 2022-07-21T01:33:30.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51db7fe700 1 -- 172.21.15.3:0/238400276 reap_dead start 2022-07-21T01:33:30.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 shutdown_connections 2022-07-21T01:33:30.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 >> 172.21.15.3:0/238400276 conn(0x7f51dc0612d0 msgr2=0x7f51dc062f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 shutdown_connections 2022-07-21T01:33:30.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.414+0000 7f51dbfff700 1 -- 172.21.15.3:0/238400276 wait complete. 2022-07-21T01:33:30.422 INFO:teuthology.orchestra.run.smithi003.stdout:64424509446 2022-07-21T01:33:30.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2022-07-21T01:33:30.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a759e700 1 Processor -- start 2022-07-21T01:33:30.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a759e700 1 -- start start 2022-07-21T01:33:30.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 0x7f16a8061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f16a80ad750 0x7f16a80b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a759e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a80aee80 con 0x7f16a81405d0 2022-07-21T01:33:30.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a80b70b0 con 0x7f16a80ad750 2022-07-21T01:33:30.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80af0b0 con 0x7f16a80614b0 2022-07-21T01:33:30.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a6d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f16a81405d0 0x7f16a81466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51046/0 (socket says 172.21.15.3:51046) 2022-07-21T01:33:30.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a6d9d700 1 -- 172.21.15.3:0/1199361659 learned_addr learned my addr 172.21.15.3:0/1199361659 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:30.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a659c700 1 --2- 172.21.15.3:0/1199361659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 0x7f16a8061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a5d9b700 1 --2- 172.21.15.3:0/1199361659 >> v2:172.21.15.143:3300/0 conn(0x7f16a80ad750 0x7f16a80b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a559a700 1 -- 172.21.15.3:0/1199361659 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189743176 0 0) 0x7f16a80aee80 con 0x7f16a81405d0 2022-07-21T01:33:30.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a559a700 1 -- 172.21.15.3:0/1199361659 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f168c003260 con 0x7f16a81405d0 2022-07-21T01:33:30.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a659c700 1 -- 172.21.15.3:0/1199361659 >> v2:172.21.15.143:3300/0 conn(0x7f16a80ad750 msgr2=0x7f16a80b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:30.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a659c700 1 --2- 172.21.15.3:0/1199361659 >> v2:172.21.15.143:3300/0 conn(0x7f16a80ad750 0x7f16a80b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a659c700 1 -- 172.21.15.3:0/1199361659 >> v1:172.21.15.3:6789/0 conn(0x7f16a81405d0 legacy=0x7f16a81466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a659c700 1 -- 172.21.15.3:0/1199361659 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a80af350 con 0x7f16a80614b0 2022-07-21T01:33:30.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a659c700 1 --2- 172.21.15.3:0/1199361659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 0x7f16a8061890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f16980032f0 tx=0x7f16980091d0).ready entity=mon.2 client_cookie=2c82c411c8225283 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a559a700 1 -- 172.21.15.3:0/1199361659 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f169800b600 con 0x7f16a80614b0 2022-07-21T01:33:30.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a559a700 1 -- 172.21.15.3:0/1199361659 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169800b760 con 0x7f16a80614b0 2022-07-21T01:33:30.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.538+0000 7f16a559a700 1 -- 172.21.15.3:0/1199361659 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f169800b8d0 con 0x7f16a80614b0 2022-07-21T01:33:30.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 -- 172.21.15.3:0/1199361659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 msgr2=0x7f16a8061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 --2- 172.21.15.3:0/1199361659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 0x7f16a8061890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f16980032f0 tx=0x7f16980091d0).stop 2022-07-21T01:33:30.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 -- 172.21.15.3:0/1199361659 shutdown_connections 2022-07-21T01:33:30.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 --2- 172.21.15.3:0/1199361659 >> v2:172.21.15.143:3300/0 conn(0x7f16a80ad750 0x7f16a80b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 --2- 172.21.15.3:0/1199361659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 0x7f16a8061890 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 -- 172.21.15.3:0/1199361659 >> 172.21.15.3:0/1199361659 conn(0x7f16a8066f70 msgr2=0x7f16a8067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 -- 172.21.15.3:0/1199361659 shutdown_connections 2022-07-21T01:33:30.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 -- 172.21.15.3:0/1199361659 wait complete. 2022-07-21T01:33:30.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 Processor -- start 2022-07-21T01:33:30.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 -- start start 2022-07-21T01:33:30.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 0x7f16a8147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16a80ad750 0x7f16a8152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16a81405d0 0x7f16a80ab560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80b00b0 con 0x7f16a81405d0 2022-07-21T01:33:30.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80ab130 con 0x7f16a80ad750 2022-07-21T01:33:30.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a81483e0 con 0x7f16a80614b0 2022-07-21T01:33:30.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16a81405d0 0x7f16a80ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 0x7f16a8147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16a81405d0 0x7f16a80ab560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42774/0 (socket says 172.21.15.3:42774) 2022-07-21T01:33:30.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a6d9d700 1 -- 172.21.15.3:0/2622101915 learned_addr learned my addr 172.21.15.3:0/2622101915 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:30.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.542+0000 7f16a5d9b700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16a80ad750 0x7f16a8152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a6d9d700 1 -- 172.21.15.3:0/2622101915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 msgr2=0x7f16a8147d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a6d9d700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16a80614b0 0x7f16a8147d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a6d9d700 1 -- 172.21.15.3:0/2622101915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16a80ad750 msgr2=0x7f16a8152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a6d9d700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16a80ad750 0x7f16a8152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a6d9d700 1 -- 172.21.15.3:0/2622101915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1698009470 con 0x7f16a81405d0 2022-07-21T01:33:30.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a5d9b700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16a80ad750 0x7f16a8152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:30.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a6d9d700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16a81405d0 0x7f16a80ab560 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f16a0001110 tx=0x7f16a0007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f169f7fe700 1 -- 172.21.15.3:0/2622101915 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a000ce50 con 0x7f16a81405d0 2022-07-21T01:33:30.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f169f7fe700 1 -- 172.21.15.3:0/2622101915 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a00070d0 con 0x7f16a81405d0 2022-07-21T01:33:30.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a81532d0 con 0x7f16a81405d0 2022-07-21T01:33:30.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f169f7fe700 1 -- 172.21.15.3:0/2622101915 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a0007430 con 0x7f16a81405d0 2022-07-21T01:33:30.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a80b1230 con 0x7f16a81405d0 2022-07-21T01:33:30.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16a80ab2c0 con 0x7f16a81405d0 2022-07-21T01:33:30.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.546+0000 7f169f7fe700 1 -- 172.21.15.3:0/2622101915 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f16a001a060 con 0x7f16a81405d0 2022-07-21T01:33:30.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f169f7fe700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1680053b00 0x7f1680055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f169f7fe700 1 -- 172.21.15.3:0/2622101915 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f16a006b720 con 0x7f16a81405d0 2022-07-21T01:33:30.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f16a659c700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1680053b00 0x7f1680055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f169f7fe700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f168005b480 0x7f168005d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f169f7fe700 1 -- 172.21.15.3:0/2622101915 --> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f168005df00 con 0x7f168005b480 2022-07-21T01:33:30.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f169f7fe700 1 -- 172.21.15.3:0/2622101915 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16a006ddb0 con 0x7f16a81405d0 2022-07-21T01:33:30.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f16a5d9b700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f168005b480 0x7f168005d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f16a659c700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1680053b00 0x7f1680055fc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f169801b000 tx=0x7f1698016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f16a5d9b700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f168005b480 0x7f168005d920 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.550+0000 7f169f7fe700 1 -- 172.21.15.3:0/2622101915 <== osd.1 v2:172.21.15.3:6810/22890 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f168005df00 con 0x7f168005b480 2022-07-21T01:33:30.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 --> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f16a8146b70 con 0x7f168005b480 2022-07-21T01:33:30.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f169f7fe700 1 -- 172.21.15.3:0/2622101915 <== osd.1 v2:172.21.15.3:6810/22890 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f16a8146b70 con 0x7f168005b480 2022-07-21T01:33:30.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f168005b480 msgr2=0x7f168005d920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f168005b480 0x7f168005d920 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1680053b00 msgr2=0x7f1680055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1680053b00 0x7f1680055fc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f169801b000 tx=0x7f1698016000).stop 2022-07-21T01:33:30.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16a81405d0 msgr2=0x7f16a80ab560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 --2- 172.21.15.3:0/2622101915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16a81405d0 0x7f16a80ab560 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f16a0001110 tx=0x7f16a0007a80).stop 2022-07-21T01:33:30.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a6d9d700 1 -- 172.21.15.3:0/2622101915 reap_dead start 2022-07-21T01:33:30.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 shutdown_connections 2022-07-21T01:33:30.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 >> 172.21.15.3:0/2622101915 conn(0x7f16a8066f70 msgr2=0x7f16a80b8750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 shutdown_connections 2022-07-21T01:33:30.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.574+0000 7f16a759e700 1 -- 172.21.15.3:0/2622101915 wait complete. 2022-07-21T01:33:30.587 INFO:teuthology.orchestra.run.smithi003.stdout:60129542150 2022-07-21T01:33:30.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2022-07-21T01:33:30.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.702+0000 7f2a59cd0700 1 Processor -- start 2022-07-21T01:33:30.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.702+0000 7f2a59cd0700 1 -- start start 2022-07-21T01:33:30.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.702+0000 7f2a59cd0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2a540b7500 0x7f2a540b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.702+0000 7f2a59cd0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2a540b31f0 0x7f2a5413e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.702+0000 7f2a59cd0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a540b2510 con 0x7f2a5413e8a0 2022-07-21T01:33:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.702+0000 7f2a59cd0700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a541443e0 con 0x7f2a540b31f0 2022-07-21T01:33:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.702+0000 7f2a59cd0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a540b3700 con 0x7f2a540b7500 2022-07-21T01:33:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.702+0000 7f2a53fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2a540b31f0 0x7f2a5413e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a53fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2a540b31f0 0x7f2a5413e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42782/0 (socket says 172.21.15.3:42782) 2022-07-21T01:33:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a53fff700 1 -- 172.21.15.3:0/2705447424 learned_addr learned my addr 172.21.15.3:0/2705447424 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:30.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a58cce700 1 --2- 172.21.15.3:0/2705447424 >> v2:172.21.15.143:3300/0 conn(0x7f2a540b7500 0x7f2a540b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a53fff700 1 -- 172.21.15.3:0/2705447424 >> v2:172.21.15.143:3300/0 conn(0x7f2a540b7500 msgr2=0x7f2a540b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:30.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a53fff700 1 --2- 172.21.15.3:0/2705447424 >> v2:172.21.15.143:3300/0 conn(0x7f2a540b7500 0x7f2a540b78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a53fff700 1 -- 172.21.15.3:0/2705447424 >> v1:172.21.15.143:6789/0 conn(0x7f2a5413e8a0 legacy=0x7f2a54064d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a53fff700 1 -- 172.21.15.3:0/2705447424 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a540b2740 con 0x7f2a540b31f0 2022-07-21T01:33:30.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a53fff700 1 --2- 172.21.15.3:0/2705447424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 0x7f2a5413e3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f2a48002f70 tx=0x7f2a48006eb0).ready entity=mon.0 client_cookie=157d0e8fb49a3d52 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a537fe700 1 -- 172.21.15.3:0/2705447424 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a4800d5c0 con 0x7f2a540b31f0 2022-07-21T01:33:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a537fe700 1 -- 172.21.15.3:0/2705447424 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a48000e90 con 0x7f2a540b31f0 2022-07-21T01:33:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a537fe700 1 -- 172.21.15.3:0/2705447424 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a4800d950 con 0x7f2a540b31f0 2022-07-21T01:33:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2705447424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 msgr2=0x7f2a5413e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a59cd0700 1 --2- 172.21.15.3:0/2705447424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 0x7f2a5413e3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f2a48002f70 tx=0x7f2a48006eb0).stop 2022-07-21T01:33:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2705447424 shutdown_connections 2022-07-21T01:33:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a59cd0700 1 --2- 172.21.15.3:0/2705447424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 0x7f2a5413e3b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a59cd0700 1 --2- 172.21.15.3:0/2705447424 >> v2:172.21.15.143:3300/0 conn(0x7f2a540b7500 0x7f2a540b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2705447424 >> 172.21.15.3:0/2705447424 conn(0x7f2a5405f840 msgr2=0x7f2a5405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2705447424 shutdown_connections 2022-07-21T01:33:30.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.706+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2705447424 wait complete. 2022-07-21T01:33:30.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 Processor -- start 2022-07-21T01:33:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 -- start start 2022-07-21T01:33:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 0x7f2a540ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a540b7500 0x7f2a540ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5413e8a0 0x7f2a54153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a540ab4c0 con 0x7f2a540b31f0 2022-07-21T01:33:30.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a540acc80 con 0x7f2a5413e8a0 2022-07-21T01:33:30.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a540ab6b0 con 0x7f2a540b7500 2022-07-21T01:33:30.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a58cce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 0x7f2a540ac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a58cce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 0x7f2a540ac3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42786/0 (socket says 172.21.15.3:42786) 2022-07-21T01:33:30.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a58cce700 1 -- 172.21.15.3:0/2633731304 learned_addr learned my addr 172.21.15.3:0/2633731304 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:30.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a594cf700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5413e8a0 0x7f2a54153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a53fff700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a540b7500 0x7f2a540ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a58cce700 1 -- 172.21.15.3:0/2633731304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a540b7500 msgr2=0x7f2a540ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a58cce700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a540b7500 0x7f2a540ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a58cce700 1 -- 172.21.15.3:0/2633731304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5413e8a0 msgr2=0x7f2a54153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a58cce700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5413e8a0 0x7f2a54153780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a58cce700 1 -- 172.21.15.3:0/2633731304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4800b1e0 con 0x7f2a540b31f0 2022-07-21T01:33:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a58cce700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 0x7f2a540ac3c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2a4400c000 tx=0x7f2a44006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a51ffb700 1 -- 172.21.15.3:0/2633731304 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a440104c0 con 0x7f2a540b31f0 2022-07-21T01:33:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a51ffb700 1 -- 172.21.15.3:0/2633731304 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a44010620 con 0x7f2a540b31f0 2022-07-21T01:33:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a540ab950 con 0x7f2a540b31f0 2022-07-21T01:33:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a51ffb700 1 -- 172.21.15.3:0/2633731304 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a44010980 con 0x7f2a540b31f0 2022-07-21T01:33:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.710+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a540b1420 con 0x7f2a540b31f0 2022-07-21T01:33:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a540acdf0 con 0x7f2a540b31f0 2022-07-21T01:33:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a51ffb700 1 -- 172.21.15.3:0/2633731304 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2a44010ae0 con 0x7f2a540b31f0 2022-07-21T01:33:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a51ffb700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2a34053980 0x7f2a34055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a51ffb700 1 -- 172.21.15.3:0/2633731304 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2a4406bb40 con 0x7f2a540b31f0 2022-07-21T01:33:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a53fff700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2a34053980 0x7f2a34055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a51ffb700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7f2a3405b3a0 0x7f2a3405d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a51ffb700 1 -- 172.21.15.3:0/2633731304 --> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a3405dec0 con 0x7f2a3405b3a0 2022-07-21T01:33:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a51ffb700 1 -- 172.21.15.3:0/2633731304 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a44070240 con 0x7f2a540b31f0 2022-07-21T01:33:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a594cf700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7f2a3405b3a0 0x7f2a3405d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a53fff700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2a34053980 0x7f2a34055e40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2a4801c000 tx=0x7f2a48015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.714+0000 7f2a594cf700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7f2a3405b3a0 0x7f2a3405d840 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.718+0000 7f2a51ffb700 1 -- 172.21.15.3:0/2633731304 <== osd.2 v2:172.21.15.3:6802/22877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f2a3405dec0 con 0x7f2a3405b3a0 2022-07-21T01:33:30.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.734+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 --> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2a540574e0 con 0x7f2a3405b3a0 2022-07-21T01:33:30.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a51ffb700 1 -- 172.21.15.3:0/2633731304 <== osd.2 v2:172.21.15.3:6802/22877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2a540574e0 con 0x7f2a3405b3a0 2022-07-21T01:33:30.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7f2a3405b3a0 msgr2=0x7f2a3405d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7f2a3405b3a0 0x7f2a3405d840 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2a34053980 msgr2=0x7f2a34055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2a34053980 0x7f2a34055e40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2a4801c000 tx=0x7f2a48015000).stop 2022-07-21T01:33:30.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 msgr2=0x7f2a540ac3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 --2- 172.21.15.3:0/2633731304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a540b31f0 0x7f2a540ac3c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2a4400c000 tx=0x7f2a44006d90).stop 2022-07-21T01:33:30.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a594cf700 1 -- 172.21.15.3:0/2633731304 reap_dead start 2022-07-21T01:33:30.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 shutdown_connections 2022-07-21T01:33:30.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 >> 172.21.15.3:0/2633731304 conn(0x7f2a5405f840 msgr2=0x7f2a5413faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 shutdown_connections 2022-07-21T01:33:30.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.738+0000 7f2a59cd0700 1 -- 172.21.15.3:0/2633731304 wait complete. 2022-07-21T01:33:30.746 INFO:teuthology.orchestra.run.smithi003.stdout:64424509445 2022-07-21T01:33:30.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2022-07-21T01:33:30.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.862+0000 7f9e4f59e700 1 Processor -- start 2022-07-21T01:33:30.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.862+0000 7f9e4f59e700 1 -- start start 2022-07-21T01:33:30.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9e500b0270 0x7f9e50143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e50067a10 con 0x7f9e500b2710 2022-07-21T01:33:30.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e50067cb0 con 0x7f9e500ac880 2022-07-21T01:33:30.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e500ad740 con 0x7f9e500b0270 2022-07-21T01:33:30.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4e59c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9e500b2710 0x7f9e501401e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51078/0 (socket says 172.21.15.3:51078) 2022-07-21T01:33:30.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4e59c700 1 -- 172.21.15.3:0/2272634387 learned_addr learned my addr 172.21.15.3:0/2272634387 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:30.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4dd9b700 1 --2- 172.21.15.3:0/2272634387 >> v2:172.21.15.143:3300/0 conn(0x7f9e500b0270 0x7f9e50143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4d59a700 1 -- 172.21.15.3:0/2272634387 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814726573 0 0) 0x7f9e50067a10 con 0x7f9e500b2710 2022-07-21T01:33:30.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4d59a700 1 -- 172.21.15.3:0/2272634387 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e34003260 con 0x7f9e500b2710 2022-07-21T01:33:30.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4dd9b700 1 -- 172.21.15.3:0/2272634387 >> v1:172.21.15.143:6789/0 conn(0x7f9e500ac880 legacy=0x7f9e501434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4dd9b700 1 -- 172.21.15.3:0/2272634387 >> v1:172.21.15.3:6789/0 conn(0x7f9e500b2710 legacy=0x7f9e501401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4dd9b700 1 -- 172.21.15.3:0/2272634387 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e50147160 con 0x7f9e500b0270 2022-07-21T01:33:30.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4dd9b700 1 --2- 172.21.15.3:0/2272634387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e500b0270 0x7f9e50143000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9e40002f70 tx=0x7f9e40006df0).ready entity=mon.1 client_cookie=dda095873cf3037e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4d59a700 1 -- 172.21.15.3:0/2272634387 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e4000d5c0 con 0x7f9e500b0270 2022-07-21T01:33:30.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4d59a700 1 -- 172.21.15.3:0/2272634387 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e40000e90 con 0x7f9e500b0270 2022-07-21T01:33:30.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4d59a700 1 -- 172.21.15.3:0/2272634387 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e4000d910 con 0x7f9e500b0270 2022-07-21T01:33:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 -- 172.21.15.3:0/2272634387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e500b0270 msgr2=0x7f9e50143000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 --2- 172.21.15.3:0/2272634387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e500b0270 0x7f9e50143000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9e40002f70 tx=0x7f9e40006df0).stop 2022-07-21T01:33:30.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 -- 172.21.15.3:0/2272634387 shutdown_connections 2022-07-21T01:33:30.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 --2- 172.21.15.3:0/2272634387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e500b0270 0x7f9e50143000 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 -- 172.21.15.3:0/2272634387 >> 172.21.15.3:0/2272634387 conn(0x7f9e50060310 msgr2=0x7f9e500b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 -- 172.21.15.3:0/2272634387 shutdown_connections 2022-07-21T01:33:30.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.866+0000 7f9e4f59e700 1 -- 172.21.15.3:0/2272634387 wait complete. 2022-07-21T01:33:30.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.870+0000 7f9e4f59e700 1 Processor -- start 2022-07-21T01:33:30.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.870+0000 7f9e4f59e700 1 -- start start 2022-07-21T01:33:30.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.870+0000 7f9e4f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e500ac880 0x7f9e50142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.870+0000 7f9e4f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e500b0270 0x7f9e50146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.870+0000 7f9e4f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e500b2710 0x7f9e501455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.870+0000 7f9e4f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e500abbf0 con 0x7f9e500ac880 2022-07-21T01:33:30.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.870+0000 7f9e4f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e500ac0c0 con 0x7f9e500b0270 2022-07-21T01:33:30.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.870+0000 7f9e4f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e501569a0 con 0x7f9e500b2710 2022-07-21T01:33:30.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.870+0000 7f9e4e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e500ac880 0x7f9e50142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e500ac880 0x7f9e50142170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42802/0 (socket says 172.21.15.3:42802) 2022-07-21T01:33:30.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4e59c700 1 -- 172.21.15.3:0/1523502620 learned_addr learned my addr 172.21.15.3:0/1523502620 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:30.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4dd9b700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e500b0270 0x7f9e50146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4ed9d700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e500b2710 0x7f9e501455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4e59c700 1 -- 172.21.15.3:0/1523502620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e500b2710 msgr2=0x7f9e501455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4e59c700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e500b2710 0x7f9e501455a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4e59c700 1 -- 172.21.15.3:0/1523502620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e500b0270 msgr2=0x7f9e50146360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4e59c700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e500b0270 0x7f9e50146360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4e59c700 1 -- 172.21.15.3:0/1523502620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4000b180 con 0x7f9e500ac880 2022-07-21T01:33:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4ed9d700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e500b2710 0x7f9e501455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:30.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4e59c700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e500ac880 0x7f9e50142170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9e44001110 tx=0x7f9e44007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e3f7fe700 1 -- 172.21.15.3:0/1523502620 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e440070d0 con 0x7f9e500ac880 2022-07-21T01:33:30.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e3f7fe700 1 -- 172.21.15.3:0/1523502620 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e44007230 con 0x7f9e500ac880 2022-07-21T01:33:30.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e500abe20 con 0x7f9e500ac880 2022-07-21T01:33:30.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e3f7fe700 1 -- 172.21.15.3:0/1523502620 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e44005610 con 0x7f9e500ac880 2022-07-21T01:33:30.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e50156620 con 0x7f9e500ac880 2022-07-21T01:33:30.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e3f7fe700 1 -- 172.21.15.3:0/1523502620 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9e44010070 con 0x7f9e500ac880 2022-07-21T01:33:30.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e501460c0 con 0x7f9e500ac880 2022-07-21T01:33:30.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e3f7fe700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e2c053a70 0x7f9e2c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e3f7fe700 1 -- 172.21.15.3:0/1523502620 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9e4406b630 con 0x7f9e500ac880 2022-07-21T01:33:30.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e3f7fe700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7f9e2c05b3f0 0x7f9e2c05d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:30.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e3f7fe700 1 -- 172.21.15.3:0/1523502620 --> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e2c05de70 con 0x7f9e2c05b3f0 2022-07-21T01:33:30.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4dd9b700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e2c053a70 0x7f9e2c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e3f7fe700 1 -- 172.21.15.3:0/1523502620 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e4406dcc0 con 0x7f9e500ac880 2022-07-21T01:33:30.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.874+0000 7f9e4ed9d700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7f9e2c05b3f0 0x7f9e2c05d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:30.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.878+0000 7f9e4dd9b700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e2c053a70 0x7f9e2c055f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9e4001a000 tx=0x7f9e40014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.878+0000 7f9e4ed9d700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7f9e2c05b3f0 0x7f9e2c05d890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:30.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.878+0000 7f9e3f7fe700 1 -- 172.21.15.3:0/1523502620 <== osd.3 v2:172.21.15.3:6814/22884 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f9e2c05de70 con 0x7f9e2c05b3f0 2022-07-21T01:33:30.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 --> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9e50143c00 con 0x7f9e2c05b3f0 2022-07-21T01:33:30.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e3f7fe700 1 -- 172.21.15.3:0/1523502620 <== osd.3 v2:172.21.15.3:6814/22884 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9e50143c00 con 0x7f9e2c05b3f0 2022-07-21T01:33:30.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7f9e2c05b3f0 msgr2=0x7f9e2c05d890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7f9e2c05b3f0 0x7f9e2c05d890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:30.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e2c053a70 msgr2=0x7f9e2c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e2c053a70 0x7f9e2c055f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9e4001a000 tx=0x7f9e40014000).stop 2022-07-21T01:33:30.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e500ac880 msgr2=0x7f9e50142170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:30.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 --2- 172.21.15.3:0/1523502620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e500ac880 0x7f9e50142170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9e44001110 tx=0x7f9e44007ad0).stop 2022-07-21T01:33:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4ed9d700 1 -- 172.21.15.3:0/1523502620 reap_dead start 2022-07-21T01:33:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 shutdown_connections 2022-07-21T01:33:30.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 >> 172.21.15.3:0/1523502620 conn(0x7f9e50060310 msgr2=0x7f9e500b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:30.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 shutdown_connections 2022-07-21T01:33:30.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:30.898+0000 7f9e4f59e700 1 -- 172.21.15.3:0/1523502620 wait complete. 2022-07-21T01:33:30.908 INFO:teuthology.orchestra.run.smithi003.stdout:64424509445 2022-07-21T01:33:30.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2022-07-21T01:33:31.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.022+0000 7fcb2e4a9700 1 Processor -- start 2022-07-21T01:33:31.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.022+0000 7fcb2e4a9700 1 -- start start 2022-07-21T01:33:31.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2e4a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 0x7fcb280b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2e4a9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcb2805c6f0 0x7fcb280a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2e4a9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb280aa630 con 0x7fcb28060330 2022-07-21T01:33:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2e4a9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb280b15e0 con 0x7fcb2805c6f0 2022-07-21T01:33:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2e4a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280af930 con 0x7fcb280af550 2022-07-21T01:33:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2cca6700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcb28060330 0x7fcb281445e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51094/0 (socket says 172.21.15.3:51094) 2022-07-21T01:33:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2cca6700 1 -- 172.21.15.3:0/1755077802 learned_addr learned my addr 172.21.15.3:0/1755077802 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:31.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2d4a7700 1 --2- 172.21.15.3:0/1755077802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 0x7fcb280b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2dca8700 1 --2- 172.21.15.3:0/1755077802 >> v2:172.21.15.143:3300/0 conn(0x7fcb2805c6f0 0x7fcb280a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb17fff700 1 -- 172.21.15.3:0/1755077802 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530321594 0 0) 0x7fcb280aa630 con 0x7fcb28060330 2022-07-21T01:33:31.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2d4a7700 1 -- 172.21.15.3:0/1755077802 >> v2:172.21.15.143:3300/0 conn(0x7fcb2805c6f0 msgr2=0x7fcb280a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:31.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2d4a7700 1 --2- 172.21.15.3:0/1755077802 >> v2:172.21.15.143:3300/0 conn(0x7fcb2805c6f0 0x7fcb280a8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2d4a7700 1 -- 172.21.15.3:0/1755077802 >> v1:172.21.15.3:6789/0 conn(0x7fcb28060330 legacy=0x7fcb281445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2d4a7700 1 -- 172.21.15.3:0/1755077802 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2805fd10 con 0x7fcb280af550 2022-07-21T01:33:31.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb2d4a7700 1 --2- 172.21.15.3:0/1755077802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 0x7fcb280b0e40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fcb24002f70 tx=0x7fcb24006df0).ready entity=mon.2 client_cookie=7fd6b432d247fb1e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb17fff700 1 -- 172.21.15.3:0/1755077802 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb2400d570 con 0x7fcb280af550 2022-07-21T01:33:31.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb17fff700 1 -- 172.21.15.3:0/1755077802 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb24000e90 con 0x7fcb280af550 2022-07-21T01:33:31.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.026+0000 7fcb17fff700 1 -- 172.21.15.3:0/1755077802 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb2400d8c0 con 0x7fcb280af550 2022-07-21T01:33:31.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1755077802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 msgr2=0x7fcb280b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 --2- 172.21.15.3:0/1755077802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 0x7fcb280b0e40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fcb24002f70 tx=0x7fcb24006df0).stop 2022-07-21T01:33:31.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1755077802 shutdown_connections 2022-07-21T01:33:31.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 --2- 172.21.15.3:0/1755077802 >> v2:172.21.15.143:3300/0 conn(0x7fcb2805c6f0 0x7fcb280a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 --2- 172.21.15.3:0/1755077802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 0x7fcb280b0e40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1755077802 >> 172.21.15.3:0/1755077802 conn(0x7fcb280612d0 msgr2=0x7fcb28062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1755077802 shutdown_connections 2022-07-21T01:33:31.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1755077802 wait complete. 2022-07-21T01:33:31.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 Processor -- start 2022-07-21T01:33:31.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.030+0000 7fcb2e4a9700 1 -- start start 2022-07-21T01:33:31.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2e4a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb2805c6f0 0x7fcb28146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2e4a9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb28060330 0x7fcb28067420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2e4a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 0x7fcb28152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2e4a9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280a89f0 con 0x7fcb28060330 2022-07-21T01:33:31.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2e4a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280af000 con 0x7fcb2805c6f0 2022-07-21T01:33:31.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2e4a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280b50e0 con 0x7fcb280af550 2022-07-21T01:33:31.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2cca6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb28060330 0x7fcb28067420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2cca6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb28060330 0x7fcb28067420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42820/0 (socket says 172.21.15.3:42820) 2022-07-21T01:33:31.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2cca6700 1 -- 172.21.15.3:0/1444773711 learned_addr learned my addr 172.21.15.3:0/1444773711 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:31.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2d4a7700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb2805c6f0 0x7fcb28146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2dca8700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 0x7fcb28152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2cca6700 1 -- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 msgr2=0x7fcb28152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2cca6700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb280af550 0x7fcb28152760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2cca6700 1 -- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb2805c6f0 msgr2=0x7fcb28146af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2cca6700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb2805c6f0 0x7fcb28146af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2cca6700 1 -- 172.21.15.3:0/1444773711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2400b180 con 0x7fcb28060330 2022-07-21T01:33:31.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2cca6700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb28060330 0x7fcb28067420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcb18000ed0 tx=0x7fcb18007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb167fc700 1 -- 172.21.15.3:0/1444773711 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb180070d0 con 0x7fcb28060330 2022-07-21T01:33:31.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb167fc700 1 -- 172.21.15.3:0/1444773711 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb18007230 con 0x7fcb28060330 2022-07-21T01:33:31.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb167fc700 1 -- 172.21.15.3:0/1444773711 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb18005610 con 0x7fcb28060330 2022-07-21T01:33:31.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2805f840 con 0x7fcb28060330 2022-07-21T01:33:31.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb28067910 con 0x7fcb28060330 2022-07-21T01:33:31.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb280b4f60 con 0x7fcb28060330 2022-07-21T01:33:31.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.034+0000 7fcb167fc700 1 -- 172.21.15.3:0/1444773711 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb18010070 con 0x7fcb28060330 2022-07-21T01:33:31.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb167fc700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 0x7fcb08055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb167fc700 1 -- 172.21.15.3:0/1444773711 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcb1806b680 con 0x7fcb28060330 2022-07-21T01:33:31.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb2d4a7700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 0x7fcb08055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb167fc700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7fcb0805b360 0x7fcb0805d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb167fc700 1 -- 172.21.15.3:0/1444773711 --> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb0805de80 con 0x7fcb0805b360 2022-07-21T01:33:31.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb167fc700 1 -- 172.21.15.3:0/1444773711 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb1806dd10 con 0x7fcb28060330 2022-07-21T01:33:31.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb2dca8700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7fcb0805b360 0x7fcb0805d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb2d4a7700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 0x7fcb08055ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcb2401a000 tx=0x7fcb24014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb2dca8700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7fcb0805b360 0x7fcb0805d800 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.038+0000 7fcb167fc700 1 -- 172.21.15.3:0/1444773711 <== osd.4 v2:172.21.15.143:6808/23339 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7fcb0805de80 con 0x7fcb0805b360 2022-07-21T01:33:31.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 --> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcb280ae9e0 con 0x7fcb0805b360 2022-07-21T01:33:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb167fc700 1 -- 172.21.15.3:0/1444773711 <== osd.4 v2:172.21.15.143:6808/23339 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcb20009da0 con 0x7fcb0805b360 2022-07-21T01:33:31.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7fcb0805b360 msgr2=0x7fcb0805d800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7fcb0805b360 0x7fcb0805d800 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 msgr2=0x7fcb08055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 0x7fcb08055ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcb2401a000 tx=0x7fcb24014000).stop 2022-07-21T01:33:31.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb28060330 msgr2=0x7fcb28067420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 --2- 172.21.15.3:0/1444773711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb28060330 0x7fcb28067420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcb18000ed0 tx=0x7fcb18007ac0).stop 2022-07-21T01:33:31.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2dca8700 1 -- 172.21.15.3:0/1444773711 reap_dead start 2022-07-21T01:33:31.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 shutdown_connections 2022-07-21T01:33:31.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 >> 172.21.15.3:0/1444773711 conn(0x7fcb280612d0 msgr2=0x7fcb280a8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 shutdown_connections 2022-07-21T01:33:31.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.058+0000 7fcb2e4a9700 1 -- 172.21.15.3:0/1444773711 wait complete. 2022-07-21T01:33:31.068 INFO:teuthology.orchestra.run.smithi003.stdout:60129542149 2022-07-21T01:33:31.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2022-07-21T01:33:31.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:33:31.158 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:33:31.161 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:33:31.164 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:33:31.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.178+0000 7f90d1043700 1 Processor -- start 2022-07-21T01:33:31.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.178+0000 7f90d1043700 1 -- start start 2022-07-21T01:33:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90cc0ad000 0x7f90cc1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc143ab0 con 0x7f90cc0b3260 2022-07-21T01:33:31.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0b0c80 con 0x7f90cc0b08a0 2022-07-21T01:33:31.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90cc0ae810 con 0x7f90cc0ad000 2022-07-21T01:33:31.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90cb7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f90cc0b3260 0x7f90cc13fa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51108/0 (socket says 172.21.15.3:51108) 2022-07-21T01:33:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90cb7fe700 1 -- 172.21.15.3:0/803429016 learned_addr learned my addr 172.21.15.3:0/803429016 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d0842700 1 --2- 172.21.15.3:0/803429016 >> v2:172.21.15.143:3300/0 conn(0x7f90cc0ad000 0x7f90cc1435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939120249 0 0) 0x7f90cc143ab0 con 0x7f90cc0b3260 2022-07-21T01:33:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b8003260 con 0x7f90cc0b3260 2022-07-21T01:33:31.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982599995 0 0) 0x7f90b8003260 con 0x7f90cc0b3260 2022-07-21T01:33:31.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 >> v2:172.21.15.143:3300/0 conn(0x7f90cc0ad000 msgr2=0x7f90cc1435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:31.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 --2- 172.21.15.3:0/803429016 >> v2:172.21.15.143:3300/0 conn(0x7f90cc0ad000 0x7f90cc1435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 >> v1:172.21.15.143:6789/0 conn(0x7f90cc0b08a0 legacy=0x7f90cc142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc0aeab0 con 0x7f90cc0b3260 2022-07-21T01:33:31.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f90bc001660 con 0x7f90cc0b3260 2022-07-21T01:33:31.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 >> v1:172.21.15.3:6789/0 conn(0x7f90cc0b3260 legacy=0x7f90cc13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b8003750 0x7f90b8003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90b8004500 0x7f90b8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b8004e70 0x7f90b800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc0ae810 con 0x7f90b8004e70 2022-07-21T01:33:31.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b800bd20 con 0x7f90b8003750 2022-07-21T01:33:31.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b800bee0 con 0x7f90b8004500 2022-07-21T01:33:31.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d0842700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b8004e70 0x7f90b800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90cb7fe700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90b8004500 0x7f90b8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90cbfff700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b8003750 0x7f90b8003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d0842700 1 -- 172.21.15.3:0/803429016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90b8004500 msgr2=0x7f90b8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d0842700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90b8004500 0x7f90b8004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d0842700 1 -- 172.21.15.3:0/803429016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b8003750 msgr2=0x7f90b8003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d0842700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b8003750 0x7f90b8003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d0842700 1 -- 172.21.15.3:0/803429016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b8003260 con 0x7f90b8004e70 2022-07-21T01:33:31.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90cbfff700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b8003750 0x7f90b8003b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:31.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d0842700 1 -- 172.21.15.3:0/803429016 reap_dead start 2022-07-21T01:33:31.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d0842700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b8004e70 0x7f90b800b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f90c400c000 tx=0x7f90c40091d0).ready entity=mon.0 client_cookie=be21b34db4818ec4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90c400e7e0 con 0x7f90b8004e70 2022-07-21T01:33:31.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c400e940 con 0x7f90b8004e70 2022-07-21T01:33:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90caffd700 1 -- 172.21.15.3:0/803429016 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90c400eca0 con 0x7f90b8004e70 2022-07-21T01:33:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 -- 172.21.15.3:0/803429016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b8004e70 msgr2=0x7f90b800b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b8004e70 0x7f90b800b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f90c400c000 tx=0x7f90c40091d0).stop 2022-07-21T01:33:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 -- 172.21.15.3:0/803429016 shutdown_connections 2022-07-21T01:33:31.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 --2- 172.21.15.3:0/803429016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90b8004e70 0x7f90b800b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 -- 172.21.15.3:0/803429016 >> 172.21.15.3:0/803429016 conn(0x7f90cc0612d0 msgr2=0x7f90cc062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 -- 172.21.15.3:0/803429016 shutdown_connections 2022-07-21T01:33:31.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.182+0000 7f90d1043700 1 -- 172.21.15.3:0/803429016 wait complete. 2022-07-21T01:33:31.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 Processor -- start 2022-07-21T01:33:31.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 -- start start 2022-07-21T01:33:31.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b8004e70 0x7f90cc0ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90cc066f60 0x7f90cc1407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc140ce0 0x7f90cc0a3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc0673c0 con 0x7f90cc140ce0 2022-07-21T01:33:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc0a4360 con 0x7f90b8004e70 2022-07-21T01:33:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc0a4600 con 0x7f90cc066f60 2022-07-21T01:33:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d0842700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc140ce0 0x7f90cc0a3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d0842700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc140ce0 0x7f90cc0a3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42844/0 (socket says 172.21.15.3:42844) 2022-07-21T01:33:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d0842700 1 -- 172.21.15.3:0/491134969 learned_addr learned my addr 172.21.15.3:0/491134969 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90cb7fe700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90cc066f60 0x7f90cc1407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90cbfff700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b8004e70 0x7f90cc0ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d0842700 1 -- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90cc066f60 msgr2=0x7f90cc1407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d0842700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90cc066f60 0x7f90cc1407f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d0842700 1 -- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b8004e70 msgr2=0x7f90cc0ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d0842700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90b8004e70 0x7f90cc0ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d0842700 1 -- 172.21.15.3:0/491134969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc0a48a0 con 0x7f90cc140ce0 2022-07-21T01:33:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90cb7fe700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90cc066f60 0x7f90cc1407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d0842700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc140ce0 0x7f90cc0a3330 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f90c400c000 tx=0x7f90c4019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90c97fa700 1 -- 172.21.15.3:0/491134969 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90c4009600 con 0x7f90cc140ce0 2022-07-21T01:33:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90c97fa700 1 -- 172.21.15.3:0/491134969 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c401de00 con 0x7f90cc140ce0 2022-07-21T01:33:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc0a4b40 con 0x7f90cc140ce0 2022-07-21T01:33:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90c97fa700 1 -- 172.21.15.3:0/491134969 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90c401b640 con 0x7f90cc140ce0 2022-07-21T01:33:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc141190 con 0x7f90cc140ce0 2022-07-21T01:33:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.186+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f90cc0673c0 con 0x7f90cc140ce0 2022-07-21T01:33:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90c97fa700 1 -- 172.21.15.3:0/491134969 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f90c4018020 con 0x7f90cc140ce0 2022-07-21T01:33:31.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90c97fa700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90ac0539e0 0x7f90ac055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90c97fa700 1 -- 172.21.15.3:0/491134969 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f90c406e0c0 con 0x7f90cc140ce0 2022-07-21T01:33:31.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90c97fa700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7f90ac05b360 0x7f90ac05d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90c97fa700 1 -- 172.21.15.3:0/491134969 --> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90ac05de80 con 0x7f90ac05b360 2022-07-21T01:33:31.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90c97fa700 1 -- 172.21.15.3:0/491134969 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90c406e4a0 con 0x7f90cc140ce0 2022-07-21T01:33:31.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90cbfff700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90ac0539e0 0x7f90ac055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90cb7fe700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7f90ac05b360 0x7f90ac05d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90cbfff700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90ac0539e0 0x7f90ac055ea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f90c000c000 tx=0x7f90c0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90cb7fe700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7f90ac05b360 0x7f90ac05d800 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.190+0000 7f90c97fa700 1 -- 172.21.15.3:0/491134969 <== osd.5 v2:172.21.15.143:6800/23340 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f90ac05de80 con 0x7f90ac05b360 2022-07-21T01:33:31.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.210+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 --> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f90cc143090 con 0x7f90ac05b360 2022-07-21T01:33:31.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.210+0000 7f90c97fa700 1 -- 172.21.15.3:0/491134969 <== osd.5 v2:172.21.15.143:6800/23340 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f90cc143090 con 0x7f90ac05b360 2022-07-21T01:33:31.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.210+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7f90ac05b360 msgr2=0x7f90ac05d800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.210+0000 7f90d1043700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7f90ac05b360 0x7f90ac05d800 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.214+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90ac0539e0 msgr2=0x7f90ac055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.214+0000 7f90d1043700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90ac0539e0 0x7f90ac055ea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f90c000c000 tx=0x7f90c0009040).stop 2022-07-21T01:33:31.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.214+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc140ce0 msgr2=0x7f90cc0a3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.214+0000 7f90d1043700 1 --2- 172.21.15.3:0/491134969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90cc140ce0 0x7f90cc0a3330 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f90c400c000 tx=0x7f90c4019000).stop 2022-07-21T01:33:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.214+0000 7f90d0842700 1 -- 172.21.15.3:0/491134969 reap_dead start 2022-07-21T01:33:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.214+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 shutdown_connections 2022-07-21T01:33:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.214+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 >> 172.21.15.3:0/491134969 conn(0x7f90cc0612d0 msgr2=0x7f90cc0ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.214+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 shutdown_connections 2022-07-21T01:33:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.214+0000 7f90d1043700 1 -- 172.21.15.3:0/491134969 wait complete. 2022-07-21T01:33:31.226 INFO:teuthology.orchestra.run.smithi003.stdout:60129542149 2022-07-21T01:33:31.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2022-07-21T01:33:31.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee5760700 1 Processor -- start 2022-07-21T01:33:31.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee5760700 1 -- start start 2022-07-21T01:33:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee5760700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcee00b2710 0x7fcee00abca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee5760700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcee00b13f0 0x7fcee00ac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee5760700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcee00b1920 0x7fcee00afd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee5760700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee00adf10 con 0x7fcee00b1920 2022-07-21T01:33:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee5760700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee00ae1b0 con 0x7fcee00b2710 2022-07-21T01:33:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee5760700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00a9520 con 0x7fcee00b13f0 2022-07-21T01:33:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee4f5f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcee00b1920 0x7fcee00afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee4f5f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcee00b1920 0x7fcee00afd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42854/0 (socket says 172.21.15.3:42854) 2022-07-21T01:33:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcedffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcee00b2710 0x7fcee00abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee4f5f700 1 -- 172.21.15.3:0/598345731 learned_addr learned my addr 172.21.15.3:0/598345731 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcedf7fe700 1 --2- 172.21.15.3:0/598345731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcee00b13f0 0x7fcee00ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee4f5f700 1 -- 172.21.15.3:0/598345731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcee00b13f0 msgr2=0x7fcee00ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee4f5f700 1 --2- 172.21.15.3:0/598345731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcee00b13f0 0x7fcee00ac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee4f5f700 1 -- 172.21.15.3:0/598345731 >> v2:172.21.15.143:3300/0 conn(0x7fcee00b2710 msgr2=0x7fcee00abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee4f5f700 1 --2- 172.21.15.3:0/598345731 >> v2:172.21.15.143:3300/0 conn(0x7fcee00b2710 0x7fcee00abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee4f5f700 1 -- 172.21.15.3:0/598345731 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee00a99c0 con 0x7fcee00b1920 2022-07-21T01:33:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.386+0000 7fcee4f5f700 1 --2- 172.21.15.3:0/598345731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 0x7fcee00afd60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcee0067bc0 tx=0x7fced800b530).ready entity=mon.0 client_cookie=21a24505423a283e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcedeffd700 1 -- 172.21.15.3:0/598345731 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fced800dd30 con 0x7fcee00b1920 2022-07-21T01:33:31.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcedeffd700 1 -- 172.21.15.3:0/598345731 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced800de90 con 0x7fcee00b1920 2022-07-21T01:33:31.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcedeffd700 1 -- 172.21.15.3:0/598345731 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fced8010660 con 0x7fcee00b1920 2022-07-21T01:33:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 -- 172.21.15.3:0/598345731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 msgr2=0x7fcee00afd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 --2- 172.21.15.3:0/598345731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 0x7fcee00afd60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcee0067bc0 tx=0x7fced800b530).stop 2022-07-21T01:33:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 -- 172.21.15.3:0/598345731 shutdown_connections 2022-07-21T01:33:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 --2- 172.21.15.3:0/598345731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 0x7fcee00afd60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 --2- 172.21.15.3:0/598345731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcee00b13f0 0x7fcee00ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 --2- 172.21.15.3:0/598345731 >> v2:172.21.15.143:3300/0 conn(0x7fcee00b2710 0x7fcee00abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 -- 172.21.15.3:0/598345731 >> 172.21.15.3:0/598345731 conn(0x7fcee0060310 msgr2=0x7fcee00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 -- 172.21.15.3:0/598345731 shutdown_connections 2022-07-21T01:33:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 -- 172.21.15.3:0/598345731 wait complete. 2022-07-21T01:33:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 Processor -- start 2022-07-21T01:33:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.390+0000 7fcee5760700 1 -- start start 2022-07-21T01:33:31.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcee5760700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcee00b13f0 0x7fcee009ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcee5760700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 0x7fcee01514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcee5760700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcee00b2710 0x7fcee01519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcee5760700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00ac480 con 0x7fcee00b1920 2022-07-21T01:33:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcee5760700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00ac1e0 con 0x7fcee00b2710 2022-07-21T01:33:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcee5760700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00a1870 con 0x7fcee00b13f0 2022-07-21T01:33:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 0x7fcee01514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 0x7fcee01514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42858/0 (socket says 172.21.15.3:42858) 2022-07-21T01:33:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedf7fe700 1 -- 172.21.15.3:0/2888509479 learned_addr learned my addr 172.21.15.3:0/2888509479 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:31.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedffff700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcee00b13f0 0x7fcee009ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcee4f5f700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcee00b2710 0x7fcee01519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedf7fe700 1 -- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcee00b13f0 msgr2=0x7fcee009ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedf7fe700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcee00b13f0 0x7fcee009ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedf7fe700 1 -- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcee00b2710 msgr2=0x7fcee01519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedf7fe700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcee00b2710 0x7fcee01519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedf7fe700 1 -- 172.21.15.3:0/2888509479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced800b7d0 con 0x7fcee00b1920 2022-07-21T01:33:31.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedf7fe700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 0x7fcee01514d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fced400c000 tx=0x7fced4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedd7fa700 1 -- 172.21.15.3:0/2888509479 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fced4010780 con 0x7fcee00b1920 2022-07-21T01:33:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedd7fa700 1 -- 172.21.15.3:0/2888509479 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced40108e0 con 0x7fcee00b1920 2022-07-21T01:33:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee00a1b10 con 0x7fcee00b1920 2022-07-21T01:33:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcedd7fa700 1 -- 172.21.15.3:0/2888509479 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fced4010c40 con 0x7fcee00b1920 2022-07-21T01:33:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.394+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee00a8a00 con 0x7fcee00b1920 2022-07-21T01:33:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcee00a22f0 con 0x7fcee00b1920 2022-07-21T01:33:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcedd7fa700 1 -- 172.21.15.3:0/2888509479 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fced401c460 con 0x7fcee00b1920 2022-07-21T01:33:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcedd7fa700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcec0053950 0x7fcec0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcedd7fa700 1 -- 172.21.15.3:0/2888509479 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fced406c4a0 con 0x7fcee00b1920 2022-07-21T01:33:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcedffff700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcec0053950 0x7fcec0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcedd7fa700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fcec005b2f0 0x7fcec005d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcedd7fa700 1 -- 172.21.15.3:0/2888509479 --> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcec005de10 con 0x7fcec005b2f0 2022-07-21T01:33:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcedd7fa700 1 -- 172.21.15.3:0/2888509479 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fced40430f0 con 0x7fcee00b1920 2022-07-21T01:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcee4f5f700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fcec005b2f0 0x7fcec005d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcedffff700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcec0053950 0x7fcec0055e10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fced000b000 tx=0x7fced0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.398+0000 7fcee4f5f700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fcec005b2f0 0x7fcec005d790 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.402+0000 7fcedd7fa700 1 -- 172.21.15.3:0/2888509479 <== osd.6 v2:172.21.15.143:6824/23342 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7fcec005de10 con 0x7fcec005b2f0 2022-07-21T01:33:31.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 --> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcee00abf20 con 0x7fcec005b2f0 2022-07-21T01:33:31.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcedd7fa700 1 -- 172.21.15.3:0/2888509479 <== osd.6 v2:172.21.15.143:6824/23342 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcee00abf20 con 0x7fcec005b2f0 2022-07-21T01:33:31.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fcec005b2f0 msgr2=0x7fcec005d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fcec005b2f0 0x7fcec005d790 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcec0053950 msgr2=0x7fcec0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcec0053950 0x7fcec0055e10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fced000b000 tx=0x7fced0006df0).stop 2022-07-21T01:33:31.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 msgr2=0x7fcee01514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 --2- 172.21.15.3:0/2888509479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcee00b1920 0x7fcee01514d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fced400c000 tx=0x7fced4006df0).stop 2022-07-21T01:33:31.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee4f5f700 1 -- 172.21.15.3:0/2888509479 reap_dead start 2022-07-21T01:33:31.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 shutdown_connections 2022-07-21T01:33:31.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 >> 172.21.15.3:0/2888509479 conn(0x7fcee0060310 msgr2=0x7fcee00b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 shutdown_connections 2022-07-21T01:33:31.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.418+0000 7fcee5760700 1 -- 172.21.15.3:0/2888509479 wait complete. 2022-07-21T01:33:31.433 INFO:teuthology.orchestra.run.smithi003.stdout:60129542149 2022-07-21T01:33:31.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2022-07-21T01:33:31.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6923b94700 1 Processor -- start 2022-07-21T01:33:31.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6923b94700 1 -- start start 2022-07-21T01:33:31.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6923b94700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 0x7f691c14a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6923b94700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0b6e70 con 0x7f691c060aa0 2022-07-21T01:33:31.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6923b94700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c061370 con 0x7f691c0b6a90 2022-07-21T01:33:31.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6923b94700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c061610 con 0x7f691c0acee0 2022-07-21T01:33:31.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f692112f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f691c060aa0 0x7f691c0aa760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51148/0 (socket says 172.21.15.3:51148) 2022-07-21T01:33:31.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f692112f700 1 -- 172.21.15.3:0/536284130 learned_addr learned my addr 172.21.15.3:0/536284130 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:31.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6922131700 1 --2- 172.21.15.3:0/536284130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 0x7f691c14a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f692092e700 1 -- 172.21.15.3:0/536284130 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880525975 0 0) 0x7f691c0b6e70 con 0x7f691c060aa0 2022-07-21T01:33:31.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f692092e700 1 -- 172.21.15.3:0/536284130 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6904003260 con 0x7f691c060aa0 2022-07-21T01:33:31.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f692092e700 1 -- 172.21.15.3:0/536284130 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42902968 0 0) 0x7f691c061370 con 0x7f691c0b6a90 2022-07-21T01:33:31.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f692092e700 1 -- 172.21.15.3:0/536284130 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c0b6e70 con 0x7f691c0b6a90 2022-07-21T01:33:31.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6922131700 1 -- 172.21.15.3:0/536284130 >> v1:172.21.15.143:6789/0 conn(0x7f691c0b6a90 legacy=0x7f691c0aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6922131700 1 -- 172.21.15.3:0/536284130 >> v1:172.21.15.3:6789/0 conn(0x7f691c060aa0 legacy=0x7f691c0aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6922131700 1 -- 172.21.15.3:0/536284130 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c0618b0 con 0x7f691c0acee0 2022-07-21T01:33:31.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.554+0000 7f6922131700 1 --2- 172.21.15.3:0/536284130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 0x7f691c14a3d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f69140032f0 tx=0x7f691400a370).ready entity=mon.2 client_cookie=d609b9b258d97123 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f692092e700 1 -- 172.21.15.3:0/536284130 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691400cba0 con 0x7f691c0acee0 2022-07-21T01:33:31.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f692092e700 1 -- 172.21.15.3:0/536284130 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691400cd00 con 0x7f691c0acee0 2022-07-21T01:33:31.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f692092e700 1 -- 172.21.15.3:0/536284130 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6914011860 con 0x7f691c0acee0 2022-07-21T01:33:31.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f6923b94700 1 -- 172.21.15.3:0/536284130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 msgr2=0x7f691c14a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f6923b94700 1 --2- 172.21.15.3:0/536284130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 0x7f691c14a3d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f69140032f0 tx=0x7f691400a370).stop 2022-07-21T01:33:31.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f6923b94700 1 -- 172.21.15.3:0/536284130 shutdown_connections 2022-07-21T01:33:31.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f6923b94700 1 --2- 172.21.15.3:0/536284130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 0x7f691c14a3d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f6923b94700 1 -- 172.21.15.3:0/536284130 >> 172.21.15.3:0/536284130 conn(0x7f691c0b3610 msgr2=0x7f691c0b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f6923b94700 1 -- 172.21.15.3:0/536284130 shutdown_connections 2022-07-21T01:33:31.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.558+0000 7f6923b94700 1 -- 172.21.15.3:0/536284130 wait complete. 2022-07-21T01:33:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 Processor -- start 2022-07-21T01:33:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 -- start start 2022-07-21T01:33:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f691c060aa0 0x7f691c13eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 0x7f691c136c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f691c0b6a90 0x7f691c133240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0b1aa0 con 0x7f691c060aa0 2022-07-21T01:33:31.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0b2180 con 0x7f691c0b6a90 2022-07-21T01:33:31.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0b1800 con 0x7f691c0acee0 2022-07-21T01:33:31.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6921930700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f691c060aa0 0x7f691c13eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6921930700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f691c060aa0 0x7f691c13eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42872/0 (socket says 172.21.15.3:42872) 2022-07-21T01:33:31.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6921930700 1 -- 172.21.15.3:0/650904776 learned_addr learned my addr 172.21.15.3:0/650904776 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:31.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f692112f700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 0x7f691c136c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6922131700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f691c0b6a90 0x7f691c133240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6921930700 1 -- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 msgr2=0x7f691c136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6921930700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 0x7f691c136c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6921930700 1 -- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f691c0b6a90 msgr2=0x7f691c133240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6921930700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f691c0b6a90 0x7f691c133240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6921930700 1 -- 172.21.15.3:0/650904776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691400a610 con 0x7f691c060aa0 2022-07-21T01:33:31.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f692112f700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f691c0acee0 0x7f691c136c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6921930700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f691c060aa0 0x7f691c13eac0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6918001110 tx=0x7f6918007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f690affd700 1 -- 172.21.15.3:0/650904776 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691800f470 con 0x7f691c060aa0 2022-07-21T01:33:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f690affd700 1 -- 172.21.15.3:0/650904776 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691800f5d0 con 0x7f691c060aa0 2022-07-21T01:33:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f690affd700 1 -- 172.21.15.3:0/650904776 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691800f930 con 0x7f691c060aa0 2022-07-21T01:33:31.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c0b23e0 con 0x7f691c060aa0 2022-07-21T01:33:31.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.562+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c13da40 con 0x7f691c060aa0 2022-07-21T01:33:31.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f691c0b2000 con 0x7f691c060aa0 2022-07-21T01:33:31.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f690affd700 1 -- 172.21.15.3:0/650904776 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6918019030 con 0x7f691c060aa0 2022-07-21T01:33:31.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f690affd700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68fc053b00 0x7f68fc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f690affd700 1 -- 172.21.15.3:0/650904776 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f691806b750 con 0x7f691c060aa0 2022-07-21T01:33:31.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f692112f700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68fc053b00 0x7f68fc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f690affd700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f68fc05b400 0x7f68fc05d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f690affd700 1 -- 172.21.15.3:0/650904776 --> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68fc05de80 con 0x7f68fc05b400 2022-07-21T01:33:31.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f690affd700 1 -- 172.21.15.3:0/650904776 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f691806dcd0 con 0x7f691c060aa0 2022-07-21T01:33:31.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f6922131700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f68fc05b400 0x7f68fc05d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f692112f700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68fc053b00 0x7f68fc055fc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f690c00b000 tx=0x7f690c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.566+0000 7f6922131700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f68fc05b400 0x7f68fc05d8a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.570+0000 7f690affd700 1 -- 172.21.15.3:0/650904776 <== osd.7 v2:172.21.15.143:6816/23341 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f68fc05de80 con 0x7f68fc05b400 2022-07-21T01:33:31.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 --> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f691c0b2570 con 0x7f68fc05b400 2022-07-21T01:33:31.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f690affd700 1 -- 172.21.15.3:0/650904776 <== osd.7 v2:172.21.15.143:6816/23341 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f691c0b2570 con 0x7f68fc05b400 2022-07-21T01:33:31.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f68fc05b400 msgr2=0x7f68fc05d8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f68fc05b400 0x7f68fc05d8a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68fc053b00 msgr2=0x7f68fc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68fc053b00 0x7f68fc055fc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f690c00b000 tx=0x7f690c009040).stop 2022-07-21T01:33:31.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f691c060aa0 msgr2=0x7f691c13eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 --2- 172.21.15.3:0/650904776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f691c060aa0 0x7f691c13eac0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6918001110 tx=0x7f6918007ae0).stop 2022-07-21T01:33:31.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6922131700 1 -- 172.21.15.3:0/650904776 reap_dead start 2022-07-21T01:33:31.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 shutdown_connections 2022-07-21T01:33:31.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 >> 172.21.15.3:0/650904776 conn(0x7f691c0b3610 msgr2=0x7f691c0ac000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 shutdown_connections 2022-07-21T01:33:31.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.590+0000 7f6923b94700 1 -- 172.21.15.3:0/650904776 wait complete. 2022-07-21T01:33:31.599 INFO:teuthology.orchestra.run.smithi003.stdout:60129542150 2022-07-21T01:33:31.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-07-21T01:33:31.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 Processor -- start 2022-07-21T01:33:31.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 -- start start 2022-07-21T01:33:31.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb0200b31d0 0x7fb020066020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb0200a9da0 0x7fb020060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb020067030 con 0x7fb0201434b0 2022-07-21T01:33:31.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb020064a70 con 0x7fb0200b31d0 2022-07-21T01:33:31.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb020061f60 con 0x7fb0200a9da0 2022-07-21T01:33:31.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01ed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb0201434b0 0x7fb020143900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51166/0 (socket says 172.21.15.3:51166) 2022-07-21T01:33:31.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01ed9d700 1 -- 172.21.15.3:0/1152104323 learned_addr learned my addr 172.21.15.3:0/1152104323 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:31.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01e59c700 1 --2- 172.21.15.3:0/1152104323 >> v2:172.21.15.3:3300/0 conn(0x7fb0200b31d0 0x7fb020066020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01dd9b700 1 --2- 172.21.15.3:0/1152104323 >> v2:172.21.15.143:3300/0 conn(0x7fb0200a9da0 0x7fb020060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01d59a700 1 -- 172.21.15.3:0/1152104323 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517671546 0 0) 0x7fb020067030 con 0x7fb0201434b0 2022-07-21T01:33:31.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01d59a700 1 -- 172.21.15.3:0/1152104323 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb004003260 con 0x7fb0201434b0 2022-07-21T01:33:31.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01e59c700 1 -- 172.21.15.3:0/1152104323 >> v2:172.21.15.143:3300/0 conn(0x7fb0200a9da0 msgr2=0x7fb020060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:31.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01d59a700 1 -- 172.21.15.3:0/1152104323 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4171823700 0 0) 0x7fb004003260 con 0x7fb0201434b0 2022-07-21T01:33:31.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01e59c700 1 --2- 172.21.15.3:0/1152104323 >> v2:172.21.15.143:3300/0 conn(0x7fb0200a9da0 0x7fb020060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01e59c700 1 -- 172.21.15.3:0/1152104323 >> v1:172.21.15.3:6789/0 conn(0x7fb0201434b0 legacy=0x7fb020143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01e59c700 1 -- 172.21.15.3:0/1152104323 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0200b35b0 con 0x7fb0200b31d0 2022-07-21T01:33:31.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01e59c700 1 --2- 172.21.15.3:0/1152104323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200b31d0 0x7fb020066020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fb018001b80 tx=0x7fb014006df0).ready entity=mon.0 client_cookie=4f5a97b4cbd2d151 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01d59a700 1 -- 172.21.15.3:0/1152104323 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0180019e0 con 0x7fb0200b31d0 2022-07-21T01:33:31.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01d59a700 1 -- 172.21.15.3:0/1152104323 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb014000e90 con 0x7fb0200b31d0 2022-07-21T01:33:31.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01d59a700 1 -- 172.21.15.3:0/1152104323 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb014012460 con 0x7fb0200b31d0 2022-07-21T01:33:31.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 -- 172.21.15.3:0/1152104323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200b31d0 msgr2=0x7fb020066020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 --2- 172.21.15.3:0/1152104323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200b31d0 0x7fb020066020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fb018001b80 tx=0x7fb014006df0).stop 2022-07-21T01:33:31.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 -- 172.21.15.3:0/1152104323 shutdown_connections 2022-07-21T01:33:31.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 --2- 172.21.15.3:0/1152104323 >> v2:172.21.15.143:3300/0 conn(0x7fb0200a9da0 0x7fb020060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 --2- 172.21.15.3:0/1152104323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200b31d0 0x7fb020066020 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 -- 172.21.15.3:0/1152104323 >> 172.21.15.3:0/1152104323 conn(0x7fb020063870 msgr2=0x7fb020063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 -- 172.21.15.3:0/1152104323 shutdown_connections 2022-07-21T01:33:31.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.722+0000 7fb01f59e700 1 -- 172.21.15.3:0/1152104323 wait complete. 2022-07-21T01:33:31.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01f59e700 1 Processor -- start 2022-07-21T01:33:31.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01f59e700 1 -- start start 2022-07-21T01:33:31.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200a9da0 0x7fb020153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0200b31d0 0x7fb0200b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0201434b0 0x7fb02014f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb020138b00 con 0x7fb0200a9da0 2022-07-21T01:33:31.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02013d810 con 0x7fb0201434b0 2022-07-21T01:33:31.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb020152720 con 0x7fb0200b31d0 2022-07-21T01:33:31.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200a9da0 0x7fb020153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200a9da0 0x7fb020153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42888/0 (socket says 172.21.15.3:42888) 2022-07-21T01:33:31.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01e59c700 1 -- 172.21.15.3:0/3493209461 learned_addr learned my addr 172.21.15.3:0/3493209461 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:31.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01ed9d700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0201434b0 0x7fb02014f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01dd9b700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0200b31d0 0x7fb0200b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01e59c700 1 -- 172.21.15.3:0/3493209461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0200b31d0 msgr2=0x7fb0200b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01e59c700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0200b31d0 0x7fb0200b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01e59c700 1 -- 172.21.15.3:0/3493209461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0201434b0 msgr2=0x7fb02014f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01e59c700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0201434b0 0x7fb02014f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01e59c700 1 -- 172.21.15.3:0/3493209461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01400b180 con 0x7fb0200a9da0 2022-07-21T01:33:31.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01ed9d700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0201434b0 0x7fb02014f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:31.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.726+0000 7fb01e59c700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200a9da0 0x7fb020153100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fb0201397a0 tx=0x7fb014014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb00f7fe700 1 -- 172.21.15.3:0/3493209461 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb014016030 con 0x7fb0200a9da0 2022-07-21T01:33:31.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb00f7fe700 1 -- 172.21.15.3:0/3493209461 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01400be90 con 0x7fb0200a9da0 2022-07-21T01:33:31.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0201529c0 con 0x7fb0200a9da0 2022-07-21T01:33:31.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb00f7fe700 1 -- 172.21.15.3:0/3493209461 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb014012460 con 0x7fb0200a9da0 2022-07-21T01:33:31.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0200b5420 con 0x7fb0200a9da0 2022-07-21T01:33:31.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb020066af0 con 0x7fb0200a9da0 2022-07-21T01:33:31.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb00f7fe700 1 -- 172.21.15.3:0/3493209461 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb014024020 con 0x7fb0200a9da0 2022-07-21T01:33:31.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb00f7fe700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faffc053a70 0x7faffc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:31.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb00f7fe700 1 -- 172.21.15.3:0/3493209461 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb014017070 con 0x7fb0200a9da0 2022-07-21T01:33:31.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb01dd9b700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faffc053a70 0x7faffc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:31.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.730+0000 7fb01dd9b700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faffc053a70 0x7faffc055f30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb01000b000 tx=0x7fb010008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:31.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.738+0000 7fb00f7fe700 1 -- 172.21.15.3:0/3493209461 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb014043cc0 con 0x7fb0200a9da0 2022-07-21T01:33:31.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb0200574e0 con 0x7fb0200a9da0 2022-07-21T01:33:31.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb00f7fe700 1 -- 172.21.15.3:0/3493209461 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb0140472e0 con 0x7fb0200a9da0 2022-07-21T01:33:31.863 INFO:teuthology.orchestra.run.smithi003.stdout:64424509445 2022-07-21T01:33:31.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faffc053a70 msgr2=0x7faffc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faffc053a70 0x7faffc055f30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb01000b000 tx=0x7fb010008040).stop 2022-07-21T01:33:31.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200a9da0 msgr2=0x7fb020153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:31.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200a9da0 0x7fb020153100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fb0201397a0 tx=0x7fb014014000).stop 2022-07-21T01:33:31.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 shutdown_connections 2022-07-21T01:33:31.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faffc053a70 0x7faffc055f30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0201434b0 0x7fb02014f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0200b31d0 0x7fb0200b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 --2- 172.21.15.3:0/3493209461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0200a9da0 0x7fb020153100 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:31.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 >> 172.21.15.3:0/3493209461 conn(0x7fb020063870 msgr2=0x7fb020065f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:31.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 shutdown_connections 2022-07-21T01:33:31.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:31.866+0000 7fb01f59e700 1 -- 172.21.15.3:0/3493209461 wait complete. 2022-07-21T01:33:31.876 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.0 2022-07-21T01:33:32.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-07-21T01:33:32.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.994+0000 7f89be1a2700 1 Processor -- start 2022-07-21T01:33:32.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.994+0000 7f89be1a2700 1 -- start start 2022-07-21T01:33:32.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 0x7f89b8060ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:32.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89b80b2730 0x7f89b8061190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:32.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f89b80666b0 0x7f89b8066b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:32.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b8062190 con 0x7f89b80b2730 2022-07-21T01:33:32.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b80654c0 con 0x7f89b80666b0 2022-07-21T01:33:32.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80b3570 con 0x7f89b80b3190 2022-07-21T01:33:32.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bc99f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89b80b2730 0x7f89b8061190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:32.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bd9a1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f89b80666b0 0x7f89b8066b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:32.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bd1a0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 0x7f89b8060ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:32.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bc99f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89b80b2730 0x7f89b8061190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42896/0 (socket says 172.21.15.3:42896) 2022-07-21T01:33:32.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bc99f700 1 -- 172.21.15.3:0/912243703 learned_addr learned my addr 172.21.15.3:0/912243703 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:32.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bd9a1700 1 -- 172.21.15.3:0/912243703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 msgr2=0x7f89b8060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:32.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bd9a1700 1 --2- 172.21.15.3:0/912243703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 0x7f89b8060ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:32.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bd9a1700 1 -- 172.21.15.3:0/912243703 >> v2:172.21.15.3:3300/0 conn(0x7f89b80b2730 msgr2=0x7f89b8061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:32.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bd9a1700 1 --2- 172.21.15.3:0/912243703 >> v2:172.21.15.3:3300/0 conn(0x7f89b80b2730 0x7f89b8061190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:32.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bd9a1700 1 -- 172.21.15.3:0/912243703 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b805fbf0 con 0x7f89b80666b0 2022-07-21T01:33:33.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bc99f700 1 --2- 172.21.15.3:0/912243703 >> v2:172.21.15.3:3300/0 conn(0x7f89b80b2730 0x7f89b8061190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:33.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89bd9a1700 1 --2- 172.21.15.3:0/912243703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89b80666b0 0x7f89b8066b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f89b00032f0 tx=0x7f89b00091d0).ready entity=mon.1 client_cookie=bf04fd8945317760 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89a7fff700 1 -- 172.21.15.3:0/912243703 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89b000b600 con 0x7f89b80666b0 2022-07-21T01:33:33.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89a7fff700 1 -- 172.21.15.3:0/912243703 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b000b760 con 0x7f89b80666b0 2022-07-21T01:33:33.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89a7fff700 1 -- 172.21.15.3:0/912243703 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89b000b8d0 con 0x7f89b80666b0 2022-07-21T01:33:33.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 -- 172.21.15.3:0/912243703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89b80666b0 msgr2=0x7f89b8066b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 --2- 172.21.15.3:0/912243703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89b80666b0 0x7f89b8066b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f89b00032f0 tx=0x7f89b00091d0).stop 2022-07-21T01:33:33.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 -- 172.21.15.3:0/912243703 shutdown_connections 2022-07-21T01:33:33.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 --2- 172.21.15.3:0/912243703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89b80666b0 0x7f89b8066b20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 --2- 172.21.15.3:0/912243703 >> v2:172.21.15.3:3300/0 conn(0x7f89b80b2730 0x7f89b8061190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 --2- 172.21.15.3:0/912243703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 0x7f89b8060ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 -- 172.21.15.3:0/912243703 >> 172.21.15.3:0/912243703 conn(0x7f89b8063870 msgr2=0x7f89b8063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:33.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 -- 172.21.15.3:0/912243703 shutdown_connections 2022-07-21T01:33:33.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:32.998+0000 7f89be1a2700 1 -- 172.21.15.3:0/912243703 wait complete. 2022-07-21T01:33:33.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89be1a2700 1 Processor -- start 2022-07-21T01:33:33.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89be1a2700 1 -- start start 2022-07-21T01:33:33.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89be1a2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89b80666b0 0x7f89b80ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89be1a2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b80b2730 0x7f89b80ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89be1a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 0x7f89b81526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89be1a2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80b0aa0 con 0x7f89b80b2730 2022-07-21T01:33:33.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89be1a2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80b0560 con 0x7f89b80666b0 2022-07-21T01:33:33.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89be1a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80b0800 con 0x7f89b80b3190 2022-07-21T01:33:33.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bc99f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b80b2730 0x7f89b80ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bc99f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b80b2730 0x7f89b80ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42902/0 (socket says 172.21.15.3:42902) 2022-07-21T01:33:33.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bc99f700 1 -- 172.21.15.3:0/1844450596 learned_addr learned my addr 172.21.15.3:0/1844450596 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:33.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bd1a0700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89b80666b0 0x7f89b80ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bd9a1700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 0x7f89b81526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bc99f700 1 -- 172.21.15.3:0/1844450596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 msgr2=0x7f89b81526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bc99f700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 0x7f89b81526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bc99f700 1 -- 172.21.15.3:0/1844450596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89b80666b0 msgr2=0x7f89b80ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bc99f700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89b80666b0 0x7f89b80ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bc99f700 1 -- 172.21.15.3:0/1844450596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a800b1e0 con 0x7f89b80b2730 2022-07-21T01:33:33.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.002+0000 7f89bc99f700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b80b2730 0x7f89b80ac710 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f89a800c000 tx=0x7f89a8006eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89a67fc700 1 -- 172.21.15.3:0/1844450596 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89a8012600 con 0x7f89b80b2730 2022-07-21T01:33:33.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89a67fc700 1 -- 172.21.15.3:0/1844450596 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a8012760 con 0x7f89b80b2730 2022-07-21T01:33:33.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b0009470 con 0x7f89b80b2730 2022-07-21T01:33:33.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89a67fc700 1 -- 172.21.15.3:0/1844450596 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89a8012ac0 con 0x7f89b80b2730 2022-07-21T01:33:33.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b8148dd0 con 0x7f89b80b2730 2022-07-21T01:33:33.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89a67fc700 1 -- 172.21.15.3:0/1844450596 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f89a8012c20 con 0x7f89b80b2730 2022-07-21T01:33:33.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b805fe90 con 0x7f89b80b2730 2022-07-21T01:33:33.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89a67fc700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8998053b30 0x7f8998055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89bd1a0700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8998053b30 0x7f8998055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89a67fc700 1 -- 172.21.15.3:0/1844450596 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f89a806c600 con 0x7f89b80b2730 2022-07-21T01:33:33.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.006+0000 7f89bd1a0700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8998053b30 0x7f8998055ff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f89b400b000 tx=0x7f89b4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.010+0000 7f89a67fc700 1 -- 172.21.15.3:0/1844450596 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f89a80432b0 con 0x7f89b80b2730 2022-07-21T01:33:33.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.134+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f89b80574e0 con 0x7f89b80b2730 2022-07-21T01:33:33.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.134+0000 7f89a67fc700 1 -- 172.21.15.3:0/1844450596 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f89a80468d0 con 0x7f89b80b2730 2022-07-21T01:33:33.133 INFO:teuthology.orchestra.run.smithi003.stdout:64424509447 2022-07-21T01:33:33.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8998053b30 msgr2=0x7f8998055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8998053b30 0x7f8998055ff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f89b400b000 tx=0x7f89b4006df0).stop 2022-07-21T01:33:33.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b80b2730 msgr2=0x7f89b80ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b80b2730 0x7f89b80ac710 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f89a800c000 tx=0x7f89a8006eb0).stop 2022-07-21T01:33:33.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 shutdown_connections 2022-07-21T01:33:33.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89b80b3190 0x7f89b81526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89b80b2730 0x7f89b80ac710 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8998053b30 0x7f8998055ff0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 --2- 172.21.15.3:0/1844450596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89b80666b0 0x7f89b80ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 >> 172.21.15.3:0/1844450596 conn(0x7f89b8063870 msgr2=0x7f89b80647a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:33.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 shutdown_connections 2022-07-21T01:33:33.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.138+0000 7f89be1a2700 1 -- 172.21.15.3:0/1844450596 wait complete. 2022-07-21T01:33:33.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.0 2022-07-21T01:33:33.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-07-21T01:33:33.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.258+0000 7f0283086700 1 Processor -- start 2022-07-21T01:33:33.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.258+0000 7f0283086700 1 -- start start 2022-07-21T01:33:33.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.258+0000 7f0283086700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c0b1f40 0x7f027c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.258+0000 7f0283086700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f027c148530 0x7f027c0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.258+0000 7f0283086700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c0b04e0 con 0x7f027c0a7250 2022-07-21T01:33:33.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.258+0000 7f0283086700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027c0a9250 con 0x7f027c148530 2022-07-21T01:33:33.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.258+0000 7f0283086700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0b0780 con 0x7f027c0b1f40 2022-07-21T01:33:33.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f027c148530 0x7f027c0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0280e22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c0b1f40 0x7f027c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0281623700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f027c0a7250 0x7f027c1448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:53306/0 (socket says 172.21.15.3:53306) 2022-07-21T01:33:33.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0281623700 1 -- 172.21.15.3:0/253681392 learned_addr learned my addr 172.21.15.3:0/253681392 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:33.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027b7fe700 1 -- 172.21.15.3:0/253681392 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610988830 0 0) 0x7f027c0b04e0 con 0x7f027c0a7250 2022-07-21T01:33:33.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027b7fe700 1 -- 172.21.15.3:0/253681392 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0260003260 con 0x7f027c0a7250 2022-07-21T01:33:33.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027bfff700 1 -- 172.21.15.3:0/253681392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c0b1f40 msgr2=0x7f027c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:33.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027bfff700 1 --2- 172.21.15.3:0/253681392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c0b1f40 0x7f027c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027bfff700 1 -- 172.21.15.3:0/253681392 >> v1:172.21.15.143:6789/0 conn(0x7f027c0a7250 legacy=0x7f027c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027bfff700 1 -- 172.21.15.3:0/253681392 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c0b0a20 con 0x7f027c148530 2022-07-21T01:33:33.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027bfff700 1 --2- 172.21.15.3:0/253681392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c148530 0x7f027c0b2320 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f026c001110 tx=0x7f0264006df0).ready entity=mon.1 client_cookie=229ceb94cde9bd3a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027b7fe700 1 -- 172.21.15.3:0/253681392 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f026400d4a0 con 0x7f027c148530 2022-07-21T01:33:33.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027b7fe700 1 -- 172.21.15.3:0/253681392 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0264000e90 con 0x7f027c148530 2022-07-21T01:33:33.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f027b7fe700 1 -- 172.21.15.3:0/253681392 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f026400d7f0 con 0x7f027c148530 2022-07-21T01:33:33.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0283086700 1 -- 172.21.15.3:0/253681392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c148530 msgr2=0x7f027c0b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0283086700 1 --2- 172.21.15.3:0/253681392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c148530 0x7f027c0b2320 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f026c001110 tx=0x7f0264006df0).stop 2022-07-21T01:33:33.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0283086700 1 -- 172.21.15.3:0/253681392 shutdown_connections 2022-07-21T01:33:33.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0283086700 1 --2- 172.21.15.3:0/253681392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c148530 0x7f027c0b2320 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0283086700 1 --2- 172.21.15.3:0/253681392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c0b1f40 0x7f027c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0283086700 1 -- 172.21.15.3:0/253681392 >> 172.21.15.3:0/253681392 conn(0x7f027c05f840 msgr2=0x7f027c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:33.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0283086700 1 -- 172.21.15.3:0/253681392 shutdown_connections 2022-07-21T01:33:33.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.262+0000 7f0283086700 1 -- 172.21.15.3:0/253681392 wait complete. 2022-07-21T01:33:33.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 Processor -- start 2022-07-21T01:33:33.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 -- start start 2022-07-21T01:33:33.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f027c0a7250 0x7f027c1469d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c0b1f40 0x7f027c0ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c148530 0x7f027c13f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0aae20 con 0x7f027c0a7250 2022-07-21T01:33:33.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0a9670 con 0x7f027c0b1f40 2022-07-21T01:33:33.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0a9910 con 0x7f027c148530 2022-07-21T01:33:33.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0280e22700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f027c0a7250 0x7f027c1469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f027bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c0b1f40 0x7f027c0ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0280e22700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f027c0a7250 0x7f027c1469d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42914/0 (socket says 172.21.15.3:42914) 2022-07-21T01:33:33.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f027bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c0b1f40 0x7f027c0ab4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:48364/0 (socket says 172.21.15.3:48364) 2022-07-21T01:33:33.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0280e22700 1 -- 172.21.15.3:0/293339032 learned_addr learned my addr 172.21.15.3:0/293339032 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:33.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0281623700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c148530 0x7f027c13f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0280e22700 1 -- 172.21.15.3:0/293339032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c148530 msgr2=0x7f027c13f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0280e22700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c148530 0x7f027c13f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0280e22700 1 -- 172.21.15.3:0/293339032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c0b1f40 msgr2=0x7f027c0ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0280e22700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c0b1f40 0x7f027c0ab4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0280e22700 1 -- 172.21.15.3:0/293339032 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026400b180 con 0x7f027c0a7250 2022-07-21T01:33:33.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0280e22700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f027c0a7250 0x7f027c1469d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f027400c000 tx=0x7f02740091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0279ffb700 1 -- 172.21.15.3:0/293339032 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f027400c6a0 con 0x7f027c0a7250 2022-07-21T01:33:33.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0279ffb700 1 -- 172.21.15.3:0/293339032 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027400c800 con 0x7f027c0a7250 2022-07-21T01:33:33.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0279ffb700 1 -- 172.21.15.3:0/293339032 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f027400cb60 con 0x7f027c0a7250 2022-07-21T01:33:33.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c0ab050 con 0x7f027c0a7250 2022-07-21T01:33:33.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.266+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c13fe00 con 0x7f027c0a7250 2022-07-21T01:33:33.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.270+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c0ab1e0 con 0x7f027c0a7250 2022-07-21T01:33:33.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.270+0000 7f0279ffb700 1 -- 172.21.15.3:0/293339032 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0274022070 con 0x7f027c0a7250 2022-07-21T01:33:33.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.270+0000 7f0279ffb700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0254053a10 0x7f0254055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.270+0000 7f0279ffb700 1 -- 172.21.15.3:0/293339032 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f027406c660 con 0x7f027c0a7250 2022-07-21T01:33:33.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.270+0000 7f027bfff700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0254053a10 0x7f0254055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.270+0000 7f027bfff700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0254053a10 0x7f0254055ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f026401a000 tx=0x7f0264014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.274+0000 7f0279ffb700 1 -- 172.21.15.3:0/293339032 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0274043420 con 0x7f027c0a7250 2022-07-21T01:33:33.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f027c0b0cc0 con 0x7f027c0a7250 2022-07-21T01:33:33.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0279ffb700 1 -- 172.21.15.3:0/293339032 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0274046a40 con 0x7f027c0a7250 2022-07-21T01:33:33.399 INFO:teuthology.orchestra.run.smithi003.stdout:60129542150 2022-07-21T01:33:33.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0254053a10 msgr2=0x7f0254055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0254053a10 0x7f0254055ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f026401a000 tx=0x7f0264014000).stop 2022-07-21T01:33:33.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f027c0a7250 msgr2=0x7f027c1469d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f027c0a7250 0x7f027c1469d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f027400c000 tx=0x7f02740091d0).stop 2022-07-21T01:33:33.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 shutdown_connections 2022-07-21T01:33:33.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c148530 0x7f027c13f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0254053a10 0x7f0254055ed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c0b1f40 0x7f027c0ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 --2- 172.21.15.3:0/293339032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f027c0a7250 0x7f027c1469d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 >> 172.21.15.3:0/293339032 conn(0x7f027c05f840 msgr2=0x7f027c147900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:33.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 shutdown_connections 2022-07-21T01:33:33.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.402+0000 7f0283086700 1 -- 172.21.15.3:0/293339032 wait complete. 2022-07-21T01:33:33.412 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.1 2022-07-21T01:33:33.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-07-21T01:33:33.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.518+0000 7f61ee704700 1 Processor -- start 2022-07-21T01:33:33.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.518+0000 7f61ee704700 1 -- start start 2022-07-21T01:33:33.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61e805d240 0x7f61e80b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e805cd40 con 0x7f61e805c960 2022-07-21T01:33:33.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e80b1920 con 0x7f61e813f520 2022-07-21T01:33:33.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e80ad6d0 con 0x7f61e805d240 2022-07-21T01:33:33.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ed702700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f61e805c960 0x7f61e80acfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51202/0 (socket says 172.21.15.3:51202) 2022-07-21T01:33:33.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ed702700 1 -- 172.21.15.3:0/2719993948 learned_addr learned my addr 172.21.15.3:0/2719993948 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:33.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ecf01700 1 --2- 172.21.15.3:0/2719993948 >> v2:172.21.15.143:3300/0 conn(0x7f61e805d240 0x7f61e80b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61d3fff700 1 -- 172.21.15.3:0/2719993948 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276983183 0 0) 0x7f61e805cd40 con 0x7f61e805c960 2022-07-21T01:33:33.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61d3fff700 1 -- 172.21.15.3:0/2719993948 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61cc004370 con 0x7f61e805c960 2022-07-21T01:33:33.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61d3fff700 1 -- 172.21.15.3:0/2719993948 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000888117 0 0) 0x7f61e80b1920 con 0x7f61e813f520 2022-07-21T01:33:33.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61d3fff700 1 -- 172.21.15.3:0/2719993948 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e805cd40 con 0x7f61e813f520 2022-07-21T01:33:33.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ecf01700 1 -- 172.21.15.3:0/2719993948 >> v1:172.21.15.143:6789/0 conn(0x7f61e813f520 legacy=0x7f61e80aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ecf01700 1 -- 172.21.15.3:0/2719993948 >> v1:172.21.15.3:6789/0 conn(0x7f61e805c960 legacy=0x7f61e80acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ecf01700 1 -- 172.21.15.3:0/2719993948 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e80b1bc0 con 0x7f61e805d240 2022-07-21T01:33:33.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ecf01700 1 --2- 172.21.15.3:0/2719993948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e805d240 0x7f61e80b1280 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f61dc002f70 tx=0x7f61dc006df0).ready entity=mon.1 client_cookie=4411fbcc610d7963 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61d3fff700 1 -- 172.21.15.3:0/2719993948 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61dc00d540 con 0x7f61e805d240 2022-07-21T01:33:33.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61d3fff700 1 -- 172.21.15.3:0/2719993948 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc000e90 con 0x7f61e805d240 2022-07-21T01:33:33.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61d3fff700 1 -- 172.21.15.3:0/2719993948 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61dc00d890 con 0x7f61e805d240 2022-07-21T01:33:33.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 -- 172.21.15.3:0/2719993948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e805d240 msgr2=0x7f61e80b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 --2- 172.21.15.3:0/2719993948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e805d240 0x7f61e80b1280 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f61dc002f70 tx=0x7f61dc006df0).stop 2022-07-21T01:33:33.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 -- 172.21.15.3:0/2719993948 shutdown_connections 2022-07-21T01:33:33.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 --2- 172.21.15.3:0/2719993948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e805d240 0x7f61e80b1280 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 -- 172.21.15.3:0/2719993948 >> 172.21.15.3:0/2719993948 conn(0x7f61e8060310 msgr2=0x7f61e80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:33.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 -- 172.21.15.3:0/2719993948 shutdown_connections 2022-07-21T01:33:33.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.522+0000 7f61ee704700 1 -- 172.21.15.3:0/2719993948 wait complete. 2022-07-21T01:33:33.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.526+0000 7f61ee704700 1 Processor -- start 2022-07-21T01:33:33.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.526+0000 7f61ee704700 1 -- start start 2022-07-21T01:33:33.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ee704700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e805c960 0x7f61e80ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ee704700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e805d240 0x7f61e8153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ee704700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e813f520 0x7f61e80afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ee704700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e8140de0 con 0x7f61e813f520 2022-07-21T01:33:33.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ee704700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e81535b0 con 0x7f61e805c960 2022-07-21T01:33:33.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ee704700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e8153850 con 0x7f61e805d240 2022-07-21T01:33:33.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61edf03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e813f520 0x7f61e80afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61edf03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e813f520 0x7f61e80afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42932/0 (socket says 172.21.15.3:42932) 2022-07-21T01:33:33.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61edf03700 1 -- 172.21.15.3:0/290816350 learned_addr learned my addr 172.21.15.3:0/290816350 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:33.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ed702700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e805c960 0x7f61e80ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ecf01700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e805d240 0x7f61e8153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61edf03700 1 -- 172.21.15.3:0/290816350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e805d240 msgr2=0x7f61e8153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61edf03700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e805d240 0x7f61e8153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61edf03700 1 -- 172.21.15.3:0/290816350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e805c960 msgr2=0x7f61e80ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61edf03700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e805c960 0x7f61e80ab820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61edf03700 1 -- 172.21.15.3:0/290816350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61dc00b180 con 0x7f61e813f520 2022-07-21T01:33:33.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61edf03700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e813f520 0x7f61e80afe00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f61e0001490 tx=0x7f61e0007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61daffd700 1 -- 172.21.15.3:0/290816350 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e0002310 con 0x7f61e813f520 2022-07-21T01:33:33.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e8153af0 con 0x7f61e813f520 2022-07-21T01:33:33.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61daffd700 1 -- 172.21.15.3:0/290816350 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e0002470 con 0x7f61e813f520 2022-07-21T01:33:33.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61daffd700 1 -- 172.21.15.3:0/290816350 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e00017a0 con 0x7f61e813f520 2022-07-21T01:33:33.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e80b02f0 con 0x7f61e813f520 2022-07-21T01:33:33.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.530+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e8153c80 con 0x7f61e813f520 2022-07-21T01:33:33.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.534+0000 7f61daffd700 1 -- 172.21.15.3:0/290816350 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f61e0010030 con 0x7f61e813f520 2022-07-21T01:33:33.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.534+0000 7f61daffd700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c40539e0 0x7f61c4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.534+0000 7f61daffd700 1 -- 172.21.15.3:0/290816350 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f61e006bff0 con 0x7f61e813f520 2022-07-21T01:33:33.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.534+0000 7f61ed702700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c40539e0 0x7f61c4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.534+0000 7f61ed702700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c40539e0 0x7f61c4055ea0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f61e400d000 tx=0x7f61e4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.538+0000 7f61daffd700 1 -- 172.21.15.3:0/290816350 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f61e0042d10 con 0x7f61e813f520 2022-07-21T01:33:33.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f61e80574e0 con 0x7f61e813f520 2022-07-21T01:33:33.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61daffd700 1 -- 172.21.15.3:0/290816350 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f61e0046330 con 0x7f61e813f520 2022-07-21T01:33:33.663 INFO:teuthology.orchestra.run.smithi003.stdout:64424509445 2022-07-21T01:33:33.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c40539e0 msgr2=0x7f61c4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c40539e0 0x7f61c4055ea0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f61e400d000 tx=0x7f61e4009040).stop 2022-07-21T01:33:33.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e813f520 msgr2=0x7f61e80afe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e813f520 0x7f61e80afe00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f61e0001490 tx=0x7f61e0007dd0).stop 2022-07-21T01:33:33.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 shutdown_connections 2022-07-21T01:33:33.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e813f520 0x7f61e80afe00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e805d240 0x7f61e8153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c40539e0 0x7f61c4055ea0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 --2- 172.21.15.3:0/290816350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e805c960 0x7f61e80ab820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 >> 172.21.15.3:0/290816350 conn(0x7f61e8060310 msgr2=0x7f61e80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:33.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 shutdown_connections 2022-07-21T01:33:33.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.666+0000 7f61ee704700 1 -- 172.21.15.3:0/290816350 wait complete. 2022-07-21T01:33:33.676 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.2 2022-07-21T01:33:33.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-07-21T01:33:33.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f95313b6700 1 Processor -- start 2022-07-21T01:33:33.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f95313b6700 1 -- start start 2022-07-21T01:33:33.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f95313b6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0b3190 0x7f952c060ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f95313b6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f952c0b2730 0x7f952c061190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f95313b6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c0654c0 con 0x7f952c0666b0 2022-07-21T01:33:33.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f95313b6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c062190 con 0x7f952c0b2730 2022-07-21T01:33:33.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f95313b6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0b3570 con 0x7f952c0b3190 2022-07-21T01:33:33.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952b7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f952c0b2730 0x7f952c061190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0b3190 0x7f952c060ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952b7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f952c0b2730 0x7f952c061190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:48384/0 (socket says 172.21.15.3:48384) 2022-07-21T01:33:33.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952b7fe700 1 -- 172.21.15.3:0/970021440 learned_addr learned my addr 172.21.15.3:0/970021440 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:33.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f9530bb5700 1 --1- 172.21.15.3:0/970021440 >> v1:172.21.15.143:6789/0 conn(0x7f952c0666b0 0x7f952c066b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:53336/0 (socket says 172.21.15.3:53336) 2022-07-21T01:33:33.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952bfff700 1 -- 172.21.15.3:0/970021440 >> v2:172.21.15.143:3300/0 conn(0x7f952c0b2730 msgr2=0x7f952c061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:33.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952bfff700 1 --2- 172.21.15.3:0/970021440 >> v2:172.21.15.143:3300/0 conn(0x7f952c0b2730 0x7f952c061190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952bfff700 1 -- 172.21.15.3:0/970021440 >> v1:172.21.15.143:6789/0 conn(0x7f952c0666b0 legacy=0x7f952c066b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952bfff700 1 -- 172.21.15.3:0/970021440 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c05fbf0 con 0x7f952c0b3190 2022-07-21T01:33:33.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952b7fe700 1 --2- 172.21.15.3:0/970021440 >> v2:172.21.15.143:3300/0 conn(0x7f952c0b2730 0x7f952c061190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:33.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952bfff700 1 --2- 172.21.15.3:0/970021440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0b3190 0x7f952c060ca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f95240032f0 tx=0x7f95240091d0).ready entity=mon.2 client_cookie=f8de89c950d9c229 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952affd700 1 -- 172.21.15.3:0/970021440 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f952400b6c0 con 0x7f952c0b3190 2022-07-21T01:33:33.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952affd700 1 -- 172.21.15.3:0/970021440 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952400b820 con 0x7f952c0b3190 2022-07-21T01:33:33.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.790+0000 7f952affd700 1 -- 172.21.15.3:0/970021440 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f952400b990 con 0x7f952c0b3190 2022-07-21T01:33:33.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 -- 172.21.15.3:0/970021440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0b3190 msgr2=0x7f952c060ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 --2- 172.21.15.3:0/970021440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0b3190 0x7f952c060ca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f95240032f0 tx=0x7f95240091d0).stop 2022-07-21T01:33:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 -- 172.21.15.3:0/970021440 shutdown_connections 2022-07-21T01:33:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 --2- 172.21.15.3:0/970021440 >> v2:172.21.15.143:3300/0 conn(0x7f952c0b2730 0x7f952c061190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 --2- 172.21.15.3:0/970021440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0b3190 0x7f952c060ca0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 -- 172.21.15.3:0/970021440 >> 172.21.15.3:0/970021440 conn(0x7f952c063870 msgr2=0x7f952c063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:33.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 -- 172.21.15.3:0/970021440 shutdown_connections 2022-07-21T01:33:33.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 -- 172.21.15.3:0/970021440 wait complete. 2022-07-21T01:33:33.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 Processor -- start 2022-07-21T01:33:33.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 -- start start 2022-07-21T01:33:33.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0666b0 0x7f952c1493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0b2730 0x7f952c152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f952c0b3190 0x7f952c155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0b0a20 con 0x7f952c0b3190 2022-07-21T01:33:33.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c1525e0 con 0x7f952c0b2730 2022-07-21T01:33:33.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f95313b6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c152880 con 0x7f952c0666b0 2022-07-21T01:33:33.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f9530bb5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f952c0b3190 0x7f952c155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f9530bb5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f952c0b3190 0x7f952c155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42946/0 (socket says 172.21.15.3:42946) 2022-07-21T01:33:33.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f9530bb5700 1 -- 172.21.15.3:0/4143061779 learned_addr learned my addr 172.21.15.3:0/4143061779 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:33.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f952b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0b2730 0x7f952c152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.794+0000 7f952bfff700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0666b0 0x7f952c1493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f952b7fe700 1 -- 172.21.15.3:0/4143061779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0666b0 msgr2=0x7f952c1493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f952b7fe700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0666b0 0x7f952c1493c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f952b7fe700 1 -- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f952c0b3190 msgr2=0x7f952c155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f952b7fe700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f952c0b3190 0x7f952c155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f952b7fe700 1 -- 172.21.15.3:0/4143061779 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951400b1e0 con 0x7f952c0b2730 2022-07-21T01:33:33.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f9530bb5700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f952c0b3190 0x7f952c155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:33.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f952b7fe700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0b2730 0x7f952c152d20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f951400c000 tx=0x7f9514006eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f95297fa700 1 -- 172.21.15.3:0/4143061779 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9514012660 con 0x7f952c0b2730 2022-07-21T01:33:33.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f95297fa700 1 -- 172.21.15.3:0/4143061779 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95140127c0 con 0x7f952c0b2730 2022-07-21T01:33:33.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f95297fa700 1 -- 172.21.15.3:0/4143061779 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9514012b20 con 0x7f952c0b2730 2022-07-21T01:33:33.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9524009470 con 0x7f952c0b2730 2022-07-21T01:33:33.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c0acd20 con 0x7f952c0b2730 2022-07-21T01:33:33.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c152a40 con 0x7f952c0b2730 2022-07-21T01:33:33.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f95297fa700 1 -- 172.21.15.3:0/4143061779 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9514012c80 con 0x7f952c0b2730 2022-07-21T01:33:33.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f95297fa700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9504053aa0 0x7f9504055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:33.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f95297fa700 1 -- 172.21.15.3:0/4143061779 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f951406c230 con 0x7f952c0b2730 2022-07-21T01:33:33.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.798+0000 7f952bfff700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9504053aa0 0x7f9504055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:33.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.802+0000 7f952bfff700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9504053aa0 0x7f9504055f60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f952401c000 tx=0x7f9524016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:33.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.806+0000 7f95297fa700 1 -- 172.21.15.3:0/4143061779 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9514042e60 con 0x7f952c0b2730 2022-07-21T01:33:33.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.934+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f952c05fe90 con 0x7f952c0b2730 2022-07-21T01:33:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.934+0000 7f95297fa700 1 -- 172.21.15.3:0/4143061779 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9514046480 con 0x7f952c0b2730 2022-07-21T01:33:33.933 INFO:teuthology.orchestra.run.smithi003.stdout:64424509445 2022-07-21T01:33:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9504053aa0 msgr2=0x7f9504055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9504053aa0 0x7f9504055f60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f952401c000 tx=0x7f9524016000).stop 2022-07-21T01:33:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0b2730 msgr2=0x7f952c152d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0b2730 0x7f952c152d20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f951400c000 tx=0x7f9514006eb0).stop 2022-07-21T01:33:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 shutdown_connections 2022-07-21T01:33:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f952c0b3190 0x7f952c155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9504053aa0 0x7f9504055f60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0b2730 0x7f952c152d20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 --2- 172.21.15.3:0/4143061779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0666b0 0x7f952c1493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:33.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 >> 172.21.15.3:0/4143061779 conn(0x7f952c063870 msgr2=0x7f952c1460e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:33.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 shutdown_connections 2022-07-21T01:33:33.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:33.938+0000 7f95313b6700 1 -- 172.21.15.3:0/4143061779 wait complete. 2022-07-21T01:33:33.945 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.3 2022-07-21T01:33:33.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-07-21T01:33:34.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.054+0000 7f6606000700 1 Processor -- start 2022-07-21T01:33:34.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.054+0000 7f6606000700 1 -- start start 2022-07-21T01:33:34.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.054+0000 7f6606000700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f66000ad670 0x7f66000b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.054+0000 7f6606000700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f66000af120 0x7f66000b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.054+0000 7f6606000700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66000b0c40 con 0x7f660005f840 2022-07-21T01:33:34.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.054+0000 7f6606000700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66000b0ee0 con 0x7f66000af120 2022-07-21T01:33:34.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.054+0000 7f6606000700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66000ae9e0 con 0x7f66000ad670 2022-07-21T01:33:34.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65fffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f66000af120 0x7f66000b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65fffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f66000af120 0x7f66000b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42950/0 (socket says 172.21.15.3:42950) 2022-07-21T01:33:34.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65fffff700 1 -- 172.21.15.3:0/331101052 learned_addr learned my addr 172.21.15.3:0/331101052 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:34.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f6604ffe700 1 --2- 172.21.15.3:0/331101052 >> v2:172.21.15.143:3300/0 conn(0x7f66000ad670 0x7f66000b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65fffff700 1 -- 172.21.15.3:0/331101052 >> v2:172.21.15.143:3300/0 conn(0x7f66000ad670 msgr2=0x7f66000b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:34.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65fffff700 1 --2- 172.21.15.3:0/331101052 >> v2:172.21.15.143:3300/0 conn(0x7f66000ad670 0x7f66000b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65fffff700 1 -- 172.21.15.3:0/331101052 >> v1:172.21.15.143:6789/0 conn(0x7f660005f840 legacy=0x7f660005fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65fffff700 1 -- 172.21.15.3:0/331101052 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66000aec80 con 0x7f66000af120 2022-07-21T01:33:34.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65fffff700 1 --2- 172.21.15.3:0/331101052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66000af120 0x7f66000b12a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f65f4002f70 tx=0x7f65f4006eb0).ready entity=mon.0 client_cookie=21c533b9ac79860f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65ff7fe700 1 -- 172.21.15.3:0/331101052 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65f400d610 con 0x7f66000af120 2022-07-21T01:33:34.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65ff7fe700 1 -- 172.21.15.3:0/331101052 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f4000e90 con 0x7f66000af120 2022-07-21T01:33:34.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f65ff7fe700 1 -- 172.21.15.3:0/331101052 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65f400d9a0 con 0x7f66000af120 2022-07-21T01:33:34.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f6606000700 1 -- 172.21.15.3:0/331101052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66000af120 msgr2=0x7f66000b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f6606000700 1 --2- 172.21.15.3:0/331101052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66000af120 0x7f66000b12a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f65f4002f70 tx=0x7f65f4006eb0).stop 2022-07-21T01:33:34.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f6606000700 1 -- 172.21.15.3:0/331101052 shutdown_connections 2022-07-21T01:33:34.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f6606000700 1 --2- 172.21.15.3:0/331101052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66000af120 0x7f66000b12a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f6606000700 1 --2- 172.21.15.3:0/331101052 >> v2:172.21.15.143:3300/0 conn(0x7f66000ad670 0x7f66000b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f6606000700 1 -- 172.21.15.3:0/331101052 >> 172.21.15.3:0/331101052 conn(0x7f6600061360 msgr2=0x7f6600062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:34.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f6606000700 1 -- 172.21.15.3:0/331101052 shutdown_connections 2022-07-21T01:33:34.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.058+0000 7f6606000700 1 -- 172.21.15.3:0/331101052 wait complete. 2022-07-21T01:33:34.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 Processor -- start 2022-07-21T01:33:34.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 -- start start 2022-07-21T01:33:34.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660005f840 0x7f660013f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66000ad670 0x7f6600155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66000af120 0x7f6600156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001400c0 con 0x7f660005f840 2022-07-21T01:33:34.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66000aae90 con 0x7f66000af120 2022-07-21T01:33:34.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66000ac5c0 con 0x7f66000ad670 2022-07-21T01:33:34.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6604ffe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660005f840 0x7f660013f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6604ffe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660005f840 0x7f660013f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42956/0 (socket says 172.21.15.3:42956) 2022-07-21T01:33:34.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6604ffe700 1 -- 172.21.15.3:0/3617070140 learned_addr learned my addr 172.21.15.3:0/3617070140 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:34.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f66057ff700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66000af120 0x7f6600156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f65fffff700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66000ad670 0x7f6600155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6604ffe700 1 -- 172.21.15.3:0/3617070140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66000ad670 msgr2=0x7f6600155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6604ffe700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66000ad670 0x7f6600155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6604ffe700 1 -- 172.21.15.3:0/3617070140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66000af120 msgr2=0x7f6600156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6604ffe700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66000af120 0x7f6600156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6604ffe700 1 -- 172.21.15.3:0/3617070140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f400b1e0 con 0x7f660005f840 2022-07-21T01:33:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6604ffe700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660005f840 0x7f660013f460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f66000ac750 tx=0x7f65ec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f65fdffb700 1 -- 172.21.15.3:0/3617070140 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65ec011780 con 0x7f660005f840 2022-07-21T01:33:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f65fdffb700 1 -- 172.21.15.3:0/3617070140 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec0118e0 con 0x7f660005f840 2022-07-21T01:33:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66000ac830 con 0x7f660005f840 2022-07-21T01:33:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f65fdffb700 1 -- 172.21.15.3:0/3617070140 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65ec011c40 con 0x7f660005f840 2022-07-21T01:33:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.062+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6600063c40 con 0x7f660005f840 2022-07-21T01:33:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.066+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66000af590 con 0x7f660005f840 2022-07-21T01:33:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.066+0000 7f65fdffb700 1 -- 172.21.15.3:0/3617070140 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f65ec012420 con 0x7f660005f840 2022-07-21T01:33:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.066+0000 7f65fdffb700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f65e00539e0 0x7f65e0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.066+0000 7f65fdffb700 1 -- 172.21.15.3:0/3617070140 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f65ec06bdb0 con 0x7f660005f840 2022-07-21T01:33:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.066+0000 7f65fffff700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f65e00539e0 0x7f65e0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.066+0000 7f65fffff700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f65e00539e0 0x7f65e0055ea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f65f401c000 tx=0x7f65f4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.070+0000 7f65fdffb700 1 -- 172.21.15.3:0/3617070140 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f65ec042b10 con 0x7f660005f840 2022-07-21T01:33:34.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.198+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f66000574e0 con 0x7f660005f840 2022-07-21T01:33:34.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.198+0000 7f65fdffb700 1 -- 172.21.15.3:0/3617070140 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f65ec046130 con 0x7f660005f840 2022-07-21T01:33:34.197 INFO:teuthology.orchestra.run.smithi003.stdout:60129542149 2022-07-21T01:33:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f65e00539e0 msgr2=0x7f65e0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f65e00539e0 0x7f65e0055ea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f65f401c000 tx=0x7f65f4015000).stop 2022-07-21T01:33:34.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660005f840 msgr2=0x7f660013f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660005f840 0x7f660013f460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f66000ac750 tx=0x7f65ec008040).stop 2022-07-21T01:33:34.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 shutdown_connections 2022-07-21T01:33:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f65e00539e0 0x7f65e0055ea0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66000af120 0x7f6600156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66000ad670 0x7f6600155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 --2- 172.21.15.3:0/3617070140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f660005f840 0x7f660013f460 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 >> 172.21.15.3:0/3617070140 conn(0x7f6600061360 msgr2=0x7f66000a8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:34.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 shutdown_connections 2022-07-21T01:33:34.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.202+0000 7f6606000700 1 -- 172.21.15.3:0/3617070140 wait complete. 2022-07-21T01:33:34.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.4 2022-07-21T01:33:34.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-07-21T01:33:34.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58f59e700 1 Processor -- start 2022-07-21T01:33:34.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58f59e700 1 -- start start 2022-07-21T01:33:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5900b7840 0x7fe590065720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58f59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe5900b3070 0x7fe590061120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5900b2c20 con 0x7fe59014b040 2022-07-21T01:33:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58f59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe590065c10 con 0x7fe5900b3070 2022-07-21T01:33:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5900b7c20 con 0x7fe5900b7840 2022-07-21T01:33:34.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58dd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe5900b3070 0x7fe590061120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58dd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe5900b3070 0x7fe590061120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42964/0 (socket says 172.21.15.3:42964) 2022-07-21T01:33:34.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58dd9b700 1 -- 172.21.15.3:0/67292436 learned_addr learned my addr 172.21.15.3:0/67292436 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:34.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58e59c700 1 --2- 172.21.15.3:0/67292436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5900b7840 0x7fe590065720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58dd9b700 1 -- 172.21.15.3:0/67292436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5900b7840 msgr2=0x7fe590065720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:34.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58dd9b700 1 --2- 172.21.15.3:0/67292436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5900b7840 0x7fe590065720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.318+0000 7fe58dd9b700 1 -- 172.21.15.3:0/67292436 >> v1:172.21.15.143:6789/0 conn(0x7fe59014b040 legacy=0x7fe59014b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58dd9b700 1 -- 172.21.15.3:0/67292436 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5900b2980 con 0x7fe5900b3070 2022-07-21T01:33:34.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58dd9b700 1 --2- 172.21.15.3:0/67292436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b3070 0x7fe590061120 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe580002f70 tx=0x7fe580006eb0).ready entity=mon.0 client_cookie=db53c2a142d258a6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58d59a700 1 -- 172.21.15.3:0/67292436 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe58000d5c0 con 0x7fe5900b3070 2022-07-21T01:33:34.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58d59a700 1 -- 172.21.15.3:0/67292436 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe580000e90 con 0x7fe5900b3070 2022-07-21T01:33:34.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58d59a700 1 -- 172.21.15.3:0/67292436 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe58000d950 con 0x7fe5900b3070 2022-07-21T01:33:34.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58f59e700 1 -- 172.21.15.3:0/67292436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b3070 msgr2=0x7fe590061120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58f59e700 1 --2- 172.21.15.3:0/67292436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b3070 0x7fe590061120 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe580002f70 tx=0x7fe580006eb0).stop 2022-07-21T01:33:34.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58f59e700 1 -- 172.21.15.3:0/67292436 shutdown_connections 2022-07-21T01:33:34.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58f59e700 1 --2- 172.21.15.3:0/67292436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b3070 0x7fe590061120 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58f59e700 1 --2- 172.21.15.3:0/67292436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5900b7840 0x7fe590065720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58f59e700 1 -- 172.21.15.3:0/67292436 >> 172.21.15.3:0/67292436 conn(0x7fe59005f840 msgr2=0x7fe59005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:34.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58f59e700 1 -- 172.21.15.3:0/67292436 shutdown_connections 2022-07-21T01:33:34.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58f59e700 1 -- 172.21.15.3:0/67292436 wait complete. 2022-07-21T01:33:34.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.322+0000 7fe58f59e700 1 Processor -- start 2022-07-21T01:33:34.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58f59e700 1 -- start start 2022-07-21T01:33:34.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5900b3070 0x7fe59014c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b7840 0x7fe590157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59014b040 0x7fe5900afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5900b13b0 con 0x7fe5900b7840 2022-07-21T01:33:34.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5900a1f40 con 0x7fe5900b3070 2022-07-21T01:33:34.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b7840 0x7fe590157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5900b15e0 con 0x7fe59014b040 2022-07-21T01:33:34.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b7840 0x7fe590157130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42972/0 (socket says 172.21.15.3:42972) 2022-07-21T01:33:34.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58dd9b700 1 -- 172.21.15.3:0/2687439898 learned_addr learned my addr 172.21.15.3:0/2687439898 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:34.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58e59c700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5900b3070 0x7fe59014c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58ed9d700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59014b040 0x7fe5900afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58dd9b700 1 -- 172.21.15.3:0/2687439898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59014b040 msgr2=0x7fe5900afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58dd9b700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59014b040 0x7fe5900afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58dd9b700 1 -- 172.21.15.3:0/2687439898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5900b3070 msgr2=0x7fe59014c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58dd9b700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5900b3070 0x7fe59014c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58dd9b700 1 -- 172.21.15.3:0/2687439898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58000b1e0 con 0x7fe5900b7840 2022-07-21T01:33:34.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58dd9b700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b7840 0x7fe590157130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe580002f70 tx=0x7fe580015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe57f7fe700 1 -- 172.21.15.3:0/2687439898 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5800117e0 con 0x7fe5900b7840 2022-07-21T01:33:34.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe57f7fe700 1 -- 172.21.15.3:0/2687439898 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe580011940 con 0x7fe5900b7840 2022-07-21T01:33:34.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe590156c90 con 0x7fe5900b7840 2022-07-21T01:33:34.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe57f7fe700 1 -- 172.21.15.3:0/2687439898 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe580011ca0 con 0x7fe5900b7840 2022-07-21T01:33:34.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.326+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59014cd70 con 0x7fe5900b7840 2022-07-21T01:33:34.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.330+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5900609e0 con 0x7fe5900b7840 2022-07-21T01:33:34.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.330+0000 7fe57f7fe700 1 -- 172.21.15.3:0/2687439898 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe580025030 con 0x7fe5900b7840 2022-07-21T01:33:34.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.330+0000 7fe57f7fe700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe56c0539e0 0x7fe56c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.330+0000 7fe57f7fe700 1 -- 172.21.15.3:0/2687439898 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe58006d430 con 0x7fe5900b7840 2022-07-21T01:33:34.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.330+0000 7fe58e59c700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe56c0539e0 0x7fe56c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.330+0000 7fe58e59c700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe56c0539e0 0x7fe56c055ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fe58400c000 tx=0x7fe584008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.334+0000 7fe57f7fe700 1 -- 172.21.15.3:0/2687439898 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe5800440e0 con 0x7fe5900b7840 2022-07-21T01:33:34.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.462+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe5900574e0 con 0x7fe5900b7840 2022-07-21T01:33:34.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.462+0000 7fe57f7fe700 1 -- 172.21.15.3:0/2687439898 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe580023090 con 0x7fe5900b7840 2022-07-21T01:33:34.460 INFO:teuthology.orchestra.run.smithi003.stdout:60129542149 2022-07-21T01:33:34.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe56c0539e0 msgr2=0x7fe56c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe56c0539e0 0x7fe56c055ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fe58400c000 tx=0x7fe584008040).stop 2022-07-21T01:33:34.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b7840 msgr2=0x7fe590157130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b7840 0x7fe590157130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe580002f70 tx=0x7fe580015000).stop 2022-07-21T01:33:34.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 shutdown_connections 2022-07-21T01:33:34.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59014b040 0x7fe5900afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5900b7840 0x7fe590157130 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe56c0539e0 0x7fe56c055ea0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 --2- 172.21.15.3:0/2687439898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5900b3070 0x7fe59014c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 >> 172.21.15.3:0/2687439898 conn(0x7fe59005f840 msgr2=0x7fe590061a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:34.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 shutdown_connections 2022-07-21T01:33:34.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.466+0000 7fe58f59e700 1 -- 172.21.15.3:0/2687439898 wait complete. 2022-07-21T01:33:34.473 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.5 2022-07-21T01:33:34.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2022-07-21T01:33:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc74359e700 1 Processor -- start 2022-07-21T01:33:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc74359e700 1 -- start start 2022-07-21T01:33:34.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc74359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 0x7fc744142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc74359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc7440b3260 0x7fc7441433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc74359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7440ad000 0x7fc74405f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc74359e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7440b0c80 con 0x7fc7440ad000 2022-07-21T01:33:34.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc74359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7440ae810 con 0x7fc7440b3260 2022-07-21T01:33:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc74359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440aeab0 con 0x7fc7440b08a0 2022-07-21T01:33:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc742d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7440ad000 0x7fc74405f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc742d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc7440ad000 0x7fc74405f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42982/0 (socket says 172.21.15.3:42982) 2022-07-21T01:33:34.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc742d9d700 1 -- 172.21.15.3:0/2050412251 learned_addr learned my addr 172.21.15.3:0/2050412251 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc74259c700 1 --2- 172.21.15.3:0/2050412251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 0x7fc744142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc73bfff700 1 --2- 172.21.15.3:0/2050412251 >> v2:172.21.15.143:3300/0 conn(0x7fc7440b3260 0x7fc7441433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc742d9d700 1 -- 172.21.15.3:0/2050412251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 msgr2=0x7fc744142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc742d9d700 1 --2- 172.21.15.3:0/2050412251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 0x7fc744142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc742d9d700 1 -- 172.21.15.3:0/2050412251 >> v2:172.21.15.143:3300/0 conn(0x7fc7440b3260 msgr2=0x7fc7441433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:34.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc742d9d700 1 --2- 172.21.15.3:0/2050412251 >> v2:172.21.15.143:3300/0 conn(0x7fc7440b3260 0x7fc7441433a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc742d9d700 1 -- 172.21.15.3:0/2050412251 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7440aed50 con 0x7fc7440ad000 2022-07-21T01:33:34.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.582+0000 7fc73bfff700 1 --2- 172.21.15.3:0/2050412251 >> v2:172.21.15.143:3300/0 conn(0x7fc7440b3260 0x7fc7441433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:34.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc742d9d700 1 --2- 172.21.15.3:0/2050412251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440ad000 0x7fc74405f840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc73c0032f0 tx=0x7fc73c00b530).ready entity=mon.0 client_cookie=1ba778ecb14a9987 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc741d9b700 1 -- 172.21.15.3:0/2050412251 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc73c0153f0 con 0x7fc7440ad000 2022-07-21T01:33:34.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc741d9b700 1 -- 172.21.15.3:0/2050412251 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c015550 con 0x7fc7440ad000 2022-07-21T01:33:34.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc741d9b700 1 -- 172.21.15.3:0/2050412251 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc73c0158b0 con 0x7fc7440ad000 2022-07-21T01:33:34.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 -- 172.21.15.3:0/2050412251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440ad000 msgr2=0x7fc74405f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 --2- 172.21.15.3:0/2050412251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440ad000 0x7fc74405f840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc73c0032f0 tx=0x7fc73c00b530).stop 2022-07-21T01:33:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 -- 172.21.15.3:0/2050412251 shutdown_connections 2022-07-21T01:33:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 --2- 172.21.15.3:0/2050412251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440ad000 0x7fc74405f840 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 --2- 172.21.15.3:0/2050412251 >> v2:172.21.15.143:3300/0 conn(0x7fc7440b3260 0x7fc7441433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 --2- 172.21.15.3:0/2050412251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 0x7fc744142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 -- 172.21.15.3:0/2050412251 >> 172.21.15.3:0/2050412251 conn(0x7fc7440612d0 msgr2=0x7fc744062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:34.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 -- 172.21.15.3:0/2050412251 shutdown_connections 2022-07-21T01:33:34.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 -- 172.21.15.3:0/2050412251 wait complete. 2022-07-21T01:33:34.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 Processor -- start 2022-07-21T01:33:34.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.586+0000 7fc74359e700 1 -- start start 2022-07-21T01:33:34.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7440ad000 0x7fc7440676b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 0x7fc744064c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440b3260 0x7fc7440a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440a5b40 con 0x7fc7440b3260 2022-07-21T01:33:34.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440a4ed0 con 0x7fc7440ad000 2022-07-21T01:33:34.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc744152e80 con 0x7fc7440b08a0 2022-07-21T01:33:34.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc742d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440b3260 0x7fc7440a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc742d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440b3260 0x7fc7440a41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42988/0 (socket says 172.21.15.3:42988) 2022-07-21T01:33:34.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc742d9d700 1 -- 172.21.15.3:0/558906438 learned_addr learned my addr 172.21.15.3:0/558906438 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74259c700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7440ad000 0x7fc7440676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc73bfff700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 0x7fc744064c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc742d9d700 1 -- 172.21.15.3:0/558906438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 msgr2=0x7fc744064c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc742d9d700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 0x7fc744064c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc742d9d700 1 -- 172.21.15.3:0/558906438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7440ad000 msgr2=0x7fc7440676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc742d9d700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7440ad000 0x7fc7440676b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc742d9d700 1 -- 172.21.15.3:0/558906438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c00b7d0 con 0x7fc7440b3260 2022-07-21T01:33:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc742d9d700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440b3260 0x7fc7440a41f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc73c0032f0 tx=0x7fc73c019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc73b7fe700 1 -- 172.21.15.3:0/558906438 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc73c009040 con 0x7fc7440b3260 2022-07-21T01:33:34.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc73b7fe700 1 -- 172.21.15.3:0/558906438 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c00b220 con 0x7fc7440b3260 2022-07-21T01:33:34.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc744153120 con 0x7fc7440b3260 2022-07-21T01:33:34.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc73b7fe700 1 -- 172.21.15.3:0/558906438 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc73c01c770 con 0x7fc7440b3260 2022-07-21T01:33:34.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc744067ba0 con 0x7fc7440b3260 2022-07-21T01:33:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.590+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7441532b0 con 0x7fc7440b3260 2022-07-21T01:33:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.594+0000 7fc73b7fe700 1 -- 172.21.15.3:0/558906438 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc73c01c8d0 con 0x7fc7440b3260 2022-07-21T01:33:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.594+0000 7fc73b7fe700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc728053a10 0x7fc728055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.594+0000 7fc73b7fe700 1 -- 172.21.15.3:0/558906438 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc73c06dd50 con 0x7fc7440b3260 2022-07-21T01:33:34.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.594+0000 7fc74259c700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc728053a10 0x7fc728055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.594+0000 7fc74259c700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc728053a10 0x7fc728055ed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc73400b000 tx=0x7fc734006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.598+0000 7fc73b7fe700 1 -- 172.21.15.3:0/558906438 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc73c044b10 con 0x7fc7440b3260 2022-07-21T01:33:34.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc7440574e0 con 0x7fc7440b3260 2022-07-21T01:33:34.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc73b7fe700 1 -- 172.21.15.3:0/558906438 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc73c048130 con 0x7fc7440b3260 2022-07-21T01:33:34.719 INFO:teuthology.orchestra.run.smithi003.stdout:60129542149 2022-07-21T01:33:34.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc728053a10 msgr2=0x7fc728055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc728053a10 0x7fc728055ed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc73400b000 tx=0x7fc734006df0).stop 2022-07-21T01:33:34.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440b3260 msgr2=0x7fc7440a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440b3260 0x7fc7440a41f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc73c0032f0 tx=0x7fc73c019000).stop 2022-07-21T01:33:34.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 shutdown_connections 2022-07-21T01:33:34.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7440b3260 0x7fc7440a41f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7440b08a0 0x7fc744064c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc728053a10 0x7fc728055ed0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 --2- 172.21.15.3:0/558906438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7440ad000 0x7fc7440676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.722+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 >> 172.21.15.3:0/558906438 conn(0x7fc7440612d0 msgr2=0x7fc744141580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:34.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.726+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 shutdown_connections 2022-07-21T01:33:34.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.726+0000 7fc74359e700 1 -- 172.21.15.3:0/558906438 wait complete. 2022-07-21T01:33:34.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.6 2022-07-21T01:33:34.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2022-07-21T01:33:34.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.842+0000 7f13e237e700 1 Processor -- start 2022-07-21T01:33:34.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.842+0000 7f13e237e700 1 -- start start 2022-07-21T01:33:34.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.842+0000 7f13e237e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc0b2730 0x7f13dc0666b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.842+0000 7f13e237e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc0654c0 con 0x7f13dc0b3190 2022-07-21T01:33:34.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.842+0000 7f13e237e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc0b3570 con 0x7f13dc066ba0 2022-07-21T01:33:34.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.842+0000 7f13e237e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc062190 con 0x7f13dc0b2730 2022-07-21T01:33:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.842+0000 7f13e137c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f13dc0b3190 0x7f13dc060ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51272/0 (socket says 172.21.15.3:51272) 2022-07-21T01:33:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.842+0000 7f13e137c700 1 -- 172.21.15.3:0/735288078 learned_addr learned my addr 172.21.15.3:0/735288078 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e0b7b700 1 --2- 172.21.15.3:0/735288078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc0b2730 0x7f13dc0666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13c7fff700 1 -- 172.21.15.3:0/735288078 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165994308 0 0) 0x7f13dc0654c0 con 0x7f13dc0b3190 2022-07-21T01:33:34.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13c7fff700 1 -- 172.21.15.3:0/735288078 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0003260 con 0x7f13dc0b3190 2022-07-21T01:33:34.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13c7fff700 1 -- 172.21.15.3:0/735288078 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409472119 0 0) 0x7f13dc0b3570 con 0x7f13dc066ba0 2022-07-21T01:33:34.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13c7fff700 1 -- 172.21.15.3:0/735288078 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13dc0654c0 con 0x7f13dc066ba0 2022-07-21T01:33:34.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e0b7b700 1 -- 172.21.15.3:0/735288078 >> v1:172.21.15.143:6789/0 conn(0x7f13dc066ba0 legacy=0x7f13dc14dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e0b7b700 1 -- 172.21.15.3:0/735288078 >> v1:172.21.15.3:6789/0 conn(0x7f13dc0b3190 legacy=0x7f13dc060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e0b7b700 1 -- 172.21.15.3:0/735288078 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc05fbf0 con 0x7f13dc0b2730 2022-07-21T01:33:34.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e0b7b700 1 --2- 172.21.15.3:0/735288078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc0b2730 0x7f13dc0666b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f13d8002f70 tx=0x7f13d8006df0).ready entity=mon.2 client_cookie=30584371a4074832 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13c7fff700 1 -- 172.21.15.3:0/735288078 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d800d4f0 con 0x7f13dc0b2730 2022-07-21T01:33:34.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13c7fff700 1 -- 172.21.15.3:0/735288078 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d8000e90 con 0x7f13dc0b2730 2022-07-21T01:33:34.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13c7fff700 1 -- 172.21.15.3:0/735288078 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d800d840 con 0x7f13dc0b2730 2022-07-21T01:33:34.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e237e700 1 -- 172.21.15.3:0/735288078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc0b2730 msgr2=0x7f13dc0666b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e237e700 1 --2- 172.21.15.3:0/735288078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc0b2730 0x7f13dc0666b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f13d8002f70 tx=0x7f13d8006df0).stop 2022-07-21T01:33:34.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e237e700 1 -- 172.21.15.3:0/735288078 shutdown_connections 2022-07-21T01:33:34.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e237e700 1 --2- 172.21.15.3:0/735288078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc0b2730 0x7f13dc0666b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e237e700 1 -- 172.21.15.3:0/735288078 >> 172.21.15.3:0/735288078 conn(0x7f13dc063870 msgr2=0x7f13dc063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:34.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e237e700 1 -- 172.21.15.3:0/735288078 shutdown_connections 2022-07-21T01:33:34.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.846+0000 7f13e237e700 1 -- 172.21.15.3:0/735288078 wait complete. 2022-07-21T01:33:34.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 Processor -- start 2022-07-21T01:33:34.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 -- start start 2022-07-21T01:33:34.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc066ba0 0x7f13dc155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13dc0b2730 0x7f13dc0a6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13dc0b3190 0x7f13dc0a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc156420 con 0x7f13dc0b3190 2022-07-21T01:33:34.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc0a6510 con 0x7f13dc0b2730 2022-07-21T01:33:34.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc156650 con 0x7f13dc066ba0 2022-07-21T01:33:34.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e1b7d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13dc0b3190 0x7f13dc0a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e1b7d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13dc0b3190 0x7f13dc0a6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43002/0 (socket says 172.21.15.3:43002) 2022-07-21T01:33:34.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e1b7d700 1 -- 172.21.15.3:0/1201350200 learned_addr learned my addr 172.21.15.3:0/1201350200 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:34.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e137c700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc066ba0 0x7f13dc155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e0b7b700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13dc0b2730 0x7f13dc0a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e1b7d700 1 -- 172.21.15.3:0/1201350200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc066ba0 msgr2=0x7f13dc155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e1b7d700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc066ba0 0x7f13dc155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e1b7d700 1 -- 172.21.15.3:0/1201350200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13dc0b2730 msgr2=0x7f13dc0a6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e1b7d700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13dc0b2730 0x7f13dc0a6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e1b7d700 1 -- 172.21.15.3:0/1201350200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d800b180 con 0x7f13dc0b3190 2022-07-21T01:33:34.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e1b7d700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13dc0b3190 0x7f13dc0a6cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f13d4001490 tx=0x7f13d4007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13ceffd700 1 -- 172.21.15.3:0/1201350200 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d4002310 con 0x7f13dc0b3190 2022-07-21T01:33:34.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13ceffd700 1 -- 172.21.15.3:0/1201350200 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d4002470 con 0x7f13dc0b3190 2022-07-21T01:33:34.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13ceffd700 1 -- 172.21.15.3:0/1201350200 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d4001840 con 0x7f13dc0b3190 2022-07-21T01:33:34.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc0aa4d0 con 0x7f13dc0b3190 2022-07-21T01:33:34.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.850+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc150280 con 0x7f13dc0b3190 2022-07-21T01:33:34.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.854+0000 7f13ceffd700 1 -- 172.21.15.3:0/1201350200 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f13d4010030 con 0x7f13dc0b3190 2022-07-21T01:33:34.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.854+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc05fe90 con 0x7f13dc0b3190 2022-07-21T01:33:34.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.854+0000 7f13ceffd700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13b80539e0 0x7f13b8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:34.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.854+0000 7f13ceffd700 1 -- 172.21.15.3:0/1201350200 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f13d406bf70 con 0x7f13dc0b3190 2022-07-21T01:33:34.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.854+0000 7f13e137c700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13b80539e0 0x7f13b8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:34.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.854+0000 7f13e137c700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13b80539e0 0x7f13b8055ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f13d000d000 tx=0x7f13d0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:34.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.862+0000 7f13ceffd700 1 -- 172.21.15.3:0/1201350200 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f13d4042ba0 con 0x7f13dc0b3190 2022-07-21T01:33:34.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.982+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f13dc0574e0 con 0x7f13dc0b3190 2022-07-21T01:33:34.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.982+0000 7f13ceffd700 1 -- 172.21.15.3:0/1201350200 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f13d40461c0 con 0x7f13dc0b3190 2022-07-21T01:33:34.986 INFO:teuthology.orchestra.run.smithi003.stdout:60129542150 2022-07-21T01:33:34.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13b80539e0 msgr2=0x7f13b8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13b80539e0 0x7f13b8055ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f13d000d000 tx=0x7f13d0009040).stop 2022-07-21T01:33:34.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13dc0b3190 msgr2=0x7f13dc0a6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:34.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13dc0b3190 0x7f13dc0a6cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f13d4001490 tx=0x7f13d4007dd0).stop 2022-07-21T01:33:34.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 shutdown_connections 2022-07-21T01:33:34.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13dc0b3190 0x7f13dc0a6cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13b80539e0 0x7f13b8055ea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13dc0b2730 0x7f13dc0a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 --2- 172.21.15.3:0/1201350200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13dc066ba0 0x7f13dc155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:34.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 >> 172.21.15.3:0/1201350200 conn(0x7f13dc063870 msgr2=0x7f13dc1474f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:34.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 shutdown_connections 2022-07-21T01:33:34.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:34.986+0000 7f13e237e700 1 -- 172.21.15.3:0/1201350200 wait complete. 2022-07-21T01:33:34.997 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2022-07-21T01:33:34.998 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2022-07-21T01:33:34.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-21T01:33:35.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884ddd6700 1 Processor -- start 2022-07-21T01:33:35.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884ddd6700 1 -- start start 2022-07-21T01:33:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884ddd6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f88480b3190 0x7f8848060ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884ddd6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8848061190 0x7f8848066dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884ddd6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88480654c0 con 0x7f88480b2730 2022-07-21T01:33:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884ddd6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8848062190 con 0x7f88480b3190 2022-07-21T01:33:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884ddd6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88480615d0 con 0x7f8848061190 2022-07-21T01:33:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884cdd4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f88480b3190 0x7f8848060ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884cdd4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f88480b3190 0x7f8848060ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43006/0 (socket says 172.21.15.3:43006) 2022-07-21T01:33:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f8847fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f88480b2730 0x7f88480666b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51288/0 (socket says 172.21.15.3:51288) 2022-07-21T01:33:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884cdd4700 1 -- 172.21.15.3:0/3586333419 learned_addr learned my addr 172.21.15.3:0/3586333419 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884d5d5700 1 --2- 172.21.15.3:0/3586333419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8848061190 0x7f8848066dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884cdd4700 1 -- 172.21.15.3:0/3586333419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8848061190 msgr2=0x7f8848066dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884cdd4700 1 --2- 172.21.15.3:0/3586333419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8848061190 0x7f8848066dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884cdd4700 1 -- 172.21.15.3:0/3586333419 >> v1:172.21.15.3:6789/0 conn(0x7f88480b2730 legacy=0x7f88480666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884cdd4700 1 -- 172.21.15.3:0/3586333419 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88480b3570 con 0x7f88480b3190 2022-07-21T01:33:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884cdd4700 1 --2- 172.21.15.3:0/3586333419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88480b3190 0x7f8848060ca0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f8848061710 tx=0x7f883c006eb0).ready entity=mon.0 client_cookie=76499b80dbdbb7de server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f88477fe700 1 -- 172.21.15.3:0/3586333419 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f883c00d5c0 con 0x7f88480b3190 2022-07-21T01:33:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f88477fe700 1 -- 172.21.15.3:0/3586333419 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883c000e90 con 0x7f88480b3190 2022-07-21T01:33:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f88477fe700 1 -- 172.21.15.3:0/3586333419 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f883c00d950 con 0x7f88480b3190 2022-07-21T01:33:35.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884ddd6700 1 -- 172.21.15.3:0/3586333419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88480b3190 msgr2=0x7f8848060ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.102+0000 7f884ddd6700 1 --2- 172.21.15.3:0/3586333419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88480b3190 0x7f8848060ca0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f8848061710 tx=0x7f883c006eb0).stop 2022-07-21T01:33:35.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.106+0000 7f884ddd6700 1 -- 172.21.15.3:0/3586333419 shutdown_connections 2022-07-21T01:33:35.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.106+0000 7f884ddd6700 1 --2- 172.21.15.3:0/3586333419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8848061190 0x7f8848066dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.106+0000 7f884ddd6700 1 --2- 172.21.15.3:0/3586333419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88480b3190 0x7f8848060ca0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.106+0000 7f884ddd6700 1 -- 172.21.15.3:0/3586333419 >> 172.21.15.3:0/3586333419 conn(0x7f8848063870 msgr2=0x7f8848063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:35.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.106+0000 7f884ddd6700 1 -- 172.21.15.3:0/3586333419 shutdown_connections 2022-07-21T01:33:35.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.106+0000 7f884ddd6700 1 -- 172.21.15.3:0/3586333419 wait complete. 2022-07-21T01:33:35.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.106+0000 7f884ddd6700 1 Processor -- start 2022-07-21T01:33:35.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.106+0000 7f884ddd6700 1 -- start start 2022-07-21T01:33:35.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884ddd6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8848061190 0x7f88480b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884ddd6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88480b2730 0x7f88480acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884ddd6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88480b3190 0x7f8848155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884ddd6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88480ac500 con 0x7f8848061190 2022-07-21T01:33:35.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884ddd6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8848143db0 con 0x7f88480b2730 2022-07-21T01:33:35.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884ddd6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88480adf10 con 0x7f88480b3190 2022-07-21T01:33:35.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884cdd4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8848061190 0x7f88480b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884cdd4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8848061190 0x7f88480b6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43012/0 (socket says 172.21.15.3:43012) 2022-07-21T01:33:35.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884cdd4700 1 -- 172.21.15.3:0/3162437995 learned_addr learned my addr 172.21.15.3:0/3162437995 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:35.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f8847fff700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88480b2730 0x7f88480acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884d5d5700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88480b3190 0x7f8848155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884cdd4700 1 -- 172.21.15.3:0/3162437995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88480b3190 msgr2=0x7f8848155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884cdd4700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88480b3190 0x7f8848155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884cdd4700 1 -- 172.21.15.3:0/3162437995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88480b2730 msgr2=0x7f88480acbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884cdd4700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88480b2730 0x7f88480acbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884cdd4700 1 -- 172.21.15.3:0/3162437995 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c00b1e0 con 0x7f8848061190 2022-07-21T01:33:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884d5d5700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88480b3190 0x7f8848155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884cdd4700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8848061190 0x7f88480b6440 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f8848061710 tx=0x7f883c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f8845ffb700 1 -- 172.21.15.3:0/3162437995 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f883c011710 con 0x7f8848061190 2022-07-21T01:33:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f8845ffb700 1 -- 172.21.15.3:0/3162437995 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883c011870 con 0x7f8848061190 2022-07-21T01:33:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88480ac730 con 0x7f8848061190 2022-07-21T01:33:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f8845ffb700 1 -- 172.21.15.3:0/3162437995 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f883c011bd0 con 0x7f8848061190 2022-07-21T01:33:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88480adb90 con 0x7f8848061190 2022-07-21T01:33:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f8845ffb700 1 -- 172.21.15.3:0/3162437995 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f883c00f070 con 0x7f8848061190 2022-07-21T01:33:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.110+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88480b6a20 con 0x7f8848061190 2022-07-21T01:33:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.114+0000 7f8845ffb700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8828053a70 0x7f8828055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.114+0000 7f8845ffb700 1 -- 172.21.15.3:0/3162437995 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f883c06d980 con 0x7f8848061190 2022-07-21T01:33:35.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.114+0000 7f8847fff700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8828053a70 0x7f8828055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.114+0000 7f8847fff700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8828053a70 0x7f8828055f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f883400d000 tx=0x7f8834007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.118+0000 7f8845ffb700 1 -- 172.21.15.3:0/3162437995 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f883c044740 con 0x7f8848061190 2022-07-21T01:33:35.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.242+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f884805fbf0 con 0x7f8828053a70 2022-07-21T01:33:35.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.250+0000 7f8845ffb700 1 -- 172.21.15.3:0/3162437995 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82411 (secure 0 0 0) 0x7f884805fbf0 con 0x7f8828053a70 2022-07-21T01:33:35.250 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:35.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8828053a70 msgr2=0x7f8828055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8828053a70 0x7f8828055f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f883400d000 tx=0x7f8834007b80).stop 2022-07-21T01:33:35.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8848061190 msgr2=0x7f88480b6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8848061190 0x7f88480b6440 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f8848061710 tx=0x7f883c015000).stop 2022-07-21T01:33:35.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 shutdown_connections 2022-07-21T01:33:35.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88480b3190 0x7f8848155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8828053a70 0x7f8828055f30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88480b2730 0x7f88480acbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 --2- 172.21.15.3:0/3162437995 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8848061190 0x7f88480b6440 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 >> 172.21.15.3:0/3162437995 conn(0x7f8848063870 msgr2=0x7f8848142b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:35.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 shutdown_connections 2022-07-21T01:33:35.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.254+0000 7f884ddd6700 1 -- 172.21.15.3:0/3162437995 wait complete. 2022-07-21T01:33:35.255 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2022-07-21T01:33:35.263 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2022-07-21T01:33:33.551497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390848,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746583872,"statfs":{"total":773094113280,"available":764501884928,"internally_reserved":8589934592,"allocated":2228224,"data_stored":868664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001894"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.106952+0000","last_change":"2022-07-21T01:33:26.099757+0000","last_active":"2022-07-21T01:33:26.106952+0000","last_peered":"2022-07-21T01:33:26.106952+0000","last_clean":"2022-07-21T01:33:26.106952+0000","last_became_active":"2022-07-21T01:33:26.098731+0000","last_became_peered":"2022-07-21T01:33:26.098731+0000","last_unstale":"2022-07-21T01:33:26.106952+0000","last_undegraded":"2022-07-21T01:33:26.106952+0000","last_fullsized":"2022-07-21T01:33:26.106952+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'2","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.095344+0000","last_change":"2022-07-21T01:33:15.640655+0000","last_active":"2022-07-21T01:33:26.095344+0000","last_peered":"2022-07-21T01:33:26.095344+0000","last_clean":"2022-07-21T01:33:26.095344+0000","last_became_active":"2022-07-21T01:33:15.639383+0000","last_became_peered":"2022-07-21T01:33:15.639383+0000","last_unstale":"2022-07-21T01:33:26.095344+0000","last_undegraded":"2022-07-21T01:33:26.095344+0000","last_fullsized":"2022-07-21T01:33:26.095344+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:10.553176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:10.553176+0000","last_clean_scrub_stamp":"2022-07-21T01:33:10.553176+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.042430+0000","last_change":"2022-07-21T01:33:19.894256+0000","last_active":"2022-07-21T01:33:26.042430+0000","last_peered":"2022-07-21T01:33:26.042430+0000","last_clean":"2022-07-21T01:33:26.042430+0000","last_became_active":"2022-07-21T01:33:19.893274+0000","last_became_peered":"2022-07-21T01:33:19.893274+0000","last_unstale":"2022-07-21T01:33:26.042430+0000","last_undegraded":"2022-07-21T01:33:26.042430+0000","last_fullsized":"2022-07-21T01:33:26.042430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"26'1","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.379252+0000","last_change":"2022-07-21T01:33:23.026022+0000","last_active":"2022-07-21T01:33:26.379252+0000","last_peered":"2022-07-21T01:33:26.379252+0000","last_clean":"2022-07-21T01:33:26.379252+0000","last_became_active":"2022-07-21T01:33:23.025460+0000","last_became_peered":"2022-07-21T01:33:23.025460+0000","last_unstale":"2022-07-21T01:33:26.379252+0000","last_undegraded":"2022-07-21T01:33:26.379252+0000","last_fullsized":"2022-07-21T01:33:26.379252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.107314+0000","last_change":"2022-07-21T01:33:26.102400+0000","last_active":"2022-07-21T01:33:26.107314+0000","last_peered":"2022-07-21T01:33:26.107314+0000","last_clean":"2022-07-21T01:33:26.107314+0000","last_became_active":"2022-07-21T01:33:26.101853+0000","last_became_peered":"2022-07-21T01:33:26.101853+0000","last_unstale":"2022-07-21T01:33:26.107314+0000","last_undegraded":"2022-07-21T01:33:26.107314+0000","last_fullsized":"2022-07-21T01:33:26.107314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.094954+0000","last_change":"2022-07-21T01:33:19.897103+0000","last_active":"2022-07-21T01:33:26.094954+0000","last_peered":"2022-07-21T01:33:26.094954+0000","last_clean":"2022-07-21T01:33:26.094954+0000","last_became_active":"2022-07-21T01:33:19.896369+0000","last_became_peered":"2022-07-21T01:33:19.896369+0000","last_unstale":"2022-07-21T01:33:26.094954+0000","last_undegraded":"2022-07-21T01:33:26.094954+0000","last_fullsized":"2022-07-21T01:33:26.094954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.053137+0000","last_change":"2022-07-21T01:33:23.033774+0000","last_active":"2022-07-21T01:33:26.053137+0000","last_peered":"2022-07-21T01:33:26.053137+0000","last_clean":"2022-07-21T01:33:26.053137+0000","last_became_active":"2022-07-21T01:33:23.033399+0000","last_became_peered":"2022-07-21T01:33:23.033399+0000","last_unstale":"2022-07-21T01:33:26.053137+0000","last_undegraded":"2022-07-21T01:33:26.053137+0000","last_fullsized":"2022-07-21T01:33:26.053137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.064880+0000","last_change":"2022-07-21T01:33:26.062702+0000","last_active":"2022-07-21T01:33:26.064880+0000","last_peered":"2022-07-21T01:33:26.064880+0000","last_clean":"2022-07-21T01:33:26.064880+0000","last_became_active":"2022-07-21T01:33:26.062150+0000","last_became_peered":"2022-07-21T01:33:26.062150+0000","last_unstale":"2022-07-21T01:33:26.064880+0000","last_undegraded":"2022-07-21T01:33:26.064880+0000","last_fullsized":"2022-07-21T01:33:26.064880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.096392+0000","last_change":"2022-07-21T01:33:19.896866+0000","last_active":"2022-07-21T01:33:26.096392+0000","last_peered":"2022-07-21T01:33:26.096392+0000","last_clean":"2022-07-21T01:33:26.096392+0000","last_became_active":"2022-07-21T01:33:19.896120+0000","last_became_peered":"2022-07-21T01:33:19.896120+0000","last_unstale":"2022-07-21T01:33:26.096392+0000","last_undegraded":"2022-07-21T01:33:26.096392+0000","last_fullsized":"2022-07-21T01:33:26.096392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"26'8","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.398930+0000","last_change":"2022-07-21T01:33:23.027820+0000","last_active":"2022-07-21T01:33:26.398930+0000","last_peered":"2022-07-21T01:33:26.398930+0000","last_clean":"2022-07-21T01:33:26.398930+0000","last_became_active":"2022-07-21T01:33:23.027431+0000","last_became_peered":"2022-07-21T01:33:23.027431+0000","last_unstale":"2022-07-21T01:33:26.398930+0000","last_undegraded":"2022-07-21T01:33:26.398930+0000","last_fullsized":"2022-07-21T01:33:26.398930+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.104927+0000","last_change":"2022-07-21T01:33:26.102179+0000","last_active":"2022-07-21T01:33:26.104927+0000","last_peered":"2022-07-21T01:33:26.104927+0000","last_clean":"2022-07-21T01:33:26.104927+0000","last_became_active":"2022-07-21T01:33:26.100679+0000","last_became_peered":"2022-07-21T01:33:26.100679+0000","last_unstale":"2022-07-21T01:33:26.104927+0000","last_undegraded":"2022-07-21T01:33:26.104927+0000","last_fullsized":"2022-07-21T01:33:26.104927+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.042426+0000","last_change":"2022-07-21T01:33:19.896530+0000","last_active":"2022-07-21T01:33:26.042426+0000","last_peered":"2022-07-21T01:33:26.042426+0000","last_clean":"2022-07-21T01:33:26.042426+0000","last_became_active":"2022-07-21T01:33:19.896110+0000","last_became_peered":"2022-07-21T01:33:19.896110+0000","last_unstale":"2022-07-21T01:33:26.042426+0000","last_undegraded":"2022-07-21T01:33:26.042426+0000","last_fullsized":"2022-07-21T01:33:26.042426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"26'5","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.395402+0000","last_change":"2022-07-21T01:33:23.030652+0000","last_active":"2022-07-21T01:33:26.395402+0000","last_peered":"2022-07-21T01:33:26.395402+0000","last_clean":"2022-07-21T01:33:26.395402+0000","last_became_active":"2022-07-21T01:33:23.030034+0000","last_became_peered":"2022-07-21T01:33:23.030034+0000","last_unstale":"2022-07-21T01:33:26.395402+0000","last_undegraded":"2022-07-21T01:33:26.395402+0000","last_fullsized":"2022-07-21T01:33:26.395402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.061662+0000","last_change":"2022-07-21T01:33:26.058400+0000","last_active":"2022-07-21T01:33:26.061662+0000","last_peered":"2022-07-21T01:33:26.061662+0000","last_clean":"2022-07-21T01:33:26.061662+0000","last_became_active":"2022-07-21T01:33:26.057650+0000","last_became_peered":"2022-07-21T01:33:26.057650+0000","last_unstale":"2022-07-21T01:33:26.061662+0000","last_undegraded":"2022-07-21T01:33:26.061662+0000","last_fullsized":"2022-07-21T01:33:26.061662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"26'6","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.392975+0000","last_change":"2022-07-21T01:33:23.022383+0000","last_active":"2022-07-21T01:33:26.392975+0000","last_peered":"2022-07-21T01:33:26.392975+0000","last_clean":"2022-07-21T01:33:26.392975+0000","last_became_active":"2022-07-21T01:33:23.021525+0000","last_became_peered":"2022-07-21T01:33:23.021525+0000","last_unstale":"2022-07-21T01:33:26.392975+0000","last_undegraded":"2022-07-21T01:33:26.392975+0000","last_fullsized":"2022-07-21T01:33:26.392975+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.050021+0000","last_change":"2022-07-21T01:33:20.650822+0000","last_active":"2022-07-21T01:33:26.050021+0000","last_peered":"2022-07-21T01:33:26.050021+0000","last_clean":"2022-07-21T01:33:26.050021+0000","last_became_active":"2022-07-21T01:33:20.649660+0000","last_became_peered":"2022-07-21T01:33:20.649660+0000","last_unstale":"2022-07-21T01:33:26.050021+0000","last_undegraded":"2022-07-21T01:33:26.050021+0000","last_fullsized":"2022-07-21T01:33:26.050021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.107397+0000","last_change":"2022-07-21T01:33:26.103108+0000","last_active":"2022-07-21T01:33:26.107397+0000","last_peered":"2022-07-21T01:33:26.107397+0000","last_clean":"2022-07-21T01:33:26.107397+0000","last_became_active":"2022-07-21T01:33:26.101966+0000","last_became_peered":"2022-07-21T01:33:26.101966+0000","last_unstale":"2022-07-21T01:33:26.107397+0000","last_undegraded":"2022-07-21T01:33:26.107397+0000","last_fullsized":"2022-07-21T01:33:26.107397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"26'8","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.399081+0000","last_change":"2022-07-21T01:33:23.025863+0000","last_active":"2022-07-21T01:33:26.399081+0000","last_peered":"2022-07-21T01:33:26.399081+0000","last_clean":"2022-07-21T01:33:26.399081+0000","last_became_active":"2022-07-21T01:33:23.024819+0000","last_became_peered":"2022-07-21T01:33:23.024819+0000","last_unstale":"2022-07-21T01:33:26.399081+0000","last_undegraded":"2022-07-21T01:33:26.399081+0000","last_fullsized":"2022-07-21T01:33:26.399081+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.094954+0000","last_change":"2022-07-21T01:33:20.648160+0000","last_active":"2022-07-21T01:33:26.094954+0000","last_peered":"2022-07-21T01:33:26.094954+0000","last_clean":"2022-07-21T01:33:26.094954+0000","last_became_active":"2022-07-21T01:33:20.647021+0000","last_became_peered":"2022-07-21T01:33:20.647021+0000","last_unstale":"2022-07-21T01:33:26.094954+0000","last_undegraded":"2022-07-21T01:33:26.094954+0000","last_fullsized":"2022-07-21T01:33:26.094954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.105341+0000","last_change":"2022-07-21T01:33:26.102665+0000","last_active":"2022-07-21T01:33:26.105341+0000","last_peered":"2022-07-21T01:33:26.105341+0000","last_clean":"2022-07-21T01:33:26.105341+0000","last_became_active":"2022-07-21T01:33:26.101807+0000","last_became_peered":"2022-07-21T01:33:26.101807+0000","last_unstale":"2022-07-21T01:33:26.105341+0000","last_undegraded":"2022-07-21T01:33:26.105341+0000","last_fullsized":"2022-07-21T01:33:26.105341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"26'7","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.398682+0000","last_change":"2022-07-21T01:33:23.247504+0000","last_active":"2022-07-21T01:33:26.398682+0000","last_peered":"2022-07-21T01:33:26.398682+0000","last_clean":"2022-07-21T01:33:26.398682+0000","last_became_active":"2022-07-21T01:33:23.246793+0000","last_became_peered":"2022-07-21T01:33:23.246793+0000","last_unstale":"2022-07-21T01:33:26.398682+0000","last_undegraded":"2022-07-21T01:33:26.398682+0000","last_fullsized":"2022-07-21T01:33:26.398682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.050281+0000","last_change":"2022-07-21T01:33:19.900435+0000","last_active":"2022-07-21T01:33:26.050281+0000","last_peered":"2022-07-21T01:33:26.050281+0000","last_clean":"2022-07-21T01:33:26.050281+0000","last_became_active":"2022-07-21T01:33:19.900054+0000","last_became_peered":"2022-07-21T01:33:19.900054+0000","last_unstale":"2022-07-21T01:33:26.050281+0000","last_undegraded":"2022-07-21T01:33:26.050281+0000","last_fullsized":"2022-07-21T01:33:26.050281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.101474+0000","last_change":"2022-07-21T01:33:26.100220+0000","last_active":"2022-07-21T01:33:26.101474+0000","last_peered":"2022-07-21T01:33:26.101474+0000","last_clean":"2022-07-21T01:33:26.101474+0000","last_became_active":"2022-07-21T01:33:26.099164+0000","last_became_peered":"2022-07-21T01:33:26.099164+0000","last_unstale":"2022-07-21T01:33:26.101474+0000","last_undegraded":"2022-07-21T01:33:26.101474+0000","last_fullsized":"2022-07-21T01:33:26.101474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"26'4","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.399245+0000","last_change":"2022-07-21T01:33:23.028971+0000","last_active":"2022-07-21T01:33:26.399245+0000","last_peered":"2022-07-21T01:33:26.399245+0000","last_clean":"2022-07-21T01:33:26.399245+0000","last_became_active":"2022-07-21T01:33:23.028634+0000","last_became_peered":"2022-07-21T01:33:23.028634+0000","last_unstale":"2022-07-21T01:33:26.399245+0000","last_undegraded":"2022-07-21T01:33:26.399245+0000","last_fullsized":"2022-07-21T01:33:26.399245+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.047375+0000","last_change":"2022-07-21T01:33:19.895590+0000","last_active":"2022-07-21T01:33:26.047375+0000","last_peered":"2022-07-21T01:33:26.047375+0000","last_clean":"2022-07-21T01:33:26.047375+0000","last_became_active":"2022-07-21T01:33:19.895046+0000","last_became_peered":"2022-07-21T01:33:19.895046+0000","last_unstale":"2022-07-21T01:33:26.047375+0000","last_undegraded":"2022-07-21T01:33:26.047375+0000","last_fullsized":"2022-07-21T01:33:26.047375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048860,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322980,"statfs":{"total":96636764160,"available":95562731520,"internally_reserved":1073741824,"allocated":282624,"data_stored":109542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":108041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048864,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322976,"statfs":{"total":96636764160,"available":95562727424,"internally_reserved":1073741824,"allocated":286720,"data_stored":108742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":108582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":108007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":108167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":108041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048860,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322980,"statfs":{"total":96636764160,"available":95562731520,"internally_reserved":1073741824,"allocated":282624,"data_stored":109542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]}]}]}} 2022-07-21T01:33:35.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-21T01:33:35.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.370+0000 7fa9b1262700 1 Processor -- start 2022-07-21T01:33:35.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.370+0000 7fa9b1262700 1 -- start start 2022-07-21T01:33:35.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa9ac0b1f40 0x7fa9ac0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0b04e0 con 0x7fa9ac0a7250 2022-07-21T01:33:35.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0b0780 con 0x7fa9ac148530 2022-07-21T01:33:35.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac0a9250 con 0x7fa9ac0b1f40 2022-07-21T01:33:35.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9aaffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa9ac0b1f40 0x7fa9ac0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9ab7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa9ac0a7250 0x7fa9ac1448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51304/0 (socket says 172.21.15.3:51304) 2022-07-21T01:33:35.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9aaffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa9ac0b1f40 0x7fa9ac0b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43022/0 (socket says 172.21.15.3:43022) 2022-07-21T01:33:35.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9ab7fe700 1 -- 172.21.15.3:0/3861948642 learned_addr learned my addr 172.21.15.3:0/3861948642 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:35.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9aaffd700 1 -- 172.21.15.3:0/3861948642 >> v1:172.21.15.143:6789/0 conn(0x7fa9ac148530 legacy=0x7fa9ac0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9a9ffb700 1 -- 172.21.15.3:0/3861948642 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984270688 0 0) 0x7fa9ac0b04e0 con 0x7fa9ac0a7250 2022-07-21T01:33:35.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9aaffd700 1 -- 172.21.15.3:0/3861948642 >> v1:172.21.15.3:6789/0 conn(0x7fa9ac0a7250 legacy=0x7fa9ac1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9aaffd700 1 -- 172.21.15.3:0/3861948642 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac0b0a20 con 0x7fa9ac0b1f40 2022-07-21T01:33:35.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9aaffd700 1 --2- 172.21.15.3:0/3861948642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0b1f40 0x7fa9ac0b1160 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fa9ac1478c0 tx=0x7fa9a0006df0).ready entity=mon.0 client_cookie=14bad8a91ae172c4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9a9ffb700 1 -- 172.21.15.3:0/3861948642 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a000d5c0 con 0x7fa9ac0b1f40 2022-07-21T01:33:35.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9a9ffb700 1 -- 172.21.15.3:0/3861948642 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a0000e90 con 0x7fa9ac0b1f40 2022-07-21T01:33:35.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9a9ffb700 1 -- 172.21.15.3:0/3861948642 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a000d910 con 0x7fa9ac0b1f40 2022-07-21T01:33:35.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 -- 172.21.15.3:0/3861948642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0b1f40 msgr2=0x7fa9ac0b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 --2- 172.21.15.3:0/3861948642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0b1f40 0x7fa9ac0b1160 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fa9ac1478c0 tx=0x7fa9a0006df0).stop 2022-07-21T01:33:35.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 -- 172.21.15.3:0/3861948642 shutdown_connections 2022-07-21T01:33:35.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 --2- 172.21.15.3:0/3861948642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0b1f40 0x7fa9ac0b1160 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 -- 172.21.15.3:0/3861948642 >> 172.21.15.3:0/3861948642 conn(0x7fa9ac05f840 msgr2=0x7fa9ac05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:35.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 -- 172.21.15.3:0/3861948642 shutdown_connections 2022-07-21T01:33:35.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.374+0000 7fa9b1262700 1 -- 172.21.15.3:0/3861948642 wait complete. 2022-07-21T01:33:35.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 Processor -- start 2022-07-21T01:33:35.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 -- start start 2022-07-21T01:33:35.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0a7250 0x7fa9ac063610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9ac0b1f40 0x7fa9ac063b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9ac148530 0x7fa9ac15a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac063ff0 con 0x7fa9ac0a7250 2022-07-21T01:33:35.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac157b10 con 0x7fa9ac148530 2022-07-21T01:33:35.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac157870 con 0x7fa9ac0b1f40 2022-07-21T01:33:35.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0a7250 0x7fa9ac063610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0a7250 0x7fa9ac063610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43026/0 (socket says 172.21.15.3:43026) 2022-07-21T01:33:35.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aaffd700 1 -- 172.21.15.3:0/3928639008 learned_addr learned my addr 172.21.15.3:0/3928639008 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:35.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9ab7fe700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9ac148530 0x7fa9ac15a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aa7fc700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9ac0b1f40 0x7fa9ac063b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aaffd700 1 -- 172.21.15.3:0/3928639008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9ac0b1f40 msgr2=0x7fa9ac063b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aaffd700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9ac0b1f40 0x7fa9ac063b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aaffd700 1 -- 172.21.15.3:0/3928639008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9ac148530 msgr2=0x7fa9ac15a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aaffd700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9ac148530 0x7fa9ac15a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aaffd700 1 -- 172.21.15.3:0/3928639008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a000b180 con 0x7fa9ac0a7250 2022-07-21T01:33:35.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9aaffd700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0a7250 0x7fa9ac063610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa9ac064130 tx=0x7fa9a0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa99bfff700 1 -- 172.21.15.3:0/3928639008 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a0002f50 con 0x7fa9ac0a7250 2022-07-21T01:33:35.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa99bfff700 1 -- 172.21.15.3:0/3928639008 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a001e400 con 0x7fa9ac0a7250 2022-07-21T01:33:35.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa99bfff700 1 -- 172.21.15.3:0/3928639008 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a001e760 con 0x7fa9ac0a7250 2022-07-21T01:33:35.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac0a9ce0 con 0x7fa9ac0a7250 2022-07-21T01:33:35.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac14b9d0 con 0x7fa9ac0a7250 2022-07-21T01:33:35.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.378+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac0aab20 con 0x7fa9ac0a7250 2022-07-21T01:33:35.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.382+0000 7fa99bfff700 1 -- 172.21.15.3:0/3928639008 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa9a0025030 con 0x7fa9ac0a7250 2022-07-21T01:33:35.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.382+0000 7fa99bfff700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa988053b00 0x7fa988055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.382+0000 7fa99bfff700 1 -- 172.21.15.3:0/3928639008 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa9a006cff0 con 0x7fa9ac0a7250 2022-07-21T01:33:35.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.382+0000 7fa9aa7fc700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa988053b00 0x7fa988055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.382+0000 7fa9aa7fc700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa988053b00 0x7fa988055fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fa99c00b000 tx=0x7fa99c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.390+0000 7fa99bfff700 1 -- 172.21.15.3:0/3928639008 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9ac0aab20 con 0x7fa9ac0a7250 2022-07-21T01:33:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.510+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9ac0aa180 con 0x7fa988053b00 2022-07-21T01:33:35.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa99bfff700 1 -- 172.21.15.3:0/3928639008 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82411 (secure 0 0 0) 0x7fa99c004100 con 0x7fa988053b00 2022-07-21T01:33:35.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:35.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa988053b00 msgr2=0x7fa988055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa988053b00 0x7fa988055fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fa99c00b000 tx=0x7fa99c009040).stop 2022-07-21T01:33:35.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0a7250 msgr2=0x7fa9ac063610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0a7250 0x7fa9ac063610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa9ac064130 tx=0x7fa9a0014000).stop 2022-07-21T01:33:35.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 shutdown_connections 2022-07-21T01:33:35.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa988053b00 0x7fa988055fc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9ac148530 0x7fa9ac15a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9ac0b1f40 0x7fa9ac063b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 --2- 172.21.15.3:0/3928639008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9ac0a7250 0x7fa9ac063610 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 >> 172.21.15.3:0/3928639008 conn(0x7fa9ac05f840 msgr2=0x7fa9ac143ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:35.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 shutdown_connections 2022-07-21T01:33:35.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.514+0000 7fa9b1262700 1 -- 172.21.15.3:0/3928639008 wait complete. 2022-07-21T01:33:35.516 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2022-07-21T01:33:35.523 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2022-07-21T01:33:33.551497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390848,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746583872,"statfs":{"total":773094113280,"available":764501884928,"internally_reserved":8589934592,"allocated":2228224,"data_stored":868664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001894"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.106952+0000","last_change":"2022-07-21T01:33:26.099757+0000","last_active":"2022-07-21T01:33:26.106952+0000","last_peered":"2022-07-21T01:33:26.106952+0000","last_clean":"2022-07-21T01:33:26.106952+0000","last_became_active":"2022-07-21T01:33:26.098731+0000","last_became_peered":"2022-07-21T01:33:26.098731+0000","last_unstale":"2022-07-21T01:33:26.106952+0000","last_undegraded":"2022-07-21T01:33:26.106952+0000","last_fullsized":"2022-07-21T01:33:26.106952+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'2","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.095344+0000","last_change":"2022-07-21T01:33:15.640655+0000","last_active":"2022-07-21T01:33:26.095344+0000","last_peered":"2022-07-21T01:33:26.095344+0000","last_clean":"2022-07-21T01:33:26.095344+0000","last_became_active":"2022-07-21T01:33:15.639383+0000","last_became_peered":"2022-07-21T01:33:15.639383+0000","last_unstale":"2022-07-21T01:33:26.095344+0000","last_undegraded":"2022-07-21T01:33:26.095344+0000","last_fullsized":"2022-07-21T01:33:26.095344+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:10.553176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:10.553176+0000","last_clean_scrub_stamp":"2022-07-21T01:33:10.553176+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.042430+0000","last_change":"2022-07-21T01:33:19.894256+0000","last_active":"2022-07-21T01:33:26.042430+0000","last_peered":"2022-07-21T01:33:26.042430+0000","last_clean":"2022-07-21T01:33:26.042430+0000","last_became_active":"2022-07-21T01:33:19.893274+0000","last_became_peered":"2022-07-21T01:33:19.893274+0000","last_unstale":"2022-07-21T01:33:26.042430+0000","last_undegraded":"2022-07-21T01:33:26.042430+0000","last_fullsized":"2022-07-21T01:33:26.042430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"26'1","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.379252+0000","last_change":"2022-07-21T01:33:23.026022+0000","last_active":"2022-07-21T01:33:26.379252+0000","last_peered":"2022-07-21T01:33:26.379252+0000","last_clean":"2022-07-21T01:33:26.379252+0000","last_became_active":"2022-07-21T01:33:23.025460+0000","last_became_peered":"2022-07-21T01:33:23.025460+0000","last_unstale":"2022-07-21T01:33:26.379252+0000","last_undegraded":"2022-07-21T01:33:26.379252+0000","last_fullsized":"2022-07-21T01:33:26.379252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.107314+0000","last_change":"2022-07-21T01:33:26.102400+0000","last_active":"2022-07-21T01:33:26.107314+0000","last_peered":"2022-07-21T01:33:26.107314+0000","last_clean":"2022-07-21T01:33:26.107314+0000","last_became_active":"2022-07-21T01:33:26.101853+0000","last_became_peered":"2022-07-21T01:33:26.101853+0000","last_unstale":"2022-07-21T01:33:26.107314+0000","last_undegraded":"2022-07-21T01:33:26.107314+0000","last_fullsized":"2022-07-21T01:33:26.107314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.094954+0000","last_change":"2022-07-21T01:33:19.897103+0000","last_active":"2022-07-21T01:33:26.094954+0000","last_peered":"2022-07-21T01:33:26.094954+0000","last_clean":"2022-07-21T01:33:26.094954+0000","last_became_active":"2022-07-21T01:33:19.896369+0000","last_became_peered":"2022-07-21T01:33:19.896369+0000","last_unstale":"2022-07-21T01:33:26.094954+0000","last_undegraded":"2022-07-21T01:33:26.094954+0000","last_fullsized":"2022-07-21T01:33:26.094954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.053137+0000","last_change":"2022-07-21T01:33:23.033774+0000","last_active":"2022-07-21T01:33:26.053137+0000","last_peered":"2022-07-21T01:33:26.053137+0000","last_clean":"2022-07-21T01:33:26.053137+0000","last_became_active":"2022-07-21T01:33:23.033399+0000","last_became_peered":"2022-07-21T01:33:23.033399+0000","last_unstale":"2022-07-21T01:33:26.053137+0000","last_undegraded":"2022-07-21T01:33:26.053137+0000","last_fullsized":"2022-07-21T01:33:26.053137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.064880+0000","last_change":"2022-07-21T01:33:26.062702+0000","last_active":"2022-07-21T01:33:26.064880+0000","last_peered":"2022-07-21T01:33:26.064880+0000","last_clean":"2022-07-21T01:33:26.064880+0000","last_became_active":"2022-07-21T01:33:26.062150+0000","last_became_peered":"2022-07-21T01:33:26.062150+0000","last_unstale":"2022-07-21T01:33:26.064880+0000","last_undegraded":"2022-07-21T01:33:26.064880+0000","last_fullsized":"2022-07-21T01:33:26.064880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.096392+0000","last_change":"2022-07-21T01:33:19.896866+0000","last_active":"2022-07-21T01:33:26.096392+0000","last_peered":"2022-07-21T01:33:26.096392+0000","last_clean":"2022-07-21T01:33:26.096392+0000","last_became_active":"2022-07-21T01:33:19.896120+0000","last_became_peered":"2022-07-21T01:33:19.896120+0000","last_unstale":"2022-07-21T01:33:26.096392+0000","last_undegraded":"2022-07-21T01:33:26.096392+0000","last_fullsized":"2022-07-21T01:33:26.096392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"26'8","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.398930+0000","last_change":"2022-07-21T01:33:23.027820+0000","last_active":"2022-07-21T01:33:26.398930+0000","last_peered":"2022-07-21T01:33:26.398930+0000","last_clean":"2022-07-21T01:33:26.398930+0000","last_became_active":"2022-07-21T01:33:23.027431+0000","last_became_peered":"2022-07-21T01:33:23.027431+0000","last_unstale":"2022-07-21T01:33:26.398930+0000","last_undegraded":"2022-07-21T01:33:26.398930+0000","last_fullsized":"2022-07-21T01:33:26.398930+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.104927+0000","last_change":"2022-07-21T01:33:26.102179+0000","last_active":"2022-07-21T01:33:26.104927+0000","last_peered":"2022-07-21T01:33:26.104927+0000","last_clean":"2022-07-21T01:33:26.104927+0000","last_became_active":"2022-07-21T01:33:26.100679+0000","last_became_peered":"2022-07-21T01:33:26.100679+0000","last_unstale":"2022-07-21T01:33:26.104927+0000","last_undegraded":"2022-07-21T01:33:26.104927+0000","last_fullsized":"2022-07-21T01:33:26.104927+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.042426+0000","last_change":"2022-07-21T01:33:19.896530+0000","last_active":"2022-07-21T01:33:26.042426+0000","last_peered":"2022-07-21T01:33:26.042426+0000","last_clean":"2022-07-21T01:33:26.042426+0000","last_became_active":"2022-07-21T01:33:19.896110+0000","last_became_peered":"2022-07-21T01:33:19.896110+0000","last_unstale":"2022-07-21T01:33:26.042426+0000","last_undegraded":"2022-07-21T01:33:26.042426+0000","last_fullsized":"2022-07-21T01:33:26.042426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"26'5","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.395402+0000","last_change":"2022-07-21T01:33:23.030652+0000","last_active":"2022-07-21T01:33:26.395402+0000","last_peered":"2022-07-21T01:33:26.395402+0000","last_clean":"2022-07-21T01:33:26.395402+0000","last_became_active":"2022-07-21T01:33:23.030034+0000","last_became_peered":"2022-07-21T01:33:23.030034+0000","last_unstale":"2022-07-21T01:33:26.395402+0000","last_undegraded":"2022-07-21T01:33:26.395402+0000","last_fullsized":"2022-07-21T01:33:26.395402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.061662+0000","last_change":"2022-07-21T01:33:26.058400+0000","last_active":"2022-07-21T01:33:26.061662+0000","last_peered":"2022-07-21T01:33:26.061662+0000","last_clean":"2022-07-21T01:33:26.061662+0000","last_became_active":"2022-07-21T01:33:26.057650+0000","last_became_peered":"2022-07-21T01:33:26.057650+0000","last_unstale":"2022-07-21T01:33:26.061662+0000","last_undegraded":"2022-07-21T01:33:26.061662+0000","last_fullsized":"2022-07-21T01:33:26.061662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"26'6","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.392975+0000","last_change":"2022-07-21T01:33:23.022383+0000","last_active":"2022-07-21T01:33:26.392975+0000","last_peered":"2022-07-21T01:33:26.392975+0000","last_clean":"2022-07-21T01:33:26.392975+0000","last_became_active":"2022-07-21T01:33:23.021525+0000","last_became_peered":"2022-07-21T01:33:23.021525+0000","last_unstale":"2022-07-21T01:33:26.392975+0000","last_undegraded":"2022-07-21T01:33:26.392975+0000","last_fullsized":"2022-07-21T01:33:26.392975+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.050021+0000","last_change":"2022-07-21T01:33:20.650822+0000","last_active":"2022-07-21T01:33:26.050021+0000","last_peered":"2022-07-21T01:33:26.050021+0000","last_clean":"2022-07-21T01:33:26.050021+0000","last_became_active":"2022-07-21T01:33:20.649660+0000","last_became_peered":"2022-07-21T01:33:20.649660+0000","last_unstale":"2022-07-21T01:33:26.050021+0000","last_undegraded":"2022-07-21T01:33:26.050021+0000","last_fullsized":"2022-07-21T01:33:26.050021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.107397+0000","last_change":"2022-07-21T01:33:26.103108+0000","last_active":"2022-07-21T01:33:26.107397+0000","last_peered":"2022-07-21T01:33:26.107397+0000","last_clean":"2022-07-21T01:33:26.107397+0000","last_became_active":"2022-07-21T01:33:26.101966+0000","last_became_peered":"2022-07-21T01:33:26.101966+0000","last_unstale":"2022-07-21T01:33:26.107397+0000","last_undegraded":"2022-07-21T01:33:26.107397+0000","last_fullsized":"2022-07-21T01:33:26.107397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"26'8","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.399081+0000","last_change":"2022-07-21T01:33:23.025863+0000","last_active":"2022-07-21T01:33:26.399081+0000","last_peered":"2022-07-21T01:33:26.399081+0000","last_clean":"2022-07-21T01:33:26.399081+0000","last_became_active":"2022-07-21T01:33:23.024819+0000","last_became_peered":"2022-07-21T01:33:23.024819+0000","last_unstale":"2022-07-21T01:33:26.399081+0000","last_undegraded":"2022-07-21T01:33:26.399081+0000","last_fullsized":"2022-07-21T01:33:26.399081+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.094954+0000","last_change":"2022-07-21T01:33:20.648160+0000","last_active":"2022-07-21T01:33:26.094954+0000","last_peered":"2022-07-21T01:33:26.094954+0000","last_clean":"2022-07-21T01:33:26.094954+0000","last_became_active":"2022-07-21T01:33:20.647021+0000","last_became_peered":"2022-07-21T01:33:20.647021+0000","last_unstale":"2022-07-21T01:33:26.094954+0000","last_undegraded":"2022-07-21T01:33:26.094954+0000","last_fullsized":"2022-07-21T01:33:26.094954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.105341+0000","last_change":"2022-07-21T01:33:26.102665+0000","last_active":"2022-07-21T01:33:26.105341+0000","last_peered":"2022-07-21T01:33:26.105341+0000","last_clean":"2022-07-21T01:33:26.105341+0000","last_became_active":"2022-07-21T01:33:26.101807+0000","last_became_peered":"2022-07-21T01:33:26.101807+0000","last_unstale":"2022-07-21T01:33:26.105341+0000","last_undegraded":"2022-07-21T01:33:26.105341+0000","last_fullsized":"2022-07-21T01:33:26.105341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"26'7","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.398682+0000","last_change":"2022-07-21T01:33:23.247504+0000","last_active":"2022-07-21T01:33:26.398682+0000","last_peered":"2022-07-21T01:33:26.398682+0000","last_clean":"2022-07-21T01:33:26.398682+0000","last_became_active":"2022-07-21T01:33:23.246793+0000","last_became_peered":"2022-07-21T01:33:23.246793+0000","last_unstale":"2022-07-21T01:33:26.398682+0000","last_undegraded":"2022-07-21T01:33:26.398682+0000","last_fullsized":"2022-07-21T01:33:26.398682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.050281+0000","last_change":"2022-07-21T01:33:19.900435+0000","last_active":"2022-07-21T01:33:26.050281+0000","last_peered":"2022-07-21T01:33:26.050281+0000","last_clean":"2022-07-21T01:33:26.050281+0000","last_became_active":"2022-07-21T01:33:19.900054+0000","last_became_peered":"2022-07-21T01:33:19.900054+0000","last_unstale":"2022-07-21T01:33:26.050281+0000","last_undegraded":"2022-07-21T01:33:26.050281+0000","last_fullsized":"2022-07-21T01:33:26.050281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.101474+0000","last_change":"2022-07-21T01:33:26.100220+0000","last_active":"2022-07-21T01:33:26.101474+0000","last_peered":"2022-07-21T01:33:26.101474+0000","last_clean":"2022-07-21T01:33:26.101474+0000","last_became_active":"2022-07-21T01:33:26.099164+0000","last_became_peered":"2022-07-21T01:33:26.099164+0000","last_unstale":"2022-07-21T01:33:26.101474+0000","last_undegraded":"2022-07-21T01:33:26.101474+0000","last_fullsized":"2022-07-21T01:33:26.101474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:25.032531+0000","last_clean_scrub_stamp":"2022-07-21T01:33:25.032531+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"26'4","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.399245+0000","last_change":"2022-07-21T01:33:23.028971+0000","last_active":"2022-07-21T01:33:26.399245+0000","last_peered":"2022-07-21T01:33:26.399245+0000","last_clean":"2022-07-21T01:33:26.399245+0000","last_became_active":"2022-07-21T01:33:23.028634+0000","last_became_peered":"2022-07-21T01:33:23.028634+0000","last_unstale":"2022-07-21T01:33:26.399245+0000","last_undegraded":"2022-07-21T01:33:26.399245+0000","last_fullsized":"2022-07-21T01:33:26.399245+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:22.002624+0000","last_clean_scrub_stamp":"2022-07-21T01:33:22.002624+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2022-07-21T01:33:26.047375+0000","last_change":"2022-07-21T01:33:19.895590+0000","last_active":"2022-07-21T01:33:26.047375+0000","last_peered":"2022-07-21T01:33:26.047375+0000","last_clean":"2022-07-21T01:33:26.047375+0000","last_became_active":"2022-07-21T01:33:19.895046+0000","last_became_peered":"2022-07-21T01:33:19.895046+0000","last_unstale":"2022-07-21T01:33:26.047375+0000","last_undegraded":"2022-07-21T01:33:26.047375+0000","last_fullsized":"2022-07-21T01:33:26.047375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048860,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322980,"statfs":{"total":96636764160,"available":95562731520,"internally_reserved":1073741824,"allocated":282624,"data_stored":109542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":108041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048864,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322976,"statfs":{"total":96636764160,"available":95562727424,"internally_reserved":1073741824,"allocated":286720,"data_stored":108742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":108582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":108007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":108167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":108041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048860,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322980,"statfs":{"total":96636764160,"available":95562731520,"internally_reserved":1073741824,"allocated":282624,"data_stored":109542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]}]}]}} 2022-07-21T01:33:35.526 INFO:tasks.ceph.ceph_manager.ceph:clean! 2022-07-21T01:33:35.526 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-07-21T01:33:35.527 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2022-07-21T01:33:35.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2022-07-21T01:33:35.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a17fa5700 1 Processor -- start 2022-07-21T01:33:35.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a17fa5700 1 -- start start 2022-07-21T01:33:35.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a17fa5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5a10067310 0x7f5a1005f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a17fa5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5a100b1b00 0x7f5a1005fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a17fa5700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a100a7c20 con 0x7f5a100a7840 2022-07-21T01:33:35.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a17fa5700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a101477b0 con 0x7f5a10067310 2022-07-21T01:33:35.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a17fa5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a10147a50 con 0x7f5a100b1b00 2022-07-21T01:33:35.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a15540700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5a10067310 0x7f5a1005f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a15540700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5a10067310 0x7f5a1005f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43034/0 (socket says 172.21.15.3:43034) 2022-07-21T01:33:35.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a15540700 1 -- 172.21.15.3:0/1455086198 learned_addr learned my addr 172.21.15.3:0/1455086198 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:35.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a15d41700 1 --1- 172.21.15.3:0/1455086198 >> v1:172.21.15.3:6789/0 conn(0x7f5a100a7840 0x7f5a10066c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51316/0 (socket says 172.21.15.3:51316) 2022-07-21T01:33:35.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a16542700 1 --2- 172.21.15.3:0/1455086198 >> v2:172.21.15.143:3300/0 conn(0x7f5a100b1b00 0x7f5a1005fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a15540700 1 -- 172.21.15.3:0/1455086198 >> v2:172.21.15.143:3300/0 conn(0x7f5a100b1b00 msgr2=0x7f5a1005fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:35.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a15540700 1 --2- 172.21.15.3:0/1455086198 >> v2:172.21.15.143:3300/0 conn(0x7f5a100b1b00 0x7f5a1005fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a15540700 1 -- 172.21.15.3:0/1455086198 >> v1:172.21.15.3:6789/0 conn(0x7f5a100a7840 legacy=0x7f5a10066c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:35.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a15540700 1 -- 172.21.15.3:0/1455086198 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a10147cf0 con 0x7f5a10067310 2022-07-21T01:33:35.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a15540700 1 --2- 172.21.15.3:0/1455086198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 0x7f5a1005f840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5a0c002f70 tx=0x7f5a0c006eb0).ready entity=mon.0 client_cookie=d4fdaab85fb32dcc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a14d3f700 1 -- 172.21.15.3:0/1455086198 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a0c00d610 con 0x7f5a10067310 2022-07-21T01:33:35.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a14d3f700 1 -- 172.21.15.3:0/1455086198 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c000e90 con 0x7f5a10067310 2022-07-21T01:33:35.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.634+0000 7f5a14d3f700 1 -- 172.21.15.3:0/1455086198 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a0c00d9a0 con 0x7f5a10067310 2022-07-21T01:33:35.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 -- 172.21.15.3:0/1455086198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 msgr2=0x7f5a1005f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 --2- 172.21.15.3:0/1455086198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 0x7f5a1005f840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5a0c002f70 tx=0x7f5a0c006eb0).stop 2022-07-21T01:33:35.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 -- 172.21.15.3:0/1455086198 shutdown_connections 2022-07-21T01:33:35.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 --2- 172.21.15.3:0/1455086198 >> v2:172.21.15.143:3300/0 conn(0x7f5a100b1b00 0x7f5a1005fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 --2- 172.21.15.3:0/1455086198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 0x7f5a1005f840 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 -- 172.21.15.3:0/1455086198 >> 172.21.15.3:0/1455086198 conn(0x7f5a100b5570 msgr2=0x7f5a100b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:35.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 -- 172.21.15.3:0/1455086198 shutdown_connections 2022-07-21T01:33:35.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 -- 172.21.15.3:0/1455086198 wait complete. 2022-07-21T01:33:35.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 Processor -- start 2022-07-21T01:33:35.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.638+0000 7f5a17fa5700 1 -- start start 2022-07-21T01:33:35.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a17fa5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 0x7f5a10146de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a17fa5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a100a7840 0x7f5a1015a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a17fa5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a100b1b00 0x7f5a100aa350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a17fa5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1015a0c0 con 0x7f5a10067310 2022-07-21T01:33:35.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a17fa5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a100ad030 con 0x7f5a100a7840 2022-07-21T01:33:35.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a17fa5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a100ad2d0 con 0x7f5a100b1b00 2022-07-21T01:33:35.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15d41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 0x7f5a10146de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15d41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 0x7f5a10146de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43040/0 (socket says 172.21.15.3:43040) 2022-07-21T01:33:35.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15d41700 1 -- 172.21.15.3:0/273280365 learned_addr learned my addr 172.21.15.3:0/273280365 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:35.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15540700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a100a7840 0x7f5a1015a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a16542700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a100b1b00 0x7f5a100aa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15d41700 1 -- 172.21.15.3:0/273280365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a100b1b00 msgr2=0x7f5a100aa350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15d41700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a100b1b00 0x7f5a100aa350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15d41700 1 -- 172.21.15.3:0/273280365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a100a7840 msgr2=0x7f5a1015a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15d41700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a100a7840 0x7f5a1015a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15d41700 1 -- 172.21.15.3:0/273280365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c00b1e0 con 0x7f5a10067310 2022-07-21T01:33:35.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a16542700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a100b1b00 0x7f5a100aa350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:35.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a15d41700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 0x7f5a10146de0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5a1006a270 tx=0x7f5a00007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f59feffd700 1 -- 172.21.15.3:0/273280365 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a0000ce30 con 0x7f5a10067310 2022-07-21T01:33:35.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f59feffd700 1 -- 172.21.15.3:0/273280365 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a000072e0 con 0x7f5a10067310 2022-07-21T01:33:35.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1015a2f0 con 0x7f5a10067310 2022-07-21T01:33:35.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f59feffd700 1 -- 172.21.15.3:0/273280365 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a00005740 con 0x7f5a10067310 2022-07-21T01:33:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.642+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a100ac200 con 0x7f5a10067310 2022-07-21T01:33:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.646+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a100574e0 con 0x7f5a10067310 2022-07-21T01:33:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.646+0000 7f59feffd700 1 -- 172.21.15.3:0/273280365 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5a00010070 con 0x7f5a10067310 2022-07-21T01:33:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.646+0000 7f59feffd700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59f0053b00 0x7f59f0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.646+0000 7f59feffd700 1 -- 172.21.15.3:0/273280365 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5a0006b610 con 0x7f5a10067310 2022-07-21T01:33:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.646+0000 7f5a15540700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59f0053b00 0x7f59f0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.646+0000 7f5a15540700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59f0053b00 0x7f59f0055fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5a0c01a000 tx=0x7f5a0c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.654+0000 7f59feffd700 1 -- 172.21.15.3:0/273280365 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5a000423d0 con 0x7f5a10067310 2022-07-21T01:33:35.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.822+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5a10069a70 con 0x7f5a10067310 2022-07-21T01:33:35.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f59feffd700 1 -- 172.21.15.3:0/273280365 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f5a000459f0 con 0x7f5a10067310 2022-07-21T01:33:35.823 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:35.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59f0053b00 msgr2=0x7f59f0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59f0053b00 0x7f59f0055fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5a0c01a000 tx=0x7f5a0c015000).stop 2022-07-21T01:33:35.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 msgr2=0x7f5a10146de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 0x7f5a10146de0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5a1006a270 tx=0x7f5a00007c00).stop 2022-07-21T01:33:35.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 shutdown_connections 2022-07-21T01:33:35.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a100b1b00 0x7f5a100aa350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59f0053b00 0x7f59f0055fc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a100a7840 0x7f5a1015a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 --2- 172.21.15.3:0/273280365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10067310 0x7f5a10146de0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 >> 172.21.15.3:0/273280365 conn(0x7f5a100b5570 msgr2=0x7f5a10143d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:35.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 shutdown_connections 2022-07-21T01:33:35.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.826+0000 7f5a17fa5700 1 -- 172.21.15.3:0/273280365 wait complete. 2022-07-21T01:33:35.835 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-07-21T01:33:35.836 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2022-07-21T01:33:35.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 Processor -- start 2022-07-21T01:33:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 -- start start 2022-07-21T01:33:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efc640b3260 0x7efc641435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efc640ad000 0x7efc6413fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc64143ab0 con 0x7efc640b08a0 2022-07-21T01:33:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc640b0c80 con 0x7efc640b3260 2022-07-21T01:33:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc640ae810 con 0x7efc640ad000 2022-07-21T01:33:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6902f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efc640b3260 0x7efc641435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a031700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efc640ad000 0x7efc6413fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6902f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efc640b3260 0x7efc641435c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43052/0 (socket says 172.21.15.3:43052) 2022-07-21T01:33:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6902f700 1 -- 172.21.15.3:0/411494689 learned_addr learned my addr 172.21.15.3:0/411494689 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc69830700 1 --1- 172.21.15.3:0/411494689 >> v1:172.21.15.143:6789/0 conn(0x7efc640b08a0 0x7efc64142eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:53444/0 (socket says 172.21.15.3:53444) 2022-07-21T01:33:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6902f700 1 -- 172.21.15.3:0/411494689 >> v2:172.21.15.143:3300/0 conn(0x7efc640ad000 msgr2=0x7efc6413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:35.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6882e700 1 -- 172.21.15.3:0/411494689 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562939958 0 0) 0x7efc64143ab0 con 0x7efc640b08a0 2022-07-21T01:33:35.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6902f700 1 --2- 172.21.15.3:0/411494689 >> v2:172.21.15.143:3300/0 conn(0x7efc640ad000 0x7efc6413fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6902f700 1 -- 172.21.15.3:0/411494689 >> v1:172.21.15.143:6789/0 conn(0x7efc640b08a0 legacy=0x7efc64142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6902f700 1 -- 172.21.15.3:0/411494689 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc640aeab0 con 0x7efc640b3260 2022-07-21T01:33:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6902f700 1 --2- 172.21.15.3:0/411494689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640b3260 0x7efc641435c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7efc58002f70 tx=0x7efc58006eb0).ready entity=mon.0 client_cookie=ddc81fdc842a7f38 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6882e700 1 -- 172.21.15.3:0/411494689 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc5800d5c0 con 0x7efc640b3260 2022-07-21T01:33:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6882e700 1 -- 172.21.15.3:0/411494689 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc58000e90 con 0x7efc640b3260 2022-07-21T01:33:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6882e700 1 -- 172.21.15.3:0/411494689 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc5800d910 con 0x7efc640b3260 2022-07-21T01:33:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 -- 172.21.15.3:0/411494689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640b3260 msgr2=0x7efc641435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 --2- 172.21.15.3:0/411494689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640b3260 0x7efc641435c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7efc58002f70 tx=0x7efc58006eb0).stop 2022-07-21T01:33:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 -- 172.21.15.3:0/411494689 shutdown_connections 2022-07-21T01:33:35.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 --2- 172.21.15.3:0/411494689 >> v2:172.21.15.143:3300/0 conn(0x7efc640ad000 0x7efc6413fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 --2- 172.21.15.3:0/411494689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640b3260 0x7efc641435c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.942+0000 7efc6a832700 1 -- 172.21.15.3:0/411494689 >> 172.21.15.3:0/411494689 conn(0x7efc640612d0 msgr2=0x7efc64062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:35.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 -- 172.21.15.3:0/411494689 shutdown_connections 2022-07-21T01:33:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 -- 172.21.15.3:0/411494689 wait complete. 2022-07-21T01:33:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 Processor -- start 2022-07-21T01:33:35.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 -- start start 2022-07-21T01:33:35.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640ad000 0x7efc64136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc640b08a0 0x7efc640ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc640b3260 0x7efc640abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc64065d00 con 0x7efc640ad000 2022-07-21T01:33:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc641389b0 con 0x7efc640b3260 2022-07-21T01:33:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc640667c0 con 0x7efc640b08a0 2022-07-21T01:33:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc69830700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640ad000 0x7efc64136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a031700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc640b3260 0x7efc640abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a031700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc640b3260 0x7efc640abed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:48502/0 (socket says 172.21.15.3:48502) 2022-07-21T01:33:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a031700 1 -- 172.21.15.3:0/564663863 learned_addr learned my addr 172.21.15.3:0/564663863 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc69830700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640ad000 0x7efc64136de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43058/0 (socket says 172.21.15.3:43058) 2022-07-21T01:33:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6902f700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc640b08a0 0x7efc640ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a031700 1 -- 172.21.15.3:0/564663863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc640b08a0 msgr2=0x7efc640ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a031700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc640b08a0 0x7efc640ab9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a031700 1 -- 172.21.15.3:0/564663863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640ad000 msgr2=0x7efc64136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:35.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a031700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640ad000 0x7efc64136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:35.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a031700 1 -- 172.21.15.3:0/564663863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5800b1e0 con 0x7efc640b3260 2022-07-21T01:33:35.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a031700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc640b3260 0x7efc640abed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7efc5400c000 tx=0x7efc54006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc62ffd700 1 -- 172.21.15.3:0/564663863 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc54011750 con 0x7efc640b3260 2022-07-21T01:33:35.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc62ffd700 1 -- 172.21.15.3:0/564663863 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc540118b0 con 0x7efc640b3260 2022-07-21T01:33:35.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc64066a60 con 0x7efc640b3260 2022-07-21T01:33:35.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc62ffd700 1 -- 172.21.15.3:0/564663863 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc54011c10 con 0x7efc640b3260 2022-07-21T01:33:35.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.946+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc64065860 con 0x7efc640b3260 2022-07-21T01:33:35.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.950+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc640ac560 con 0x7efc640b3260 2022-07-21T01:33:35.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.950+0000 7efc62ffd700 1 -- 172.21.15.3:0/564663863 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efc54011d70 con 0x7efc640b3260 2022-07-21T01:33:35.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.950+0000 7efc62ffd700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc44053950 0x7efc44055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:35.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.950+0000 7efc62ffd700 1 -- 172.21.15.3:0/564663863 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efc5406c4b0 con 0x7efc640b3260 2022-07-21T01:33:35.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.950+0000 7efc69830700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc44053950 0x7efc44055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:35.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.950+0000 7efc69830700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc44053950 0x7efc44055e10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7efc5c00d000 tx=0x7efc5c007d80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:35.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:35.958+0000 7efc62ffd700 1 -- 172.21.15.3:0/564663863 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc640ac560 con 0x7efc640b3260 2022-07-21T01:33:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.090+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc640ac560 con 0x7efc640b3260 2022-07-21T01:33:36.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.090+0000 7efc62ffd700 1 -- 172.21.15.3:0/564663863 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7efc640ac560 con 0x7efc640b3260 2022-07-21T01:33:36.088 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:36.088 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:36.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc44053950 msgr2=0x7efc44055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc44053950 0x7efc44055e10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7efc5c00d000 tx=0x7efc5c007d80).stop 2022-07-21T01:33:36.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc640b3260 msgr2=0x7efc640abed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc640b3260 0x7efc640abed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7efc5400c000 tx=0x7efc54006df0).stop 2022-07-21T01:33:36.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 shutdown_connections 2022-07-21T01:33:36.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc44053950 0x7efc44055e10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc640b3260 0x7efc640abed0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc640b08a0 0x7efc640ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 --2- 172.21.15.3:0/564663863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc640ad000 0x7efc64136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 >> 172.21.15.3:0/564663863 conn(0x7efc640612d0 msgr2=0x7efc640b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 shutdown_connections 2022-07-21T01:33:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.094+0000 7efc6a832700 1 -- 172.21.15.3:0/564663863 wait complete. 2022-07-21T01:33:36.102 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:33:21.299936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:33:24.075599+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:33:36.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:33:36.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.218+0000 7f5c5e97b700 1 Processor -- start 2022-07-21T01:33:36.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.218+0000 7f5c5e97b700 1 -- start start 2022-07-21T01:33:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.218+0000 7f5c5e97b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5c580aebd0 0x7f5c58143940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5c580ad190 0x7f5c58143e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c580b0b20 con 0x7f5c580b0610 2022-07-21T01:33:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c580adde0 con 0x7f5c580aebd0 2022-07-21T01:33:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c580aefb0 con 0x7f5c580ad190 2022-07-21T01:33:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e17a700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5c580b0610 0x7f5c58146cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51346/0 (socket says 172.21.15.3:51346) 2022-07-21T01:33:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5d979700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5c580aebd0 0x7f5c58143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e17a700 1 -- 172.21.15.3:0/4033710006 learned_addr learned my addr 172.21.15.3:0/4033710006 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5d178700 1 --2- 172.21.15.3:0/4033710006 >> v2:172.21.15.143:3300/0 conn(0x7f5c580ad190 0x7f5c58143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5c977700 1 -- 172.21.15.3:0/4033710006 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661697553 0 0) 0x7f5c580b0b20 con 0x7f5c580b0610 2022-07-21T01:33:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5c977700 1 -- 172.21.15.3:0/4033710006 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c40003260 con 0x7f5c580b0610 2022-07-21T01:33:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5d979700 1 -- 172.21.15.3:0/4033710006 >> v2:172.21.15.143:3300/0 conn(0x7f5c580ad190 msgr2=0x7f5c58143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5d979700 1 --2- 172.21.15.3:0/4033710006 >> v2:172.21.15.143:3300/0 conn(0x7f5c580ad190 0x7f5c58143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5d979700 1 -- 172.21.15.3:0/4033710006 >> v1:172.21.15.3:6789/0 conn(0x7f5c580b0610 legacy=0x7f5c58146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5d979700 1 -- 172.21.15.3:0/4033710006 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c58146890 con 0x7f5c580aebd0 2022-07-21T01:33:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5d979700 1 --2- 172.21.15.3:0/4033710006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580aebd0 0x7f5c58143940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f5c54002f70 tx=0x7f5c54006df0).ready entity=mon.0 client_cookie=6640b320aa5f1a06 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5c977700 1 -- 172.21.15.3:0/4033710006 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c5400d4f0 con 0x7f5c580aebd0 2022-07-21T01:33:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5c977700 1 -- 172.21.15.3:0/4033710006 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54000e90 con 0x7f5c580aebd0 2022-07-21T01:33:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5c977700 1 -- 172.21.15.3:0/4033710006 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c5400d840 con 0x7f5c580aebd0 2022-07-21T01:33:36.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 -- 172.21.15.3:0/4033710006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580aebd0 msgr2=0x7f5c58143940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 --2- 172.21.15.3:0/4033710006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580aebd0 0x7f5c58143940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f5c54002f70 tx=0x7f5c54006df0).stop 2022-07-21T01:33:36.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 -- 172.21.15.3:0/4033710006 shutdown_connections 2022-07-21T01:33:36.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 --2- 172.21.15.3:0/4033710006 >> v2:172.21.15.143:3300/0 conn(0x7f5c580ad190 0x7f5c58143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 --2- 172.21.15.3:0/4033710006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580aebd0 0x7f5c58143940 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 -- 172.21.15.3:0/4033710006 >> 172.21.15.3:0/4033710006 conn(0x7f5c580613c0 msgr2=0x7f5c58062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 -- 172.21.15.3:0/4033710006 shutdown_connections 2022-07-21T01:33:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.222+0000 7f5c5e97b700 1 -- 172.21.15.3:0/4033710006 wait complete. 2022-07-21T01:33:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e97b700 1 Processor -- start 2022-07-21T01:33:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e97b700 1 -- start start 2022-07-21T01:33:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e97b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c580ad190 0x7f5c5813d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e97b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c580aebd0 0x7f5c580b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e97b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580b0610 0x7f5c580b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e97b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5813c1d0 con 0x7f5c580b0610 2022-07-21T01:33:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e97b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c580b4930 con 0x7f5c580ad190 2022-07-21T01:33:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e97b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c580b4e30 con 0x7f5c580aebd0 2022-07-21T01:33:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e17a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580b0610 0x7f5c580b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e17a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580b0610 0x7f5c580b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43072/0 (socket says 172.21.15.3:43072) 2022-07-21T01:33:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5d979700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c580ad190 0x7f5c5813d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e17a700 1 -- 172.21.15.3:0/935706211 learned_addr learned my addr 172.21.15.3:0/935706211 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:36.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5d178700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c580aebd0 0x7f5c580b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e17a700 1 -- 172.21.15.3:0/935706211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c580aebd0 msgr2=0x7f5c580b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e17a700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c580aebd0 0x7f5c580b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e17a700 1 -- 172.21.15.3:0/935706211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c580ad190 msgr2=0x7f5c5813d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e17a700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c580ad190 0x7f5c5813d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e17a700 1 -- 172.21.15.3:0/935706211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5400b180 con 0x7f5c580b0610 2022-07-21T01:33:36.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c5e17a700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580b0610 0x7f5c580b4440 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5c58143620 tx=0x7f5c50007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.226+0000 7f5c4affd700 1 -- 172.21.15.3:0/935706211 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c50005ba0 con 0x7f5c580b0610 2022-07-21T01:33:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c4affd700 1 -- 172.21.15.3:0/935706211 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c50005d00 con 0x7f5c580b0610 2022-07-21T01:33:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c580b50d0 con 0x7f5c580b0610 2022-07-21T01:33:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c4affd700 1 -- 172.21.15.3:0/935706211 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c500072c0 con 0x7f5c580b0610 2022-07-21T01:33:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c580b37f0 con 0x7f5c580b0610 2022-07-21T01:33:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c581412c0 con 0x7f5c580b0610 2022-07-21T01:33:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c4affd700 1 -- 172.21.15.3:0/935706211 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5c5001a060 con 0x7f5c580b0610 2022-07-21T01:33:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c4affd700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c380539e0 0x7f5c38055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c4affd700 1 -- 172.21.15.3:0/935706211 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5c5006bbb0 con 0x7f5c580b0610 2022-07-21T01:33:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c5d979700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c380539e0 0x7f5c38055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.230+0000 7f5c5d979700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c380539e0 0x7f5c38055ea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5c5401a000 tx=0x7f5c54014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.238+0000 7f5c4affd700 1 -- 172.21.15.3:0/935706211 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5c50042860 con 0x7f5c580b0610 2022-07-21T01:33:36.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5c580b0d50 con 0x7f5c580b0610 2022-07-21T01:33:36.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c4affd700 1 -- 172.21.15.3:0/935706211 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v26) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5c50018030 con 0x7f5c580b0610 2022-07-21T01:33:36.363 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:33:36.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c380539e0 msgr2=0x7f5c38055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c380539e0 0x7f5c38055ea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5c5401a000 tx=0x7f5c54014000).stop 2022-07-21T01:33:36.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580b0610 msgr2=0x7f5c580b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580b0610 0x7f5c580b4440 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5c58143620 tx=0x7f5c50007dc0).stop 2022-07-21T01:33:36.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 shutdown_connections 2022-07-21T01:33:36.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c580b0610 0x7f5c580b4440 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c580aebd0 0x7f5c580b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c380539e0 0x7f5c38055ea0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 --2- 172.21.15.3:0/935706211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c580ad190 0x7f5c5813d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 >> 172.21.15.3:0/935706211 conn(0x7f5c580613c0 msgr2=0x7f5c580b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 shutdown_connections 2022-07-21T01:33:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.366+0000 7f5c5e97b700 1 -- 172.21.15.3:0/935706211 wait complete. 2022-07-21T01:33:36.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:33:36.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.490+0000 7f60420f8700 1 Processor -- start 2022-07-21T01:33:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.490+0000 7f60420f8700 1 -- start start 2022-07-21T01:33:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f603c066a50 0x7f603c0b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c0b0e60 0x7f603c0b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f603c05fe20 0x7f603c0b6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c0b1700 con 0x7f603c05fe20 2022-07-21T01:33:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c066e30 con 0x7f603c066a50 2022-07-21T01:33:36.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0b1500 con 0x7f603c0b0e60 2022-07-21T01:33:36.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60418f7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f603c05fe20 0x7f603c0b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60418f7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f603c05fe20 0x7f603c0b6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43080/0 (socket says 172.21.15.3:43080) 2022-07-21T01:33:36.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60418f7700 1 -- 172.21.15.3:0/1604486926 learned_addr learned my addr 172.21.15.3:0/1604486926 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:36.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60410f6700 1 --2- 172.21.15.3:0/1604486926 >> v2:172.21.15.143:3300/0 conn(0x7f603c066a50 0x7f603c0b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60408f5700 1 --2- 172.21.15.3:0/1604486926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c0b0e60 0x7f603c0b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60418f7700 1 -- 172.21.15.3:0/1604486926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c0b0e60 msgr2=0x7f603c0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:36.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60418f7700 1 --2- 172.21.15.3:0/1604486926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c0b0e60 0x7f603c0b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60418f7700 1 -- 172.21.15.3:0/1604486926 >> v2:172.21.15.143:3300/0 conn(0x7f603c066a50 msgr2=0x7f603c0b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:36.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60418f7700 1 --2- 172.21.15.3:0/1604486926 >> v2:172.21.15.143:3300/0 conn(0x7f603c066a50 0x7f603c0b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60418f7700 1 -- 172.21.15.3:0/1604486926 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c05f9c0 con 0x7f603c05fe20 2022-07-21T01:33:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60410f6700 1 --2- 172.21.15.3:0/1604486926 >> v2:172.21.15.143:3300/0 conn(0x7f603c066a50 0x7f603c0b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60418f7700 1 --2- 172.21.15.3:0/1604486926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c05fe20 0x7f603c0b6370 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f60340032f0 tx=0x7f6034009290).ready entity=mon.0 client_cookie=5a90dc6e4eada6f0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f602bfff700 1 -- 172.21.15.3:0/1604486926 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f603400b870 con 0x7f603c05fe20 2022-07-21T01:33:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f602bfff700 1 -- 172.21.15.3:0/1604486926 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603400b9d0 con 0x7f603c05fe20 2022-07-21T01:33:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f602bfff700 1 -- 172.21.15.3:0/1604486926 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f603400bb80 con 0x7f603c05fe20 2022-07-21T01:33:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- 172.21.15.3:0/1604486926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c05fe20 msgr2=0x7f603c0b6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- 172.21.15.3:0/1604486926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c05fe20 0x7f603c0b6370 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f60340032f0 tx=0x7f6034009290).stop 2022-07-21T01:33:36.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- 172.21.15.3:0/1604486926 shutdown_connections 2022-07-21T01:33:36.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- 172.21.15.3:0/1604486926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c05fe20 0x7f603c0b6370 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- 172.21.15.3:0/1604486926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c0b0e60 0x7f603c0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- 172.21.15.3:0/1604486926 >> v2:172.21.15.143:3300/0 conn(0x7f603c066a50 0x7f603c0b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- 172.21.15.3:0/1604486926 >> 172.21.15.3:0/1604486926 conn(0x7f603c063800 msgr2=0x7f603c063be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:36.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- 172.21.15.3:0/1604486926 shutdown_connections 2022-07-21T01:33:36.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- 172.21.15.3:0/1604486926 wait complete. 2022-07-21T01:33:36.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 Processor -- start 2022-07-21T01:33:36.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- start start 2022-07-21T01:33:36.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f603c05fe20 0x7f603c0b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c066a50 0x7f603c064880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c0b0e60 0x7f603c064d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c152350 con 0x7f603c0b0e60 2022-07-21T01:33:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c153310 con 0x7f603c05fe20 2022-07-21T01:33:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.494+0000 7f60420f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0659f0 con 0x7f603c066a50 2022-07-21T01:33:36.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60418f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c0b0e60 0x7f603c064d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60418f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c0b0e60 0x7f603c064d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43086/0 (socket says 172.21.15.3:43086) 2022-07-21T01:33:36.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60418f7700 1 -- 172.21.15.3:0/213618636 learned_addr learned my addr 172.21.15.3:0/213618636 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:36.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60410f6700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f603c05fe20 0x7f603c0b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60408f5700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c066a50 0x7f603c064880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60418f7700 1 -- 172.21.15.3:0/213618636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c066a50 msgr2=0x7f603c064880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60418f7700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c066a50 0x7f603c064880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60418f7700 1 -- 172.21.15.3:0/213618636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f603c05fe20 msgr2=0x7f603c0b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60418f7700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f603c05fe20 0x7f603c0b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60418f7700 1 -- 172.21.15.3:0/213618636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6034009530 con 0x7f603c0b0e60 2022-07-21T01:33:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60408f5700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c066a50 0x7f603c064880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60418f7700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c0b0e60 0x7f603c064d70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f60340032f0 tx=0x7f6034016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f602a7fc700 1 -- 172.21.15.3:0/213618636 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f603400b870 con 0x7f603c0b0e60 2022-07-21T01:33:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f602a7fc700 1 -- 172.21.15.3:0/213618636 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034020470 con 0x7f603c0b0e60 2022-07-21T01:33:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c0af460 con 0x7f603c0b0e60 2022-07-21T01:33:36.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f602a7fc700 1 -- 172.21.15.3:0/213618636 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60340207d0 con 0x7f603c0b0e60 2022-07-21T01:33:36.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c0aec60 con 0x7f603c0b0e60 2022-07-21T01:33:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c0b1930 con 0x7f603c0b0e60 2022-07-21T01:33:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f602a7fc700 1 -- 172.21.15.3:0/213618636 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6034029070 con 0x7f603c0b0e60 2022-07-21T01:33:36.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f602a7fc700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f601c053950 0x7f601c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f602a7fc700 1 -- 172.21.15.3:0/213618636 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f603406cca0 con 0x7f603c0b0e60 2022-07-21T01:33:36.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.498+0000 7f60410f6700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f601c053950 0x7f601c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.502+0000 7f60410f6700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f601c053950 0x7f601c055e10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f603800c000 tx=0x7f6038008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:36.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.506+0000 7f602a7fc700 1 -- 172.21.15.3:0/213618636 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6034071070 con 0x7f603c0b0e60 2022-07-21T01:33:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.634+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f603c0574e0 con 0x7f603c0b0e60 2022-07-21T01:33:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.634+0000 7f602a7fc700 1 -- 172.21.15.3:0/213618636 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6034015070 con 0x7f603c0b0e60 2022-07-21T01:33:36.632 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:36.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f601c053950 msgr2=0x7f601c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f601c053950 0x7f601c055e10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f603800c000 tx=0x7f6038008040).stop 2022-07-21T01:33:36.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c0b0e60 msgr2=0x7f603c064d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c0b0e60 0x7f603c064d70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f60340032f0 tx=0x7f6034016000).stop 2022-07-21T01:33:36.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 shutdown_connections 2022-07-21T01:33:36.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f603c0b0e60 0x7f603c064d70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f603c066a50 0x7f603c064880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f601c053950 0x7f601c055e10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 --2- 172.21.15.3:0/213618636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f603c05fe20 0x7f603c0b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 >> 172.21.15.3:0/213618636 conn(0x7f603c063800 msgr2=0x7f603c0b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:36.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 shutdown_connections 2022-07-21T01:33:36.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.638+0000 7f60420f8700 1 -- 172.21.15.3:0/213618636 wait complete. 2022-07-21T01:33:36.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:33:36.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f07366aa700 1 Processor -- start 2022-07-21T01:33:36.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f07366aa700 1 -- start start 2022-07-21T01:33:36.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f07366aa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f07300b2340 0x7f073005f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f07366aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730143e20 0x7f073012c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f07366aa700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07300b3dd0 con 0x7f0730060eb0 2022-07-21T01:33:36.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f07366aa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0730063f80 con 0x7f07300b2340 2022-07-21T01:33:36.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f07366aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07300b2720 con 0x7f0730143e20 2022-07-21T01:33:36.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f0734ea7700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0730060eb0 0x7f07300b36c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51372/0 (socket says 172.21.15.3:51372) 2022-07-21T01:33:36.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f0734ea7700 1 -- 172.21.15.3:0/2061791876 learned_addr learned my addr 172.21.15.3:0/2061791876 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:36.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f0735ea9700 1 --2- 172.21.15.3:0/2061791876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730143e20 0x7f073012c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.770+0000 7f07356a8700 1 --2- 172.21.15.3:0/2061791876 >> v2:172.21.15.143:3300/0 conn(0x7f07300b2340 0x7f073005f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f071ffff700 1 -- 172.21.15.3:0/2061791876 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201757798 0 0) 0x7f07300b3dd0 con 0x7f0730060eb0 2022-07-21T01:33:36.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f071ffff700 1 -- 172.21.15.3:0/2061791876 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0718003260 con 0x7f0730060eb0 2022-07-21T01:33:36.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f0735ea9700 1 -- 172.21.15.3:0/2061791876 >> v2:172.21.15.143:3300/0 conn(0x7f07300b2340 msgr2=0x7f073005f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:36.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f0735ea9700 1 --2- 172.21.15.3:0/2061791876 >> v2:172.21.15.143:3300/0 conn(0x7f07300b2340 0x7f073005f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f0735ea9700 1 -- 172.21.15.3:0/2061791876 >> v1:172.21.15.3:6789/0 conn(0x7f0730060eb0 legacy=0x7f07300b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f0735ea9700 1 -- 172.21.15.3:0/2061791876 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07300b2fc0 con 0x7f0730143e20 2022-07-21T01:33:36.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f0735ea9700 1 --2- 172.21.15.3:0/2061791876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730143e20 0x7f073012c1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f07280032f0 tx=0x7f07280091d0).ready entity=mon.2 client_cookie=8c16659ef47c1833 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:36.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f071ffff700 1 -- 172.21.15.3:0/2061791876 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f072800b600 con 0x7f0730143e20 2022-07-21T01:33:36.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f071ffff700 1 -- 172.21.15.3:0/2061791876 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072800b760 con 0x7f0730143e20 2022-07-21T01:33:36.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f071ffff700 1 -- 172.21.15.3:0/2061791876 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f072800b8d0 con 0x7f0730143e20 2022-07-21T01:33:36.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 -- 172.21.15.3:0/2061791876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730143e20 msgr2=0x7f073012c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 --2- 172.21.15.3:0/2061791876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730143e20 0x7f073012c1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f07280032f0 tx=0x7f07280091d0).stop 2022-07-21T01:33:36.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 -- 172.21.15.3:0/2061791876 shutdown_connections 2022-07-21T01:33:36.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 --2- 172.21.15.3:0/2061791876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730143e20 0x7f073012c1b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 --2- 172.21.15.3:0/2061791876 >> v2:172.21.15.143:3300/0 conn(0x7f07300b2340 0x7f073005f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 -- 172.21.15.3:0/2061791876 >> 172.21.15.3:0/2061791876 conn(0x7f073005c620 msgr2=0x7f073005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:36.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 -- 172.21.15.3:0/2061791876 shutdown_connections 2022-07-21T01:33:36.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 -- 172.21.15.3:0/2061791876 wait complete. 2022-07-21T01:33:36.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 Processor -- start 2022-07-21T01:33:36.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 -- start start 2022-07-21T01:33:36.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730060eb0 0x7f073012abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07300b2340 0x7f0730128170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0730143e20 0x7f0730128660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0730145fe0 con 0x7f07300b2340 2022-07-21T01:33:36.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0730146810 con 0x7f0730143e20 2022-07-21T01:33:36.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f07366aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0730128b50 con 0x7f0730060eb0 2022-07-21T01:33:36.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f0734ea7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07300b2340 0x7f0730128170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f0734ea7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07300b2340 0x7f0730128170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43098/0 (socket says 172.21.15.3:43098) 2022-07-21T01:33:36.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.774+0000 7f0734ea7700 1 -- 172.21.15.3:0/1019721325 learned_addr learned my addr 172.21.15.3:0/1019721325 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:36.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f0735ea9700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0730143e20 0x7f0730128660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f07356a8700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730060eb0 0x7f073012abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f0734ea7700 1 -- 172.21.15.3:0/1019721325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730060eb0 msgr2=0x7f073012abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f0734ea7700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730060eb0 0x7f073012abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f0734ea7700 1 -- 172.21.15.3:0/1019721325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0730143e20 msgr2=0x7f0730128660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f0734ea7700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0730143e20 0x7f0730128660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f0734ea7700 1 -- 172.21.15.3:0/1019721325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0728009470 con 0x7f07300b2340 2022-07-21T01:33:36.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f0734ea7700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07300b2340 0x7f0730128170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f07301503e0 tx=0x7f0720007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:36.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f071e7fc700 1 -- 172.21.15.3:0/1019721325 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f072000ce10 con 0x7f07300b2340 2022-07-21T01:33:36.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f071e7fc700 1 -- 172.21.15.3:0/1019721325 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07200072e0 con 0x7f07300b2340 2022-07-21T01:33:36.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07301452d0 con 0x7f07300b2340 2022-07-21T01:33:36.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f071e7fc700 1 -- 172.21.15.3:0/1019721325 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0720005800 con 0x7f07300b2340 2022-07-21T01:33:36.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0730146d10 con 0x7f07300b2340 2022-07-21T01:33:36.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f071e7fc700 1 -- 172.21.15.3:0/1019721325 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f072001a060 con 0x7f07300b2340 2022-07-21T01:33:36.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0730146ed0 con 0x7f07300b2340 2022-07-21T01:33:36.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f071e7fc700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0710053a70 0x7f0710055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:36.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f071e7fc700 1 -- 172.21.15.3:0/1019721325 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f072006b980 con 0x7f07300b2340 2022-07-21T01:33:36.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.778+0000 7f07356a8700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0710053a70 0x7f0710055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:36.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.782+0000 7f07356a8700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0710053a70 0x7f0710055f30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f072c00b000 tx=0x7f072c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:36.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.786+0000 7f071e7fc700 1 -- 172.21.15.3:0/1019721325 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0720042630 con 0x7f07300b2340 2022-07-21T01:33:36.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.914+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0730063a40 con 0x7f07300b2340 2022-07-21T01:33:36.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.914+0000 7f071e7fc700 1 -- 172.21.15.3:0/1019721325 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+10 (secure 0 0 0) 0x7f07200180a0 con 0x7f07300b2340 2022-07-21T01:33:36.912 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:36.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0710053a70 msgr2=0x7f0710055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0710053a70 0x7f0710055f30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f072c00b000 tx=0x7f072c006df0).stop 2022-07-21T01:33:36.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07300b2340 msgr2=0x7f0730128170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:36.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07300b2340 0x7f0730128170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f07301503e0 tx=0x7f0720007ad0).stop 2022-07-21T01:33:36.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 shutdown_connections 2022-07-21T01:33:36.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0710053a70 0x7f0710055f30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0730143e20 0x7f0730128660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07300b2340 0x7f0730128170 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 --2- 172.21.15.3:0/1019721325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0730060eb0 0x7f073012abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:36.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 >> 172.21.15.3:0/1019721325 conn(0x7f073005c620 msgr2=0x7f073012b650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:36.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 shutdown_connections 2022-07-21T01:33:36.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:36.918+0000 7f07366aa700 1 -- 172.21.15.3:0/1019721325 wait complete. 2022-07-21T01:33:36.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:33:37.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeaeb1b700 1 Processor -- start 2022-07-21T01:33:37.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeaeb1b700 1 -- start start 2022-07-21T01:33:37.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeaeb1b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 0x7fdea814ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeaeb1b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea8067030 con 0x7fdea80a9da0 2022-07-21T01:33:37.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeaeb1b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea80b35b0 con 0x7fdea80b31d0 2022-07-21T01:33:37.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeaeb1b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea8064a70 con 0x7fdea81489f0 2022-07-21T01:33:37.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdead318700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdea80a9da0 0x7fdea8060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51386/0 (socket says 172.21.15.3:51386) 2022-07-21T01:33:37.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdead318700 1 -- 172.21.15.3:0/3556544134 learned_addr learned my addr 172.21.15.3:0/3556544134 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:37.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeae31a700 1 --2- 172.21.15.3:0/3556544134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 0x7fdea814ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeacb17700 1 -- 172.21.15.3:0/3556544134 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806684597 0 0) 0x7fdea8067030 con 0x7fdea80a9da0 2022-07-21T01:33:37.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeacb17700 1 -- 172.21.15.3:0/3556544134 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde90003260 con 0x7fdea80a9da0 2022-07-21T01:33:37.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeacb17700 1 -- 172.21.15.3:0/3556544134 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714202 0 0) 0x7fdea80b35b0 con 0x7fdea80b31d0 2022-07-21T01:33:37.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeacb17700 1 -- 172.21.15.3:0/3556544134 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea8067030 con 0x7fdea80b31d0 2022-07-21T01:33:37.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeae31a700 1 -- 172.21.15.3:0/3556544134 >> v1:172.21.15.143:6789/0 conn(0x7fdea80b31d0 legacy=0x7fdea8066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeae31a700 1 -- 172.21.15.3:0/3556544134 >> v1:172.21.15.3:6789/0 conn(0x7fdea80a9da0 legacy=0x7fdea8060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeacb17700 1 -- 172.21.15.3:0/3556544134 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1132543754 0 0) 0x7fde90003260 con 0x7fdea80a9da0 2022-07-21T01:33:37.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeae31a700 1 -- 172.21.15.3:0/3556544134 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea8066af0 con 0x7fdea81489f0 2022-07-21T01:33:37.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeae31a700 1 --2- 172.21.15.3:0/3556544134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 0x7fdea814ade0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fdea00032f0 tx=0x7fdea00091d0).ready entity=mon.2 client_cookie=a1ab171d484b3867 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:37.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeacb17700 1 -- 172.21.15.3:0/3556544134 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdea000b640 con 0x7fdea81489f0 2022-07-21T01:33:37.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.038+0000 7fdeacb17700 1 -- 172.21.15.3:0/3556544134 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea000b7a0 con 0x7fdea81489f0 2022-07-21T01:33:37.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeacb17700 1 -- 172.21.15.3:0/3556544134 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdea000b910 con 0x7fdea81489f0 2022-07-21T01:33:37.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/3556544134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 msgr2=0x7fdea814ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 --2- 172.21.15.3:0/3556544134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 0x7fdea814ade0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fdea00032f0 tx=0x7fdea00091d0).stop 2022-07-21T01:33:37.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/3556544134 shutdown_connections 2022-07-21T01:33:37.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 --2- 172.21.15.3:0/3556544134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 0x7fdea814ade0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/3556544134 >> 172.21.15.3:0/3556544134 conn(0x7fdea8063870 msgr2=0x7fdea8063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:37.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/3556544134 shutdown_connections 2022-07-21T01:33:37.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/3556544134 wait complete. 2022-07-21T01:33:37.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 Processor -- start 2022-07-21T01:33:37.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 -- start start 2022-07-21T01:33:37.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdea80a9da0 0x7fdea813bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdea80b31d0 0x7fdea81557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 0x7fdea8155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea80aed00 con 0x7fdea80b31d0 2022-07-21T01:33:37.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea80ae080 con 0x7fdea80a9da0 2022-07-21T01:33:37.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.042+0000 7fdeaeb1b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea80b02d0 con 0x7fdea81489f0 2022-07-21T01:33:37.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdead318700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdea80b31d0 0x7fdea81557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdeadb19700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdea80a9da0 0x7fdea813bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdead318700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdea80b31d0 0x7fdea81557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43112/0 (socket says 172.21.15.3:43112) 2022-07-21T01:33:37.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdead318700 1 -- 172.21.15.3:0/593274828 learned_addr learned my addr 172.21.15.3:0/593274828 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:37.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdeae31a700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 0x7fdea8155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdead318700 1 -- 172.21.15.3:0/593274828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 msgr2=0x7fdea8155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdead318700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 0x7fdea8155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdead318700 1 -- 172.21.15.3:0/593274828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdea80a9da0 msgr2=0x7fdea813bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdead318700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdea80a9da0 0x7fdea813bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdead318700 1 -- 172.21.15.3:0/593274828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea0009470 con 0x7fdea80b31d0 2022-07-21T01:33:37.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdead318700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdea80b31d0 0x7fdea81557f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fdea80aeaf0 tx=0x7fde98001800).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:37.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fde96ffd700 1 -- 172.21.15.3:0/593274828 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde98007170 con 0x7fdea80b31d0 2022-07-21T01:33:37.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea80ad140 con 0x7fdea80b31d0 2022-07-21T01:33:37.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fde96ffd700 1 -- 172.21.15.3:0/593274828 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde980072d0 con 0x7fdea80b31d0 2022-07-21T01:33:37.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fde96ffd700 1 -- 172.21.15.3:0/593274828 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde98004710 con 0x7fdea80b31d0 2022-07-21T01:33:37.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea80adbe0 con 0x7fdea80b31d0 2022-07-21T01:33:37.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.046+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea80adda0 con 0x7fdea80b31d0 2022-07-21T01:33:37.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.050+0000 7fde96ffd700 1 -- 172.21.15.3:0/593274828 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fde98004870 con 0x7fdea80b31d0 2022-07-21T01:33:37.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.050+0000 7fde96ffd700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde88053a70 0x7fde88055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.050+0000 7fde96ffd700 1 -- 172.21.15.3:0/593274828 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fde9806ba20 con 0x7fdea80b31d0 2022-07-21T01:33:37.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.050+0000 7fdeadb19700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde88053a70 0x7fde88055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.050+0000 7fdeadb19700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde88053a70 0x7fde88055f30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdea400d000 tx=0x7fdea4007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:37.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.058+0000 7fde96ffd700 1 -- 172.21.15.3:0/593274828 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fde98042650 con 0x7fdea80b31d0 2022-07-21T01:33:37.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.182+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdea8066d90 con 0x7fdea80b31d0 2022-07-21T01:33:37.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fde96ffd700 1 -- 172.21.15.3:0/593274828 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+10 (secure 0 0 0) 0x7fde98045c70 con 0x7fdea80b31d0 2022-07-21T01:33:37.182 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:37.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde88053a70 msgr2=0x7fde88055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde88053a70 0x7fde88055f30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdea400d000 tx=0x7fdea4007a20).stop 2022-07-21T01:33:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdea80b31d0 msgr2=0x7fdea81557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdea80b31d0 0x7fdea81557f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fdea80aeaf0 tx=0x7fde98001800).stop 2022-07-21T01:33:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 shutdown_connections 2022-07-21T01:33:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdea81489f0 0x7fdea8155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdea80b31d0 0x7fdea81557f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde88053a70 0x7fde88055f30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 --2- 172.21.15.3:0/593274828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdea80a9da0 0x7fdea813bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 >> 172.21.15.3:0/593274828 conn(0x7fdea8063870 msgr2=0x7fdea80b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:37.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 shutdown_connections 2022-07-21T01:33:37.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.186+0000 7fdeaeb1b700 1 -- 172.21.15.3:0/593274828 wait complete. 2022-07-21T01:33:37.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:33:37.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e50956700 1 Processor -- start 2022-07-21T01:33:37.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e50956700 1 -- start start 2022-07-21T01:33:37.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e50956700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 0x7f2e4c0a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e50956700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4c0a7250 con 0x7f2e4c0afd60 2022-07-21T01:33:37.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e50956700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4c0a7480 con 0x7f2e4c0b2710 2022-07-21T01:33:37.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e50956700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c0a7720 con 0x7f2e4c0b01a0 2022-07-21T01:33:37.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2e4c0afd60 0x7f2e4c151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51402/0 (socket says 172.21.15.3:51402) 2022-07-21T01:33:37.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4bfff700 1 -- 172.21.15.3:0/858036789 learned_addr learned my addr 172.21.15.3:0/858036789 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:37.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4affd700 1 --2- 172.21.15.3:0/858036789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 0x7f2e4c0a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4a7fc700 1 -- 172.21.15.3:0/858036789 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217634540 0 0) 0x7f2e4c0a7250 con 0x7f2e4c0afd60 2022-07-21T01:33:37.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4a7fc700 1 -- 172.21.15.3:0/858036789 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e38003260 con 0x7f2e4c0afd60 2022-07-21T01:33:37.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4affd700 1 -- 172.21.15.3:0/858036789 >> v1:172.21.15.143:6789/0 conn(0x7f2e4c0b2710 legacy=0x7f2e4c14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4affd700 1 -- 172.21.15.3:0/858036789 >> v1:172.21.15.3:6789/0 conn(0x7f2e4c0afd60 legacy=0x7f2e4c151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4a7fc700 1 -- 172.21.15.3:0/858036789 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604695469 0 0) 0x7f2e4c0a7480 con 0x7f2e4c0b2710 2022-07-21T01:33:37.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4affd700 1 -- 172.21.15.3:0/858036789 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c0a79c0 con 0x7f2e4c0b01a0 2022-07-21T01:33:37.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.302+0000 7f2e4affd700 1 --2- 172.21.15.3:0/858036789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 0x7f2e4c0a8540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f2e3c0032f0 tx=0x7f2e3c0091d0).ready entity=mon.2 client_cookie=70a9b8fe6abb70e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:37.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e4a7fc700 1 -- 172.21.15.3:0/858036789 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e3c00b6c0 con 0x7f2e4c0b01a0 2022-07-21T01:33:37.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e4a7fc700 1 -- 172.21.15.3:0/858036789 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c00b820 con 0x7f2e4c0b01a0 2022-07-21T01:33:37.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e4a7fc700 1 -- 172.21.15.3:0/858036789 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e3c00b990 con 0x7f2e4c0b01a0 2022-07-21T01:33:37.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 -- 172.21.15.3:0/858036789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 msgr2=0x7f2e4c0a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 --2- 172.21.15.3:0/858036789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 0x7f2e4c0a8540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f2e3c0032f0 tx=0x7f2e3c0091d0).stop 2022-07-21T01:33:37.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 -- 172.21.15.3:0/858036789 shutdown_connections 2022-07-21T01:33:37.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 --2- 172.21.15.3:0/858036789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 0x7f2e4c0a8540 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 -- 172.21.15.3:0/858036789 >> 172.21.15.3:0/858036789 conn(0x7f2e4c060310 msgr2=0x7f2e4c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:37.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 -- 172.21.15.3:0/858036789 shutdown_connections 2022-07-21T01:33:37.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 -- 172.21.15.3:0/858036789 wait complete. 2022-07-21T01:33:37.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 Processor -- start 2022-07-21T01:33:37.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 -- start start 2022-07-21T01:33:37.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4c0afd60 0x7f2e4c14cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 0x7f2e4c0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e4c0b2710 0x7f2e4c0b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c0b3e70 con 0x7f2e4c0b2710 2022-07-21T01:33:37.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c0b43d0 con 0x7f2e4c0afd60 2022-07-21T01:33:37.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e50956700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c0b2f60 con 0x7f2e4c0b01a0 2022-07-21T01:33:37.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e4c0b2710 0x7f2e4c0b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e4affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 0x7f2e4c0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e4c0b2710 0x7f2e4c0b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43128/0 (socket says 172.21.15.3:43128) 2022-07-21T01:33:37.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e4bfff700 1 -- 172.21.15.3:0/461273102 learned_addr learned my addr 172.21.15.3:0/461273102 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:37.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.306+0000 7f2e4b7fe700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4c0afd60 0x7f2e4c14cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e4bfff700 1 -- 172.21.15.3:0/461273102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 msgr2=0x7f2e4c0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e4bfff700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 0x7f2e4c0b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e4bfff700 1 -- 172.21.15.3:0/461273102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4c0afd60 msgr2=0x7f2e4c14cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e4bfff700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4c0afd60 0x7f2e4c14cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e4bfff700 1 -- 172.21.15.3:0/461273102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c009470 con 0x7f2e4c0b2710 2022-07-21T01:33:37.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e4bfff700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e4c0b2710 0x7f2e4c0b4d60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2e44001110 tx=0x7f2e44007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:37.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e48ff9700 1 -- 172.21.15.3:0/461273102 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e4400ce80 con 0x7f2e4c0b2710 2022-07-21T01:33:37.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e48ff9700 1 -- 172.21.15.3:0/461273102 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e440070d0 con 0x7f2e4c0b2710 2022-07-21T01:33:37.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4c0a9fd0 con 0x7f2e4c0b2710 2022-07-21T01:33:37.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e48ff9700 1 -- 172.21.15.3:0/461273102 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e44007430 con 0x7f2e4c0b2710 2022-07-21T01:33:37.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c0b3a30 con 0x7f2e4c0b2710 2022-07-21T01:33:37.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c0b4590 con 0x7f2e4c0b2710 2022-07-21T01:33:37.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e48ff9700 1 -- 172.21.15.3:0/461273102 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2e4401a060 con 0x7f2e4c0b2710 2022-07-21T01:33:37.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e48ff9700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e2c0539e0 0x7f2e2c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:37.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.310+0000 7f2e48ff9700 1 -- 172.21.15.3:0/461273102 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2e4406b730 con 0x7f2e4c0b2710 2022-07-21T01:33:37.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.314+0000 7f2e4b7fe700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e2c0539e0 0x7f2e2c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:37.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.314+0000 7f2e4b7fe700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e2c0539e0 0x7f2e2c055ea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f2e4000d000 tx=0x7f2e40007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:37.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.318+0000 7f2e48ff9700 1 -- 172.21.15.3:0/461273102 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2e440424f0 con 0x7f2e4c0b2710 2022-07-21T01:33:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.482+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e4c0574e0 con 0x7f2e4c0b2710 2022-07-21T01:33:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.482+0000 7f2e48ff9700 1 -- 172.21.15.3:0/461273102 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2474 (secure 0 0 0) 0x7f2e44018070 con 0x7f2e4c0b2710 2022-07-21T01:33:37.481 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:37.481 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/22093665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":22093665},{"type":"v1","addr":"172.21.15.143:6833","nonce":22093665}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/140525254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140525254},{"type":"v1","addr":"172.21.15.3:6835","nonce":140525254}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:33:26.371191+0000","modified":"2022-07-21T01:33:27.386296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/4030643578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":4030643578},{"type":"v1","addr":"172.21.15.3:6837","nonce":4030643578}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-21T01:33:37.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e2c0539e0 msgr2=0x7f2e2c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e2c0539e0 0x7f2e2c055ea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f2e4000d000 tx=0x7f2e40007b90).stop 2022-07-21T01:33:37.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e4c0b2710 msgr2=0x7f2e4c0b4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:37.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e4c0b2710 0x7f2e4c0b4d60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2e44001110 tx=0x7f2e44007a80).stop 2022-07-21T01:33:37.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 shutdown_connections 2022-07-21T01:33:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e4c0b2710 0x7f2e4c0b4d60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e4c0b01a0 0x7f2e4c0b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e2c0539e0 0x7f2e2c055ea0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 --2- 172.21.15.3:0/461273102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4c0afd60 0x7f2e4c14cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 >> 172.21.15.3:0/461273102 conn(0x7f2e4c060310 msgr2=0x7f2e4c145ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 shutdown_connections 2022-07-21T01:33:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:37.486+0000 7f2e50956700 1 -- 172.21.15.3:0/461273102 wait complete. 2022-07-21T01:33:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 6 2022-07-21T01:33:37.796 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:33:26.371191+0000', 'modified': '2022-07-21T01:33:27.386296+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4281': {'gid': 4281, 'name': 'a', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/4030643578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 4030643578}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 4030643578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:33:37.797 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:33:37.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:33:37.916 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":4281,"want_state":"up:active","state":"up:active","rank_uptime":11.540773868,"mdsmap_epoch":6,"osdmap_epoch":26,"osdmap_epoch_barrier":26,"uptime":17.921256923000001} 2022-07-21T01:33:37.916 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 4281, "mdsmap_epoch": 6, "osdmap_epoch": 26, "osdmap_epoch_barrier": 26, "rank_uptime": 11.540773868, "state": "up:active", "uptime": 17.921256923, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:33:37.917 INFO:teuthology.run_tasks:Running task ceph-fuse... 2022-07-21T01:33:37.941 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2022-07-21T01:33:37.941 INFO:tasks.ceph_fuse:config is None 2022-07-21T01:33:37.942 INFO:tasks.ceph_fuse:new config is {'client.0': None, 'client.1': None} 2022-07-21T01:33:37.942 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2022-07-21T01:33:37.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:38.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.054+0000 7f73467e7700 1 Processor -- start 2022-07-21T01:33:38.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.054+0000 7f73467e7700 1 -- start start 2022-07-21T01:33:38.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f73400b1160 0x7f73401448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73400b04e0 con 0x7f73400b1f40 2022-07-21T01:33:38.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73400b0780 con 0x7f7340148530 2022-07-21T01:33:38.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73400a9250 con 0x7f73400b1160 2022-07-21T01:33:38.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f733ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f73400b1f40 0x7f73400b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51414/0 (socket says 172.21.15.3:51414) 2022-07-21T01:33:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f733ffff700 1 -- 172.21.15.3:0/1038479270 learned_addr learned my addr 172.21.15.3:0/1038479270 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f7344d84700 1 --2- 172.21.15.3:0/1038479270 >> v2:172.21.15.143:3300/0 conn(0x7f73400b1160 0x7f73401448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f733effd700 1 -- 172.21.15.3:0/1038479270 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475510987 0 0) 0x7f73400b04e0 con 0x7f73400b1f40 2022-07-21T01:33:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f733effd700 1 -- 172.21.15.3:0/1038479270 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f732c003260 con 0x7f73400b1f40 2022-07-21T01:33:38.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f733effd700 1 -- 172.21.15.3:0/1038479270 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246561782 0 0) 0x7f73400b0780 con 0x7f7340148530 2022-07-21T01:33:38.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f733effd700 1 -- 172.21.15.3:0/1038479270 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73400b04e0 con 0x7f7340148530 2022-07-21T01:33:38.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f7344d84700 1 -- 172.21.15.3:0/1038479270 >> v1:172.21.15.143:6789/0 conn(0x7f7340148530 legacy=0x7f73400a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f7344d84700 1 -- 172.21.15.3:0/1038479270 >> v1:172.21.15.3:6789/0 conn(0x7f73400b1f40 legacy=0x7f73400b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f7344d84700 1 -- 172.21.15.3:0/1038479270 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73400b0a20 con 0x7f73400b1160 2022-07-21T01:33:38.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f7344d84700 1 --2- 172.21.15.3:0/1038479270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 0x7f73401448b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f73401478c0 tx=0x7f7338006df0).ready entity=mon.1 client_cookie=6a95a7efe43410f3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f733effd700 1 -- 172.21.15.3:0/1038479270 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f733800d610 con 0x7f73400b1160 2022-07-21T01:33:38.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f733effd700 1 -- 172.21.15.3:0/1038479270 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7338000e90 con 0x7f73400b1160 2022-07-21T01:33:38.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f733effd700 1 -- 172.21.15.3:0/1038479270 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f733800d960 con 0x7f73400b1160 2022-07-21T01:33:38.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 -- 172.21.15.3:0/1038479270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 msgr2=0x7f73401448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 --2- 172.21.15.3:0/1038479270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 0x7f73401448b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f73401478c0 tx=0x7f7338006df0).stop 2022-07-21T01:33:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 -- 172.21.15.3:0/1038479270 shutdown_connections 2022-07-21T01:33:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 --2- 172.21.15.3:0/1038479270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 0x7f73401448b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 -- 172.21.15.3:0/1038479270 >> 172.21.15.3:0/1038479270 conn(0x7f734005f840 msgr2=0x7f734005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 -- 172.21.15.3:0/1038479270 shutdown_connections 2022-07-21T01:33:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.058+0000 7f73467e7700 1 -- 172.21.15.3:0/1038479270 wait complete. 2022-07-21T01:33:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 Processor -- start 2022-07-21T01:33:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 -- start start 2022-07-21T01:33:38.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 0x7f73400aa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73400b1f40 0x7f73400af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7340148530 0x7f73400af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734014ae90 con 0x7f73400b1f40 2022-07-21T01:33:38.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734014b080 con 0x7f73400b1160 2022-07-21T01:33:38.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734014b320 con 0x7f7340148530 2022-07-21T01:33:38.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73400b1f40 0x7f73400af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73400b1f40 0x7f73400af030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43140/0 (socket says 172.21.15.3:43140) 2022-07-21T01:33:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733f7fe700 1 -- 172.21.15.3:0/3485766023 learned_addr learned my addr 172.21.15.3:0/3485766023 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f7344d84700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7340148530 0x7f73400af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733ffff700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 0x7f73400aa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733f7fe700 1 -- 172.21.15.3:0/3485766023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7340148530 msgr2=0x7f73400af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733f7fe700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7340148530 0x7f73400af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733f7fe700 1 -- 172.21.15.3:0/3485766023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 msgr2=0x7f73400aa8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733f7fe700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 0x7f73400aa8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733f7fe700 1 -- 172.21.15.3:0/3485766023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733800b180 con 0x7f73400b1f40 2022-07-21T01:33:38.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733ffff700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 0x7f73400aa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:38.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733f7fe700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73400b1f40 0x7f73400af030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7330001490 tx=0x7f7330007de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733d7fa700 1 -- 172.21.15.3:0/3485766023 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7330002310 con 0x7f73400b1f40 2022-07-21T01:33:38.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733d7fa700 1 -- 172.21.15.3:0/3485766023 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7330002470 con 0x7f73400b1f40 2022-07-21T01:33:38.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734014b640 con 0x7f73400b1f40 2022-07-21T01:33:38.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f733d7fa700 1 -- 172.21.15.3:0/3485766023 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73300017a0 con 0x7f73400b1f40 2022-07-21T01:33:38.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.062+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73400ab640 con 0x7f73400b1f40 2022-07-21T01:33:38.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.066+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73400b0cc0 con 0x7f73400b1f40 2022-07-21T01:33:38.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.066+0000 7f733d7fa700 1 -- 172.21.15.3:0/3485766023 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7330010030 con 0x7f73400b1f40 2022-07-21T01:33:38.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.066+0000 7f733d7fa700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7320053aa0 0x7f7320055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.066+0000 7f733ffff700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7320053aa0 0x7f7320055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.066+0000 7f733d7fa700 1 -- 172.21.15.3:0/3485766023 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f733006bf80 con 0x7f73400b1f40 2022-07-21T01:33:38.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.066+0000 7f733ffff700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7320053aa0 0x7f7320055f60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f733400b000 tx=0x7f7334009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.070+0000 7f733d7fa700 1 -- 172.21.15.3:0/3485766023 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7330042c30 con 0x7f73400b1f40 2022-07-21T01:33:38.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.198+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73400574e0 con 0x7f73400b1f40 2022-07-21T01:33:38.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.198+0000 7f733d7fa700 1 -- 172.21.15.3:0/3485766023 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7f7330046250 con 0x7f73400b1f40 2022-07-21T01:33:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:38.198 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7320053aa0 msgr2=0x7f7320055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7320053aa0 0x7f7320055f60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f733400b000 tx=0x7f7334009040).stop 2022-07-21T01:33:38.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73400b1f40 msgr2=0x7f73400af030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73400b1f40 0x7f73400af030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7330001490 tx=0x7f7330007de0).stop 2022-07-21T01:33:38.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 shutdown_connections 2022-07-21T01:33:38.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7340148530 0x7f73400af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73400b1f40 0x7f73400af030 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7320053aa0 0x7f7320055f60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 --2- 172.21.15.3:0/3485766023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73400b1160 0x7f73400aa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 >> 172.21.15.3:0/3485766023 conn(0x7f734005f840 msgr2=0x7f73400b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:38.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 shutdown_connections 2022-07-21T01:33:38.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.202+0000 7f73467e7700 1 -- 172.21.15.3:0/3485766023 wait complete. 2022-07-21T01:33:38.210 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:33:21.299936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:33:24.075599+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:33:38.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:33:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe242464700 1 Processor -- start 2022-07-21T01:33:38.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe242464700 1 -- start start 2022-07-21T01:33:38.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe242464700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe23c060eb0 0x7fe23c05f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe242464700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe23c143e20 0x7fe23c12c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe242464700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c0b3dd0 con 0x7fe23c0b2340 2022-07-21T01:33:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe242464700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23c063f80 con 0x7fe23c060eb0 2022-07-21T01:33:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe242464700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23c0b2720 con 0x7fe23c143e20 2022-07-21T01:33:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe241462700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe23c0b2340 0x7fe23c0b36c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51426/0 (socket says 172.21.15.3:51426) 2022-07-21T01:33:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe241462700 1 -- 172.21.15.3:0/2032144551 learned_addr learned my addr 172.21.15.3:0/2032144551 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe240c61700 1 --2- 172.21.15.3:0/2032144551 >> v2:172.21.15.3:3300/0 conn(0x7fe23c060eb0 0x7fe23c05f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe241c63700 1 --2- 172.21.15.3:0/2032144551 >> v2:172.21.15.143:3300/0 conn(0x7fe23c143e20 0x7fe23c12c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe22ffff700 1 -- 172.21.15.3:0/2032144551 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168168152 0 0) 0x7fe23c0b3dd0 con 0x7fe23c0b2340 2022-07-21T01:33:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe22ffff700 1 -- 172.21.15.3:0/2032144551 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe220003260 con 0x7fe23c0b2340 2022-07-21T01:33:38.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe240c61700 1 -- 172.21.15.3:0/2032144551 >> v2:172.21.15.143:3300/0 conn(0x7fe23c143e20 msgr2=0x7fe23c12c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe240c61700 1 --2- 172.21.15.3:0/2032144551 >> v2:172.21.15.143:3300/0 conn(0x7fe23c143e20 0x7fe23c12c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe240c61700 1 -- 172.21.15.3:0/2032144551 >> v1:172.21.15.3:6789/0 conn(0x7fe23c0b2340 legacy=0x7fe23c0b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe240c61700 1 -- 172.21.15.3:0/2032144551 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23c0b2fc0 con 0x7fe23c060eb0 2022-07-21T01:33:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe240c61700 1 --2- 172.21.15.3:0/2032144551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c060eb0 0x7fe23c05f840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fe238002f70 tx=0x7fe238006df0).ready entity=mon.0 client_cookie=861072e2ab8d7208 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe22ffff700 1 -- 172.21.15.3:0/2032144551 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe23800d4f0 con 0x7fe23c060eb0 2022-07-21T01:33:38.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe22ffff700 1 -- 172.21.15.3:0/2032144551 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe238000e90 con 0x7fe23c060eb0 2022-07-21T01:33:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.326+0000 7fe22ffff700 1 -- 172.21.15.3:0/2032144551 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe23800d840 con 0x7fe23c060eb0 2022-07-21T01:33:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 -- 172.21.15.3:0/2032144551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c060eb0 msgr2=0x7fe23c05f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 --2- 172.21.15.3:0/2032144551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c060eb0 0x7fe23c05f840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fe238002f70 tx=0x7fe238006df0).stop 2022-07-21T01:33:38.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 -- 172.21.15.3:0/2032144551 shutdown_connections 2022-07-21T01:33:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 --2- 172.21.15.3:0/2032144551 >> v2:172.21.15.143:3300/0 conn(0x7fe23c143e20 0x7fe23c12c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 --2- 172.21.15.3:0/2032144551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c060eb0 0x7fe23c05f840 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 -- 172.21.15.3:0/2032144551 >> 172.21.15.3:0/2032144551 conn(0x7fe23c05c620 msgr2=0x7fe23c05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 -- 172.21.15.3:0/2032144551 shutdown_connections 2022-07-21T01:33:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 -- 172.21.15.3:0/2032144551 wait complete. 2022-07-21T01:33:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 Processor -- start 2022-07-21T01:33:38.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 -- start start 2022-07-21T01:33:38.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe23c060eb0 0x7fe23c156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe23c0b2340 0x7fe23c0648a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c143e20 0x7fe23c152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c144690 con 0x7fe23c143e20 2022-07-21T01:33:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c155c60 con 0x7fe23c0b2340 2022-07-21T01:33:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.330+0000 7fe242464700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c1448e0 con 0x7fe23c060eb0 2022-07-21T01:33:38.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241c63700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c143e20 0x7fe23c152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241c63700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c143e20 0x7fe23c152690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43156/0 (socket says 172.21.15.3:43156) 2022-07-21T01:33:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241c63700 1 -- 172.21.15.3:0/2015241970 learned_addr learned my addr 172.21.15.3:0/2015241970 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241462700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe23c060eb0 0x7fe23c156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe240c61700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe23c0b2340 0x7fe23c0648a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241c63700 1 -- 172.21.15.3:0/2015241970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe23c060eb0 msgr2=0x7fe23c156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241c63700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe23c060eb0 0x7fe23c156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241c63700 1 -- 172.21.15.3:0/2015241970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe23c0b2340 msgr2=0x7fe23c0648a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241c63700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe23c0b2340 0x7fe23c0648a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241c63700 1 -- 172.21.15.3:0/2015241970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23800b180 con 0x7fe23c143e20 2022-07-21T01:33:38.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241c63700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c143e20 0x7fe23c152690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fe234006e00 tx=0x7fe2340091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe22e7fc700 1 -- 172.21.15.3:0/2015241970 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe234012600 con 0x7fe23c143e20 2022-07-21T01:33:38.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe22e7fc700 1 -- 172.21.15.3:0/2015241970 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234012760 con 0x7fe23c143e20 2022-07-21T01:33:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe22e7fc700 1 -- 172.21.15.3:0/2015241970 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe234012ac0 con 0x7fe23c143e20 2022-07-21T01:33:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe23c155ec0 con 0x7fe23c143e20 2022-07-21T01:33:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe23c156af0 con 0x7fe23c143e20 2022-07-21T01:33:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23c061d60 con 0x7fe23c143e20 2022-07-21T01:33:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe22e7fc700 1 -- 172.21.15.3:0/2015241970 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe234022070 con 0x7fe23c143e20 2022-07-21T01:33:38.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe22e7fc700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe224053aa0 0x7fe224055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe22e7fc700 1 -- 172.21.15.3:0/2015241970 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe23406c5f0 con 0x7fe23c143e20 2022-07-21T01:33:38.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.334+0000 7fe241462700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe224053aa0 0x7fe224055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.338+0000 7fe241462700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe224053aa0 0x7fe224055f60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe23000d000 tx=0x7fe230009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.338+0000 7fe22e7fc700 1 -- 172.21.15.3:0/2015241970 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe234043310 con 0x7fe23c143e20 2022-07-21T01:33:38.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.466+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe23c0574e0 con 0x7fe23c143e20 2022-07-21T01:33:38.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.466+0000 7fe22e7fc700 1 -- 172.21.15.3:0/2015241970 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v26) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe2340160b0 con 0x7fe23c143e20 2022-07-21T01:33:38.466 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:33:38.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe224053aa0 msgr2=0x7fe224055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe224053aa0 0x7fe224055f60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe23000d000 tx=0x7fe230009040).stop 2022-07-21T01:33:38.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c143e20 msgr2=0x7fe23c152690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c143e20 0x7fe23c152690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fe234006e00 tx=0x7fe2340091d0).stop 2022-07-21T01:33:38.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 shutdown_connections 2022-07-21T01:33:38.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe23c143e20 0x7fe23c152690 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe224053aa0 0x7fe224055f60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe23c0b2340 0x7fe23c0648a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 --2- 172.21.15.3:0/2015241970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe23c060eb0 0x7fe23c156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 >> 172.21.15.3:0/2015241970 conn(0x7fe23c05c620 msgr2=0x7fe23c0b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:38.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 shutdown_connections 2022-07-21T01:33:38.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.470+0000 7fe242464700 1 -- 172.21.15.3:0/2015241970 wait complete. 2022-07-21T01:33:38.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:33:38.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44f59e700 1 Processor -- start 2022-07-21T01:33:38.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44f59e700 1 -- start start 2022-07-21T01:33:38.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b41f0 0x7fc450061d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44f59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4500b60e0 con 0x7fc450142ff0 2022-07-21T01:33:38.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4500b2100 con 0x7fc4500b6420 2022-07-21T01:33:38.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500b23a0 con 0x7fc4500b41f0 2022-07-21T01:33:38.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44ed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc450142ff0 0x7fc4501434a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51444/0 (socket says 172.21.15.3:51444) 2022-07-21T01:33:38.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44ed9d700 1 -- 172.21.15.3:0/3872408076 learned_addr learned my addr 172.21.15.3:0/3872408076 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:38.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44dd9b700 1 --2- 172.21.15.3:0/3872408076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b41f0 0x7fc450061d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44d59a700 1 -- 172.21.15.3:0/3872408076 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273912078 0 0) 0x7fc448001490 con 0x7fc450142ff0 2022-07-21T01:33:38.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44d59a700 1 -- 172.21.15.3:0/3872408076 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc434003260 con 0x7fc450142ff0 2022-07-21T01:33:38.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44d59a700 1 -- 172.21.15.3:0/3872408076 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825072910 0 0) 0x7fc4500b2100 con 0x7fc4500b6420 2022-07-21T01:33:38.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.590+0000 7fc44d59a700 1 -- 172.21.15.3:0/3872408076 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc448001490 con 0x7fc4500b6420 2022-07-21T01:33:38.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44dd9b700 1 -- 172.21.15.3:0/3872408076 >> v1:172.21.15.143:6789/0 conn(0x7fc4500b6420 legacy=0x7fc450061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44dd9b700 1 -- 172.21.15.3:0/3872408076 >> v1:172.21.15.3:6789/0 conn(0x7fc450142ff0 legacy=0x7fc4501434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44d59a700 1 -- 172.21.15.3:0/3872408076 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3825113977 0 0) 0x7fc434003260 con 0x7fc450142ff0 2022-07-21T01:33:38.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44dd9b700 1 -- 172.21.15.3:0/3872408076 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4500b2640 con 0x7fc4500b41f0 2022-07-21T01:33:38.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44dd9b700 1 --2- 172.21.15.3:0/3872408076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b41f0 0x7fc450061d10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc448001620 tx=0x7fc444006df0).ready entity=mon.2 client_cookie=d7f3973892cf0dd5 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44d59a700 1 -- 172.21.15.3:0/3872408076 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc448001b70 con 0x7fc4500b41f0 2022-07-21T01:33:38.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44d59a700 1 -- 172.21.15.3:0/3872408076 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc444000e90 con 0x7fc4500b41f0 2022-07-21T01:33:38.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44d59a700 1 -- 172.21.15.3:0/3872408076 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc444012460 con 0x7fc4500b41f0 2022-07-21T01:33:38.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 -- 172.21.15.3:0/3872408076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b41f0 msgr2=0x7fc450061d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 --2- 172.21.15.3:0/3872408076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b41f0 0x7fc450061d10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc448001620 tx=0x7fc444006df0).stop 2022-07-21T01:33:38.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 -- 172.21.15.3:0/3872408076 shutdown_connections 2022-07-21T01:33:38.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 --2- 172.21.15.3:0/3872408076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b41f0 0x7fc450061d10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 -- 172.21.15.3:0/3872408076 >> 172.21.15.3:0/3872408076 conn(0x7fc45005c620 msgr2=0x7fc45005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:38.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 -- 172.21.15.3:0/3872408076 shutdown_connections 2022-07-21T01:33:38.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 -- 172.21.15.3:0/3872408076 wait complete. 2022-07-21T01:33:38.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 Processor -- start 2022-07-21T01:33:38.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 -- start start 2022-07-21T01:33:38.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4500b41f0 0x7fc450141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b6420 0x7fc450155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc450142ff0 0x7fc4501560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500b0b00 con 0x7fc4500b41f0 2022-07-21T01:33:38.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500b0020 con 0x7fc450142ff0 2022-07-21T01:33:38.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500af5e0 con 0x7fc4500b6420 2022-07-21T01:33:38.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4500b41f0 0x7fc450141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4500b41f0 0x7fc450141350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43166/0 (socket says 172.21.15.3:43166) 2022-07-21T01:33:38.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44e59c700 1 -- 172.21.15.3:0/3537746225 learned_addr learned my addr 172.21.15.3:0/3537746225 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:38.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44dd9b700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b6420 0x7fc450155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44e59c700 1 -- 172.21.15.3:0/3537746225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b6420 msgr2=0x7fc450155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44e59c700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b6420 0x7fc450155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.594+0000 7fc44e59c700 1 -- 172.21.15.3:0/3537746225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc450142ff0 msgr2=0x7fc4501560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc44ed9d700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc450142ff0 0x7fc4501560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc44e59c700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc450142ff0 0x7fc4501560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc44e59c700 1 -- 172.21.15.3:0/3537746225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44400b180 con 0x7fc4500b41f0 2022-07-21T01:33:38.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc44ed9d700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc450142ff0 0x7fc4501560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:38.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc44e59c700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4500b41f0 0x7fc450141350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc440001110 tx=0x7fc440007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc43f7fe700 1 -- 172.21.15.3:0/3537746225 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44000f470 con 0x7fc4500b41f0 2022-07-21T01:33:38.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc43f7fe700 1 -- 172.21.15.3:0/3537746225 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44000f5d0 con 0x7fc4500b41f0 2022-07-21T01:33:38.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4501565a0 con 0x7fc4500b41f0 2022-07-21T01:33:38.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc43f7fe700 1 -- 172.21.15.3:0/3537746225 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44000f930 con 0x7fc4500b41f0 2022-07-21T01:33:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc450140380 con 0x7fc4500b41f0 2022-07-21T01:33:38.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4500b0220 con 0x7fc4500b41f0 2022-07-21T01:33:38.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc43f7fe700 1 -- 172.21.15.3:0/3537746225 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc440019030 con 0x7fc4500b41f0 2022-07-21T01:33:38.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc43f7fe700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc42c053b00 0x7fc42c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc44dd9b700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc42c053b00 0x7fc42c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.598+0000 7fc43f7fe700 1 -- 172.21.15.3:0/3537746225 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc44006b660 con 0x7fc4500b41f0 2022-07-21T01:33:38.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.602+0000 7fc44dd9b700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc42c053b00 0x7fc42c055fc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fc444015000 tx=0x7fc444014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.602+0000 7fc43f7fe700 1 -- 172.21.15.3:0/3537746225 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc440042320 con 0x7fc4500b41f0 2022-07-21T01:33:38.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc4500afe80 con 0x7fc4500b41f0 2022-07-21T01:33:38.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc43f7fe700 1 -- 172.21.15.3:0/3537746225 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc4500afe80 con 0x7fc4500b41f0 2022-07-21T01:33:38.727 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:38.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc42c053b00 msgr2=0x7fc42c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc42c053b00 0x7fc42c055fc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fc444015000 tx=0x7fc444014000).stop 2022-07-21T01:33:38.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4500b41f0 msgr2=0x7fc450141350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4500b41f0 0x7fc450141350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc440001110 tx=0x7fc440007ae0).stop 2022-07-21T01:33:38.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 shutdown_connections 2022-07-21T01:33:38.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc42c053b00 0x7fc42c055fc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc450142ff0 0x7fc4501560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4500b6420 0x7fc450155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 --2- 172.21.15.3:0/3537746225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4500b41f0 0x7fc450141350 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 >> 172.21.15.3:0/3537746225 conn(0x7fc45005c620 msgr2=0x7fc4500b8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:38.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 shutdown_connections 2022-07-21T01:33:38.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.730+0000 7fc44f59e700 1 -- 172.21.15.3:0/3537746225 wait complete. 2022-07-21T01:33:38.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:33:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f70148700 1 Processor -- start 2022-07-21T01:33:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f70148700 1 -- start start 2022-07-21T01:33:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f70148700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f68143760 0x7f9f680b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f70148700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9f680611c0 0x7f9f68140580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f70148700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9f68140a70 0x7f9f680b1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f70148700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f680ac110 con 0x7f9f680611c0 2022-07-21T01:33:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f70148700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f680ac340 con 0x7f9f68140a70 2022-07-21T01:33:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f70148700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f68143e10 con 0x7f9f68143760 2022-07-21T01:33:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6d6e3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9f680611c0 0x7f9f68140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6dee4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f68143760 0x7f9f680b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6d6e3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9f680611c0 0x7f9f68140580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43176/0 (socket says 172.21.15.3:43176) 2022-07-21T01:33:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6e6e5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9f68140a70 0x7f9f680b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6dee4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f68143760 0x7f9f680b00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:50090/0 (socket says 172.21.15.3:50090) 2022-07-21T01:33:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6dee4700 1 -- 172.21.15.3:0/623767858 learned_addr learned my addr 172.21.15.3:0/623767858 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6dee4700 1 -- 172.21.15.3:0/623767858 >> v2:172.21.15.143:3300/0 conn(0x7f9f68140a70 msgr2=0x7f9f680b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6dee4700 1 --2- 172.21.15.3:0/623767858 >> v2:172.21.15.143:3300/0 conn(0x7f9f68140a70 0x7f9f680b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6dee4700 1 -- 172.21.15.3:0/623767858 >> v2:172.21.15.3:3300/0 conn(0x7f9f680611c0 msgr2=0x7f9f68140580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6dee4700 1 --2- 172.21.15.3:0/623767858 >> v2:172.21.15.3:3300/0 conn(0x7f9f680611c0 0x7f9f68140580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6dee4700 1 -- 172.21.15.3:0/623767858 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f681440b0 con 0x7f9f68143760 2022-07-21T01:33:38.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6dee4700 1 --2- 172.21.15.3:0/623767858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f68143760 0x7f9f680b00a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9f64002f70 tx=0x7f9f64006eb0).ready entity=mon.2 client_cookie=c87a1ecff114b641 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6cee2700 1 -- 172.21.15.3:0/623767858 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f6400d570 con 0x7f9f68143760 2022-07-21T01:33:38.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6cee2700 1 -- 172.21.15.3:0/623767858 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f64000e90 con 0x7f9f68143760 2022-07-21T01:33:38.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.850+0000 7f9f6cee2700 1 -- 172.21.15.3:0/623767858 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f6400d900 con 0x7f9f68143760 2022-07-21T01:33:38.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- 172.21.15.3:0/623767858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f68143760 msgr2=0x7f9f680b00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 --2- 172.21.15.3:0/623767858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f68143760 0x7f9f680b00a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9f64002f70 tx=0x7f9f64006eb0).stop 2022-07-21T01:33:38.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- 172.21.15.3:0/623767858 shutdown_connections 2022-07-21T01:33:38.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 --2- 172.21.15.3:0/623767858 >> v2:172.21.15.143:3300/0 conn(0x7f9f68140a70 0x7f9f680b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 --2- 172.21.15.3:0/623767858 >> v2:172.21.15.3:3300/0 conn(0x7f9f680611c0 0x7f9f68140580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 --2- 172.21.15.3:0/623767858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f68143760 0x7f9f680b00a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- 172.21.15.3:0/623767858 >> 172.21.15.3:0/623767858 conn(0x7f9f680b5700 msgr2=0x7f9f680b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:38.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- 172.21.15.3:0/623767858 shutdown_connections 2022-07-21T01:33:38.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- 172.21.15.3:0/623767858 wait complete. 2022-07-21T01:33:38.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 Processor -- start 2022-07-21T01:33:38.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- start start 2022-07-21T01:33:38.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f680611c0 0x7f9f68142270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f68140a70 0x7f9f680b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f68143760 0x7f9f680b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680b54d0 con 0x7f9f68143760 2022-07-21T01:33:38.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680b5230 con 0x7f9f68140a70 2022-07-21T01:33:38.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f68147300 con 0x7f9f680611c0 2022-07-21T01:33:38.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6e6e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f68143760 0x7f9f680b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6e6e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f68143760 0x7f9f680b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43184/0 (socket says 172.21.15.3:43184) 2022-07-21T01:33:38.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6e6e5700 1 -- 172.21.15.3:0/1686555933 learned_addr learned my addr 172.21.15.3:0/1686555933 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:38.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6dee4700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f680611c0 0x7f9f68142270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6d6e3700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f68140a70 0x7f9f680b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6e6e5700 1 -- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f680611c0 msgr2=0x7f9f68142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6e6e5700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f680611c0 0x7f9f68142270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6e6e5700 1 -- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f68140a70 msgr2=0x7f9f680b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6e6e5700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f68140a70 0x7f9f680b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6e6e5700 1 -- 172.21.15.3:0/1686555933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6400b1e0 con 0x7f9f68143760 2022-07-21T01:33:38.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6dee4700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f680611c0 0x7f9f68142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:38.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6d6e3700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f68140a70 0x7f9f680b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:38.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f6e6e5700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f68143760 0x7f9f680b37d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f9f60006e00 tx=0x7f9f600091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f56ffd700 1 -- 172.21.15.3:0/1686555933 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f60009620 con 0x7f9f68143760 2022-07-21T01:33:38.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f56ffd700 1 -- 172.21.15.3:0/1686555933 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6000c850 con 0x7f9f68143760 2022-07-21T01:33:38.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f681475a0 con 0x7f9f68143760 2022-07-21T01:33:38.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.854+0000 7f9f56ffd700 1 -- 172.21.15.3:0/1686555933 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f6000cbb0 con 0x7f9f68143760 2022-07-21T01:33:38.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.858+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f680b4d20 con 0x7f9f68143760 2022-07-21T01:33:38.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.858+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f68143b40 con 0x7f9f68143760 2022-07-21T01:33:38.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.858+0000 7f9f56ffd700 1 -- 172.21.15.3:0/1686555933 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9f60022070 con 0x7f9f68143760 2022-07-21T01:33:38.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.858+0000 7f9f56ffd700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f480539e0 0x7f9f48055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:38.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.858+0000 7f9f56ffd700 1 -- 172.21.15.3:0/1686555933 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9f6006c830 con 0x7f9f68143760 2022-07-21T01:33:38.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.858+0000 7f9f6dee4700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f480539e0 0x7f9f48055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:38.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.862+0000 7f9f6dee4700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f480539e0 0x7f9f48055ea0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9f6401a000 tx=0x7f9f64015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:38.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.866+0000 7f9f56ffd700 1 -- 172.21.15.3:0/1686555933 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f600434e0 con 0x7f9f68143760 2022-07-21T01:33:38.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.994+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9f680574e0 con 0x7f9f68143760 2022-07-21T01:33:38.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f56ffd700 1 -- 172.21.15.3:0/1686555933 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9f60046b00 con 0x7f9f68143760 2022-07-21T01:33:38.993 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:38.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f480539e0 msgr2=0x7f9f48055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f480539e0 0x7f9f48055ea0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9f6401a000 tx=0x7f9f64015000).stop 2022-07-21T01:33:38.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f68143760 msgr2=0x7f9f680b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:38.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f68143760 0x7f9f680b37d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f9f60006e00 tx=0x7f9f600091d0).stop 2022-07-21T01:33:38.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 shutdown_connections 2022-07-21T01:33:38.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f68143760 0x7f9f680b37d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f480539e0 0x7f9f48055ea0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f68140a70 0x7f9f680b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 --2- 172.21.15.3:0/1686555933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f680611c0 0x7f9f68142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:38.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 >> 172.21.15.3:0/1686555933 conn(0x7f9f680b5700 msgr2=0x7f9f6813e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:38.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 shutdown_connections 2022-07-21T01:33:38.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:38.998+0000 7f9f70148700 1 -- 172.21.15.3:0/1686555933 wait complete. 2022-07-21T01:33:39.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:33:39.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 Processor -- start 2022-07-21T01:33:39.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 -- start start 2022-07-21T01:33:39.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd5a40b6420 0x7fd5a4061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4142ff0 0x7fd5a41434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a40b60e0 con 0x7fd5a40b41f0 2022-07-21T01:33:39.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a40b2100 con 0x7fd5a40b6420 2022-07-21T01:33:39.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a40b23a0 con 0x7fd5a4142ff0 2022-07-21T01:33:39.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd5a40b6420 0x7fd5a4061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd5a40b6420 0x7fd5a4061600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43188/0 (socket says 172.21.15.3:43188) 2022-07-21T01:33:39.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a259c700 1 -- 172.21.15.3:0/3272753014 learned_addr learned my addr 172.21.15.3:0/3272753014 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:39.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a2d9d700 1 --2- 172.21.15.3:0/3272753014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4142ff0 0x7fd5a41434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a259c700 1 -- 172.21.15.3:0/3272753014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4142ff0 msgr2=0x7fd5a41434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:39.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a259c700 1 --2- 172.21.15.3:0/3272753014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4142ff0 0x7fd5a41434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a259c700 1 -- 172.21.15.3:0/3272753014 >> v1:172.21.15.3:6789/0 conn(0x7fd5a40b41f0 legacy=0x7fd5a4061af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a259c700 1 -- 172.21.15.3:0/3272753014 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a40b2640 con 0x7fd5a40b6420 2022-07-21T01:33:39.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a2d9d700 1 --2- 172.21.15.3:0/3272753014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4142ff0 0x7fd5a41434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:39.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a259c700 1 --2- 172.21.15.3:0/3272753014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a40b6420 0x7fd5a4061600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fd58c002f70 tx=0x7fd58c006eb0).ready entity=mon.0 client_cookie=a89c6419c470b603 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:39.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a159a700 1 -- 172.21.15.3:0/3272753014 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58c00d5c0 con 0x7fd5a40b6420 2022-07-21T01:33:39.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a159a700 1 -- 172.21.15.3:0/3272753014 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c000e90 con 0x7fd5a40b6420 2022-07-21T01:33:39.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a159a700 1 -- 172.21.15.3:0/3272753014 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58c00d950 con 0x7fd5a40b6420 2022-07-21T01:33:39.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 -- 172.21.15.3:0/3272753014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a40b6420 msgr2=0x7fd5a4061600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 --2- 172.21.15.3:0/3272753014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a40b6420 0x7fd5a4061600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fd58c002f70 tx=0x7fd58c006eb0).stop 2022-07-21T01:33:39.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 -- 172.21.15.3:0/3272753014 shutdown_connections 2022-07-21T01:33:39.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 --2- 172.21.15.3:0/3272753014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4142ff0 0x7fd5a41434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 --2- 172.21.15.3:0/3272753014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a40b6420 0x7fd5a4061600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 -- 172.21.15.3:0/3272753014 >> 172.21.15.3:0/3272753014 conn(0x7fd5a405c620 msgr2=0x7fd5a405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:39.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 -- 172.21.15.3:0/3272753014 shutdown_connections 2022-07-21T01:33:39.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.122+0000 7fd5a359e700 1 -- 172.21.15.3:0/3272753014 wait complete. 2022-07-21T01:33:39.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 Processor -- start 2022-07-21T01:33:39.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 -- start start 2022-07-21T01:33:39.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40b41f0 0x7fd5a40b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a40b6420 0x7fd5a4153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a4142ff0 0x7fd5a413f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a41409d0 con 0x7fd5a4142ff0 2022-07-21T01:33:39.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a40afcc0 con 0x7fd5a40b6420 2022-07-21T01:33:39.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a40aff20 con 0x7fd5a40b41f0 2022-07-21T01:33:39.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a4142ff0 0x7fd5a413f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a4142ff0 0x7fd5a413f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43198/0 (socket says 172.21.15.3:43198) 2022-07-21T01:33:39.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a2d9d700 1 -- 172.21.15.3:0/1113579658 learned_addr learned my addr 172.21.15.3:0/1113579658 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:39.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a259c700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40b41f0 0x7fd5a40b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a1d9b700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a40b6420 0x7fd5a4153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a2d9d700 1 -- 172.21.15.3:0/1113579658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40b41f0 msgr2=0x7fd5a40b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a2d9d700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40b41f0 0x7fd5a40b4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a2d9d700 1 -- 172.21.15.3:0/1113579658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a40b6420 msgr2=0x7fd5a4153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a2d9d700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a40b6420 0x7fd5a4153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a2d9d700 1 -- 172.21.15.3:0/1113579658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c00b1e0 con 0x7fd5a4142ff0 2022-07-21T01:33:39.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a2d9d700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a4142ff0 0x7fd5a413f300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fd59c007000 tx=0x7fd59c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:39.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd59b7fe700 1 -- 172.21.15.3:0/1113579658 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd59c007540 con 0x7fd5a4142ff0 2022-07-21T01:33:39.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd59b7fe700 1 -- 172.21.15.3:0/1113579658 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c0076a0 con 0x7fd5a4142ff0 2022-07-21T01:33:39.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a40b01c0 con 0x7fd5a4142ff0 2022-07-21T01:33:39.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd59b7fe700 1 -- 172.21.15.3:0/1113579658 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd59c007a00 con 0x7fd5a4142ff0 2022-07-21T01:33:39.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a41537c0 con 0x7fd5a4142ff0 2022-07-21T01:33:39.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.126+0000 7fd59b7fe700 1 -- 172.21.15.3:0/1113579658 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd59c019030 con 0x7fd5a4142ff0 2022-07-21T01:33:39.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.130+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a40b28e0 con 0x7fd5a4142ff0 2022-07-21T01:33:39.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.130+0000 7fd59b7fe700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd588053950 0x7fd588055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.130+0000 7fd59b7fe700 1 -- 172.21.15.3:0/1113579658 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd59c06c100 con 0x7fd5a4142ff0 2022-07-21T01:33:39.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.130+0000 7fd5a259c700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd588053950 0x7fd588055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.130+0000 7fd5a259c700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd588053950 0x7fd588055e10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd58c017000 tx=0x7fd58c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:39.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.134+0000 7fd59b7fe700 1 -- 172.21.15.3:0/1113579658 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd59c042d30 con 0x7fd5a4142ff0 2022-07-21T01:33:39.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.254+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd5a40574e0 con 0x7fd5a4142ff0 2022-07-21T01:33:39.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.254+0000 7fd59b7fe700 1 -- 172.21.15.3:0/1113579658 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd59c046350 con 0x7fd5a4142ff0 2022-07-21T01:33:39.252 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:39.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd588053950 msgr2=0x7fd588055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd588053950 0x7fd588055e10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd58c017000 tx=0x7fd58c015000).stop 2022-07-21T01:33:39.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a4142ff0 msgr2=0x7fd5a413f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a4142ff0 0x7fd5a413f300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fd59c007000 tx=0x7fd59c008040).stop 2022-07-21T01:33:39.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 shutdown_connections 2022-07-21T01:33:39.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5a4142ff0 0x7fd5a413f300 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd588053950 0x7fd588055e10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a40b6420 0x7fd5a4153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 --2- 172.21.15.3:0/1113579658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40b41f0 0x7fd5a40b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 >> 172.21.15.3:0/1113579658 conn(0x7fd5a405c620 msgr2=0x7fd5a4142510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:39.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 shutdown_connections 2022-07-21T01:33:39.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.258+0000 7fd5a359e700 1 -- 172.21.15.3:0/1113579658 wait complete. 2022-07-21T01:33:39.265 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:33:39.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:33:39.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.370+0000 7ff203c86700 1 Processor -- start 2022-07-21T01:33:39.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.370+0000 7ff203c86700 1 -- start start 2022-07-21T01:33:39.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff1fc0b1f40 0x7ff1fc0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff1fc0a7250 0x7ff1fc1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc0b04e0 con 0x7ff1fc148530 2022-07-21T01:33:39.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1fc0a9250 con 0x7ff1fc0b1f40 2022-07-21T01:33:39.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1fc0b0780 con 0x7ff1fc0a7250 2022-07-21T01:33:39.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff201a22700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff1fc0b1f40 0x7ff1fc0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff201221700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff1fc148530 0x7ff1fc0b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51484/0 (socket says 172.21.15.3:51484) 2022-07-21T01:33:39.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff201221700 1 -- 172.21.15.3:0/3875956981 learned_addr learned my addr 172.21.15.3:0/3875956981 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:39.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff202223700 1 --2- 172.21.15.3:0/3875956981 >> v2:172.21.15.143:3300/0 conn(0x7ff1fc0a7250 0x7ff1fc1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff200a20700 1 -- 172.21.15.3:0/3875956981 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086094832 0 0) 0x7ff1fc0b04e0 con 0x7ff1fc148530 2022-07-21T01:33:39.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff201a22700 1 -- 172.21.15.3:0/3875956981 >> v2:172.21.15.143:3300/0 conn(0x7ff1fc0a7250 msgr2=0x7ff1fc1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:39.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff201a22700 1 --2- 172.21.15.3:0/3875956981 >> v2:172.21.15.143:3300/0 conn(0x7ff1fc0a7250 0x7ff1fc1448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff201a22700 1 -- 172.21.15.3:0/3875956981 >> v1:172.21.15.3:6789/0 conn(0x7ff1fc148530 legacy=0x7ff1fc0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff201a22700 1 -- 172.21.15.3:0/3875956981 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0b0a20 con 0x7ff1fc0b1f40 2022-07-21T01:33:39.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff202223700 1 --2- 172.21.15.3:0/3875956981 >> v2:172.21.15.143:3300/0 conn(0x7ff1fc0a7250 0x7ff1fc1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:39.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff201a22700 1 --2- 172.21.15.3:0/3875956981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0b1f40 0x7ff1fc0b1160 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ff1fc1478c0 tx=0x7ff1f40091d0).ready entity=mon.0 client_cookie=75fad7fd770ae90a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:39.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff200a20700 1 -- 172.21.15.3:0/3875956981 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1f4009ea0 con 0x7ff1fc0b1f40 2022-07-21T01:33:39.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff200a20700 1 -- 172.21.15.3:0/3875956981 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f400b860 con 0x7ff1fc0b1f40 2022-07-21T01:33:39.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff200a20700 1 -- 172.21.15.3:0/3875956981 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1f400b9d0 con 0x7ff1fc0b1f40 2022-07-21T01:33:39.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 -- 172.21.15.3:0/3875956981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0b1f40 msgr2=0x7ff1fc0b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 --2- 172.21.15.3:0/3875956981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0b1f40 0x7ff1fc0b1160 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ff1fc1478c0 tx=0x7ff1f40091d0).stop 2022-07-21T01:33:39.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 -- 172.21.15.3:0/3875956981 shutdown_connections 2022-07-21T01:33:39.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 --2- 172.21.15.3:0/3875956981 >> v2:172.21.15.143:3300/0 conn(0x7ff1fc0a7250 0x7ff1fc1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 --2- 172.21.15.3:0/3875956981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0b1f40 0x7ff1fc0b1160 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 -- 172.21.15.3:0/3875956981 >> 172.21.15.3:0/3875956981 conn(0x7ff1fc05f840 msgr2=0x7ff1fc05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:39.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 -- 172.21.15.3:0/3875956981 shutdown_connections 2022-07-21T01:33:39.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.374+0000 7ff203c86700 1 -- 172.21.15.3:0/3875956981 wait complete. 2022-07-21T01:33:39.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 Processor -- start 2022-07-21T01:33:39.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 -- start start 2022-07-21T01:33:39.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0a7250 0x7ff1fc14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1fc0b1f40 0x7ff1fc1572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1fc148530 0x7ff1fc1577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc156c30 con 0x7ff1fc0a7250 2022-07-21T01:33:39.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc148c90 con 0x7ff1fc0b1f40 2022-07-21T01:33:39.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc148ec0 con 0x7ff1fc148530 2022-07-21T01:33:39.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201a22700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0a7250 0x7ff1fc14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201a22700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0a7250 0x7ff1fc14b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43208/0 (socket says 172.21.15.3:43208) 2022-07-21T01:33:39.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201a22700 1 -- 172.21.15.3:0/2428663866 learned_addr learned my addr 172.21.15.3:0/2428663866 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:39.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201221700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1fc0b1f40 0x7ff1fc1572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff202223700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1fc148530 0x7ff1fc1577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201a22700 1 -- 172.21.15.3:0/2428663866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1fc148530 msgr2=0x7ff1fc1577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201a22700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1fc148530 0x7ff1fc1577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201a22700 1 -- 172.21.15.3:0/2428663866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1fc0b1f40 msgr2=0x7ff1fc1572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201a22700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1fc0b1f40 0x7ff1fc1572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201a22700 1 -- 172.21.15.3:0/2428663866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4009470 con 0x7ff1fc0a7250 2022-07-21T01:33:39.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201221700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1fc0b1f40 0x7ff1fc1572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:39.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff201a22700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0a7250 0x7ff1fc14b550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7ff1fc0a9a30 tx=0x7ff1f4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:39.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff1faffd700 1 -- 172.21.15.3:0/2428663866 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1f40107a0 con 0x7ff1fc0a7250 2022-07-21T01:33:39.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff1faffd700 1 -- 172.21.15.3:0/2428663866 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f4010900 con 0x7ff1fc0a7250 2022-07-21T01:33:39.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc156e20 con 0x7ff1fc0a7250 2022-07-21T01:33:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff1faffd700 1 -- 172.21.15.3:0/2428663866 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1f4010c60 con 0x7ff1fc0a7250 2022-07-21T01:33:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc0a95d0 con 0x7ff1fc0a7250 2022-07-21T01:33:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.378+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc1490c0 con 0x7ff1fc0a7250 2022-07-21T01:33:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.382+0000 7ff1faffd700 1 -- 172.21.15.3:0/2428663866 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff1f4027020 con 0x7ff1fc0a7250 2022-07-21T01:33:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.382+0000 7ff1faffd700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1dc053a70 0x7ff1dc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.382+0000 7ff1faffd700 1 -- 172.21.15.3:0/2428663866 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff1f406d700 con 0x7ff1fc0a7250 2022-07-21T01:33:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.382+0000 7ff201221700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1dc053a70 0x7ff1dc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.382+0000 7ff201221700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1dc053a70 0x7ff1dc055f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ff1f000b000 tx=0x7ff1f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:39.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.386+0000 7ff1faffd700 1 -- 172.21.15.3:0/2428663866 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff1f40443b0 con 0x7ff1fc0a7250 2022-07-21T01:33:39.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.546+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1fc0b0cc0 con 0x7ff1fc0a7250 2022-07-21T01:33:39.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.546+0000 7ff1faffd700 1 -- 172.21.15.3:0/2428663866 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2474 (secure 0 0 0) 0x7ff1f4015090 con 0x7ff1fc0a7250 2022-07-21T01:33:39.544 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:39.545 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/22093665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":22093665},{"type":"v1","addr":"172.21.15.143:6833","nonce":22093665}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/140525254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140525254},{"type":"v1","addr":"172.21.15.3:6835","nonce":140525254}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:33:26.371191+0000","modified":"2022-07-21T01:33:27.386296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/4030643578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":4030643578},{"type":"v1","addr":"172.21.15.3:6837","nonce":4030643578}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-21T01:33:39.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1dc053a70 msgr2=0x7ff1dc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1dc053a70 0x7ff1dc055f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ff1f000b000 tx=0x7ff1f0009040).stop 2022-07-21T01:33:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0a7250 msgr2=0x7ff1fc14b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0a7250 0x7ff1fc14b550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7ff1fc0a9a30 tx=0x7ff1f4016000).stop 2022-07-21T01:33:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 shutdown_connections 2022-07-21T01:33:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1fc148530 0x7ff1fc1577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1dc053a70 0x7ff1dc055f30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1fc0b1f40 0x7ff1fc1572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 --2- 172.21.15.3:0/2428663866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1fc0a7250 0x7ff1fc14b550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:39.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 >> 172.21.15.3:0/2428663866 conn(0x7ff1fc05f840 msgr2=0x7ff1fc144a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:39.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 shutdown_connections 2022-07-21T01:33:39.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:39.550+0000 7ff203c86700 1 -- 172.21.15.3:0/2428663866 wait complete. 2022-07-21T01:33:39.550 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 6 2022-07-21T01:33:39.864 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:33:26.371191+0000', 'modified': '2022-07-21T01:33:27.386296+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4281': {'gid': 4281, 'name': 'a', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/4030643578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 4030643578}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 4030643578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:33:39.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:33:39.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:33:39.994 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":4281,"want_state":"up:active","state":"up:active","rank_uptime":13.618113221,"mdsmap_epoch":6,"osdmap_epoch":26,"osdmap_epoch_barrier":26,"uptime":19.998605803} 2022-07-21T01:33:39.995 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 4281, "mdsmap_epoch": 6, "osdmap_epoch": 26, "osdmap_epoch_barrier": 26, "rank_uptime": 13.618113221, "state": "up:active", "uptime": 19.998605803, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:33:39.995 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:33:39.996 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:33:39.996 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:33:39.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:39.997 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:40.007 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:33:40.068 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:33:40.069 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:33:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:33:40.089 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:33:40.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:40.090 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:33:40.144 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:33:40.145 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:40.147 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:33:40.167 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:33:40.167 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:33:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:33:40.236 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:33:40.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:40.238 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:33:40.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:33:40.235+0000 7fccf660c080 -1 init, newargv = 0x55e84fa41790 newargc=9 2022-07-21T01:33:40.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22152]: starting ceph client 2022-07-21T01:33:40.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22152]: starting fuse 2022-07-21T01:33:40.256 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:33:40.257 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:33:40.257 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:40.258 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:33:40.258 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:33:40.258 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:33:40.259 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:33:40.259 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:33:40.259 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:33:40.260 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:33:40.261 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:33:40.261 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:33:40.261 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:33:40.262 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:33:40.262 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:33:40.263 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:33:40.263 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:33:40.264 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:33:40.265 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:33:40.266 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:33:40.266 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:33:40.267 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:33:40.268 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:33:40.268 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:33:40.269 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:33:40.270 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:33:40.364 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22152.asok 2022-07-21T01:33:40.369 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22152.asok 2022-07-21T01:33:40.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:40.370 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22152.asok status 2022-07-21T01:33:40.459 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:33:40.460 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:33:40.460 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:33:40.460 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:33:40.461 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:33:40.461 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:33:40.461 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:33:40.461 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22152", 2022-07-21T01:33:40.462 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:33:40.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:33:40.463 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:33:40.463 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:33:40.464 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 4659, 2022-07-21T01:33:40.464 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:33:40.464 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:33:40.465 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:33:40.465 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 4659 2022-07-21T01:33:40.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:33:40.466 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:33:40.466 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:33:40.467 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:33:40.467 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2177557537 2022-07-21T01:33:40.468 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:33:40.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:33:40.470 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:33:40.471 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:33:40.471 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:33:40.474 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2177557537 2022-07-21T01:33:40.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:33:40.475 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.4659 172.21.15.171:0/2177557537", 2022-07-21T01:33:40.476 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/2177557537", 2022-07-21T01:33:40.477 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:33:40.478 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 6, 2022-07-21T01:33:40.478 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 26, 2022-07-21T01:33:40.479 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:33:40.480 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:33:40.480 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:33:40.481 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:33:40.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:40.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.614+0000 7f1abc8e8700 1 Processor -- start 2022-07-21T01:33:40.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.614+0000 7f1abc8e8700 1 -- start start 2022-07-21T01:33:40.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ab80614b0 0x7f1ab8061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 0x7f1ab80b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab80aee80 con 0x7f1ab81405d0 2022-07-21T01:33:40.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab80b70b0 con 0x7f1ab80614b0 2022-07-21T01:33:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80af0b0 con 0x7f1ab80ad750 2022-07-21T01:33:40.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ab80614b0 0x7f1ab8061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ab80614b0 0x7f1ab8061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43218/0 (socket says 172.21.15.3:43218) 2022-07-21T01:33:40.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab77fe700 1 -- 172.21.15.3:0/25942395 learned_addr learned my addr 172.21.15.3:0/25942395 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:40.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab6ffd700 1 --2- 172.21.15.3:0/25942395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 0x7f1ab80b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab77fe700 1 -- 172.21.15.3:0/25942395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 msgr2=0x7f1ab80b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:40.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab67fc700 1 -- 172.21.15.3:0/25942395 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535365672 0 0) 0x7f1ab80aee80 con 0x7f1ab81405d0 2022-07-21T01:33:40.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab77fe700 1 --2- 172.21.15.3:0/25942395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 0x7f1ab80b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab77fe700 1 -- 172.21.15.3:0/25942395 >> v1:172.21.15.143:6789/0 conn(0x7f1ab81405d0 legacy=0x7f1ab81466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab77fe700 1 -- 172.21.15.3:0/25942395 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab80af350 con 0x7f1ab80614b0 2022-07-21T01:33:40.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab6ffd700 1 --2- 172.21.15.3:0/25942395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 0x7f1ab80b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:40.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab77fe700 1 --2- 172.21.15.3:0/25942395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 0x7f1ab8061890 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1aa8002f70 tx=0x7f1aa8006eb0).ready entity=mon.0 client_cookie=e870f5bdfe6b037c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:40.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab67fc700 1 -- 172.21.15.3:0/25942395 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa800d570 con 0x7f1ab80614b0 2022-07-21T01:33:40.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab67fc700 1 -- 172.21.15.3:0/25942395 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa8000e90 con 0x7f1ab80614b0 2022-07-21T01:33:40.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1ab67fc700 1 -- 172.21.15.3:0/25942395 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa800d8c0 con 0x7f1ab80614b0 2022-07-21T01:33:40.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 -- 172.21.15.3:0/25942395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 msgr2=0x7f1ab8061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 --2- 172.21.15.3:0/25942395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 0x7f1ab8061890 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1aa8002f70 tx=0x7f1aa8006eb0).stop 2022-07-21T01:33:40.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 -- 172.21.15.3:0/25942395 shutdown_connections 2022-07-21T01:33:40.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 --2- 172.21.15.3:0/25942395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 0x7f1ab80b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 --2- 172.21.15.3:0/25942395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 0x7f1ab8061890 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 -- 172.21.15.3:0/25942395 >> 172.21.15.3:0/25942395 conn(0x7f1ab8066f70 msgr2=0x7f1ab8067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:40.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 -- 172.21.15.3:0/25942395 shutdown_connections 2022-07-21T01:33:40.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.618+0000 7f1abc8e8700 1 -- 172.21.15.3:0/25942395 wait complete. 2022-07-21T01:33:40.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 Processor -- start 2022-07-21T01:33:40.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 -- start start 2022-07-21T01:33:40.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 0x7f1ab8155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 0x7f1ab80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab81405d0 0x7f1ab813f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab8156340 con 0x7f1ab80614b0 2022-07-21T01:33:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab81428e0 con 0x7f1ab81405d0 2022-07-21T01:33:40.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab8142b80 con 0x7f1ab80ad750 2022-07-21T01:33:40.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 0x7f1ab8155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 0x7f1ab8155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43222/0 (socket says 172.21.15.3:43222) 2022-07-21T01:33:40.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab77fe700 1 -- 172.21.15.3:0/2622668468 learned_addr learned my addr 172.21.15.3:0/2622668468 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:40.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab6ffd700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 0x7f1ab80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab7fff700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab81405d0 0x7f1ab813f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab77fe700 1 -- 172.21.15.3:0/2622668468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 msgr2=0x7f1ab80b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab77fe700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 0x7f1ab80b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab77fe700 1 -- 172.21.15.3:0/2622668468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab81405d0 msgr2=0x7f1ab813f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab77fe700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab81405d0 0x7f1ab813f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab77fe700 1 -- 172.21.15.3:0/2622668468 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac00b180 con 0x7f1ab80614b0 2022-07-21T01:33:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab77fe700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 0x7f1ab8155e50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1aa8002f70 tx=0x7f1aa8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab4ff9700 1 -- 172.21.15.3:0/2622668468 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa8011990 con 0x7f1ab80614b0 2022-07-21T01:33:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab4ff9700 1 -- 172.21.15.3:0/2622668468 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa8011af0 con 0x7f1ab80614b0 2022-07-21T01:33:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa800b1e0 con 0x7f1ab80614b0 2022-07-21T01:33:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1ab4ff9700 1 -- 172.21.15.3:0/2622668468 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa8011e50 con 0x7f1ab80614b0 2022-07-21T01:33:40.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.622+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab80b3b50 con 0x7f1ab80614b0 2022-07-21T01:33:40.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.626+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab8142720 con 0x7f1ab80614b0 2022-07-21T01:33:40.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.626+0000 7f1ab4ff9700 1 -- 172.21.15.3:0/2622668468 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1aa8025030 con 0x7f1ab80614b0 2022-07-21T01:33:40.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.626+0000 7f1ab4ff9700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1a980539e0 0x7f1a98055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.626+0000 7f1ab4ff9700 1 -- 172.21.15.3:0/2622668468 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1aa806d2e0 con 0x7f1ab80614b0 2022-07-21T01:33:40.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.626+0000 7f1ab6ffd700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1a980539e0 0x7f1a98055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.626+0000 7f1ab6ffd700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1a980539e0 0x7f1a98055ea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1aac00b000 tx=0x7f1aac008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:40.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.630+0000 7f1ab4ff9700 1 -- 172.21.15.3:0/2622668468 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1aa80440a0 con 0x7f1ab80614b0 2022-07-21T01:33:40.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.754+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ab8141900 con 0x7f1ab80614b0 2022-07-21T01:33:40.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.754+0000 7f1ab4ff9700 1 -- 172.21.15.3:0/2622668468 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7f1aa8023020 con 0x7f1ab80614b0 2022-07-21T01:33:40.752 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:40.752 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:40.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1a980539e0 msgr2=0x7f1a98055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1a980539e0 0x7f1a98055ea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1aac00b000 tx=0x7f1aac008040).stop 2022-07-21T01:33:40.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 msgr2=0x7f1ab8155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 0x7f1ab8155e50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1aa8002f70 tx=0x7f1aa8015000).stop 2022-07-21T01:33:40.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 shutdown_connections 2022-07-21T01:33:40.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1a980539e0 0x7f1a98055ea0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab81405d0 0x7f1ab813f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab80ad750 0x7f1ab80b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 --2- 172.21.15.3:0/2622668468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ab80614b0 0x7f1ab8155e50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 >> 172.21.15.3:0/2622668468 conn(0x7f1ab8066f70 msgr2=0x7f1ab805cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:40.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 shutdown_connections 2022-07-21T01:33:40.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.758+0000 7f1abc8e8700 1 -- 172.21.15.3:0/2622668468 wait complete. 2022-07-21T01:33:40.765 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:33:21.299936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:33:24.075599+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:33:40.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:33:40.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e2401700 1 Processor -- start 2022-07-21T01:33:40.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e2401700 1 -- start start 2022-07-21T01:33:40.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e2401700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa8dc0b07a0 0x7fa8dc0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e2401700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa8dc147730 0x7fa8dc0acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e2401700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8dc147b80 con 0x7fa8dc0b5630 2022-07-21T01:33:40.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e2401700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8dc145470 con 0x7fa8dc147730 2022-07-21T01:33:40.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e2401700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8dc0ae180 con 0x7fa8dc0b07a0 2022-07-21T01:33:40.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e1c00700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa8dc147730 0x7fa8dc0acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e13ff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa8dc0b5630 0x7fa8dc067730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51510/0 (socket says 172.21.15.3:51510) 2022-07-21T01:33:40.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e13ff700 1 -- 172.21.15.3:0/3325496580 learned_addr learned my addr 172.21.15.3:0/3325496580 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:40.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.882+0000 7fa8e0bfe700 1 --2- 172.21.15.3:0/3325496580 >> v2:172.21.15.143:3300/0 conn(0x7fa8dc0b07a0 0x7fa8dc0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e1c00700 1 -- 172.21.15.3:0/3325496580 >> v2:172.21.15.143:3300/0 conn(0x7fa8dc0b07a0 msgr2=0x7fa8dc0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:40.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e1c00700 1 --2- 172.21.15.3:0/3325496580 >> v2:172.21.15.143:3300/0 conn(0x7fa8dc0b07a0 0x7fa8dc0af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e1c00700 1 -- 172.21.15.3:0/3325496580 >> v1:172.21.15.3:6789/0 conn(0x7fa8dc0b5630 legacy=0x7fa8dc067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e1c00700 1 -- 172.21.15.3:0/3325496580 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8dc0b5a10 con 0x7fa8dc147730 2022-07-21T01:33:40.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8d3fff700 1 -- 172.21.15.3:0/3325496580 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557017340 0 0) 0x7fa8dc147b80 con 0x7fa8dc0b5630 2022-07-21T01:33:40.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e0bfe700 1 --2- 172.21.15.3:0/3325496580 >> v2:172.21.15.143:3300/0 conn(0x7fa8dc0b07a0 0x7fa8dc0af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:40.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e1c00700 1 --2- 172.21.15.3:0/3325496580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 0x7fa8dc0acdf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fa8d40032f0 tx=0x7fa8d40091d0).ready entity=mon.0 client_cookie=121247db140e2a7e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:40.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8d3fff700 1 -- 172.21.15.3:0/3325496580 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d400b680 con 0x7fa8dc147730 2022-07-21T01:33:40.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8d3fff700 1 -- 172.21.15.3:0/3325496580 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d400b7e0 con 0x7fa8dc147730 2022-07-21T01:33:40.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8d3fff700 1 -- 172.21.15.3:0/3325496580 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d400b950 con 0x7fa8dc147730 2022-07-21T01:33:40.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 -- 172.21.15.3:0/3325496580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 msgr2=0x7fa8dc0acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 --2- 172.21.15.3:0/3325496580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 0x7fa8dc0acdf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fa8d40032f0 tx=0x7fa8d40091d0).stop 2022-07-21T01:33:40.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 -- 172.21.15.3:0/3325496580 shutdown_connections 2022-07-21T01:33:40.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 --2- 172.21.15.3:0/3325496580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 0x7fa8dc0acdf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 --2- 172.21.15.3:0/3325496580 >> v2:172.21.15.143:3300/0 conn(0x7fa8dc0b07a0 0x7fa8dc0af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 -- 172.21.15.3:0/3325496580 >> 172.21.15.3:0/3325496580 conn(0x7fa8dc0626f0 msgr2=0x7fa8dc062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:40.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 -- 172.21.15.3:0/3325496580 shutdown_connections 2022-07-21T01:33:40.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 -- 172.21.15.3:0/3325496580 wait complete. 2022-07-21T01:33:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 Processor -- start 2022-07-21T01:33:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.886+0000 7fa8e2401700 1 -- start start 2022-07-21T01:33:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e2401700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8dc0b07a0 0x7fa8dc1455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e2401700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8dc0b5630 0x7fa8dc13f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e2401700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 0x7fa8dc140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e2401700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc141ec0 con 0x7fa8dc147730 2022-07-21T01:33:40.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e2401700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc13f980 con 0x7fa8dc0b07a0 2022-07-21T01:33:40.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e2401700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc142560 con 0x7fa8dc0b5630 2022-07-21T01:33:40.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e1c00700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 0x7fa8dc140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e1c00700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 0x7fa8dc140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43240/0 (socket says 172.21.15.3:43240) 2022-07-21T01:33:40.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e1c00700 1 -- 172.21.15.3:0/3221534357 learned_addr learned my addr 172.21.15.3:0/3221534357 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:40.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e0bfe700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8dc0b5630 0x7fa8dc13f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e13ff700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8dc0b07a0 0x7fa8dc1455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e1c00700 1 -- 172.21.15.3:0/3221534357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8dc0b5630 msgr2=0x7fa8dc13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e1c00700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8dc0b5630 0x7fa8dc13f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e1c00700 1 -- 172.21.15.3:0/3221534357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8dc0b07a0 msgr2=0x7fa8dc1455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:40.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e1c00700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8dc0b07a0 0x7fa8dc1455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:40.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e1c00700 1 -- 172.21.15.3:0/3221534357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d800b180 con 0x7fa8dc147730 2022-07-21T01:33:40.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e1c00700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 0x7fa8dc140f50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fa8d40032f0 tx=0x7fa8d4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:40.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8d27fc700 1 -- 172.21.15.3:0/3221534357 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d400b680 con 0x7fa8dc147730 2022-07-21T01:33:40.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8d27fc700 1 -- 172.21.15.3:0/3221534357 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d40107a0 con 0x7fa8dc147730 2022-07-21T01:33:40.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d4009470 con 0x7fa8dc147730 2022-07-21T01:33:40.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8d27fc700 1 -- 172.21.15.3:0/3221534357 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d4010b00 con 0x7fa8dc147730 2022-07-21T01:33:40.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8dc142bf0 con 0x7fa8dc147730 2022-07-21T01:33:40.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8dc142030 con 0x7fa8dc147730 2022-07-21T01:33:40.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8d27fc700 1 -- 172.21.15.3:0/3221534357 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa8d4027020 con 0x7fa8dc147730 2022-07-21T01:33:40.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8d27fc700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8c4053a70 0x7fa8c4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:40.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.890+0000 7fa8d27fc700 1 -- 172.21.15.3:0/3221534357 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa8d406d670 con 0x7fa8dc147730 2022-07-21T01:33:40.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.894+0000 7fa8e13ff700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8c4053a70 0x7fa8c4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:40.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.894+0000 7fa8e13ff700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8c4053a70 0x7fa8c4055f30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fa8cc00b000 tx=0x7fa8cc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:40.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:40.898+0000 7fa8d27fc700 1 -- 172.21.15.3:0/3221534357 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa8d4044390 con 0x7fa8dc147730 2022-07-21T01:33:41.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa8dc0574e0 con 0x7fa8dc147730 2022-07-21T01:33:41.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8d27fc700 1 -- 172.21.15.3:0/3221534357 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v26) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa8d40150b0 con 0x7fa8dc147730 2022-07-21T01:33:41.022 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:33:41.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8c4053a70 msgr2=0x7fa8c4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8c4053a70 0x7fa8c4055f30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fa8cc00b000 tx=0x7fa8cc009040).stop 2022-07-21T01:33:41.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 msgr2=0x7fa8dc140f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 0x7fa8dc140f50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fa8d40032f0 tx=0x7fa8d4016000).stop 2022-07-21T01:33:41.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 shutdown_connections 2022-07-21T01:33:41.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8dc147730 0x7fa8dc140f50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8dc0b5630 0x7fa8dc13f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8c4053a70 0x7fa8c4055f30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 --2- 172.21.15.3:0/3221534357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8dc0b07a0 0x7fa8dc1455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 >> 172.21.15.3:0/3221534357 conn(0x7fa8dc0626f0 msgr2=0x7fa8dc0acd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:41.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 shutdown_connections 2022-07-21T01:33:41.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.026+0000 7fa8e2401700 1 -- 172.21.15.3:0/3221534357 wait complete. 2022-07-21T01:33:41.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:33:41.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.150+0000 7f4f1bfff700 1 Processor -- start 2022-07-21T01:33:41.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.150+0000 7f4f1bfff700 1 -- start start 2022-07-21T01:33:41.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0b08a0 0x7f4f1c142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 0x7f4f1c1433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1bfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c0b0c80 con 0x7f4f1c0ad000 2022-07-21T01:33:41.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1bfff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c0ae810 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0aeab0 con 0x7f4f1c0b3260 2022-07-21T01:33:41.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f127fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 0x7f4f1c1433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1affd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0b08a0 0x7f4f1c142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1b7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4f1c0ad000 0x7f4f1c05f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:53642/0 (socket says 172.21.15.3:53642) 2022-07-21T01:33:41.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1b7fe700 1 -- 172.21.15.3:0/518901224 learned_addr learned my addr 172.21.15.3:0/518901224 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:41.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1a7fc700 1 -- 172.21.15.3:0/518901224 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067518457 0 0) 0x7f4f1c0b0c80 con 0x7f4f1c0ad000 2022-07-21T01:33:41.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1a7fc700 1 -- 172.21.15.3:0/518901224 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4efc003260 con 0x7f4f1c0ad000 2022-07-21T01:33:41.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f127fc700 1 -- 172.21.15.3:0/518901224 >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0b08a0 msgr2=0x7f4f1c142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:41.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f127fc700 1 --2- 172.21.15.3:0/518901224 >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0b08a0 0x7f4f1c142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f127fc700 1 -- 172.21.15.3:0/518901224 >> v1:172.21.15.143:6789/0 conn(0x7f4f1c0ad000 legacy=0x7f4f1c05f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f127fc700 1 -- 172.21.15.3:0/518901224 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c0aed50 con 0x7f4f1c0b3260 2022-07-21T01:33:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f127fc700 1 --2- 172.21.15.3:0/518901224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 0x7f4f1c1433a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4f04002f70 tx=0x7f4f04006df0).ready entity=mon.2 client_cookie=e48fc53ea381bc26 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1a7fc700 1 -- 172.21.15.3:0/518901224 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f0400d4a0 con 0x7f4f1c0b3260 2022-07-21T01:33:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1a7fc700 1 -- 172.21.15.3:0/518901224 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f04000e90 con 0x7f4f1c0b3260 2022-07-21T01:33:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1a7fc700 1 -- 172.21.15.3:0/518901224 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f0400d7f0 con 0x7f4f1c0b3260 2022-07-21T01:33:41.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1bfff700 1 -- 172.21.15.3:0/518901224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 msgr2=0x7f4f1c1433a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.154+0000 7f4f1bfff700 1 --2- 172.21.15.3:0/518901224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 0x7f4f1c1433a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4f04002f70 tx=0x7f4f04006df0).stop 2022-07-21T01:33:41.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.158+0000 7f4f1bfff700 1 -- 172.21.15.3:0/518901224 shutdown_connections 2022-07-21T01:33:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.158+0000 7f4f1bfff700 1 --2- 172.21.15.3:0/518901224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 0x7f4f1c1433a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.158+0000 7f4f1bfff700 1 --2- 172.21.15.3:0/518901224 >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0b08a0 0x7f4f1c142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.158+0000 7f4f1bfff700 1 -- 172.21.15.3:0/518901224 >> 172.21.15.3:0/518901224 conn(0x7f4f1c0612d0 msgr2=0x7f4f1c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.158+0000 7f4f1bfff700 1 -- 172.21.15.3:0/518901224 shutdown_connections 2022-07-21T01:33:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.158+0000 7f4f1bfff700 1 -- 172.21.15.3:0/518901224 wait complete. 2022-07-21T01:33:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.158+0000 7f4f1bfff700 1 Processor -- start 2022-07-21T01:33:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.158+0000 7f4f1bfff700 1 -- start start 2022-07-21T01:33:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c0ad000 0x7f4f1c065930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f1c0b08a0 0x7f4f1c152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 0x7f4f1c152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0a9400 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c05f170 con 0x7f4f1c0ad000 2022-07-21T01:33:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c067070 con 0x7f4f1c0b3260 2022-07-21T01:33:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f127fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f1c0b08a0 0x7f4f1c152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f127fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f1c0b08a0 0x7f4f1c152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43254/0 (socket says 172.21.15.3:43254) 2022-07-21T01:33:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f127fc700 1 -- 172.21.15.3:0/1701476619 learned_addr learned my addr 172.21.15.3:0/1701476619 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1b7fe700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 0x7f4f1c152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1affd700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c0ad000 0x7f4f1c065930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f127fc700 1 -- 172.21.15.3:0/1701476619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 msgr2=0x7f4f1c152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f127fc700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 0x7f4f1c152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f127fc700 1 -- 172.21.15.3:0/1701476619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c0ad000 msgr2=0x7f4f1c065930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f127fc700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c0ad000 0x7f4f1c065930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f127fc700 1 -- 172.21.15.3:0/1701476619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0400b180 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f127fc700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f1c0b08a0 0x7f4f1c152ab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f4f04002f70 tx=0x7f4f04014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f18ff9700 1 -- 172.21.15.3:0/1701476619 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f04016070 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f18ff9700 1 -- 172.21.15.3:0/1701476619 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f040138d0 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c064570 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f18ff9700 1 -- 172.21.15.3:0/1701476619 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f04013c30 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c0642d0 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.162+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c0574e0 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.166+0000 7f4f18ff9700 1 -- 172.21.15.3:0/1701476619 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4f0400f030 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.166+0000 7f4f18ff9700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f000539e0 0x7f4f00055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.166+0000 7f4f18ff9700 1 -- 172.21.15.3:0/1701476619 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4f0406ce50 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.166+0000 7f4f1affd700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f000539e0 0x7f4f00055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.166+0000 7f4f1affd700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f000539e0 0x7f4f00055ea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f4f0c00b000 tx=0x7f4f0c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.174+0000 7f4f18ff9700 1 -- 172.21.15.3:0/1701476619 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f04043b70 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.302+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4f1c0b2e00 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.302+0000 7f4f18ff9700 1 -- 172.21.15.3:0/1701476619 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4f040230a0 con 0x7f4f1c0b08a0 2022-07-21T01:33:41.299 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:41.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f000539e0 msgr2=0x7f4f00055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f000539e0 0x7f4f00055ea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f4f0c00b000 tx=0x7f4f0c006df0).stop 2022-07-21T01:33:41.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f1c0b08a0 msgr2=0x7f4f1c152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f1c0b08a0 0x7f4f1c152ab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f4f04002f70 tx=0x7f4f04014000).stop 2022-07-21T01:33:41.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 shutdown_connections 2022-07-21T01:33:41.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0b3260 0x7f4f1c152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f1c0b08a0 0x7f4f1c152ab0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f000539e0 0x7f4f00055ea0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 --2- 172.21.15.3:0/1701476619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c0ad000 0x7f4f1c065930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 >> 172.21.15.3:0/1701476619 conn(0x7f4f1c0612d0 msgr2=0x7f4f1c143780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:41.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 shutdown_connections 2022-07-21T01:33:41.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.306+0000 7f4f1bfff700 1 -- 172.21.15.3:0/1701476619 wait complete. 2022-07-21T01:33:41.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:33:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f2c60700 1 Processor -- start 2022-07-21T01:33:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f2c60700 1 -- start start 2022-07-21T01:33:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f2c60700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11ec0b2340 0x7f11ec05f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f2c60700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec060eb0 0x7f11ec0b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f2c60700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0b2720 con 0x7f11ec143e20 2022-07-21T01:33:41.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f2c60700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ec063f80 con 0x7f11ec0b2340 2022-07-21T01:33:41.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f2c60700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec0b2fc0 con 0x7f11ec060eb0 2022-07-21T01:33:41.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f145d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec060eb0 0x7f11ec0b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f1c5e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11ec0b2340 0x7f11ec05f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f145d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec060eb0 0x7f11ec0b36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:50176/0 (socket says 172.21.15.3:50176) 2022-07-21T01:33:41.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f145d700 1 -- 172.21.15.3:0/3499643812 learned_addr learned my addr 172.21.15.3:0/3499643812 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:41.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f1c5e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11ec0b2340 0x7f11ec05f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:48706/0 (socket says 172.21.15.3:48706) 2022-07-21T01:33:41.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f245f700 1 --1- 172.21.15.3:0/3499643812 >> v1:172.21.15.143:6789/0 conn(0x7f11ec143e20 0x7f11ec12c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:53658/0 (socket says 172.21.15.3:53658) 2022-07-21T01:33:41.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f145d700 1 -- 172.21.15.3:0/3499643812 >> v2:172.21.15.143:3300/0 conn(0x7f11ec0b2340 msgr2=0x7f11ec05f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:41.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f145d700 1 --2- 172.21.15.3:0/3499643812 >> v2:172.21.15.143:3300/0 conn(0x7f11ec0b2340 0x7f11ec05f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f145d700 1 -- 172.21.15.3:0/3499643812 >> v1:172.21.15.143:6789/0 conn(0x7f11ec143e20 legacy=0x7f11ec12c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f0c5c700 1 -- 172.21.15.3:0/3499643812 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839696083 0 0) 0x7f11ec0b2720 con 0x7f11ec143e20 2022-07-21T01:33:41.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f145d700 1 -- 172.21.15.3:0/3499643812 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec063a40 con 0x7f11ec060eb0 2022-07-21T01:33:41.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f1c5e700 1 --2- 172.21.15.3:0/3499643812 >> v2:172.21.15.143:3300/0 conn(0x7f11ec0b2340 0x7f11ec05f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:41.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f145d700 1 --2- 172.21.15.3:0/3499643812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec060eb0 0x7f11ec0b36c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f11e80032f0 tx=0x7f11e8009290).ready entity=mon.2 client_cookie=4b078b680c85d79f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f0c5c700 1 -- 172.21.15.3:0/3499643812 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11e800b8a0 con 0x7f11ec060eb0 2022-07-21T01:33:41.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f0c5c700 1 -- 172.21.15.3:0/3499643812 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e800ba00 con 0x7f11ec060eb0 2022-07-21T01:33:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.426+0000 7f11f0c5c700 1 -- 172.21.15.3:0/3499643812 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11e800bb70 con 0x7f11ec060eb0 2022-07-21T01:33:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 -- 172.21.15.3:0/3499643812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec060eb0 msgr2=0x7f11ec0b36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 --2- 172.21.15.3:0/3499643812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec060eb0 0x7f11ec0b36c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f11e80032f0 tx=0x7f11e8009290).stop 2022-07-21T01:33:41.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 -- 172.21.15.3:0/3499643812 shutdown_connections 2022-07-21T01:33:41.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 --2- 172.21.15.3:0/3499643812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec060eb0 0x7f11ec0b36c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 --2- 172.21.15.3:0/3499643812 >> v2:172.21.15.143:3300/0 conn(0x7f11ec0b2340 0x7f11ec05f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 -- 172.21.15.3:0/3499643812 >> 172.21.15.3:0/3499643812 conn(0x7f11ec05c620 msgr2=0x7f11ec05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:41.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 -- 172.21.15.3:0/3499643812 shutdown_connections 2022-07-21T01:33:41.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 -- 172.21.15.3:0/3499643812 wait complete. 2022-07-21T01:33:41.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 Processor -- start 2022-07-21T01:33:41.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 -- start start 2022-07-21T01:33:41.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11ec060eb0 0x7f11ec12b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.430+0000 7f11f2c60700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11ec0b2340 0x7f11ec145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f2c60700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec143e20 0x7f11ec146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f2c60700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec146b60 con 0x7f11ec0b2340 2022-07-21T01:33:41.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f2c60700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec0af1d0 con 0x7f11ec060eb0 2022-07-21T01:33:41.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f2c60700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec146860 con 0x7f11ec143e20 2022-07-21T01:33:41.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f145d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11ec0b2340 0x7f11ec145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f145d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11ec0b2340 0x7f11ec145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43266/0 (socket says 172.21.15.3:43266) 2022-07-21T01:33:41.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f145d700 1 -- 172.21.15.3:0/1489556773 learned_addr learned my addr 172.21.15.3:0/1489556773 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:41.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f1c5e700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11ec060eb0 0x7f11ec12b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f245f700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec143e20 0x7f11ec146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f145d700 1 -- 172.21.15.3:0/1489556773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec143e20 msgr2=0x7f11ec146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f145d700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec143e20 0x7f11ec146370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f145d700 1 -- 172.21.15.3:0/1489556773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11ec060eb0 msgr2=0x7f11ec12b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f145d700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11ec060eb0 0x7f11ec12b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f145d700 1 -- 172.21.15.3:0/1489556773 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e400b180 con 0x7f11ec0b2340 2022-07-21T01:33:41.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f145d700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11ec0b2340 0x7f11ec145e80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f11e80032f0 tx=0x7f11e8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11e2ffd700 1 -- 172.21.15.3:0/1489556773 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11e800bb70 con 0x7f11ec0b2340 2022-07-21T01:33:41.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11e2ffd700 1 -- 172.21.15.3:0/1489556773 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e8021630 con 0x7f11ec0b2340 2022-07-21T01:33:41.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e8009530 con 0x7f11ec0b2340 2022-07-21T01:33:41.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11e2ffd700 1 -- 172.21.15.3:0/1489556773 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11e800b8a0 con 0x7f11ec0b2340 2022-07-21T01:33:41.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ec0afc80 con 0x7f11ec0b2340 2022-07-21T01:33:41.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11e2ffd700 1 -- 172.21.15.3:0/1489556773 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f11e8027020 con 0x7f11ec0b2340 2022-07-21T01:33:41.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.434+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ec146f10 con 0x7f11ec0b2340 2022-07-21T01:33:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.438+0000 7f11e2ffd700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f11cc053a70 0x7f11cc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.438+0000 7f11e2ffd700 1 -- 172.21.15.3:0/1489556773 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f11e806d350 con 0x7f11ec0b2340 2022-07-21T01:33:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.438+0000 7f11f1c5e700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f11cc053a70 0x7f11cc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.438+0000 7f11f1c5e700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f11cc053a70 0x7f11cc055f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f11e400b000 tx=0x7f11e4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.442+0000 7f11e2ffd700 1 -- 172.21.15.3:0/1489556773 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f11e8044110 con 0x7f11ec0b2340 2022-07-21T01:33:41.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.566+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11ec063ce0 con 0x7f11ec0b2340 2022-07-21T01:33:41.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11e2ffd700 1 -- 172.21.15.3:0/1489556773 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+10 (secure 0 0 0) 0x7f11e80150a0 con 0x7f11ec0b2340 2022-07-21T01:33:41.565 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:41.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f11cc053a70 msgr2=0x7f11cc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f11cc053a70 0x7f11cc055f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f11e400b000 tx=0x7f11e4006df0).stop 2022-07-21T01:33:41.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11ec0b2340 msgr2=0x7f11ec145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11ec0b2340 0x7f11ec145e80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f11e80032f0 tx=0x7f11e8016000).stop 2022-07-21T01:33:41.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 shutdown_connections 2022-07-21T01:33:41.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11ec143e20 0x7f11ec146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11ec0b2340 0x7f11ec145e80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f11cc053a70 0x7f11cc055f30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 --2- 172.21.15.3:0/1489556773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11ec060eb0 0x7f11ec12b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 >> 172.21.15.3:0/1489556773 conn(0x7f11ec05c620 msgr2=0x7f11ec1430d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:41.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 shutdown_connections 2022-07-21T01:33:41.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.570+0000 7f11f2c60700 1 -- 172.21.15.3:0/1489556773 wait complete. 2022-07-21T01:33:41.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:33:41.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64c16ae700 1 Processor -- start 2022-07-21T01:33:41.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64c16ae700 1 -- start start 2022-07-21T01:33:41.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64c16ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc1445d0 0x7f64bc144b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64c16ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f64bc0655b0 0x7f64bc144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64c16ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc1454e0 con 0x7f64bc060290 2022-07-21T01:33:41.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64c16ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64bc065e10 con 0x7f64bc0655b0 2022-07-21T01:33:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64c16ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc060670 con 0x7f64bc1445d0 2022-07-21T01:33:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64ba7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc1445d0 0x7f64bc144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64bb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f64bc0655b0 0x7f64bc144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64baffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f64bc060290 0x7f64bc064ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:53668/0 (socket says 172.21.15.3:53668) 2022-07-21T01:33:41.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64ba7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc1445d0 0x7f64bc144b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:50190/0 (socket says 172.21.15.3:50190) 2022-07-21T01:33:41.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64ba7fc700 1 -- 172.21.15.3:0/1721863311 learned_addr learned my addr 172.21.15.3:0/1721863311 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:41.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64b9ffb700 1 -- 172.21.15.3:0/1721863311 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688394218 0 0) 0x7f64bc1454e0 con 0x7f64bc060290 2022-07-21T01:33:41.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64b9ffb700 1 -- 172.21.15.3:0/1721863311 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a0003260 con 0x7f64bc060290 2022-07-21T01:33:41.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64ba7fc700 1 -- 172.21.15.3:0/1721863311 >> v2:172.21.15.143:3300/0 conn(0x7f64bc0655b0 msgr2=0x7f64bc144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64ba7fc700 1 --2- 172.21.15.3:0/1721863311 >> v2:172.21.15.143:3300/0 conn(0x7f64bc0655b0 0x7f64bc144ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64ba7fc700 1 -- 172.21.15.3:0/1721863311 >> v1:172.21.15.143:6789/0 conn(0x7f64bc060290 legacy=0x7f64bc064ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.694+0000 7f64ba7fc700 1 -- 172.21.15.3:0/1721863311 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc0b2540 con 0x7f64bc1445d0 2022-07-21T01:33:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64ba7fc700 1 --2- 172.21.15.3:0/1721863311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc1445d0 0x7f64bc144b00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f64ac002f70 tx=0x7f64ac006eb0).ready entity=mon.2 client_cookie=6b7453d09685a23c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64b9ffb700 1 -- 172.21.15.3:0/1721863311 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64ac00d5c0 con 0x7f64bc1445d0 2022-07-21T01:33:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64b9ffb700 1 -- 172.21.15.3:0/1721863311 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac000e90 con 0x7f64bc1445d0 2022-07-21T01:33:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64b9ffb700 1 -- 172.21.15.3:0/1721863311 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64ac00d950 con 0x7f64bc1445d0 2022-07-21T01:33:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 -- 172.21.15.3:0/1721863311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc1445d0 msgr2=0x7f64bc144b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 --2- 172.21.15.3:0/1721863311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc1445d0 0x7f64bc144b00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f64ac002f70 tx=0x7f64ac006eb0).stop 2022-07-21T01:33:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 -- 172.21.15.3:0/1721863311 shutdown_connections 2022-07-21T01:33:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 --2- 172.21.15.3:0/1721863311 >> v2:172.21.15.143:3300/0 conn(0x7f64bc0655b0 0x7f64bc144ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 --2- 172.21.15.3:0/1721863311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc1445d0 0x7f64bc144b00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 -- 172.21.15.3:0/1721863311 >> 172.21.15.3:0/1721863311 conn(0x7f64bc05d0f0 msgr2=0x7f64bc05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 -- 172.21.15.3:0/1721863311 shutdown_connections 2022-07-21T01:33:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 -- 172.21.15.3:0/1721863311 wait complete. 2022-07-21T01:33:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 Processor -- start 2022-07-21T01:33:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.698+0000 7f64c16ae700 1 -- start start 2022-07-21T01:33:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64c16ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc060290 0x7f64bc0b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64c16ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64bc0655b0 0x7f64bc1543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64c16ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64bc1411e0 0x7f64bc150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64c16ae700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc13eea0 con 0x7f64bc0655b0 2022-07-21T01:33:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64c16ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc153a50 con 0x7f64bc1411e0 2022-07-21T01:33:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64c16ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc13f0d0 con 0x7f64bc060290 2022-07-21T01:33:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64ba7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64bc0655b0 0x7f64bc1543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64ba7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64bc0655b0 0x7f64bc1543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43278/0 (socket says 172.21.15.3:43278) 2022-07-21T01:33:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64ba7fc700 1 -- 172.21.15.3:0/1180862325 learned_addr learned my addr 172.21.15.3:0/1180862325 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64baffd700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc060290 0x7f64bc0b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64bb7fe700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64bc1411e0 0x7f64bc150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64ba7fc700 1 -- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc060290 msgr2=0x7f64bc0b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64ba7fc700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc060290 0x7f64bc0b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64ba7fc700 1 -- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64bc1411e0 msgr2=0x7f64bc150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64ba7fc700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64bc1411e0 0x7f64bc150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64ba7fc700 1 -- 172.21.15.3:0/1180862325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac00b1e0 con 0x7f64bc0655b0 2022-07-21T01:33:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64baffd700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc060290 0x7f64bc0b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64bb7fe700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64bc1411e0 0x7f64bc150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64ba7fc700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64bc0655b0 0x7f64bc1543f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f64ac002f70 tx=0x7f64ac015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64abfff700 1 -- 172.21.15.3:0/1180862325 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64ac0117e0 con 0x7f64bc0655b0 2022-07-21T01:33:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64abfff700 1 -- 172.21.15.3:0/1180862325 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac011940 con 0x7f64bc0655b0 2022-07-21T01:33:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc153cb0 con 0x7f64bc0655b0 2022-07-21T01:33:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64abfff700 1 -- 172.21.15.3:0/1180862325 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64ac011ca0 con 0x7f64bc0655b0 2022-07-21T01:33:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc1548e0 con 0x7f64bc0655b0 2022-07-21T01:33:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64bc154150 con 0x7f64bc0655b0 2022-07-21T01:33:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.702+0000 7f64abfff700 1 -- 172.21.15.3:0/1180862325 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f64ac025030 con 0x7f64bc0655b0 2022-07-21T01:33:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.706+0000 7f64abfff700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6498053a70 0x7f6498055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.706+0000 7f64abfff700 1 -- 172.21.15.3:0/1180862325 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f64ac06d490 con 0x7f64bc0655b0 2022-07-21T01:33:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.706+0000 7f64baffd700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6498053a70 0x7f6498055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.706+0000 7f64baffd700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6498053a70 0x7f6498055f30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f64b0008000 tx=0x7f64b0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.706+0000 7f64abfff700 1 -- 172.21.15.3:0/1180862325 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f64bc154150 con 0x7f64bc0655b0 2022-07-21T01:33:41.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f64bc154150 con 0x7f64bc0655b0 2022-07-21T01:33:41.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64abfff700 1 -- 172.21.15.3:0/1180862325 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+10 (secure 0 0 0) 0x7f64bc154150 con 0x7f64bc0655b0 2022-07-21T01:33:41.831 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:41.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6498053a70 msgr2=0x7f6498055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6498053a70 0x7f6498055f30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f64b0008000 tx=0x7f64b0009040).stop 2022-07-21T01:33:41.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64bc0655b0 msgr2=0x7f64bc1543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64bc0655b0 0x7f64bc1543f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f64ac002f70 tx=0x7f64ac015000).stop 2022-07-21T01:33:41.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 shutdown_connections 2022-07-21T01:33:41.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6498053a70 0x7f6498055f30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64bc1411e0 0x7f64bc150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64bc0655b0 0x7f64bc1543f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 --2- 172.21.15.3:0/1180862325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64bc060290 0x7f64bc0b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.834+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 >> 172.21.15.3:0/1180862325 conn(0x7f64bc05d0f0 msgr2=0x7f64bc0b5650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:41.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.838+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 shutdown_connections 2022-07-21T01:33:41.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.838+0000 7f64c16ae700 1 -- 172.21.15.3:0/1180862325 wait complete. 2022-07-21T01:33:41.844 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:33:41.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:33:41.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd9649c0700 1 Processor -- start 2022-07-21T01:33:41.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd9649c0700 1 -- start start 2022-07-21T01:33:41.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd9649c0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9600a8c50 0x7fd960152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd9649c0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9600a7250 con 0x7fd9600b01a0 2022-07-21T01:33:41.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd9649c0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9600a7480 con 0x7fd9600b2710 2022-07-21T01:33:41.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd9649c0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9600a7720 con 0x7fd9600a8c50 2022-07-21T01:33:41.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95effd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd9600b01a0 0x7fd9600a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51566/0 (socket says 172.21.15.3:51566) 2022-07-21T01:33:41.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95effd700 1 -- 172.21.15.3:0/3257132283 learned_addr learned my addr 172.21.15.3:0/3257132283 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:41.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95ffff700 1 --2- 172.21.15.3:0/3257132283 >> v2:172.21.15.143:3300/0 conn(0x7fd9600a8c50 0x7fd960152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24062234 0 0) 0x7fd9600a7250 con 0x7fd9600b01a0 2022-07-21T01:33:41.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd94c003260 con 0x7fd9600b01a0 2022-07-21T01:33:41.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4111618609 0 0) 0x7fd94c003260 con 0x7fd9600b01a0 2022-07-21T01:33:41.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 >> v2:172.21.15.143:3300/0 conn(0x7fd9600a8c50 msgr2=0x7fd960152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:41.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 --2- 172.21.15.3:0/3257132283 >> v2:172.21.15.143:3300/0 conn(0x7fd9600a8c50 0x7fd960152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 >> v1:172.21.15.143:6789/0 conn(0x7fd9600b2710 legacy=0x7fd96014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9600a79c0 con 0x7fd9600b01a0 2022-07-21T01:33:41.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fd9500019e0 con 0x7fd9600b01a0 2022-07-21T01:33:41.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 >> v1:172.21.15.3:6789/0 conn(0x7fd9600b01a0 legacy=0x7fd9600a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd94c003750 0x7fd94c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd94c004500 0x7fd94c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd94c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9600a7720 con 0x7fd94c004e70 2022-07-21T01:33:41.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c00bd20 con 0x7fd94c004500 2022-07-21T01:33:41.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c00bee0 con 0x7fd94c003750 2022-07-21T01:33:41.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95ffff700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd94c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95f7fe700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd94c003750 0x7fd94c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95effd700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd94c004500 0x7fd94c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95ffff700 1 -- 172.21.15.3:0/3257132283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd94c003750 msgr2=0x7fd94c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:41.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95ffff700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd94c003750 0x7fd94c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95ffff700 1 -- 172.21.15.3:0/3257132283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd94c004500 msgr2=0x7fd94c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:41.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.954+0000 7fd95ffff700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd94c004500 0x7fd94c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd95ffff700 1 -- 172.21.15.3:0/3257132283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c003260 con 0x7fd94c004e70 2022-07-21T01:33:41.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd95ffff700 1 -- 172.21.15.3:0/3257132283 reap_dead start 2022-07-21T01:33:41.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd95ffff700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd94c00b750 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd95800c000 tx=0x7fd958006df0).ready entity=mon.0 client_cookie=51ffd0899ba9fef4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd958016070 con 0x7fd94c004e70 2022-07-21T01:33:41.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95800e5f0 con 0x7fd94c004e70 2022-07-21T01:33:41.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd95e7fc700 1 -- 172.21.15.3:0/3257132283 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd95800e950 con 0x7fd94c004e70 2022-07-21T01:33:41.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd9649c0700 1 -- 172.21.15.3:0/3257132283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 msgr2=0x7fd94c00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd9649c0700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd94c00b750 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd95800c000 tx=0x7fd958006df0).stop 2022-07-21T01:33:41.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd9649c0700 1 -- 172.21.15.3:0/3257132283 shutdown_connections 2022-07-21T01:33:41.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd9649c0700 1 --2- 172.21.15.3:0/3257132283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd94c00b750 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd9649c0700 1 -- 172.21.15.3:0/3257132283 >> 172.21.15.3:0/3257132283 conn(0x7fd960060310 msgr2=0x7fd9600b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:41.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd9649c0700 1 -- 172.21.15.3:0/3257132283 shutdown_connections 2022-07-21T01:33:41.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd9649c0700 1 -- 172.21.15.3:0/3257132283 wait complete. 2022-07-21T01:33:41.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd9649c0700 1 Processor -- start 2022-07-21T01:33:41.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.958+0000 7fd9649c0700 1 -- start start 2022-07-21T01:33:41.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd9649c0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd96005c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd9649c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9600aba50 0x7fd960063700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd9649c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd960063bf0 0x7fd960067190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd9649c0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd960064070 con 0x7fd94c004e70 2022-07-21T01:33:41.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd9649c0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9600632a0 con 0x7fd9600aba50 2022-07-21T01:33:41.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd9649c0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9600ab5f0 con 0x7fd960063bf0 2022-07-21T01:33:41.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd96005c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9600aba50 0x7fd960063700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd960063bf0 0x7fd960067190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9600aba50 0x7fd960063700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:48746/0 (socket says 172.21.15.3:48746) 2022-07-21T01:33:41.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95effd700 1 -- 172.21.15.3:0/1705057597 learned_addr learned my addr 172.21.15.3:0/1705057597 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:41.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95f7fe700 1 -- 172.21.15.3:0/1705057597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd960063bf0 msgr2=0x7fd960067190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95f7fe700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd960063bf0 0x7fd960067190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95f7fe700 1 -- 172.21.15.3:0/1705057597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9600aba50 msgr2=0x7fd960063700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:41.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95f7fe700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9600aba50 0x7fd960063700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:41.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95f7fe700 1 -- 172.21.15.3:0/1705057597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9600ab0b0 con 0x7fd94c004e70 2022-07-21T01:33:41.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95f7fe700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd96005c170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fd9600ab780 tx=0x7fd954007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95cff9700 1 -- 172.21.15.3:0/1705057597 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd954002e10 con 0x7fd94c004e70 2022-07-21T01:33:41.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95cff9700 1 -- 172.21.15.3:0/1705057597 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9540114c0 con 0x7fd94c004e70 2022-07-21T01:33:41.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9600ab350 con 0x7fd94c004e70 2022-07-21T01:33:41.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd95cff9700 1 -- 172.21.15.3:0/1705057597 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd954011820 con 0x7fd94c004e70 2022-07-21T01:33:41.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.962+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9600aa370 con 0x7fd94c004e70 2022-07-21T01:33:41.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.966+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9600574e0 con 0x7fd94c004e70 2022-07-21T01:33:41.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.966+0000 7fd95cff9700 1 -- 172.21.15.3:0/1705057597 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd95401a030 con 0x7fd94c004e70 2022-07-21T01:33:41.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.966+0000 7fd95cff9700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd940053aa0 0x7fd940055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:41.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.966+0000 7fd95cff9700 1 -- 172.21.15.3:0/1705057597 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd95406b700 con 0x7fd94c004e70 2022-07-21T01:33:41.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.966+0000 7fd95effd700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd940053aa0 0x7fd940055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:41.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.966+0000 7fd95effd700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd940053aa0 0x7fd940055f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd950005000 tx=0x7fd9600a7b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:41.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:41.974+0000 7fd95cff9700 1 -- 172.21.15.3:0/1705057597 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd954068e90 con 0x7fd94c004e70 2022-07-21T01:33:42.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.134+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd960069a70 con 0x7fd94c004e70 2022-07-21T01:33:42.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.134+0000 7fd95cff9700 1 -- 172.21.15.3:0/1705057597 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2474 (secure 0 0 0) 0x7fd954042510 con 0x7fd94c004e70 2022-07-21T01:33:42.134 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:42.134 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/22093665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":22093665},{"type":"v1","addr":"172.21.15.143:6833","nonce":22093665}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/140525254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140525254},{"type":"v1","addr":"172.21.15.3:6835","nonce":140525254}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:33:26.371191+0000","modified":"2022-07-21T01:33:27.386296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/4030643578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":4030643578},{"type":"v1","addr":"172.21.15.3:6837","nonce":4030643578}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-21T01:33:42.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd940053aa0 msgr2=0x7fd940055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:42.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd940053aa0 0x7fd940055f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd950005000 tx=0x7fd9600a7b50).stop 2022-07-21T01:33:42.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 msgr2=0x7fd96005c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:42.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd96005c170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fd9600ab780 tx=0x7fd954007a00).stop 2022-07-21T01:33:42.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 shutdown_connections 2022-07-21T01:33:42.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd960063bf0 0x7fd960067190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:42.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd940053aa0 0x7fd940055f60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:42.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9600aba50 0x7fd960063700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:42.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 --2- 172.21.15.3:0/1705057597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd94c004e70 0x7fd96005c170 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:42.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 >> 172.21.15.3:0/1705057597 conn(0x7fd960060310 msgr2=0x7fd960148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:42.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 shutdown_connections 2022-07-21T01:33:42.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:42.138+0000 7fd9649c0700 1 -- 172.21.15.3:0/1705057597 wait complete. 2022-07-21T01:33:42.140 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 6 2022-07-21T01:33:42.448 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:33:26.371191+0000', 'modified': '2022-07-21T01:33:27.386296+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4281': {'gid': 4281, 'name': 'a', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/4030643578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 4030643578}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 4030643578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:33:42.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:33:42.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:33:42.568 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":4281,"want_state":"up:active","state":"up:active","rank_uptime":16.192553451999999,"mdsmap_epoch":6,"osdmap_epoch":26,"osdmap_epoch_barrier":26,"uptime":22.573042857000001} 2022-07-21T01:33:42.568 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 4281, "mdsmap_epoch": 6, "osdmap_epoch": 26, "osdmap_epoch_barrier": 26, "rank_uptime": 16.192553452, "state": "up:active", "uptime": 22.573042857, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:33:42.569 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:33:42.569 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:33:42.569 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:33:42.570 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:42.570 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:42.579 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:33:42.638 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:33:42.638 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:33:42.653 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:33:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:33:42.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:42.656 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:33:42.663 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:33:42.663 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:42.666 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:33:42.682 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:33:42.682 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:33:42.693 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:33:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:33:42.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:42.695 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:33:42.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:33:42.778+0000 7f6d46e84080 -1 init, newargv = 0x560378f1abf0 newargc=9 2022-07-21T01:33:42.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22249]: starting ceph client 2022-07-21T01:33:42.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22249]: starting fuse 2022-07-21T01:33:43.849 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:33:43.868 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:33:43.869 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:33:43.928 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:33:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:33:43.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:43.931 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:33:43.936 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:33:43.936 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:33:43.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:43.937 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:33:43.937 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:33:43.938 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:33:43.938 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:33:43.938 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:33:43.939 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:33:43.939 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:33:43.939 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:33:43.940 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:33:43.940 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:33:43.940 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:33:43.941 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:33:43.941 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:33:43.941 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:33:43.942 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:33:43.942 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:33:43.943 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:33:43.943 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:33:43.943 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:33:43.944 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:33:43.944 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:33:43.944 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:33:43.945 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:33:44.021 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.22249.asok 2022-07-21T01:33:44.026 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22249.asok 2022-07-21T01:33:44.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:44.026 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22249.asok status 2022-07-21T01:33:44.115 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:33:44.116 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:33:44.116 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:33:44.116 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:33:44.116 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:33:44.117 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:33:44.117 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:33:44.117 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "22249", 2022-07-21T01:33:44.118 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:33:44.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:33:44.118 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:33:44.118 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:33:44.119 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 4695, 2022-07-21T01:33:44.119 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:33:44.119 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:33:44.119 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:33:44.119 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 4695 2022-07-21T01:33:44.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:33:44.120 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:33:44.120 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:33:44.120 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:33:44.121 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3773055576 2022-07-21T01:33:44.121 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:33:44.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:33:44.121 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:33:44.122 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:33:44.122 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:33:44.122 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3773055576 2022-07-21T01:33:44.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:33:44.123 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.4695 172.21.15.174:0/3773055576", 2022-07-21T01:33:44.123 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/3773055576", 2022-07-21T01:33:44.123 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:33:44.123 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 6, 2022-07-21T01:33:44.124 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 26, 2022-07-21T01:33:44.124 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:33:44.124 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:33:44.124 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:33:44.125 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:33:44.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:44.127 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:44.127 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:44.136 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:33:44.138 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:33:44.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:44.139 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:44.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:44.163 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:44.164 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:44.172 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:33:44.173 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:33:44.173 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:44.174 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:44.239 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2022-07-21T01:33:44.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:44.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.378+0000 7f19c3fff700 1 Processor -- start 2022-07-21T01:33:44.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 -- start start 2022-07-21T01:33:44.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f19c40b28b0 0x7f19c413e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4141920 0x7f19c413e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c40aac20 con 0x7f19c40b1180 2022-07-21T01:33:44.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c4140e20 con 0x7f19c40b28b0 2022-07-21T01:33:44.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c40b2c90 con 0x7f19c4141920 2022-07-21T01:33:44.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c2ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f19c40b28b0 0x7f19c413e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4141920 0x7f19c413e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19bbfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f19c40b1180 0x7f19c4141210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:53702/0 (socket says 172.21.15.3:53702) 2022-07-21T01:33:44.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19bbfff700 1 -- 172.21.15.3:0/974815877 learned_addr learned my addr 172.21.15.3:0/974815877 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:44.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c27fc700 1 -- 172.21.15.3:0/974815877 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741487550 0 0) 0x7f19ac001110 con 0x7f19c40b1180 2022-07-21T01:33:44.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c37fe700 1 -- 172.21.15.3:0/974815877 >> v2:172.21.15.143:3300/0 conn(0x7f19c40b28b0 msgr2=0x7f19c413e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:44.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c37fe700 1 --2- 172.21.15.3:0/974815877 >> v2:172.21.15.143:3300/0 conn(0x7f19c40b28b0 0x7f19c413e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c37fe700 1 -- 172.21.15.3:0/974815877 >> v1:172.21.15.143:6789/0 conn(0x7f19c40b1180 legacy=0x7f19c4141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c37fe700 1 -- 172.21.15.3:0/974815877 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c40aa440 con 0x7f19c4141920 2022-07-21T01:33:44.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c37fe700 1 --2- 172.21.15.3:0/974815877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4141920 0x7f19c413e6b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f19bc0032f0 tx=0x7f19bc0091d0).ready entity=mon.2 client_cookie=e91b83ef767b72fa server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:44.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c27fc700 1 -- 172.21.15.3:0/974815877 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19bc00b6c0 con 0x7f19c4141920 2022-07-21T01:33:44.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c27fc700 1 -- 172.21.15.3:0/974815877 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc00b820 con 0x7f19c4141920 2022-07-21T01:33:44.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c27fc700 1 -- 172.21.15.3:0/974815877 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19bc00b990 con 0x7f19c4141920 2022-07-21T01:33:44.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 -- 172.21.15.3:0/974815877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4141920 msgr2=0x7f19c413e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 --2- 172.21.15.3:0/974815877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4141920 0x7f19c413e6b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f19bc0032f0 tx=0x7f19bc0091d0).stop 2022-07-21T01:33:44.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 -- 172.21.15.3:0/974815877 shutdown_connections 2022-07-21T01:33:44.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 --2- 172.21.15.3:0/974815877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c4141920 0x7f19c413e6b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 --2- 172.21.15.3:0/974815877 >> v2:172.21.15.143:3300/0 conn(0x7f19c40b28b0 0x7f19c413e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 -- 172.21.15.3:0/974815877 >> 172.21.15.3:0/974815877 conn(0x7f19c4060310 msgr2=0x7f19c40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:44.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 -- 172.21.15.3:0/974815877 shutdown_connections 2022-07-21T01:33:44.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.382+0000 7f19c3fff700 1 -- 172.21.15.3:0/974815877 wait complete. 2022-07-21T01:33:44.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c3fff700 1 Processor -- start 2022-07-21T01:33:44.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c3fff700 1 -- start start 2022-07-21T01:33:44.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c40b1180 0x7f19c4142f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c40b28b0 0x7f19c4143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19c4141920 0x7f19c4144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c3fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c41511b0 con 0x7f19c4141920 2022-07-21T01:33:44.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c3fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c4150f10 con 0x7f19c40b1180 2022-07-21T01:33:44.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c3fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c4063fc0 con 0x7f19c40b28b0 2022-07-21T01:33:44.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19c4141920 0x7f19c4144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19c4141920 0x7f19c4144030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43316/0 (socket says 172.21.15.3:43316) 2022-07-21T01:33:44.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c37fe700 1 -- 172.21.15.3:0/1822863297 learned_addr learned my addr 172.21.15.3:0/1822863297 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:44.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19bbfff700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c40b28b0 0x7f19c4143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.386+0000 7f19c2ffd700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c40b1180 0x7f19c4142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c37fe700 1 -- 172.21.15.3:0/1822863297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c40b28b0 msgr2=0x7f19c4143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c37fe700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c40b28b0 0x7f19c4143b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c37fe700 1 -- 172.21.15.3:0/1822863297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c40b1180 msgr2=0x7f19c4142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c37fe700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c40b1180 0x7f19c4142f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c37fe700 1 -- 172.21.15.3:0/1822863297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc009470 con 0x7f19c4141920 2022-07-21T01:33:44.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19bbfff700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c40b28b0 0x7f19c4143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:44.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c37fe700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19c4141920 0x7f19c4144030 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f19bc0032f0 tx=0x7f19bc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:44.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c0ff9700 1 -- 172.21.15.3:0/1822863297 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19bc00b6c0 con 0x7f19c4141920 2022-07-21T01:33:44.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c0ff9700 1 -- 172.21.15.3:0/1822863297 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc0205f0 con 0x7f19c4141920 2022-07-21T01:33:44.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c0ff9700 1 -- 172.21.15.3:0/1822863297 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19bc020950 con 0x7f19c4141920 2022-07-21T01:33:44.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c4064260 con 0x7f19c4141920 2022-07-21T01:33:44.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c40671a0 con 0x7f19c4141920 2022-07-21T01:33:44.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.390+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c4064420 con 0x7f19c4141920 2022-07-21T01:33:44.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.394+0000 7f19c0ff9700 1 -- 172.21.15.3:0/1822863297 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f19bc01e070 con 0x7f19c4141920 2022-07-21T01:33:44.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.394+0000 7f19c0ff9700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19a8053a70 0x7f19a8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.394+0000 7f19c0ff9700 1 -- 172.21.15.3:0/1822863297 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f19bc06ced0 con 0x7f19c4141920 2022-07-21T01:33:44.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.394+0000 7f19c2ffd700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19a8053a70 0x7f19a8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.394+0000 7f19c2ffd700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19a8053a70 0x7f19a8055f30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f19b400b000 tx=0x7f19b4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:44.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.402+0000 7f19c0ff9700 1 -- 172.21.15.3:0/1822863297 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f19bc043d90 con 0x7f19c4141920 2022-07-21T01:33:44.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.530+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19c40574e0 con 0x7f19c4141920 2022-07-21T01:33:44.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.530+0000 7f19c0ff9700 1 -- 172.21.15.3:0/1822863297 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7f19bc031fc0 con 0x7f19c4141920 2022-07-21T01:33:44.528 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:44.529 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:44.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19a8053a70 msgr2=0x7f19a8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19a8053a70 0x7f19a8055f30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f19b400b000 tx=0x7f19b4006df0).stop 2022-07-21T01:33:44.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19c4141920 msgr2=0x7f19c4144030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19c4141920 0x7f19c4144030 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f19bc0032f0 tx=0x7f19bc016000).stop 2022-07-21T01:33:44.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 shutdown_connections 2022-07-21T01:33:44.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19c4141920 0x7f19c4144030 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19c40b28b0 0x7f19c4143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19a8053a70 0x7f19a8055f30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 --2- 172.21.15.3:0/1822863297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19c40b1180 0x7f19c4142f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 >> 172.21.15.3:0/1822863297 conn(0x7f19c4060310 msgr2=0x7f19c41405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:44.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 shutdown_connections 2022-07-21T01:33:44.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.534+0000 7f19c3fff700 1 -- 172.21.15.3:0/1822863297 wait complete. 2022-07-21T01:33:44.541 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:33:21.299936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:33:24.075599+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:33:44.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:33:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.654+0000 7f74c4c19700 1 Processor -- start 2022-07-21T01:33:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.654+0000 7f74c4c19700 1 -- start start 2022-07-21T01:33:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.654+0000 7f74c4c19700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f74c00b0c60 0x7f74c00afad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.654+0000 7f74c4c19700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00ad3c0 0x7f74c00affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.654+0000 7f74c4c19700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f74c00b1640 0x7f74c0143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.654+0000 7f74c4c19700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c00b1040 con 0x7f74c00b1640 2022-07-21T01:33:44.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.654+0000 7f74c4c19700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c00b4f90 con 0x7f74c00b0c60 2022-07-21T01:33:44.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c00b5230 con 0x7f74c00ad3c0 2022-07-21T01:33:44.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f74c00b1640 0x7f74c0143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f74c00b1640 0x7f74c0143690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43324/0 (socket says 172.21.15.3:43324) 2022-07-21T01:33:44.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bed9d700 1 -- 172.21.15.3:0/3307115756 learned_addr learned my addr 172.21.15.3:0/3307115756 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:44.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74be59c700 1 --2- 172.21.15.3:0/3307115756 >> v2:172.21.15.143:3300/0 conn(0x7f74c00b0c60 0x7f74c00afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74b5d9b700 1 --2- 172.21.15.3:0/3307115756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00ad3c0 0x7f74c00affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bed9d700 1 -- 172.21.15.3:0/3307115756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00ad3c0 msgr2=0x7f74c00affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:44.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bed9d700 1 --2- 172.21.15.3:0/3307115756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00ad3c0 0x7f74c00affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bed9d700 1 -- 172.21.15.3:0/3307115756 >> v2:172.21.15.143:3300/0 conn(0x7f74c00b0c60 msgr2=0x7f74c00afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:44.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bed9d700 1 --2- 172.21.15.3:0/3307115756 >> v2:172.21.15.143:3300/0 conn(0x7f74c00b0c60 0x7f74c00afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bed9d700 1 -- 172.21.15.3:0/3307115756 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c00b54d0 con 0x7f74c00b1640 2022-07-21T01:33:44.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74be59c700 1 --2- 172.21.15.3:0/3307115756 >> v2:172.21.15.143:3300/0 conn(0x7f74c00b0c60 0x7f74c00afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:44.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bed9d700 1 --2- 172.21.15.3:0/3307115756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 0x7f74c0143690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f74b80032f0 tx=0x7f74b800b530).ready entity=mon.0 client_cookie=f837a5cf39f2ed3a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:44.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bdd9b700 1 -- 172.21.15.3:0/3307115756 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b800be70 con 0x7f74c00b1640 2022-07-21T01:33:44.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bdd9b700 1 -- 172.21.15.3:0/3307115756 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b8010470 con 0x7f74c00b1640 2022-07-21T01:33:44.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74bdd9b700 1 -- 172.21.15.3:0/3307115756 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b80107d0 con 0x7f74c00b1640 2022-07-21T01:33:44.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 -- 172.21.15.3:0/3307115756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 msgr2=0x7f74c0143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 --2- 172.21.15.3:0/3307115756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 0x7f74c0143690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f74b80032f0 tx=0x7f74b800b530).stop 2022-07-21T01:33:44.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 -- 172.21.15.3:0/3307115756 shutdown_connections 2022-07-21T01:33:44.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 --2- 172.21.15.3:0/3307115756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 0x7f74c0143690 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 --2- 172.21.15.3:0/3307115756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00ad3c0 0x7f74c00affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 --2- 172.21.15.3:0/3307115756 >> v2:172.21.15.143:3300/0 conn(0x7f74c00b0c60 0x7f74c00afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 -- 172.21.15.3:0/3307115756 >> 172.21.15.3:0/3307115756 conn(0x7f74c006a2a0 msgr2=0x7f74c005ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:44.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 -- 172.21.15.3:0/3307115756 shutdown_connections 2022-07-21T01:33:44.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.658+0000 7f74c4c19700 1 -- 172.21.15.3:0/3307115756 wait complete. 2022-07-21T01:33:44.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74c4c19700 1 Processor -- start 2022-07-21T01:33:44.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74c4c19700 1 -- start start 2022-07-21T01:33:44.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74c4c19700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74c00ad3c0 0x7f74c009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74c4c19700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00b0c60 0x7f74c00b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74c4c19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 0x7f74c00a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74c4c19700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c01400d0 con 0x7f74c00b1640 2022-07-21T01:33:44.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74c4c19700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c013f220 con 0x7f74c00ad3c0 2022-07-21T01:33:44.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74c4c19700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c013f4c0 con 0x7f74c00b0c60 2022-07-21T01:33:44.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74bed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 0x7f74c00a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74bed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 0x7f74c00a0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43330/0 (socket says 172.21.15.3:43330) 2022-07-21T01:33:44.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74bed9d700 1 -- 172.21.15.3:0/175739163 learned_addr learned my addr 172.21.15.3:0/175739163 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:44.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74b5d9b700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00b0c60 0x7f74c00b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74be59c700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74c00ad3c0 0x7f74c009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74bed9d700 1 -- 172.21.15.3:0/175739163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00b0c60 msgr2=0x7f74c00b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74bed9d700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00b0c60 0x7f74c00b5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74bed9d700 1 -- 172.21.15.3:0/175739163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74c00ad3c0 msgr2=0x7f74c009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74bed9d700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74c00ad3c0 0x7f74c009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.662+0000 7f74bed9d700 1 -- 172.21.15.3:0/175739163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b800b7d0 con 0x7f74c00b1640 2022-07-21T01:33:44.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74bed9d700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 0x7f74c00a0500 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f74b80032f0 tx=0x7f74b800b530).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:44.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74b7fff700 1 -- 172.21.15.3:0/175739163 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b800be70 con 0x7f74c00b1640 2022-07-21T01:33:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74b7fff700 1 -- 172.21.15.3:0/175739163 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b8018610 con 0x7f74c00b1640 2022-07-21T01:33:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74b7fff700 1 -- 172.21.15.3:0/175739163 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b8018970 con 0x7f74c00b1640 2022-07-21T01:33:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c013f760 con 0x7f74c00b1640 2022-07-21T01:33:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c013fca0 con 0x7f74c00b1640 2022-07-21T01:33:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c013ff20 con 0x7f74c00b1640 2022-07-21T01:33:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74b7fff700 1 -- 172.21.15.3:0/175739163 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f74b8021020 con 0x7f74c00b1640 2022-07-21T01:33:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74b7fff700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f74a4053a10 0x7f74a4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74b7fff700 1 -- 172.21.15.3:0/175739163 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f74b806ffc0 con 0x7f74c00b1640 2022-07-21T01:33:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74be59c700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f74a4053a10 0x7f74a4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.666+0000 7f74be59c700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f74a4053a10 0x7f74a4055ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f74b000c000 tx=0x7f74b0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.674+0000 7f74b7fff700 1 -- 172.21.15.3:0/175739163 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f74b8046d80 con 0x7f74c00b1640 2022-07-21T01:33:44.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.802+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f74c00574e0 con 0x7f74c00b1640 2022-07-21T01:33:44.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.802+0000 7f74b7fff700 1 -- 172.21.15.3:0/175739163 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v26) v1 ==== 110+0+10 (secure 0 0 0) 0x7f74b804a3a0 con 0x7f74c00b1640 2022-07-21T01:33:44.800 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:33:44.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f74a4053a10 msgr2=0x7f74a4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f74a4053a10 0x7f74a4055ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f74b000c000 tx=0x7f74b0008040).stop 2022-07-21T01:33:44.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 msgr2=0x7f74c00a0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 0x7f74c00a0500 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f74b80032f0 tx=0x7f74b800b530).stop 2022-07-21T01:33:44.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 shutdown_connections 2022-07-21T01:33:44.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74c00b1640 0x7f74c00a0500 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c00b0c60 0x7f74c00b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f74a4053a10 0x7f74a4055ed0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 --2- 172.21.15.3:0/175739163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74c00ad3c0 0x7f74c009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 >> 172.21.15.3:0/175739163 conn(0x7f74c006a2a0 msgr2=0x7f74c0066da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:44.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 shutdown_connections 2022-07-21T01:33:44.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.806+0000 7f74c4c19700 1 -- 172.21.15.3:0/175739163 wait complete. 2022-07-21T01:33:44.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:33:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.925+0000 7fb2268bd700 1 Processor -- start 2022-07-21T01:33:44.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.925+0000 7fb2268bd700 1 -- start start 2022-07-21T01:33:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200612f0 0x7fb220143b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2200671e0 con 0x7fb2200b7840 2022-07-21T01:33:44.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb220065540 con 0x7fb2200b2930 2022-07-21T01:33:44.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2200b7c20 con 0x7fb2200612f0 2022-07-21T01:33:44.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2258bb700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb2200b7840 0x7fb220060be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51614/0 (socket says 172.21.15.3:51614) 2022-07-21T01:33:44.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2258bb700 1 -- 172.21.15.3:0/1247272986 learned_addr learned my addr 172.21.15.3:0/1247272986 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:44.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2260bc700 1 --2- 172.21.15.3:0/1247272986 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200612f0 0x7fb220143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2248b9700 1 -- 172.21.15.3:0/1247272986 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930756335 0 0) 0x7fb2200671e0 con 0x7fb2200b7840 2022-07-21T01:33:44.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2248b9700 1 -- 172.21.15.3:0/1247272986 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb208003260 con 0x7fb2200b7840 2022-07-21T01:33:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2248b9700 1 -- 172.21.15.3:0/1247272986 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466115478 0 0) 0x7fb220065540 con 0x7fb2200b2930 2022-07-21T01:33:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2248b9700 1 -- 172.21.15.3:0/1247272986 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2200671e0 con 0x7fb2200b2930 2022-07-21T01:33:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2260bc700 1 -- 172.21.15.3:0/1247272986 >> v1:172.21.15.143:6789/0 conn(0x7fb2200b2930 legacy=0x7fb220143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2260bc700 1 -- 172.21.15.3:0/1247272986 >> v1:172.21.15.3:6789/0 conn(0x7fb2200b7840 legacy=0x7fb220060be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2260bc700 1 -- 172.21.15.3:0/1247272986 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb220066ce0 con 0x7fb2200612f0 2022-07-21T01:33:44.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2260bc700 1 --2- 172.21.15.3:0/1247272986 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200612f0 0x7fb220143b10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb2180032f0 tx=0x7fb2180091d0).ready entity=mon.2 client_cookie=7635f85049acba45 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:44.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2248b9700 1 -- 172.21.15.3:0/1247272986 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb21800b640 con 0x7fb2200612f0 2022-07-21T01:33:44.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2248b9700 1 -- 172.21.15.3:0/1247272986 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21800b7a0 con 0x7fb2200612f0 2022-07-21T01:33:44.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2248b9700 1 -- 172.21.15.3:0/1247272986 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb21800b910 con 0x7fb2200612f0 2022-07-21T01:33:44.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 -- 172.21.15.3:0/1247272986 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200612f0 msgr2=0x7fb220143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 --2- 172.21.15.3:0/1247272986 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200612f0 0x7fb220143b10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb2180032f0 tx=0x7fb2180091d0).stop 2022-07-21T01:33:44.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 -- 172.21.15.3:0/1247272986 shutdown_connections 2022-07-21T01:33:44.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 --2- 172.21.15.3:0/1247272986 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200612f0 0x7fb220143b10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 -- 172.21.15.3:0/1247272986 >> 172.21.15.3:0/1247272986 conn(0x7fb22005f840 msgr2=0x7fb22005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:44.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 -- 172.21.15.3:0/1247272986 shutdown_connections 2022-07-21T01:33:44.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.929+0000 7fb2268bd700 1 -- 172.21.15.3:0/1247272986 wait complete. 2022-07-21T01:33:44.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2268bd700 1 Processor -- start 2022-07-21T01:33:44.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2268bd700 1 -- start start 2022-07-21T01:33:44.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2268bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2200612f0 0x7fb220148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2268bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2200b2930 0x7fb220141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2268bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200b7840 0x7fb220141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2268bd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22013f840 con 0x7fb2200612f0 2022-07-21T01:33:44.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2268bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb220140070 con 0x7fb2200b2930 2022-07-21T01:33:44.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2268bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb220140c70 con 0x7fb2200b7840 2022-07-21T01:33:44.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2258bb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2200612f0 0x7fb220148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2258bb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2200612f0 0x7fb220148d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43340/0 (socket says 172.21.15.3:43340) 2022-07-21T01:33:44.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.933+0000 7fb2258bb700 1 -- 172.21.15.3:0/1971517207 learned_addr learned my addr 172.21.15.3:0/1971517207 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:44.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2250ba700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2200b2930 0x7fb220141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2260bc700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200b7840 0x7fb220141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2258bb700 1 -- 172.21.15.3:0/1971517207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200b7840 msgr2=0x7fb220141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2258bb700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200b7840 0x7fb220141520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2258bb700 1 -- 172.21.15.3:0/1971517207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2200b2930 msgr2=0x7fb220141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:44.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2258bb700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2200b2930 0x7fb220141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:44.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2258bb700 1 -- 172.21.15.3:0/1971517207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb218009470 con 0x7fb2200612f0 2022-07-21T01:33:44.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2258bb700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2200612f0 0x7fb220148d00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb214001290 tx=0x7fb2140028f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:44.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb212ffd700 1 -- 172.21.15.3:0/1971517207 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2140070d0 con 0x7fb2200612f0 2022-07-21T01:33:44.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb212ffd700 1 -- 172.21.15.3:0/1971517207 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb214007230 con 0x7fb2200612f0 2022-07-21T01:33:44.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22013fa70 con 0x7fb2200612f0 2022-07-21T01:33:44.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb212ffd700 1 -- 172.21.15.3:0/1971517207 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2140073a0 con 0x7fb2200612f0 2022-07-21T01:33:44.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22005d0c0 con 0x7fb2200612f0 2022-07-21T01:33:44.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.937+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2200574e0 con 0x7fb2200612f0 2022-07-21T01:33:44.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.941+0000 7fb212ffd700 1 -- 172.21.15.3:0/1971517207 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb214010070 con 0x7fb2200612f0 2022-07-21T01:33:44.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.941+0000 7fb212ffd700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb2000539e0 0x7fb200055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:44.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.941+0000 7fb212ffd700 1 -- 172.21.15.3:0/1971517207 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb21406b710 con 0x7fb2200612f0 2022-07-21T01:33:44.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.941+0000 7fb2250ba700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb2000539e0 0x7fb200055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:44.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.941+0000 7fb2250ba700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb2000539e0 0x7fb200055ea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb21c00d000 tx=0x7fb21c007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:44.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:44.945+0000 7fb212ffd700 1 -- 172.21.15.3:0/1971517207 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb214042430 con 0x7fb2200612f0 2022-07-21T01:33:45.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.073+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb22005d460 con 0x7fb2200612f0 2022-07-21T01:33:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.073+0000 7fb212ffd700 1 -- 172.21.15.3:0/1971517207 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb214045a50 con 0x7fb2200612f0 2022-07-21T01:33:45.073 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb2000539e0 msgr2=0x7fb200055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb2000539e0 0x7fb200055ea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb21c00d000 tx=0x7fb21c007a20).stop 2022-07-21T01:33:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2200612f0 msgr2=0x7fb220148d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2200612f0 0x7fb220148d00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb214001290 tx=0x7fb2140028f0).stop 2022-07-21T01:33:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 shutdown_connections 2022-07-21T01:33:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2200b7840 0x7fb220141520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb2000539e0 0x7fb200055ea0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2200b2930 0x7fb220141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 --2- 172.21.15.3:0/1971517207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2200612f0 0x7fb220148d00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 >> 172.21.15.3:0/1971517207 conn(0x7fb22005f840 msgr2=0x7fb2200b6410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 shutdown_connections 2022-07-21T01:33:45.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.077+0000 7fb2268bd700 1 -- 172.21.15.3:0/1971517207 wait complete. 2022-07-21T01:33:45.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:33:45.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.205+0000 7fcda5346700 1 Processor -- start 2022-07-21T01:33:45.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.205+0000 7fcda5346700 1 -- start start 2022-07-21T01:33:45.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcda5346700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda00619b0 0x7fcda00b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcda5346700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda0061d90 con 0x7fcda00b6ee0 2022-07-21T01:33:45.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcda5346700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00ba5a0 con 0x7fcda00b5ed0 2022-07-21T01:33:45.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcda5346700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda0143fa0 con 0x7fcda00619b0 2022-07-21T01:33:45.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd97fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcda00b6ee0 0x7fcda0143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51628/0 (socket says 172.21.15.3:51628) 2022-07-21T01:33:45.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd97fff700 1 -- 172.21.15.3:0/1085643373 learned_addr learned my addr 172.21.15.3:0/1085643373 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:45.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9ffff700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda00619b0 0x7fcda00b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903013397 0 0) 0x7fcda0061d90 con 0x7fcda00b6ee0 2022-07-21T01:33:45.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd80003260 con 0x7fcda00b6ee0 2022-07-21T01:33:45.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389702162 0 0) 0x7fcda00ba5a0 con 0x7fcda00b5ed0 2022-07-21T01:33:45.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda0061d90 con 0x7fcda00b5ed0 2022-07-21T01:33:45.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3943520407 0 0) 0x7fcd80003260 con 0x7fcda00b6ee0 2022-07-21T01:33:45.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda00619b0 msgr2=0x7fcda00b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:45.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda00619b0 0x7fcda00b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 >> v1:172.21.15.143:6789/0 conn(0x7fcda00b5ed0 legacy=0x7fcda00b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda0143150 con 0x7fcda00b6ee0 2022-07-21T01:33:45.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fcd88001680 con 0x7fcda00b6ee0 2022-07-21T01:33:45.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 >> v1:172.21.15.3:6789/0 conn(0x7fcda00b6ee0 legacy=0x7fcda0143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda00619b0 0x7fcd80003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd80003e20 0x7fcd80004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd800049f0 0x7fcd80004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda0143fa0 con 0x7fcd80003e20 2022-07-21T01:33:45.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8000b800 con 0x7fcd800049f0 2022-07-21T01:33:45.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8000b9c0 con 0x7fcda00619b0 2022-07-21T01:33:45.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd97fff700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd80003e20 0x7fcd80004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9ffff700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda00619b0 0x7fcd80003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcda4b45700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd800049f0 0x7fcd80004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd97fff700 1 -- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda00619b0 msgr2=0x7fcd80003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:45.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd97fff700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda00619b0 0x7fcd80003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd97fff700 1 -- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd800049f0 msgr2=0x7fcd80004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:45.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd97fff700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd800049f0 0x7fcd80004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd97fff700 1 -- 172.21.15.3:0/1085643373 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd80003260 con 0x7fcd80003e20 2022-07-21T01:33:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd97fff700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd80003e20 0x7fcd80004500 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fcd88001450 tx=0x7fcda00b2040).ready entity=mon.0 client_cookie=90341ba559380dcd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd88007170 con 0x7fcd80003e20 2022-07-21T01:33:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd880072d0 con 0x7fcd80003e20 2022-07-21T01:33:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.209+0000 7fcd9f7fe700 1 -- 172.21.15.3:0/1085643373 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8800c8c0 con 0x7fcd80003e20 2022-07-21T01:33:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.213+0000 7fcda5346700 1 -- 172.21.15.3:0/1085643373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd80003e20 msgr2=0x7fcd80004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.213+0000 7fcda5346700 1 --2- 172.21.15.3:0/1085643373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd80003e20 0x7fcd80004500 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fcd88001450 tx=0x7fcda00b2040).stop 2022-07-21T01:33:45.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.213+0000 7fcda4b45700 1 -- 172.21.15.3:0/1085643373 reap_dead start 2022-07-21T01:33:45.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.213+0000 7fcda5346700 1 -- 172.21.15.3:0/1085643373 shutdown_connections 2022-07-21T01:33:45.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.213+0000 7fcda5346700 1 -- 172.21.15.3:0/1085643373 >> 172.21.15.3:0/1085643373 conn(0x7fcda006af80 msgr2=0x7fcda005fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:45.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.213+0000 7fcda5346700 1 -- 172.21.15.3:0/1085643373 shutdown_connections 2022-07-21T01:33:45.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.213+0000 7fcda5346700 1 -- 172.21.15.3:0/1085643373 wait complete. 2022-07-21T01:33:45.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.213+0000 7fcda5346700 1 Processor -- start 2022-07-21T01:33:45.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.213+0000 7fcda5346700 1 -- start start 2022-07-21T01:33:45.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda5346700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd80003e20 0x7fcda014a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda5346700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcda00afc30 0x7fcda00aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda5346700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda00af450 0x7fcda01400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda5346700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda01559d0 con 0x7fcda00afc30 2022-07-21T01:33:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda5346700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda00ad990 con 0x7fcda00af450 2022-07-21T01:33:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda5346700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda00ae2f0 con 0x7fcd80003e20 2022-07-21T01:33:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd97fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcda00afc30 0x7fcda00aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd97fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcda00afc30 0x7fcda00aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43362/0 (socket says 172.21.15.3:43362) 2022-07-21T01:33:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd97fff700 1 -- 172.21.15.3:0/2433865614 learned_addr learned my addr 172.21.15.3:0/2433865614 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda4b45700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda00af450 0x7fcda01400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd9ffff700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd80003e20 0x7fcda014a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd97fff700 1 -- 172.21.15.3:0/2433865614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd80003e20 msgr2=0x7fcda014a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd97fff700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd80003e20 0x7fcda014a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd97fff700 1 -- 172.21.15.3:0/2433865614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda00af450 msgr2=0x7fcda01400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd97fff700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda00af450 0x7fcda01400c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd97fff700 1 -- 172.21.15.3:0/2433865614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda00ae590 con 0x7fcda00afc30 2022-07-21T01:33:45.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda4b45700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda00af450 0x7fcda01400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:45.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd97fff700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcda00afc30 0x7fcda00aef60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fcd88001450 tx=0x7fcd88015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:45.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd9dffb700 1 -- 172.21.15.3:0/2433865614 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd88004730 con 0x7fcda00afc30 2022-07-21T01:33:45.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd9dffb700 1 -- 172.21.15.3:0/2433865614 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd88018070 con 0x7fcda00afc30 2022-07-21T01:33:45.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda0113b10 con 0x7fcda00afc30 2022-07-21T01:33:45.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd9dffb700 1 -- 172.21.15.3:0/2433865614 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd88004a80 con 0x7fcda00afc30 2022-07-21T01:33:45.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda00af8d0 con 0x7fcda00afc30 2022-07-21T01:33:45.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda00574e0 con 0x7fcda00afc30 2022-07-21T01:33:45.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.217+0000 7fcd9dffb700 1 -- 172.21.15.3:0/2433865614 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd8801d070 con 0x7fcda00afc30 2022-07-21T01:33:45.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.221+0000 7fcd9dffb700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd84053a70 0x7fcd84055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.221+0000 7fcd9ffff700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd84053a70 0x7fcd84055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.221+0000 7fcd9dffb700 1 -- 172.21.15.3:0/2433865614 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcd8806d170 con 0x7fcda00afc30 2022-07-21T01:33:45.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.221+0000 7fcd9ffff700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd84053a70 0x7fcd84055f30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fcd9000b000 tx=0x7fcd90008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:45.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.225+0000 7fcd9dffb700 1 -- 172.21.15.3:0/2433865614 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd88043f30 con 0x7fcda00afc30 2022-07-21T01:33:45.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.353+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcda0060070 con 0x7fcda00afc30 2022-07-21T01:33:45.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.353+0000 7fcd9dffb700 1 -- 172.21.15.3:0/2433865614 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcd8801a030 con 0x7fcda00afc30 2022-07-21T01:33:45.353 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:45.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd84053a70 msgr2=0x7fcd84055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd84053a70 0x7fcd84055f30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fcd9000b000 tx=0x7fcd90008040).stop 2022-07-21T01:33:45.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcda00afc30 msgr2=0x7fcda00aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcda00afc30 0x7fcda00aef60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fcd88001450 tx=0x7fcd88015000).stop 2022-07-21T01:33:45.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 shutdown_connections 2022-07-21T01:33:45.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd84053a70 0x7fcd84055f30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda00af450 0x7fcda01400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcda00afc30 0x7fcda00aef60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 --2- 172.21.15.3:0/2433865614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd80003e20 0x7fcda014a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 >> 172.21.15.3:0/2433865614 conn(0x7fcda006af80 msgr2=0x7fcda0147670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:45.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 shutdown_connections 2022-07-21T01:33:45.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.357+0000 7fcda5346700 1 -- 172.21.15.3:0/2433865614 wait complete. 2022-07-21T01:33:45.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:33:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.477+0000 7f82556dd700 1 Processor -- start 2022-07-21T01:33:45.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.477+0000 7f82556dd700 1 -- start start 2022-07-21T01:33:45.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.477+0000 7f82556dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 0x7f8250061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.477+0000 7f82556dd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500aee80 con 0x7f82500614b0 2022-07-21T01:33:45.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.477+0000 7f82556dd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500b70b0 con 0x7f82501405d0 2022-07-21T01:33:45.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.477+0000 7f82556dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82500af0b0 con 0x7f82500ad750 2022-07-21T01:33:45.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.477+0000 7f824ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f82500614b0 0x7f82500b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51648/0 (socket says 172.21.15.3:51648) 2022-07-21T01:33:45.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.477+0000 7f824ffff700 1 -- 172.21.15.3:0/545751140 learned_addr learned my addr 172.21.15.3:0/545751140 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:45.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.477+0000 7f824f7fe700 1 --2- 172.21.15.3:0/545751140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 0x7f8250061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824effd700 1 -- 172.21.15.3:0/545751140 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180795109 0 0) 0x7f82500aee80 con 0x7f82500614b0 2022-07-21T01:33:45.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824effd700 1 -- 172.21.15.3:0/545751140 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f823c003260 con 0x7f82500614b0 2022-07-21T01:33:45.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824effd700 1 -- 172.21.15.3:0/545751140 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7307306 0 0) 0x7f82500b70b0 con 0x7f82501405d0 2022-07-21T01:33:45.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824f7fe700 1 -- 172.21.15.3:0/545751140 >> v1:172.21.15.143:6789/0 conn(0x7f82501405d0 legacy=0x7f82501466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824f7fe700 1 -- 172.21.15.3:0/545751140 >> v1:172.21.15.3:6789/0 conn(0x7f82500614b0 legacy=0x7f82500b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824f7fe700 1 -- 172.21.15.3:0/545751140 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82500af350 con 0x7f82500ad750 2022-07-21T01:33:45.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824f7fe700 1 --2- 172.21.15.3:0/545751140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 0x7f8250061890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8244002f70 tx=0x7f8244006df0).ready entity=mon.2 client_cookie=f5d4b7f0e9a14cf6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:45.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824effd700 1 -- 172.21.15.3:0/545751140 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f824400d570 con 0x7f82500ad750 2022-07-21T01:33:45.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824effd700 1 -- 172.21.15.3:0/545751140 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8244000e90 con 0x7f82500ad750 2022-07-21T01:33:45.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f824effd700 1 -- 172.21.15.3:0/545751140 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f824400d8c0 con 0x7f82500ad750 2022-07-21T01:33:45.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f82556dd700 1 -- 172.21.15.3:0/545751140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 msgr2=0x7f8250061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f82556dd700 1 --2- 172.21.15.3:0/545751140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 0x7f8250061890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8244002f70 tx=0x7f8244006df0).stop 2022-07-21T01:33:45.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f82556dd700 1 -- 172.21.15.3:0/545751140 shutdown_connections 2022-07-21T01:33:45.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f82556dd700 1 --2- 172.21.15.3:0/545751140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 0x7f8250061890 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f82556dd700 1 -- 172.21.15.3:0/545751140 >> 172.21.15.3:0/545751140 conn(0x7f8250066f70 msgr2=0x7f8250067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:45.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f82556dd700 1 -- 172.21.15.3:0/545751140 shutdown_connections 2022-07-21T01:33:45.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f82556dd700 1 -- 172.21.15.3:0/545751140 wait complete. 2022-07-21T01:33:45.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f82556dd700 1 Processor -- start 2022-07-21T01:33:45.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.481+0000 7f82556dd700 1 -- start start 2022-07-21T01:33:45.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f82556dd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82500614b0 0x7f8250142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f82556dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 0x7f82500b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f82556dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82501405d0 0x7f82500b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f82556dd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82500b2620 con 0x7f82500614b0 2022-07-21T01:33:45.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f82556dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82500b3fc0 con 0x7f82501405d0 2022-07-21T01:33:45.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f82556dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82500b0fa0 con 0x7f82500ad750 2022-07-21T01:33:45.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82500614b0 0x7f8250142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82500614b0 0x7f8250142630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43376/0 (socket says 172.21.15.3:43376) 2022-07-21T01:33:45.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824ffff700 1 -- 172.21.15.3:0/3149920529 learned_addr learned my addr 172.21.15.3:0/3149920529 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:45.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824f7fe700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 0x7f82500b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f8254edc700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82501405d0 0x7f82500b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824ffff700 1 -- 172.21.15.3:0/3149920529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 msgr2=0x7f82500b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824ffff700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 0x7f82500b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824ffff700 1 -- 172.21.15.3:0/3149920529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82501405d0 msgr2=0x7f82500b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824ffff700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82501405d0 0x7f82500b2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824ffff700 1 -- 172.21.15.3:0/3149920529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824400b180 con 0x7f82500614b0 2022-07-21T01:33:45.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824ffff700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82500614b0 0x7f8250142630 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8240001110 tx=0x7f8240007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:45.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824d7fa700 1 -- 172.21.15.3:0/3149920529 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f824000ce80 con 0x7f82500614b0 2022-07-21T01:33:45.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824d7fa700 1 -- 172.21.15.3:0/3149920529 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82400070d0 con 0x7f82500614b0 2022-07-21T01:33:45.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f824d7fa700 1 -- 172.21.15.3:0/3149920529 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8240007430 con 0x7f82500614b0 2022-07-21T01:33:45.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82500b0210 con 0x7f82500614b0 2022-07-21T01:33:45.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82500b0750 con 0x7f82500614b0 2022-07-21T01:33:45.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.485+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82500af5f0 con 0x7f82500614b0 2022-07-21T01:33:45.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.489+0000 7f824d7fa700 1 -- 172.21.15.3:0/3149920529 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8240010070 con 0x7f82500614b0 2022-07-21T01:33:45.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.489+0000 7f824d7fa700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8230053a70 0x7f8230055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.489+0000 7f824d7fa700 1 -- 172.21.15.3:0/3149920529 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f824006b710 con 0x7f82500614b0 2022-07-21T01:33:45.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.489+0000 7f824f7fe700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8230053a70 0x7f8230055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.489+0000 7f824f7fe700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8230053a70 0x7f8230055f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f824401a000 tx=0x7f8244014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:45.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.493+0000 7f824d7fa700 1 -- 172.21.15.3:0/3149920529 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f82400423c0 con 0x7f82500614b0 2022-07-21T01:33:45.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.625+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f82500b1a40 con 0x7f82500614b0 2022-07-21T01:33:45.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.625+0000 7f824d7fa700 1 -- 172.21.15.3:0/3149920529 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8240018070 con 0x7f82500614b0 2022-07-21T01:33:45.624 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:45.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8230053a70 msgr2=0x7f8230055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8230053a70 0x7f8230055f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f824401a000 tx=0x7f8244014000).stop 2022-07-21T01:33:45.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82500614b0 msgr2=0x7f8250142630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82500614b0 0x7f8250142630 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8240001110 tx=0x7f8240007aa0).stop 2022-07-21T01:33:45.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 shutdown_connections 2022-07-21T01:33:45.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8230053a70 0x7f8230055f30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82501405d0 0x7f82500b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82500ad750 0x7f82500b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 --2- 172.21.15.3:0/3149920529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82500614b0 0x7f8250142630 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 >> 172.21.15.3:0/3149920529 conn(0x7f8250066f70 msgr2=0x7f8250146880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:45.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 shutdown_connections 2022-07-21T01:33:45.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.629+0000 7f82556dd700 1 -- 172.21.15.3:0/3149920529 wait complete. 2022-07-21T01:33:45.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:45.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24d0b9e700 1 Processor -- start 2022-07-21T01:33:45.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24d0b9e700 1 -- start start 2022-07-21T01:33:45.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24d0b9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc066780 0x7f24cc066b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24d0b9e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f24cc0b1bc0 0x7f24cc05ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24d0b9e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24cc05f8e0 con 0x7f24cc140e40 2022-07-21T01:33:45.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24d0b9e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24cc0b1630 con 0x7f24cc0b1bc0 2022-07-21T01:33:45.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24d0b9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc05fb10 con 0x7f24cc066780 2022-07-21T01:33:45.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24c3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f24cc0b1bc0 0x7f24cc05ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24cbfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f24cc140e40 0x7f24cc13e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51666/0 (socket says 172.21.15.3:51666) 2022-07-21T01:33:45.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.749+0000 7f24cbfff700 1 -- 172.21.15.3:0/1285323165 learned_addr learned my addr 172.21.15.3:0/1285323165 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:45.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24cb7fe700 1 --2- 172.21.15.3:0/1285323165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc066780 0x7f24cc066b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24c3fff700 1 -- 172.21.15.3:0/1285323165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc066780 msgr2=0x7f24cc066b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:45.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24c3fff700 1 --2- 172.21.15.3:0/1285323165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc066780 0x7f24cc066b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24c3fff700 1 -- 172.21.15.3:0/1285323165 >> v1:172.21.15.3:6789/0 conn(0x7f24cc140e40 legacy=0x7f24cc13e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24caffd700 1 -- 172.21.15.3:0/1285323165 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886872318 0 0) 0x7f24cc05f8e0 con 0x7f24cc140e40 2022-07-21T01:33:45.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24c3fff700 1 -- 172.21.15.3:0/1285323165 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc0604a0 con 0x7f24cc0b1bc0 2022-07-21T01:33:45.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24c3fff700 1 --2- 172.21.15.3:0/1285323165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc0b1bc0 0x7f24cc05ffb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f24b4002f70 tx=0x7f24b4006df0).ready entity=mon.0 client_cookie=d276b1589312f574 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:45.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24caffd700 1 -- 172.21.15.3:0/1285323165 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24b400d570 con 0x7f24cc0b1bc0 2022-07-21T01:33:45.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24caffd700 1 -- 172.21.15.3:0/1285323165 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b4000e90 con 0x7f24cc0b1bc0 2022-07-21T01:33:45.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24caffd700 1 -- 172.21.15.3:0/1285323165 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24b400d8c0 con 0x7f24cc0b1bc0 2022-07-21T01:33:45.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 -- 172.21.15.3:0/1285323165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc0b1bc0 msgr2=0x7f24cc05ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 --2- 172.21.15.3:0/1285323165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc0b1bc0 0x7f24cc05ffb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f24b4002f70 tx=0x7f24b4006df0).stop 2022-07-21T01:33:45.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 -- 172.21.15.3:0/1285323165 shutdown_connections 2022-07-21T01:33:45.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 --2- 172.21.15.3:0/1285323165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc0b1bc0 0x7f24cc05ffb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 --2- 172.21.15.3:0/1285323165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc066780 0x7f24cc066b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 -- 172.21.15.3:0/1285323165 >> 172.21.15.3:0/1285323165 conn(0x7f24cc063870 msgr2=0x7f24cc063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:45.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 -- 172.21.15.3:0/1285323165 shutdown_connections 2022-07-21T01:33:45.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 -- 172.21.15.3:0/1285323165 wait complete. 2022-07-21T01:33:45.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 Processor -- start 2022-07-21T01:33:45.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.753+0000 7f24d0b9e700 1 -- start start 2022-07-21T01:33:45.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24d0b9e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24cc066780 0x7f24cc1537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24d0b9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc0b1bc0 0x7f24cc150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24d0b9e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc140e40 0x7f24cc0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24d0b9e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc150210 con 0x7f24cc140e40 2022-07-21T01:33:45.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24d0b9e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc153cb0 con 0x7f24cc066780 2022-07-21T01:33:45.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24d0b9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc1426a0 con 0x7f24cc0b1bc0 2022-07-21T01:33:45.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc140e40 0x7f24cc0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc140e40 0x7f24cc0b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43392/0 (socket says 172.21.15.3:43392) 2022-07-21T01:33:45.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cbfff700 1 -- 172.21.15.3:0/2878002680 learned_addr learned my addr 172.21.15.3:0/2878002680 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:45.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24c3fff700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc0b1bc0 0x7f24cc150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cb7fe700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24cc066780 0x7f24cc1537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cbfff700 1 -- 172.21.15.3:0/2878002680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc0b1bc0 msgr2=0x7f24cc150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cbfff700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc0b1bc0 0x7f24cc150b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cbfff700 1 -- 172.21.15.3:0/2878002680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24cc066780 msgr2=0x7f24cc1537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cbfff700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24cc066780 0x7f24cc1537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cbfff700 1 -- 172.21.15.3:0/2878002680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b400b180 con 0x7f24cc140e40 2022-07-21T01:33:45.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24cbfff700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc140e40 0x7f24cc0b2fb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f24c4001250 tx=0x7f24c4001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:45.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24c97fa700 1 -- 172.21.15.3:0/2878002680 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24c4005ba0 con 0x7f24cc140e40 2022-07-21T01:33:45.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24c97fa700 1 -- 172.21.15.3:0/2878002680 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c4005d00 con 0x7f24cc140e40 2022-07-21T01:33:45.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24cc150400 con 0x7f24cc140e40 2022-07-21T01:33:45.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24c97fa700 1 -- 172.21.15.3:0/2878002680 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24c4004a90 con 0x7f24cc140e40 2022-07-21T01:33:45.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24cc0b34a0 con 0x7f24cc140e40 2022-07-21T01:33:45.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc0b18d0 con 0x7f24cc140e40 2022-07-21T01:33:45.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.757+0000 7f24c97fa700 1 -- 172.21.15.3:0/2878002680 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f24c4004bf0 con 0x7f24cc140e40 2022-07-21T01:33:45.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.761+0000 7f24c97fa700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f24a4053b00 0x7f24a4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:45.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.761+0000 7f24c97fa700 1 -- 172.21.15.3:0/2878002680 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f24c406bba0 con 0x7f24cc140e40 2022-07-21T01:33:45.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.761+0000 7f24cb7fe700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f24a4053b00 0x7f24a4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:45.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.761+0000 7f24cb7fe700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f24a4053b00 0x7f24a4055fc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f24bc00c000 tx=0x7f24bc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:45.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.765+0000 7f24c97fa700 1 -- 172.21.15.3:0/2878002680 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f24c40428c0 con 0x7f24cc140e40 2022-07-21T01:33:45.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.893+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24cc0574e0 con 0x7f24cc140e40 2022-07-21T01:33:45.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.893+0000 7f24c97fa700 1 -- 172.21.15.3:0/2878002680 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7f24c4045ee0 con 0x7f24cc140e40 2022-07-21T01:33:45.892 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:45.893 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.893+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f24a4053b00 msgr2=0x7f24a4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.893+0000 7f24d0b9e700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f24a4053b00 0x7f24a4055fc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f24bc00c000 tx=0x7f24bc008040).stop 2022-07-21T01:33:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.893+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc140e40 msgr2=0x7f24cc0b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.893+0000 7f24d0b9e700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc140e40 0x7f24cc0b2fb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f24c4001250 tx=0x7f24c4001360).stop 2022-07-21T01:33:45.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.897+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 shutdown_connections 2022-07-21T01:33:45.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.897+0000 7f24d0b9e700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc140e40 0x7f24cc0b2fb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.897+0000 7f24d0b9e700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24cc0b1bc0 0x7f24cc150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.897+0000 7f24d0b9e700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f24a4053b00 0x7f24a4055fc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.897+0000 7f24d0b9e700 1 --2- 172.21.15.3:0/2878002680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24cc066780 0x7f24cc1537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:45.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.897+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 >> 172.21.15.3:0/2878002680 conn(0x7f24cc063870 msgr2=0x7f24cc063c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:45.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.897+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 shutdown_connections 2022-07-21T01:33:45.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:45.897+0000 7f24d0b9e700 1 -- 172.21.15.3:0/2878002680 wait complete. 2022-07-21T01:33:45.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:33:21.299936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:33:24.075599+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:33:45.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:33:46.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.021+0000 7fb813fff700 1 Processor -- start 2022-07-21T01:33:46.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.021+0000 7fb813fff700 1 -- start start 2022-07-21T01:33:46.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.021+0000 7fb813fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb8140aaf10 0x7fb814067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.021+0000 7fb813fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140b4ee0 0x7fb8140b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.021+0000 7fb813fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb814065af0 con 0x7fb8140ab2f0 2022-07-21T01:33:46.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.021+0000 7fb813fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8140650a0 con 0x7fb8140aaf10 2022-07-21T01:33:46.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.021+0000 7fb813fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140b3e70 con 0x7fb8140b4ee0 2022-07-21T01:33:46.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.021+0000 7fb8127fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb8140ab2f0 0x7fb814064990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51678/0 (socket says 172.21.15.3:51678) 2022-07-21T01:33:46.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.021+0000 7fb8127fc700 1 -- 172.21.15.3:0/1278164487 learned_addr learned my addr 172.21.15.3:0/1278164487 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:46.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb812ffd700 1 --2- 172.21.15.3:0/1278164487 >> v2:172.21.15.143:3300/0 conn(0x7fb8140aaf10 0x7fb814067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb8137fe700 1 --2- 172.21.15.3:0/1278164487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140b4ee0 0x7fb8140b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb811ffb700 1 -- 172.21.15.3:0/1278164487 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14697252 0 0) 0x7fb814065af0 con 0x7fb8140ab2f0 2022-07-21T01:33:46.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb811ffb700 1 -- 172.21.15.3:0/1278164487 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f8003260 con 0x7fb8140ab2f0 2022-07-21T01:33:46.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb812ffd700 1 -- 172.21.15.3:0/1278164487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140b4ee0 msgr2=0x7fb8140b76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:46.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb812ffd700 1 --2- 172.21.15.3:0/1278164487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140b4ee0 0x7fb8140b76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb812ffd700 1 -- 172.21.15.3:0/1278164487 >> v1:172.21.15.3:6789/0 conn(0x7fb8140ab2f0 legacy=0x7fb814064990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb812ffd700 1 -- 172.21.15.3:0/1278164487 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb814066ff0 con 0x7fb8140aaf10 2022-07-21T01:33:46.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb812ffd700 1 --2- 172.21.15.3:0/1278164487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8140aaf10 0x7fb814067490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb808002f70 tx=0x7fb808006df0).ready entity=mon.1 client_cookie=719e6e8b6a5f139d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb811ffb700 1 -- 172.21.15.3:0/1278164487 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb80800d4a0 con 0x7fb8140aaf10 2022-07-21T01:33:46.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb811ffb700 1 -- 172.21.15.3:0/1278164487 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb808000e90 con 0x7fb8140aaf10 2022-07-21T01:33:46.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb811ffb700 1 -- 172.21.15.3:0/1278164487 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb80800d7f0 con 0x7fb8140aaf10 2022-07-21T01:33:46.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb813fff700 1 -- 172.21.15.3:0/1278164487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8140aaf10 msgr2=0x7fb814067490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb813fff700 1 --2- 172.21.15.3:0/1278164487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8140aaf10 0x7fb814067490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb808002f70 tx=0x7fb808006df0).stop 2022-07-21T01:33:46.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb813fff700 1 -- 172.21.15.3:0/1278164487 shutdown_connections 2022-07-21T01:33:46.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb813fff700 1 --2- 172.21.15.3:0/1278164487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140b4ee0 0x7fb8140b76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb813fff700 1 --2- 172.21.15.3:0/1278164487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8140aaf10 0x7fb814067490 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb813fff700 1 -- 172.21.15.3:0/1278164487 >> 172.21.15.3:0/1278164487 conn(0x7fb814062d70 msgr2=0x7fb81405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:46.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb813fff700 1 -- 172.21.15.3:0/1278164487 shutdown_connections 2022-07-21T01:33:46.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.025+0000 7fb813fff700 1 -- 172.21.15.3:0/1278164487 wait complete. 2022-07-21T01:33:46.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb813fff700 1 Processor -- start 2022-07-21T01:33:46.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb813fff700 1 -- start start 2022-07-21T01:33:46.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb813fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8140aaf10 0x7fb8141415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb813fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140ab2f0 0x7fb81413b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb813fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8140b4ee0 0x7fb8140ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb813fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140ac0b0 con 0x7fb8140b4ee0 2022-07-21T01:33:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb813fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb814141ae0 con 0x7fb8140aaf10 2022-07-21T01:33:46.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb813fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81413f630 con 0x7fb8140ab2f0 2022-07-21T01:33:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb8137fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8140b4ee0 0x7fb8140ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb8137fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8140b4ee0 0x7fb8140ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43406/0 (socket says 172.21.15.3:43406) 2022-07-21T01:33:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb8137fe700 1 -- 172.21.15.3:0/4226165837 learned_addr learned my addr 172.21.15.3:0/4226165837 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb812ffd700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8140aaf10 0x7fb8141415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.029+0000 7fb8127fc700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140ab2f0 0x7fb81413b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb8137fe700 1 -- 172.21.15.3:0/4226165837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140ab2f0 msgr2=0x7fb81413b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb8137fe700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140ab2f0 0x7fb81413b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb8137fe700 1 -- 172.21.15.3:0/4226165837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8140aaf10 msgr2=0x7fb8141415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb8137fe700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8140aaf10 0x7fb8141415f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb8137fe700 1 -- 172.21.15.3:0/4226165837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80800b180 con 0x7fb8140b4ee0 2022-07-21T01:33:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb8137fe700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8140b4ee0 0x7fb8140ac8c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb8140b7c80 tx=0x7fb80c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb803fff700 1 -- 172.21.15.3:0/4226165837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb80c012600 con 0x7fb8140b4ee0 2022-07-21T01:33:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb803fff700 1 -- 172.21.15.3:0/4226165837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80c012760 con 0x7fb8140b4ee0 2022-07-21T01:33:46.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81413f890 con 0x7fb8140b4ee0 2022-07-21T01:33:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb803fff700 1 -- 172.21.15.3:0/4226165837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb80c012ac0 con 0x7fb8140b4ee0 2022-07-21T01:33:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81413b9d0 con 0x7fb8140b4ee0 2022-07-21T01:33:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb814140170 con 0x7fb8140b4ee0 2022-07-21T01:33:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb803fff700 1 -- 172.21.15.3:0/4226165837 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb80c022070 con 0x7fb8140b4ee0 2022-07-21T01:33:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.033+0000 7fb803fff700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7f0053aa0 0x7fb7f0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.037+0000 7fb803fff700 1 -- 172.21.15.3:0/4226165837 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb80c06c600 con 0x7fb8140b4ee0 2022-07-21T01:33:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.037+0000 7fb812ffd700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7f0053aa0 0x7fb7f0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.037+0000 7fb812ffd700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7f0053aa0 0x7fb7f0055f60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb80801a000 tx=0x7fb808014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.041+0000 7fb803fff700 1 -- 172.21.15.3:0/4226165837 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb80c043250 con 0x7fb8140b4ee0 2022-07-21T01:33:46.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.165+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb8140574e0 con 0x7fb8140b4ee0 2022-07-21T01:33:46.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.165+0000 7fb803fff700 1 -- 172.21.15.3:0/4226165837 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v26) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb80c0160b0 con 0x7fb8140b4ee0 2022-07-21T01:33:46.162 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:33:46.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7f0053aa0 msgr2=0x7fb7f0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7f0053aa0 0x7fb7f0055f60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb80801a000 tx=0x7fb808014000).stop 2022-07-21T01:33:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8140b4ee0 msgr2=0x7fb8140ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8140b4ee0 0x7fb8140ac8c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb8140b7c80 tx=0x7fb80c0091d0).stop 2022-07-21T01:33:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 shutdown_connections 2022-07-21T01:33:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8140b4ee0 0x7fb8140ac8c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8140ab2f0 0x7fb81413b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb7f0053aa0 0x7fb7f0055f60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 --2- 172.21.15.3:0/4226165837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8140aaf10 0x7fb8141415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 >> 172.21.15.3:0/4226165837 conn(0x7fb814062d70 msgr2=0x7fb81413d400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 shutdown_connections 2022-07-21T01:33:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.169+0000 7fb813fff700 1 -- 172.21.15.3:0/4226165837 wait complete. 2022-07-21T01:33:46.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:33:46.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 Processor -- start 2022-07-21T01:33:46.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 -- start start 2022-07-21T01:33:46.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6ac00b22b0 0x7f6ac00b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6ac00ad1c0 0x7f6ac0143d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 0x7f6ac00b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac00ae650 con 0x7f6ac00b22b0 2022-07-21T01:33:46.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac00ae880 con 0x7f6ac00ad1c0 2022-07-21T01:33:46.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac00aeb20 con 0x7f6ac0144230 2022-07-21T01:33:46.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac4d69700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6ac00b22b0 0x7f6ac00b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac4d69700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6ac00b22b0 0x7f6ac00b0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43410/0 (socket says 172.21.15.3:43410) 2022-07-21T01:33:46.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac4d69700 1 -- 172.21.15.3:0/2709823566 learned_addr learned my addr 172.21.15.3:0/2709823566 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:46.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ab7fff700 1 --2- 172.21.15.3:0/2709823566 >> v2:172.21.15.143:3300/0 conn(0x7f6ac00ad1c0 0x7f6ac0143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac556a700 1 --2- 172.21.15.3:0/2709823566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 0x7f6ac00b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac4d69700 1 -- 172.21.15.3:0/2709823566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 msgr2=0x7f6ac00b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:46.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac4d69700 1 --2- 172.21.15.3:0/2709823566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 0x7f6ac00b0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac4d69700 1 -- 172.21.15.3:0/2709823566 >> v2:172.21.15.143:3300/0 conn(0x7f6ac00ad1c0 msgr2=0x7f6ac0143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:46.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac4d69700 1 --2- 172.21.15.3:0/2709823566 >> v2:172.21.15.143:3300/0 conn(0x7f6ac00ad1c0 0x7f6ac0143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac4d69700 1 -- 172.21.15.3:0/2709823566 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac00aedc0 con 0x7f6ac00b22b0 2022-07-21T01:33:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac4d69700 1 --2- 172.21.15.3:0/2709823566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 0x7f6ac00b0140 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6abc002f70 tx=0x7f6abc006eb0).ready entity=mon.0 client_cookie=353354fba553fd11 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ab77fe700 1 -- 172.21.15.3:0/2709823566 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6abc00d570 con 0x7f6ac00b22b0 2022-07-21T01:33:46.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ab77fe700 1 -- 172.21.15.3:0/2709823566 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc000e90 con 0x7f6ac00b22b0 2022-07-21T01:33:46.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ab77fe700 1 -- 172.21.15.3:0/2709823566 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6abc00d900 con 0x7f6ac00b22b0 2022-07-21T01:33:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/2709823566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 msgr2=0x7f6ac00b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/2709823566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 0x7f6ac00b0140 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6abc002f70 tx=0x7f6abc006eb0).stop 2022-07-21T01:33:46.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/2709823566 shutdown_connections 2022-07-21T01:33:46.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/2709823566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 0x7f6ac00b0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/2709823566 >> v2:172.21.15.143:3300/0 conn(0x7f6ac00ad1c0 0x7f6ac0143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/2709823566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 0x7f6ac00b0140 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.289+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/2709823566 >> 172.21.15.3:0/2709823566 conn(0x7f6ac005f840 msgr2=0x7f6ac005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:46.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/2709823566 shutdown_connections 2022-07-21T01:33:46.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/2709823566 wait complete. 2022-07-21T01:33:46.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 Processor -- start 2022-07-21T01:33:46.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 -- start start 2022-07-21T01:33:46.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ac00ad1c0 0x7f6ac00b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 0x7f6ac0152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 0x7f6ac00a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac0153190 con 0x7f6ac00b22b0 2022-07-21T01:33:46.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac00a1350 con 0x7f6ac00ad1c0 2022-07-21T01:33:46.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac6fcd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac00a0610 con 0x7f6ac0144230 2022-07-21T01:33:46.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ab7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 0x7f6ac0152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ab7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 0x7f6ac0152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43418/0 (socket says 172.21.15.3:43418) 2022-07-21T01:33:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ab7fff700 1 -- 172.21.15.3:0/837275175 learned_addr learned my addr 172.21.15.3:0/837275175 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac4d69700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ac00ad1c0 0x7f6ac00b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ac556a700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 0x7f6ac00a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ab7fff700 1 -- 172.21.15.3:0/837275175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 msgr2=0x7f6ac00a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ab7fff700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 0x7f6ac00a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ab7fff700 1 -- 172.21.15.3:0/837275175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ac00ad1c0 msgr2=0x7f6ac00b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ab7fff700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ac00ad1c0 0x7f6ac00b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ab7fff700 1 -- 172.21.15.3:0/837275175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc00b1e0 con 0x7f6ac00b22b0 2022-07-21T01:33:46.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.293+0000 7f6ab7fff700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 0x7f6ac0152ca0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6ab000c000 tx=0x7f6ab0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ab5ffb700 1 -- 172.21.15.3:0/837275175 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ab0010780 con 0x7f6ac00b22b0 2022-07-21T01:33:46.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ab5ffb700 1 -- 172.21.15.3:0/837275175 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab00108e0 con 0x7f6ac00b22b0 2022-07-21T01:33:46.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac00a08b0 con 0x7f6ac00b22b0 2022-07-21T01:33:46.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ab5ffb700 1 -- 172.21.15.3:0/837275175 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ab0010c40 con 0x7f6ac00b22b0 2022-07-21T01:33:46.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac00a4120 con 0x7f6ac00b22b0 2022-07-21T01:33:46.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac00b2690 con 0x7f6ac00b22b0 2022-07-21T01:33:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ab5ffb700 1 -- 172.21.15.3:0/837275175 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6ab001c460 con 0x7f6ac00b22b0 2022-07-21T01:33:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ab5ffb700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6aa00539e0 0x7f6aa0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ab5ffb700 1 -- 172.21.15.3:0/837275175 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6ab006c4f0 con 0x7f6ac00b22b0 2022-07-21T01:33:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ac4d69700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6aa00539e0 0x7f6aa0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.297+0000 7f6ac4d69700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6aa00539e0 0x7f6aa0055ea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6abc01c000 tx=0x7f6abc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.305+0000 7f6ab5ffb700 1 -- 172.21.15.3:0/837275175 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6ab00431b0 con 0x7f6ac00b22b0 2022-07-21T01:33:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.433+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6ac00574e0 con 0x7f6ac00b22b0 2022-07-21T01:33:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.433+0000 7f6ab5ffb700 1 -- 172.21.15.3:0/837275175 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6ab00467d0 con 0x7f6ac00b22b0 2022-07-21T01:33:46.427 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:46.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6aa00539e0 msgr2=0x7f6aa0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6aa00539e0 0x7f6aa0055ea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6abc01c000 tx=0x7f6abc015000).stop 2022-07-21T01:33:46.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 msgr2=0x7f6ac0152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 0x7f6ac0152ca0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6ab000c000 tx=0x7f6ab0006df0).stop 2022-07-21T01:33:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 shutdown_connections 2022-07-21T01:33:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ac0144230 0x7f6ac00a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ac00b22b0 0x7f6ac0152ca0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6aa00539e0 0x7f6aa0055ea0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 --2- 172.21.15.3:0/837275175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ac00ad1c0 0x7f6ac00b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 >> 172.21.15.3:0/837275175 conn(0x7f6ac005f840 msgr2=0x7f6ac0146970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 shutdown_connections 2022-07-21T01:33:46.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.437+0000 7f6ac6fcd700 1 -- 172.21.15.3:0/837275175 wait complete. 2022-07-21T01:33:46.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:33:46.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3eab8ab700 1 Processor -- start 2022-07-21T01:33:46.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3eab8ab700 1 -- start start 2022-07-21T01:33:46.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3eab8ab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 0x7f3ea40b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3eab8ab700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3ea40b2770 0x7f3ea4147d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3eab8ab700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ea41481f0 0x7f3ea414b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3eab8ab700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea4148660 con 0x7f3ea41481f0 2022-07-21T01:33:46.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3eab8ab700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea4060d80 con 0x7f3ea40b2770 2022-07-21T01:33:46.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3eab8ab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea4061020 con 0x7f3ea40b77b0 2022-07-21T01:33:46.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9e48700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ea41481f0 0x7f3ea414b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9e48700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ea41481f0 0x7f3ea414b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43428/0 (socket says 172.21.15.3:43428) 2022-07-21T01:33:46.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9e48700 1 -- 172.21.15.3:0/2852467045 learned_addr learned my addr 172.21.15.3:0/2852467045 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:46.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea8e46700 1 --2- 172.21.15.3:0/2852467045 >> v2:172.21.15.143:3300/0 conn(0x7f3ea40b2770 0x7f3ea4147d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9647700 1 --2- 172.21.15.3:0/2852467045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 0x7f3ea40b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9e48700 1 -- 172.21.15.3:0/2852467045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 msgr2=0x7f3ea40b7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9e48700 1 --2- 172.21.15.3:0/2852467045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 0x7f3ea40b7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9e48700 1 -- 172.21.15.3:0/2852467045 >> v2:172.21.15.143:3300/0 conn(0x7f3ea40b2770 msgr2=0x7f3ea4147d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9e48700 1 --2- 172.21.15.3:0/2852467045 >> v2:172.21.15.143:3300/0 conn(0x7f3ea40b2770 0x7f3ea4147d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9e48700 1 -- 172.21.15.3:0/2852467045 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea40612c0 con 0x7f3ea41481f0 2022-07-21T01:33:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3ea9e48700 1 --2- 172.21.15.3:0/2852467045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 0x7f3ea414b0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f3e9c0032f0 tx=0x7f3e9c009290).ready entity=mon.0 client_cookie=985901ea15b7b374 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3e97fff700 1 -- 172.21.15.3:0/2852467045 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e9c00b680 con 0x7f3ea41481f0 2022-07-21T01:33:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3e97fff700 1 -- 172.21.15.3:0/2852467045 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9c00b7e0 con 0x7f3ea41481f0 2022-07-21T01:33:46.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.561+0000 7f3e97fff700 1 -- 172.21.15.3:0/2852467045 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e9c00b950 con 0x7f3ea41481f0 2022-07-21T01:33:46.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- 172.21.15.3:0/2852467045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 msgr2=0x7f3ea414b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/2852467045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 0x7f3ea414b0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f3e9c0032f0 tx=0x7f3e9c009290).stop 2022-07-21T01:33:46.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- 172.21.15.3:0/2852467045 shutdown_connections 2022-07-21T01:33:46.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/2852467045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 0x7f3ea414b0e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/2852467045 >> v2:172.21.15.143:3300/0 conn(0x7f3ea40b2770 0x7f3ea4147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/2852467045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 0x7f3ea40b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- 172.21.15.3:0/2852467045 >> 172.21.15.3:0/2852467045 conn(0x7f3ea4062b60 msgr2=0x7f3ea4062f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:46.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- 172.21.15.3:0/2852467045 shutdown_connections 2022-07-21T01:33:46.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- 172.21.15.3:0/2852467045 wait complete. 2022-07-21T01:33:46.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 Processor -- start 2022-07-21T01:33:46.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- start start 2022-07-21T01:33:46.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ea40b2770 0x7f3ea415a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 0x7f3ea415a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 0x7f3ea415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea415b010 con 0x7f3ea41481f0 2022-07-21T01:33:46.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea40b0070 con 0x7f3ea40b2770 2022-07-21T01:33:46.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea414a4d0 con 0x7f3ea40b77b0 2022-07-21T01:33:46.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9e48700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 0x7f3ea415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9e48700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 0x7f3ea415ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43434/0 (socket says 172.21.15.3:43434) 2022-07-21T01:33:46.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9e48700 1 -- 172.21.15.3:0/4112780076 learned_addr learned my addr 172.21.15.3:0/4112780076 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:46.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea8e46700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 0x7f3ea415a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9647700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ea40b2770 0x7f3ea415a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9e48700 1 -- 172.21.15.3:0/4112780076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 msgr2=0x7f3ea415a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9e48700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 0x7f3ea415a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9e48700 1 -- 172.21.15.3:0/4112780076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ea40b2770 msgr2=0x7f3ea415a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9e48700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ea40b2770 0x7f3ea415a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9e48700 1 -- 172.21.15.3:0/4112780076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c009530 con 0x7f3ea41481f0 2022-07-21T01:33:46.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9647700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ea40b2770 0x7f3ea415a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:46.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3ea9e48700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 0x7f3ea415ab20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3e9c0032f0 tx=0x7f3e9c017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3e96ffd700 1 -- 172.21.15.3:0/4112780076 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e9c01f070 con 0x7f3ea41481f0 2022-07-21T01:33:46.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3e96ffd700 1 -- 172.21.15.3:0/4112780076 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9c011e80 con 0x7f3ea41481f0 2022-07-21T01:33:46.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea40aec80 con 0x7f3ea41481f0 2022-07-21T01:33:46.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.565+0000 7f3e96ffd700 1 -- 172.21.15.3:0/4112780076 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e9c00b680 con 0x7f3ea41481f0 2022-07-21T01:33:46.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.569+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea41574f0 con 0x7f3ea41481f0 2022-07-21T01:33:46.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.569+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea41579b0 con 0x7f3ea41481f0 2022-07-21T01:33:46.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.569+0000 7f3e96ffd700 1 -- 172.21.15.3:0/4112780076 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3e9c00f070 con 0x7f3ea41481f0 2022-07-21T01:33:46.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.569+0000 7f3e96ffd700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e80053950 0x7f3e80055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.569+0000 7f3ea9647700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e80053950 0x7f3e80055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.569+0000 7f3e96ffd700 1 -- 172.21.15.3:0/4112780076 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3e9c06d540 con 0x7f3ea41481f0 2022-07-21T01:33:46.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.569+0000 7f3ea9647700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e80053950 0x7f3e80055e10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3ea000b000 tx=0x7f3ea0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.577+0000 7f3e96ffd700 1 -- 172.21.15.3:0/4112780076 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ea41579b0 con 0x7f3ea41481f0 2022-07-21T01:33:46.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.705+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3ea41579b0 con 0x7f3ea41481f0 2022-07-21T01:33:46.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.705+0000 7f3e96ffd700 1 -- 172.21.15.3:0/4112780076 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3ea41579b0 con 0x7f3ea41481f0 2022-07-21T01:33:46.701 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:46.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e80053950 msgr2=0x7f3e80055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e80053950 0x7f3e80055e10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3ea000b000 tx=0x7f3ea0008040).stop 2022-07-21T01:33:46.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 msgr2=0x7f3ea415ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 0x7f3ea415ab20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3e9c0032f0 tx=0x7f3e9c017000).stop 2022-07-21T01:33:46.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 shutdown_connections 2022-07-21T01:33:46.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ea41481f0 0x7f3ea415ab20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ea40b77b0 0x7f3ea415a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e80053950 0x7f3e80055e10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 --2- 172.21.15.3:0/4112780076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ea40b2770 0x7f3ea415a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 >> 172.21.15.3:0/4112780076 conn(0x7f3ea4062b60 msgr2=0x7f3ea4064a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:46.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 shutdown_connections 2022-07-21T01:33:46.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.709+0000 7f3eab8ab700 1 -- 172.21.15.3:0/4112780076 wait complete. 2022-07-21T01:33:46.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:33:46.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 Processor -- start 2022-07-21T01:33:46.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 -- start start 2022-07-21T01:33:46.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa108060330 0x7fa1080b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa10805c6f0 0x7fa1080a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1080aa630 con 0x7fa1080af550 2022-07-21T01:33:46.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1080b15e0 con 0x7fa10805c6f0 2022-07-21T01:33:46.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1080af930 con 0x7fa108060330 2022-07-21T01:33:46.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa106d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa10805c6f0 0x7fa1080a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa106d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa10805c6f0 0x7fa1080a8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43440/0 (socket says 172.21.15.3:43440) 2022-07-21T01:33:46.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa106d9d700 1 -- 172.21.15.3:0/1226814642 learned_addr learned my addr 172.21.15.3:0/1226814642 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:46.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa106d9d700 1 -- 172.21.15.3:0/1226814642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa108060330 msgr2=0x7fa1080b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:46.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa0fdd9b700 1 --2- 172.21.15.3:0/1226814642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa108060330 0x7fa1080b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa106d9d700 1 --2- 172.21.15.3:0/1226814642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa108060330 0x7fa1080b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa106d9d700 1 -- 172.21.15.3:0/1226814642 >> v1:172.21.15.3:6789/0 conn(0x7fa1080af550 legacy=0x7fa1081445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa106d9d700 1 -- 172.21.15.3:0/1226814642 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10805fd10 con 0x7fa10805c6f0 2022-07-21T01:33:46.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa0fdd9b700 1 --2- 172.21.15.3:0/1226814642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa108060330 0x7fa1080b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:46.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa106d9d700 1 --2- 172.21.15.3:0/1226814642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa10805c6f0 0x7fa1080a8d80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa1000032f0 tx=0x7fa100009290).ready entity=mon.0 client_cookie=66c37e90be4838ec server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa105d9b700 1 -- 172.21.15.3:0/1226814642 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa10000b8a0 con 0x7fa10805c6f0 2022-07-21T01:33:46.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa105d9b700 1 -- 172.21.15.3:0/1226814642 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10000ba00 con 0x7fa10805c6f0 2022-07-21T01:33:46.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa105d9b700 1 -- 172.21.15.3:0/1226814642 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa10000bb70 con 0x7fa10805c6f0 2022-07-21T01:33:46.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 -- 172.21.15.3:0/1226814642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa10805c6f0 msgr2=0x7fa1080a8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 --2- 172.21.15.3:0/1226814642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa10805c6f0 0x7fa1080a8d80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa1000032f0 tx=0x7fa100009290).stop 2022-07-21T01:33:46.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 -- 172.21.15.3:0/1226814642 shutdown_connections 2022-07-21T01:33:46.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 --2- 172.21.15.3:0/1226814642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa10805c6f0 0x7fa1080a8d80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 --2- 172.21.15.3:0/1226814642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa108060330 0x7fa1080b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 -- 172.21.15.3:0/1226814642 >> 172.21.15.3:0/1226814642 conn(0x7fa1080612d0 msgr2=0x7fa108062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:46.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 -- 172.21.15.3:0/1226814642 shutdown_connections 2022-07-21T01:33:46.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.829+0000 7fa10759e700 1 -- 172.21.15.3:0/1226814642 wait complete. 2022-07-21T01:33:46.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 Processor -- start 2022-07-21T01:33:46.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 -- start start 2022-07-21T01:33:46.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10805c6f0 0x7fa1081403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa108060330 0x7fa108144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1080af550 0x7fa108142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa108144020 con 0x7fa108060330 2022-07-21T01:33:46.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa108145120 con 0x7fa1080af550 2022-07-21T01:33:46.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa108141190 con 0x7fa10805c6f0 2022-07-21T01:33:46.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fdd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa108060330 0x7fa108144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fdd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa108060330 0x7fa108144c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43446/0 (socket says 172.21.15.3:43446) 2022-07-21T01:33:46.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fdd9b700 1 -- 172.21.15.3:0/2857836245 learned_addr learned my addr 172.21.15.3:0/2857836245 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:46.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa106d9d700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1080af550 0x7fa108142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10659c700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10805c6f0 0x7fa1081403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fdd9b700 1 -- 172.21.15.3:0/2857836245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10805c6f0 msgr2=0x7fa1081403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fdd9b700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10805c6f0 0x7fa1081403f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fdd9b700 1 -- 172.21.15.3:0/2857836245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1080af550 msgr2=0x7fa108142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fdd9b700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1080af550 0x7fa108142410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fdd9b700 1 -- 172.21.15.3:0/2857836245 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100009530 con 0x7fa108060330 2022-07-21T01:33:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa106d9d700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1080af550 0x7fa108142410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fdd9b700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa108060330 0x7fa108144c30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fa0f0007000 tx=0x7fa0f0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fffff700 1 -- 172.21.15.3:0/2857836245 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0f0010580 con 0x7fa108060330 2022-07-21T01:33:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10805f840 con 0x7fa108060330 2022-07-21T01:33:46.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fffff700 1 -- 172.21.15.3:0/2857836245 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f00106e0 con 0x7fa108060330 2022-07-21T01:33:46.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa108144250 con 0x7fa108060330 2022-07-21T01:33:46.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.833+0000 7fa0fffff700 1 -- 172.21.15.3:0/2857836245 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0f0010a40 con 0x7fa108060330 2022-07-21T01:33:46.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.837+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa108144930 con 0x7fa108060330 2022-07-21T01:33:46.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.837+0000 7fa0fffff700 1 -- 172.21.15.3:0/2857836245 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa0f0010ba0 con 0x7fa108060330 2022-07-21T01:33:46.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.837+0000 7fa0fffff700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ec053950 0x7fa0ec055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:46.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.837+0000 7fa0fffff700 1 -- 172.21.15.3:0/2857836245 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa0f006c1a0 con 0x7fa108060330 2022-07-21T01:33:46.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.837+0000 7fa10659c700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ec053950 0x7fa0ec055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:46.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.837+0000 7fa10659c700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ec053950 0x7fa0ec055e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fa0f800d000 tx=0x7fa0f8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:46.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.841+0000 7fa0fffff700 1 -- 172.21.15.3:0/2857836245 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa0f0042e60 con 0x7fa108060330 2022-07-21T01:33:46.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.968+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa10805fa70 con 0x7fa108060330 2022-07-21T01:33:46.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.968+0000 7fa0fffff700 1 -- 172.21.15.3:0/2857836245 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa0f0046480 con 0x7fa108060330 2022-07-21T01:33:46.968 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:46.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ec053950 msgr2=0x7fa0ec055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ec053950 0x7fa0ec055e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fa0f800d000 tx=0x7fa0f8009040).stop 2022-07-21T01:33:46.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa108060330 msgr2=0x7fa108144c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:46.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa108060330 0x7fa108144c30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fa0f0007000 tx=0x7fa0f0008040).stop 2022-07-21T01:33:46.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 shutdown_connections 2022-07-21T01:33:46.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ec053950 0x7fa0ec055e10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1080af550 0x7fa108142410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa108060330 0x7fa108144c30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 --2- 172.21.15.3:0/2857836245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10805c6f0 0x7fa1081403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:46.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 >> 172.21.15.3:0/2857836245 conn(0x7fa1080612d0 msgr2=0x7fa1080b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:46.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 shutdown_connections 2022-07-21T01:33:46.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:46.972+0000 7fa10759e700 1 -- 172.21.15.3:0/2857836245 wait complete. 2022-07-21T01:33:46.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:47.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.088+0000 7fa08a5af700 1 Processor -- start 2022-07-21T01:33:47.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.088+0000 7fa08a5af700 1 -- start start 2022-07-21T01:33:47.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa0840614b0 0x7fa084061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0841405d0 0x7fa084140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0840aee80 con 0x7fa0840ad750 2022-07-21T01:33:47.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0840b70b0 con 0x7fa0840614b0 2022-07-21T01:33:47.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0840af0b0 con 0x7fa0841405d0 2022-07-21T01:33:47.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa089dae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0841405d0 0x7fa084140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa0895ad700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa0840614b0 0x7fa084061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa089dae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0841405d0 0x7fa084140a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:50372/0 (socket says 172.21.15.3:50372) 2022-07-21T01:33:47.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa088dac700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa0840ad750 0x7fa0840b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:53848/0 (socket says 172.21.15.3:53848) 2022-07-21T01:33:47.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa089dae700 1 -- 172.21.15.3:0/2429446165 learned_addr learned my addr 172.21.15.3:0/2429446165 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:47.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa089dae700 1 -- 172.21.15.3:0/2429446165 >> v2:172.21.15.143:3300/0 conn(0x7fa0840614b0 msgr2=0x7fa084061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:47.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa089dae700 1 --2- 172.21.15.3:0/2429446165 >> v2:172.21.15.143:3300/0 conn(0x7fa0840614b0 0x7fa084061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa089dae700 1 -- 172.21.15.3:0/2429446165 >> v1:172.21.15.143:6789/0 conn(0x7fa0840ad750 legacy=0x7fa0840b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa089dae700 1 -- 172.21.15.3:0/2429446165 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0840af350 con 0x7fa0841405d0 2022-07-21T01:33:47.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa0895ad700 1 --2- 172.21.15.3:0/2429446165 >> v2:172.21.15.143:3300/0 conn(0x7fa0840614b0 0x7fa084061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:47.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa089dae700 1 --2- 172.21.15.3:0/2429446165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0841405d0 0x7fa084140a60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fa0840aed30 tx=0x7fa07c009290).ready entity=mon.2 client_cookie=beb812515297d0e1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa07bfff700 1 -- 172.21.15.3:0/2429446165 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa07c00b860 con 0x7fa0841405d0 2022-07-21T01:33:47.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa07bfff700 1 -- 172.21.15.3:0/2429446165 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07c00b9c0 con 0x7fa0841405d0 2022-07-21T01:33:47.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa07bfff700 1 -- 172.21.15.3:0/2429446165 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa07c00bb30 con 0x7fa0841405d0 2022-07-21T01:33:47.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 -- 172.21.15.3:0/2429446165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0841405d0 msgr2=0x7fa084140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 --2- 172.21.15.3:0/2429446165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0841405d0 0x7fa084140a60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fa0840aed30 tx=0x7fa07c009290).stop 2022-07-21T01:33:47.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 -- 172.21.15.3:0/2429446165 shutdown_connections 2022-07-21T01:33:47.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 --2- 172.21.15.3:0/2429446165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0841405d0 0x7fa084140a60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 --2- 172.21.15.3:0/2429446165 >> v2:172.21.15.143:3300/0 conn(0x7fa0840614b0 0x7fa084061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 -- 172.21.15.3:0/2429446165 >> 172.21.15.3:0/2429446165 conn(0x7fa084066f70 msgr2=0x7fa084067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:47.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 -- 172.21.15.3:0/2429446165 shutdown_connections 2022-07-21T01:33:47.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.092+0000 7fa08a5af700 1 -- 172.21.15.3:0/2429446165 wait complete. 2022-07-21T01:33:47.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 Processor -- start 2022-07-21T01:33:47.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 -- start start 2022-07-21T01:33:47.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0840614b0 0x7fa08409ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0840ad750 0x7fa08409f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0841405d0 0x7fa084152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0840b1780 con 0x7fa0841405d0 2022-07-21T01:33:47.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0840b21d0 con 0x7fa0840614b0 2022-07-21T01:33:47.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0840b19b0 con 0x7fa0840ad750 2022-07-21T01:33:47.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa089dae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0841405d0 0x7fa084152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa089dae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0841405d0 0x7fa084152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43462/0 (socket says 172.21.15.3:43462) 2022-07-21T01:33:47.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa089dae700 1 -- 172.21.15.3:0/3659869698 learned_addr learned my addr 172.21.15.3:0/3659869698 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:47.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa0895ad700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0840614b0 0x7fa08409ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa088dac700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0840ad750 0x7fa08409f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa089dae700 1 -- 172.21.15.3:0/3659869698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0840ad750 msgr2=0x7fa08409f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa089dae700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0840ad750 0x7fa08409f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa089dae700 1 -- 172.21.15.3:0/3659869698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0840614b0 msgr2=0x7fa08409ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa089dae700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0840614b0 0x7fa08409ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa089dae700 1 -- 172.21.15.3:0/3659869698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07400b180 con 0x7fa0841405d0 2022-07-21T01:33:47.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa089dae700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0841405d0 0x7fa084152490 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa0840aed30 tx=0x7fa07c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa07a7fc700 1 -- 172.21.15.3:0/3659869698 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa07c00bb30 con 0x7fa0841405d0 2022-07-21T01:33:47.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa07a7fc700 1 -- 172.21.15.3:0/3659869698 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07c021630 con 0x7fa0841405d0 2022-07-21T01:33:47.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07c009530 con 0x7fa0841405d0 2022-07-21T01:33:47.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa07a7fc700 1 -- 172.21.15.3:0/3659869698 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa07c00b860 con 0x7fa0841405d0 2022-07-21T01:33:47.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.096+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0840b26f0 con 0x7fa0841405d0 2022-07-21T01:33:47.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.100+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0840b28b0 con 0x7fa0841405d0 2022-07-21T01:33:47.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.100+0000 7fa07a7fc700 1 -- 172.21.15.3:0/3659869698 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa07c00f070 con 0x7fa0841405d0 2022-07-21T01:33:47.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.100+0000 7fa07a7fc700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa06c053950 0x7fa06c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.100+0000 7fa07a7fc700 1 -- 172.21.15.3:0/3659869698 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa07c06cec0 con 0x7fa0841405d0 2022-07-21T01:33:47.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.100+0000 7fa0895ad700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa06c053950 0x7fa06c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.100+0000 7fa0895ad700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa06c053950 0x7fa06c055e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa07400b000 tx=0x7fa074006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.104+0000 7fa07a7fc700 1 -- 172.21.15.3:0/3659869698 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa07c043c80 con 0x7fa0841405d0 2022-07-21T01:33:47.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.232+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0840af5f0 con 0x7fa0841405d0 2022-07-21T01:33:47.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa07a7fc700 1 -- 172.21.15.3:0/3659869698 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7fa07c015030 con 0x7fa0841405d0 2022-07-21T01:33:47.232 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:47.232 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:47.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa06c053950 msgr2=0x7fa06c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa06c053950 0x7fa06c055e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa07400b000 tx=0x7fa074006df0).stop 2022-07-21T01:33:47.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0841405d0 msgr2=0x7fa084152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0841405d0 0x7fa084152490 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa0840aed30 tx=0x7fa07c016000).stop 2022-07-21T01:33:47.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 shutdown_connections 2022-07-21T01:33:47.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0841405d0 0x7fa084152490 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0840ad750 0x7fa08409f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa06c053950 0x7fa06c055e10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 --2- 172.21.15.3:0/3659869698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0840614b0 0x7fa08409ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 >> 172.21.15.3:0/3659869698 conn(0x7fa084066f70 msgr2=0x7fa084143830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:47.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 shutdown_connections 2022-07-21T01:33:47.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.236+0000 7fa08a5af700 1 -- 172.21.15.3:0/3659869698 wait complete. 2022-07-21T01:33:47.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:33:21.299936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:33:24.075599+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:33:47.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:33:47.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.360+0000 7ff55df27700 1 Processor -- start 2022-07-21T01:33:47.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 -- start start 2022-07-21T01:33:47.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5581499d0 0x7ff55814bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580af840 con 0x7ff5581437d0 2022-07-21T01:33:47.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff558144020 con 0x7ff5580ac310 2022-07-21T01:33:47.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580b63b0 con 0x7ff5581499d0 2022-07-21T01:33:47.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5577fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff5581437d0 0x7ff55813d550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51748/0 (socket says 172.21.15.3:51748) 2022-07-21T01:33:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5577fe700 1 -- 172.21.15.3:0/4258419304 learned_addr learned my addr 172.21.15.3:0/4258419304 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff557fff700 1 --2- 172.21.15.3:0/4258419304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5581499d0 0x7ff55814bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5567fc700 1 -- 172.21.15.3:0/4258419304 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745092941 0 0) 0x7ff5580af840 con 0x7ff5581437d0 2022-07-21T01:33:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5567fc700 1 -- 172.21.15.3:0/4258419304 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff538003260 con 0x7ff5581437d0 2022-07-21T01:33:47.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5567fc700 1 -- 172.21.15.3:0/4258419304 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111302870 0 0) 0x7ff558144020 con 0x7ff5580ac310 2022-07-21T01:33:47.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5567fc700 1 -- 172.21.15.3:0/4258419304 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5580af840 con 0x7ff5580ac310 2022-07-21T01:33:47.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff557fff700 1 -- 172.21.15.3:0/4258419304 >> v1:172.21.15.143:6789/0 conn(0x7ff5580ac310 legacy=0x7ff5580ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5567fc700 1 -- 172.21.15.3:0/4258419304 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154609062 0 0) 0x7ff538003260 con 0x7ff5581437d0 2022-07-21T01:33:47.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff557fff700 1 -- 172.21.15.3:0/4258419304 >> v1:172.21.15.3:6789/0 conn(0x7ff5581437d0 legacy=0x7ff55813d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff557fff700 1 -- 172.21.15.3:0/4258419304 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5581442c0 con 0x7ff5581499d0 2022-07-21T01:33:47.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff557fff700 1 --2- 172.21.15.3:0/4258419304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5581499d0 0x7ff55814bdc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff540001110 tx=0x7ff5500091d0).ready entity=mon.2 client_cookie=40b868cd92f0bce3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5567fc700 1 -- 172.21.15.3:0/4258419304 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff540001660 con 0x7ff5581499d0 2022-07-21T01:33:47.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5567fc700 1 -- 172.21.15.3:0/4258419304 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55000f510 con 0x7ff5581499d0 2022-07-21T01:33:47.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff5567fc700 1 -- 172.21.15.3:0/4258419304 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff55000f680 con 0x7ff5581499d0 2022-07-21T01:33:47.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 -- 172.21.15.3:0/4258419304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5581499d0 msgr2=0x7ff55814bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 --2- 172.21.15.3:0/4258419304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5581499d0 0x7ff55814bdc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff540001110 tx=0x7ff5500091d0).stop 2022-07-21T01:33:47.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 -- 172.21.15.3:0/4258419304 shutdown_connections 2022-07-21T01:33:47.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 --2- 172.21.15.3:0/4258419304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5581499d0 0x7ff55814bdc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 -- 172.21.15.3:0/4258419304 >> 172.21.15.3:0/4258419304 conn(0x7ff5580b5700 msgr2=0x7ff5580b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:47.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.364+0000 7ff55df27700 1 -- 172.21.15.3:0/4258419304 shutdown_connections 2022-07-21T01:33:47.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 -- 172.21.15.3:0/4258419304 wait complete. 2022-07-21T01:33:47.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 Processor -- start 2022-07-21T01:33:47.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 -- start start 2022-07-21T01:33:47.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580ac310 0x7ff5581369b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5581437d0 0x7ff558065f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5581499d0 0x7ff558066420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff558067860 con 0x7ff5581499d0 2022-07-21T01:33:47.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff558065ad0 con 0x7ff5581437d0 2022-07-21T01:33:47.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580675c0 con 0x7ff5580ac310 2022-07-21T01:33:47.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff557fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5581499d0 0x7ff558066420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff557fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5581499d0 0x7ff558066420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43476/0 (socket says 172.21.15.3:43476) 2022-07-21T01:33:47.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff557fff700 1 -- 172.21.15.3:0/2567588517 learned_addr learned my addr 172.21.15.3:0/2567588517 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:47.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff5577fe700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580ac310 0x7ff5581369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff556ffd700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5581437d0 0x7ff558065f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff557fff700 1 -- 172.21.15.3:0/2567588517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580ac310 msgr2=0x7ff5581369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff557fff700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580ac310 0x7ff5581369b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff557fff700 1 -- 172.21.15.3:0/2567588517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5581437d0 msgr2=0x7ff558065f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff557fff700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5581437d0 0x7ff558065f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff557fff700 1 -- 172.21.15.3:0/2567588517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff550009470 con 0x7ff5581499d0 2022-07-21T01:33:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff5577fe700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580ac310 0x7ff5581369b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff557fff700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5581499d0 0x7ff558066420 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff540001110 tx=0x7ff550016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff554ff9700 1 -- 172.21.15.3:0/2567588517 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550011bf0 con 0x7ff5581499d0 2022-07-21T01:33:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff558065420 con 0x7ff5581499d0 2022-07-21T01:33:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff554ff9700 1 -- 172.21.15.3:0/2567588517 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550011d50 con 0x7ff5581499d0 2022-07-21T01:33:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff554ff9700 1 -- 172.21.15.3:0/2567588517 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550021670 con 0x7ff5581499d0 2022-07-21T01:33:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.368+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff558146010 con 0x7ff5581499d0 2022-07-21T01:33:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.372+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55813a830 con 0x7ff5581499d0 2022-07-21T01:33:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.372+0000 7ff554ff9700 1 -- 172.21.15.3:0/2567588517 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff55000f070 con 0x7ff5581499d0 2022-07-21T01:33:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.372+0000 7ff554ff9700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff53c0539e0 0x7ff53c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.372+0000 7ff554ff9700 1 -- 172.21.15.3:0/2567588517 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff55006cbf0 con 0x7ff5581499d0 2022-07-21T01:33:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.372+0000 7ff5577fe700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff53c0539e0 0x7ff53c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.372+0000 7ff5577fe700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff53c0539e0 0x7ff53c055ea0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff54000b000 tx=0x7ff540000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.384+0000 7ff554ff9700 1 -- 172.21.15.3:0/2567588517 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff55813a830 con 0x7ff5581499d0 2022-07-21T01:33:47.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.508+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff55813a830 con 0x7ff5581499d0 2022-07-21T01:33:47.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.508+0000 7ff554ff9700 1 -- 172.21.15.3:0/2567588517 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v26) v1 ==== 110+0+10 (secure 0 0 0) 0x7ff55813a830 con 0x7ff5581499d0 2022-07-21T01:33:47.507 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:33:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff53c0539e0 msgr2=0x7ff53c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff53c0539e0 0x7ff53c055ea0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff54000b000 tx=0x7ff540000bc0).stop 2022-07-21T01:33:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5581499d0 msgr2=0x7ff558066420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5581499d0 0x7ff558066420 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff540001110 tx=0x7ff550016000).stop 2022-07-21T01:33:47.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 shutdown_connections 2022-07-21T01:33:47.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5581499d0 0x7ff558066420 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff53c0539e0 0x7ff53c055ea0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5581437d0 0x7ff558065f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 --2- 172.21.15.3:0/2567588517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580ac310 0x7ff5581369b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 >> 172.21.15.3:0/2567588517 conn(0x7ff5580b5700 msgr2=0x7ff55813a120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:47.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 shutdown_connections 2022-07-21T01:33:47.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.512+0000 7ff55df27700 1 -- 172.21.15.3:0/2567588517 wait complete. 2022-07-21T01:33:47.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:33:47.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.640+0000 7f200e7c6700 1 Processor -- start 2022-07-21T01:33:47.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.640+0000 7f200e7c6700 1 -- start start 2022-07-21T01:33:47.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f20080614b0 0x7f2008061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f20081405d0 0x7f2008140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20080aee80 con 0x7f20080ad750 2022-07-21T01:33:47.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20080b70b0 con 0x7f20081405d0 2022-07-21T01:33:47.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20080af0b0 con 0x7f20080614b0 2022-07-21T01:33:47.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200dfc5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f20081405d0 0x7f2008140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200dfc5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f20081405d0 0x7f2008140a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43484/0 (socket says 172.21.15.3:43484) 2022-07-21T01:33:47.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200dfc5700 1 -- 172.21.15.3:0/645248111 learned_addr learned my addr 172.21.15.3:0/645248111 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:47.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200d7c4700 1 --2- 172.21.15.3:0/645248111 >> v2:172.21.15.143:3300/0 conn(0x7f20080614b0 0x7f2008061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200dfc5700 1 -- 172.21.15.3:0/645248111 >> v2:172.21.15.143:3300/0 conn(0x7f20080614b0 msgr2=0x7f2008061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:47.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200dfc5700 1 --2- 172.21.15.3:0/645248111 >> v2:172.21.15.143:3300/0 conn(0x7f20080614b0 0x7f2008061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200dfc5700 1 -- 172.21.15.3:0/645248111 >> v1:172.21.15.143:6789/0 conn(0x7f20080ad750 legacy=0x7f20080b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200dfc5700 1 -- 172.21.15.3:0/645248111 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20080af350 con 0x7f20081405d0 2022-07-21T01:33:47.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200dfc5700 1 --2- 172.21.15.3:0/645248111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20081405d0 0x7f2008140a60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f20080aed30 tx=0x7f2000009290).ready entity=mon.0 client_cookie=70543c17931b20c2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f1ffffff700 1 -- 172.21.15.3:0/645248111 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f200000b860 con 0x7f20081405d0 2022-07-21T01:33:47.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f1ffffff700 1 -- 172.21.15.3:0/645248111 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200000b9c0 con 0x7f20081405d0 2022-07-21T01:33:47.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f1ffffff700 1 -- 172.21.15.3:0/645248111 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f200000bb30 con 0x7f20081405d0 2022-07-21T01:33:47.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 -- 172.21.15.3:0/645248111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20081405d0 msgr2=0x7f2008140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 --2- 172.21.15.3:0/645248111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20081405d0 0x7f2008140a60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f20080aed30 tx=0x7f2000009290).stop 2022-07-21T01:33:47.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 -- 172.21.15.3:0/645248111 shutdown_connections 2022-07-21T01:33:47.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 --2- 172.21.15.3:0/645248111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20081405d0 0x7f2008140a60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 --2- 172.21.15.3:0/645248111 >> v2:172.21.15.143:3300/0 conn(0x7f20080614b0 0x7f2008061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 -- 172.21.15.3:0/645248111 >> 172.21.15.3:0/645248111 conn(0x7f2008066f70 msgr2=0x7f2008067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:47.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 -- 172.21.15.3:0/645248111 shutdown_connections 2022-07-21T01:33:47.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.644+0000 7f200e7c6700 1 -- 172.21.15.3:0/645248111 wait complete. 2022-07-21T01:33:47.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200e7c6700 1 Processor -- start 2022-07-21T01:33:47.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200e7c6700 1 -- start start 2022-07-21T01:33:47.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200e7c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20080614b0 0x7f200809ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200e7c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20080ad750 0x7f20080b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200e7c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20081405d0 0x7f20080b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200e7c6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20080b2dc0 con 0x7f20080614b0 2022-07-21T01:33:47.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200e7c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20080b01f0 con 0x7f20081405d0 2022-07-21T01:33:47.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200e7c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20080afc80 con 0x7f20080ad750 2022-07-21T01:33:47.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200d7c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20080614b0 0x7f200809ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200d7c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20080614b0 0x7f200809ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43486/0 (socket says 172.21.15.3:43486) 2022-07-21T01:33:47.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200d7c4700 1 -- 172.21.15.3:0/4262803105 learned_addr learned my addr 172.21.15.3:0/4262803105 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:47.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200dfc5700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20081405d0 0x7f20080b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200cfc3700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20080ad750 0x7f20080b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200d7c4700 1 -- 172.21.15.3:0/4262803105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20080ad750 msgr2=0x7f20080b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200d7c4700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20080ad750 0x7f20080b1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200d7c4700 1 -- 172.21.15.3:0/4262803105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20081405d0 msgr2=0x7f20080b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200d7c4700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20081405d0 0x7f20080b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200d7c4700 1 -- 172.21.15.3:0/4262803105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2000009530 con 0x7f20080614b0 2022-07-21T01:33:47.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f200d7c4700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20080614b0 0x7f200809ed80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1ff800c000 tx=0x7f1ff8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.648+0000 7f1ffe7fc700 1 -- 172.21.15.3:0/4262803105 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ff8010780 con 0x7f20080614b0 2022-07-21T01:33:47.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f1ffe7fc700 1 -- 172.21.15.3:0/4262803105 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff80108e0 con 0x7f20080614b0 2022-07-21T01:33:47.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f1ffe7fc700 1 -- 172.21.15.3:0/4262803105 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ff8010c40 con 0x7f20080614b0 2022-07-21T01:33:47.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20080aff50 con 0x7f20080614b0 2022-07-21T01:33:47.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20081457e0 con 0x7f20080614b0 2022-07-21T01:33:47.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20080af5f0 con 0x7f20080614b0 2022-07-21T01:33:47.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f1ffe7fc700 1 -- 172.21.15.3:0/4262803105 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ff801c460 con 0x7f20080614b0 2022-07-21T01:33:47.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f1ffe7fc700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ff0053950 0x7f1ff0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f1ffe7fc700 1 -- 172.21.15.3:0/4262803105 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1ff806c470 con 0x7f20080614b0 2022-07-21T01:33:47.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f200cfc3700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ff0053950 0x7f1ff0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.652+0000 7f200cfc3700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ff0053950 0x7f1ff0055e10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f200400d000 tx=0x7f2004000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.656+0000 7f1ffe7fc700 1 -- 172.21.15.3:0/4262803105 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ff8043130 con 0x7f20080614b0 2022-07-21T01:33:47.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.784+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f20080574e0 con 0x7f20080614b0 2022-07-21T01:33:47.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.784+0000 7f1ffe7fc700 1 -- 172.21.15.3:0/4262803105 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1ff8046750 con 0x7f20080614b0 2022-07-21T01:33:47.781 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:47.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.784+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ff0053950 msgr2=0x7f1ff0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.784+0000 7f200e7c6700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ff0053950 0x7f1ff0055e10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f200400d000 tx=0x7f2004000bc0).stop 2022-07-21T01:33:47.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.784+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20080614b0 msgr2=0x7f200809ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.784+0000 7f200e7c6700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20080614b0 0x7f200809ed80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1ff800c000 tx=0x7f1ff8006df0).stop 2022-07-21T01:33:47.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.788+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 shutdown_connections 2022-07-21T01:33:47.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.788+0000 7f200e7c6700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ff0053950 0x7f1ff0055e10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.788+0000 7f200e7c6700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20081405d0 0x7f20080b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.788+0000 7f200e7c6700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20080ad750 0x7f20080b1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.788+0000 7f200e7c6700 1 --2- 172.21.15.3:0/4262803105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20080614b0 0x7f200809ed80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.788+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 >> 172.21.15.3:0/4262803105 conn(0x7f2008066f70 msgr2=0x7f20080b8870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:47.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.788+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 shutdown_connections 2022-07-21T01:33:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.788+0000 7f200e7c6700 1 -- 172.21.15.3:0/4262803105 wait complete. 2022-07-21T01:33:47.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:33:47.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.904+0000 7fa899025700 1 Processor -- start 2022-07-21T01:33:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.904+0000 7fa899025700 1 -- start start 2022-07-21T01:33:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa8940b13c0 0x7fa8940b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa8940afd80 0x7fa894142910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa894142e00 0x7fa8940ae870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8940ac520 con 0x7fa8940b13c0 2022-07-21T01:33:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8940b17a0 con 0x7fa8940afd80 2022-07-21T01:33:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940b0450 con 0x7fa894142e00 2022-07-21T01:33:47.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa893fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa8940b13c0 0x7fa8940b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa893fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa8940b13c0 0x7fa8940b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43494/0 (socket says 172.21.15.3:43494) 2022-07-21T01:33:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa893fff700 1 -- 172.21.15.3:0/2687814015 learned_addr learned my addr 172.21.15.3:0/2687814015 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa8937fe700 1 --2- 172.21.15.3:0/2687814015 >> v2:172.21.15.143:3300/0 conn(0x7fa8940afd80 0x7fa894142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa898824700 1 --2- 172.21.15.3:0/2687814015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa894142e00 0x7fa8940ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa893fff700 1 -- 172.21.15.3:0/2687814015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa894142e00 msgr2=0x7fa8940ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:47.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa893fff700 1 --2- 172.21.15.3:0/2687814015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa894142e00 0x7fa8940ae870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa893fff700 1 -- 172.21.15.3:0/2687814015 >> v2:172.21.15.143:3300/0 conn(0x7fa8940afd80 msgr2=0x7fa894142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:47.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa893fff700 1 --2- 172.21.15.3:0/2687814015 >> v2:172.21.15.143:3300/0 conn(0x7fa8940afd80 0x7fa894142910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa893fff700 1 -- 172.21.15.3:0/2687814015 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940b06f0 con 0x7fa8940b13c0 2022-07-21T01:33:47.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa893fff700 1 --2- 172.21.15.3:0/2687814015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8940b13c0 0x7fa8940b0b20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fa888002f70 tx=0x7fa888006eb0).ready entity=mon.0 client_cookie=b9024661a4af494b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa892ffd700 1 -- 172.21.15.3:0/2687814015 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa88800d570 con 0x7fa8940b13c0 2022-07-21T01:33:47.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa892ffd700 1 -- 172.21.15.3:0/2687814015 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa888000e90 con 0x7fa8940b13c0 2022-07-21T01:33:47.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa892ffd700 1 -- 172.21.15.3:0/2687814015 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa88800d900 con 0x7fa8940b13c0 2022-07-21T01:33:47.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 -- 172.21.15.3:0/2687814015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8940b13c0 msgr2=0x7fa8940b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 --2- 172.21.15.3:0/2687814015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8940b13c0 0x7fa8940b0b20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fa888002f70 tx=0x7fa888006eb0).stop 2022-07-21T01:33:47.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 -- 172.21.15.3:0/2687814015 shutdown_connections 2022-07-21T01:33:47.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 --2- 172.21.15.3:0/2687814015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa894142e00 0x7fa8940ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 --2- 172.21.15.3:0/2687814015 >> v2:172.21.15.143:3300/0 conn(0x7fa8940afd80 0x7fa894142910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 --2- 172.21.15.3:0/2687814015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8940b13c0 0x7fa8940b0b20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 -- 172.21.15.3:0/2687814015 >> 172.21.15.3:0/2687814015 conn(0x7fa8940626f0 msgr2=0x7fa894062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:47.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 -- 172.21.15.3:0/2687814015 shutdown_connections 2022-07-21T01:33:47.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.908+0000 7fa899025700 1 -- 172.21.15.3:0/2687814015 wait complete. 2022-07-21T01:33:47.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 Processor -- start 2022-07-21T01:33:47.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 -- start start 2022-07-21T01:33:47.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8940afd80 0x7fa8940b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8940b13c0 0x7fa894152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa894142e00 0x7fa894153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa894156370 con 0x7fa894142e00 2022-07-21T01:33:47.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa894146100 con 0x7fa8940b13c0 2022-07-21T01:33:47.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8941463a0 con 0x7fa8940afd80 2022-07-21T01:33:47.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa898824700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa894142e00 0x7fa894153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa898824700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa894142e00 0x7fa894153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43502/0 (socket says 172.21.15.3:43502) 2022-07-21T01:33:47.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa898824700 1 -- 172.21.15.3:0/3205639744 learned_addr learned my addr 172.21.15.3:0/3205639744 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:47.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa893fff700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8940afd80 0x7fa8940b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa8937fe700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8940b13c0 0x7fa894152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa898824700 1 -- 172.21.15.3:0/3205639744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8940afd80 msgr2=0x7fa8940b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa898824700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8940afd80 0x7fa8940b1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa898824700 1 -- 172.21.15.3:0/3205639744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8940b13c0 msgr2=0x7fa894152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:47.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa898824700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8940b13c0 0x7fa894152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:47.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa898824700 1 -- 172.21.15.3:0/3205639744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88800b1e0 con 0x7fa894142e00 2022-07-21T01:33:47.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa8937fe700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8940b13c0 0x7fa894152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:47.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa898824700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa894142e00 0x7fa894153290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fa88c006e00 tx=0x7fa88c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa8917fa700 1 -- 172.21.15.3:0/3205639744 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa88c00c670 con 0x7fa894142e00 2022-07-21T01:33:47.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa8917fa700 1 -- 172.21.15.3:0/3205639744 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88c00c7d0 con 0x7fa894142e00 2022-07-21T01:33:47.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa8917fa700 1 -- 172.21.15.3:0/3205639744 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa88c00cb30 con 0x7fa894142e00 2022-07-21T01:33:47.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8941560d0 con 0x7fa894142e00 2022-07-21T01:33:47.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.912+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8940aed50 con 0x7fa894142e00 2022-07-21T01:33:47.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.916+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8940574e0 con 0x7fa894142e00 2022-07-21T01:33:47.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.916+0000 7fa8917fa700 1 -- 172.21.15.3:0/3205639744 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa88c022070 con 0x7fa894142e00 2022-07-21T01:33:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.916+0000 7fa8917fa700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8740539e0 0x7fa874055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.916+0000 7fa893fff700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8740539e0 0x7fa874055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.916+0000 7fa8917fa700 1 -- 172.21.15.3:0/3205639744 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa88c06c5d0 con 0x7fa894142e00 2022-07-21T01:33:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.916+0000 7fa893fff700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8740539e0 0x7fa874055ea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa88801c000 tx=0x7fa888015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:47.920+0000 7fa8917fa700 1 -- 172.21.15.3:0/3205639744 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa88c043390 con 0x7fa894142e00 2022-07-21T01:33:48.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.044+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa89405f3d0 con 0x7fa894142e00 2022-07-21T01:33:48.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa8917fa700 1 -- 172.21.15.3:0/3205639744 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa88c0469b0 con 0x7fa894142e00 2022-07-21T01:33:48.044 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:48.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8740539e0 msgr2=0x7fa874055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8740539e0 0x7fa874055ea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa88801c000 tx=0x7fa888015000).stop 2022-07-21T01:33:48.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa894142e00 msgr2=0x7fa894153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa894142e00 0x7fa894153290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fa88c006e00 tx=0x7fa88c0091d0).stop 2022-07-21T01:33:48.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 shutdown_connections 2022-07-21T01:33:48.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa894142e00 0x7fa894153290 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8740539e0 0x7fa874055ea0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8940b13c0 0x7fa894152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 --2- 172.21.15.3:0/3205639744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8940afd80 0x7fa8940b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 >> 172.21.15.3:0/3205639744 conn(0x7fa8940626f0 msgr2=0x7fa89413d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:48.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 shutdown_connections 2022-07-21T01:33:48.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.048+0000 7fa899025700 1 -- 172.21.15.3:0/3205639744 wait complete. 2022-07-21T01:33:48.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:33:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485bfff700 1 Processor -- start 2022-07-21T01:33:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485bfff700 1 -- start start 2022-07-21T01:33:48.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c0b4640 0x7f485c0b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485bfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c061bd0 con 0x7f485c0617f0 2022-07-21T01:33:48.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485bfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0a88e0 con 0x7f485c144d20 2022-07-21T01:33:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0a9fd0 con 0x7f485c0b4640 2022-07-21T01:33:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485affd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f485c0617f0 0x7f485c144610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51786/0 (socket says 172.21.15.3:51786) 2022-07-21T01:33:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485affd700 1 -- 172.21.15.3:0/1296008902 learned_addr learned my addr 172.21.15.3:0/1296008902 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f4853fff700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c0b4640 0x7f485c0b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370943454 0 0) 0x7f485c061bd0 con 0x7f485c0617f0 2022-07-21T01:33:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f483c003260 con 0x7f485c0617f0 2022-07-21T01:33:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499225286 0 0) 0x7f485c0a88e0 con 0x7f485c144d20 2022-07-21T01:33:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485c061bd0 con 0x7f485c144d20 2022-07-21T01:33:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 49575159 0 0) 0x7f483c003260 con 0x7f485c0617f0 2022-07-21T01:33:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c0b4640 msgr2=0x7f485c0b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c0b4640 0x7f485c0b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 >> v1:172.21.15.143:6789/0 conn(0x7f485c144d20 legacy=0x7f485c14aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c0a8b80 con 0x7f485c0617f0 2022-07-21T01:33:48.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.172+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f484c001680 con 0x7f485c0617f0 2022-07-21T01:33:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 >> v1:172.21.15.3:6789/0 conn(0x7f485c0617f0 legacy=0x7f485c144610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f483c003910 0x7f483c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c0b4640 0x7f483c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f483c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0a9fd0 con 0x7f483c0049f0 2022-07-21T01:33:48.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c00b800 con 0x7f483c003910 2022-07-21T01:33:48.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c00b970 con 0x7f485c0b4640 2022-07-21T01:33:48.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485b7fe700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f483c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485affd700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f483c003910 0x7f483c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f4853fff700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c0b4640 0x7f483c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485b7fe700 1 -- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c0b4640 msgr2=0x7f483c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:48.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485b7fe700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c0b4640 0x7f483c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485b7fe700 1 -- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f483c003910 msgr2=0x7f483c003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:48.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485b7fe700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f483c003910 0x7f483c003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485b7fe700 1 -- 172.21.15.3:0/1296008902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c003260 con 0x7f483c0049f0 2022-07-21T01:33:48.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485b7fe700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f483c004e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4854001490 tx=0x7f4854007d80).ready entity=mon.0 client_cookie=88f7b686da03ee6a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4854001780 con 0x7f483c0049f0 2022-07-21T01:33:48.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4854006890 con 0x7f483c0049f0 2022-07-21T01:33:48.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485a7fc700 1 -- 172.21.15.3:0/1296008902 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4854006bf0 con 0x7f483c0049f0 2022-07-21T01:33:48.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485bfff700 1 -- 172.21.15.3:0/1296008902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 msgr2=0x7f483c004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485bfff700 1 --2- 172.21.15.3:0/1296008902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f483c004e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4854001490 tx=0x7f4854007d80).stop 2022-07-21T01:33:48.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485b7fe700 1 -- 172.21.15.3:0/1296008902 reap_dead start 2022-07-21T01:33:48.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485bfff700 1 -- 172.21.15.3:0/1296008902 shutdown_connections 2022-07-21T01:33:48.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485bfff700 1 -- 172.21.15.3:0/1296008902 >> 172.21.15.3:0/1296008902 conn(0x7f485c066f70 msgr2=0x7f485c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:48.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485bfff700 1 -- 172.21.15.3:0/1296008902 shutdown_connections 2022-07-21T01:33:48.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485bfff700 1 -- 172.21.15.3:0/1296008902 wait complete. 2022-07-21T01:33:48.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485bfff700 1 Processor -- start 2022-07-21T01:33:48.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.176+0000 7f485bfff700 1 -- start start 2022-07-21T01:33:48.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f485c149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f485c066af0 0x7f485c0b8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c066190 0x7f485c149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0b3d50 con 0x7f483c0049f0 2022-07-21T01:33:48.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c14b320 con 0x7f485c066af0 2022-07-21T01:33:48.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f485c149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c14b550 con 0x7f485c066190 2022-07-21T01:33:48.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f485c149860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43518/0 (socket says 172.21.15.3:43518) 2022-07-21T01:33:48.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485affd700 1 -- 172.21.15.3:0/2497294452 learned_addr learned my addr 172.21.15.3:0/2497294452 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:48.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f4853fff700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f485c066af0 0x7f485c0b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485b7fe700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c066190 0x7f485c149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485affd700 1 -- 172.21.15.3:0/2497294452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c066190 msgr2=0x7f485c149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485affd700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c066190 0x7f485c149fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485affd700 1 -- 172.21.15.3:0/2497294452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f485c066af0 msgr2=0x7f485c0b8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485affd700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f485c066af0 0x7f485c0b8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485affd700 1 -- 172.21.15.3:0/2497294452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c157780 con 0x7f483c0049f0 2022-07-21T01:33:48.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485affd700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f485c149860 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f485c0b3e90 tx=0x7f485c0b6720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f4858ff9700 1 -- 172.21.15.3:0/2497294452 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f484c0116b0 con 0x7f483c0049f0 2022-07-21T01:33:48.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f4858ff9700 1 -- 172.21.15.3:0/2497294452 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c011810 con 0x7f483c0049f0 2022-07-21T01:33:48.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c157a20 con 0x7f483c0049f0 2022-07-21T01:33:48.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f4858ff9700 1 -- 172.21.15.3:0/2497294452 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f484c011b70 con 0x7f483c0049f0 2022-07-21T01:33:48.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.180+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c154880 con 0x7f483c0049f0 2022-07-21T01:33:48.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.184+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c0a8e20 con 0x7f483c0049f0 2022-07-21T01:33:48.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.184+0000 7f4858ff9700 1 -- 172.21.15.3:0/2497294452 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f484c01b030 con 0x7f483c0049f0 2022-07-21T01:33:48.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.184+0000 7f4858ff9700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4840053b00 0x7f4840055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.184+0000 7f4858ff9700 1 -- 172.21.15.3:0/2497294452 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f484c06b940 con 0x7f483c0049f0 2022-07-21T01:33:48.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.184+0000 7f4853fff700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4840053b00 0x7f4840055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.184+0000 7f4853fff700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4840053b00 0x7f4840055fc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f484400b000 tx=0x7f4844008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.188+0000 7f4858ff9700 1 -- 172.21.15.3:0/2497294452 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f484c0425f0 con 0x7f483c0049f0 2022-07-21T01:33:48.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.312+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f485c0574e0 con 0x7f483c0049f0 2022-07-21T01:33:48.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.312+0000 7f4858ff9700 1 -- 172.21.15.3:0/2497294452 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+10 (secure 0 0 0) 0x7f484c019030 con 0x7f483c0049f0 2022-07-21T01:33:48.310 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:48.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.312+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4840053b00 msgr2=0x7f4840055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.312+0000 7f485bfff700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4840053b00 0x7f4840055fc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f484400b000 tx=0x7f4844008040).stop 2022-07-21T01:33:48.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.312+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 msgr2=0x7f485c149860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.312+0000 7f485bfff700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f485c149860 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f485c0b3e90 tx=0x7f485c0b6720).stop 2022-07-21T01:33:48.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.316+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 shutdown_connections 2022-07-21T01:33:48.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.316+0000 7f485bfff700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f485c066190 0x7f485c149fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.316+0000 7f485bfff700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4840053b00 0x7f4840055fc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.316+0000 7f485bfff700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f485c066af0 0x7f485c0b8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.316+0000 7f485bfff700 1 --2- 172.21.15.3:0/2497294452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f483c0049f0 0x7f485c149860 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.316+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 >> 172.21.15.3:0/2497294452 conn(0x7f485c066f70 msgr2=0x7f485c146080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:48.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.316+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 shutdown_connections 2022-07-21T01:33:48.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.316+0000 7f485bfff700 1 -- 172.21.15.3:0/2497294452 wait complete. 2022-07-21T01:33:48.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:33:48.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2cd7d700 1 Processor -- start 2022-07-21T01:33:48.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2cd7d700 1 -- start start 2022-07-21T01:33:48.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2cd7d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9e280b4550 0x7f9e280b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2cd7d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e280b6af0 0x7f9e2812bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2cd7d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e28144240 con 0x7f9e280b6710 2022-07-21T01:33:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2cd7d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e28143d00 con 0x7f9e280b6af0 2022-07-21T01:33:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2cd7d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e28143fa0 con 0x7f9e280b4550 2022-07-21T01:33:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e26d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e280b6af0 0x7f9e2812bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e26d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e280b6af0 0x7f9e2812bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43530/0 (socket says 172.21.15.3:43530) 2022-07-21T01:33:48.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e26d9d700 1 -- 172.21.15.3:0/657009303 learned_addr learned my addr 172.21.15.3:0/657009303 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e25d9b700 1 --2- 172.21.15.3:0/657009303 >> v2:172.21.15.143:3300/0 conn(0x7f9e280b4550 0x7f9e280b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e26d9d700 1 -- 172.21.15.3:0/657009303 >> v2:172.21.15.143:3300/0 conn(0x7f9e280b4550 msgr2=0x7f9e280b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e26d9d700 1 --2- 172.21.15.3:0/657009303 >> v2:172.21.15.143:3300/0 conn(0x7f9e280b4550 0x7f9e280b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e26d9d700 1 -- 172.21.15.3:0/657009303 >> v1:172.21.15.143:6789/0 conn(0x7f9e280b6710 legacy=0x7f9e28061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e26d9d700 1 -- 172.21.15.3:0/657009303 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e28061780 con 0x7f9e280b6af0 2022-07-21T01:33:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e26d9d700 1 --2- 172.21.15.3:0/657009303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b6af0 0x7f9e2812bb90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f9e281432b0 tx=0x7f9e20009290).ready entity=mon.0 client_cookie=f05a627eba8afcbe server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2559a700 1 -- 172.21.15.3:0/657009303 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e2000b860 con 0x7f9e280b6af0 2022-07-21T01:33:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2559a700 1 -- 172.21.15.3:0/657009303 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2000b9c0 con 0x7f9e280b6af0 2022-07-21T01:33:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2559a700 1 -- 172.21.15.3:0/657009303 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e2000bb30 con 0x7f9e280b6af0 2022-07-21T01:33:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/657009303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b6af0 msgr2=0x7f9e2812bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.428+0000 7f9e2cd7d700 1 --2- 172.21.15.3:0/657009303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b6af0 0x7f9e2812bb90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f9e281432b0 tx=0x7f9e20009290).stop 2022-07-21T01:33:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.432+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/657009303 shutdown_connections 2022-07-21T01:33:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.432+0000 7f9e2cd7d700 1 --2- 172.21.15.3:0/657009303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b6af0 0x7f9e2812bb90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.432+0000 7f9e2cd7d700 1 --2- 172.21.15.3:0/657009303 >> v2:172.21.15.143:3300/0 conn(0x7f9e280b4550 0x7f9e280b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.432+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/657009303 >> 172.21.15.3:0/657009303 conn(0x7f9e2805c620 msgr2=0x7f9e2805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.432+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/657009303 shutdown_connections 2022-07-21T01:33:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.432+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/657009303 wait complete. 2022-07-21T01:33:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.432+0000 7f9e2cd7d700 1 Processor -- start 2022-07-21T01:33:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.432+0000 7f9e2cd7d700 1 -- start start 2022-07-21T01:33:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2cd7d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b4550 0x7f9e281419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2cd7d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280b6710 0x7f9e28125030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2cd7d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b6af0 0x7f9e2813d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2cd7d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e281459e0 con 0x7f9e280b4550 2022-07-21T01:33:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2cd7d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2813c9d0 con 0x7f9e280b6710 2022-07-21T01:33:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2cd7d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2813bec0 con 0x7f9e280b6af0 2022-07-21T01:33:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b4550 0x7f9e281419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b4550 0x7f9e281419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43532/0 (socket says 172.21.15.3:43532) 2022-07-21T01:33:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2659c700 1 -- 172.21.15.3:0/1169986209 learned_addr learned my addr 172.21.15.3:0/1169986209 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e25d9b700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280b6710 0x7f9e28125030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e26d9d700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b6af0 0x7f9e2813d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2659c700 1 -- 172.21.15.3:0/1169986209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b6af0 msgr2=0x7f9e2813d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2659c700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b6af0 0x7f9e2813d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2659c700 1 -- 172.21.15.3:0/1169986209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280b6710 msgr2=0x7f9e28125030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2659c700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280b6710 0x7f9e28125030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2659c700 1 -- 172.21.15.3:0/1169986209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e20009530 con 0x7f9e280b4550 2022-07-21T01:33:48.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e26d9d700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b6af0 0x7f9e2813d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:48.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2659c700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b4550 0x7f9e281419b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f9e2813cb10 tx=0x7f9e18007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e177fe700 1 -- 172.21.15.3:0/1169986209 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1800ce20 con 0x7f9e280b4550 2022-07-21T01:33:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e177fe700 1 -- 172.21.15.3:0/1169986209 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e18007240 con 0x7f9e280b4550 2022-07-21T01:33:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e281450c0 con 0x7f9e280b4550 2022-07-21T01:33:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e177fe700 1 -- 172.21.15.3:0/1169986209 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e18006700 con 0x7f9e280b4550 2022-07-21T01:33:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2813cbf0 con 0x7f9e280b4550 2022-07-21T01:33:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.436+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2813bd30 con 0x7f9e280b4550 2022-07-21T01:33:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.440+0000 7f9e177fe700 1 -- 172.21.15.3:0/1169986209 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9e18010030 con 0x7f9e280b4550 2022-07-21T01:33:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.440+0000 7f9e177fe700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.440+0000 7f9e177fe700 1 -- 172.21.15.3:0/1169986209 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9e1806bbb0 con 0x7f9e280b4550 2022-07-21T01:33:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.440+0000 7f9e25d9b700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.440+0000 7f9e25d9b700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9e1c00b000 tx=0x7f9e1c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.444+0000 7f9e177fe700 1 -- 172.21.15.3:0/1169986209 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9e180428f0 con 0x7f9e280b4550 2022-07-21T01:33:48.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.572+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e281450c0 con 0x7f9e280b4550 2022-07-21T01:33:48.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.576+0000 7f9e177fe700 1 -- 172.21.15.3:0/1169986209 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+16299 (secure 0 0 0) 0x7f9e281450c0 con 0x7f9e280b4550 2022-07-21T01:33:48.573 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:48.573 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:26.370831+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:33:21.299936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-21T01:33:24.075599+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:33:48.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 msgr2=0x7f9e04055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9e1c00b000 tx=0x7f9e1c006df0).stop 2022-07-21T01:33:48.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b4550 msgr2=0x7f9e281419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b4550 0x7f9e281419b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f9e2813cb10 tx=0x7f9e18007c10).stop 2022-07-21T01:33:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 shutdown_connections 2022-07-21T01:33:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b6af0 0x7f9e2813d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280b6710 0x7f9e28125030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 --2- 172.21.15.3:0/1169986209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b4550 0x7f9e281419b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 >> 172.21.15.3:0/1169986209 conn(0x7f9e2805c620 msgr2=0x7f9e28128c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 shutdown_connections 2022-07-21T01:33:48.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.580+0000 7f9e2cd7d700 1 -- 172.21.15.3:0/1169986209 wait complete. 2022-07-21T01:33:48.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:33:21.299936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:33:24.075599+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:33:48.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:33:48.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f0210979700 1 Processor -- start 2022-07-21T01:33:48.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f0210979700 1 -- start start 2022-07-21T01:33:48.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f0210979700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 0x7f020c0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f0210979700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c061d90 con 0x7f020c0b5ed0 2022-07-21T01:33:48.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f0210979700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c0ba5a0 con 0x7f020c0619b0 2022-07-21T01:33:48.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f0210979700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c143fa0 con 0x7f020c0b6ee0 2022-07-21T01:33:48.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f020b59e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f020c0b5ed0 0x7f020c0b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51822/0 (socket says 172.21.15.3:51822) 2022-07-21T01:33:48.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f020b59e700 1 -- 172.21.15.3:0/1067646889 learned_addr learned my addr 172.21.15.3:0/1067646889 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:48.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f020a59c700 1 --2- 172.21.15.3:0/1067646889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 0x7f020c0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.708+0000 7f0209d9b700 1 -- 172.21.15.3:0/1067646889 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267150153 0 0) 0x7f020c061d90 con 0x7f020c0b5ed0 2022-07-21T01:33:48.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0209d9b700 1 -- 172.21.15.3:0/1067646889 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ec003260 con 0x7f020c0b5ed0 2022-07-21T01:33:48.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0209d9b700 1 -- 172.21.15.3:0/1067646889 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781460580 0 0) 0x7f020c0ba5a0 con 0x7f020c0619b0 2022-07-21T01:33:48.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0209d9b700 1 -- 172.21.15.3:0/1067646889 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c061d90 con 0x7f020c0619b0 2022-07-21T01:33:48.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f020a59c700 1 -- 172.21.15.3:0/1067646889 >> v1:172.21.15.143:6789/0 conn(0x7f020c0619b0 legacy=0x7f020c143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f020a59c700 1 -- 172.21.15.3:0/1067646889 >> v1:172.21.15.3:6789/0 conn(0x7f020c0b5ed0 legacy=0x7f020c0b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f020a59c700 1 -- 172.21.15.3:0/1067646889 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c143150 con 0x7f020c0b6ee0 2022-07-21T01:33:48.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f020a59c700 1 --2- 172.21.15.3:0/1067646889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 0x7f020c0b48a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f01fc002f70 tx=0x7f01fc006df0).ready entity=mon.2 client_cookie=202aa3e11e302655 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0209d9b700 1 -- 172.21.15.3:0/1067646889 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01fc00d5c0 con 0x7f020c0b6ee0 2022-07-21T01:33:48.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0209d9b700 1 -- 172.21.15.3:0/1067646889 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc000e90 con 0x7f020c0b6ee0 2022-07-21T01:33:48.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0209d9b700 1 -- 172.21.15.3:0/1067646889 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01fc00d910 con 0x7f020c0b6ee0 2022-07-21T01:33:48.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 -- 172.21.15.3:0/1067646889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 msgr2=0x7f020c0b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 --2- 172.21.15.3:0/1067646889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 0x7f020c0b48a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f01fc002f70 tx=0x7f01fc006df0).stop 2022-07-21T01:33:48.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 -- 172.21.15.3:0/1067646889 shutdown_connections 2022-07-21T01:33:48.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 --2- 172.21.15.3:0/1067646889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 0x7f020c0b48a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 -- 172.21.15.3:0/1067646889 >> 172.21.15.3:0/1067646889 conn(0x7f020c06af80 msgr2=0x7f020c05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 -- 172.21.15.3:0/1067646889 shutdown_connections 2022-07-21T01:33:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 -- 172.21.15.3:0/1067646889 wait complete. 2022-07-21T01:33:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 Processor -- start 2022-07-21T01:33:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 -- start start 2022-07-21T01:33:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f020c0619b0 0x7f020c0adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020c0b5ed0 0x7f020c1463b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 0x7f020c155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0b07c0 con 0x7f020c0b5ed0 2022-07-21T01:33:48.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c156940 con 0x7f020c0619b0 2022-07-21T01:33:48.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f0210979700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c144c90 con 0x7f020c0b6ee0 2022-07-21T01:33:48.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f020a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020c0b5ed0 0x7f020c1463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f020a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020c0b5ed0 0x7f020c1463b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43548/0 (socket says 172.21.15.3:43548) 2022-07-21T01:33:48.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f020a59c700 1 -- 172.21.15.3:0/1272852898 learned_addr learned my addr 172.21.15.3:0/1272852898 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:48.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f020b59e700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 0x7f020c155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.712+0000 7f020ad9d700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f020c0619b0 0x7f020c0adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f020a59c700 1 -- 172.21.15.3:0/1272852898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 msgr2=0x7f020c155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f020a59c700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 0x7f020c155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f020a59c700 1 -- 172.21.15.3:0/1272852898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f020c0619b0 msgr2=0x7f020c0adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f020a59c700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f020c0619b0 0x7f020c0adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f020a59c700 1 -- 172.21.15.3:0/1272852898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc00b180 con 0x7f020c0b5ed0 2022-07-21T01:33:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f020ad9d700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f020c0619b0 0x7f020c0adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f020a59c700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020c0b5ed0 0x7f020c1463b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f01fc002f70 tx=0x7f01fc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f0203fff700 1 -- 172.21.15.3:0/1272852898 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01fc013e90 con 0x7f020c0b5ed0 2022-07-21T01:33:48.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f0203fff700 1 -- 172.21.15.3:0/1272852898 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc01e400 con 0x7f020c0b5ed0 2022-07-21T01:33:48.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020c144f30 con 0x7f020c0b5ed0 2022-07-21T01:33:48.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f0203fff700 1 -- 172.21.15.3:0/1272852898 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01fc01e760 con 0x7f020c0b5ed0 2022-07-21T01:33:48.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020c0ae020 con 0x7f020c0b5ed0 2022-07-21T01:33:48.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f0203fff700 1 -- 172.21.15.3:0/1272852898 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f01fc01c070 con 0x7f020c0b5ed0 2022-07-21T01:33:48.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020c1461b0 con 0x7f020c0b5ed0 2022-07-21T01:33:48.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.716+0000 7f0203fff700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f01f0053b00 0x7f01f0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.720+0000 7f0203fff700 1 -- 172.21.15.3:0/1272852898 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f01fc06cad0 con 0x7f020c0b5ed0 2022-07-21T01:33:48.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.720+0000 7f020ad9d700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f01f0053b00 0x7f01f0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.720+0000 7f020ad9d700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f01f0053b00 0x7f01f0055fc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f01f400b000 tx=0x7f01f4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.724+0000 7f0203fff700 1 -- 172.21.15.3:0/1272852898 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f01fc0437f0 con 0x7f020c0b5ed0 2022-07-21T01:33:48.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f020c0574e0 con 0x7f020c0b5ed0 2022-07-21T01:33:48.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0203fff700 1 -- 172.21.15.3:0/1272852898 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v26) v1 ==== 110+0+10 (secure 0 0 0) 0x7f01fc023070 con 0x7f020c0b5ed0 2022-07-21T01:33:48.848 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:33:48.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f01f0053b00 msgr2=0x7f01f0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f01f0053b00 0x7f01f0055fc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f01f400b000 tx=0x7f01f4009040).stop 2022-07-21T01:33:48.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020c0b5ed0 msgr2=0x7f020c1463b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020c0b5ed0 0x7f020c1463b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f01fc002f70 tx=0x7f01fc014000).stop 2022-07-21T01:33:48.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 shutdown_connections 2022-07-21T01:33:48.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f020c0b6ee0 0x7f020c155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f020c0b5ed0 0x7f020c1463b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f01f0053b00 0x7f01f0055fc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 --2- 172.21.15.3:0/1272852898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f020c0619b0 0x7f020c0adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 >> 172.21.15.3:0/1272852898 conn(0x7f020c06af80 msgr2=0x7f020c05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:48.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 shutdown_connections 2022-07-21T01:33:48.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.851+0000 7f0210979700 1 -- 172.21.15.3:0/1272852898 wait complete. 2022-07-21T01:33:48.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:33:48.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff447909700 1 Processor -- start 2022-07-21T01:33:48.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff447909700 1 -- start start 2022-07-21T01:33:48.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff447909700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff440146ae0 0x7ff4401403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff447909700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b4310 0x7ff4401408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff447909700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff4400b4820 0x7ff4400b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff447909700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4401476a0 con 0x7ff4400b4820 2022-07-21T01:33:48.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff447909700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff440143cf0 con 0x7ff440146ae0 2022-07-21T01:33:48.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff447909700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff440143f90 con 0x7ff4400b4310 2022-07-21T01:33:48.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff445ea6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff4400b4820 0x7ff4400b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff445ea6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff4400b4820 0x7ff4400b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43558/0 (socket says 172.21.15.3:43558) 2022-07-21T01:33:48.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.975+0000 7ff445ea6700 1 -- 172.21.15.3:0/46653886 learned_addr learned my addr 172.21.15.3:0/46653886 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:48.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff4456a5700 1 --2- 172.21.15.3:0/46653886 >> v2:172.21.15.143:3300/0 conn(0x7ff440146ae0 0x7ff4401403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff444ea4700 1 --2- 172.21.15.3:0/46653886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b4310 0x7ff4401408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff445ea6700 1 -- 172.21.15.3:0/46653886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b4310 msgr2=0x7ff4401408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:48.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff445ea6700 1 --2- 172.21.15.3:0/46653886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b4310 0x7ff4401408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff445ea6700 1 -- 172.21.15.3:0/46653886 >> v2:172.21.15.143:3300/0 conn(0x7ff440146ae0 msgr2=0x7ff4401403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:48.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff445ea6700 1 --2- 172.21.15.3:0/46653886 >> v2:172.21.15.143:3300/0 conn(0x7ff440146ae0 0x7ff4401403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff445ea6700 1 -- 172.21.15.3:0/46653886 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400b0e20 con 0x7ff4400b4820 2022-07-21T01:33:48.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff444ea4700 1 --2- 172.21.15.3:0/46653886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b4310 0x7ff4401408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:48.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff445ea6700 1 --2- 172.21.15.3:0/46653886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 0x7ff4400b5060 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff4380032f0 tx=0x7ff438009290).ready entity=mon.0 client_cookie=ac563b0b7126b1ca server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff437fff700 1 -- 172.21.15.3:0/46653886 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff43800b870 con 0x7ff4400b4820 2022-07-21T01:33:48.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff437fff700 1 -- 172.21.15.3:0/46653886 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43800b9d0 con 0x7ff4400b4820 2022-07-21T01:33:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff437fff700 1 -- 172.21.15.3:0/46653886 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff43800bb40 con 0x7ff4400b4820 2022-07-21T01:33:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff447909700 1 -- 172.21.15.3:0/46653886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 msgr2=0x7ff4400b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff447909700 1 --2- 172.21.15.3:0/46653886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 0x7ff4400b5060 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff4380032f0 tx=0x7ff438009290).stop 2022-07-21T01:33:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff447909700 1 -- 172.21.15.3:0/46653886 shutdown_connections 2022-07-21T01:33:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff447909700 1 --2- 172.21.15.3:0/46653886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 0x7ff4400b5060 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff447909700 1 --2- 172.21.15.3:0/46653886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b4310 0x7ff4401408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff447909700 1 --2- 172.21.15.3:0/46653886 >> v2:172.21.15.143:3300/0 conn(0x7ff440146ae0 0x7ff4401403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff447909700 1 -- 172.21.15.3:0/46653886 >> 172.21.15.3:0/46653886 conn(0x7ff44005c620 msgr2=0x7ff44005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:48.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff447909700 1 -- 172.21.15.3:0/46653886 shutdown_connections 2022-07-21T01:33:48.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.979+0000 7ff447909700 1 -- 172.21.15.3:0/46653886 wait complete. 2022-07-21T01:33:48.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff447909700 1 Processor -- start 2022-07-21T01:33:48.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff447909700 1 -- start start 2022-07-21T01:33:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff447909700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b4310 0x7ff4400b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff447909700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 0x7ff4400ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff447909700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff440146ae0 0x7ff4400adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff447909700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400ae430 con 0x7ff4400b4820 2022-07-21T01:33:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff447909700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff440142380 con 0x7ff4400b4310 2022-07-21T01:33:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff447909700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff440141e40 con 0x7ff440146ae0 2022-07-21T01:33:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff444ea4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 0x7ff4400ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff444ea4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 0x7ff4400ada50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43562/0 (socket says 172.21.15.3:43562) 2022-07-21T01:33:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff444ea4700 1 -- 172.21.15.3:0/70412116 learned_addr learned my addr 172.21.15.3:0/70412116 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff4456a5700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b4310 0x7ff4400b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff445ea6700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff440146ae0 0x7ff4400adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff444ea4700 1 -- 172.21.15.3:0/70412116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff440146ae0 msgr2=0x7ff4400adf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff444ea4700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff440146ae0 0x7ff4400adf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff444ea4700 1 -- 172.21.15.3:0/70412116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b4310 msgr2=0x7ff4400b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff444ea4700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b4310 0x7ff4400b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff444ea4700 1 -- 172.21.15.3:0/70412116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff438009530 con 0x7ff4400b4820 2022-07-21T01:33:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff444ea4700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 0x7ff4400ada50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7ff440142510 tx=0x7ff430008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff4367fc700 1 -- 172.21.15.3:0/70412116 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff430011780 con 0x7ff4400b4820 2022-07-21T01:33:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.983+0000 7ff4367fc700 1 -- 172.21.15.3:0/70412116 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4300118e0 con 0x7ff4400b4820 2022-07-21T01:33:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.987+0000 7ff4367fc700 1 -- 172.21.15.3:0/70412116 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff430011c40 con 0x7ff4400b4820 2022-07-21T01:33:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.987+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4401420e0 con 0x7ff4400b4820 2022-07-21T01:33:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.987+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff440155a00 con 0x7ff4400b4820 2022-07-21T01:33:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.987+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4400574e0 con 0x7ff4400b4820 2022-07-21T01:33:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.987+0000 7ff4367fc700 1 -- 172.21.15.3:0/70412116 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff430006030 con 0x7ff4400b4820 2022-07-21T01:33:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.987+0000 7ff4367fc700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff4280539e0 0x7ff428055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.987+0000 7ff4367fc700 1 -- 172.21.15.3:0/70412116 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff43006c0f0 con 0x7ff4400b4820 2022-07-21T01:33:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.987+0000 7ff4456a5700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff4280539e0 0x7ff428055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.991+0000 7ff4456a5700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff4280539e0 0x7ff428055ea0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff43c00b000 tx=0x7ff43c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:48.995+0000 7ff4367fc700 1 -- 172.21.15.3:0/70412116 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff430042db0 con 0x7ff4400b4820 2022-07-21T01:33:49.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.123+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff4400ad780 con 0x7ff4400b4820 2022-07-21T01:33:49.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff4367fc700 1 -- 172.21.15.3:0/70412116 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff4300463d0 con 0x7ff4400b4820 2022-07-21T01:33:49.124 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:49.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff4280539e0 msgr2=0x7ff428055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff4280539e0 0x7ff428055ea0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff43c00b000 tx=0x7ff43c006df0).stop 2022-07-21T01:33:49.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 msgr2=0x7ff4400ada50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 0x7ff4400ada50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7ff440142510 tx=0x7ff430008040).stop 2022-07-21T01:33:49.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 shutdown_connections 2022-07-21T01:33:49.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff440146ae0 0x7ff4400adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b4820 0x7ff4400ada50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff4280539e0 0x7ff428055ea0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 --2- 172.21.15.3:0/70412116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b4310 0x7ff4400b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 >> 172.21.15.3:0/70412116 conn(0x7ff44005c620 msgr2=0x7ff440145d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:49.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 shutdown_connections 2022-07-21T01:33:49.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.127+0000 7ff447909700 1 -- 172.21.15.3:0/70412116 wait complete. 2022-07-21T01:33:49.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:33:49.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fdf80700 1 Processor -- start 2022-07-21T01:33:49.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fdf80700 1 -- start start 2022-07-21T01:33:49.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fdf80700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f80af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fdf80700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f10f8147730 0x7f10f80acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fdf80700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80b5a10 con 0x7f10f80b07a0 2022-07-21T01:33:49.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fdf80700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f8145470 con 0x7f10f8147730 2022-07-21T01:33:49.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fdf80700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80ae180 con 0x7f10f80b5630 2022-07-21T01:33:49.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fd77f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f10f8147730 0x7f10f80acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10f7fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f10f80b07a0 0x7f10f8067730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:53964/0 (socket says 172.21.15.3:53964) 2022-07-21T01:33:49.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10f7fff700 1 -- 172.21.15.3:0/1336883681 learned_addr learned my addr 172.21.15.3:0/1336883681 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:49.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fcf7e700 1 --2- 172.21.15.3:0/1336883681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f80af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10f77fe700 1 -- 172.21.15.3:0/1336883681 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025208987 0 0) 0x7f10f80b5a10 con 0x7f10f80b07a0 2022-07-21T01:33:49.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10f77fe700 1 -- 172.21.15.3:0/1336883681 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10dc003260 con 0x7f10f80b07a0 2022-07-21T01:33:49.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fd77f700 1 -- 172.21.15.3:0/1336883681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 msgr2=0x7f10f80af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:49.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fd77f700 1 --2- 172.21.15.3:0/1336883681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f80af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fd77f700 1 -- 172.21.15.3:0/1336883681 >> v1:172.21.15.143:6789/0 conn(0x7f10f80b07a0 legacy=0x7f10f8067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fd77f700 1 -- 172.21.15.3:0/1336883681 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80ac990 con 0x7f10f8147730 2022-07-21T01:33:49.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fcf7e700 1 --2- 172.21.15.3:0/1336883681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f80af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:49.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10fd77f700 1 --2- 172.21.15.3:0/1336883681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f8147730 0x7f10f80acdf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f10f00032f0 tx=0x7f10f00091d0).ready entity=mon.0 client_cookie=235a1e1655729beb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:49.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10f77fe700 1 -- 172.21.15.3:0/1336883681 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10f000b600 con 0x7f10f8147730 2022-07-21T01:33:49.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10f77fe700 1 -- 172.21.15.3:0/1336883681 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f000b760 con 0x7f10f8147730 2022-07-21T01:33:49.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.251+0000 7f10f77fe700 1 -- 172.21.15.3:0/1336883681 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10f000b8d0 con 0x7f10f8147730 2022-07-21T01:33:49.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 -- 172.21.15.3:0/1336883681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f8147730 msgr2=0x7f10f80acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 --2- 172.21.15.3:0/1336883681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f8147730 0x7f10f80acdf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f10f00032f0 tx=0x7f10f00091d0).stop 2022-07-21T01:33:49.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 -- 172.21.15.3:0/1336883681 shutdown_connections 2022-07-21T01:33:49.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 --2- 172.21.15.3:0/1336883681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f8147730 0x7f10f80acdf0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 --2- 172.21.15.3:0/1336883681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f80af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 -- 172.21.15.3:0/1336883681 >> 172.21.15.3:0/1336883681 conn(0x7f10f80626f0 msgr2=0x7f10f8062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:49.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 -- 172.21.15.3:0/1336883681 shutdown_connections 2022-07-21T01:33:49.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 -- 172.21.15.3:0/1336883681 wait complete. 2022-07-21T01:33:49.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 Processor -- start 2022-07-21T01:33:49.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 -- start start 2022-07-21T01:33:49.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f80b07a0 0x7f10f8144be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f8156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f8147730 0x7f10f8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80b4f20 con 0x7f10f80b07a0 2022-07-21T01:33:49.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8067080 con 0x7f10f8147730 2022-07-21T01:33:49.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fdf80700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8066b60 con 0x7f10f80b5630 2022-07-21T01:33:49.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fcf7e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f80b07a0 0x7f10f8144be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fcf7e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f80b07a0 0x7f10f8144be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43574/0 (socket says 172.21.15.3:43574) 2022-07-21T01:33:49.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fcf7e700 1 -- 172.21.15.3:0/4264161836 learned_addr learned my addr 172.21.15.3:0/4264161836 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:49.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10f7fff700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f8156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fd77f700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f8147730 0x7f10f8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fcf7e700 1 -- 172.21.15.3:0/4264161836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 msgr2=0x7f10f8156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fcf7e700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f8156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fcf7e700 1 -- 172.21.15.3:0/4264161836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f8147730 msgr2=0x7f10f8156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fcf7e700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f8147730 0x7f10f8156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fcf7e700 1 -- 172.21.15.3:0/4264161836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e400b180 con 0x7f10f80b07a0 2022-07-21T01:33:49.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10f7fff700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f8156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:49.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.255+0000 7f10fcf7e700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f80b07a0 0x7f10f8144be0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f10f814a6e0 tx=0x7f10e4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:49.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10f5ffb700 1 -- 172.21.15.3:0/4264161836 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10e400c890 con 0x7f10f80b07a0 2022-07-21T01:33:49.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10f5ffb700 1 -- 172.21.15.3:0/4264161836 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e400c9f0 con 0x7f10f80b07a0 2022-07-21T01:33:49.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f0009470 con 0x7f10f80b07a0 2022-07-21T01:33:49.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10f5ffb700 1 -- 172.21.15.3:0/4264161836 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10e400cd50 con 0x7f10f80b07a0 2022-07-21T01:33:49.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f80b4c80 con 0x7f10f80b07a0 2022-07-21T01:33:49.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f80574e0 con 0x7f10f80b07a0 2022-07-21T01:33:49.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10f5ffb700 1 -- 172.21.15.3:0/4264161836 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f10e401c460 con 0x7f10f80b07a0 2022-07-21T01:33:49.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10f5ffb700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10d8053aa0 0x7f10d8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10f5ffb700 1 -- 172.21.15.3:0/4264161836 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f10e406c6d0 con 0x7f10f80b07a0 2022-07-21T01:33:49.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10f7fff700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10d8053aa0 0x7f10d8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.259+0000 7f10f7fff700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10d8053aa0 0x7f10d8055f60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f10ec00b000 tx=0x7f10ec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:49.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.263+0000 7f10f5ffb700 1 -- 172.21.15.3:0/4264161836 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f10e4043320 con 0x7f10f80b07a0 2022-07-21T01:33:49.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.387+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10f805f3d0 con 0x7f10f80b07a0 2022-07-21T01:33:49.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.387+0000 7f10f5ffb700 1 -- 172.21.15.3:0/4264161836 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+10 (secure 0 0 0) 0x7f10e4046940 con 0x7f10f80b07a0 2022-07-21T01:33:49.387 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10d8053aa0 msgr2=0x7f10d8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10d8053aa0 0x7f10d8055f60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f10ec00b000 tx=0x7f10ec009040).stop 2022-07-21T01:33:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f80b07a0 msgr2=0x7f10f8144be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f80b07a0 0x7f10f8144be0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f10f814a6e0 tx=0x7f10e4006df0).stop 2022-07-21T01:33:49.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 shutdown_connections 2022-07-21T01:33:49.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10d8053aa0 0x7f10d8055f60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f8147730 0x7f10f8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f80b5630 0x7f10f8156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 --2- 172.21.15.3:0/4264161836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10f80b07a0 0x7f10f8144be0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 >> 172.21.15.3:0/4264161836 conn(0x7f10f80626f0 msgr2=0x7f10f80acd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:49.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 shutdown_connections 2022-07-21T01:33:49.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.391+0000 7f10fdf80700 1 -- 172.21.15.3:0/4264161836 wait complete. 2022-07-21T01:33:49.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:33:49.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847b331700 1 Processor -- start 2022-07-21T01:33:49.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847b331700 1 -- start start 2022-07-21T01:33:49.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847b331700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84740b6420 0x7f8474061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847b331700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b41f0 0x7f8474061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847b331700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8474061fe0 0x7f84741465c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847b331700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84740b60e0 con 0x7f84740b6420 2022-07-21T01:33:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847b331700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84740b2100 con 0x7f8474061fe0 2022-07-21T01:33:49.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847b331700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84740b23a0 con 0x7f84740b41f0 2022-07-21T01:33:49.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847a32f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84740b6420 0x7f8474061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847a32f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84740b6420 0x7f8474061600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43582/0 (socket says 172.21.15.3:43582) 2022-07-21T01:33:49.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847a32f700 1 -- 172.21.15.3:0/3882201881 learned_addr learned my addr 172.21.15.3:0/3882201881 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:49.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f847ab30700 1 --2- 172.21.15.3:0/3882201881 >> v2:172.21.15.143:3300/0 conn(0x7f8474061fe0 0x7f84741465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.523+0000 7f8479b2e700 1 --2- 172.21.15.3:0/3882201881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b41f0 0x7f8474061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847a32f700 1 -- 172.21.15.3:0/3882201881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b41f0 msgr2=0x7f8474061af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:49.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847a32f700 1 --2- 172.21.15.3:0/3882201881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b41f0 0x7f8474061af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847a32f700 1 -- 172.21.15.3:0/3882201881 >> v2:172.21.15.143:3300/0 conn(0x7f8474061fe0 msgr2=0x7f84741465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:49.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847a32f700 1 --2- 172.21.15.3:0/3882201881 >> v2:172.21.15.143:3300/0 conn(0x7f8474061fe0 0x7f84741465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847a32f700 1 -- 172.21.15.3:0/3882201881 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84740b2640 con 0x7f84740b6420 2022-07-21T01:33:49.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847ab30700 1 --2- 172.21.15.3:0/3882201881 >> v2:172.21.15.143:3300/0 conn(0x7f8474061fe0 0x7f84741465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:49.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847a32f700 1 --2- 172.21.15.3:0/3882201881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84740b6420 0x7f8474061600 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f8468002f70 tx=0x7f8468006eb0).ready entity=mon.0 client_cookie=bb869c0316c1b10f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:49.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847932d700 1 -- 172.21.15.3:0/3882201881 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f846800d570 con 0x7f84740b6420 2022-07-21T01:33:49.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847932d700 1 -- 172.21.15.3:0/3882201881 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8468000e90 con 0x7f84740b6420 2022-07-21T01:33:49.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847932d700 1 -- 172.21.15.3:0/3882201881 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f846800d900 con 0x7f84740b6420 2022-07-21T01:33:49.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 -- 172.21.15.3:0/3882201881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84740b6420 msgr2=0x7f8474061600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 --2- 172.21.15.3:0/3882201881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84740b6420 0x7f8474061600 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f8468002f70 tx=0x7f8468006eb0).stop 2022-07-21T01:33:49.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 -- 172.21.15.3:0/3882201881 shutdown_connections 2022-07-21T01:33:49.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 --2- 172.21.15.3:0/3882201881 >> v2:172.21.15.143:3300/0 conn(0x7f8474061fe0 0x7f84741465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 --2- 172.21.15.3:0/3882201881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b41f0 0x7f8474061af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 --2- 172.21.15.3:0/3882201881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84740b6420 0x7f8474061600 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 -- 172.21.15.3:0/3882201881 >> 172.21.15.3:0/3882201881 conn(0x7f847405c620 msgr2=0x7f847405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:49.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 -- 172.21.15.3:0/3882201881 shutdown_connections 2022-07-21T01:33:49.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 -- 172.21.15.3:0/3882201881 wait complete. 2022-07-21T01:33:49.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 Processor -- start 2022-07-21T01:33:49.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.527+0000 7f847b331700 1 -- start start 2022-07-21T01:33:49.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847b331700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8474061fe0 0x7f8474144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847b331700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84740b41f0 0x7f84740b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847b331700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b6420 0x7f8474141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847b331700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8474153300 con 0x7f8474061fe0 2022-07-21T01:33:49.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847b331700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84741535a0 con 0x7f84740b41f0 2022-07-21T01:33:49.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847b331700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84741438c0 con 0x7f84740b6420 2022-07-21T01:33:49.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847a32f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8474061fe0 0x7f8474144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847a32f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8474061fe0 0x7f8474144000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43588/0 (socket says 172.21.15.3:43588) 2022-07-21T01:33:49.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847a32f700 1 -- 172.21.15.3:0/2307098532 learned_addr learned my addr 172.21.15.3:0/2307098532 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:49.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847ab30700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b6420 0x7f8474141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f8479b2e700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84740b41f0 0x7f84740b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847a32f700 1 -- 172.21.15.3:0/2307098532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b6420 msgr2=0x7f8474141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847a32f700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b6420 0x7f8474141420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847a32f700 1 -- 172.21.15.3:0/2307098532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84740b41f0 msgr2=0x7f84740b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847a32f700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84740b41f0 0x7f84740b06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847a32f700 1 -- 172.21.15.3:0/2307098532 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c009470 con 0x7f8474061fe0 2022-07-21T01:33:49.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f8479b2e700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84740b41f0 0x7f84740b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:49.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847a32f700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8474061fe0 0x7f8474144000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f8468002f70 tx=0x7f8468015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:49.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f84677fe700 1 -- 172.21.15.3:0/2307098532 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84680117e0 con 0x7f8474061fe0 2022-07-21T01:33:49.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f84677fe700 1 -- 172.21.15.3:0/2307098532 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8468011940 con 0x7f8474061fe0 2022-07-21T01:33:49.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846800b1e0 con 0x7f8474061fe0 2022-07-21T01:33:49.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f84677fe700 1 -- 172.21.15.3:0/2307098532 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8468011ca0 con 0x7f8474061fe0 2022-07-21T01:33:49.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.531+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84740af5f0 con 0x7f8474061fe0 2022-07-21T01:33:49.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.535+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84740b28e0 con 0x7f8474061fe0 2022-07-21T01:33:49.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.535+0000 7f84677fe700 1 -- 172.21.15.3:0/2307098532 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8468025030 con 0x7f8474061fe0 2022-07-21T01:33:49.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.535+0000 7f84677fe700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8454053950 0x7f8454055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.535+0000 7f84677fe700 1 -- 172.21.15.3:0/2307098532 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f846806d2a0 con 0x7f8474061fe0 2022-07-21T01:33:49.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.535+0000 7f8479b2e700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8454053950 0x7f8454055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.535+0000 7f8479b2e700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8454053950 0x7f8454055e10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f847000b000 tx=0x7f8470008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:49.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.539+0000 7f84677fe700 1 -- 172.21.15.3:0/2307098532 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8468043f60 con 0x7f8474061fe0 2022-07-21T01:33:49.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.663+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84740574e0 con 0x7f8474061fe0 2022-07-21T01:33:49.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f84677fe700 1 -- 172.21.15.3:0/2307098532 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+10 (secure 0 0 0) 0x7f84680230a0 con 0x7f8474061fe0 2022-07-21T01:33:49.665 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:33:49.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8454053950 msgr2=0x7f8454055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8454053950 0x7f8454055e10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f847000b000 tx=0x7f8470008040).stop 2022-07-21T01:33:49.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8474061fe0 msgr2=0x7f8474144000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8474061fe0 0x7f8474144000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f8468002f70 tx=0x7f8468015000).stop 2022-07-21T01:33:49.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 shutdown_connections 2022-07-21T01:33:49.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84740b6420 0x7f8474141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8454053950 0x7f8454055e10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84740b41f0 0x7f84740b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 --2- 172.21.15.3:0/2307098532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8474061fe0 0x7f8474144000 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 >> 172.21.15.3:0/2307098532 conn(0x7f847405c620 msgr2=0x7f84740b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:49.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 shutdown_connections 2022-07-21T01:33:49.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.667+0000 7f847b331700 1 -- 172.21.15.3:0/2307098532 wait complete. 2022-07-21T01:33:49.688 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-21T01:33:49.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-07-21T01:33:49.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a6131700 1 Processor -- start 2022-07-21T01:33:49.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a6131700 1 -- start start 2022-07-21T01:33:49.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a6131700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f84a00b2710 0x7f84a014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a6131700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 0x7f84a014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a6131700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a00a7250 con 0x7f84a00b01a0 2022-07-21T01:33:49.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a6131700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a00a7480 con 0x7f84a00b2710 2022-07-21T01:33:49.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a6131700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00a7720 con 0x7f84a00a8c50 2022-07-21T01:33:49.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a492e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f84a00b01a0 0x7f84a00a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51878/0 (socket says 172.21.15.3:51878) 2022-07-21T01:33:49.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a492e700 1 -- 172.21.15.3:0/3100781742 learned_addr learned my addr 172.21.15.3:0/3100781742 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:49.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a512f700 1 --2- 172.21.15.3:0/3100781742 >> v2:172.21.15.143:3300/0 conn(0x7f84a00b2710 0x7f84a014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a5930700 1 --2- 172.21.15.3:0/3100781742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 0x7f84a014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f8497fff700 1 -- 172.21.15.3:0/3100781742 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935872811 0 0) 0x7f84a00a7250 con 0x7f84a00b01a0 2022-07-21T01:33:49.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f8497fff700 1 -- 172.21.15.3:0/3100781742 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8484003260 con 0x7f84a00b01a0 2022-07-21T01:33:49.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a512f700 1 -- 172.21.15.3:0/3100781742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 msgr2=0x7f84a014b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:49.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a512f700 1 --2- 172.21.15.3:0/3100781742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 0x7f84a014b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a512f700 1 -- 172.21.15.3:0/3100781742 >> v1:172.21.15.3:6789/0 conn(0x7f84a00b01a0 legacy=0x7f84a00a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a512f700 1 -- 172.21.15.3:0/3100781742 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a00a79c0 con 0x7f84a00b2710 2022-07-21T01:33:49.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f84a512f700 1 --2- 172.21.15.3:0/3100781742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a00b2710 0x7f84a014afd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f8490002f70 tx=0x7f8490006df0).ready entity=mon.1 client_cookie=52d5d0eadbb8007f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:49.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f8497fff700 1 -- 172.21.15.3:0/3100781742 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f849000d4a0 con 0x7f84a00b2710 2022-07-21T01:33:49.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f8497fff700 1 -- 172.21.15.3:0/3100781742 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8490000e90 con 0x7f84a00b2710 2022-07-21T01:33:49.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.807+0000 7f8497fff700 1 -- 172.21.15.3:0/3100781742 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f849000d7f0 con 0x7f84a00b2710 2022-07-21T01:33:49.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 -- 172.21.15.3:0/3100781742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a00b2710 msgr2=0x7f84a014afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 --2- 172.21.15.3:0/3100781742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a00b2710 0x7f84a014afd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f8490002f70 tx=0x7f8490006df0).stop 2022-07-21T01:33:49.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 -- 172.21.15.3:0/3100781742 shutdown_connections 2022-07-21T01:33:49.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 --2- 172.21.15.3:0/3100781742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 0x7f84a014b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 --2- 172.21.15.3:0/3100781742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a00b2710 0x7f84a014afd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 -- 172.21.15.3:0/3100781742 >> 172.21.15.3:0/3100781742 conn(0x7f84a0060310 msgr2=0x7f84a00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:49.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 -- 172.21.15.3:0/3100781742 shutdown_connections 2022-07-21T01:33:49.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 -- 172.21.15.3:0/3100781742 wait complete. 2022-07-21T01:33:49.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 Processor -- start 2022-07-21T01:33:49.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 -- start start 2022-07-21T01:33:49.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 0x7f84a0142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a00b01a0 0x7f84a00af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b2710 0x7f84a00b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00b3a70 con 0x7f84a00b2710 2022-07-21T01:33:49.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00b37d0 con 0x7f84a00b01a0 2022-07-21T01:33:49.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a6131700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00b3ef0 con 0x7f84a00a8c50 2022-07-21T01:33:49.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a5930700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b2710 0x7f84a00b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a5930700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b2710 0x7f84a00b5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43606/0 (socket says 172.21.15.3:43606) 2022-07-21T01:33:49.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a5930700 1 -- 172.21.15.3:0/2047776914 learned_addr learned my addr 172.21.15.3:0/2047776914 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:49.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a492e700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a00b01a0 0x7f84a00af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a512f700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 0x7f84a0142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a5930700 1 -- 172.21.15.3:0/2047776914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 msgr2=0x7f84a0142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a5930700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 0x7f84a0142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a5930700 1 -- 172.21.15.3:0/2047776914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a00b01a0 msgr2=0x7f84a00af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:49.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a5930700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a00b01a0 0x7f84a00af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:49.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.811+0000 7f84a5930700 1 -- 172.21.15.3:0/2047776914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849000b180 con 0x7f84a00b2710 2022-07-21T01:33:49.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.815+0000 7f84a5930700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b2710 0x7f84a00b5480 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f849800c000 tx=0x7f84980091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:49.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.815+0000 7f84967fc700 1 -- 172.21.15.3:0/2047776914 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8498012600 con 0x7f84a00b2710 2022-07-21T01:33:49.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.815+0000 7f84967fc700 1 -- 172.21.15.3:0/2047776914 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498012760 con 0x7f84a00b2710 2022-07-21T01:33:49.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.815+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a00b4190 con 0x7f84a00b2710 2022-07-21T01:33:49.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.815+0000 7f84967fc700 1 -- 172.21.15.3:0/2047776914 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8498012ac0 con 0x7f84a00b2710 2022-07-21T01:33:49.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.815+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a00aa060 con 0x7f84a00b2710 2022-07-21T01:33:49.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.815+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a00b4320 con 0x7f84a00b2710 2022-07-21T01:33:49.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.815+0000 7f84967fc700 1 -- 172.21.15.3:0/2047776914 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8498022070 con 0x7f84a00b2710 2022-07-21T01:33:49.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.815+0000 7f84967fc700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84880539e0 0x7f8488055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:49.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.819+0000 7f84967fc700 1 -- 172.21.15.3:0/2047776914 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f849806c600 con 0x7f84a00b2710 2022-07-21T01:33:49.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.819+0000 7f84a512f700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84880539e0 0x7f8488055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:49.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.819+0000 7f84a512f700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84880539e0 0x7f8488055ea0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f849001a000 tx=0x7f8490014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:49.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.823+0000 7f84967fc700 1 -- 172.21.15.3:0/2047776914 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8498043360 con 0x7f84a00b2710 2022-07-21T01:33:49.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:49.999+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7f84a00574e0 con 0x7f84a00b2710 2022-07-21T01:33:50.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.655+0000 7f84967fc700 1 -- 172.21.15.3:0/2047776914 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v43) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8498016020 con 0x7f84a00b2710 2022-07-21T01:33:50.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84880539e0 msgr2=0x7f8488055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:50.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84880539e0 0x7f8488055ea0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f849001a000 tx=0x7f8490014000).stop 2022-07-21T01:33:50.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b2710 msgr2=0x7f84a00b5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:50.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b2710 0x7f84a00b5480 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f849800c000 tx=0x7f84980091d0).stop 2022-07-21T01:33:50.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 shutdown_connections 2022-07-21T01:33:50.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b2710 0x7f84a00b5480 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84880539e0 0x7f8488055ea0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a00b01a0 0x7f84a00af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 --2- 172.21.15.3:0/2047776914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00a8c50 0x7f84a0142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 >> 172.21.15.3:0/2047776914 conn(0x7f84a0060310 msgr2=0x7f84a0148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:50.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 shutdown_connections 2022-07-21T01:33:50.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.659+0000 7f84a6131700 1 -- 172.21.15.3:0/2047776914 wait complete. 2022-07-21T01:33:50.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:33:50.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.787+0000 7fc71aa34700 1 Processor -- start 2022-07-21T01:33:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.787+0000 7fc71aa34700 1 -- start start 2022-07-21T01:33:50.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 0x7fc7140b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:50.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc71413f520 0x7fc7140aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:50.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71405cd40 con 0x7fc71405c960 2022-07-21T01:33:50.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7140ad6d0 con 0x7fc71413f520 2022-07-21T01:33:50.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc714067f00 con 0x7fc71405d240 2022-07-21T01:33:50.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc719231700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 0x7fc7140b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:50.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71a233700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc71413f520 0x7fc7140aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:50.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc719a32700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fc71405c960 0x7fc7140acfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:54006/0 (socket says 172.21.15.3:54006) 2022-07-21T01:33:50.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc719a32700 1 -- 172.21.15.3:0/3607117411 learned_addr learned my addr 172.21.15.3:0/3607117411 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc718a30700 1 -- 172.21.15.3:0/3607117411 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601015657 0 0) 0x7fc71405cd40 con 0x7fc71405c960 2022-07-21T01:33:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc718a30700 1 -- 172.21.15.3:0/3607117411 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6fc003260 con 0x7fc71405c960 2022-07-21T01:33:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc719231700 1 -- 172.21.15.3:0/3607117411 >> v2:172.21.15.143:3300/0 conn(0x7fc71413f520 msgr2=0x7fc7140aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc719231700 1 --2- 172.21.15.3:0/3607117411 >> v2:172.21.15.143:3300/0 conn(0x7fc71413f520 0x7fc7140aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc719231700 1 -- 172.21.15.3:0/3607117411 >> v1:172.21.15.143:6789/0 conn(0x7fc71405c960 legacy=0x7fc7140acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc719231700 1 -- 172.21.15.3:0/3607117411 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7140b1920 con 0x7fc71405d240 2022-07-21T01:33:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc719231700 1 --2- 172.21.15.3:0/3607117411 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 0x7fc7140b1280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc710002f70 tx=0x7fc710006df0).ready entity=mon.2 client_cookie=d62bc61acf0a80fb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc718a30700 1 -- 172.21.15.3:0/3607117411 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71000d4a0 con 0x7fc71405d240 2022-07-21T01:33:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc718a30700 1 -- 172.21.15.3:0/3607117411 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc710000e90 con 0x7fc71405d240 2022-07-21T01:33:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc718a30700 1 -- 172.21.15.3:0/3607117411 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71000d7f0 con 0x7fc71405d240 2022-07-21T01:33:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 -- 172.21.15.3:0/3607117411 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 msgr2=0x7fc7140b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 --2- 172.21.15.3:0/3607117411 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 0x7fc7140b1280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc710002f70 tx=0x7fc710006df0).stop 2022-07-21T01:33:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 -- 172.21.15.3:0/3607117411 shutdown_connections 2022-07-21T01:33:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 --2- 172.21.15.3:0/3607117411 >> v2:172.21.15.143:3300/0 conn(0x7fc71413f520 0x7fc7140aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 --2- 172.21.15.3:0/3607117411 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 0x7fc7140b1280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 -- 172.21.15.3:0/3607117411 >> 172.21.15.3:0/3607117411 conn(0x7fc714060310 msgr2=0x7fc7140b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 -- 172.21.15.3:0/3607117411 shutdown_connections 2022-07-21T01:33:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.791+0000 7fc71aa34700 1 -- 172.21.15.3:0/3607117411 wait complete. 2022-07-21T01:33:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71aa34700 1 Processor -- start 2022-07-21T01:33:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71aa34700 1 -- start start 2022-07-21T01:33:50.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71aa34700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc71405c960 0x7fc714152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:50.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71aa34700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 0x7fc7140b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:50.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71aa34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71413f520 0x7fc714141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:50.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71aa34700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714063550 con 0x7fc71405c960 2022-07-21T01:33:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71aa34700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714066a50 con 0x7fc71413f520 2022-07-21T01:33:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71aa34700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140ab080 con 0x7fc71405d240 2022-07-21T01:33:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc719a32700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc71405c960 0x7fc714152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc719a32700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc71405c960 0x7fc714152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43616/0 (socket says 172.21.15.3:43616) 2022-07-21T01:33:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc719a32700 1 -- 172.21.15.3:0/2721190624 learned_addr learned my addr 172.21.15.3:0/2721190624 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc719231700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 0x7fc7140b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71a233700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71413f520 0x7fc714141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71a233700 1 -- 172.21.15.3:0/2721190624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 msgr2=0x7fc7140b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:50.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71a233700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 0x7fc7140b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71a233700 1 -- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc71405c960 msgr2=0x7fc714152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:50.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71a233700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc71405c960 0x7fc714152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71a233700 1 -- 172.21.15.3:0/2721190624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71000b180 con 0x7fc71413f520 2022-07-21T01:33:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc719a32700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc71405c960 0x7fc714152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71a233700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71413f520 0x7fc714141c10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc7140aa920 tx=0x7fc70c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc702ffd700 1 -- 172.21.15.3:0/2721190624 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc70c012600 con 0x7fc71413f520 2022-07-21T01:33:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc702ffd700 1 -- 172.21.15.3:0/2721190624 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c012760 con 0x7fc71413f520 2022-07-21T01:33:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.795+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc714143510 con 0x7fc71413f520 2022-07-21T01:33:50.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.799+0000 7fc702ffd700 1 -- 172.21.15.3:0/2721190624 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc70c012ac0 con 0x7fc71413f520 2022-07-21T01:33:50.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.799+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7140ab4b0 con 0x7fc71413f520 2022-07-21T01:33:50.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.799+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7140ab1f0 con 0x7fc71413f520 2022-07-21T01:33:50.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.799+0000 7fc702ffd700 1 -- 172.21.15.3:0/2721190624 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc70c012c20 con 0x7fc71413f520 2022-07-21T01:33:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.799+0000 7fc702ffd700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6f40539e0 0x7fc6f4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.799+0000 7fc719a32700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6f40539e0 0x7fc6f4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.799+0000 7fc702ffd700 1 -- 172.21.15.3:0/2721190624 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc70c06c160 con 0x7fc71413f520 2022-07-21T01:33:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.799+0000 7fc719a32700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6f40539e0 0x7fc6f4055ea0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc70400d000 tx=0x7fc704009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:50.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.807+0000 7fc702ffd700 1 -- 172.21.15.3:0/2721190624 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc70c042f20 con 0x7fc71413f520 2022-07-21T01:33:50.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.934+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc7140b1690 con 0x7fc71413f520 2022-07-21T01:33:50.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.942+0000 7fc702ffd700 1 -- 172.21.15.3:0/2721190624 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc70c0160c0 con 0x7fc71413f520 2022-07-21T01:33:50.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6f40539e0 msgr2=0x7fc6f4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:50.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6f40539e0 0x7fc6f4055ea0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc70400d000 tx=0x7fc704009040).stop 2022-07-21T01:33:50.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71413f520 msgr2=0x7fc714141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:50.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71413f520 0x7fc714141c10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc7140aa920 tx=0x7fc70c0091d0).stop 2022-07-21T01:33:50.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 shutdown_connections 2022-07-21T01:33:50.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6f40539e0 0x7fc6f4055ea0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71413f520 0x7fc714141c10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc71405d240 0x7fc7140b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 --2- 172.21.15.3:0/2721190624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc71405c960 0x7fc714152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:50.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 >> 172.21.15.3:0/2721190624 conn(0x7fc714060310 msgr2=0x7fc7141335b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:50.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 shutdown_connections 2022-07-21T01:33:50.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:50.946+0000 7fc71aa34700 1 -- 172.21.15.3:0/2721190624 wait complete. 2022-07-21T01:33:50.960 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:33:50.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:33:50.978 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:33:51.000 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:33:51.022 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:33:51.045 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:33:51.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:51.047 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:51.048 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:51.057 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:33:51.059 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:33:51.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:51.061 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:51.061 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:51.107 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:33:51.110 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:33:51.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:51.111 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:51.111 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:51.119 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-21T01:33:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:33:51.120 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-21T01:33:51.121 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:33:51.121 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:33:51.122 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:51.161 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:51.161 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:51.162 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:51.210 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:33:51.212 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:33:51.212 INFO:teuthology.orchestra.run:waiting for 900 2022-07-21T01:33:51.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:33:51.213 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:33:51.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:51.233 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:51.234 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:51.242 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:33:51.243 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:33:51.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:51.244 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:51.245 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:51.292 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:33:51.293 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:33:51.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:51.295 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:51.295 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:51.303 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': Transport endpoint is not connected 2022-07-21T01:33:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:33:51.304 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.1 2022-07-21T01:33:51.305 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:33:51.305 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:33:51.305 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:51.328 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:33:51.328 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:33:51.329 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:51.375 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:33:51.376 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:33:51.377 INFO:teuthology.orchestra.run:waiting for 900 2022-07-21T01:33:51.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:33:51.378 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:33:51.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:33:51.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf3d32700 1 Processor -- start 2022-07-21T01:33:51.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf3d32700 1 -- start start 2022-07-21T01:33:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf3d32700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1bec0b1f40 0x7f1bec0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf3d32700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec148530 0x7f1bec0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:51.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf3d32700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bec0b04e0 con 0x7f1bec0a7250 2022-07-21T01:33:51.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf3d32700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bec0a9250 con 0x7f1bec0b1f40 2022-07-21T01:33:51.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf3d32700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec0b0780 con 0x7f1bec148530 2022-07-21T01:33:51.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf22cf700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1bec0a7250 0x7f1bec1448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51906/0 (socket says 172.21.15.3:51906) 2022-07-21T01:33:51.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf22cf700 1 -- 172.21.15.3:0/1867035183 learned_addr learned my addr 172.21.15.3:0/1867035183 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:51.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf1ace700 1 --2- 172.21.15.3:0/1867035183 >> v2:172.21.15.143:3300/0 conn(0x7f1bec0b1f40 0x7f1bec0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:51.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf12cd700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec148530 0x7f1bec0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:51.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738981935 0 0) 0x7f1bec0b04e0 con 0x7f1bec0a7250 2022-07-21T01:33:51.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bcc003260 con 0x7f1bec0a7250 2022-07-21T01:33:51.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1912759577 0 0) 0x7f1bcc003260 con 0x7f1bec0a7250 2022-07-21T01:33:51.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec148530 msgr2=0x7f1bec0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:51.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec148530 0x7f1bec0b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 >> v2:172.21.15.143:3300/0 conn(0x7f1bec0b1f40 msgr2=0x7f1bec0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:51.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 --2- 172.21.15.3:0/1867035183 >> v2:172.21.15.143:3300/0 conn(0x7f1bec0b1f40 0x7f1bec0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec0b0a20 con 0x7f1bec0a7250 2022-07-21T01:33:51.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f1be4001660 con 0x7f1bec0a7250 2022-07-21T01:33:51.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 >> v1:172.21.15.3:6789/0 conn(0x7f1bec0a7250 legacy=0x7f1bec1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:51.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec148530 0x7f1bcc003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:51.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bcc003c80 0x7f1bcc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:51.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bcc0049f0 0x7f1bcc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:51.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec0b0780 con 0x7f1bcc0049f0 2022-07-21T01:33:51.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc00b750 con 0x7f1bcc003c80 2022-07-21T01:33:51.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc00b890 con 0x7f1bec148530 2022-07-21T01:33:51.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf12cd700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bcc003c80 0x7f1bcc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:51.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf1ace700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec148530 0x7f1bcc003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:51.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf22cf700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bcc0049f0 0x7f1bcc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:51.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf1ace700 1 -- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bcc003c80 msgr2=0x7f1bcc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:51.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf1ace700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bcc003c80 0x7f1bcc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf1ace700 1 -- 172.21.15.3:0/1867035183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bcc0049f0 msgr2=0x7f1bcc004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:51.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf1ace700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bcc0049f0 0x7f1bcc004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf1ace700 1 -- 172.21.15.3:0/1867035183 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc003260 con 0x7f1bec148530 2022-07-21T01:33:51.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf22cf700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bcc0049f0 0x7f1bcc004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:51.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.502+0000 7f1bf12cd700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bcc003c80 0x7f1bcc004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:51.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf1ace700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec148530 0x7f1bcc003750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f1bdc00c000 tx=0x7f1bdc0091d0).ready entity=mon.2 client_cookie=4b033505531035fc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:51.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bdc00e7e0 con 0x7f1bec148530 2022-07-21T01:33:51.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bdc00e940 con 0x7f1bec148530 2022-07-21T01:33:51.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf0acc700 1 -- 172.21.15.3:0/1867035183 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bdc00eca0 con 0x7f1bec148530 2022-07-21T01:33:51.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf3d32700 1 -- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec148530 msgr2=0x7f1bcc003750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:51.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf3d32700 1 --2- 172.21.15.3:0/1867035183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec148530 0x7f1bcc003750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f1bdc00c000 tx=0x7f1bdc0091d0).stop 2022-07-21T01:33:51.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf22cf700 1 -- 172.21.15.3:0/1867035183 reap_dead start 2022-07-21T01:33:51.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf3d32700 1 -- 172.21.15.3:0/1867035183 shutdown_connections 2022-07-21T01:33:51.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf3d32700 1 -- 172.21.15.3:0/1867035183 >> 172.21.15.3:0/1867035183 conn(0x7f1bec05f840 msgr2=0x7f1bec05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:51.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf3d32700 1 -- 172.21.15.3:0/1867035183 shutdown_connections 2022-07-21T01:33:51.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.506+0000 7f1bf3d32700 1 -- 172.21.15.3:0/1867035183 wait complete. 2022-07-21T01:33:51.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf3d32700 1 Processor -- start 2022-07-21T01:33:51.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf3d32700 1 -- start start 2022-07-21T01:33:51.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf3d32700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bec148530 0x7f1bec14afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:51.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf3d32700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bec142260 0x7f1bec145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:51.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf3d32700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec0adf50 0x7f1bec0ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:51.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf3d32700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec145850 con 0x7f1bec142260 2022-07-21T01:33:51.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf3d32700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec0ac810 con 0x7f1bec148530 2022-07-21T01:33:51.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf3d32700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec146760 con 0x7f1bec0adf50 2022-07-21T01:33:51.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf12cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bec142260 0x7f1bec145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:51.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf12cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bec142260 0x7f1bec145e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43638/0 (socket says 172.21.15.3:43638) 2022-07-21T01:33:51.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf12cd700 1 -- 172.21.15.3:0/3101233967 learned_addr learned my addr 172.21.15.3:0/3101233967 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:51.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf22cf700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec0adf50 0x7f1bec0ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:51.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf1ace700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bec148530 0x7f1bec14afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:51.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf12cd700 1 -- 172.21.15.3:0/3101233967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec0adf50 msgr2=0x7f1bec0ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:51.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf12cd700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec0adf50 0x7f1bec0ae3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf12cd700 1 -- 172.21.15.3:0/3101233967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bec148530 msgr2=0x7f1bec14afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:51.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf12cd700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bec148530 0x7f1bec14afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf12cd700 1 -- 172.21.15.3:0/3101233967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec0aca40 con 0x7f1bec142260 2022-07-21T01:33:51.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf12cd700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bec142260 0x7f1bec145e50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f1bec146d40 tx=0x7f1bd4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:51.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1beaffd700 1 -- 172.21.15.3:0/3101233967 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bd400fa10 con 0x7f1bec142260 2022-07-21T01:33:51.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1beaffd700 1 -- 172.21.15.3:0/3101233967 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd400fb70 con 0x7f1bec142260 2022-07-21T01:33:51.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bec1469c0 con 0x7f1bec142260 2022-07-21T01:33:51.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.510+0000 7f1beaffd700 1 -- 172.21.15.3:0/3101233967 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bd400fce0 con 0x7f1bec142260 2022-07-21T01:33:51.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.514+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bec142940 con 0x7f1bec142260 2022-07-21T01:33:51.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.514+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bec0b0cc0 con 0x7f1bec142260 2022-07-21T01:33:51.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.514+0000 7f1beaffd700 1 -- 172.21.15.3:0/3101233967 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1bd4024070 con 0x7f1bec142260 2022-07-21T01:33:51.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.514+0000 7f1beaffd700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1bd0053a70 0x7f1bd0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:51.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.514+0000 7f1beaffd700 1 -- 172.21.15.3:0/3101233967 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1bd406cab0 con 0x7f1bec142260 2022-07-21T01:33:51.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.514+0000 7f1bf1ace700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1bd0053a70 0x7f1bd0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:51.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.514+0000 7f1bf1ace700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1bd0053a70 0x7f1bd0055f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f1bdc01e000 tx=0x7f1bdc01a000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:51.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.518+0000 7f1beaffd700 1 -- 172.21.15.3:0/3101233967 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1bd4043870 con 0x7f1bec142260 2022-07-21T01:33:51.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.650+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f1bec0574e0 con 0x7f1bec142260 2022-07-21T01:33:51.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.650+0000 7f1beaffd700 1 -- 172.21.15.3:0/3101233967 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v26) v1 ==== 81+0+34065 (secure 0 0 0) 0x7f1bd4046e90 con 0x7f1bec142260 2022-07-21T01:33:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:51.653 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:33:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 26, 2022-07-21T01:33:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:33:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:33:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:33:26.370831+0000", 2022-07-21T01:33:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:33:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:33:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:33:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:33:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:33:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:33:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:33:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:33:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:33:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:33:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:33:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:33:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:33:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:33:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:33:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 4, 2022-07-21T01:33:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:33:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:33:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:33:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:33:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:33:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:33:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:33:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:33:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:33:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:33:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:33:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:33:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:33:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:33:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:33:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:33:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:33:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:33:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:33:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:33:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:33:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:33:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:33:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:33:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:33:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:33:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:33:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:33:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:33:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:33:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:33:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:33:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:33:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:33:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:33:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:33:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:33:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:33:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:33:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:33:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:33:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:33:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:33:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:33:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:33:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:33:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:33:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:33:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:33:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:33:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:33:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:33:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:33:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:33:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:33:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:33:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:33:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:33:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:33:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:33:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:33:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:33:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:33:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:33:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:33:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:33:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:33:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:33:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:33:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:33:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:33:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:33:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:33:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:33:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:33:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:33:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:33:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:33:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:33:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:33:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:33:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:33:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:33:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:33:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:33:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:33:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:33:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:33:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:33:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:33:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:33:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:33:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:33:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:33:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:33:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:33:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:33:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:33:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:33:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:33:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:33:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:33:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:33:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:33:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:33:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:33:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:33:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:33:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:33:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:33:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:33:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:33:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:33:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:33:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:33:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:33:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:33:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:33:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:33:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:33:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:33:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:33:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:33:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:33:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:33:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:33:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:33:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:33:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:33:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:33:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:33:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:33:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:33:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:33:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 3, 2022-07-21T01:33:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:33:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:21.299936+0000", 2022-07-21T01:33:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:33:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:33:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:33:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:33:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:33:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:33:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:33:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:33:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:33:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:33:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:33:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:33:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:33:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:33:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:33:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:33:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:33:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:33:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:33:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:33:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "26", 2022-07-21T01:33:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:33:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:33:51.720 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:33:51.721 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:33:51.721 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:33:51.721 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:33:51.721 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:33:51.722 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:33:51.722 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:33:51.722 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:33:51.722 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:33:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:33:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:33:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:33:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:33:51.724 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:33:51.724 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:33:51.724 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:33:51.724 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:33:51.725 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:33:51.725 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:33:51.725 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:33:51.726 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:33:51.726 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:33:51.726 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:33:51.727 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:33:51.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.727 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:33:51.727 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:33:51.728 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:33:51.728 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:33:51.728 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:33:51.728 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:33:51.729 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:33:51.729 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:33:51.729 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:33:51.730 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:33:51.730 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:33:51.730 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:33:51.730 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:33:51.731 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:33:51.731 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:33:51.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.731 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:33:51.732 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:33:51.732 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:33:51.732 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.732 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.733 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 4, 2022-07-21T01:33:51.733 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:33:51.734 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:24.075599+0000", 2022-07-21T01:33:51.734 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 5, 2022-07-21T01:33:51.734 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool,ec_overwrites", 2022-07-21T01:33:51.734 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 3, 2022-07-21T01:33:51.735 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 4, 2022-07-21T01:33:51.735 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 3, 2022-07-21T01:33:51.735 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 1, 2022-07-21T01:33:51.735 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:33:51.735 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:33:51.736 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:33:51.736 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:33:51.736 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:33:51.736 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:33:51.737 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:33:51.737 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:33:51.737 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:33:51.737 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:33:51.738 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:33:51.738 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:33:51.738 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:33:51.738 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:33:51.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.739 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "26", 2022-07-21T01:33:51.739 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:33:51.739 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:33:51.740 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:33:51.740 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:33:51.740 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:33:51.740 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:33:51.741 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:33:51.741 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:33:51.741 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:33:51.742 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:33:51.742 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:33:51.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:33:51.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:33:51.743 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:33:51.743 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:33:51.743 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:33:51.743 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:33:51.744 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:33:51.744 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:33:51.744 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:33:51.744 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:33:51.745 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:33:51.745 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:33:51.745 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "cephfs_data", 2022-07-21T01:33:51.745 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:33:51.746 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:33:51.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.746 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:33:51.746 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:33:51.746 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:33:51.747 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:33:51.747 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:33:51.747 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:33:51.747 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:33:51.748 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:33:51.748 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 8192, 2022-07-21T01:33:51.748 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:33:51.748 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:33:51.749 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:33:51.749 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:33:51.749 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:33:51.749 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:33:51.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.750 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:33:51.751 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:33:51.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.751 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:33:51.751 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:33:51.752 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:33:51.752 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:33:51.752 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:33:51.752 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:33:51.753 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:33:51.753 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:33:51.753 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:33:51.753 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 24, 2022-07-21T01:33:51.754 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:33:51.754 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:33:51.754 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:33:51.754 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.755 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.755 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:33:51.755 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:33:51.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.756 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.757 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:33:51.757 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:33:51.757 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.757 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.758 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:33:51.758 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.759 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.759 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:33:51.759 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:33:51.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.760 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:33:51.761 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:33:51.761 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.761 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.762 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:33:51.762 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.762 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.763 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:33:51.763 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:33:51.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.764 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.764 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:33:51.764 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:33:51.765 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.765 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.765 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:33:51.766 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.766 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.766 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:33:51.767 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:33:51.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.767 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.768 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:33:51.768 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:33:51.768 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.768 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.769 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:33:51.769 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:33:51.769 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:33:51.769 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:33:51.770 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:33:51.770 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:33:51.770 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:33:51.770 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:33:51.771 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:33:51.772 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:33:51.772 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:33:51.772 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:33:51.772 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:33:51.773 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:33:51.773 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:33:51.773 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:33:51.773 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 24, 2022-07-21T01:33:51.774 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:33:51.774 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:33:51.774 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:33:51.774 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.775 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.775 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:33:51.775 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:33:51.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.776 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.776 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:33:51.777 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:33:51.777 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.777 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.778 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:33:51.778 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.779 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.779 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:33:51.779 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:33:51.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.780 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.780 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:33:51.780 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:33:51.781 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.781 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.781 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:33:51.782 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.782 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.782 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:33:51.783 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:33:51.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.784 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.784 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:33:51.784 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:33:51.784 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.785 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.785 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:33:51.785 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.786 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.786 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:33:51.786 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:33:51.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.787 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.787 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:33:51.788 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:33:51.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.789 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:33:51.789 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:33:51.789 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:33:51.789 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:33:51.790 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:33:51.790 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:33:51.790 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:33:51.790 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.791 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:33:51.792 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:33:51.792 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:33:51.792 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:33:51.792 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:33:51.793 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:33:51.793 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:33:51.793 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:33:51.793 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:33:51.794 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 18, 2022-07-21T01:33:51.794 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:33:51.794 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:33:51.794 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:33:51.795 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.795 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.795 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:33:51.796 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:33:51.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.796 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.797 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:33:51.797 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:33:51.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:33:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.799 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.799 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:33:51.800 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:33:51.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.800 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.801 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:33:51.801 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:33:51.801 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.801 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.802 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:33:51.802 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.803 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.803 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:33:51.803 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:33:51.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.804 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.804 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:33:51.805 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:33:51.805 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.805 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.806 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:33:51.806 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.807 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.807 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:33:51.807 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:33:51.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.808 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.808 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:33:51.808 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:33:51.809 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.809 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.810 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:33:51.810 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:33:51.810 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:33:51.810 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:33:51.811 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:33:51.811 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:33:51.811 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:33:51.811 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.812 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:33:51.813 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:33:51.813 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:33:51.813 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:33:51.813 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:33:51.814 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:33:51.814 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:33:51.814 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:33:51.814 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:33:51.815 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 24, 2022-07-21T01:33:51.815 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:33:51.815 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:33:51.815 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:33:51.816 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.816 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.816 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:33:51.817 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:33:51.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.817 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.818 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:33:51.818 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:33:51.818 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.818 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.819 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:33:51.819 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.820 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.820 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:33:51.820 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:33:51.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.821 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.821 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:33:51.821 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:33:51.822 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.822 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.822 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:33:51.823 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.823 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.823 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:33:51.824 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:33:51.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.824 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.825 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:33:51.825 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:33:51.825 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.825 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.826 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:33:51.826 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.827 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:33:51.827 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:33:51.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.828 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.828 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:33:51.829 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:33:51.829 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.829 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.830 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:33:51.830 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:33:51.830 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:33:51.830 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:33:51.831 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:33:51.831 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:33:51.831 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:33:51.831 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.832 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:33:51.832 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:33:51.833 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:33:51.833 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:33:51.833 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:33:51.833 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:33:51.834 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:33:51.834 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:33:51.834 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:33:51.834 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 24, 2022-07-21T01:33:51.835 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:33:51.835 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:33:51.835 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:33:51.835 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:33:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:33:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:33:51.838 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:33:51.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.838 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.839 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:33:51.839 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.839 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.840 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:33:51.840 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:33:51.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.841 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.841 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:33:51.841 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:33:51.841 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.842 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.842 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:33:51.843 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.843 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.843 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:33:51.844 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:33:51.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.844 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.845 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:33:51.845 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:33:51.845 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.845 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.846 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:33:51.846 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.847 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.847 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:33:51.847 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:33:51.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.848 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.848 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:33:51.848 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:33:51.849 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.849 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.849 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:33:51.850 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:33:51.850 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:33:51.850 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:33:51.851 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:33:51.851 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:33:51.851 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:33:51.851 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.852 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:33:51.852 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:33:51.853 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:33:51.853 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:33:51.853 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:33:51.853 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:33:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:33:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:33:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:33:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 21, 2022-07-21T01:33:51.855 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:33:51.855 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:33:51.855 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:33:51.855 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.856 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.856 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:33:51.856 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:33:51.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:33:51.858 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:33:51.858 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.858 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.859 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:33:51.859 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.859 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:33:51.860 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:33:51.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.861 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:33:51.861 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:33:51.861 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.862 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.862 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:33:51.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:33:51.863 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:33:51.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.864 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.864 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:33:51.865 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:33:51.865 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.865 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.866 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:33:51.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.866 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:33:51.867 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:33:51.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.868 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.868 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:33:51.868 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:33:51.869 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.869 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.869 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:33:51.870 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:33:51.870 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:33:51.870 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:33:51.870 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:33:51.871 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:33:51.871 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:33:51.871 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.872 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:33:51.872 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:33:51.872 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:33:51.873 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:33:51.873 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:33:51.873 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:33:51.873 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:33:51.874 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:33:51.874 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:33:51.874 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 24, 2022-07-21T01:33:51.874 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:33:51.875 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:33:51.875 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:33:51.875 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.876 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.876 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:33:51.876 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:33:51.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.877 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.877 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:33:51.877 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:33:51.878 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.878 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.878 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:33:51.879 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.879 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.880 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:33:51.880 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:33:51.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.881 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.881 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:33:51.881 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:33:51.881 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.882 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.882 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:33:51.882 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.883 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.883 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:33:51.883 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:33:51.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.884 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.884 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:33:51.885 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:33:51.885 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.885 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.886 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:33:51.886 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.887 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.887 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:33:51.887 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:33:51.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:33:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:33:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:33:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:33:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:33:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:33:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:33:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:33:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:33:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:33:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:33:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:33:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:33:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:33:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:33:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:33:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:33:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:33:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 18, 2022-07-21T01:33:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:33:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:33:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:33:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.896 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.896 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:33:51.896 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:33:51.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.897 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.897 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:33:51.898 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:33:51.898 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.898 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.899 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:33:51.899 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.899 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.900 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:33:51.900 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:33:51.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.901 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.901 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:33:51.901 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:33:51.901 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.902 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.902 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:33:51.903 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.903 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.903 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:33:51.904 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:33:51.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.904 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.905 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:33:51.905 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:33:51.905 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.905 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.906 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:33:51.906 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:33:51.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.907 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:33:51.907 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:33:51.907 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:33:51.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.908 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:33:51.908 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:33:51.909 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:33:51.909 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.909 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.910 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:33:51.910 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:33:51.910 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:33:51.910 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:33:51.911 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:33:51.911 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:33:51.911 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:33:51.911 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:51.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.912 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:33:51.912 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:33:51.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.913 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:33:51.913 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:33:51.913 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:33:51.913 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:33:51.914 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:33:51.914 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:33:51.914 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:33:51.914 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:33:51.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.915 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:33:51.915 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:33:51.916 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:33:51.916 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:33:51.916 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:33:51.916 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:33:51.917 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:33:51.917 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:33:51.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.918 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:33:51.918 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:33:51.918 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:33:51.918 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:33:51.919 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:33:51.919 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:33:51.919 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:33:51.919 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:33:51.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.920 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:33:51.921 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:33:51.921 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:33:51.921 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:33:51.921 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:33:51.922 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:33:51.922 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:33:51.922 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:33:51.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.923 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:33:51.923 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:33:51.923 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:33:51.924 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:33:51.924 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:33:51.924 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:33:51.924 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:33:51.925 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:33:51.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.925 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:33:51.926 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:33:51.926 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:33:51.926 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:33:51.927 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:33:51.927 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:33:51.927 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:33:51.927 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:33:51.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.928 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:33:51.928 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:33:51.929 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:33:51.929 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:33:51.929 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:33:51.929 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:33:51.930 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:33:51.930 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:33:51.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:51.931 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:33:51.931 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:33:51.931 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:33:51.931 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:33:51.932 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:33:51.932 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:33:51.932 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:33:51.932 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:33:51.933 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.933 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:33:51.933 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:33:51.933 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:33:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:33:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:33:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:33:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:33:51.935 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:33:51.935 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:33:51.935 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:33:51.935 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:33:51.936 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:33:51.936 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:33:51.936 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:33:51.936 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:33:51.937 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:33:51.937 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:33:51.937 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:33:51.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.938 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:33:51.938 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:33:51.938 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:33:51.939 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:33:51.939 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:33:51.939 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:33:51.939 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:33:51.940 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:51.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:51.940 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:33:51.940 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:33:51.941 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:33:51.941 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:33:51.941 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:33:51.941 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:33:51.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1bd0053a70 msgr2=0x7f1bd0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:51.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1bd0053a70 0x7f1bd0055f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f1bdc01e000 tx=0x7f1bdc01a000).stop 2022-07-21T01:33:51.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bec142260 msgr2=0x7f1bec145e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:51.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bec142260 0x7f1bec145e50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f1bec146d40 tx=0x7f1bd4006df0).stop 2022-07-21T01:33:51.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 shutdown_connections 2022-07-21T01:33:51.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bec0adf50 0x7f1bec0ae3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bec142260 0x7f1bec145e50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1bd0053a70 0x7f1bd0055f30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 --2- 172.21.15.3:0/3101233967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bec148530 0x7f1bec14afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:51.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 >> 172.21.15.3:0/3101233967 conn(0x7f1bec05f840 msgr2=0x7f1bec142ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:51.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 shutdown_connections 2022-07-21T01:33:51.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:51.654+0000 7f1bf3d32700 1 -- 172.21.15.3:0/3101233967 wait complete. 2022-07-21T01:33:51.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:33:52.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfbfff700 1 Processor -- start 2022-07-21T01:33:52.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfbfff700 1 -- start start 2022-07-21T01:33:52.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b01a0 0x7f1dfc0a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfbfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1dfc0afd60 0x7f1dfc0a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfbfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dfc0a7250 con 0x7f1dfc0b2710 2022-07-21T01:33:52.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfbfff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dfc0a7480 con 0x7f1dfc0afd60 2022-07-21T01:33:52.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc0a7720 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfaffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1dfc0b2710 0x7f1dfc14afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51926/0 (socket says 172.21.15.3:51926) 2022-07-21T01:33:52.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfaffd700 1 -- 172.21.15.3:0/1235414561 learned_addr learned my addr 172.21.15.3:0/1235414561 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:52.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfa7fc700 1 --2- 172.21.15.3:0/1235414561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b01a0 0x7f1dfc0a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.062+0000 7f1dfb7fe700 1 --2- 172.21.15.3:0/1235414561 >> v2:172.21.15.143:3300/0 conn(0x7f1dfc0afd60 0x7f1dfc0a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1df9ffb700 1 -- 172.21.15.3:0/1235414561 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516523 0 0) 0x7f1dfc0a7250 con 0x7f1dfc0b2710 2022-07-21T01:33:52.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1df9ffb700 1 -- 172.21.15.3:0/1235414561 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1de0003260 con 0x7f1dfc0b2710 2022-07-21T01:33:52.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfa7fc700 1 -- 172.21.15.3:0/1235414561 >> v2:172.21.15.143:3300/0 conn(0x7f1dfc0afd60 msgr2=0x7f1dfc0a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:52.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfa7fc700 1 --2- 172.21.15.3:0/1235414561 >> v2:172.21.15.143:3300/0 conn(0x7f1dfc0afd60 0x7f1dfc0a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfa7fc700 1 -- 172.21.15.3:0/1235414561 >> v1:172.21.15.3:6789/0 conn(0x7f1dfc0b2710 legacy=0x7f1dfc14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfa7fc700 1 -- 172.21.15.3:0/1235414561 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc0a79c0 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfa7fc700 1 --2- 172.21.15.3:0/1235414561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b01a0 0x7f1dfc0a8540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f1df40032f0 tx=0x7f1df40091d0).ready entity=mon.2 client_cookie=e2541bb87682e57a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:52.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1df9ffb700 1 -- 172.21.15.3:0/1235414561 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1df400b600 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1df9ffb700 1 -- 172.21.15.3:0/1235414561 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df400b760 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1df9ffb700 1 -- 172.21.15.3:0/1235414561 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1df400b8d0 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfbfff700 1 -- 172.21.15.3:0/1235414561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b01a0 msgr2=0x7f1dfc0a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfbfff700 1 --2- 172.21.15.3:0/1235414561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b01a0 0x7f1dfc0a8540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f1df40032f0 tx=0x7f1df40091d0).stop 2022-07-21T01:33:52.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfbfff700 1 -- 172.21.15.3:0/1235414561 shutdown_connections 2022-07-21T01:33:52.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfbfff700 1 --2- 172.21.15.3:0/1235414561 >> v2:172.21.15.143:3300/0 conn(0x7f1dfc0afd60 0x7f1dfc0a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfbfff700 1 --2- 172.21.15.3:0/1235414561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b01a0 0x7f1dfc0a8540 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfbfff700 1 -- 172.21.15.3:0/1235414561 >> 172.21.15.3:0/1235414561 conn(0x7f1dfc060310 msgr2=0x7f1dfc0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:52.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfbfff700 1 -- 172.21.15.3:0/1235414561 shutdown_connections 2022-07-21T01:33:52.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.066+0000 7f1dfbfff700 1 -- 172.21.15.3:0/1235414561 wait complete. 2022-07-21T01:33:52.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfbfff700 1 Processor -- start 2022-07-21T01:33:52.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfbfff700 1 -- start start 2022-07-21T01:33:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dfc0afd60 0x7f1dfc13d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dfc0b01a0 0x7f1dfc13f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b2710 0x7f1dfc0b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfbfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc13ffb0 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfbfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc0aef20 con 0x7f1dfc0afd60 2022-07-21T01:33:52.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc157030 con 0x7f1dfc0b2710 2022-07-21T01:33:52.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfa7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dfc0b01a0 0x7f1dfc13f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfa7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dfc0b01a0 0x7f1dfc13f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43652/0 (socket says 172.21.15.3:43652) 2022-07-21T01:33:52.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfa7fc700 1 -- 172.21.15.3:0/3405847525 learned_addr learned my addr 172.21.15.3:0/3405847525 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:52.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfb7fe700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b2710 0x7f1dfc0b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfaffd700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dfc0afd60 0x7f1dfc13d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfa7fc700 1 -- 172.21.15.3:0/3405847525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b2710 msgr2=0x7f1dfc0b4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfa7fc700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b2710 0x7f1dfc0b4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfa7fc700 1 -- 172.21.15.3:0/3405847525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dfc0afd60 msgr2=0x7f1dfc13d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfa7fc700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dfc0afd60 0x7f1dfc13d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfa7fc700 1 -- 172.21.15.3:0/3405847525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df4009470 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.070+0000 7f1dfa7fc700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dfc0b01a0 0x7f1dfc13f470 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f1df40032f0 tx=0x7f1df4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:52.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1df3fff700 1 -- 172.21.15.3:0/3405847525 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1df4011c10 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dfc1572d0 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1df3fff700 1 -- 172.21.15.3:0/3405847525 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df4011d70 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1df3fff700 1 -- 172.21.15.3:0/3405847525 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1df401f5e0 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dfc13e820 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1df3fff700 1 -- 172.21.15.3:0/3405847525 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1df4026030 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dfc157460 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1df3fff700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dd4053a70 0x7f1dd4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1df3fff700 1 -- 172.21.15.3:0/3405847525 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1df406cca0 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1dfaffd700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dd4053a70 0x7f1dd4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.074+0000 7f1dfaffd700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dd4053a70 0x7f1dd4055f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1de400d000 tx=0x7f1de4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:52.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.082+0000 7f1df3fff700 1 -- 172.21.15.3:0/3405847525 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1df4071070 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.238+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dfc0574e0 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1df3fff700 1 -- 172.21.15.3:0/3405847525 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2474 (secure 0 0 0) 0x7f1df4024090 con 0x7f1dfc0b01a0 2022-07-21T01:33:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:52.243 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/22093665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":22093665},{"type":"v1","addr":"172.21.15.143:6833","nonce":22093665}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/140525254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140525254},{"type":"v1","addr":"172.21.15.3:6835","nonce":140525254}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:33:26.371191+0000","modified":"2022-07-21T01:33:27.386296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/4030643578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":4030643578},{"type":"v1","addr":"172.21.15.3:6837","nonce":4030643578}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-21T01:33:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dd4053a70 msgr2=0x7f1dd4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dd4053a70 0x7f1dd4055f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1de400d000 tx=0x7f1de4009040).stop 2022-07-21T01:33:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dfc0b01a0 msgr2=0x7f1dfc13f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dfc0b01a0 0x7f1dfc13f470 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f1df40032f0 tx=0x7f1df4016000).stop 2022-07-21T01:33:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 shutdown_connections 2022-07-21T01:33:52.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dfc0b2710 0x7f1dfc0b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dfc0b01a0 0x7f1dfc13f470 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dd4053a70 0x7f1dd4055f30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 --2- 172.21.15.3:0/3405847525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dfc0afd60 0x7f1dfc13d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 >> 172.21.15.3:0/3405847525 conn(0x7f1dfc060310 msgr2=0x7f1dfc05be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:52.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 shutdown_connections 2022-07-21T01:33:52.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.242+0000 7f1dfbfff700 1 -- 172.21.15.3:0/3405847525 wait complete. 2022-07-21T01:33:52.252 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 6 2022-07-21T01:33:52.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:33:52.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff145d57700 1 Processor -- start 2022-07-21T01:33:52.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff145d57700 1 -- start start 2022-07-21T01:33:52.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff145d57700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff1400b79a0 0x7ff140144730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff145d57700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b1300 0x7ff140144c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff145d57700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400b3360 con 0x7ff140064a00 2022-07-21T01:33:52.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff145d57700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1400b7d80 con 0x7ff1400b79a0 2022-07-21T01:33:52.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff145d57700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff140067750 con 0x7ff1400b1300 2022-07-21T01:33:52.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff140064a00 0x7ff140064e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51940/0 (socket says 172.21.15.3:51940) 2022-07-21T01:33:52.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13ffff700 1 -- 172.21.15.3:0/1697442902 learned_addr learned my addr 172.21.15.3:0/1697442902 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:52.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13effd700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b1300 0x7ff140144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13f7fe700 1 --2- 172.21.15.3:0/1697442902 >> v2:172.21.15.143:3300/0 conn(0x7ff1400b79a0 0x7ff140144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328457937 0 0) 0x7ff1400b3360 con 0x7ff140064a00 2022-07-21T01:33:52.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff12c003260 con 0x7ff140064a00 2022-07-21T01:33:52.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 801585484 0 0) 0x7ff12c003260 con 0x7ff140064a00 2022-07-21T01:33:52.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b1300 msgr2=0x7ff140144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:52.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b1300 0x7ff140144c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 >> v2:172.21.15.143:3300/0 conn(0x7ff1400b79a0 msgr2=0x7ff140144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:52.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 --2- 172.21.15.3:0/1697442902 >> v2:172.21.15.143:3300/0 conn(0x7ff1400b79a0 0x7ff140144730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400ab000 con 0x7ff140064a00 2022-07-21T01:33:52.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7ff138001660 con 0x7ff140064a00 2022-07-21T01:33:52.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 >> v1:172.21.15.3:6789/0 conn(0x7ff140064a00 legacy=0x7ff140064e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b1300 0x7ff12c003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff12c003c80 0x7ff12c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff12c0049f0 0x7ff12c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400b7d80 con 0x7ff12c003c80 2022-07-21T01:33:52.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff140067750 con 0x7ff12c0049f0 2022-07-21T01:33:52.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c00b750 con 0x7ff1400b1300 2022-07-21T01:33:52.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13f7fe700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b1300 0x7ff12c003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13ffff700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff12c0049f0 0x7ff12c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13effd700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff12c003c80 0x7ff12c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13f7fe700 1 -- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff12c0049f0 msgr2=0x7ff12c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:52.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13f7fe700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff12c0049f0 0x7ff12c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13f7fe700 1 -- 172.21.15.3:0/1697442902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff12c003c80 msgr2=0x7ff12c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:52.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13f7fe700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff12c003c80 0x7ff12c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.362+0000 7ff13f7fe700 1 -- 172.21.15.3:0/1697442902 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c003260 con 0x7ff1400b1300 2022-07-21T01:33:52.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff13f7fe700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b1300 0x7ff12c003750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7ff13400c000 tx=0x7ff134006df0).ready entity=mon.2 client_cookie=d0938882c983e54e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:52.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff13400e830 con 0x7ff1400b1300 2022-07-21T01:33:52.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13400e990 con 0x7ff1400b1300 2022-07-21T01:33:52.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff13e7fc700 1 -- 172.21.15.3:0/1697442902 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff13400ecf0 con 0x7ff1400b1300 2022-07-21T01:33:52.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 -- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b1300 msgr2=0x7ff12c003750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 --2- 172.21.15.3:0/1697442902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b1300 0x7ff12c003750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7ff13400c000 tx=0x7ff134006df0).stop 2022-07-21T01:33:52.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff13ffff700 1 -- 172.21.15.3:0/1697442902 reap_dead start 2022-07-21T01:33:52.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 -- 172.21.15.3:0/1697442902 shutdown_connections 2022-07-21T01:33:52.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 -- 172.21.15.3:0/1697442902 >> 172.21.15.3:0/1697442902 conn(0x7ff14005f840 msgr2=0x7ff14005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:52.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 -- 172.21.15.3:0/1697442902 shutdown_connections 2022-07-21T01:33:52.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 -- 172.21.15.3:0/1697442902 wait complete. 2022-07-21T01:33:52.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 Processor -- start 2022-07-21T01:33:52.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 -- start start 2022-07-21T01:33:52.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1400b1300 0x7ff140156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1400b02b0 0x7ff1400b5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b5a10 0x7ff14009ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14013f1a0 con 0x7ff1400b1300 2022-07-21T01:33:52.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400618d0 con 0x7ff1400b02b0 2022-07-21T01:33:52.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff145d57700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400afa10 con 0x7ff1400b5a10 2022-07-21T01:33:52.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff13f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1400b1300 0x7ff140156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff13f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1400b1300 0x7ff140156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43668/0 (socket says 172.21.15.3:43668) 2022-07-21T01:33:52.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.366+0000 7ff13f7fe700 1 -- 172.21.15.3:0/2304108899 learned_addr learned my addr 172.21.15.3:0/2304108899 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:52.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13ffff700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b5a10 0x7ff14009ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13effd700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1400b02b0 0x7ff1400b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13f7fe700 1 -- 172.21.15.3:0/2304108899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b5a10 msgr2=0x7ff14009ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13f7fe700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b5a10 0x7ff14009ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13f7fe700 1 -- 172.21.15.3:0/2304108899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1400b02b0 msgr2=0x7ff1400b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:52.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13f7fe700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1400b02b0 0x7ff1400b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:52.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13f7fe700 1 -- 172.21.15.3:0/2304108899 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff140061630 con 0x7ff1400b1300 2022-07-21T01:33:52.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13f7fe700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1400b1300 0x7ff140156260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7ff14013f350 tx=0x7ff134019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:52.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13cff9700 1 -- 172.21.15.3:0/2304108899 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff13401b820 con 0x7ff1400b1300 2022-07-21T01:33:52.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13cff9700 1 -- 172.21.15.3:0/2304108899 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff134000e90 con 0x7ff1400b1300 2022-07-21T01:33:52.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14013fab0 con 0x7ff1400b1300 2022-07-21T01:33:52.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13cff9700 1 -- 172.21.15.3:0/2304108899 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff13401bb70 con 0x7ff1400b1300 2022-07-21T01:33:52.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1400b0a60 con 0x7ff1400b1300 2022-07-21T01:33:52.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1400ab2a0 con 0x7ff1400b1300 2022-07-21T01:33:52.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13cff9700 1 -- 172.21.15.3:0/2304108899 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff13402d070 con 0x7ff1400b1300 2022-07-21T01:33:52.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13cff9700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1200539e0 0x7ff120055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:52.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13cff9700 1 -- 172.21.15.3:0/2304108899 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff13406d4b0 con 0x7ff1400b1300 2022-07-21T01:33:52.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.370+0000 7ff13effd700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1200539e0 0x7ff120055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:52.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.374+0000 7ff13effd700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1200539e0 0x7ff120055ea0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7ff13000b000 tx=0x7ff130008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:52.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.382+0000 7ff13cff9700 1 -- 172.21.15.3:0/2304108899 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff134044270 con 0x7ff1400b1300 2022-07-21T01:33:52.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:52.546+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff1400574e0 con 0x7ff1400b1300 2022-07-21T01:33:53.393 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:33:53.389+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:33:53.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.397+0000 7ff13cff9700 1 -- 172.21.15.3:0/2304108899 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff134047890 con 0x7ff1400b1300 2022-07-21T01:33:53.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1200539e0 msgr2=0x7ff120055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1200539e0 0x7ff120055ea0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7ff13000b000 tx=0x7ff130008040).stop 2022-07-21T01:33:53.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1400b1300 msgr2=0x7ff140156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1400b1300 0x7ff140156260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7ff14013f350 tx=0x7ff134019000).stop 2022-07-21T01:33:53.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 shutdown_connections 2022-07-21T01:33:53.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400b5a10 0x7ff14009ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff1200539e0 0x7ff120055ea0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1400b02b0 0x7ff1400b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 --2- 172.21.15.3:0/2304108899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1400b1300 0x7ff140156260 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 >> 172.21.15.3:0/2304108899 conn(0x7ff14005f840 msgr2=0x7ff1400b6050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:53.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 shutdown_connections 2022-07-21T01:33:53.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.401+0000 7ff145d57700 1 -- 172.21.15.3:0/2304108899 wait complete. 2022-07-21T01:33:53.408 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:33:53.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:33:53.431 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:33:53.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.525+0000 7f546e1cc700 1 Processor -- start 2022-07-21T01:33:53.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.525+0000 7f546e1cc700 1 -- start start 2022-07-21T01:33:53.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f54680b2710 0x7f546814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54680b01a0 0x7f546814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54680a7250 con 0x7f54680afd60 2022-07-21T01:33:53.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54680a7480 con 0x7f54680b01a0 2022-07-21T01:33:53.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54680a7720 con 0x7f54680b2710 2022-07-21T01:33:53.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546c9c9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54680b01a0 0x7f546814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546c9c9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54680b01a0 0x7f546814b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43688/0 (socket says 172.21.15.3:43688) 2022-07-21T01:33:53.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546c9c9700 1 -- 172.21.15.3:0/1719751791 learned_addr learned my addr 172.21.15.3:0/1719751791 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:53.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546d1ca700 1 --2- 172.21.15.3:0/1719751791 >> v2:172.21.15.143:3300/0 conn(0x7f54680b2710 0x7f546814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546d9cb700 1 --1- 172.21.15.3:0/1719751791 >> v1:172.21.15.3:6789/0 conn(0x7f54680afd60 0x7f54680a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51970/0 (socket says 172.21.15.3:51970) 2022-07-21T01:33:53.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546c9c9700 1 -- 172.21.15.3:0/1719751791 >> v2:172.21.15.143:3300/0 conn(0x7f54680b2710 msgr2=0x7f546814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:53.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546c9c9700 1 --2- 172.21.15.3:0/1719751791 >> v2:172.21.15.143:3300/0 conn(0x7f54680b2710 0x7f546814afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546c9c9700 1 -- 172.21.15.3:0/1719751791 >> v1:172.21.15.3:6789/0 conn(0x7f54680afd60 legacy=0x7f54680a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546c9c9700 1 -- 172.21.15.3:0/1719751791 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54680a79c0 con 0x7f54680b01a0 2022-07-21T01:33:53.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546c9c9700 1 --2- 172.21.15.3:0/1719751791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b01a0 0x7f546814b4c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f5458002f70 tx=0x7f5458006eb0).ready entity=mon.0 client_cookie=205088430d0250d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:53.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f545ffff700 1 -- 172.21.15.3:0/1719751791 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f545800d5c0 con 0x7f54680b01a0 2022-07-21T01:33:53.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f545ffff700 1 -- 172.21.15.3:0/1719751791 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5458000e90 con 0x7f54680b01a0 2022-07-21T01:33:53.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f545ffff700 1 -- 172.21.15.3:0/1719751791 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f545800d950 con 0x7f54680b01a0 2022-07-21T01:33:53.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 -- 172.21.15.3:0/1719751791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b01a0 msgr2=0x7f546814b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 --2- 172.21.15.3:0/1719751791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b01a0 0x7f546814b4c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f5458002f70 tx=0x7f5458006eb0).stop 2022-07-21T01:33:53.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 -- 172.21.15.3:0/1719751791 shutdown_connections 2022-07-21T01:33:53.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 --2- 172.21.15.3:0/1719751791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b01a0 0x7f546814b4c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 --2- 172.21.15.3:0/1719751791 >> v2:172.21.15.143:3300/0 conn(0x7f54680b2710 0x7f546814afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 -- 172.21.15.3:0/1719751791 >> 172.21.15.3:0/1719751791 conn(0x7f5468060310 msgr2=0x7f54680b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:53.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 -- 172.21.15.3:0/1719751791 shutdown_connections 2022-07-21T01:33:53.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 -- 172.21.15.3:0/1719751791 wait complete. 2022-07-21T01:33:53.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 Processor -- start 2022-07-21T01:33:53.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.529+0000 7f546e1cc700 1 -- start start 2022-07-21T01:33:53.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546e1cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54680afd60 0x7f54681565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546e1cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54680b01a0 0x7f546814ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546e1cc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b2710 0x7f5468064c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546e1cc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54680ab100 con 0x7f54680b2710 2022-07-21T01:33:53.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546e1cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5468156a90 con 0x7f54680b01a0 2022-07-21T01:33:53.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546e1cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54680aab90 con 0x7f54680afd60 2022-07-21T01:33:53.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d9cb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b2710 0x7f5468064c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d1ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54680afd60 0x7f54681565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d1ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54680afd60 0x7f54681565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:50608/0 (socket says 172.21.15.3:50608) 2022-07-21T01:33:53.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d1ca700 1 -- 172.21.15.3:0/3570101032 learned_addr learned my addr 172.21.15.3:0/3570101032 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:53.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546c9c9700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54680b01a0 0x7f546814ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d1ca700 1 -- 172.21.15.3:0/3570101032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54680b01a0 msgr2=0x7f546814ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d1ca700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54680b01a0 0x7f546814ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d1ca700 1 -- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b2710 msgr2=0x7f5468064c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d1ca700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b2710 0x7f5468064c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d1ca700 1 -- 172.21.15.3:0/3570101032 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545800b1e0 con 0x7f54680afd60 2022-07-21T01:33:53.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d9cb700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b2710 0x7f5468064c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:53.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546d1ca700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54680afd60 0x7f54681565a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f54680b22f0 tx=0x7f54600091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:53.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f545e7fc700 1 -- 172.21.15.3:0/3570101032 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f546000c610 con 0x7f54680afd60 2022-07-21T01:33:53.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f545e7fc700 1 -- 172.21.15.3:0/3570101032 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546000c770 con 0x7f54680afd60 2022-07-21T01:33:53.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54680ab330 con 0x7f54680afd60 2022-07-21T01:33:53.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f545e7fc700 1 -- 172.21.15.3:0/3570101032 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f546000cad0 con 0x7f54680afd60 2022-07-21T01:33:53.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.533+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54680aca50 con 0x7f54680afd60 2022-07-21T01:33:53.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.537+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54680574e0 con 0x7f54680afd60 2022-07-21T01:33:53.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.537+0000 7f545e7fc700 1 -- 172.21.15.3:0/3570101032 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f546000cc30 con 0x7f54680afd60 2022-07-21T01:33:53.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.537+0000 7f545e7fc700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5450053b00 0x7f5450055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.537+0000 7f546c9c9700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5450053b00 0x7f5450055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.537+0000 7f545e7fc700 1 -- 172.21.15.3:0/3570101032 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f546006c340 con 0x7f54680afd60 2022-07-21T01:33:53.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.537+0000 7f546c9c9700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5450053b00 0x7f5450055fc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f545801c000 tx=0x7f5458015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:53.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.545+0000 7f545e7fc700 1 -- 172.21.15.3:0/3570101032 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5460042f30 con 0x7f54680afd60 2022-07-21T01:33:53.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54680aacd0 con 0x7f54680afd60 2022-07-21T01:33:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f545e7fc700 1 -- 172.21.15.3:0/3570101032 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2117 (secure 0 0 0) 0x7f5460046550 con 0x7f54680afd60 2022-07-21T01:33:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:53.715 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/22093665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":22093665},{"type":"v1","addr":"172.21.15.143:6833","nonce":22093665}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/140525254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140525254},{"type":"v1","addr":"172.21.15.3:6835","nonce":140525254}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:33:26.371191+0000","modified":"2022-07-21T01:33:53.396412+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-21T01:33:53.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5450053b00 msgr2=0x7f5450055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5450053b00 0x7f5450055fc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f545801c000 tx=0x7f5458015000).stop 2022-07-21T01:33:53.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54680afd60 msgr2=0x7f54681565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54680afd60 0x7f54681565a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f54680b22f0 tx=0x7f54600091d0).stop 2022-07-21T01:33:53.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 shutdown_connections 2022-07-21T01:33:53.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54680b2710 0x7f5468064c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5450053b00 0x7f5450055fc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54680b01a0 0x7f546814ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 --2- 172.21.15.3:0/3570101032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54680afd60 0x7f54681565a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.713+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 >> 172.21.15.3:0/3570101032 conn(0x7f5468060310 msgr2=0x7f54680673e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:53.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.717+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 shutdown_connections 2022-07-21T01:33:53.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.717+0000 7f546e1cc700 1 -- 172.21.15.3:0/3570101032 wait complete. 2022-07-21T01:33:53.721 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 7 2022-07-21T01:33:53.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:33:53.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 Processor -- start 2022-07-21T01:33:53.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 -- start start 2022-07-21T01:33:53.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b1460 0x7f81600b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81600a7960 con 0x7f81600b2050 2022-07-21T01:33:53.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81600b07e0 con 0x7f81600b2430 2022-07-21T01:33:53.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81600b18d0 con 0x7f81600b1460 2022-07-21T01:33:53.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f81663c8700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f81600b2050 0x7f81600a7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51980/0 (socket says 172.21.15.3:51980) 2022-07-21T01:33:53.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f81663c8700 1 -- 172.21.15.3:0/1072607902 learned_addr learned my addr 172.21.15.3:0/1072607902 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:53.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f8165bc7700 1 --2- 172.21.15.3:0/1072607902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b1460 0x7f81600b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f81653c6700 1 -- 172.21.15.3:0/1072607902 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213867641 0 0) 0x7f81600a7960 con 0x7f81600b2050 2022-07-21T01:33:53.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f81653c6700 1 -- 172.21.15.3:0/1072607902 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f814c003260 con 0x7f81600b2050 2022-07-21T01:33:53.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f81653c6700 1 -- 172.21.15.3:0/1072607902 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728120272 0 0) 0x7f81600b07e0 con 0x7f81600b2430 2022-07-21T01:33:53.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f81653c6700 1 -- 172.21.15.3:0/1072607902 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81600a7960 con 0x7f81600b2430 2022-07-21T01:33:53.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f8165bc7700 1 -- 172.21.15.3:0/1072607902 >> v1:172.21.15.143:6789/0 conn(0x7f81600b2430 legacy=0x7f81601413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f8165bc7700 1 -- 172.21.15.3:0/1072607902 >> v1:172.21.15.3:6789/0 conn(0x7f81600b2050 legacy=0x7f81600a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f8165bc7700 1 -- 172.21.15.3:0/1072607902 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81600b0a80 con 0x7f81600b1460 2022-07-21T01:33:53.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f8165bc7700 1 --2- 172.21.15.3:0/1072607902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b1460 0x7f81600b0140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f8154002f70 tx=0x7f8154006df0).ready entity=mon.2 client_cookie=115da24aef1d9e25 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:53.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f81653c6700 1 -- 172.21.15.3:0/1072607902 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f815400d5c0 con 0x7f81600b1460 2022-07-21T01:33:53.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f81653c6700 1 -- 172.21.15.3:0/1072607902 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8154000e90 con 0x7f81600b1460 2022-07-21T01:33:53.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f81653c6700 1 -- 172.21.15.3:0/1072607902 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f815400d910 con 0x7f81600b1460 2022-07-21T01:33:53.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 -- 172.21.15.3:0/1072607902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b1460 msgr2=0x7f81600b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 --2- 172.21.15.3:0/1072607902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b1460 0x7f81600b0140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f8154002f70 tx=0x7f8154006df0).stop 2022-07-21T01:33:53.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 -- 172.21.15.3:0/1072607902 shutdown_connections 2022-07-21T01:33:53.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 --2- 172.21.15.3:0/1072607902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b1460 0x7f81600b0140 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 -- 172.21.15.3:0/1072607902 >> 172.21.15.3:0/1072607902 conn(0x7f816005f840 msgr2=0x7f816005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:53.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 -- 172.21.15.3:0/1072607902 shutdown_connections 2022-07-21T01:33:53.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.885+0000 7f816862c700 1 -- 172.21.15.3:0/1072607902 wait complete. 2022-07-21T01:33:53.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 Processor -- start 2022-07-21T01:33:53.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 -- start start 2022-07-21T01:33:53.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81600b1460 0x7f816014ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b2050 0x7f81600b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81600b2430 0x7f81600b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81600b3fa0 con 0x7f81600b1460 2022-07-21T01:33:53.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816014b320 con 0x7f81600b2430 2022-07-21T01:33:53.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816014b7f0 con 0x7f81600b2050 2022-07-21T01:33:53.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f81663c8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81600b1460 0x7f816014ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f81663c8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81600b1460 0x7f816014ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43706/0 (socket says 172.21.15.3:43706) 2022-07-21T01:33:53.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f81663c8700 1 -- 172.21.15.3:0/3964290672 learned_addr learned my addr 172.21.15.3:0/3964290672 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:53.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f8165bc7700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b2050 0x7f81600b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f8166bc9700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81600b2430 0x7f81600b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f81663c8700 1 -- 172.21.15.3:0/3964290672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b2050 msgr2=0x7f81600b5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f81663c8700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b2050 0x7f81600b5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f81663c8700 1 -- 172.21.15.3:0/3964290672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81600b2430 msgr2=0x7f81600b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:53.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f81663c8700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81600b2430 0x7f81600b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:53.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f81663c8700 1 -- 172.21.15.3:0/3964290672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815400b180 con 0x7f81600b1460 2022-07-21T01:33:53.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f81663c8700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81600b1460 0x7f816014ad60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f8150001550 tx=0x7f8150007ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:53.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f815f7fe700 1 -- 172.21.15.3:0/3964290672 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8150005ba0 con 0x7f81600b1460 2022-07-21T01:33:53.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f815f7fe700 1 -- 172.21.15.3:0/3964290672 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8150005d00 con 0x7f81600b1460 2022-07-21T01:33:53.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816014b550 con 0x7f81600b1460 2022-07-21T01:33:53.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f815f7fe700 1 -- 172.21.15.3:0/3964290672 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81500072c0 con 0x7f81600b1460 2022-07-21T01:33:53.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.889+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81600b5ff0 con 0x7f81600b1460 2022-07-21T01:33:53.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.893+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81600574e0 con 0x7f81600b1460 2022-07-21T01:33:53.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.893+0000 7f815f7fe700 1 -- 172.21.15.3:0/3964290672 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f815001a060 con 0x7f81600b1460 2022-07-21T01:33:53.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.893+0000 7f815f7fe700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8140053a70 0x7f8140055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:53.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.893+0000 7f815f7fe700 1 -- 172.21.15.3:0/3964290672 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f815006bd00 con 0x7f81600b1460 2022-07-21T01:33:53.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.893+0000 7f8165bc7700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8140053a70 0x7f8140055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:53.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.893+0000 7f8165bc7700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8140053a70 0x7f8140055f30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f8154018000 tx=0x7f8154014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:53.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:53.901+0000 7f815f7fe700 1 -- 172.21.15.3:0/3964290672 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8150042950 con 0x7f81600b1460 2022-07-21T01:33:54.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.073+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f81600ac8f0 con 0x7f81600b1460 2022-07-21T01:33:54.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.405+0000 7f815f7fe700 1 -- 172.21.15.3:0/3964290672 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8150018020 con 0x7f81600b1460 2022-07-21T01:33:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8140053a70 msgr2=0x7f8140055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8140053a70 0x7f8140055f30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f8154018000 tx=0x7f8154014000).stop 2022-07-21T01:33:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81600b1460 msgr2=0x7f816014ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81600b1460 0x7f816014ad60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f8150001550 tx=0x7f8150007ed0).stop 2022-07-21T01:33:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 shutdown_connections 2022-07-21T01:33:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8140053a70 0x7f8140055f30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:54.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81600b2430 0x7f81600b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:54.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81600b2050 0x7f81600b5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:54.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 --2- 172.21.15.3:0/3964290672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81600b1460 0x7f816014ad60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:54.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 >> 172.21.15.3:0/3964290672 conn(0x7f816005f840 msgr2=0x7f8160148120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:54.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 shutdown_connections 2022-07-21T01:33:54.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.409+0000 7f816862c700 1 -- 172.21.15.3:0/3964290672 wait complete. 2022-07-21T01:33:54.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:33:54.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.549+0000 7f6d333a5700 1 Processor -- start 2022-07-21T01:33:54.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.549+0000 7f6d333a5700 1 -- start start 2022-07-21T01:33:54.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6d2c0b08a0 0x7f6d2c142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:54.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6d2c0ad000 0x7f6d2c13fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:54.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d2c143ab0 con 0x7f6d2c0b3260 2022-07-21T01:33:54.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c0b0c80 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c0ae810 con 0x7f6d2c0ad000 2022-07-21T01:33:54.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d323a3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6d2c0b08a0 0x7f6d2c142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:54.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d323a3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6d2c0b08a0 0x7f6d2c142eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43716/0 (socket says 172.21.15.3:43716) 2022-07-21T01:33:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d323a3700 1 -- 172.21.15.3:0/2264322542 learned_addr learned my addr 172.21.15.3:0/2264322542 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d32ba4700 1 --2- 172.21.15.3:0/2264322542 >> v2:172.21.15.143:3300/0 conn(0x7f6d2c0ad000 0x7f6d2c13fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d323a3700 1 -- 172.21.15.3:0/2264322542 >> v2:172.21.15.143:3300/0 conn(0x7f6d2c0ad000 msgr2=0x7f6d2c13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d323a3700 1 --2- 172.21.15.3:0/2264322542 >> v2:172.21.15.143:3300/0 conn(0x7f6d2c0ad000 0x7f6d2c13fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d323a3700 1 -- 172.21.15.3:0/2264322542 >> v1:172.21.15.143:6789/0 conn(0x7f6d2c0b3260 legacy=0x7f6d2c1433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d323a3700 1 -- 172.21.15.3:0/2264322542 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c0aeab0 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d32ba4700 1 --2- 172.21.15.3:0/2264322542 >> v2:172.21.15.143:3300/0 conn(0x7f6d2c0ad000 0x7f6d2c13fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d323a3700 1 --2- 172.21.15.3:0/2264322542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 0x7f6d2c142eb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f6d28002f70 tx=0x7f6d28006eb0).ready entity=mon.0 client_cookie=5ad0355acb7feb0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d313a1700 1 -- 172.21.15.3:0/2264322542 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d2800d5c0 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d313a1700 1 -- 172.21.15.3:0/2264322542 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d28000e90 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d313a1700 1 -- 172.21.15.3:0/2264322542 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d2800d950 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 -- 172.21.15.3:0/2264322542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 msgr2=0x7f6d2c142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:54.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 --2- 172.21.15.3:0/2264322542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 0x7f6d2c142eb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f6d28002f70 tx=0x7f6d28006eb0).stop 2022-07-21T01:33:54.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 -- 172.21.15.3:0/2264322542 shutdown_connections 2022-07-21T01:33:54.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 --2- 172.21.15.3:0/2264322542 >> v2:172.21.15.143:3300/0 conn(0x7f6d2c0ad000 0x7f6d2c13fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:54.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 --2- 172.21.15.3:0/2264322542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 0x7f6d2c142eb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:54.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 -- 172.21.15.3:0/2264322542 >> 172.21.15.3:0/2264322542 conn(0x7f6d2c0612d0 msgr2=0x7f6d2c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:54.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 -- 172.21.15.3:0/2264322542 shutdown_connections 2022-07-21T01:33:54.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.553+0000 7f6d333a5700 1 -- 172.21.15.3:0/2264322542 wait complete. 2022-07-21T01:33:54.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 Processor -- start 2022-07-21T01:33:54.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 -- start start 2022-07-21T01:33:54.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d2c0ad000 0x7f6d2c0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:54.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 0x7f6d2c156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:54.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d2c0b3260 0x7f6d2c067390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:54.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c140920 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c156b30 con 0x7f6d2c0ad000 2022-07-21T01:33:54.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0ac9c0 con 0x7f6d2c0b3260 2022-07-21T01:33:54.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d31ba2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 0x7f6d2c156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:54.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d31ba2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 0x7f6d2c156640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43724/0 (socket says 172.21.15.3:43724) 2022-07-21T01:33:54.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d31ba2700 1 -- 172.21.15.3:0/2889081554 learned_addr learned my addr 172.21.15.3:0/2889081554 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:54.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d32ba4700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d2c0b3260 0x7f6d2c067390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:54.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d323a3700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d2c0ad000 0x7f6d2c0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:54.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d31ba2700 1 -- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d2c0b3260 msgr2=0x7f6d2c067390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:54.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d31ba2700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d2c0b3260 0x7f6d2c067390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:54.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d31ba2700 1 -- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d2c0ad000 msgr2=0x7f6d2c0b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:54.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d31ba2700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d2c0ad000 0x7f6d2c0b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:54.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d31ba2700 1 -- 172.21.15.3:0/2889081554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d24009470 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d323a3700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d2c0ad000 0x7f6d2c0b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:54.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d32ba4700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d2c0b3260 0x7f6d2c067390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:54.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d31ba2700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 0x7f6d2c156640 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f6d2c0b6bd0 tx=0x7f6d1c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:54.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d1b7fe700 1 -- 172.21.15.3:0/2889081554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d1c00ce30 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d1b7fe700 1 -- 172.21.15.3:0/2889081554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c00f470 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2800b1e0 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d1b7fe700 1 -- 172.21.15.3:0/2889081554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d1c00f7d0 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.557+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c063e80 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.561+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c0ac6f0 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.561+0000 7f6d1b7fe700 1 -- 172.21.15.3:0/2889081554 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6d1c019030 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.561+0000 7f6d1b7fe700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d0c053980 0x7f6d0c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:54.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.561+0000 7f6d323a3700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d0c053980 0x7f6d0c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:54.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.561+0000 7f6d1b7fe700 1 -- 172.21.15.3:0/2889081554 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f6d1c06b4c0 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.561+0000 7f6d323a3700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d0c053980 0x7f6d0c055e40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f6d2801a000 tx=0x7f6d28015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:54.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.565+0000 7f6d1b7fe700 1 -- 172.21.15.3:0/2889081554 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d1c0421a0 con 0x7f6d2c0b08a0 2022-07-21T01:33:54.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:54.697+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6d2c0574e0 con 0x7f6d2c0b08a0 2022-07-21T01:33:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.676+0000 7f6d1b7fe700 1 -- 172.21.15.3:0/2889081554 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 185+0+0 (secure 0 0 0) 0x7f6d1c0457c0 con 0x7f6d2c0b08a0 2022-07-21T01:33:55.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d0c053980 msgr2=0x7f6d0c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:55.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d0c053980 0x7f6d0c055e40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f6d2801a000 tx=0x7f6d28015000).stop 2022-07-21T01:33:55.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 msgr2=0x7f6d2c156640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:55.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 0x7f6d2c156640 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f6d2c0b6bd0 tx=0x7f6d1c007c50).stop 2022-07-21T01:33:55.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 shutdown_connections 2022-07-21T01:33:55.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d2c0b3260 0x7f6d2c067390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:55.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d2c0b08a0 0x7f6d2c156640 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:55.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d0c053980 0x7f6d0c055e40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:55.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 --2- 172.21.15.3:0/2889081554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d2c0ad000 0x7f6d2c0b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:55.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 >> 172.21.15.3:0/2889081554 conn(0x7f6d2c0612d0 msgr2=0x7f6d2c1421a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:55.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 shutdown_connections 2022-07-21T01:33:55.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.684+0000 7f6d333a5700 1 -- 172.21.15.3:0/2889081554 wait complete. 2022-07-21T01:33:55.687 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:33:55.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:33:55.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1deac5d700 1 Processor -- start 2022-07-21T01:33:55.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1deac5d700 1 -- start start 2022-07-21T01:33:55.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1deac5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 0x7f1de4144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:55.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1deac5d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1de40b36e0 0x7f1de4141370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:55.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1deac5d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de4064ef0 con 0x7f1de4141860 2022-07-21T01:33:55.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1deac5d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de413ebf0 con 0x7f1de40b36e0 2022-07-21T01:33:55.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1deac5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4066690 con 0x7f1de4064b10 2022-07-21T01:33:55.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1dea45c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1de4141860 0x7f1de4066ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52012/0 (socket says 172.21.15.3:52012) 2022-07-21T01:33:55.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1dea45c700 1 -- 172.21.15.3:0/1463451858 learned_addr learned my addr 172.21.15.3:0/1463451858 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:55.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1de945a700 1 --2- 172.21.15.3:0/1463451858 >> v2:172.21.15.3:3300/0 conn(0x7f1de40b36e0 0x7f1de4141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:55.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1de9c5b700 1 --2- 172.21.15.3:0/1463451858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 0x7f1de4144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:55.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1de8c59700 1 -- 172.21.15.3:0/1463451858 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703011498 0 0) 0x7f1de4064ef0 con 0x7f1de4141860 2022-07-21T01:33:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1de8c59700 1 -- 172.21.15.3:0/1463451858 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dd0003260 con 0x7f1de4141860 2022-07-21T01:33:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1de945a700 1 -- 172.21.15.3:0/1463451858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 msgr2=0x7f1de4144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1de945a700 1 --2- 172.21.15.3:0/1463451858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 0x7f1de4144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1de945a700 1 -- 172.21.15.3:0/1463451858 >> v1:172.21.15.3:6789/0 conn(0x7f1de4141860 legacy=0x7f1de4066ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1de945a700 1 -- 172.21.15.3:0/1463451858 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de40b4850 con 0x7f1de40b36e0 2022-07-21T01:33:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.804+0000 7f1de945a700 1 --2- 172.21.15.3:0/1463451858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de40b36e0 0x7f1de4141370 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1dd8002f70 tx=0x7f1dd8006df0).ready entity=mon.0 client_cookie=73789b1d983eadb7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1de8c59700 1 -- 172.21.15.3:0/1463451858 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dd800d4a0 con 0x7f1de40b36e0 2022-07-21T01:33:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1de8c59700 1 -- 172.21.15.3:0/1463451858 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd8000e90 con 0x7f1de40b36e0 2022-07-21T01:33:55.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1de8c59700 1 -- 172.21.15.3:0/1463451858 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dd800d7f0 con 0x7f1de40b36e0 2022-07-21T01:33:55.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 -- 172.21.15.3:0/1463451858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de40b36e0 msgr2=0x7f1de4141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:55.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 --2- 172.21.15.3:0/1463451858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de40b36e0 0x7f1de4141370 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1dd8002f70 tx=0x7f1dd8006df0).stop 2022-07-21T01:33:55.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 -- 172.21.15.3:0/1463451858 shutdown_connections 2022-07-21T01:33:55.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 --2- 172.21.15.3:0/1463451858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de40b36e0 0x7f1de4141370 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:55.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 --2- 172.21.15.3:0/1463451858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 0x7f1de4144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:55.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 -- 172.21.15.3:0/1463451858 >> 172.21.15.3:0/1463451858 conn(0x7f1de405f840 msgr2=0x7f1de405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:55.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 -- 172.21.15.3:0/1463451858 shutdown_connections 2022-07-21T01:33:55.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 -- 172.21.15.3:0/1463451858 wait complete. 2022-07-21T01:33:55.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 Processor -- start 2022-07-21T01:33:55.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.808+0000 7f1deac5d700 1 -- start start 2022-07-21T01:33:55.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1deac5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 0x7f1de40af700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:55.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1deac5d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de40b36e0 0x7f1de40b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:55.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1deac5d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de4141860 0x7f1de40b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:55.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1deac5d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4062350 con 0x7f1de4141860 2022-07-21T01:33:55.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1deac5d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4155c30 con 0x7f1de40b36e0 2022-07-21T01:33:55.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1deac5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4155ed0 con 0x7f1de4064b10 2022-07-21T01:33:55.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1dea45c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de4141860 0x7f1de40b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:55.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1dea45c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de4141860 0x7f1de40b0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43740/0 (socket says 172.21.15.3:43740) 2022-07-21T01:33:55.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1dea45c700 1 -- 172.21.15.3:0/1842843394 learned_addr learned my addr 172.21.15.3:0/1842843394 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:55.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1de945a700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de40b36e0 0x7f1de40b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:55.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1de9c5b700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 0x7f1de40af700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:55.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1dea45c700 1 -- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 msgr2=0x7f1de40af700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:55.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1dea45c700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 0x7f1de40af700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:55.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1dea45c700 1 -- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de40b36e0 msgr2=0x7f1de40b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:55.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1dea45c700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de40b36e0 0x7f1de40b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:55.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1dea45c700 1 -- 172.21.15.3:0/1842843394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd800b180 con 0x7f1de4141860 2022-07-21T01:33:55.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1de9c5b700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 0x7f1de40af700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:55.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1de945a700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de40b36e0 0x7f1de40b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:55.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1dea45c700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de4141860 0x7f1de40b0590 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f1ddc001110 tx=0x7f1ddc007a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1de2ffd700 1 -- 172.21.15.3:0/1842843394 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ddc00ce00 con 0x7f1de4141860 2022-07-21T01:33:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1de2ffd700 1 -- 172.21.15.3:0/1842843394 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc0072e0 con 0x7f1de4141860 2022-07-21T01:33:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de4156170 con 0x7f1de4141860 2022-07-21T01:33:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1de2ffd700 1 -- 172.21.15.3:0/1842843394 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ddc005800 con 0x7f1de4141860 2022-07-21T01:33:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de41558b0 con 0x7f1de4141860 2022-07-21T01:33:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de40b4af0 con 0x7f1de4141860 2022-07-21T01:33:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1de2ffd700 1 -- 172.21.15.3:0/1842843394 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ddc01a060 con 0x7f1de4141860 2022-07-21T01:33:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.812+0000 7f1de2ffd700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dc4053a70 0x7f1dc4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.816+0000 7f1de2ffd700 1 -- 172.21.15.3:0/1842843394 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f1ddc06b6f0 con 0x7f1de4141860 2022-07-21T01:33:55.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.816+0000 7f1de9c5b700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dc4053a70 0x7f1dc4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:55.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.816+0000 7f1de9c5b700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dc4053a70 0x7f1dc4055f30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f1dd400b000 tx=0x7f1dd4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:55.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.820+0000 7f1de2ffd700 1 -- 172.21.15.3:0/1842843394 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ddc042610 con 0x7f1de4141860 2022-07-21T01:33:55.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:55.948+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1de40574e0 con 0x7f1de4141860 2022-07-21T01:33:56.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.688+0000 7f1de2ffd700 1 -- 172.21.15.3:0/1842843394 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 173+0+0 (secure 0 0 0) 0x7f1ddc018070 con 0x7f1de4141860 2022-07-21T01:33:56.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dc4053a70 msgr2=0x7f1dc4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:56.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dc4053a70 0x7f1dc4055f30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f1dd400b000 tx=0x7f1dd4008040).stop 2022-07-21T01:33:56.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de4141860 msgr2=0x7f1de40b0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:56.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de4141860 0x7f1de40b0590 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f1ddc001110 tx=0x7f1ddc007a40).stop 2022-07-21T01:33:56.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 shutdown_connections 2022-07-21T01:33:56.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de4141860 0x7f1de40b0590 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:56.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1dc4053a70 0x7f1dc4055f30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:56.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de40b36e0 0x7f1de40b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:56.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 --2- 172.21.15.3:0/1842843394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de4064b10 0x7f1de40af700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:56.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 >> 172.21.15.3:0/1842843394 conn(0x7f1de405f840 msgr2=0x7f1de40b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:56.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 shutdown_connections 2022-07-21T01:33:56.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.692+0000 7f1deac5d700 1 -- 172.21.15.3:0/1842843394 wait complete. 2022-07-21T01:33:56.700 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:33:56.707 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:33:56.707 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:33:56.708 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:33:56.708 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:33:56.709 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:33:56.709 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:33:56.709 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:33:56.710 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:33:56.710 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:33:56.710 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:33:56.711 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:33:56.711 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:33:56.723 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:33:56.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:33:56.725 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:33:56.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:33:56.727 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:33:56.728 DEBUG:teuthology.parallel:result is None 2022-07-21T01:33:56.729 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:33:56.729 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:33:56.730 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:33:56.731 DEBUG:teuthology.parallel:result is None 2022-07-21T01:33:56.774 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:33:56.775 DEBUG:teuthology.parallel:result is None 2022-07-21T01:33:56.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:33:56.795 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:33:56.798 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:33:56.850 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:33:56.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3fe47700 1 Processor -- start 2022-07-21T01:33:56.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3fe47700 1 -- start start 2022-07-21T01:33:56.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3fe47700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa38061a20 0x7faa380b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:56.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3fe47700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faa380b7080 0x7faa3812b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:56.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3fe47700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa38061e00 con 0x7faa3812b7e0 2022-07-21T01:33:56.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3fe47700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa380b2e40 con 0x7faa380b7080 2022-07-21T01:33:56.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3fe47700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3812c090 con 0x7faa38061a20 2022-07-21T01:33:56.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3d3e2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faa380b7080 0x7faa3812b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:56.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3d3e2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faa380b7080 0x7faa3812b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43770/0 (socket says 172.21.15.3:43770) 2022-07-21T01:33:56.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3d3e2700 1 -- 172.21.15.3:0/912278780 learned_addr learned my addr 172.21.15.3:0/912278780 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:56.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3dbe3700 1 --2- 172.21.15.3:0/912278780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa38061a20 0x7faa380b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:56.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3cbe1700 1 -- 172.21.15.3:0/912278780 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061272077 0 0) 0x7faa38061e00 con 0x7faa3812b7e0 2022-07-21T01:33:56.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3d3e2700 1 -- 172.21.15.3:0/912278780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa38061a20 msgr2=0x7faa380b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:56.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3d3e2700 1 --2- 172.21.15.3:0/912278780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa38061a20 0x7faa380b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:56.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3d3e2700 1 -- 172.21.15.3:0/912278780 >> v1:172.21.15.143:6789/0 conn(0x7faa3812b7e0 legacy=0x7faa38146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:56.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.867+0000 7faa3d3e2700 1 -- 172.21.15.3:0/912278780 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3812c330 con 0x7faa380b7080 2022-07-21T01:33:56.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3d3e2700 1 --2- 172.21.15.3:0/912278780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa380b7080 0x7faa3812b2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7faa28002f70 tx=0x7faa28006eb0).ready entity=mon.0 client_cookie=dfb1966d0952fa19 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:56.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3cbe1700 1 -- 172.21.15.3:0/912278780 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa2800d570 con 0x7faa380b7080 2022-07-21T01:33:56.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3cbe1700 1 -- 172.21.15.3:0/912278780 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa28000e90 con 0x7faa380b7080 2022-07-21T01:33:56.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3cbe1700 1 -- 172.21.15.3:0/912278780 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa2800d8c0 con 0x7faa380b7080 2022-07-21T01:33:56.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 -- 172.21.15.3:0/912278780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa380b7080 msgr2=0x7faa3812b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:56.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 --2- 172.21.15.3:0/912278780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa380b7080 0x7faa3812b2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7faa28002f70 tx=0x7faa28006eb0).stop 2022-07-21T01:33:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 -- 172.21.15.3:0/912278780 shutdown_connections 2022-07-21T01:33:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 --2- 172.21.15.3:0/912278780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa380b7080 0x7faa3812b2f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 --2- 172.21.15.3:0/912278780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa38061a20 0x7faa380b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 -- 172.21.15.3:0/912278780 >> 172.21.15.3:0/912278780 conn(0x7faa38066f70 msgr2=0x7faa38067380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 -- 172.21.15.3:0/912278780 shutdown_connections 2022-07-21T01:33:56.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 -- 172.21.15.3:0/912278780 wait complete. 2022-07-21T01:33:56.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 Processor -- start 2022-07-21T01:33:56.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 -- start start 2022-07-21T01:33:56.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa38061a20 0x7faa380b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:56.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa380b7080 0x7faa381559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3812b7e0 0x7faa38152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa381524a0 con 0x7faa3812b7e0 2022-07-21T01:33:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa38155ee0 con 0x7faa38061a20 2022-07-21T01:33:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3fe47700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa38142c90 con 0x7faa380b7080 2022-07-21T01:33:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.871+0000 7faa3e3e4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3812b7e0 0x7faa38152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3e3e4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3812b7e0 0x7faa38152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43778/0 (socket says 172.21.15.3:43778) 2022-07-21T01:33:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3e3e4700 1 -- 172.21.15.3:0/4112683138 learned_addr learned my addr 172.21.15.3:0/4112683138 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3d3e2700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa380b7080 0x7faa381559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3dbe3700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa38061a20 0x7faa380b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3e3e4700 1 -- 172.21.15.3:0/4112683138 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa380b7080 msgr2=0x7faa381559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3e3e4700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa380b7080 0x7faa381559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3e3e4700 1 -- 172.21.15.3:0/4112683138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa38061a20 msgr2=0x7faa380b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3e3e4700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa38061a20 0x7faa380b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3e3e4700 1 -- 172.21.15.3:0/4112683138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2800b1e0 con 0x7faa3812b7e0 2022-07-21T01:33:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3e3e4700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3812b7e0 0x7faa38152dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7faa30000ed0 tx=0x7faa30007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa26ffd700 1 -- 172.21.15.3:0/4112683138 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa30007e20 con 0x7faa3812b7e0 2022-07-21T01:33:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa26ffd700 1 -- 172.21.15.3:0/4112683138 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa30007030 con 0x7faa3812b7e0 2022-07-21T01:33:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa38152690 con 0x7faa3812b7e0 2022-07-21T01:33:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa26ffd700 1 -- 172.21.15.3:0/4112683138 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa30007390 con 0x7faa3812b7e0 2022-07-21T01:33:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa381532c0 con 0x7faa3812b7e0 2022-07-21T01:33:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa380b8a50 con 0x7faa3812b7e0 2022-07-21T01:33:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa26ffd700 1 -- 172.21.15.3:0/4112683138 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faa30006510 con 0x7faa3812b7e0 2022-07-21T01:33:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa26ffd700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa18053b00 0x7faa18055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa26ffd700 1 -- 172.21.15.3:0/4112683138 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5686+0+0 (secure 0 0 0) 0x7faa3006b5b0 con 0x7faa3812b7e0 2022-07-21T01:33:56.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3dbe3700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa18053b00 0x7faa18055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:56.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.875+0000 7faa3dbe3700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa18053b00 0x7faa18055fc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7faa3400b000 tx=0x7faa340091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:56.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:56.883+0000 7faa26ffd700 1 -- 172.21.15.3:0/4112683138 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faa300425f0 con 0x7faa3812b7e0 2022-07-21T01:33:57.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.019+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7faa38142e50 con 0x7faa3812b7e0 2022-07-21T01:33:57.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.879+0000 7faa26ffd700 1 -- 172.21.15.3:0/4112683138 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v32) v1 ==== 97+0+0 (secure 0 0 0) 0x7faa30045c10 con 0x7faa3812b7e0 2022-07-21T01:33:57.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa18053b00 msgr2=0x7faa18055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:57.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa18053b00 0x7faa18055fc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7faa3400b000 tx=0x7faa340091d0).stop 2022-07-21T01:33:57.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3812b7e0 msgr2=0x7faa38152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:57.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3812b7e0 0x7faa38152dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7faa30000ed0 tx=0x7faa30007bd0).stop 2022-07-21T01:33:57.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 shutdown_connections 2022-07-21T01:33:57.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa3812b7e0 0x7faa38152dd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:57.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa380b7080 0x7faa381559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:57.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa18053b00 0x7faa18055fc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:57.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 --2- 172.21.15.3:0/4112683138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa38061a20 0x7faa380b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:57.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 >> 172.21.15.3:0/4112683138 conn(0x7faa38066f70 msgr2=0x7faa3805be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:57.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 shutdown_connections 2022-07-21T01:33:57.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:57.883+0000 7faa3fe47700 1 -- 172.21.15.3:0/4112683138 wait complete. 2022-07-21T01:33:57.890 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:33:57.900 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:33:57.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:33:58.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42b59e700 1 Processor -- start 2022-07-21T01:33:58.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42b59e700 1 -- start start 2022-07-21T01:33:58.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42b59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb42c0ad000 0x7fb42c1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42b59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c143ab0 con 0x7fb42c0b08a0 2022-07-21T01:33:58.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c0b0c80 con 0x7fb42c0b3260 2022-07-21T01:33:58.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42b59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c0ae810 con 0x7fb42c0ad000 2022-07-21T01:33:58.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42ad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb42c0ad000 0x7fb42c1435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42a59c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb42c0b08a0 0x7fb42c142eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52066/0 (socket says 172.21.15.3:52066) 2022-07-21T01:33:58.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42a59c700 1 -- 172.21.15.3:0/1482732204 learned_addr learned my addr 172.21.15.3:0/1482732204 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:58.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42ad9d700 1 -- 172.21.15.3:0/1482732204 >> v1:172.21.15.143:6789/0 conn(0x7fb42c0b3260 legacy=0x7fb42c13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42ad9d700 1 -- 172.21.15.3:0/1482732204 >> v1:172.21.15.3:6789/0 conn(0x7fb42c0b08a0 legacy=0x7fb42c142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42ad9d700 1 -- 172.21.15.3:0/1482732204 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c0aeab0 con 0x7fb42c0ad000 2022-07-21T01:33:58.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.051+0000 7fb42ad9d700 1 --2- 172.21.15.3:0/1482732204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 0x7fb42c1435c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb4240032f0 tx=0x7fb42400a370).ready entity=mon.0 client_cookie=b101d5a51addf373 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:58.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42959a700 1 -- 172.21.15.3:0/1482732204 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42400cc40 con 0x7fb42c0ad000 2022-07-21T01:33:58.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42959a700 1 -- 172.21.15.3:0/1482732204 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42400cda0 con 0x7fb42c0ad000 2022-07-21T01:33:58.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42959a700 1 -- 172.21.15.3:0/1482732204 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb424010600 con 0x7fb42c0ad000 2022-07-21T01:33:58.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 -- 172.21.15.3:0/1482732204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 msgr2=0x7fb42c1435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 --2- 172.21.15.3:0/1482732204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 0x7fb42c1435c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb4240032f0 tx=0x7fb42400a370).stop 2022-07-21T01:33:58.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 -- 172.21.15.3:0/1482732204 shutdown_connections 2022-07-21T01:33:58.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 --2- 172.21.15.3:0/1482732204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 0x7fb42c1435c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 -- 172.21.15.3:0/1482732204 >> 172.21.15.3:0/1482732204 conn(0x7fb42c0612d0 msgr2=0x7fb42c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:58.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 -- 172.21.15.3:0/1482732204 shutdown_connections 2022-07-21T01:33:58.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 -- 172.21.15.3:0/1482732204 wait complete. 2022-07-21T01:33:58.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 Processor -- start 2022-07-21T01:33:58.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 -- start start 2022-07-21T01:33:58.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 0x7fb42c0a3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb42c0b08a0 0x7fb42c1563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb42c0b3260 0x7fb42c1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c156dd0 con 0x7fb42c0ad000 2022-07-21T01:33:58.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c067a30 con 0x7fb42c0b08a0 2022-07-21T01:33:58.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c065d50 con 0x7fb42c0b3260 2022-07-21T01:33:58.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 0x7fb42c0a3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 0x7fb42c0a3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43790/0 (socket says 172.21.15.3:43790) 2022-07-21T01:33:58.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42a59c700 1 -- 172.21.15.3:0/330491787 learned_addr learned my addr 172.21.15.3:0/330491787 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:58.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb429d9b700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb42c0b08a0 0x7fb42c1563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.055+0000 7fb42ad9d700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb42c0b3260 0x7fb42c1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42a59c700 1 -- 172.21.15.3:0/330491787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb42c0b3260 msgr2=0x7fb42c1568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42a59c700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb42c0b3260 0x7fb42c1568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42a59c700 1 -- 172.21.15.3:0/330491787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb42c0b08a0 msgr2=0x7fb42c1563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42a59c700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb42c0b08a0 0x7fb42c1563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42a59c700 1 -- 172.21.15.3:0/330491787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42400a610 con 0x7fb42c0ad000 2022-07-21T01:33:58.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42ad9d700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb42c0b3260 0x7fb42c1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:58.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42a59c700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 0x7fb42c0a3190 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb42c0b6bd0 tx=0x7fb41c007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:58.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb4237fe700 1 -- 172.21.15.3:0/330491787 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb41c0070d0 con 0x7fb42c0ad000 2022-07-21T01:33:58.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb4237fe700 1 -- 172.21.15.3:0/330491787 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c007230 con 0x7fb42c0ad000 2022-07-21T01:33:58.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb4237fe700 1 -- 172.21.15.3:0/330491787 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb41c005610 con 0x7fb42c0ad000 2022-07-21T01:33:58.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c065ff0 con 0x7fb42c0ad000 2022-07-21T01:33:58.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c065190 con 0x7fb42c0ad000 2022-07-21T01:33:58.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c0574e0 con 0x7fb42c0ad000 2022-07-21T01:33:58.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb4237fe700 1 -- 172.21.15.3:0/330491787 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb41c010070 con 0x7fb42c0ad000 2022-07-21T01:33:58.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb4237fe700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb404053aa0 0x7fb404055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb429d9b700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb404053aa0 0x7fb404055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.059+0000 7fb4237fe700 1 -- 172.21.15.3:0/330491787 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fb41c06b340 con 0x7fb42c0ad000 2022-07-21T01:33:58.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.063+0000 7fb429d9b700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb404053aa0 0x7fb404055f60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fb41400d000 tx=0x7fb414007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:58.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.067+0000 7fb4237fe700 1 -- 172.21.15.3:0/330491787 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb41c042330 con 0x7fb42c0ad000 2022-07-21T01:33:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.235+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb42c067dd0 con 0x7fb42c0ad000 2022-07-21T01:33:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.235+0000 7fb4237fe700 1 -- 172.21.15.3:0/330491787 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) v1 ==== 80+0+5621 (secure 0 0 0) 0x7fb41c018070 con 0x7fb42c0ad000 2022-07-21T01:33:58.238 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:33:58.239 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:33:58.239 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:33:58.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.239 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:33:58.240 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:33:58.240 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.240 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:33:58.240 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.241 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.241 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.241 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.242 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:33:58.242 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:33:58.242 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.243 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:33:58.243 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.243 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.243 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.244 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.244 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:33:58.245 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:33:58.245 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.246 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:33:58.246 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.246 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.247 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.247 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.248 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:33:58.248 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:33:58.248 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.249 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.249 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.249 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.249 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.250 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:33:58.250 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:33:58.250 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.251 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.251 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.251 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.251 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.252 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:33:58.252 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:33:58.253 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.253 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.253 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.253 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.254 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.254 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:33:58.255 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:33:58.255 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.255 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.255 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.256 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.256 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.257 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:33:58.257 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:33:58.257 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.257 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.258 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.258 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.258 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.260 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:33:58.260 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:33:58.260 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.261 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.261 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.262 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.262 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.263 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:33:58.263 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:33:58.263 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.263 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.264 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.264 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.264 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.265 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:33:58.265 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:33:58.265 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.266 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.266 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.266 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.266 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.267 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:33:58.268 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:33:58.268 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.269 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:33:58.269 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow r", 2022-07-21T01:33:58.270 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow rw", 2022-07-21T01:33:58.270 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rwx" 2022-07-21T01:33:58.271 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.272 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:33:58.272 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:33:58.272 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.273 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:33:58.273 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow r", 2022-07-21T01:33:58.274 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow rw", 2022-07-21T01:33:58.274 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rwx" 2022-07-21T01:33:58.275 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.276 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:33:58.276 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:33:58.277 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.277 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:33:58.278 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:33:58.278 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:33:58.279 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.279 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.280 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-mds", 2022-07-21T01:33:58.281 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDSrNhiIUM3JxAAWxE5Go6R4u8FU9MKf5zbJQ==", 2022-07-21T01:33:58.281 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.282 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-mds" 2022-07-21T01:33:58.282 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.283 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-mgr", 2022-07-21T01:33:58.284 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDSrNhi21s3JxAA39GsHgs+kel2XzzzH1iJlA==", 2022-07-21T01:33:58.284 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.285 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-mgr" 2022-07-21T01:33:58.285 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.287 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-osd", 2022-07-21T01:33:58.287 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDSrNhi/nE3JxAAAJg6IvLAi4mzBHWdMj+7AQ==", 2022-07-21T01:33:58.287 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.288 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-osd" 2022-07-21T01:33:58.288 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.289 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-rbd", 2022-07-21T01:33:58.290 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDSrNhiF6s3JxAA9NlIgfgeGFazi2A0dnieCg==", 2022-07-21T01:33:58.290 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.291 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-rbd" 2022-07-21T01:33:58.291 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.291 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-rbd-mirror", 2022-07-21T01:33:58.292 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDSrNhiJMQ3JxAAHSTStruJmlNXefyUmFKQ9Q==", 2022-07-21T01:33:58.292 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.292 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2022-07-21T01:33:58.292 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.294 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.bootstrap-rgw", 2022-07-21T01:33:58.294 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDSrNhiRdw3JxAAh0y162QhFvbYVM7BNTjBmA==", 2022-07-21T01:33:58.294 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.294 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile bootstrap-rgw" 2022-07-21T01:33:58.295 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.295 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:33:58.296 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:33:58.296 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.296 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:33:58.297 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:33:58.297 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.297 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:33:58.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:33:58.298 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:33:58.298 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:33:58.299 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:33:58.299 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:33:58.299 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:33:58.299 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:33:58.300 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.300 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:33:58.300 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:33:58.300 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:33:58.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb404053aa0 msgr2=0x7fb404055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb404053aa0 0x7fb404055f60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fb41400d000 tx=0x7fb414007b90).stop 2022-07-21T01:33:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 msgr2=0x7fb42c0a3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 0x7fb42c0a3190 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb42c0b6bd0 tx=0x7fb41c007c40).stop 2022-07-21T01:33:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 shutdown_connections 2022-07-21T01:33:58.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb42c0b3260 0x7fb42c1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb404053aa0 0x7fb404055f60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb42c0b08a0 0x7fb42c1563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 --2- 172.21.15.3:0/330491787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb42c0ad000 0x7fb42c0a3190 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 >> 172.21.15.3:0/330491787 conn(0x7fb42c0612d0 msgr2=0x7fb42c0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:58.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 shutdown_connections 2022-07-21T01:33:58.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.239+0000 7fb42b59e700 1 -- 172.21.15.3:0/330491787 wait complete. 2022-07-21T01:33:58.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2022-07-21T01:33:58.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc747b14700 1 Processor -- start 2022-07-21T01:33:58.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc747b14700 1 -- start start 2022-07-21T01:33:58.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc747b14700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc740148530 0x7fc7400b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc747b14700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7400b04e0 con 0x7fc7400a7250 2022-07-21T01:33:58.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc747b14700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7400b0780 con 0x7fc7400b1f40 2022-07-21T01:33:58.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc747b14700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7400a9250 con 0x7fc740148530 2022-07-21T01:33:58.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc7460b1700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc7400a7250 0x7fc7401448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52082/0 (socket says 172.21.15.3:52082) 2022-07-21T01:33:58.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc7460b1700 1 -- 172.21.15.3:0/193512138 learned_addr learned my addr 172.21.15.3:0/193512138 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:58.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc7450af700 1 --2- 172.21.15.3:0/193512138 >> v2:172.21.15.143:3300/0 conn(0x7fc740148530 0x7fc7400b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc7448ae700 1 -- 172.21.15.3:0/193512138 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921539899 0 0) 0x7fc7400b04e0 con 0x7fc7400a7250 2022-07-21T01:33:58.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.411+0000 7fc7448ae700 1 -- 172.21.15.3:0/193512138 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc728003260 con 0x7fc7400a7250 2022-07-21T01:33:58.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc7448ae700 1 -- 172.21.15.3:0/193512138 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638303740 0 0) 0x7fc728003260 con 0x7fc7400a7250 2022-07-21T01:33:58.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc7450af700 1 -- 172.21.15.3:0/193512138 >> v1:172.21.15.143:6789/0 conn(0x7fc7400b1f40 legacy=0x7fc7400b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc7450af700 1 -- 172.21.15.3:0/193512138 >> v1:172.21.15.3:6789/0 conn(0x7fc7400a7250 legacy=0x7fc7401448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc7450af700 1 -- 172.21.15.3:0/193512138 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7400b0a20 con 0x7fc740148530 2022-07-21T01:33:58.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc7450af700 1 --2- 172.21.15.3:0/193512138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740148530 0x7fc7400b1160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fc738001b80 tx=0x7fc73c006df0).ready entity=mon.1 client_cookie=feb9bdc4add0815e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:58.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc7448ae700 1 -- 172.21.15.3:0/193512138 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7380019e0 con 0x7fc740148530 2022-07-21T01:33:58.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc7448ae700 1 -- 172.21.15.3:0/193512138 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c000e90 con 0x7fc740148530 2022-07-21T01:33:58.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc7448ae700 1 -- 172.21.15.3:0/193512138 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc73c00d4f0 con 0x7fc740148530 2022-07-21T01:33:58.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc747b14700 1 -- 172.21.15.3:0/193512138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740148530 msgr2=0x7fc7400b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc747b14700 1 --2- 172.21.15.3:0/193512138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740148530 0x7fc7400b1160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fc738001b80 tx=0x7fc73c006df0).stop 2022-07-21T01:33:58.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc747b14700 1 -- 172.21.15.3:0/193512138 shutdown_connections 2022-07-21T01:33:58.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc747b14700 1 --2- 172.21.15.3:0/193512138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740148530 0x7fc7400b1160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc747b14700 1 -- 172.21.15.3:0/193512138 >> 172.21.15.3:0/193512138 conn(0x7fc74005f840 msgr2=0x7fc74005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:58.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc747b14700 1 -- 172.21.15.3:0/193512138 shutdown_connections 2022-07-21T01:33:58.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc747b14700 1 -- 172.21.15.3:0/193512138 wait complete. 2022-07-21T01:33:58.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc747b14700 1 Processor -- start 2022-07-21T01:33:58.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.415+0000 7fc747b14700 1 -- start start 2022-07-21T01:33:58.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc747b14700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7400a7250 0x7fc74014b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc747b14700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7400b1f40 0x7fc740149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc747b14700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740148530 0x7fc7400abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc747b14700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74014a620 con 0x7fc7400b1f40 2022-07-21T01:33:58.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc747b14700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7400ab240 con 0x7fc740148530 2022-07-21T01:33:58.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc747b14700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7400ab4e0 con 0x7fc7400a7250 2022-07-21T01:33:58.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7450af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7400b1f40 0x7fc740149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7450af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7400b1f40 0x7fc740149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43806/0 (socket says 172.21.15.3:43806) 2022-07-21T01:33:58.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7450af700 1 -- 172.21.15.3:0/3873341576 learned_addr learned my addr 172.21.15.3:0/3873341576 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:58.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7458b0700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7400a7250 0x7fc74014b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7460b1700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740148530 0x7fc7400abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7450af700 1 -- 172.21.15.3:0/3873341576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7400a7250 msgr2=0x7fc74014b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7450af700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7400a7250 0x7fc74014b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7450af700 1 -- 172.21.15.3:0/3873341576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740148530 msgr2=0x7fc7400abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7450af700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740148530 0x7fc7400abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7450af700 1 -- 172.21.15.3:0/3873341576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c00b180 con 0x7fc7400b1f40 2022-07-21T01:33:58.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc7450af700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7400b1f40 0x7fc740149d80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc738001b80 tx=0x7fc73c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:58.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc72effd700 1 -- 172.21.15.3:0/3873341576 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc73c016070 con 0x7fc7400b1f40 2022-07-21T01:33:58.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc72effd700 1 -- 172.21.15.3:0/3873341576 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c013af0 con 0x7fc7400b1f40 2022-07-21T01:33:58.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7400ab780 con 0x7fc7400b1f40 2022-07-21T01:33:58.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc72effd700 1 -- 172.21.15.3:0/3873341576 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc73c00d4f0 con 0x7fc7400b1f40 2022-07-21T01:33:58.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.419+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74014a270 con 0x7fc7400b1f40 2022-07-21T01:33:58.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.423+0000 7fc72effd700 1 -- 172.21.15.3:0/3873341576 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc73c00f030 con 0x7fc7400b1f40 2022-07-21T01:33:58.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.423+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7400ab910 con 0x7fc7400b1f40 2022-07-21T01:33:58.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.423+0000 7fc72effd700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc720053950 0x7fc720055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.423+0000 7fc72effd700 1 -- 172.21.15.3:0/3873341576 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fc73c06bfe0 con 0x7fc7400b1f40 2022-07-21T01:33:58.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.423+0000 7fc7458b0700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc720053950 0x7fc720055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.423+0000 7fc7458b0700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc720053950 0x7fc720055e10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc73000d000 tx=0x7fc730009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:58.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.427+0000 7fc72effd700 1 -- 172.21.15.3:0/3873341576 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc73c0430e0 con 0x7fc7400b1f40 2022-07-21T01:33:58.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.591+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fc7400b0cc0 con 0x7fc7400b1f40 2022-07-21T01:33:58.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.599+0000 7fc72effd700 1 -- 172.21.15.3:0/3873341576 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc73c046700 con 0x7fc7400b1f40 2022-07-21T01:33:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc720053950 msgr2=0x7fc720055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc720053950 0x7fc720055e10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc73000d000 tx=0x7fc730009040).stop 2022-07-21T01:33:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7400b1f40 msgr2=0x7fc740149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7400b1f40 0x7fc740149d80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc738001b80 tx=0x7fc73c014000).stop 2022-07-21T01:33:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 shutdown_connections 2022-07-21T01:33:58.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc720053950 0x7fc720055e10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740148530 0x7fc7400abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7400b1f40 0x7fc740149d80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 --2- 172.21.15.3:0/3873341576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7400a7250 0x7fc74014b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 >> 172.21.15.3:0/3873341576 conn(0x7fc74005f840 msgr2=0x7fc740147990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 shutdown_connections 2022-07-21T01:33:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.603+0000 7fc747b14700 1 -- 172.21.15.3:0/3873341576 wait complete. 2022-07-21T01:33:58.613 INFO:teuthology.orchestra.run.smithi003.stderr:updated 2022-07-21T01:33:58.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2022-07-21T01:33:58.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135756e700 1 Processor -- start 2022-07-21T01:33:58.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135756e700 1 -- start start 2022-07-21T01:33:58.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135756e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f13500b6710 0x7f13500b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135756e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1350144240 con 0x7f13500b4550 2022-07-21T01:33:58.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135756e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1350143d00 con 0x7f13500b6af0 2022-07-21T01:33:58.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135756e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1350143fa0 con 0x7f13500b6710 2022-07-21T01:33:58.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f1354b09700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f13500b4550 0x7f1350061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52094/0 (socket says 172.21.15.3:52094) 2022-07-21T01:33:58.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f1354b09700 1 -- 172.21.15.3:0/2713954070 learned_addr learned my addr 172.21.15.3:0/2713954070 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:58.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135530a700 1 --2- 172.21.15.3:0/2713954070 >> v2:172.21.15.143:3300/0 conn(0x7f13500b6710 0x7f13500b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f1347fff700 1 -- 172.21.15.3:0/2713954070 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659006940 0 0) 0x7f1350144240 con 0x7f13500b4550 2022-07-21T01:33:58.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f1347fff700 1 -- 172.21.15.3:0/2713954070 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1334003260 con 0x7f13500b4550 2022-07-21T01:33:58.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135530a700 1 -- 172.21.15.3:0/2713954070 >> v1:172.21.15.143:6789/0 conn(0x7f13500b6af0 legacy=0x7f135012bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f1347fff700 1 -- 172.21.15.3:0/2713954070 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684891247 0 0) 0x7f1350143d00 con 0x7f13500b6af0 2022-07-21T01:33:58.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135530a700 1 -- 172.21.15.3:0/2713954070 >> v1:172.21.15.3:6789/0 conn(0x7f13500b4550 legacy=0x7f1350061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135530a700 1 -- 172.21.15.3:0/2713954070 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1350061780 con 0x7f13500b6710 2022-07-21T01:33:58.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f135530a700 1 --2- 172.21.15.3:0/2713954070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b6710 0x7f13500b5d80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f1340001800 tx=0x7f134c006df0).ready entity=mon.1 client_cookie=4da02457e2e4dfa9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:58.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f1347fff700 1 -- 172.21.15.3:0/2713954070 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1340001660 con 0x7f13500b6710 2022-07-21T01:33:58.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f1347fff700 1 -- 172.21.15.3:0/2713954070 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c000e90 con 0x7f13500b6710 2022-07-21T01:33:58.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.719+0000 7f1347fff700 1 -- 172.21.15.3:0/2713954070 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f134c00d4f0 con 0x7f13500b6710 2022-07-21T01:33:58.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- 172.21.15.3:0/2713954070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b6710 msgr2=0x7f13500b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 --2- 172.21.15.3:0/2713954070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b6710 0x7f13500b5d80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f1340001800 tx=0x7f134c006df0).stop 2022-07-21T01:33:58.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- 172.21.15.3:0/2713954070 shutdown_connections 2022-07-21T01:33:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 --2- 172.21.15.3:0/2713954070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b6710 0x7f13500b5d80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- 172.21.15.3:0/2713954070 >> 172.21.15.3:0/2713954070 conn(0x7f135005c620 msgr2=0x7f135005ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- 172.21.15.3:0/2713954070 shutdown_connections 2022-07-21T01:33:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- 172.21.15.3:0/2713954070 wait complete. 2022-07-21T01:33:58.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 Processor -- start 2022-07-21T01:33:58.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- start start 2022-07-21T01:33:58.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b4550 0x7f135012acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13500b6710 0x7f1350140460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13500b6af0 0x7f1350140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135012b9e0 con 0x7f13500b6af0 2022-07-21T01:33:58.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13500b0490 con 0x7f13500b4550 2022-07-21T01:33:58.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13500ae9f0 con 0x7f13500b6710 2022-07-21T01:33:58.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1355b0b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13500b6af0 0x7f1350140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1355b0b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13500b6af0 0x7f1350140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43820/0 (socket says 172.21.15.3:43820) 2022-07-21T01:33:58.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1355b0b700 1 -- 172.21.15.3:0/2567605551 learned_addr learned my addr 172.21.15.3:0/2567605551 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:58.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135530a700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b4550 0x7f135012acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1354b09700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13500b6710 0x7f1350140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1355b0b700 1 -- 172.21.15.3:0/2567605551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13500b6710 msgr2=0x7f1350140460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1355b0b700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13500b6710 0x7f1350140460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1355b0b700 1 -- 172.21.15.3:0/2567605551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b4550 msgr2=0x7f135012acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1355b0b700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b4550 0x7f135012acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1355b0b700 1 -- 172.21.15.3:0/2567605551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c00b180 con 0x7f13500b6af0 2022-07-21T01:33:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135530a700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b4550 0x7f135012acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:33:58.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f1355b0b700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13500b6af0 0x7f1350140950 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f1348000f40 tx=0x7f1348007d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:58.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f13467fc700 1 -- 172.21.15.3:0/2567605551 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f134800e4a0 con 0x7f13500b6af0 2022-07-21T01:33:58.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f13467fc700 1 -- 172.21.15.3:0/2567605551 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134800e600 con 0x7f13500b6af0 2022-07-21T01:33:58.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13500afd50 con 0x7f13500b6af0 2022-07-21T01:33:58.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.723+0000 7f13467fc700 1 -- 172.21.15.3:0/2567605551 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f134800e960 con 0x7f13500b6af0 2022-07-21T01:33:58.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.727+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1350140e40 con 0x7f13500b6af0 2022-07-21T01:33:58.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.727+0000 7f13467fc700 1 -- 172.21.15.3:0/2567605551 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1348010070 con 0x7f13500b6af0 2022-07-21T01:33:58.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.727+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13500aebf0 con 0x7f13500b6af0 2022-07-21T01:33:58.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.727+0000 7f13467fc700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1338053b00 0x7f1338055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:58.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.727+0000 7f135530a700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1338053b00 0x7f1338055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:58.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.727+0000 7f13467fc700 1 -- 172.21.15.3:0/2567605551 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f134806b190 con 0x7f13500b6af0 2022-07-21T01:33:58.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.727+0000 7f135530a700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1338053b00 0x7f1338055fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f134c017000 tx=0x7f1350143e90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:58.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.731+0000 7f13467fc700 1 -- 172.21.15.3:0/2567605551 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1348042290 con 0x7f13500b6af0 2022-07-21T01:33:58.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.898+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f1350061a20 con 0x7f13500b6af0 2022-07-21T01:33:58.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f13467fc700 1 -- 172.21.15.3:0/2567605551 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7f13480458b0 con 0x7f13500b6af0 2022-07-21T01:33:58.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1338053b00 msgr2=0x7f1338055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1338053b00 0x7f1338055fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f134c017000 tx=0x7f1350143e90).stop 2022-07-21T01:33:58.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13500b6af0 msgr2=0x7f1350140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13500b6af0 0x7f1350140950 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f1348000f40 tx=0x7f1348007d60).stop 2022-07-21T01:33:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 shutdown_connections 2022-07-21T01:33:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13500b6af0 0x7f1350140950 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13500b6710 0x7f1350140460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1338053b00 0x7f1338055fc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 --2- 172.21.15.3:0/2567605551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13500b4550 0x7f135012acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:58.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 >> 172.21.15.3:0/2567605551 conn(0x7f135005c620 msgr2=0x7f13501432f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:58.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 shutdown_connections 2022-07-21T01:33:58.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:58.906+0000 7f135756e700 1 -- 172.21.15.3:0/2567605551 wait complete. 2022-07-21T01:33:58.915 INFO:teuthology.orchestra.run.smithi003.stderr:updated 2022-07-21T01:33:58.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2022-07-21T01:33:59.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 Processor -- start 2022-07-21T01:33:59.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 -- start start 2022-07-21T01:33:59.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5bb40b31d0 0x7f5bb4066020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb41402d0 0x7f5bb4140780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb4067030 con 0x7f5bb40a9da0 2022-07-21T01:33:59.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb4064a70 con 0x7f5bb40b31d0 2022-07-21T01:33:59.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4061f60 con 0x7f5bb41402d0 2022-07-21T01:33:59.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5ba9d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5bb40a9da0 0x7f5bb4060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52106/0 (socket says 172.21.15.3:52106) 2022-07-21T01:33:59.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5ba9d9b700 1 -- 172.21.15.3:0/780898755 learned_addr learned my addr 172.21.15.3:0/780898755 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:59.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb259c700 1 --2- 172.21.15.3:0/780898755 >> v2:172.21.15.143:3300/0 conn(0x7f5bb40b31d0 0x7f5bb4066020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb2d9d700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb41402d0 0x7f5bb4140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403361371 0 0) 0x7f5bb4067030 con 0x7f5bb40a9da0 2022-07-21T01:33:59.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b94003260 con 0x7f5bb40a9da0 2022-07-21T01:33:59.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2559715916 0 0) 0x7f5b94003260 con 0x7f5bb40a9da0 2022-07-21T01:33:59.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb41402d0 msgr2=0x7f5bb4140780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb41402d0 0x7f5bb4140780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 >> v2:172.21.15.143:3300/0 conn(0x7f5bb40b31d0 msgr2=0x7f5bb4066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 --2- 172.21.15.3:0/780898755 >> v2:172.21.15.143:3300/0 conn(0x7f5bb40b31d0 0x7f5bb4066020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb40b35b0 con 0x7f5bb40a9da0 2022-07-21T01:33:59.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f5b9c001660 con 0x7f5bb40a9da0 2022-07-21T01:33:59.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 >> v1:172.21.15.3:6789/0 conn(0x7f5bb40a9da0 legacy=0x7f5bb4060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb41402d0 0x7f5b94003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b94003c80 0x7f5b94004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b940049f0 0x7f5b94004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9400fb60 con 0x7f5b94003c80 2022-07-21T01:33:59.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9400fca0 con 0x7f5b940049f0 2022-07-21T01:33:59.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9400fde0 con 0x7f5bb41402d0 2022-07-21T01:33:59.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5ba9d9b700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b94003c80 0x7f5b94004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb2d9d700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b940049f0 0x7f5b94004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb259c700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb41402d0 0x7f5b94003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5ba9d9b700 1 -- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb41402d0 msgr2=0x7f5b94003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5ba9d9b700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb41402d0 0x7f5b94003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5ba9d9b700 1 -- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b940049f0 msgr2=0x7f5b94004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5ba9d9b700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b940049f0 0x7f5b94004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5ba9d9b700 1 -- 172.21.15.3:0/780898755 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b94003260 con 0x7f5b94003c80 2022-07-21T01:33:59.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb259c700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb41402d0 0x7f5b94003750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:33:59.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5ba9d9b700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b94003c80 0x7f5b94004500 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f5b940054d0 tx=0x7f5b9c001e50).ready entity=mon.0 client_cookie=6282f00d7ab1f9b2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b9c0070d0 con 0x7f5b94003c80 2022-07-21T01:33:59.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c007230 con 0x7f5b94003c80 2022-07-21T01:33:59.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb1d9b700 1 -- 172.21.15.3:0/780898755 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b9c00c980 con 0x7f5b94003c80 2022-07-21T01:33:59.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 -- 172.21.15.3:0/780898755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b94003c80 msgr2=0x7f5b94004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 --2- 172.21.15.3:0/780898755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b94003c80 0x7f5b94004500 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f5b940054d0 tx=0x7f5b9c001e50).stop 2022-07-21T01:33:59.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb2d9d700 1 -- 172.21.15.3:0/780898755 reap_dead start 2022-07-21T01:33:59.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 -- 172.21.15.3:0/780898755 shutdown_connections 2022-07-21T01:33:59.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.026+0000 7f5bb359e700 1 -- 172.21.15.3:0/780898755 >> 172.21.15.3:0/780898755 conn(0x7f5bb4063870 msgr2=0x7f5bb4063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:59.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 -- 172.21.15.3:0/780898755 shutdown_connections 2022-07-21T01:33:59.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 -- 172.21.15.3:0/780898755 wait complete. 2022-07-21T01:33:59.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 Processor -- start 2022-07-21T01:33:59.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 -- start start 2022-07-21T01:33:59.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b94003c80 0x7f5bb4143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bb413bb20 0x7f5bb414dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb4142360 0x7f5bb414dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb413b6f0 con 0x7f5bb413bb20 2022-07-21T01:33:59.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb41519d0 con 0x7f5b94003c80 2022-07-21T01:33:59.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4151c40 con 0x7f5bb4142360 2022-07-21T01:33:59.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5ba9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bb413bb20 0x7f5bb414dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5ba9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bb413bb20 0x7f5bb414dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43840/0 (socket says 172.21.15.3:43840) 2022-07-21T01:33:59.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5ba9d9b700 1 -- 172.21.15.3:0/1230567008 learned_addr learned my addr 172.21.15.3:0/1230567008 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:59.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb259c700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b94003c80 0x7f5bb4143e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb2d9d700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb4142360 0x7f5bb414dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5ba9d9b700 1 -- 172.21.15.3:0/1230567008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb4142360 msgr2=0x7f5bb414dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5ba9d9b700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb4142360 0x7f5bb414dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5ba9d9b700 1 -- 172.21.15.3:0/1230567008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b94003c80 msgr2=0x7f5bb4143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5ba9d9b700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b94003c80 0x7f5bb4143e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5ba9d9b700 1 -- 172.21.15.3:0/1230567008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb4151ee0 con 0x7f5bb413bb20 2022-07-21T01:33:59.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5ba9d9b700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bb413bb20 0x7f5bb414dae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f5bb4151b60 tx=0x7f5b9c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5babfff700 1 -- 172.21.15.3:0/1230567008 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b9c0070d0 con 0x7f5bb413bb20 2022-07-21T01:33:59.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5babfff700 1 -- 172.21.15.3:0/1230567008 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c004730 con 0x7f5bb413bb20 2022-07-21T01:33:59.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5babfff700 1 -- 172.21.15.3:0/1230567008 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b9c004a90 con 0x7f5bb413bb20 2022-07-21T01:33:59.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb4146060 con 0x7f5bb413bb20 2022-07-21T01:33:59.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.030+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb413c510 con 0x7f5bb413bb20 2022-07-21T01:33:59.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.034+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb413b860 con 0x7f5bb413bb20 2022-07-21T01:33:59.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.034+0000 7f5babfff700 1 -- 172.21.15.3:0/1230567008 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5b9c010070 con 0x7f5bb413bb20 2022-07-21T01:33:59.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.034+0000 7f5babfff700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b98053aa0 0x7f5b98055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.034+0000 7f5babfff700 1 -- 172.21.15.3:0/1230567008 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f5b9c06cde0 con 0x7f5bb413bb20 2022-07-21T01:33:59.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.034+0000 7f5bb259c700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b98053aa0 0x7f5b98055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.034+0000 7f5bb259c700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b98053aa0 0x7f5b98055f60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f5ba400b000 tx=0x7f5ba4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.038+0000 7f5babfff700 1 -- 172.21.15.3:0/1230567008 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5bb413b860 con 0x7f5bb413bb20 2022-07-21T01:33:59.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.202+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f5bb4066af0 con 0x7f5bb413bb20 2022-07-21T01:33:59.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5babfff700 1 -- 172.21.15.3:0/1230567008 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5b9c025030 con 0x7f5bb413bb20 2022-07-21T01:33:59.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b98053aa0 msgr2=0x7f5b98055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b98053aa0 0x7f5b98055f60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f5ba400b000 tx=0x7f5ba4008040).stop 2022-07-21T01:33:59.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bb413bb20 msgr2=0x7f5bb414dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bb413bb20 0x7f5bb414dae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f5bb4151b60 tx=0x7f5b9c015000).stop 2022-07-21T01:33:59.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 shutdown_connections 2022-07-21T01:33:59.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bb4142360 0x7f5bb414dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bb413bb20 0x7f5bb414dae0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b98053aa0 0x7f5b98055f60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 --2- 172.21.15.3:0/1230567008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b94003c80 0x7f5bb4143e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 >> 172.21.15.3:0/1230567008 conn(0x7f5bb4063870 msgr2=0x7f5bb40b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:59.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 shutdown_connections 2022-07-21T01:33:59.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.210+0000 7f5bb359e700 1 -- 172.21.15.3:0/1230567008 wait complete. 2022-07-21T01:33:59.218 INFO:teuthology.orchestra.run.smithi003.stderr:updated 2022-07-21T01:33:59.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2022-07-21T01:33:59.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.330+0000 7f0d7b59e700 1 Processor -- start 2022-07-21T01:33:59.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.330+0000 7f0d7b59e700 1 -- start start 2022-07-21T01:33:59.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.330+0000 7f0d7b59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0d7c0b2710 0x7f0d7c14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.330+0000 7f0d7b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0d7c0b01a0 0x7f0d7c14b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.330+0000 7f0d7b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 0x7f0d7c0a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.330+0000 7f0d7b59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7c0a7250 con 0x7f0d7c0b2710 2022-07-21T01:33:59.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.330+0000 7f0d7b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7c0a7480 con 0x7f0d7c0b01a0 2022-07-21T01:33:59.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.330+0000 7f0d7b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0a7720 con 0x7f0d7c0afd60 2022-07-21T01:33:59.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0d7c0b2710 0x7f0d7c14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0d7c0b2710 0x7f0d7c14afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43846/0 (socket says 172.21.15.3:43846) 2022-07-21T01:33:59.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7a59c700 1 -- 172.21.15.3:0/1159888391 learned_addr learned my addr 172.21.15.3:0/1159888391 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:59.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d79d9b700 1 --2- 172.21.15.3:0/1159888391 >> v2:172.21.15.143:3300/0 conn(0x7f0d7c0b01a0 0x7f0d7c14b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7ad9d700 1 --2- 172.21.15.3:0/1159888391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 0x7f0d7c0a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d79d9b700 1 -- 172.21.15.3:0/1159888391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 msgr2=0x7f0d7c0a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d79d9b700 1 --2- 172.21.15.3:0/1159888391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 0x7f0d7c0a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d79d9b700 1 -- 172.21.15.3:0/1159888391 >> v2:172.21.15.3:3300/0 conn(0x7f0d7c0b2710 msgr2=0x7f0d7c14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d79d9b700 1 --2- 172.21.15.3:0/1159888391 >> v2:172.21.15.3:3300/0 conn(0x7f0d7c0b2710 0x7f0d7c14afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d79d9b700 1 -- 172.21.15.3:0/1159888391 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c0a79c0 con 0x7f0d7c0b01a0 2022-07-21T01:33:59.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7a59c700 1 --2- 172.21.15.3:0/1159888391 >> v2:172.21.15.3:3300/0 conn(0x7f0d7c0b2710 0x7f0d7c14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:59.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d79d9b700 1 --2- 172.21.15.3:0/1159888391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 0x7f0d7c14b4c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0d6c0032f0 tx=0x7f0d6c0091d0).ready entity=mon.1 client_cookie=38f7dd22cdcf92da server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7959a700 1 -- 172.21.15.3:0/1159888391 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d6c00b600 con 0x7f0d7c0b01a0 2022-07-21T01:33:59.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7959a700 1 -- 172.21.15.3:0/1159888391 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c00b760 con 0x7f0d7c0b01a0 2022-07-21T01:33:59.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7959a700 1 -- 172.21.15.3:0/1159888391 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d6c00b8d0 con 0x7f0d7c0b01a0 2022-07-21T01:33:59.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1159888391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 msgr2=0x7f0d7c14b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1159888391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 0x7f0d7c14b4c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0d6c0032f0 tx=0x7f0d6c0091d0).stop 2022-07-21T01:33:59.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1159888391 shutdown_connections 2022-07-21T01:33:59.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1159888391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 0x7f0d7c0a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1159888391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 0x7f0d7c14b4c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1159888391 >> v2:172.21.15.3:3300/0 conn(0x7f0d7c0b2710 0x7f0d7c14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1159888391 >> 172.21.15.3:0/1159888391 conn(0x7f0d7c060310 msgr2=0x7f0d7c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:59.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1159888391 shutdown_connections 2022-07-21T01:33:59.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.334+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1159888391 wait complete. 2022-07-21T01:33:59.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7b59e700 1 Processor -- start 2022-07-21T01:33:59.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7b59e700 1 -- start start 2022-07-21T01:33:59.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 0x7f0d7c0abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 0x7f0d7c0af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d7c0b2710 0x7f0d7c0b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0aa190 con 0x7f0d7c0b2710 2022-07-21T01:33:59.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0a92d0 con 0x7f0d7c0b01a0 2022-07-21T01:33:59.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0aa3c0 con 0x7f0d7c0afd60 2022-07-21T01:33:59.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d7c0b2710 0x7f0d7c0b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d7c0b2710 0x7f0d7c0b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43856/0 (socket says 172.21.15.3:43856) 2022-07-21T01:33:59.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7ad9d700 1 -- 172.21.15.3:0/1842974219 learned_addr learned my addr 172.21.15.3:0/1842974219 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:59.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d79d9b700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 0x7f0d7c0af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.338+0000 7f0d7a59c700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 0x7f0d7c0abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7ad9d700 1 -- 172.21.15.3:0/1842974219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 msgr2=0x7f0d7c0abfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7ad9d700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 0x7f0d7c0abfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7ad9d700 1 -- 172.21.15.3:0/1842974219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 msgr2=0x7f0d7c0af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7ad9d700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 0x7f0d7c0af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7ad9d700 1 -- 172.21.15.3:0/1842974219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7000b1e0 con 0x7f0d7c0b2710 2022-07-21T01:33:59.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d79d9b700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 0x7f0d7c0af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:33:59.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7ad9d700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d7c0b2710 0x7f0d7c0b5cd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f0d7400c000 tx=0x7f0d74006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d6b7fe700 1 -- 172.21.15.3:0/1842974219 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d74010780 con 0x7f0d7c0b2710 2022-07-21T01:33:59.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c009470 con 0x7f0d7c0b2710 2022-07-21T01:33:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d6b7fe700 1 -- 172.21.15.3:0/1842974219 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d740108e0 con 0x7f0d7c0b2710 2022-07-21T01:33:59.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d6b7fe700 1 -- 172.21.15.3:0/1842974219 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d74010c40 con 0x7f0d7c0b2710 2022-07-21T01:33:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7c0a97a0 con 0x7f0d7c0b2710 2022-07-21T01:33:59.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7c14d9c0 con 0x7f0d7c0b2710 2022-07-21T01:33:59.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d6b7fe700 1 -- 172.21.15.3:0/1842974219 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0d7401c460 con 0x7f0d7c0b2710 2022-07-21T01:33:59.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d6b7fe700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0d58053950 0x7f0d58055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d6b7fe700 1 -- 172.21.15.3:0/1842974219 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f0d7406c270 con 0x7f0d7c0b2710 2022-07-21T01:33:59.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.342+0000 7f0d7a59c700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0d58053950 0x7f0d58055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.346+0000 7f0d7a59c700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0d58053950 0x7f0d58055e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f0d7000b000 tx=0x7f0d70006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.350+0000 7f0d6b7fe700 1 -- 172.21.15.3:0/1842974219 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0d7c14d9c0 con 0x7f0d7c0b2710 2022-07-21T01:33:59.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.514+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f0d7c14d9c0 con 0x7f0d7c0b2710 2022-07-21T01:33:59.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d6b7fe700 1 -- 172.21.15.3:0/1842974219 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v12) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0d7c14d9c0 con 0x7f0d7c0b2710 2022-07-21T01:33:59.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0d58053950 msgr2=0x7f0d58055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0d58053950 0x7f0d58055e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f0d7000b000 tx=0x7f0d70006eb0).stop 2022-07-21T01:33:59.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d7c0b2710 msgr2=0x7f0d7c0b5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d7c0b2710 0x7f0d7c0b5cd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f0d7400c000 tx=0x7f0d74006df0).stop 2022-07-21T01:33:59.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 shutdown_connections 2022-07-21T01:33:59.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d7c0b2710 0x7f0d7c0b5cd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0d58053950 0x7f0d58055e10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d7c0b01a0 0x7f0d7c0af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 --2- 172.21.15.3:0/1842974219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d7c0afd60 0x7f0d7c0abfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 >> 172.21.15.3:0/1842974219 conn(0x7f0d7c060310 msgr2=0x7f0d7c144cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:59.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 shutdown_connections 2022-07-21T01:33:59.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.522+0000 7f0d7b59e700 1 -- 172.21.15.3:0/1842974219 wait complete. 2022-07-21T01:33:59.535 INFO:teuthology.orchestra.run.smithi003.stderr:updated 2022-07-21T01:33:59.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2022-07-21T01:33:59.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21b4aec700 1 Processor -- start 2022-07-21T01:33:59.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21b4aec700 1 -- start start 2022-07-21T01:33:59.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21b4aec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b00b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21b4aec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f21b00b6af0 0x7f21b012bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21b4aec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b0144240 con 0x7f21b00b4550 2022-07-21T01:33:59.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21b4aec700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b0143d00 con 0x7f21b00b6af0 2022-07-21T01:33:59.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21b4aec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0143fa0 con 0x7f21b00b6710 2022-07-21T01:33:59.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21ae59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b00b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21aed9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f21b00b6af0 0x7f21b012bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21add9b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f21b00b4550 0x7f21b0061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:54258/0 (socket says 172.21.15.3:54258) 2022-07-21T01:33:59.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.638+0000 7f21add9b700 1 -- 172.21.15.3:0/2903923497 learned_addr learned my addr 172.21.15.3:0/2903923497 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:59.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ad59a700 1 -- 172.21.15.3:0/2903923497 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529205596 0 0) 0x7f21b0144240 con 0x7f21b00b4550 2022-07-21T01:33:59.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ad59a700 1 -- 172.21.15.3:0/2903923497 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2194003260 con 0x7f21b00b4550 2022-07-21T01:33:59.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ae59c700 1 -- 172.21.15.3:0/2903923497 >> v2:172.21.15.143:3300/0 conn(0x7f21b00b6af0 msgr2=0x7f21b012bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ae59c700 1 --2- 172.21.15.3:0/2903923497 >> v2:172.21.15.143:3300/0 conn(0x7f21b00b6af0 0x7f21b012bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ae59c700 1 -- 172.21.15.3:0/2903923497 >> v1:172.21.15.143:6789/0 conn(0x7f21b00b4550 legacy=0x7f21b0061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ae59c700 1 -- 172.21.15.3:0/2903923497 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b0061780 con 0x7f21b00b6710 2022-07-21T01:33:59.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ae59c700 1 --2- 172.21.15.3:0/2903923497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b00b5d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f21a0002f70 tx=0x7f21a0006df0).ready entity=mon.2 client_cookie=7ae34e2556b38da3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ad59a700 1 -- 172.21.15.3:0/2903923497 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a000d4a0 con 0x7f21b00b6710 2022-07-21T01:33:59.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ad59a700 1 -- 172.21.15.3:0/2903923497 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a0000e90 con 0x7f21b00b6710 2022-07-21T01:33:59.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21ad59a700 1 -- 172.21.15.3:0/2903923497 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a000d7f0 con 0x7f21b00b6710 2022-07-21T01:33:59.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21b4aec700 1 -- 172.21.15.3:0/2903923497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 msgr2=0x7f21b00b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21b4aec700 1 --2- 172.21.15.3:0/2903923497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b00b5d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f21a0002f70 tx=0x7f21a0006df0).stop 2022-07-21T01:33:59.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21b4aec700 1 -- 172.21.15.3:0/2903923497 shutdown_connections 2022-07-21T01:33:59.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21b4aec700 1 --2- 172.21.15.3:0/2903923497 >> v2:172.21.15.143:3300/0 conn(0x7f21b00b6af0 0x7f21b012bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21b4aec700 1 --2- 172.21.15.3:0/2903923497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b00b5d80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21b4aec700 1 -- 172.21.15.3:0/2903923497 >> 172.21.15.3:0/2903923497 conn(0x7f21b005c620 msgr2=0x7f21b005ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:59.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21b4aec700 1 -- 172.21.15.3:0/2903923497 shutdown_connections 2022-07-21T01:33:59.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.642+0000 7f21b4aec700 1 -- 172.21.15.3:0/2903923497 wait complete. 2022-07-21T01:33:59.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 Processor -- start 2022-07-21T01:33:59.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 -- start start 2022-07-21T01:33:59.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21b00b4550 0x7f21b00ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b01477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21b00b6af0 0x7f21b0147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0129910 con 0x7f21b00b6af0 2022-07-21T01:33:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0149660 con 0x7f21b00b4550 2022-07-21T01:33:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b01530e0 con 0x7f21b00b6710 2022-07-21T01:33:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21aed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21b00b6af0 0x7f21b0147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21ae59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21b00b4550 0x7f21b00ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21add9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b01477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21add9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b01477f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:50784/0 (socket says 172.21.15.3:50784) 2022-07-21T01:33:59.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21add9b700 1 -- 172.21.15.3:0/3345760882 learned_addr learned my addr 172.21.15.3:0/3345760882 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:59.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21aed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21b00b6af0 0x7f21b0147ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43870/0 (socket says 172.21.15.3:43870) 2022-07-21T01:33:59.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21add9b700 1 -- 172.21.15.3:0/3345760882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21b00b4550 msgr2=0x7f21b00ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21add9b700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21b00b4550 0x7f21b00ad950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21add9b700 1 -- 172.21.15.3:0/3345760882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21b00b6af0 msgr2=0x7f21b0147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21add9b700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21b00b6af0 0x7f21b0147ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21add9b700 1 -- 172.21.15.3:0/3345760882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a000b180 con 0x7f21b00b6710 2022-07-21T01:33:59.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21add9b700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b01477f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f21a4001110 tx=0x7f21a4007a60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f219f7fe700 1 -- 172.21.15.3:0/3345760882 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a40070d0 con 0x7f21b00b6710 2022-07-21T01:33:59.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f219f7fe700 1 -- 172.21.15.3:0/3345760882 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a4007230 con 0x7f21b00b6710 2022-07-21T01:33:59.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b0153380 con 0x7f21b00b6710 2022-07-21T01:33:59.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f219f7fe700 1 -- 172.21.15.3:0/3345760882 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a4005610 con 0x7f21b00b6710 2022-07-21T01:33:59.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.646+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b00ade40 con 0x7f21b00b6710 2022-07-21T01:33:59.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.650+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b01491a0 con 0x7f21b00b6710 2022-07-21T01:33:59.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.650+0000 7f219f7fe700 1 -- 172.21.15.3:0/3345760882 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f21a401a070 con 0x7f21b00b6710 2022-07-21T01:33:59.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.650+0000 7f219f7fe700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f218c053b30 0x7f218c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.650+0000 7f219f7fe700 1 -- 172.21.15.3:0/3345760882 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f21a406b060 con 0x7f21b00b6710 2022-07-21T01:33:59.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.650+0000 7f21ae59c700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f218c053b30 0x7f218c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.650+0000 7f21ae59c700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f218c053b30 0x7f218c055ff0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f21a001a000 tx=0x7f21a0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.662+0000 7f219f7fe700 1 -- 172.21.15.3:0/3345760882 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f21b01491a0 con 0x7f21b00b6710 2022-07-21T01:33:59.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.830+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f21b01491a0 con 0x7f21b00b6710 2022-07-21T01:33:59.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.842+0000 7f219f7fe700 1 -- 172.21.15.3:0/3345760882 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f21b01491a0 con 0x7f21b00b6710 2022-07-21T01:33:59.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f218c053b30 msgr2=0x7f218c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f218c053b30 0x7f218c055ff0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f21a001a000 tx=0x7f21a0014000).stop 2022-07-21T01:33:59.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 msgr2=0x7f21b01477f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b01477f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f21a4001110 tx=0x7f21a4007a60).stop 2022-07-21T01:33:59.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 shutdown_connections 2022-07-21T01:33:59.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21b00b6af0 0x7f21b0147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21b00b6710 0x7f21b01477f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f218c053b30 0x7f218c055ff0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 --2- 172.21.15.3:0/3345760882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21b00b4550 0x7f21b00ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 >> 172.21.15.3:0/3345760882 conn(0x7f21b005c620 msgr2=0x7f21b0128b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:59.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 shutdown_connections 2022-07-21T01:33:59.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.846+0000 7f21b4aec700 1 -- 172.21.15.3:0/3345760882 wait complete. 2022-07-21T01:33:59.856 INFO:teuthology.orchestra.run.smithi003.stderr:updated 2022-07-21T01:33:59.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2022-07-21T01:33:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13f59e700 1 Processor -- start 2022-07-21T01:33:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13f59e700 1 -- start start 2022-07-21T01:33:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff140066dc0 0x7ff14014eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13f59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400654c0 con 0x7ff1400b2730 2022-07-21T01:33:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400b3570 con 0x7ff1400b3190 2022-07-21T01:33:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff140062190 con 0x7ff140066dc0 2022-07-21T01:33:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff137fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff1400b2730 0x7ff1400666b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52158/0 (socket says 172.21.15.3:52158) 2022-07-21T01:33:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff137fff700 1 -- 172.21.15.3:0/197892278 learned_addr learned my addr 172.21.15.3:0/197892278 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:33:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13ed9d700 1 --2- 172.21.15.3:0/197892278 >> v2:172.21.15.143:3300/0 conn(0x7ff140066dc0 0x7ff14014eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760066335 0 0) 0x7ff1400654c0 con 0x7ff1400b2730 2022-07-21T01:33:59.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff120003260 con 0x7ff1400b2730 2022-07-21T01:33:59.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2848218433 0 0) 0x7ff120003260 con 0x7ff1400b2730 2022-07-21T01:33:59.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 >> v2:172.21.15.143:3300/0 conn(0x7ff140066dc0 msgr2=0x7ff14014eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 --2- 172.21.15.3:0/197892278 >> v2:172.21.15.143:3300/0 conn(0x7ff140066dc0 0x7ff14014eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 >> v1:172.21.15.143:6789/0 conn(0x7ff1400b3190 legacy=0x7ff140060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14005fbf0 con 0x7ff1400b2730 2022-07-21T01:33:59.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7ff128001660 con 0x7ff1400b2730 2022-07-21T01:33:59.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 >> v1:172.21.15.3:6789/0 conn(0x7ff1400b2730 legacy=0x7ff1400666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120003750 0x7ff120003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff120004500 0x7ff120004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff120004e70 0x7ff12000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff140062190 con 0x7ff120003750 2022-07-21T01:33:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12000bd20 con 0x7ff120004500 2022-07-21T01:33:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12000bee0 con 0x7ff120004e70 2022-07-21T01:33:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13e59c700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120003750 0x7ff120003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff137fff700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff120004500 0x7ff120004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13ed9d700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff120004e70 0x7ff12000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff137fff700 1 -- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff120004e70 msgr2=0x7ff12000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff137fff700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff120004e70 0x7ff12000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff137fff700 1 -- 172.21.15.3:0/197892278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120003750 msgr2=0x7ff120003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:33:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff137fff700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120003750 0x7ff120003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff137fff700 1 -- 172.21.15.3:0/197892278 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff120003260 con 0x7ff120004500 2022-07-21T01:33:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13ed9d700 1 -- 172.21.15.3:0/197892278 reap_dead start 2022-07-21T01:33:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff137fff700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff120004500 0x7ff120004980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7ff128001370 tx=0x7ff14005fd80).ready entity=mon.1 client_cookie=65a0c856a6ffdcfe server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1280070d0 con 0x7ff120004500 2022-07-21T01:33:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff128007230 con 0x7ff120004500 2022-07-21T01:33:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.970+0000 7ff13dd9b700 1 -- 172.21.15.3:0/197892278 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff12800c840 con 0x7ff120004500 2022-07-21T01:33:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 -- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff120004500 msgr2=0x7ff120004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff120004500 0x7ff120004980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7ff128001370 tx=0x7ff14005fd80).stop 2022-07-21T01:33:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 -- 172.21.15.3:0/197892278 shutdown_connections 2022-07-21T01:33:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 --2- 172.21.15.3:0/197892278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff120004500 0x7ff120004980 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 -- 172.21.15.3:0/197892278 >> 172.21.15.3:0/197892278 conn(0x7ff140063870 msgr2=0x7ff140063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:33:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 -- 172.21.15.3:0/197892278 shutdown_connections 2022-07-21T01:33:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 -- 172.21.15.3:0/197892278 wait complete. 2022-07-21T01:33:59.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 Processor -- start 2022-07-21T01:33:59.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 -- start start 2022-07-21T01:33:59.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120004500 0x7ff1400a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1401466f0 0x7ff140140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400a8c30 0x7ff1400a90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:33:59.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1401533d0 con 0x7ff120004500 2022-07-21T01:33:59.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff140153130 con 0x7ff1401466f0 2022-07-21T01:33:59.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff140145ef0 con 0x7ff1400a8c30 2022-07-21T01:33:59.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120004500 0x7ff1400a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120004500 0x7ff1400a4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43886/0 (socket says 172.21.15.3:43886) 2022-07-21T01:33:59.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13e59c700 1 -- 172.21.15.3:0/407434595 learned_addr learned my addr 172.21.15.3:0/407434595 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:33:59.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff13ed9d700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400a8c30 0x7ff1400a90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.974+0000 7ff137fff700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1401466f0 0x7ff140140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:33:59.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff13e59c700 1 -- 172.21.15.3:0/407434595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400a8c30 msgr2=0x7ff1400a90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff13e59c700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400a8c30 0x7ff1400a90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff13e59c700 1 -- 172.21.15.3:0/407434595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1401466f0 msgr2=0x7ff140140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:33:59.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff13e59c700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1401466f0 0x7ff140140f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:33:59.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff13e59c700 1 -- 172.21.15.3:0/407434595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff140146190 con 0x7ff120004500 2022-07-21T01:33:59.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff13e59c700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120004500 0x7ff1400a4310 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff13000d000 tx=0x7ff130007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:33:59.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff1377fe700 1 -- 172.21.15.3:0/407434595 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff130002e10 con 0x7ff120004500 2022-07-21T01:33:59.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff1377fe700 1 -- 172.21.15.3:0/407434595 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1300114c0 con 0x7ff120004500 2022-07-21T01:34:00.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff140152d60 con 0x7ff120004500 2022-07-21T01:34:00.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff1377fe700 1 -- 172.21.15.3:0/407434595 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff130011820 con 0x7ff120004500 2022-07-21T01:34:00.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1400a9740 con 0x7ff120004500 2022-07-21T01:34:00.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14005fe90 con 0x7ff120004500 2022-07-21T01:34:00.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff1377fe700 1 -- 172.21.15.3:0/407434595 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff13001a030 con 0x7ff120004500 2022-07-21T01:34:00.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff1377fe700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff124053aa0 0x7ff124055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.978+0000 7ff1377fe700 1 -- 172.21.15.3:0/407434595 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5600+0+0 (secure 0 0 0) 0x7ff13006b310 con 0x7ff120004500 2022-07-21T01:34:00.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.982+0000 7ff137fff700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff124053aa0 0x7ff124055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.982+0000 7ff137fff700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff124053aa0 0x7ff124055f60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff128019000 tx=0x7ff128013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:33:59.986+0000 7ff1377fe700 1 -- 172.21.15.3:0/407434595 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff130042460 con 0x7ff120004500 2022-07-21T01:34:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.154+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7ff1400a9740 con 0x7ff120004500 2022-07-21T01:34:00.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff1377fe700 1 -- 172.21.15.3:0/407434595 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v14) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff1400a9740 con 0x7ff120004500 2022-07-21T01:34:00.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff124053aa0 msgr2=0x7ff124055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff124053aa0 0x7ff124055f60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff128019000 tx=0x7ff128013000).stop 2022-07-21T01:34:00.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120004500 msgr2=0x7ff1400a4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120004500 0x7ff1400a4310 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff13000d000 tx=0x7ff130007a00).stop 2022-07-21T01:34:00.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 shutdown_connections 2022-07-21T01:34:00.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1400a8c30 0x7ff1400a90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff124053aa0 0x7ff124055f60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1401466f0 0x7ff140140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 --2- 172.21.15.3:0/407434595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff120004500 0x7ff1400a4310 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 >> 172.21.15.3:0/407434595 conn(0x7ff140063870 msgr2=0x7ff140143210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:00.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 shutdown_connections 2022-07-21T01:34:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.162+0000 7ff13f59e700 1 -- 172.21.15.3:0/407434595 wait complete. 2022-07-21T01:34:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:updated 2022-07-21T01:34:00.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:34:00.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672b157700 1 Processor -- start 2022-07-21T01:34:00.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672b157700 1 -- start start 2022-07-21T01:34:00.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672b157700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f67240b01a0 0x7f67240a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672b157700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 0x7f67240a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672b157700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67240a7250 con 0x7f67240b2710 2022-07-21T01:34:00.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672b157700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67240a7480 con 0x7f67240b01a0 2022-07-21T01:34:00.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672b157700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67240a7720 con 0x7f67240afd60 2022-07-21T01:34:00.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672a155700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f67240b2710 0x7f672414afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52176/0 (socket says 172.21.15.3:52176) 2022-07-21T01:34:00.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672a155700 1 -- 172.21.15.3:0/721818749 learned_addr learned my addr 172.21.15.3:0/721818749 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:00.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f6729954700 1 --2- 172.21.15.3:0/721818749 >> v2:172.21.15.143:3300/0 conn(0x7f67240b01a0 0x7f67240a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.282+0000 7f672a956700 1 --2- 172.21.15.3:0/721818749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 0x7f67240a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729153700 1 -- 172.21.15.3:0/721818749 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080068603 0 0) 0x7f67240a7250 con 0x7f67240b2710 2022-07-21T01:34:00.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729153700 1 -- 172.21.15.3:0/721818749 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f670c003260 con 0x7f67240b2710 2022-07-21T01:34:00.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729954700 1 -- 172.21.15.3:0/721818749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 msgr2=0x7f67240a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:00.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729954700 1 --2- 172.21.15.3:0/721818749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 0x7f67240a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729954700 1 -- 172.21.15.3:0/721818749 >> v1:172.21.15.3:6789/0 conn(0x7f67240b2710 legacy=0x7f672414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729954700 1 -- 172.21.15.3:0/721818749 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67240a79c0 con 0x7f67240b01a0 2022-07-21T01:34:00.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729954700 1 --2- 172.21.15.3:0/721818749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 0x7f67240a8540 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f6720002f70 tx=0x7f6720006df0).ready entity=mon.1 client_cookie=3738ca0485cb6ebe server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729153700 1 -- 172.21.15.3:0/721818749 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f672000d4a0 con 0x7f67240b01a0 2022-07-21T01:34:00.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729153700 1 -- 172.21.15.3:0/721818749 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720000e90 con 0x7f67240b01a0 2022-07-21T01:34:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729153700 1 -- 172.21.15.3:0/721818749 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f672000d7f0 con 0x7f67240b01a0 2022-07-21T01:34:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 -- 172.21.15.3:0/721818749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 msgr2=0x7f67240a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 --2- 172.21.15.3:0/721818749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 0x7f67240a8540 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f6720002f70 tx=0x7f6720006df0).stop 2022-07-21T01:34:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 -- 172.21.15.3:0/721818749 shutdown_connections 2022-07-21T01:34:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 --2- 172.21.15.3:0/721818749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 0x7f67240a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 --2- 172.21.15.3:0/721818749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 0x7f67240a8540 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 -- 172.21.15.3:0/721818749 >> 172.21.15.3:0/721818749 conn(0x7f6724060310 msgr2=0x7f67240b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 -- 172.21.15.3:0/721818749 shutdown_connections 2022-07-21T01:34:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 -- 172.21.15.3:0/721818749 wait complete. 2022-07-21T01:34:00.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 Processor -- start 2022-07-21T01:34:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 -- start start 2022-07-21T01:34:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 0x7f67240b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 0x7f67240b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b2710 0x7f67241411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67240b3c10 con 0x7f67240b2710 2022-07-21T01:34:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67240ac830 con 0x7f67240b01a0 2022-07-21T01:34:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672b157700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672414b0f0 con 0x7f67240afd60 2022-07-21T01:34:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672a956700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b2710 0x7f67241411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f672a155700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 0x7f67240b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.286+0000 7f6729954700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 0x7f67240b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f6729954700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 0x7f67240b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:49350/0 (socket says 172.21.15.3:49350) 2022-07-21T01:34:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672a155700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 0x7f67240b54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:50816/0 (socket says 172.21.15.3:50816) 2022-07-21T01:34:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f6729954700 1 -- 172.21.15.3:0/1332373074 learned_addr learned my addr 172.21.15.3:0/1332373074 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672a956700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b2710 0x7f67241411f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43904/0 (socket says 172.21.15.3:43904) 2022-07-21T01:34:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672a956700 1 -- 172.21.15.3:0/1332373074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 msgr2=0x7f67240b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672a956700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 0x7f67240b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672a956700 1 -- 172.21.15.3:0/1332373074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 msgr2=0x7f67240b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672a956700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 0x7f67240b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672a956700 1 -- 172.21.15.3:0/1332373074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672000b180 con 0x7f67240b2710 2022-07-21T01:34:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672a956700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b2710 0x7f67241411f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f67240a9ca0 tx=0x7f671c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f67137fe700 1 -- 172.21.15.3:0/1332373074 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f671c012600 con 0x7f67240b2710 2022-07-21T01:34:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f67137fe700 1 -- 172.21.15.3:0/1332373074 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671c012760 con 0x7f67240b2710 2022-07-21T01:34:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67240b47c0 con 0x7f67240b2710 2022-07-21T01:34:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f67137fe700 1 -- 172.21.15.3:0/1332373074 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f671c012ac0 con 0x7f67240b2710 2022-07-21T01:34:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67240b4d00 con 0x7f67240b2710 2022-07-21T01:34:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.290+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67240574e0 con 0x7f67240b2710 2022-07-21T01:34:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.294+0000 7f67137fe700 1 -- 172.21.15.3:0/1332373074 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f671c022070 con 0x7f67240b2710 2022-07-21T01:34:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.294+0000 7f67137fe700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053b30 0x7f6704055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.294+0000 7f67137fe700 1 -- 172.21.15.3:0/1332373074 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f671c06c2d0 con 0x7f67240b2710 2022-07-21T01:34:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.294+0000 7f672a155700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053b30 0x7f6704055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.294+0000 7f672a155700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053b30 0x7f6704055ff0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f671400d000 tx=0x7f6714009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.302+0000 7f67137fe700 1 -- 172.21.15.3:0/1332373074 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f671c043370 con 0x7f67240b2710 2022-07-21T01:34:00.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.422+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67240b3d60 con 0x7f67240b2710 2022-07-21T01:34:00.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f67137fe700 1 -- 172.21.15.3:0/1332373074 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+13414 (secure 0 0 0) 0x7f671c016020 con 0x7f67240b2710 2022-07-21T01:34:00.430 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:00.430 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":32,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:33:57.868136+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:34:00.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053b30 msgr2=0x7f6704055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053b30 0x7f6704055ff0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f671400d000 tx=0x7f6714009040).stop 2022-07-21T01:34:00.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b2710 msgr2=0x7f67241411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b2710 0x7f67241411f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f67240a9ca0 tx=0x7f671c0091d0).stop 2022-07-21T01:34:00.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 shutdown_connections 2022-07-21T01:34:00.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b2710 0x7f67241411f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053b30 0x7f6704055ff0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67240b01a0 0x7f67240b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 --2- 172.21.15.3:0/1332373074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240afd60 0x7f67240b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 >> 172.21.15.3:0/1332373074 conn(0x7f6724060310 msgr2=0x7f6724148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:00.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 shutdown_connections 2022-07-21T01:34:00.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.426+0000 7f672b157700 1 -- 172.21.15.3:0/1332373074 wait complete. 2022-07-21T01:34:00.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:34:00.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:34:00.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.550+0000 7f475946a700 1 Processor -- start 2022-07-21T01:34:00.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.550+0000 7f475946a700 1 -- start start 2022-07-21T01:34:00.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4754063a60 0x7f4754062050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47540b1940 0x7f475413e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47540b10d0 con 0x7f475413e5b0 2022-07-21T01:34:00.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4754063e40 con 0x7f4754063a60 2022-07-21T01:34:00.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4754060200 con 0x7f47540b1940 2022-07-21T01:34:00.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f4753fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4754063a60 0x7f4754062050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f4753fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4754063a60 0x7f4754062050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43908/0 (socket says 172.21.15.3:43908) 2022-07-21T01:34:00.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f4753fff700 1 -- 172.21.15.3:0/1766122297 learned_addr learned my addr 172.21.15.3:0/1766122297 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:00.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f474b7fe700 1 --2- 172.21.15.3:0/1766122297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47540b1940 0x7f475413e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f4753fff700 1 -- 172.21.15.3:0/1766122297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47540b1940 msgr2=0x7f475413e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:00.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f4753fff700 1 --2- 172.21.15.3:0/1766122297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47540b1940 0x7f475413e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f4753fff700 1 -- 172.21.15.3:0/1766122297 >> v1:172.21.15.143:6789/0 conn(0x7f475413e5b0 legacy=0x7f47541414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f4753fff700 1 -- 172.21.15.3:0/1766122297 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47540b1300 con 0x7f4754063a60 2022-07-21T01:34:00.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f4753fff700 1 --2- 172.21.15.3:0/1766122297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 0x7f4754062050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f4744002f70 tx=0x7f4744006eb0).ready entity=mon.0 client_cookie=58a001367ee09ec2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f47537fe700 1 -- 172.21.15.3:0/1766122297 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f474400d5c0 con 0x7f4754063a60 2022-07-21T01:34:00.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f47537fe700 1 -- 172.21.15.3:0/1766122297 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4744000e90 con 0x7f4754063a60 2022-07-21T01:34:00.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f47537fe700 1 -- 172.21.15.3:0/1766122297 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f474400d950 con 0x7f4754063a60 2022-07-21T01:34:00.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 -- 172.21.15.3:0/1766122297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 msgr2=0x7f4754062050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 --2- 172.21.15.3:0/1766122297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 0x7f4754062050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f4744002f70 tx=0x7f4744006eb0).stop 2022-07-21T01:34:00.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 -- 172.21.15.3:0/1766122297 shutdown_connections 2022-07-21T01:34:00.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 --2- 172.21.15.3:0/1766122297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47540b1940 0x7f475413e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 --2- 172.21.15.3:0/1766122297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 0x7f4754062050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 -- 172.21.15.3:0/1766122297 >> 172.21.15.3:0/1766122297 conn(0x7f475405c620 msgr2=0x7f475405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:00.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 -- 172.21.15.3:0/1766122297 shutdown_connections 2022-07-21T01:34:00.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.554+0000 7f475946a700 1 -- 172.21.15.3:0/1766122297 wait complete. 2022-07-21T01:34:00.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 Processor -- start 2022-07-21T01:34:00.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 -- start start 2022-07-21T01:34:00.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 0x7f475413bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47540b1940 0x7f4754133910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f475413e5b0 0x7f4754133e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540aeba0 con 0x7f4754063a60 2022-07-21T01:34:00.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b5d40 con 0x7f47540b1940 2022-07-21T01:34:00.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475413b950 con 0x7f475413e5b0 2022-07-21T01:34:00.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4753fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 0x7f475413bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4753fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 0x7f475413bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43914/0 (socket says 172.21.15.3:43914) 2022-07-21T01:34:00.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4753fff700 1 -- 172.21.15.3:0/1464675450 learned_addr learned my addr 172.21.15.3:0/1464675450 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:00.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f474b7fe700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47540b1940 0x7f4754133910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4758c69700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f475413e5b0 0x7f4754133e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4753fff700 1 -- 172.21.15.3:0/1464675450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f475413e5b0 msgr2=0x7f4754133e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4753fff700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f475413e5b0 0x7f4754133e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4753fff700 1 -- 172.21.15.3:0/1464675450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47540b1940 msgr2=0x7f4754133910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4753fff700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47540b1940 0x7f4754133910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4753fff700 1 -- 172.21.15.3:0/1464675450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474400b1e0 con 0x7f4754063a60 2022-07-21T01:34:00.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4753fff700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 0x7f475413bf20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f4744002f70 tx=0x7f4744015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4751ffb700 1 -- 172.21.15.3:0/1464675450 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47440117e0 con 0x7f4754063a60 2022-07-21T01:34:00.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4751ffb700 1 -- 172.21.15.3:0/1464675450 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4744011940 con 0x7f4754063a60 2022-07-21T01:34:00.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47540aedd0 con 0x7f4754063a60 2022-07-21T01:34:00.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f4751ffb700 1 -- 172.21.15.3:0/1464675450 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4744011ca0 con 0x7f4754063a60 2022-07-21T01:34:00.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.558+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47540b5aa0 con 0x7f4754063a60 2022-07-21T01:34:00.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.562+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47540b15a0 con 0x7f4754063a60 2022-07-21T01:34:00.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.562+0000 7f4751ffb700 1 -- 172.21.15.3:0/1464675450 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4744025030 con 0x7f4754063a60 2022-07-21T01:34:00.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.562+0000 7f4751ffb700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4738053a70 0x7f4738055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.562+0000 7f474b7fe700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4738053a70 0x7f4738055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.562+0000 7f4751ffb700 1 -- 172.21.15.3:0/1464675450 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f474406cfa0 con 0x7f4754063a60 2022-07-21T01:34:00.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.562+0000 7f474b7fe700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4738053a70 0x7f4738055f30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f473c00c000 tx=0x7f473c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.566+0000 7f4751ffb700 1 -- 172.21.15.3:0/1464675450 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f47440440e0 con 0x7f4754063a60 2022-07-21T01:34:00.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.690+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f47540574e0 con 0x7f4754063a60 2022-07-21T01:34:00.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.690+0000 7f4751ffb700 1 -- 172.21.15.3:0/1464675450 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v32) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4744023020 con 0x7f4754063a60 2022-07-21T01:34:00.696 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:34:00.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4738053a70 msgr2=0x7f4738055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4738053a70 0x7f4738055f30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f473c00c000 tx=0x7f473c008040).stop 2022-07-21T01:34:00.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 msgr2=0x7f475413bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 0x7f475413bf20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f4744002f70 tx=0x7f4744015000).stop 2022-07-21T01:34:00.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 shutdown_connections 2022-07-21T01:34:00.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f475413e5b0 0x7f4754133e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4738053a70 0x7f4738055f30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47540b1940 0x7f4754133910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 --2- 172.21.15.3:0/1464675450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4754063a60 0x7f475413bf20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 >> 172.21.15.3:0/1464675450 conn(0x7f475405c620 msgr2=0x7f4754136e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:00.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 shutdown_connections 2022-07-21T01:34:00.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.694+0000 7f475946a700 1 -- 172.21.15.3:0/1464675450 wait complete. 2022-07-21T01:34:00.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:34:00.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f1940dd6700 1 Processor -- start 2022-07-21T01:34:00.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f1940dd6700 1 -- start start 2022-07-21T01:34:00.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f1940dd6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f193c0b28b0 0x7f193c13e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f1940dd6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f193c0b1180 0x7f193c13e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f1940dd6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193c0aac20 con 0x7f193c141210 2022-07-21T01:34:00.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f1940dd6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c141db0 con 0x7f193c0b1180 2022-07-21T01:34:00.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f1940dd6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c140e20 con 0x7f193c0b28b0 2022-07-21T01:34:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193affd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f193c0b1180 0x7f193c13e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193affd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f193c0b1180 0x7f193c13e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43924/0 (socket says 172.21.15.3:43924) 2022-07-21T01:34:00.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193affd700 1 -- 172.21.15.3:0/2515357914 learned_addr learned my addr 172.21.15.3:0/2515357914 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:00.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193b7fe700 1 --2- 172.21.15.3:0/2515357914 >> v2:172.21.15.143:3300/0 conn(0x7f193c0b28b0 0x7f193c13e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193affd700 1 -- 172.21.15.3:0/2515357914 >> v2:172.21.15.143:3300/0 conn(0x7f193c0b28b0 msgr2=0x7f193c13e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:00.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193affd700 1 --2- 172.21.15.3:0/2515357914 >> v2:172.21.15.143:3300/0 conn(0x7f193c0b28b0 0x7f193c13e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193affd700 1 -- 172.21.15.3:0/2515357914 >> v1:172.21.15.143:6789/0 conn(0x7f193c141210 legacy=0x7f193c1416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193affd700 1 -- 172.21.15.3:0/2515357914 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c0b2c90 con 0x7f193c0b1180 2022-07-21T01:34:00.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193affd700 1 --2- 172.21.15.3:0/2515357914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 0x7f193c13e6b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f1930002f70 tx=0x7f1930006eb0).ready entity=mon.0 client_cookie=56be6281ff725f27 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193a7fc700 1 -- 172.21.15.3:0/2515357914 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f193000d610 con 0x7f193c0b1180 2022-07-21T01:34:00.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193a7fc700 1 -- 172.21.15.3:0/2515357914 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1930000e90 con 0x7f193c0b1180 2022-07-21T01:34:00.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.969+0000 7f193a7fc700 1 -- 172.21.15.3:0/2515357914 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f193000d9a0 con 0x7f193c0b1180 2022-07-21T01:34:00.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 -- 172.21.15.3:0/2515357914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 msgr2=0x7f193c13e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 --2- 172.21.15.3:0/2515357914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 0x7f193c13e6b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f1930002f70 tx=0x7f1930006eb0).stop 2022-07-21T01:34:00.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 -- 172.21.15.3:0/2515357914 shutdown_connections 2022-07-21T01:34:00.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 --2- 172.21.15.3:0/2515357914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 0x7f193c13e6b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 --2- 172.21.15.3:0/2515357914 >> v2:172.21.15.143:3300/0 conn(0x7f193c0b28b0 0x7f193c13e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 -- 172.21.15.3:0/2515357914 >> 172.21.15.3:0/2515357914 conn(0x7f193c060310 msgr2=0x7f193c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:00.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 -- 172.21.15.3:0/2515357914 shutdown_connections 2022-07-21T01:34:00.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 -- 172.21.15.3:0/2515357914 wait complete. 2022-07-21T01:34:00.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 Processor -- start 2022-07-21T01:34:00.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.973+0000 7f1940dd6700 1 -- start start 2022-07-21T01:34:00.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1940dd6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 0x7f193c150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1940dd6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c0b28b0 0x7f193c0642f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1940dd6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c141210 0x7f193c14d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1940dd6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1511a0 con 0x7f193c0b1180 2022-07-21T01:34:00.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1940dd6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c142cc0 con 0x7f193c141210 2022-07-21T01:34:00.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1940dd6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c142f60 con 0x7f193c0b28b0 2022-07-21T01:34:00.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 0x7f193c150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 0x7f193c150cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43928/0 (socket says 172.21.15.3:43928) 2022-07-21T01:34:00.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c141210 0x7f193c14d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193b7fe700 1 -- 172.21.15.3:0/3194904756 learned_addr learned my addr 172.21.15.3:0/3194904756 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:00.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193affd700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c0b28b0 0x7f193c0642f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193bfff700 1 -- 172.21.15.3:0/3194904756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c0b28b0 msgr2=0x7f193c0642f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193bfff700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c0b28b0 0x7f193c0642f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193bfff700 1 -- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 msgr2=0x7f193c150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:00.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193bfff700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 0x7f193c150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:00.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193bfff700 1 -- 172.21.15.3:0/3194904756 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193000b1e0 con 0x7f193c141210 2022-07-21T01:34:00.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193b7fe700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 0x7f193c150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:34:00.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f193bfff700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c141210 0x7f193c14d4f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f193c0b15f0 tx=0x7f1934007f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1938ff9700 1 -- 172.21.15.3:0/3194904756 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f193400ce10 con 0x7f193c141210 2022-07-21T01:34:00.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1938ff9700 1 -- 172.21.15.3:0/3194904756 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1934006aa0 con 0x7f193c141210 2022-07-21T01:34:00.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c143200 con 0x7f193c141210 2022-07-21T01:34:00.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1938ff9700 1 -- 172.21.15.3:0/3194904756 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1934006e00 con 0x7f193c141210 2022-07-21T01:34:00.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c0647e0 con 0x7f193c141210 2022-07-21T01:34:00.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c0574e0 con 0x7f193c141210 2022-07-21T01:34:00.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.977+0000 7f1938ff9700 1 -- 172.21.15.3:0/3194904756 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1934010030 con 0x7f193c141210 2022-07-21T01:34:00.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.981+0000 7f1938ff9700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f191c053980 0x7f191c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:00.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.981+0000 7f1938ff9700 1 -- 172.21.15.3:0/3194904756 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f193406b780 con 0x7f193c141210 2022-07-21T01:34:00.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.981+0000 7f193b7fe700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f191c053980 0x7f191c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:00.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.981+0000 7f193b7fe700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f191c053980 0x7f191c055e40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f192c008000 tx=0x7f192c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:00.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:00.985+0000 7f1938ff9700 1 -- 172.21.15.3:0/3194904756 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f19340427e0 con 0x7f193c141210 2022-07-21T01:34:01.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.117+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f193c0aac20 con 0x7f193c141210 2022-07-21T01:34:01.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.117+0000 7f1938ff9700 1 -- 172.21.15.3:0/3194904756 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v33) v1 ==== 92+0+10 (secure 0 0 0) 0x7f193c0aac20 con 0x7f193c141210 2022-07-21T01:34:01.124 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:34:01.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f191c053980 msgr2=0x7f191c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f191c053980 0x7f191c055e40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f192c008000 tx=0x7f192c008040).stop 2022-07-21T01:34:01.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c141210 msgr2=0x7f193c14d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c141210 0x7f193c14d4f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f193c0b15f0 tx=0x7f1934007f10).stop 2022-07-21T01:34:01.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 shutdown_connections 2022-07-21T01:34:01.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f191c053980 0x7f191c055e40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c141210 0x7f193c14d4f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c0b28b0 0x7f193c0642f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 --2- 172.21.15.3:0/3194904756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f193c0b1180 0x7f193c150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 >> 172.21.15.3:0/3194904756 conn(0x7f193c060310 msgr2=0x7f193c0ab320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:01.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 shutdown_connections 2022-07-21T01:34:01.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.121+0000 7f1940dd6700 1 -- 172.21.15.3:0/3194904756 wait complete. 2022-07-21T01:34:01.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:34:01.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:34:01.225 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:34:01.234 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:34:01.243 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:34:01.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272fb91700 1 Processor -- start 2022-07-21T01:34:01.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272fb91700 1 -- start start 2022-07-21T01:34:01.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272fb91700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 0x7f27280b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272fb91700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f27280b6af0 0x7f272812bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272fb91700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2728144240 con 0x7f27280b6710 2022-07-21T01:34:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272fb91700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2728143d00 con 0x7f27280b6af0 2022-07-21T01:34:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272fb91700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2728143fa0 con 0x7f27280b4550 2022-07-21T01:34:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272e12e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f27280b6af0 0x7f272812bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272d92d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f27280b6710 0x7f2728061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52216/0 (socket says 172.21.15.3:52216) 2022-07-21T01:34:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.257+0000 7f272d92d700 1 -- 172.21.15.3:0/1365102843 learned_addr learned my addr 172.21.15.3:0/1365102843 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272d12c700 1 --2- 172.21.15.3:0/1365102843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 0x7f27280b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272e12e700 1 -- 172.21.15.3:0/1365102843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 msgr2=0x7f27280b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:01.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272e12e700 1 --2- 172.21.15.3:0/1365102843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 0x7f27280b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272e12e700 1 -- 172.21.15.3:0/1365102843 >> v1:172.21.15.3:6789/0 conn(0x7f27280b6710 legacy=0x7f2728061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272e12e700 1 -- 172.21.15.3:0/1365102843 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2728061780 con 0x7f27280b6af0 2022-07-21T01:34:01.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272e12e700 1 --2- 172.21.15.3:0/1365102843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 0x7f272812bb90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f2720002f70 tx=0x7f2720006df0).ready entity=mon.0 client_cookie=5e148dcd36828d01 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:01.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272c92b700 1 -- 172.21.15.3:0/1365102843 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f272000d570 con 0x7f27280b6af0 2022-07-21T01:34:01.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272c92b700 1 -- 172.21.15.3:0/1365102843 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2720000e90 con 0x7f27280b6af0 2022-07-21T01:34:01.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272c92b700 1 -- 172.21.15.3:0/1365102843 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f272000d8c0 con 0x7f27280b6af0 2022-07-21T01:34:01.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272fb91700 1 -- 172.21.15.3:0/1365102843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 msgr2=0x7f272812bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272fb91700 1 --2- 172.21.15.3:0/1365102843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 0x7f272812bb90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f2720002f70 tx=0x7f2720006df0).stop 2022-07-21T01:34:01.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272fb91700 1 -- 172.21.15.3:0/1365102843 shutdown_connections 2022-07-21T01:34:01.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272fb91700 1 --2- 172.21.15.3:0/1365102843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 0x7f272812bb90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272fb91700 1 --2- 172.21.15.3:0/1365102843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 0x7f27280b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272fb91700 1 -- 172.21.15.3:0/1365102843 >> 172.21.15.3:0/1365102843 conn(0x7f272805c620 msgr2=0x7f272805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:01.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272fb91700 1 -- 172.21.15.3:0/1365102843 shutdown_connections 2022-07-21T01:34:01.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.261+0000 7f272fb91700 1 -- 172.21.15.3:0/1365102843 wait complete. 2022-07-21T01:34:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 Processor -- start 2022-07-21T01:34:01.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 -- start start 2022-07-21T01:34:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 0x7f2728155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27280b6710 0x7f272814a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 0x7f2728147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2728147a70 con 0x7f27280b6af0 2022-07-21T01:34:01.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280b3910 con 0x7f27280b6710 2022-07-21T01:34:01.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27280b3bb0 con 0x7f27280b4550 2022-07-21T01:34:01.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272e12e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 0x7f2728147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272d92d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 0x7f2728155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272e12e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 0x7f2728147e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43946/0 (socket says 172.21.15.3:43946) 2022-07-21T01:34:01.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272e12e700 1 -- 172.21.15.3:0/541176070 learned_addr learned my addr 172.21.15.3:0/541176070 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:01.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272d12c700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27280b6710 0x7f272814a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272e12e700 1 -- 172.21.15.3:0/541176070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 msgr2=0x7f2728155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272e12e700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 0x7f2728155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272e12e700 1 -- 172.21.15.3:0/541176070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27280b6710 msgr2=0x7f272814a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272e12e700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27280b6710 0x7f272814a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272e12e700 1 -- 172.21.15.3:0/541176070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272000b180 con 0x7f27280b6af0 2022-07-21T01:34:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272d92d700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 0x7f2728155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:34:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272e12e700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 0x7f2728147e60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f2720002f70 tx=0x7f2720015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:01.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f2726ffd700 1 -- 172.21.15.3:0/541176070 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2720011760 con 0x7f27280b6af0 2022-07-21T01:34:01.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f2726ffd700 1 -- 172.21.15.3:0/541176070 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27200118c0 con 0x7f27280b6af0 2022-07-21T01:34:01.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27280b3650 con 0x7f27280b6af0 2022-07-21T01:34:01.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f2726ffd700 1 -- 172.21.15.3:0/541176070 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2720011c20 con 0x7f27280b6af0 2022-07-21T01:34:01.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.265+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2728146b20 con 0x7f27280b6af0 2022-07-21T01:34:01.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.269+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27280b3d70 con 0x7f27280b6af0 2022-07-21T01:34:01.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.269+0000 7f2726ffd700 1 -- 172.21.15.3:0/541176070 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2720025030 con 0x7f27280b6af0 2022-07-21T01:34:01.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.269+0000 7f2726ffd700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2708053a70 0x7f2708055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.269+0000 7f2726ffd700 1 -- 172.21.15.3:0/541176070 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f272006cfc0 con 0x7f27280b6af0 2022-07-21T01:34:01.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.269+0000 7f272d92d700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2708053a70 0x7f2708055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.269+0000 7f272d92d700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2708053a70 0x7f2708055f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f271800d000 tx=0x7f2718009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.273+0000 7f2726ffd700 1 -- 172.21.15.3:0/541176070 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2720044020 con 0x7f27280b6af0 2022-07-21T01:34:01.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.401+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2728147880 con 0x7f27280b6af0 2022-07-21T01:34:01.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.401+0000 7f2726ffd700 1 -- 172.21.15.3:0/541176070 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v33) v1 ==== 84+0+143 (secure 0 0 0) 0x7f2720023020 con 0x7f27280b6af0 2022-07-21T01:34:01.410 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:01.411 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:34:01.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:34:01.412 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:34:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:34:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:34:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:34:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:34:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:34:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:34:01.414 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:34:01.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2708053a70 msgr2=0x7f2708055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2708053a70 0x7f2708055f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f271800d000 tx=0x7f2718009040).stop 2022-07-21T01:34:01.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 msgr2=0x7f2728147e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 0x7f2728147e60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f2720002f70 tx=0x7f2720015000).stop 2022-07-21T01:34:01.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 shutdown_connections 2022-07-21T01:34:01.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27280b6af0 0x7f2728147e60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2708053a70 0x7f2708055f30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27280b6710 0x7f272814a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 --2- 172.21.15.3:0/541176070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27280b4550 0x7f2728155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 >> 172.21.15.3:0/541176070 conn(0x7f272805c620 msgr2=0x7f2728061e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:01.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 shutdown_connections 2022-07-21T01:34:01.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.405+0000 7f272fb91700 1 -- 172.21.15.3:0/541176070 wait complete. 2022-07-21T01:34:01.423 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:34:01.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:34:01.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.529+0000 7f98153e3700 1 Processor -- start 2022-07-21T01:34:01.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.529+0000 7f98153e3700 1 -- start start 2022-07-21T01:34:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9810143ee0 0x7f9810140b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9810063e40 con 0x7f9810061ab0 2022-07-21T01:34:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9810062180 con 0x7f9810063a60 2022-07-21T01:34:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981013ff40 con 0x7f9810143ee0 2022-07-21T01:34:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f9814be2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9810143ee0 0x7f9810140b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f980f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9810061ab0 0x7f9810128e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52232/0 (socket says 172.21.15.3:52232) 2022-07-21T01:34:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f980f7fe700 1 -- 172.21.15.3:0/2340310335 learned_addr learned my addr 172.21.15.3:0/2340310335 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f9814be2700 1 -- 172.21.15.3:0/2340310335 >> v1:172.21.15.143:6789/0 conn(0x7f9810063a60 legacy=0x7f98100b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f9814be2700 1 -- 172.21.15.3:0/2340310335 >> v1:172.21.15.3:6789/0 conn(0x7f9810061ab0 legacy=0x7f9810128e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f9814be2700 1 -- 172.21.15.3:0/2340310335 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810140130 con 0x7f9810143ee0 2022-07-21T01:34:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f9814be2700 1 --2- 172.21.15.3:0/2340310335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810143ee0 0x7f9810140b10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f98080032f0 tx=0x7f98080091d0).ready entity=mon.0 client_cookie=b3bba16177d4503c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f980effd700 1 -- 172.21.15.3:0/2340310335 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9808009ea0 con 0x7f9810143ee0 2022-07-21T01:34:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f980effd700 1 -- 172.21.15.3:0/2340310335 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980800b8a0 con 0x7f9810143ee0 2022-07-21T01:34:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f980effd700 1 -- 172.21.15.3:0/2340310335 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f980800ba10 con 0x7f9810143ee0 2022-07-21T01:34:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 -- 172.21.15.3:0/2340310335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810143ee0 msgr2=0x7f9810140b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 --2- 172.21.15.3:0/2340310335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810143ee0 0x7f9810140b10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f98080032f0 tx=0x7f98080091d0).stop 2022-07-21T01:34:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 -- 172.21.15.3:0/2340310335 shutdown_connections 2022-07-21T01:34:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 --2- 172.21.15.3:0/2340310335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810143ee0 0x7f9810140b10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 -- 172.21.15.3:0/2340310335 >> 172.21.15.3:0/2340310335 conn(0x7f981005c620 msgr2=0x7f981005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 -- 172.21.15.3:0/2340310335 shutdown_connections 2022-07-21T01:34:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.533+0000 7f98153e3700 1 -- 172.21.15.3:0/2340310335 wait complete. 2022-07-21T01:34:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 Processor -- start 2022-07-21T01:34:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 -- start start 2022-07-21T01:34:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9810061ab0 0x7f9810141750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810063a60 0x7f9810153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9810143ee0 0x7f98101544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9810064dc0 con 0x7f9810063a60 2022-07-21T01:34:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100b3120 con 0x7f9810061ab0 2022-07-21T01:34:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98100649f0 con 0x7f9810143ee0 2022-07-21T01:34:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810063a60 0x7f9810153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810063a60 0x7f9810153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43958/0 (socket says 172.21.15.3:43958) 2022-07-21T01:34:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980f7fe700 1 -- 172.21.15.3:0/3020128585 learned_addr learned my addr 172.21.15.3:0/3020128585 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980ffff700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9810061ab0 0x7f9810141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f9814be2700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9810143ee0 0x7f98101544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980f7fe700 1 -- 172.21.15.3:0/3020128585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9810143ee0 msgr2=0x7f98101544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980f7fe700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9810143ee0 0x7f98101544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980f7fe700 1 -- 172.21.15.3:0/3020128585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9810061ab0 msgr2=0x7f9810141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980f7fe700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9810061ab0 0x7f9810141750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980f7fe700 1 -- 172.21.15.3:0/3020128585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9808009470 con 0x7f9810063a60 2022-07-21T01:34:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980ffff700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9810061ab0 0x7f9810141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:34:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980f7fe700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810063a60 0x7f9810153ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f981006ac50 tx=0x7f97f8007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980d7fa700 1 -- 172.21.15.3:0/3020128585 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97f800ce30 con 0x7f9810063a60 2022-07-21T01:34:01.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980d7fa700 1 -- 172.21.15.3:0/3020128585 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f80072e0 con 0x7f9810063a60 2022-07-21T01:34:01.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98100b3380 con 0x7f9810063a60 2022-07-21T01:34:01.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f980d7fa700 1 -- 172.21.15.3:0/3020128585 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97f80057c0 con 0x7f9810063a60 2022-07-21T01:34:01.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.537+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9810145540 con 0x7f9810063a60 2022-07-21T01:34:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.541+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98100574e0 con 0x7f9810063a60 2022-07-21T01:34:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.541+0000 7f980d7fa700 1 -- 172.21.15.3:0/3020128585 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f97f8010070 con 0x7f9810063a60 2022-07-21T01:34:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.541+0000 7f980d7fa700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97f40539e0 0x7f97f4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.541+0000 7f980ffff700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97f40539e0 0x7f97f4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.541+0000 7f980d7fa700 1 -- 172.21.15.3:0/3020128585 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f97f806b3d0 con 0x7f9810063a60 2022-07-21T01:34:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.541+0000 7f980ffff700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97f40539e0 0x7f97f4055ea0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f980000b000 tx=0x7f9800009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.549+0000 7f980d7fa700 1 -- 172.21.15.3:0/3020128585 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f97f80423c0 con 0x7f9810063a60 2022-07-21T01:34:01.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:01.673+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f9810067530 con 0x7f9810063a60 2022-07-21T01:34:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.409+0000 7f980d7fa700 1 -- 172.21.15.3:0/3020128585 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v34) v1 ==== 133+0+0 (secure 0 0 0) 0x7f97f80459e0 con 0x7f9810063a60 2022-07-21T01:34:02.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.413+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97f40539e0 msgr2=0x7f97f4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97f40539e0 0x7f97f4055ea0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f980000b000 tx=0x7f9800009040).stop 2022-07-21T01:34:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810063a60 msgr2=0x7f9810153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810063a60 0x7f9810153ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f981006ac50 tx=0x7f97f8007c30).stop 2022-07-21T01:34:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 shutdown_connections 2022-07-21T01:34:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9810143ee0 0x7f98101544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9810063a60 0x7f9810153ff0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97f40539e0 0x7f97f4055ea0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 --2- 172.21.15.3:0/3020128585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9810061ab0 0x7f9810141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 >> 172.21.15.3:0/3020128585 conn(0x7f981005c620 msgr2=0x7f981013ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 shutdown_connections 2022-07-21T01:34:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.417+0000 7f98153e3700 1 -- 172.21.15.3:0/3020128585 wait complete. 2022-07-21T01:34:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:34:02.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:34:02.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.545+0000 7fd28afad700 1 Processor -- start 2022-07-21T01:34:02.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.545+0000 7fd28afad700 1 -- start start 2022-07-21T01:34:02.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840af450 0x7fd2840af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:02.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2840aff30 con 0x7fd2840ad4b0 2022-07-21T01:34:02.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28405c7a0 con 0x7fd2840ad9e0 2022-07-21T01:34:02.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd284140990 con 0x7fd2840af450 2022-07-21T01:34:02.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd2897aa700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd2840ad4b0 0x7fd2841432e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52248/0 (socket says 172.21.15.3:52248) 2022-07-21T01:34:02.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd2897aa700 1 -- 172.21.15.3:0/1899867670 learned_addr learned my addr 172.21.15.3:0/1899867670 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:02.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd289fab700 1 --2- 172.21.15.3:0/1899867670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840af450 0x7fd2840af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:02.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd288fa9700 1 -- 172.21.15.3:0/1899867670 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426009280 0 0) 0x7fd2840aff30 con 0x7fd2840ad4b0 2022-07-21T01:34:02.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd288fa9700 1 -- 172.21.15.3:0/1899867670 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd26c003260 con 0x7fd2840ad4b0 2022-07-21T01:34:02.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd288fa9700 1 -- 172.21.15.3:0/1899867670 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648795507 0 0) 0x7fd28405c7a0 con 0x7fd2840ad9e0 2022-07-21T01:34:02.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd288fa9700 1 -- 172.21.15.3:0/1899867670 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2840aff30 con 0x7fd2840ad9e0 2022-07-21T01:34:02.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd289fab700 1 -- 172.21.15.3:0/1899867670 >> v1:172.21.15.143:6789/0 conn(0x7fd2840ad9e0 legacy=0x7fd2840a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:02.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd289fab700 1 -- 172.21.15.3:0/1899867670 >> v1:172.21.15.3:6789/0 conn(0x7fd2840ad4b0 legacy=0x7fd2841432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:02.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd289fab700 1 -- 172.21.15.3:0/1899867670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2841439f0 con 0x7fd2840af450 2022-07-21T01:34:02.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd289fab700 1 --2- 172.21.15.3:0/1899867670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840af450 0x7fd2840af830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fd280002f70 tx=0x7fd280006df0).ready entity=mon.2 client_cookie=d3f17d5c56f7fc21 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:02.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd288fa9700 1 -- 172.21.15.3:0/1899867670 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd28000d5c0 con 0x7fd2840af450 2022-07-21T01:34:02.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd288fa9700 1 -- 172.21.15.3:0/1899867670 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd280000e90 con 0x7fd2840af450 2022-07-21T01:34:02.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd288fa9700 1 -- 172.21.15.3:0/1899867670 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd28000d910 con 0x7fd2840af450 2022-07-21T01:34:02.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 -- 172.21.15.3:0/1899867670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840af450 msgr2=0x7fd2840af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:02.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 --2- 172.21.15.3:0/1899867670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840af450 0x7fd2840af830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fd280002f70 tx=0x7fd280006df0).stop 2022-07-21T01:34:02.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 -- 172.21.15.3:0/1899867670 shutdown_connections 2022-07-21T01:34:02.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 --2- 172.21.15.3:0/1899867670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840af450 0x7fd2840af830 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:02.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 -- 172.21.15.3:0/1899867670 >> 172.21.15.3:0/1899867670 conn(0x7fd2840612d0 msgr2=0x7fd284062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:02.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 -- 172.21.15.3:0/1899867670 shutdown_connections 2022-07-21T01:34:02.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.549+0000 7fd28afad700 1 -- 172.21.15.3:0/1899867670 wait complete. 2022-07-21T01:34:02.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28afad700 1 Processor -- start 2022-07-21T01:34:02.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28afad700 1 -- start start 2022-07-21T01:34:02.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28afad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2840ad4b0 0x7fd2840b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:02.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28afad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840ad9e0 0x7fd2840b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:02.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28afad700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2840af450 0x7fd2840abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:02.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28afad700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840aa570 con 0x7fd2840af450 2022-07-21T01:34:02.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28afad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840b4d00 con 0x7fd2840ad4b0 2022-07-21T01:34:02.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28afad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840b4fa0 con 0x7fd2840ad9e0 2022-07-21T01:34:02.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28a7ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2840af450 0x7fd2840abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:02.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28a7ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2840af450 0x7fd2840abbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43976/0 (socket says 172.21.15.3:43976) 2022-07-21T01:34:02.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28a7ac700 1 -- 172.21.15.3:0/3546485710 learned_addr learned my addr 172.21.15.3:0/3546485710 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:02.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd289fab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2840ad4b0 0x7fd2840b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:02.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd2897aa700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840ad9e0 0x7fd2840b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:02.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28a7ac700 1 -- 172.21.15.3:0/3546485710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840ad9e0 msgr2=0x7fd2840b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:02.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28a7ac700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840ad9e0 0x7fd2840b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:02.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28a7ac700 1 -- 172.21.15.3:0/3546485710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2840ad4b0 msgr2=0x7fd2840b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:02.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28a7ac700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2840ad4b0 0x7fd2840b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:02.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.553+0000 7fd28a7ac700 1 -- 172.21.15.3:0/3546485710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28000b180 con 0x7fd2840af450 2022-07-21T01:34:02.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd28a7ac700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2840af450 0x7fd2840abbf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fd27c001490 tx=0x7fd27c007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:02.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd272ffd700 1 -- 172.21.15.3:0/3546485710 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd27c002240 con 0x7fd2840af450 2022-07-21T01:34:02.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd272ffd700 1 -- 172.21.15.3:0/3546485710 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c0023a0 con 0x7fd2840af450 2022-07-21T01:34:02.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2840b5240 con 0x7fd2840af450 2022-07-21T01:34:02.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd272ffd700 1 -- 172.21.15.3:0/3546485710 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd27c001650 con 0x7fd2840af450 2022-07-21T01:34:02.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2840ac550 con 0x7fd2840af450 2022-07-21T01:34:02.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2840b53d0 con 0x7fd2840af450 2022-07-21T01:34:02.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd272ffd700 1 -- 172.21.15.3:0/3546485710 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd27c010030 con 0x7fd2840af450 2022-07-21T01:34:02.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd272ffd700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd264053b00 0x7fd264055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:02.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd272ffd700 1 -- 172.21.15.3:0/3546485710 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5954+0+0 (secure 0 0 0) 0x7fd27c06bdc0 con 0x7fd2840af450 2022-07-21T01:34:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd289fab700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd264053b00 0x7fd264055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.557+0000 7fd289fab700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd264053b00 0x7fd264055fc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd28001a000 tx=0x7fd280014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:02.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.565+0000 7fd272ffd700 1 -- 172.21.15.3:0/3546485710 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd27c042cb0 con 0x7fd2840af450 2022-07-21T01:34:02.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:02.697+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fd2840574e0 con 0x7fd2840af450 2022-07-21T01:34:03.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.413+0000 7fd272ffd700 1 -- 172.21.15.3:0/3546485710 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v35) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd27c0462d0 con 0x7fd2840af450 2022-07-21T01:34:03.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.417+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd264053b00 msgr2=0x7fd264055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.417+0000 7fd28afad700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd264053b00 0x7fd264055fc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd28001a000 tx=0x7fd280014000).stop 2022-07-21T01:34:03.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.417+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2840af450 msgr2=0x7fd2840abbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.417+0000 7fd28afad700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2840af450 0x7fd2840abbf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fd27c001490 tx=0x7fd27c007dd0).stop 2022-07-21T01:34:03.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.421+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 shutdown_connections 2022-07-21T01:34:03.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.421+0000 7fd28afad700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2840af450 0x7fd2840abbf0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.421+0000 7fd28afad700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2840ad9e0 0x7fd2840b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.421+0000 7fd28afad700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd264053b00 0x7fd264055fc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.421+0000 7fd28afad700 1 --2- 172.21.15.3:0/3546485710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2840ad4b0 0x7fd2840b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.421+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 >> 172.21.15.3:0/3546485710 conn(0x7fd2840612d0 msgr2=0x7fd284144650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:03.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.421+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 shutdown_connections 2022-07-21T01:34:03.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.421+0000 7fd28afad700 1 -- 172.21.15.3:0/3546485710 wait complete. 2022-07-21T01:34:03.426 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:34:03.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:34:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f720dc0e700 1 Processor -- start 2022-07-21T01:34:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f720dc0e700 1 -- start start 2022-07-21T01:34:03.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f720dc0e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7208139e50 0x7f72080acd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f720dc0e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72080b2350 0x7f7208143680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f720dc0e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72080ad200 con 0x7f720813a230 2022-07-21T01:34:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f720dc0e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72080b28c0 con 0x7f72080b2350 2022-07-21T01:34:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f720dc0e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7208060ed0 con 0x7f7208139e50 2022-07-21T01:34:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f7206ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72080b2350 0x7f7208143680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f7206ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72080b2350 0x7f7208143680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43982/0 (socket says 172.21.15.3:43982) 2022-07-21T01:34:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f7206ffd700 1 -- 172.21.15.3:0/3324490051 learned_addr learned my addr 172.21.15.3:0/3324490051 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f72077fe700 1 --2- 172.21.15.3:0/3324490051 >> v2:172.21.15.143:3300/0 conn(0x7f7208139e50 0x7f72080acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f7206ffd700 1 -- 172.21.15.3:0/3324490051 >> v2:172.21.15.143:3300/0 conn(0x7f7208139e50 msgr2=0x7f72080acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f7206ffd700 1 --2- 172.21.15.3:0/3324490051 >> v2:172.21.15.143:3300/0 conn(0x7f7208139e50 0x7f72080acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f7206ffd700 1 -- 172.21.15.3:0/3324490051 >> v1:172.21.15.143:6789/0 conn(0x7f720813a230 legacy=0x7f72080b6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f7206ffd700 1 -- 172.21.15.3:0/3324490051 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72080b6e60 con 0x7f72080b2350 2022-07-21T01:34:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f7206ffd700 1 --2- 172.21.15.3:0/3324490051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72080b2350 0x7f7208143680 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f71f0002f70 tx=0x7f71f0006f40).ready entity=mon.0 client_cookie=f8300a7f53461812 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f72067fc700 1 -- 172.21.15.3:0/3324490051 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f000dc70 con 0x7f72080b2350 2022-07-21T01:34:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f72067fc700 1 -- 172.21.15.3:0/3324490051 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f0000e90 con 0x7f72080b2350 2022-07-21T01:34:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.533+0000 7f72067fc700 1 -- 172.21.15.3:0/3324490051 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f000be90 con 0x7f72080b2350 2022-07-21T01:34:03.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 -- 172.21.15.3:0/3324490051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72080b2350 msgr2=0x7f7208143680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 --2- 172.21.15.3:0/3324490051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72080b2350 0x7f7208143680 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f71f0002f70 tx=0x7f71f0006f40).stop 2022-07-21T01:34:03.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 -- 172.21.15.3:0/3324490051 shutdown_connections 2022-07-21T01:34:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 --2- 172.21.15.3:0/3324490051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72080b2350 0x7f7208143680 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 --2- 172.21.15.3:0/3324490051 >> v2:172.21.15.143:3300/0 conn(0x7f7208139e50 0x7f72080acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 -- 172.21.15.3:0/3324490051 >> 172.21.15.3:0/3324490051 conn(0x7f72080b3610 msgr2=0x7f72080b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 -- 172.21.15.3:0/3324490051 shutdown_connections 2022-07-21T01:34:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 -- 172.21.15.3:0/3324490051 wait complete. 2022-07-21T01:34:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 Processor -- start 2022-07-21T01:34:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 -- start start 2022-07-21T01:34:03.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72080b2350 0x7f72081527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208139e50 0x7f72081425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f720813a230 0x7f720813afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720813b7e0 con 0x7f7208139e50 2022-07-21T01:34:03.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72080b1690 con 0x7f72080b2350 2022-07-21T01:34:03.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f720dc0e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7208151e10 con 0x7f720813a230 2022-07-21T01:34:03.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f7206ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208139e50 0x7f72081425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f7206ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208139e50 0x7f72081425e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43988/0 (socket says 172.21.15.3:43988) 2022-07-21T01:34:03.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f7206ffd700 1 -- 172.21.15.3:0/1832279176 learned_addr learned my addr 172.21.15.3:0/1832279176 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:03.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f72077fe700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72080b2350 0x7f72081527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.537+0000 7f7207fff700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f720813a230 0x7f720813afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7206ffd700 1 -- 172.21.15.3:0/1832279176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f720813a230 msgr2=0x7f720813afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7206ffd700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f720813a230 0x7f720813afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7206ffd700 1 -- 172.21.15.3:0/1832279176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72080b2350 msgr2=0x7f72081527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7206ffd700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72080b2350 0x7f72081527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7206ffd700 1 -- 172.21.15.3:0/1832279176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f000b460 con 0x7f7208139e50 2022-07-21T01:34:03.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7207fff700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f720813a230 0x7f720813afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:34:03.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7206ffd700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208139e50 0x7f72081425e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f71f0002f70 tx=0x7f72081400b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:03.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7204ff9700 1 -- 172.21.15.3:0/1832279176 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f0010e40 con 0x7f7208139e50 2022-07-21T01:34:03.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7204ff9700 1 -- 172.21.15.3:0/1832279176 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f0024450 con 0x7f7208139e50 2022-07-21T01:34:03.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72081520b0 con 0x7f7208139e50 2022-07-21T01:34:03.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7204ff9700 1 -- 172.21.15.3:0/1832279176 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f00247b0 con 0x7f7208139e50 2022-07-21T01:34:03.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7208142ad0 con 0x7f7208139e50 2022-07-21T01:34:03.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7208061130 con 0x7f7208139e50 2022-07-21T01:34:03.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7204ff9700 1 -- 172.21.15.3:0/1832279176 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f71f0021070 con 0x7f7208139e50 2022-07-21T01:34:03.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7204ff9700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f71ec053950 0x7f71ec055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f7204ff9700 1 -- 172.21.15.3:0/1832279176 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f71f006f9f0 con 0x7f7208139e50 2022-07-21T01:34:03.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.541+0000 7f72077fe700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f71ec053950 0x7f71ec055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.545+0000 7f72077fe700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f71ec053950 0x7f71ec055e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f71f800c000 tx=0x7f71f8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:03.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.549+0000 7f7204ff9700 1 -- 172.21.15.3:0/1832279176 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f71f0071e60 con 0x7f7208139e50 2022-07-21T01:34:03.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.709+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f72080574e0 con 0x7f7208139e50 2022-07-21T01:34:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.717+0000 7f7204ff9700 1 -- 172.21.15.3:0/1832279176 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 5 and data pool 6 v11) v1 ==== 187+0+0 (secure 0 0 0) 0x7f71f00468f0 con 0x7f7208139e50 2022-07-21T01:34:03.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.725+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f71ec053950 msgr2=0x7f71ec055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.725+0000 7f720dc0e700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f71ec053950 0x7f71ec055e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f71f800c000 tx=0x7f71f8008040).stop 2022-07-21T01:34:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.725+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208139e50 msgr2=0x7f72081425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.725+0000 7f720dc0e700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208139e50 0x7f72081425e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f71f0002f70 tx=0x7f72081400b0).stop 2022-07-21T01:34:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.729+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 shutdown_connections 2022-07-21T01:34:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.729+0000 7f720dc0e700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f720813a230 0x7f720813afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.729+0000 7f720dc0e700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208139e50 0x7f72081425e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.729+0000 7f720dc0e700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f71ec053950 0x7f71ec055e10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.729+0000 7f720dc0e700 1 --2- 172.21.15.3:0/1832279176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72080b2350 0x7f72081527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.729+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 >> 172.21.15.3:0/1832279176 conn(0x7f72080b3610 msgr2=0x7f7208138870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.729+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 shutdown_connections 2022-07-21T01:34:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.729+0000 7f720dc0e700 1 -- 172.21.15.3:0/1832279176 wait complete. 2022-07-21T01:34:03.734 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 5 and data pool 6 2022-07-21T01:34:03.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:34:03.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43c0a9e700 1 Processor -- start 2022-07-21T01:34:03.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43c0a9e700 1 -- start start 2022-07-21T01:34:03.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43c0a9e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f43bc0b38d0 0x7f43bc060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43c0a9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060e40 0x7f43bc0612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43c0a9e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43bc060670 con 0x7f43bc060290 2022-07-21T01:34:03.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43c0a9e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43bc065c30 con 0x7f43bc0b38d0 2022-07-21T01:34:03.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43c0a9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc065ed0 con 0x7f43bc060e40 2022-07-21T01:34:03.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43ba59c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f43bc060290 0x7f43bc066610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52286/0 (socket says 172.21.15.3:52286) 2022-07-21T01:34:03.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43ba59c700 1 -- 172.21.15.3:0/3496557179 learned_addr learned my addr 172.21.15.3:0/3496557179 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:03.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b9d9b700 1 --2- 172.21.15.3:0/3496557179 >> v2:172.21.15.3:3300/0 conn(0x7f43bc0b38d0 0x7f43bc060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43bad9d700 1 --2- 172.21.15.3:0/3496557179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060e40 0x7f43bc0612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b959a700 1 -- 172.21.15.3:0/3496557179 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39406722 0 0) 0x7f43bc060670 con 0x7f43bc060290 2022-07-21T01:34:03.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b959a700 1 -- 172.21.15.3:0/3496557179 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43a0003260 con 0x7f43bc060290 2022-07-21T01:34:03.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b9d9b700 1 -- 172.21.15.3:0/3496557179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060e40 msgr2=0x7f43bc0612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:03.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b9d9b700 1 --2- 172.21.15.3:0/3496557179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060e40 0x7f43bc0612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b9d9b700 1 -- 172.21.15.3:0/3496557179 >> v1:172.21.15.3:6789/0 conn(0x7f43bc060290 legacy=0x7f43bc066610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b9d9b700 1 -- 172.21.15.3:0/3496557179 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43bc066170 con 0x7f43bc0b38d0 2022-07-21T01:34:03.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b9d9b700 1 --2- 172.21.15.3:0/3496557179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc0b38d0 0x7f43bc060950 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f43b0002f70 tx=0x7f43b0006df0).ready entity=mon.0 client_cookie=8625db4626288961 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:03.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b959a700 1 -- 172.21.15.3:0/3496557179 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b000d4a0 con 0x7f43bc0b38d0 2022-07-21T01:34:03.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b959a700 1 -- 172.21.15.3:0/3496557179 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b0000e90 con 0x7f43bc0b38d0 2022-07-21T01:34:03.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43b959a700 1 -- 172.21.15.3:0/3496557179 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b000d7f0 con 0x7f43bc0b38d0 2022-07-21T01:34:03.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43c0a9e700 1 -- 172.21.15.3:0/3496557179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc0b38d0 msgr2=0x7f43bc060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.837+0000 7f43c0a9e700 1 --2- 172.21.15.3:0/3496557179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc0b38d0 0x7f43bc060950 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f43b0002f70 tx=0x7f43b0006df0).stop 2022-07-21T01:34:03.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 -- 172.21.15.3:0/3496557179 shutdown_connections 2022-07-21T01:34:03.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 --2- 172.21.15.3:0/3496557179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060e40 0x7f43bc0612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 --2- 172.21.15.3:0/3496557179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc0b38d0 0x7f43bc060950 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 -- 172.21.15.3:0/3496557179 >> 172.21.15.3:0/3496557179 conn(0x7f43bc05d0c0 msgr2=0x7f43bc05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:03.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 -- 172.21.15.3:0/3496557179 shutdown_connections 2022-07-21T01:34:03.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 -- 172.21.15.3:0/3496557179 wait complete. 2022-07-21T01:34:03.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 Processor -- start 2022-07-21T01:34:03.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 -- start start 2022-07-21T01:34:03.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060290 0x7f43bc0ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc060e40 0x7f43bc14e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43bc0b38d0 0x7f43bc156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc148380 con 0x7f43bc060e40 2022-07-21T01:34:03.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc0b1ca0 con 0x7f43bc0b38d0 2022-07-21T01:34:03.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43c0a9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc0ad3a0 con 0x7f43bc060290 2022-07-21T01:34:03.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43b9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc060e40 0x7f43bc14e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43b9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc060e40 0x7f43bc14e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44014/0 (socket says 172.21.15.3:44014) 2022-07-21T01:34:03.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43b9d9b700 1 -- 172.21.15.3:0/1646709564 learned_addr learned my addr 172.21.15.3:0/1646709564 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:03.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43ba59c700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060290 0x7f43bc0ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43bad9d700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43bc0b38d0 0x7f43bc156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43b9d9b700 1 -- 172.21.15.3:0/1646709564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060290 msgr2=0x7f43bc0ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43b9d9b700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060290 0x7f43bc0ab5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43b9d9b700 1 -- 172.21.15.3:0/1646709564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43bc0b38d0 msgr2=0x7f43bc156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43b9d9b700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43bc0b38d0 0x7f43bc156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43b9d9b700 1 -- 172.21.15.3:0/1646709564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b000b180 con 0x7f43bc060e40 2022-07-21T01:34:03.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.841+0000 7f43b9d9b700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc060e40 0x7f43bc14e370 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f43b0002f70 tx=0x7f43b0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:03.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43ab7fe700 1 -- 172.21.15.3:0/1646709564 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b0012930 con 0x7f43bc060e40 2022-07-21T01:34:03.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43ab7fe700 1 -- 172.21.15.3:0/1646709564 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b0012a90 con 0x7f43bc060e40 2022-07-21T01:34:03.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43bc0ad640 con 0x7f43bc060e40 2022-07-21T01:34:03.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43ab7fe700 1 -- 172.21.15.3:0/1646709564 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b0012df0 con 0x7f43bc060e40 2022-07-21T01:34:03.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43bc0aca20 con 0x7f43bc060e40 2022-07-21T01:34:03.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43bc1568b0 con 0x7f43bc060e40 2022-07-21T01:34:03.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43ab7fe700 1 -- 172.21.15.3:0/1646709564 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f43b001c070 con 0x7f43bc060e40 2022-07-21T01:34:03.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43ab7fe700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4398053950 0x7f4398055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:03.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43ab7fe700 1 -- 172.21.15.3:0/1646709564 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f43b006cbb0 con 0x7f43bc060e40 2022-07-21T01:34:03.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.845+0000 7f43ba59c700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4398053950 0x7f4398055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:03.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.849+0000 7f43ba59c700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4398053950 0x7f4398055e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f43ac00d000 tx=0x7f43ac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:03.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.853+0000 7f43ab7fe700 1 -- 172.21.15.3:0/1646709564 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43bc1568b0 con 0x7f43bc060e40 2022-07-21T01:34:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.973+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43bc1568b0 con 0x7f43bc060e40 2022-07-21T01:34:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.973+0000 7f43ab7fe700 1 -- 172.21.15.3:0/1646709564 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16284 (secure 0 0 0) 0x7f43bc1568b0 con 0x7f43bc060e40 2022-07-21T01:34:03.978 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:03.978 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":36,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:34:03.710370+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:34:01.732226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-21T01:34:02.749428+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:34:03.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4398053950 msgr2=0x7f4398055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4398053950 0x7f4398055e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f43ac00d000 tx=0x7f43ac009040).stop 2022-07-21T01:34:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc060e40 msgr2=0x7f43bc14e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc060e40 0x7f43bc14e370 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f43b0002f70 tx=0x7f43b0014000).stop 2022-07-21T01:34:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 shutdown_connections 2022-07-21T01:34:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4398053950 0x7f4398055e10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43bc0b38d0 0x7f43bc156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43bc060e40 0x7f43bc14e370 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 --2- 172.21.15.3:0/1646709564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43bc060290 0x7f43bc0ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 >> 172.21.15.3:0/1646709564 conn(0x7f43bc05d0c0 msgr2=0x7f43bc0b7090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 shutdown_connections 2022-07-21T01:34:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:03.977+0000 7f43c0a9e700 1 -- 172.21.15.3:0/1646709564 wait complete. 2022-07-21T01:34:03.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:34:04.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.089+0000 7f206a269700 1 Processor -- start 2022-07-21T01:34:04.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.089+0000 7f206a269700 1 -- start start 2022-07-21T01:34:04.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.089+0000 7f206a269700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f20640b01a0 0x7f20640a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.089+0000 7f206a269700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20640a7250 con 0x7f20640afd60 2022-07-21T01:34:04.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.089+0000 7f206a269700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20640a7480 con 0x7f20640b2710 2022-07-21T01:34:04.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.089+0000 7f206a269700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20640a7720 con 0x7f20640b01a0 2022-07-21T01:34:04.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.089+0000 7f2069a68700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f20640afd60 0x7f2064151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52304/0 (socket says 172.21.15.3:52304) 2022-07-21T01:34:04.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.089+0000 7f2069a68700 1 -- 172.21.15.3:0/2269846557 learned_addr learned my addr 172.21.15.3:0/2269846557 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:04.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.089+0000 7f2068a66700 1 --2- 172.21.15.3:0/2269846557 >> v2:172.21.15.143:3300/0 conn(0x7f20640b01a0 0x7f20640a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f205bfff700 1 -- 172.21.15.3:0/2269846557 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956751273 0 0) 0x7f20640a7250 con 0x7f20640afd60 2022-07-21T01:34:04.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f205bfff700 1 -- 172.21.15.3:0/2269846557 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2048003260 con 0x7f20640afd60 2022-07-21T01:34:04.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f205bfff700 1 -- 172.21.15.3:0/2269846557 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936564662 0 0) 0x7f20640a7480 con 0x7f20640b2710 2022-07-21T01:34:04.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f205bfff700 1 -- 172.21.15.3:0/2269846557 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20640a7250 con 0x7f20640b2710 2022-07-21T01:34:04.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f2068a66700 1 -- 172.21.15.3:0/2269846557 >> v1:172.21.15.143:6789/0 conn(0x7f20640b2710 legacy=0x7f206414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f2068a66700 1 -- 172.21.15.3:0/2269846557 >> v1:172.21.15.3:6789/0 conn(0x7f20640afd60 legacy=0x7f2064151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f2068a66700 1 -- 172.21.15.3:0/2269846557 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20640a79c0 con 0x7f20640b01a0 2022-07-21T01:34:04.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f2068a66700 1 --2- 172.21.15.3:0/2269846557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b01a0 0x7f20640a8540 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f2054002f70 tx=0x7f2054006df0).ready entity=mon.1 client_cookie=ea98b1bef31be0fc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:04.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f205bfff700 1 -- 172.21.15.3:0/2269846557 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f205400d540 con 0x7f20640b01a0 2022-07-21T01:34:04.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f205bfff700 1 -- 172.21.15.3:0/2269846557 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2054000e90 con 0x7f20640b01a0 2022-07-21T01:34:04.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f205bfff700 1 -- 172.21.15.3:0/2269846557 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f205400d890 con 0x7f20640b01a0 2022-07-21T01:34:04.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f206a269700 1 -- 172.21.15.3:0/2269846557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b01a0 msgr2=0x7f20640a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f206a269700 1 --2- 172.21.15.3:0/2269846557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b01a0 0x7f20640a8540 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f2054002f70 tx=0x7f2054006df0).stop 2022-07-21T01:34:04.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f206a269700 1 -- 172.21.15.3:0/2269846557 shutdown_connections 2022-07-21T01:34:04.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f206a269700 1 --2- 172.21.15.3:0/2269846557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b01a0 0x7f20640a8540 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f206a269700 1 -- 172.21.15.3:0/2269846557 >> 172.21.15.3:0/2269846557 conn(0x7f2064060310 msgr2=0x7f20640b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:04.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f206a269700 1 -- 172.21.15.3:0/2269846557 shutdown_connections 2022-07-21T01:34:04.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.093+0000 7f206a269700 1 -- 172.21.15.3:0/2269846557 wait complete. 2022-07-21T01:34:04.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 Processor -- start 2022-07-21T01:34:04.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 -- start start 2022-07-21T01:34:04.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20640afd60 0x7f20640aa530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20640b01a0 0x7f20640b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b2710 0x7f20640b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20640a9360 con 0x7f20640b01a0 2022-07-21T01:34:04.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20640b4150 con 0x7f20640b2710 2022-07-21T01:34:04.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20640a9e30 con 0x7f20640afd60 2022-07-21T01:34:04.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2068a66700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20640b01a0 0x7f20640b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2068a66700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20640b01a0 0x7f20640b5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44028/0 (socket says 172.21.15.3:44028) 2022-07-21T01:34:04.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2068a66700 1 -- 172.21.15.3:0/3797730104 learned_addr learned my addr 172.21.15.3:0/3797730104 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:04.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2069a68700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b2710 0x7f20640b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2069267700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20640afd60 0x7f20640aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2068a66700 1 -- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20640afd60 msgr2=0x7f20640aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2068a66700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20640afd60 0x7f20640aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2068a66700 1 -- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b2710 msgr2=0x7f20640b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2068a66700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b2710 0x7f20640b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2068a66700 1 -- 172.21.15.3:0/3797730104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205400b180 con 0x7f20640b01a0 2022-07-21T01:34:04.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2069a68700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b2710 0x7f20640b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:34:04.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2069267700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20640afd60 0x7f20640aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:34:04.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f2068a66700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20640b01a0 0x7f20640b5390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f2054002f70 tx=0x7f2054014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:04.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f205a7fc700 1 -- 172.21.15.3:0/3797730104 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2054016070 con 0x7f20640b01a0 2022-07-21T01:34:04.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f205a7fc700 1 -- 172.21.15.3:0/3797730104 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2054013980 con 0x7f20640b01a0 2022-07-21T01:34:04.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f205a7fc700 1 -- 172.21.15.3:0/3797730104 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2054013ce0 con 0x7f20640b01a0 2022-07-21T01:34:04.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206414d720 con 0x7f20640b01a0 2022-07-21T01:34:04.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.097+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20640ac410 con 0x7f20640b01a0 2022-07-21T01:34:04.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.101+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20640aa230 con 0x7f20640b01a0 2022-07-21T01:34:04.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.101+0000 7f205a7fc700 1 -- 172.21.15.3:0/3797730104 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f205400f030 con 0x7f20640b01a0 2022-07-21T01:34:04.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.101+0000 7f205a7fc700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f204c053a70 0x7f204c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.101+0000 7f205a7fc700 1 -- 172.21.15.3:0/3797730104 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f205406cdd0 con 0x7f20640b01a0 2022-07-21T01:34:04.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.101+0000 7f2069267700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f204c053a70 0x7f204c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.101+0000 7f2069267700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f204c053a70 0x7f204c055f30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2060008000 tx=0x7f2060009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:04.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.105+0000 7f205a7fc700 1 -- 172.21.15.3:0/3797730104 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2054043af0 con 0x7f20640b01a0 2022-07-21T01:34:04.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.233+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20640ac740 con 0x7f20640b01a0 2022-07-21T01:34:04.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.233+0000 7f205a7fc700 1 -- 172.21.15.3:0/3797730104 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16284 (secure 0 0 0) 0x7f2054023070 con 0x7f20640b01a0 2022-07-21T01:34:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:04.238 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":36,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:34:03.710370+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:34:01.732226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-21T01:34:02.749428+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:34:04.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f204c053a70 msgr2=0x7f204c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f204c053a70 0x7f204c055f30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2060008000 tx=0x7f2060009040).stop 2022-07-21T01:34:04.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20640b01a0 msgr2=0x7f20640b5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20640b01a0 0x7f20640b5390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f2054002f70 tx=0x7f2054014000).stop 2022-07-21T01:34:04.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 shutdown_connections 2022-07-21T01:34:04.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f204c053a70 0x7f204c055f30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20640b2710 0x7f20640b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20640b01a0 0x7f20640b5390 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 --2- 172.21.15.3:0/3797730104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f20640afd60 0x7f20640aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 >> 172.21.15.3:0/3797730104 conn(0x7f2064060310 msgr2=0x7f2064148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:04.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 shutdown_connections 2022-07-21T01:34:04.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.237+0000 7f206a269700 1 -- 172.21.15.3:0/3797730104 wait complete. 2022-07-21T01:34:04.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:34:04.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.361+0000 7f68afc6b700 1 Processor -- start 2022-07-21T01:34:04.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.361+0000 7f68afc6b700 1 -- start start 2022-07-21T01:34:04.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f68a80b66f0 0x7f68a80b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f68a80b5b20 0x7f68a80b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a81440f0 con 0x7f68a805fd90 2022-07-21T01:34:04.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a80ba840 con 0x7f68a80b66f0 2022-07-21T01:34:04.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a8060170 con 0x7f68a80b5b20 2022-07-21T01:34:04.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68ad206700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f68a80b66f0 0x7f68a80b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68ada07700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f68a805fd90 0x7f68a80ba130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52314/0 (socket says 172.21.15.3:52314) 2022-07-21T01:34:04.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68ada07700 1 -- 172.21.15.3:0/95649554 learned_addr learned my addr 172.21.15.3:0/95649554 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:04.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68ae208700 1 --2- 172.21.15.3:0/95649554 >> v2:172.21.15.143:3300/0 conn(0x7f68a80b5b20 0x7f68a80b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68aca05700 1 -- 172.21.15.3:0/95649554 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608051566 0 0) 0x7f68a81440f0 con 0x7f68a805fd90 2022-07-21T01:34:04.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68aca05700 1 -- 172.21.15.3:0/95649554 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6890003260 con 0x7f68a805fd90 2022-07-21T01:34:04.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68ad206700 1 -- 172.21.15.3:0/95649554 >> v2:172.21.15.143:3300/0 conn(0x7f68a80b5b20 msgr2=0x7f68a80b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:04.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68ad206700 1 --2- 172.21.15.3:0/95649554 >> v2:172.21.15.143:3300/0 conn(0x7f68a80b5b20 0x7f68a80b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68ad206700 1 -- 172.21.15.3:0/95649554 >> v1:172.21.15.3:6789/0 conn(0x7f68a805fd90 legacy=0x7f68a80ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68ad206700 1 -- 172.21.15.3:0/95649554 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8143be0 con 0x7f68a80b66f0 2022-07-21T01:34:04.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68ad206700 1 --2- 172.21.15.3:0/95649554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 0x7f68a80b4340 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f68a4002f70 tx=0x7f68a4006df0).ready entity=mon.0 client_cookie=7893b4496ff57dac server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:04.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68aca05700 1 -- 172.21.15.3:0/95649554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a400d4f0 con 0x7f68a80b66f0 2022-07-21T01:34:04.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68aca05700 1 -- 172.21.15.3:0/95649554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a4000e90 con 0x7f68a80b66f0 2022-07-21T01:34:04.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68aca05700 1 -- 172.21.15.3:0/95649554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a400d840 con 0x7f68a80b66f0 2022-07-21T01:34:04.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 -- 172.21.15.3:0/95649554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 msgr2=0x7f68a80b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 --2- 172.21.15.3:0/95649554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 0x7f68a80b4340 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f68a4002f70 tx=0x7f68a4006df0).stop 2022-07-21T01:34:04.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 -- 172.21.15.3:0/95649554 shutdown_connections 2022-07-21T01:34:04.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 --2- 172.21.15.3:0/95649554 >> v2:172.21.15.143:3300/0 conn(0x7f68a80b5b20 0x7f68a80b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 --2- 172.21.15.3:0/95649554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 0x7f68a80b4340 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 -- 172.21.15.3:0/95649554 >> 172.21.15.3:0/95649554 conn(0x7f68a8062530 msgr2=0x7f68a8066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:04.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 -- 172.21.15.3:0/95649554 shutdown_connections 2022-07-21T01:34:04.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.365+0000 7f68afc6b700 1 -- 172.21.15.3:0/95649554 wait complete. 2022-07-21T01:34:04.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68afc6b700 1 Processor -- start 2022-07-21T01:34:04.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68afc6b700 1 -- start start 2022-07-21T01:34:04.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68afc6b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a805fd90 0x7f68a8142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68afc6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a80b5b20 0x7f68a8155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68afc6b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 0x7f68a8155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68afc6b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a813fe50 con 0x7f68a80b66f0 2022-07-21T01:34:04.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68afc6b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a81400f0 con 0x7f68a805fd90 2022-07-21T01:34:04.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68afc6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8140390 con 0x7f68a80b5b20 2022-07-21T01:34:04.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68ae208700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 0x7f68a8155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68ae208700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 0x7f68a8155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44044/0 (socket says 172.21.15.3:44044) 2022-07-21T01:34:04.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68ae208700 1 -- 172.21.15.3:0/2673857761 learned_addr learned my addr 172.21.15.3:0/2673857761 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:04.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68ada07700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a805fd90 0x7f68a8142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68ad206700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a80b5b20 0x7f68a8155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.369+0000 7f68ae208700 1 -- 172.21.15.3:0/2673857761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a80b5b20 msgr2=0x7f68a8155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68ae208700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a80b5b20 0x7f68a8155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68ae208700 1 -- 172.21.15.3:0/2673857761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a805fd90 msgr2=0x7f68a8142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68ae208700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a805fd90 0x7f68a8142400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68ae208700 1 -- 172.21.15.3:0/2673857761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a400b180 con 0x7f68a80b66f0 2022-07-21T01:34:04.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68ae208700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 0x7f68a8155f40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f68a0006e00 tx=0x7f68a00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:04.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f689affd700 1 -- 172.21.15.3:0/2673857761 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a000c640 con 0x7f68a80b66f0 2022-07-21T01:34:04.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f689affd700 1 -- 172.21.15.3:0/2673857761 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a000c7a0 con 0x7f68a80b66f0 2022-07-21T01:34:04.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a8140630 con 0x7f68a80b66f0 2022-07-21T01:34:04.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f689affd700 1 -- 172.21.15.3:0/2673857761 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a000cb00 con 0x7f68a80b66f0 2022-07-21T01:34:04.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a8143030 con 0x7f68a80b66f0 2022-07-21T01:34:04.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f689affd700 1 -- 172.21.15.3:0/2673857761 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f68a000cc60 con 0x7f68a80b66f0 2022-07-21T01:34:04.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f689affd700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6888053a70 0x7f6888055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f689affd700 1 -- 172.21.15.3:0/2673857761 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f68a006c250 con 0x7f68a80b66f0 2022-07-21T01:34:04.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68ada07700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6888053a70 0x7f6888055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a81407c0 con 0x7f68a80b66f0 2022-07-21T01:34:04.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.373+0000 7f68ada07700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6888053a70 0x7f6888055f30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f689c00d000 tx=0x7f689c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:04.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.381+0000 7f689affd700 1 -- 172.21.15.3:0/2673857761 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f68a0043010 con 0x7f68a80b66f0 2022-07-21T01:34:04.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.545+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f68a8143e50 con 0x7f68a80b66f0 2022-07-21T01:34:04.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.717+0000 7f689affd700 1 -- 172.21.15.3:0/2673857761 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v12) v1 ==== 118+0+0 (secure 0 0 0) 0x7f68a0046630 con 0x7f68a80b66f0 2022-07-21T01:34:04.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6888053a70 msgr2=0x7f6888055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6888053a70 0x7f6888055f30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f689c00d000 tx=0x7f689c009040).stop 2022-07-21T01:34:04.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 msgr2=0x7f68a8155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 0x7f68a8155f40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f68a0006e00 tx=0x7f68a00091d0).stop 2022-07-21T01:34:04.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 shutdown_connections 2022-07-21T01:34:04.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68a80b66f0 0x7f68a8155f40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a80b5b20 0x7f68a8155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6888053a70 0x7f6888055f30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 --2- 172.21.15.3:0/2673857761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a805fd90 0x7f68a8142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 >> 172.21.15.3:0/2673857761 conn(0x7f68a8062530 msgr2=0x7f68a812c340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:04.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 shutdown_connections 2022-07-21T01:34:04.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.725+0000 7f68afc6b700 1 -- 172.21.15.3:0/2673857761 wait complete. 2022-07-21T01:34:04.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:34:04.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.853+0000 7fb9095f0700 1 Processor -- start 2022-07-21T01:34:04.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.853+0000 7fb9095f0700 1 -- start start 2022-07-21T01:34:04.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb9040b7500 0x7fb9040b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040aadb0 0x7fb904064d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9040b2510 con 0x7fb9040b31f0 2022-07-21T01:34:04.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9041443e0 con 0x7fb9040b7500 2022-07-21T01:34:04.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9040b3700 con 0x7fb9040aadb0 2022-07-21T01:34:04.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb903fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb9040b7500 0x7fb9040b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb903fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb9040b7500 0x7fb9040b78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44048/0 (socket says 172.21.15.3:44048) 2022-07-21T01:34:04.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb903fff700 1 -- 172.21.15.3:0/830604729 learned_addr learned my addr 172.21.15.3:0/830604729 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:04.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb908def700 1 --2- 172.21.15.3:0/830604729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040aadb0 0x7fb904064d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb903fff700 1 -- 172.21.15.3:0/830604729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040aadb0 msgr2=0x7fb904064d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:04.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb903fff700 1 --2- 172.21.15.3:0/830604729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040aadb0 0x7fb904064d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb903fff700 1 -- 172.21.15.3:0/830604729 >> v1:172.21.15.143:6789/0 conn(0x7fb9040b31f0 legacy=0x7fb90413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb903fff700 1 -- 172.21.15.3:0/830604729 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9040b2740 con 0x7fb9040b7500 2022-07-21T01:34:04.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb908def700 1 --2- 172.21.15.3:0/830604729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040aadb0 0x7fb904064d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:34:04.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb903fff700 1 --2- 172.21.15.3:0/830604729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040b7500 0x7fb9040b78e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fb8f4002f70 tx=0x7fb8f4006e50).ready entity=mon.0 client_cookie=4e2f559e5280321d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:04.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb902ffd700 1 -- 172.21.15.3:0/830604729 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8f400d5c0 con 0x7fb9040b7500 2022-07-21T01:34:04.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb902ffd700 1 -- 172.21.15.3:0/830604729 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f4000e90 con 0x7fb9040b7500 2022-07-21T01:34:04.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb902ffd700 1 -- 172.21.15.3:0/830604729 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8f400d950 con 0x7fb9040b7500 2022-07-21T01:34:04.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 -- 172.21.15.3:0/830604729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040b7500 msgr2=0x7fb9040b78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 --2- 172.21.15.3:0/830604729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040b7500 0x7fb9040b78e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fb8f4002f70 tx=0x7fb8f4006e50).stop 2022-07-21T01:34:04.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 -- 172.21.15.3:0/830604729 shutdown_connections 2022-07-21T01:34:04.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 --2- 172.21.15.3:0/830604729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040aadb0 0x7fb904064d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 --2- 172.21.15.3:0/830604729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040b7500 0x7fb9040b78e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 -- 172.21.15.3:0/830604729 >> 172.21.15.3:0/830604729 conn(0x7fb90405f840 msgr2=0x7fb90405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:04.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 -- 172.21.15.3:0/830604729 shutdown_connections 2022-07-21T01:34:04.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.857+0000 7fb9095f0700 1 -- 172.21.15.3:0/830604729 wait complete. 2022-07-21T01:34:04.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 Processor -- start 2022-07-21T01:34:04.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 -- start start 2022-07-21T01:34:04.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040aadb0 0x7fb904147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9040b31f0 0x7fb9040adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040b7500 0x7fb9040ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9040af6e0 con 0x7fb9040aadb0 2022-07-21T01:34:04.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9040b09a0 con 0x7fb9040b31f0 2022-07-21T01:34:04.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9040afd70 con 0x7fb9040b7500 2022-07-21T01:34:04.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb903fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040aadb0 0x7fb904147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb903fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040aadb0 0x7fb904147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44054/0 (socket says 172.21.15.3:44054) 2022-07-21T01:34:04.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb903fff700 1 -- 172.21.15.3:0/2302779701 learned_addr learned my addr 172.21.15.3:0/2302779701 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:04.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb908def700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040b7500 0x7fb9040ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9037fe700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9040b31f0 0x7fb9040adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb903fff700 1 -- 172.21.15.3:0/2302779701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040b7500 msgr2=0x7fb9040ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb903fff700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040b7500 0x7fb9040ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb903fff700 1 -- 172.21.15.3:0/2302779701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9040b31f0 msgr2=0x7fb9040adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:04.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb903fff700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9040b31f0 0x7fb9040adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:04.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb903fff700 1 -- 172.21.15.3:0/2302779701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f400b1e0 con 0x7fb9040aadb0 2022-07-21T01:34:04.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb903fff700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040aadb0 0x7fb904147e30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fb8f4002f70 tx=0x7fb8f4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:04.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9017fa700 1 -- 172.21.15.3:0/2302779701 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8f4006b80 con 0x7fb9040aadb0 2022-07-21T01:34:04.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9017fa700 1 -- 172.21.15.3:0/2302779701 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f400d940 con 0x7fb9040aadb0 2022-07-21T01:34:04.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9040b0010 con 0x7fb9040aadb0 2022-07-21T01:34:04.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9017fa700 1 -- 172.21.15.3:0/2302779701 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8f4011d70 con 0x7fb9040aadb0 2022-07-21T01:34:04.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.861+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9040b0500 con 0x7fb9040aadb0 2022-07-21T01:34:04.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.865+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9040574e0 con 0x7fb9040aadb0 2022-07-21T01:34:04.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.865+0000 7fb9017fa700 1 -- 172.21.15.3:0/2302779701 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb8f4025030 con 0x7fb9040aadb0 2022-07-21T01:34:04.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.865+0000 7fb9017fa700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8e40539e0 0x7fb8e4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:04.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.865+0000 7fb9017fa700 1 -- 172.21.15.3:0/2302779701 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb8f406d110 con 0x7fb9040aadb0 2022-07-21T01:34:04.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.865+0000 7fb9037fe700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8e40539e0 0x7fb8e4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:04.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.869+0000 7fb9037fe700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8e40539e0 0x7fb8e4055ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fb8f800d000 tx=0x7fb8f8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:04.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:04.873+0000 7fb9017fa700 1 -- 172.21.15.3:0/2302779701 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb8f4043ed0 con 0x7fb9040aadb0 2022-07-21T01:34:05.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.037+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb904060330 con 0x7fb9040aadb0 2022-07-21T01:34:05.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9017fa700 1 -- 172.21.15.3:0/2302779701 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2480 (secure 0 0 0) 0x7fb8f4023090 con 0x7fb9040aadb0 2022-07-21T01:34:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:05.044 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:34:03.710770+0000","modified":"2022-07-21T01:34:04.713666+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4896":{"gid":4896,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/24695956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":24695956},{"type":"v1","addr":"172.21.15.3:6837","nonce":24695956}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-21T01:34:05.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8e40539e0 msgr2=0x7fb8e4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8e40539e0 0x7fb8e4055ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fb8f800d000 tx=0x7fb8f8000bc0).stop 2022-07-21T01:34:05.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040aadb0 msgr2=0x7fb904147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040aadb0 0x7fb904147e30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fb8f4002f70 tx=0x7fb8f4015000).stop 2022-07-21T01:34:05.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 shutdown_connections 2022-07-21T01:34:05.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9040b7500 0x7fb9040ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8e40539e0 0x7fb8e4055ea0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9040b31f0 0x7fb9040adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 --2- 172.21.15.3:0/2302779701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9040aadb0 0x7fb904147e30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 >> 172.21.15.3:0/2302779701 conn(0x7fb90405f840 msgr2=0x7fb90405be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:05.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 shutdown_connections 2022-07-21T01:34:05.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.041+0000 7fb9095f0700 1 -- 172.21.15.3:0/2302779701 wait complete. 2022-07-21T01:34:05.049 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 12 2022-07-21T01:34:05.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:34:05.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 Processor -- start 2022-07-21T01:34:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 -- start start 2022-07-21T01:34:05.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ee80aa2b0 0x7f8ee80ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80b0a10 0x7f8ee8140730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee80b2c80 con 0x7f8ee80b0630 2022-07-21T01:34:05.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee80b2900 con 0x7f8ee80aa2b0 2022-07-21T01:34:05.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80b2eb0 con 0x7f8ee80b0a10 2022-07-21T01:34:05.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ee6ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8ee80b0630 0x7f8ee80adec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52342/0 (socket says 172.21.15.3:52342) 2022-07-21T01:34:05.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ee6ffd700 1 -- 172.21.15.3:0/3372521438 learned_addr learned my addr 172.21.15.3:0/3372521438 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:05.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ede7fc700 1 --2- 172.21.15.3:0/3372521438 >> v2:172.21.15.143:3300/0 conn(0x7f8ee80aa2b0 0x7f8ee80ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ee77fe700 1 --2- 172.21.15.3:0/3372521438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80b0a10 0x7f8ee8140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ee67fc700 1 -- 172.21.15.3:0/3372521438 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993321561 0 0) 0x7f8ee80b2c80 con 0x7f8ee80b0630 2022-07-21T01:34:05.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ee67fc700 1 -- 172.21.15.3:0/3372521438 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec8003260 con 0x7f8ee80b0630 2022-07-21T01:34:05.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ede7fc700 1 -- 172.21.15.3:0/3372521438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80b0a10 msgr2=0x7f8ee8140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:05.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ede7fc700 1 --2- 172.21.15.3:0/3372521438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80b0a10 0x7f8ee8140730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ede7fc700 1 -- 172.21.15.3:0/3372521438 >> v1:172.21.15.3:6789/0 conn(0x7f8ee80b0630 legacy=0x7f8ee80adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ede7fc700 1 -- 172.21.15.3:0/3372521438 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee80a9d00 con 0x7f8ee80aa2b0 2022-07-21T01:34:05.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ede7fc700 1 --2- 172.21.15.3:0/3372521438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ee80aa2b0 0x7f8ee80ae5d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f8ed0002f70 tx=0x7f8ed0006df0).ready entity=mon.1 client_cookie=a11f035cf1b200b2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:05.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ee67fc700 1 -- 172.21.15.3:0/3372521438 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ed0014070 con 0x7f8ee80aa2b0 2022-07-21T01:34:05.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ee67fc700 1 -- 172.21.15.3:0/3372521438 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed0000e90 con 0x7f8ee80aa2b0 2022-07-21T01:34:05.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8ee67fc700 1 -- 172.21.15.3:0/3372521438 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ed000fa90 con 0x7f8ee80aa2b0 2022-07-21T01:34:05.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3372521438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ee80aa2b0 msgr2=0x7f8ee80ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 --2- 172.21.15.3:0/3372521438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ee80aa2b0 0x7f8ee80ae5d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f8ed0002f70 tx=0x7f8ed0006df0).stop 2022-07-21T01:34:05.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3372521438 shutdown_connections 2022-07-21T01:34:05.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 --2- 172.21.15.3:0/3372521438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80b0a10 0x7f8ee8140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 --2- 172.21.15.3:0/3372521438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ee80aa2b0 0x7f8ee80ae5d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3372521438 >> 172.21.15.3:0/3372521438 conn(0x7f8ee8060950 msgr2=0x7f8ee8060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:05.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3372521438 shutdown_connections 2022-07-21T01:34:05.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.153+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3372521438 wait complete. 2022-07-21T01:34:05.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 Processor -- start 2022-07-21T01:34:05.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 -- start start 2022-07-21T01:34:05.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80aa2b0 0x7f8ee80ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ee80b0630 0x7f8ee813bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ee80b0a10 0x7f8ee815a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80abf80 con 0x7f8ee80b0a10 2022-07-21T01:34:05.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee813e940 con 0x7f8ee80b0630 2022-07-21T01:34:05.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee813e6a0 con 0x7f8ee80aa2b0 2022-07-21T01:34:05.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ee80b0a10 0x7f8ee815a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ee80b0a10 0x7f8ee815a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44072/0 (socket says 172.21.15.3:44072) 2022-07-21T01:34:05.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee77fe700 1 -- 172.21.15.3:0/3607690119 learned_addr learned my addr 172.21.15.3:0/3607690119 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:05.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ede7fc700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ee80b0630 0x7f8ee813bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee6ffd700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80aa2b0 0x7f8ee80ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee77fe700 1 -- 172.21.15.3:0/3607690119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80aa2b0 msgr2=0x7f8ee80ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee77fe700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80aa2b0 0x7f8ee80ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee77fe700 1 -- 172.21.15.3:0/3607690119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ee80b0630 msgr2=0x7f8ee813bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee77fe700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ee80b0630 0x7f8ee813bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee77fe700 1 -- 172.21.15.3:0/3607690119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed000b180 con 0x7f8ee80b0a10 2022-07-21T01:34:05.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee77fe700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ee80b0a10 0x7f8ee815a1b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f8ee000c000 tx=0x7f8ee00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:05.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee4ff9700 1 -- 172.21.15.3:0/3607690119 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ee0012600 con 0x7f8ee80b0a10 2022-07-21T01:34:05.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee4ff9700 1 -- 172.21.15.3:0/3607690119 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee0012760 con 0x7f8ee80b0a10 2022-07-21T01:34:05.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee8064b70 con 0x7f8ee80b0a10 2022-07-21T01:34:05.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8ee4ff9700 1 -- 172.21.15.3:0/3607690119 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ee0012ac0 con 0x7f8ee80b0a10 2022-07-21T01:34:05.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.157+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee80aa6d0 con 0x7f8ee80b0a10 2022-07-21T01:34:05.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.161+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee80ab170 con 0x7f8ee80b0a10 2022-07-21T01:34:05.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.161+0000 7f8ee4ff9700 1 -- 172.21.15.3:0/3607690119 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8ee0022070 con 0x7f8ee80b0a10 2022-07-21T01:34:05.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.161+0000 7f8ee4ff9700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ecc053a70 0x7f8ecc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.161+0000 7f8ee4ff9700 1 -- 172.21.15.3:0/3607690119 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8ee006c5d0 con 0x7f8ee80b0a10 2022-07-21T01:34:05.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.161+0000 7f8ee6ffd700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ecc053a70 0x7f8ecc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.161+0000 7f8ee6ffd700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ecc053a70 0x7f8ecc055f30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f8ed800d000 tx=0x7f8ed8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:05.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.169+0000 7f8ee4ff9700 1 -- 172.21.15.3:0/3607690119 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8ee0043330 con 0x7f8ee80b0a10 2022-07-21T01:34:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.293+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ee80574e0 con 0x7f8ee80b0a10 2022-07-21T01:34:05.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8ee4ff9700 1 -- 172.21.15.3:0/3607690119 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16271 (secure 0 0 0) 0x7f8ee0016020 con 0x7f8ee80b0a10 2022-07-21T01:34:05.298 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:05.299 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":37,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:34:04.714529+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:34:01.732226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-21T01:34:02.749428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:34:05.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ecc053a70 msgr2=0x7f8ecc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ecc053a70 0x7f8ecc055f30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f8ed800d000 tx=0x7f8ed8009040).stop 2022-07-21T01:34:05.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ee80b0a10 msgr2=0x7f8ee815a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ee80b0a10 0x7f8ee815a1b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f8ee000c000 tx=0x7f8ee00091d0).stop 2022-07-21T01:34:05.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 shutdown_connections 2022-07-21T01:34:05.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ee80b0a10 0x7f8ee815a1b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ecc053a70 0x7f8ecc055f30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ee80b0630 0x7f8ee813bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 --2- 172.21.15.3:0/3607690119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ee80aa2b0 0x7f8ee80ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 >> 172.21.15.3:0/3607690119 conn(0x7f8ee8060950 msgr2=0x7f8ee80b5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:05.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 shutdown_connections 2022-07-21T01:34:05.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.297+0000 7f8eed3d7700 1 -- 172.21.15.3:0/3607690119 wait complete. 2022-07-21T01:34:05.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:34:05.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dc4ed2700 1 Processor -- start 2022-07-21T01:34:05.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dc4ed2700 1 -- start start 2022-07-21T01:34:05.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dc4ed2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2dc00b08a0 0x7f2dc0142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dc4ed2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2dc00b3260 0x7f2dc01433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dc4ed2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00ad000 0x7f2dc005f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dc4ed2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc00b0c80 con 0x7f2dc00b08a0 2022-07-21T01:34:05.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dc4ed2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc00ae810 con 0x7f2dc00b3260 2022-07-21T01:34:05.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dc4ed2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc00aeab0 con 0x7f2dc00ad000 2022-07-21T01:34:05.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbf7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2dc00b08a0 0x7f2dc0142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbf7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2dc00b08a0 0x7f2dc0142eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44076/0 (socket says 172.21.15.3:44076) 2022-07-21T01:34:05.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbf7fe700 1 -- 172.21.15.3:0/2838887602 learned_addr learned my addr 172.21.15.3:0/2838887602 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:05.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbeffd700 1 --2- 172.21.15.3:0/2838887602 >> v2:172.21.15.143:3300/0 conn(0x7f2dc00b3260 0x7f2dc01433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbffff700 1 --2- 172.21.15.3:0/2838887602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00ad000 0x7f2dc005f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbf7fe700 1 -- 172.21.15.3:0/2838887602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00ad000 msgr2=0x7f2dc005f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:05.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbf7fe700 1 --2- 172.21.15.3:0/2838887602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00ad000 0x7f2dc005f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbf7fe700 1 -- 172.21.15.3:0/2838887602 >> v2:172.21.15.143:3300/0 conn(0x7f2dc00b3260 msgr2=0x7f2dc01433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:05.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbf7fe700 1 --2- 172.21.15.3:0/2838887602 >> v2:172.21.15.143:3300/0 conn(0x7f2dc00b3260 0x7f2dc01433a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbf7fe700 1 -- 172.21.15.3:0/2838887602 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc00aed50 con 0x7f2dc00b08a0 2022-07-21T01:34:05.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.429+0000 7f2dbeffd700 1 --2- 172.21.15.3:0/2838887602 >> v2:172.21.15.143:3300/0 conn(0x7f2dc00b3260 0x7f2dc01433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:34:05.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dbf7fe700 1 --2- 172.21.15.3:0/2838887602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b08a0 0x7f2dc0142eb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f2db0002f70 tx=0x7f2db0006f40).ready entity=mon.0 client_cookie=ae3bc48a72dc6f72 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:05.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dbe7fc700 1 -- 172.21.15.3:0/2838887602 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2db0010810 con 0x7f2dc00b08a0 2022-07-21T01:34:05.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dbe7fc700 1 -- 172.21.15.3:0/2838887602 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db0000e90 con 0x7f2dc00b08a0 2022-07-21T01:34:05.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dbe7fc700 1 -- 172.21.15.3:0/2838887602 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2db0010ba0 con 0x7f2dc00b08a0 2022-07-21T01:34:05.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2838887602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b08a0 msgr2=0x7f2dc0142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2838887602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b08a0 0x7f2dc0142eb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f2db0002f70 tx=0x7f2db0006f40).stop 2022-07-21T01:34:05.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2838887602 shutdown_connections 2022-07-21T01:34:05.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2838887602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00ad000 0x7f2dc005f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2838887602 >> v2:172.21.15.143:3300/0 conn(0x7f2dc00b3260 0x7f2dc01433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2838887602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b08a0 0x7f2dc0142eb0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2838887602 >> 172.21.15.3:0/2838887602 conn(0x7f2dc00612d0 msgr2=0x7f2dc0062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:05.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2838887602 shutdown_connections 2022-07-21T01:34:05.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.433+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2838887602 wait complete. 2022-07-21T01:34:05.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dc4ed2700 1 Processor -- start 2022-07-21T01:34:05.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dc4ed2700 1 -- start start 2022-07-21T01:34:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dc4ed2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dc00ad000 0x7f2dc00b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dc4ed2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00b08a0 0x7f2dc013e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dc4ed2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b3260 0x7f2dc013ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dc4ed2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc00671c0 con 0x7f2dc00b3260 2022-07-21T01:34:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dc4ed2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc00674f0 con 0x7f2dc00ad000 2022-07-21T01:34:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dc4ed2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc0064f20 con 0x7f2dc00b08a0 2022-07-21T01:34:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b3260 0x7f2dc013ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b3260 0x7f2dc013ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44086/0 (socket says 172.21.15.3:44086) 2022-07-21T01:34:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbffff700 1 -- 172.21.15.3:0/2447814995 learned_addr learned my addr 172.21.15.3:0/2447814995 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbf7fe700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dc00ad000 0x7f2dc00b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbeffd700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00b08a0 0x7f2dc013e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbffff700 1 -- 172.21.15.3:0/2447814995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00b08a0 msgr2=0x7f2dc013e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbffff700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00b08a0 0x7f2dc013e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbffff700 1 -- 172.21.15.3:0/2447814995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dc00ad000 msgr2=0x7f2dc00b6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbffff700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dc00ad000 0x7f2dc00b6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbffff700 1 -- 172.21.15.3:0/2447814995 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db400b180 con 0x7f2dc00b3260 2022-07-21T01:34:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbffff700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b3260 0x7f2dc013ee90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f2db8006e00 tx=0x7f2db80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.437+0000 7f2dbcff9700 1 -- 172.21.15.3:0/2447814995 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2db800c640 con 0x7f2dc00b3260 2022-07-21T01:34:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dbcff9700 1 -- 172.21.15.3:0/2447814995 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db800c7a0 con 0x7f2dc00b3260 2022-07-21T01:34:05.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db000a360 con 0x7f2dc00b3260 2022-07-21T01:34:05.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dbcff9700 1 -- 172.21.15.3:0/2447814995 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2db800cb00 con 0x7f2dc00b3260 2022-07-21T01:34:05.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc013f380 con 0x7f2dc00b3260 2022-07-21T01:34:05.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc0065850 con 0x7f2dc00b3260 2022-07-21T01:34:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dbcff9700 1 -- 172.21.15.3:0/2447814995 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2db8022070 con 0x7f2dc00b3260 2022-07-21T01:34:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dbcff9700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2da0053a10 0x7f2da0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dbf7fe700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2da0053a10 0x7f2da0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dbcff9700 1 -- 172.21.15.3:0/2447814995 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2db806c660 con 0x7f2dc00b3260 2022-07-21T01:34:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.441+0000 7f2dbf7fe700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2da0053a10 0x7f2da0055ed0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f2db001b000 tx=0x7f2dc00a95b0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:05.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.449+0000 7f2dbcff9700 1 -- 172.21.15.3:0/2447814995 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2db8043420 con 0x7f2dc00b3260 2022-07-21T01:34:05.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.621+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2dc00574e0 con 0x7f2dc00b3260 2022-07-21T01:34:05.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dbcff9700 1 -- 172.21.15.3:0/2447814995 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2db8046a40 con 0x7f2dc00b3260 2022-07-21T01:34:05.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2da0053a10 msgr2=0x7f2da0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2da0053a10 0x7f2da0055ed0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f2db001b000 tx=0x7f2dc00a95b0).stop 2022-07-21T01:34:05.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b3260 msgr2=0x7f2dc013ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b3260 0x7f2dc013ee90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f2db8006e00 tx=0x7f2db80091d0).stop 2022-07-21T01:34:05.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 shutdown_connections 2022-07-21T01:34:05.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2dc00b3260 0x7f2dc013ee90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dc00b08a0 0x7f2dc013e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2da0053a10 0x7f2da0055ed0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 --2- 172.21.15.3:0/2447814995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dc00ad000 0x7f2dc00b6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 >> 172.21.15.3:0/2447814995 conn(0x7f2dc00612d0 msgr2=0x7f2dc0142340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 shutdown_connections 2022-07-21T01:34:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.629+0000 7f2dc4ed2700 1 -- 172.21.15.3:0/2447814995 wait complete. 2022-07-21T01:34:05.637 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:34:05.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:34:05.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.757+0000 7f353a98b700 1 Processor -- start 2022-07-21T01:34:05.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.757+0000 7f353a98b700 1 -- start start 2022-07-21T01:34:05.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.757+0000 7f353a98b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3534060290 0x7f35340617e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.757+0000 7f353a98b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 0x7f3534066e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.757+0000 7f353a98b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3534060670 con 0x7f3534065b00 2022-07-21T01:34:05.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.757+0000 7f353a98b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35340663f0 con 0x7f3534060290 2022-07-21T01:34:05.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.757+0000 7f353a98b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3534060e00 con 0x7f3534065fb0 2022-07-21T01:34:05.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.757+0000 7f3533fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3534060290 0x7f35340617e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3538f28700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 0x7f3534066e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3533fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3534060290 0x7f35340617e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44090/0 (socket says 172.21.15.3:44090) 2022-07-21T01:34:05.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3533fff700 1 -- 172.21.15.3:0/2678842790 learned_addr learned my addr 172.21.15.3:0/2678842790 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:05.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3533fff700 1 -- 172.21.15.3:0/2678842790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 msgr2=0x7f3534066e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:05.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3533fff700 1 --2- 172.21.15.3:0/2678842790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 0x7f3534066e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3533fff700 1 -- 172.21.15.3:0/2678842790 >> v1:172.21.15.143:6789/0 conn(0x7f3534065b00 legacy=0x7f3534066760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3533fff700 1 -- 172.21.15.3:0/2678842790 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35340610a0 con 0x7f3534060290 2022-07-21T01:34:05.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3533fff700 1 --2- 172.21.15.3:0/2678842790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534060290 0x7f35340617e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f3524002f70 tx=0x7f3524006eb0).ready entity=mon.0 client_cookie=b6e67e79515b846c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:05.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3532ffd700 1 -- 172.21.15.3:0/2678842790 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f352400d570 con 0x7f3534060290 2022-07-21T01:34:05.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3532ffd700 1 -- 172.21.15.3:0/2678842790 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3524000e90 con 0x7f3534060290 2022-07-21T01:34:05.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f3532ffd700 1 -- 172.21.15.3:0/2678842790 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f352400d900 con 0x7f3534060290 2022-07-21T01:34:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f353a98b700 1 -- 172.21.15.3:0/2678842790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534060290 msgr2=0x7f35340617e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f353a98b700 1 --2- 172.21.15.3:0/2678842790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534060290 0x7f35340617e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f3524002f70 tx=0x7f3524006eb0).stop 2022-07-21T01:34:05.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f353a98b700 1 -- 172.21.15.3:0/2678842790 shutdown_connections 2022-07-21T01:34:05.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f353a98b700 1 --2- 172.21.15.3:0/2678842790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 0x7f3534066e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f353a98b700 1 --2- 172.21.15.3:0/2678842790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534060290 0x7f35340617e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f353a98b700 1 -- 172.21.15.3:0/2678842790 >> 172.21.15.3:0/2678842790 conn(0x7f353405d0c0 msgr2=0x7f353405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f353a98b700 1 -- 172.21.15.3:0/2678842790 shutdown_connections 2022-07-21T01:34:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.761+0000 7f353a98b700 1 -- 172.21.15.3:0/2678842790 wait complete. 2022-07-21T01:34:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f353a98b700 1 Processor -- start 2022-07-21T01:34:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f353a98b700 1 -- start start 2022-07-21T01:34:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f353a98b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534060290 0x7f3534146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f353a98b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534065b00 0x7f3534153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f353a98b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 0x7f3534155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f353a98b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35340a4ba0 con 0x7f3534065b00 2022-07-21T01:34:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f353a98b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35340a4e40 con 0x7f3534060290 2022-07-21T01:34:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f353a98b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3534152bf0 con 0x7f3534065fb0 2022-07-21T01:34:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f35337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534065b00 0x7f3534153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f3538f28700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 0x7f3534155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f3533fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534060290 0x7f3534146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f35337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534065b00 0x7f3534153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44098/0 (socket says 172.21.15.3:44098) 2022-07-21T01:34:05.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f35337fe700 1 -- 172.21.15.3:0/2487976801 learned_addr learned my addr 172.21.15.3:0/2487976801 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:05.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f35337fe700 1 -- 172.21.15.3:0/2487976801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 msgr2=0x7f3534155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f35337fe700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 0x7f3534155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f35337fe700 1 -- 172.21.15.3:0/2487976801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534060290 msgr2=0x7f3534146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f35337fe700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534060290 0x7f3534146c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f35337fe700 1 -- 172.21.15.3:0/2487976801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352400b1e0 con 0x7f3534065b00 2022-07-21T01:34:05.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.765+0000 7f35337fe700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534065b00 0x7f3534153090 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f35340aeac0 tx=0x7f3528007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:05.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f35317fa700 1 -- 172.21.15.3:0/2487976801 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3528012400 con 0x7f3534065b00 2022-07-21T01:34:05.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3534150540 con 0x7f3534065b00 2022-07-21T01:34:05.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f35317fa700 1 -- 172.21.15.3:0/2487976801 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3528012560 con 0x7f3534065b00 2022-07-21T01:34:05.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f35317fa700 1 -- 172.21.15.3:0/2487976801 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35280128c0 con 0x7f3534065b00 2022-07-21T01:34:05.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35340b5d90 con 0x7f3534065b00 2022-07-21T01:34:05.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f35317fa700 1 -- 172.21.15.3:0/2487976801 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3528019030 con 0x7f3534065b00 2022-07-21T01:34:05.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3534152db0 con 0x7f3534065b00 2022-07-21T01:34:05.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f35317fa700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3514053aa0 0x7f3514055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:05.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f35317fa700 1 -- 172.21.15.3:0/2487976801 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f352806b550 con 0x7f3534065b00 2022-07-21T01:34:05.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f3533fff700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3514053aa0 0x7f3514055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:05.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.769+0000 7f3533fff700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3514053aa0 0x7f3514055f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f352401c000 tx=0x7f3524015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:05.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.773+0000 7f35317fa700 1 -- 172.21.15.3:0/2487976801 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3528042270 con 0x7f3534065b00 2022-07-21T01:34:05.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.937+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3534061340 con 0x7f3534065b00 2022-07-21T01:34:05.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.945+0000 7f35317fa700 1 -- 172.21.15.3:0/2487976801 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3528045890 con 0x7f3534065b00 2022-07-21T01:34:05.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3514053aa0 msgr2=0x7f3514055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3514053aa0 0x7f3514055f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f352401c000 tx=0x7f3524015000).stop 2022-07-21T01:34:05.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534065b00 msgr2=0x7f3534153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:05.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534065b00 0x7f3534153090 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f35340aeac0 tx=0x7f3528007c50).stop 2022-07-21T01:34:05.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 shutdown_connections 2022-07-21T01:34:05.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3534065fb0 0x7f3534155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3534065b00 0x7f3534153090 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3514053aa0 0x7f3514055f60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 --2- 172.21.15.3:0/2487976801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534060290 0x7f3534146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:05.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 >> 172.21.15.3:0/2487976801 conn(0x7f353405d0c0 msgr2=0x7f35340b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:05.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 shutdown_connections 2022-07-21T01:34:05.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:05.949+0000 7f353a98b700 1 -- 172.21.15.3:0/2487976801 wait complete. 2022-07-21T01:34:05.955 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:34:05.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:34:06.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.069+0000 7fabaa230700 1 Processor -- start 2022-07-21T01:34:06.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.069+0000 7fabaa230700 1 -- start start 2022-07-21T01:34:06.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba40ad1c0 0x7faba4144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba4067560 con 0x7faba4067180 2022-07-21T01:34:06.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba40b0160 con 0x7faba40b0870 2022-07-21T01:34:06.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba40b0400 con 0x7faba40ad1c0 2022-07-21T01:34:06.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba37fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7faba4067180 0x7faba4143a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52384/0 (socket says 172.21.15.3:52384) 2022-07-21T01:34:06.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba37fe700 1 -- 172.21.15.3:0/3418030054 learned_addr learned my addr 172.21.15.3:0/3418030054 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:06.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba2ffd700 1 --2- 172.21.15.3:0/3418030054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba40ad1c0 0x7faba4144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba27fc700 1 -- 172.21.15.3:0/3418030054 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201028962 0 0) 0x7faba4067560 con 0x7faba4067180 2022-07-21T01:34:06.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba27fc700 1 -- 172.21.15.3:0/3418030054 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab84003260 con 0x7faba4067180 2022-07-21T01:34:06.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba27fc700 1 -- 172.21.15.3:0/3418030054 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458280914 0 0) 0x7faba40b0160 con 0x7faba40b0870 2022-07-21T01:34:06.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba2ffd700 1 -- 172.21.15.3:0/3418030054 >> v1:172.21.15.143:6789/0 conn(0x7faba40b0870 legacy=0x7faba40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba2ffd700 1 -- 172.21.15.3:0/3418030054 >> v1:172.21.15.3:6789/0 conn(0x7faba4067180 legacy=0x7faba4143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba2ffd700 1 -- 172.21.15.3:0/3418030054 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40af530 con 0x7faba40ad1c0 2022-07-21T01:34:06.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba2ffd700 1 --2- 172.21.15.3:0/3418030054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba40ad1c0 0x7faba4144170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fab8c0032f0 tx=0x7fab8c0091d0).ready entity=mon.2 client_cookie=b4e9d6b6f47a7ca9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:06.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba27fc700 1 -- 172.21.15.3:0/3418030054 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab8c00b6c0 con 0x7faba40ad1c0 2022-07-21T01:34:06.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba27fc700 1 -- 172.21.15.3:0/3418030054 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8c00b820 con 0x7faba40ad1c0 2022-07-21T01:34:06.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7faba27fc700 1 -- 172.21.15.3:0/3418030054 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab8c00b990 con 0x7faba40ad1c0 2022-07-21T01:34:06.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 -- 172.21.15.3:0/3418030054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba40ad1c0 msgr2=0x7faba4144170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 --2- 172.21.15.3:0/3418030054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba40ad1c0 0x7faba4144170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fab8c0032f0 tx=0x7fab8c0091d0).stop 2022-07-21T01:34:06.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 -- 172.21.15.3:0/3418030054 shutdown_connections 2022-07-21T01:34:06.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 --2- 172.21.15.3:0/3418030054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba40ad1c0 0x7faba4144170 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 -- 172.21.15.3:0/3418030054 >> 172.21.15.3:0/3418030054 conn(0x7faba405f840 msgr2=0x7faba405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:06.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 -- 172.21.15.3:0/3418030054 shutdown_connections 2022-07-21T01:34:06.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.073+0000 7fabaa230700 1 -- 172.21.15.3:0/3418030054 wait complete. 2022-07-21T01:34:06.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 Processor -- start 2022-07-21T01:34:06.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 -- start start 2022-07-21T01:34:06.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba4067180 0x7faba40a6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faba40b0870 0x7faba40a2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faba40a2d80 0x7faba41526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba40a3ac0 con 0x7faba40b0870 2022-07-21T01:34:06.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba40a5f90 con 0x7faba40a2d80 2022-07-21T01:34:06.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba40a3cf0 con 0x7faba4067180 2022-07-21T01:34:06.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faba40b0870 0x7faba40a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faba40b0870 0x7faba40a2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44112/0 (socket says 172.21.15.3:44112) 2022-07-21T01:34:06.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba2ffd700 1 -- 172.21.15.3:0/3680284841 learned_addr learned my addr 172.21.15.3:0/3680284841 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:06.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba37fe700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba4067180 0x7faba40a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba3fff700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faba40a2d80 0x7faba41526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba2ffd700 1 -- 172.21.15.3:0/3680284841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba4067180 msgr2=0x7faba40a6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba2ffd700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba4067180 0x7faba40a6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba2ffd700 1 -- 172.21.15.3:0/3680284841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faba40a2d80 msgr2=0x7faba41526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba2ffd700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faba40a2d80 0x7faba41526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba2ffd700 1 -- 172.21.15.3:0/3680284841 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8c009470 con 0x7faba40b0870 2022-07-21T01:34:06.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba37fe700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba4067180 0x7faba40a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:34:06.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba2ffd700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faba40b0870 0x7faba40a2400 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fab8c0032f0 tx=0x7fab8c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:06.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba0ff9700 1 -- 172.21.15.3:0/3680284841 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab8c00b990 con 0x7faba40b0870 2022-07-21T01:34:06.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba0ff9700 1 -- 172.21.15.3:0/3680284841 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8c011e80 con 0x7faba40b0870 2022-07-21T01:34:06.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba40a3f90 con 0x7faba40b0870 2022-07-21T01:34:06.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7faba0ff9700 1 -- 172.21.15.3:0/3680284841 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab8c00b6c0 con 0x7faba40b0870 2022-07-21T01:34:06.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.077+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba40a1780 con 0x7faba40b0870 2022-07-21T01:34:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.081+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba40a33b0 con 0x7faba40b0870 2022-07-21T01:34:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.081+0000 7faba0ff9700 1 -- 172.21.15.3:0/3680284841 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fab8c00f070 con 0x7faba40b0870 2022-07-21T01:34:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.081+0000 7faba0ff9700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab88053a70 0x7fab88055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.081+0000 7faba0ff9700 1 -- 172.21.15.3:0/3680284841 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fab8c06cb70 con 0x7faba40b0870 2022-07-21T01:34:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.081+0000 7faba37fe700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab88053a70 0x7fab88055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.081+0000 7faba37fe700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab88053a70 0x7fab88055f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fab9400d000 tx=0x7fab94009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.089+0000 7faba0ff9700 1 -- 172.21.15.3:0/3680284841 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fab8c043820 con 0x7faba40b0870 2022-07-21T01:34:06.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.253+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faba40574e0 con 0x7faba40b0870 2022-07-21T01:34:06.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.253+0000 7faba0ff9700 1 -- 172.21.15.3:0/3680284841 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2480 (secure 0 0 0) 0x7fab8c015070 con 0x7faba40b0870 2022-07-21T01:34:06.257 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:06.258 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:34:03.710770+0000","modified":"2022-07-21T01:34:04.713666+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4896":{"gid":4896,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/24695956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":24695956},{"type":"v1","addr":"172.21.15.3:6837","nonce":24695956}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-21T01:34:06.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab88053a70 msgr2=0x7fab88055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab88053a70 0x7fab88055f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fab9400d000 tx=0x7fab94009040).stop 2022-07-21T01:34:06.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faba40b0870 msgr2=0x7faba40a2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faba40b0870 0x7faba40a2400 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fab8c0032f0 tx=0x7fab8c016000).stop 2022-07-21T01:34:06.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 shutdown_connections 2022-07-21T01:34:06.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab88053a70 0x7fab88055f30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faba40a2d80 0x7faba41526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faba40b0870 0x7faba40a2400 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 --2- 172.21.15.3:0/3680284841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faba4067180 0x7faba40a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 >> 172.21.15.3:0/3680284841 conn(0x7faba405f840 msgr2=0x7faba4147dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:06.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 shutdown_connections 2022-07-21T01:34:06.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.257+0000 7fabaa230700 1 -- 172.21.15.3:0/3680284841 wait complete. 2022-07-21T01:34:06.268 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 12 2022-07-21T01:34:06.577 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:34:03.710770+0000', 'modified': '2022-07-21T01:34:04.713666+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4896}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4896': {'gid': 4896, 'name': 'a', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/24695956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 24695956}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 24695956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:34:06.578 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:34:06.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:34:06.696 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":4896,"want_state":"up:active","state":"up:active","rank_uptime":2.9723532229999998,"mdsmap_epoch":12,"osdmap_epoch":37,"osdmap_epoch_barrier":36,"uptime":9.8838440240000001} 2022-07-21T01:34:06.697 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 4896, "mdsmap_epoch": 12, "osdmap_epoch": 37, "osdmap_epoch_barrier": 36, "rank_uptime": 2.972353223, "state": "up:active", "uptime": 9.883844024, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:34:06.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:34:06.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.801+0000 7f81aa59a700 1 Processor -- start 2022-07-21T01:34:06.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.801+0000 7f81aa59a700 1 -- start start 2022-07-21T01:34:06.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a40b41f0 0x7f81a4061d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f81a4142ff0 0x7f81a41434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a40b60e0 con 0x7f81a40b6420 2022-07-21T01:34:06.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a40b2100 con 0x7f81a4142ff0 2022-07-21T01:34:06.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a40b23a0 con 0x7f81a40b41f0 2022-07-21T01:34:06.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a9d99700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f81a4142ff0 0x7f81a41434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a9598700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f81a40b6420 0x7f81a4061600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52398/0 (socket says 172.21.15.3:52398) 2022-07-21T01:34:06.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a9d99700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f81a4142ff0 0x7f81a41434a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44122/0 (socket says 172.21.15.3:44122) 2022-07-21T01:34:06.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a9598700 1 -- 172.21.15.3:0/2095432488 learned_addr learned my addr 172.21.15.3:0/2095432488 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:06.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a8d97700 1 --2- 172.21.15.3:0/2095432488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a40b41f0 0x7f81a4061d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a9d99700 1 -- 172.21.15.3:0/2095432488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a40b41f0 msgr2=0x7f81a4061d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:06.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a9d99700 1 --2- 172.21.15.3:0/2095432488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a40b41f0 0x7f81a4061d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a9d99700 1 -- 172.21.15.3:0/2095432488 >> v1:172.21.15.3:6789/0 conn(0x7f81a40b6420 legacy=0x7f81a4061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a9d99700 1 -- 172.21.15.3:0/2095432488 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a40b2640 con 0x7f81a4142ff0 2022-07-21T01:34:06.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81a9d99700 1 --2- 172.21.15.3:0/2095432488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a4142ff0 0x7f81a41434a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f819c0032f0 tx=0x7f819c0091d0).ready entity=mon.0 client_cookie=3463f8164cda34c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:06.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f818ffff700 1 -- 172.21.15.3:0/2095432488 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f819c009ea0 con 0x7f81a4142ff0 2022-07-21T01:34:06.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f818ffff700 1 -- 172.21.15.3:0/2095432488 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c00b8a0 con 0x7f81a4142ff0 2022-07-21T01:34:06.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f818ffff700 1 -- 172.21.15.3:0/2095432488 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f819c00ba10 con 0x7f81a4142ff0 2022-07-21T01:34:06.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 -- 172.21.15.3:0/2095432488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a4142ff0 msgr2=0x7f81a41434a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 --2- 172.21.15.3:0/2095432488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a4142ff0 0x7f81a41434a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f819c0032f0 tx=0x7f819c0091d0).stop 2022-07-21T01:34:06.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 -- 172.21.15.3:0/2095432488 shutdown_connections 2022-07-21T01:34:06.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 --2- 172.21.15.3:0/2095432488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a4142ff0 0x7f81a41434a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 --2- 172.21.15.3:0/2095432488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a40b41f0 0x7f81a4061d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 -- 172.21.15.3:0/2095432488 >> 172.21.15.3:0/2095432488 conn(0x7f81a405c620 msgr2=0x7f81a405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:06.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 -- 172.21.15.3:0/2095432488 shutdown_connections 2022-07-21T01:34:06.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.805+0000 7f81aa59a700 1 -- 172.21.15.3:0/2095432488 wait complete. 2022-07-21T01:34:06.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81aa59a700 1 Processor -- start 2022-07-21T01:34:06.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81aa59a700 1 -- start start 2022-07-21T01:34:06.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81aa59a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81a40b41f0 0x7f81a40b9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81aa59a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a40b6420 0x7f81a412af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81aa59a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a4142ff0 0x7f81a412b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81aa59a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a413f8b0 con 0x7f81a40b6420 2022-07-21T01:34:06.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81aa59a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a413fe30 con 0x7f81a40b41f0 2022-07-21T01:34:06.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81aa59a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a41400d0 con 0x7f81a4142ff0 2022-07-21T01:34:06.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a8d97700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a40b6420 0x7f81a412af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a8d97700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a40b6420 0x7f81a412af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44126/0 (socket says 172.21.15.3:44126) 2022-07-21T01:34:06.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a8d97700 1 -- 172.21.15.3:0/3195017711 learned_addr learned my addr 172.21.15.3:0/3195017711 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:06.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a9d99700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a4142ff0 0x7f81a412b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a9598700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81a40b41f0 0x7f81a40b9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a8d97700 1 -- 172.21.15.3:0/3195017711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a4142ff0 msgr2=0x7f81a412b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a8d97700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a4142ff0 0x7f81a412b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a8d97700 1 -- 172.21.15.3:0/3195017711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81a40b41f0 msgr2=0x7f81a40b9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a8d97700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81a40b41f0 0x7f81a40b9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a8d97700 1 -- 172.21.15.3:0/3195017711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c009470 con 0x7f81a40b6420 2022-07-21T01:34:06.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81a8d97700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a40b6420 0x7f81a412af80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f819800a000 tx=0x7f8198008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f8196ffd700 1 -- 172.21.15.3:0/3195017711 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f819800dea0 con 0x7f81a40b6420 2022-07-21T01:34:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f8196ffd700 1 -- 172.21.15.3:0/3195017711 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8198011780 con 0x7f81a40b6420 2022-07-21T01:34:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a412b960 con 0x7f81a40b6420 2022-07-21T01:34:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.809+0000 7f8196ffd700 1 -- 172.21.15.3:0/3195017711 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8198011ae0 con 0x7f81a40b6420 2022-07-21T01:34:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.813+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a413f3a0 con 0x7f81a40b6420 2022-07-21T01:34:06.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.813+0000 7f8196ffd700 1 -- 172.21.15.3:0/3195017711 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8198011c40 con 0x7f81a40b6420 2022-07-21T01:34:06.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.813+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a4140290 con 0x7f81a40b6420 2022-07-21T01:34:06.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.813+0000 7f8196ffd700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f81800539e0 0x7f8180055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:06.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.813+0000 7f8196ffd700 1 -- 172.21.15.3:0/3195017711 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f819806c440 con 0x7f81a40b6420 2022-07-21T01:34:06.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.813+0000 7f81a9598700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f81800539e0 0x7f8180055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:06.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.813+0000 7f81a9598700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f81800539e0 0x7f8180055ea0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f81a000d000 tx=0x7f81a0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:06.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.817+0000 7f8196ffd700 1 -- 172.21.15.3:0/3195017711 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8198043090 con 0x7f81a40b6420 2022-07-21T01:34:06.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.945+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81a40b15c0 con 0x7f81a40b6420 2022-07-21T01:34:06.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.945+0000 7f8196ffd700 1 -- 172.21.15.3:0/3195017711 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16271 (secure 0 0 0) 0x7f81980466b0 con 0x7f81a40b6420 2022-07-21T01:34:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:06.951 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":37,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:34:04.714529+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:34:01.732226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-21T01:34:02.749428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:34:06.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f81800539e0 msgr2=0x7f8180055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f81800539e0 0x7f8180055ea0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f81a000d000 tx=0x7f81a0009040).stop 2022-07-21T01:34:06.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a40b6420 msgr2=0x7f81a412af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:06.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a40b6420 0x7f81a412af80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f819800a000 tx=0x7f8198008040).stop 2022-07-21T01:34:06.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 shutdown_connections 2022-07-21T01:34:06.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81a4142ff0 0x7f81a412b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81a40b6420 0x7f81a412af80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f81800539e0 0x7f8180055ea0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 --2- 172.21.15.3:0/3195017711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81a40b41f0 0x7f81a40b9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:06.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 >> 172.21.15.3:0/3195017711 conn(0x7f81a405c620 msgr2=0x7f81a40b4830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:06.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 shutdown_connections 2022-07-21T01:34:06.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:06.949+0000 7f81aa59a700 1 -- 172.21.15.3:0/3195017711 wait complete. 2022-07-21T01:34:06.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:34:01.732226+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:34:02.749428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:34:06.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:34:07.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada2219700 1 Processor -- start 2022-07-21T01:34:07.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada2219700 1 -- start start 2022-07-21T01:34:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada2219700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 0x7fad9c143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada2219700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad9c140b10 0x7fad9c128e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada2219700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9c063e40 con 0x7fad9c061ab0 2022-07-21T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada2219700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad9c13ff40 con 0x7fad9c140b10 2022-07-21T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada2219700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c062180 con 0x7fad9c063a60 2022-07-21T01:34:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1217700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 0x7fad9c143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1a18700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad9c140b10 0x7fad9c128e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1a18700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad9c140b10 0x7fad9c128e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:49580/0 (socket says 172.21.15.3:49580) 2022-07-21T01:34:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1217700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 0x7fad9c143ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:51048/0 (socket says 172.21.15.3:51048) 2022-07-21T01:34:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1a18700 1 -- 172.21.15.3:0/71109240 learned_addr learned my addr 172.21.15.3:0/71109240 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada0a16700 1 --1- 172.21.15.3:0/71109240 >> v1:172.21.15.143:6789/0 conn(0x7fad9c061ab0 0x7fad9c0b1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:54532/0 (socket says 172.21.15.3:54532) 2022-07-21T01:34:07.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1a18700 1 -- 172.21.15.3:0/71109240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 msgr2=0x7fad9c143ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:07.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1a18700 1 --2- 172.21.15.3:0/71109240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 0x7fad9c143ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1a18700 1 -- 172.21.15.3:0/71109240 >> v1:172.21.15.143:6789/0 conn(0x7fad9c061ab0 legacy=0x7fad9c0b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1a18700 1 -- 172.21.15.3:0/71109240 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c140130 con 0x7fad9c140b10 2022-07-21T01:34:07.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fada1a18700 1 --2- 172.21.15.3:0/71109240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c140b10 0x7fad9c128e10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fad940032f0 tx=0x7fad9400b530).ready entity=mon.1 client_cookie=aa75f143208fc268 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fad93fff700 1 -- 172.21.15.3:0/71109240 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad9400de20 con 0x7fad9c140b10 2022-07-21T01:34:07.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fad93fff700 1 -- 172.21.15.3:0/71109240 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad94010470 con 0x7fad9c140b10 2022-07-21T01:34:07.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.073+0000 7fad93fff700 1 -- 172.21.15.3:0/71109240 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad940107d0 con 0x7fad9c140b10 2022-07-21T01:34:07.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 -- 172.21.15.3:0/71109240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c140b10 msgr2=0x7fad9c128e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 --2- 172.21.15.3:0/71109240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c140b10 0x7fad9c128e10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fad940032f0 tx=0x7fad9400b530).stop 2022-07-21T01:34:07.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 -- 172.21.15.3:0/71109240 shutdown_connections 2022-07-21T01:34:07.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 --2- 172.21.15.3:0/71109240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c140b10 0x7fad9c128e10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 --2- 172.21.15.3:0/71109240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 0x7fad9c143ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 -- 172.21.15.3:0/71109240 >> 172.21.15.3:0/71109240 conn(0x7fad9c05c620 msgr2=0x7fad9c05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:07.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 -- 172.21.15.3:0/71109240 shutdown_connections 2022-07-21T01:34:07.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 -- 172.21.15.3:0/71109240 wait complete. 2022-07-21T01:34:07.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 Processor -- start 2022-07-21T01:34:07.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.077+0000 7fada2219700 1 -- start start 2022-07-21T01:34:07.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada2219700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c061ab0 0x7fad9c13f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada2219700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 0x7fad9c0b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada2219700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad9c140b10 0x7fad9c0b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada2219700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c153810 con 0x7fad9c140b10 2022-07-21T01:34:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada2219700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c1528f0 con 0x7fad9c061ab0 2022-07-21T01:34:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada2219700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c152b90 con 0x7fad9c063a60 2022-07-21T01:34:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1a18700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad9c140b10 0x7fad9c0b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1217700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c061ab0 0x7fad9c13f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1a18700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad9c140b10 0x7fad9c0b45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44142/0 (socket says 172.21.15.3:44142) 2022-07-21T01:34:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1a18700 1 -- 172.21.15.3:0/1481835237 learned_addr learned my addr 172.21.15.3:0/1481835237 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada0a16700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 0x7fad9c0b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1217700 1 -- 172.21.15.3:0/1481835237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 msgr2=0x7fad9c0b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1217700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 0x7fad9c0b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1217700 1 -- 172.21.15.3:0/1481835237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad9c140b10 msgr2=0x7fad9c0b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1217700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad9c140b10 0x7fad9c0b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1217700 1 -- 172.21.15.3:0/1481835237 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9400b7d0 con 0x7fad9c061ab0 2022-07-21T01:34:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada1217700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c061ab0 0x7fad9c13f5f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fad9800c000 tx=0x7fad98006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fad927fc700 1 -- 172.21.15.3:0/1481835237 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad98011750 con 0x7fad9c061ab0 2022-07-21T01:34:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fad927fc700 1 -- 172.21.15.3:0/1481835237 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad980118b0 con 0x7fad9c061ab0 2022-07-21T01:34:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9c152e30 con 0x7fad9c061ab0 2022-07-21T01:34:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fad927fc700 1 -- 172.21.15.3:0/1481835237 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad98011c10 con 0x7fad9c061ab0 2022-07-21T01:34:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9c0b4ae0 con 0x7fad9c061ab0 2022-07-21T01:34:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9c0574e0 con 0x7fad9c061ab0 2022-07-21T01:34:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.081+0000 7fad927fc700 1 -- 172.21.15.3:0/1481835237 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fad98011d70 con 0x7fad9c061ab0 2022-07-21T01:34:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.085+0000 7fad927fc700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad84053b30 0x7fad84055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.085+0000 7fad927fc700 1 -- 172.21.15.3:0/1481835237 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fad9806bfc0 con 0x7fad9c061ab0 2022-07-21T01:34:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.085+0000 7fada0a16700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad84053b30 0x7fad84055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.085+0000 7fada0a16700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad84053b30 0x7fad84055ff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fad8c00d000 tx=0x7fad8c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.093+0000 7fad927fc700 1 -- 172.21.15.3:0/1481835237 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad98042d20 con 0x7fad9c061ab0 2022-07-21T01:34:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.217+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fad9c129530 con 0x7fad9c061ab0 2022-07-21T01:34:07.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fad927fc700 1 -- 172.21.15.3:0/1481835237 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v37) v1 ==== 110+0+10 (secure 0 0 0) 0x7fad98046340 con 0x7fad9c061ab0 2022-07-21T01:34:07.222 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:34:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad84053b30 msgr2=0x7fad84055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad84053b30 0x7fad84055ff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fad8c00d000 tx=0x7fad8c009040).stop 2022-07-21T01:34:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c061ab0 msgr2=0x7fad9c13f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c061ab0 0x7fad9c13f5f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fad9800c000 tx=0x7fad98006df0).stop 2022-07-21T01:34:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 shutdown_connections 2022-07-21T01:34:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad9c140b10 0x7fad9c0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad9c063a60 0x7fad9c0b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad84053b30 0x7fad84055ff0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 --2- 172.21.15.3:0/1481835237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad9c061ab0 0x7fad9c13f5f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 >> 172.21.15.3:0/1481835237 conn(0x7fad9c05c620 msgr2=0x7fad9c1421a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 shutdown_connections 2022-07-21T01:34:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.221+0000 7fada2219700 1 -- 172.21.15.3:0/1481835237 wait complete. 2022-07-21T01:34:07.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:34:07.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.337+0000 7fb1f8805700 1 Processor -- start 2022-07-21T01:34:07.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.337+0000 7fb1f8805700 1 -- start start 2022-07-21T01:34:07.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.337+0000 7fb1f8805700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb1f40b36e0 0x7fb1f4141a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.337+0000 7fb1f8805700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f4064ef0 con 0x7fb1f4144e60 2022-07-21T01:34:07.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.337+0000 7fb1f8805700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f4066690 con 0x7fb1f4064b10 2022-07-21T01:34:07.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.337+0000 7fb1f8805700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f413ebf0 con 0x7fb1f40b36e0 2022-07-21T01:34:07.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f3fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb1f4144e60 0x7fb1f4066ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52430/0 (socket says 172.21.15.3:52430) 2022-07-21T01:34:07.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f3fff700 1 -- 172.21.15.3:0/2194470509 learned_addr learned my addr 172.21.15.3:0/2194470509 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:07.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f2ffd700 1 --2- 172.21.15.3:0/2194470509 >> v2:172.21.15.143:3300/0 conn(0x7fb1f40b36e0 0x7fb1f4141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f27fc700 1 -- 172.21.15.3:0/2194470509 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507643861 0 0) 0x7fb1f4064ef0 con 0x7fb1f4144e60 2022-07-21T01:34:07.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f27fc700 1 -- 172.21.15.3:0/2194470509 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e0003260 con 0x7fb1f4144e60 2022-07-21T01:34:07.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f27fc700 1 -- 172.21.15.3:0/2194470509 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452414379 0 0) 0x7fb1f4066690 con 0x7fb1f4064b10 2022-07-21T01:34:07.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f27fc700 1 -- 172.21.15.3:0/2194470509 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f4064ef0 con 0x7fb1f4064b10 2022-07-21T01:34:07.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f2ffd700 1 -- 172.21.15.3:0/2194470509 >> v1:172.21.15.143:6789/0 conn(0x7fb1f4064b10 legacy=0x7fb1f4141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f2ffd700 1 -- 172.21.15.3:0/2194470509 >> v1:172.21.15.3:6789/0 conn(0x7fb1f4144e60 legacy=0x7fb1f4066ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f27fc700 1 -- 172.21.15.3:0/2194470509 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 33358145 0 0) 0x7fb1e0003260 con 0x7fb1f4144e60 2022-07-21T01:34:07.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f2ffd700 1 -- 172.21.15.3:0/2194470509 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f40b4850 con 0x7fb1f40b36e0 2022-07-21T01:34:07.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f2ffd700 1 --2- 172.21.15.3:0/2194470509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f40b36e0 0x7fb1f4141a80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fb1e8002f70 tx=0x7fb1e8006df0).ready entity=mon.1 client_cookie=bd19da2a641f9ef2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f27fc700 1 -- 172.21.15.3:0/2194470509 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1e800d540 con 0x7fb1f40b36e0 2022-07-21T01:34:07.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f27fc700 1 -- 172.21.15.3:0/2194470509 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e8000e90 con 0x7fb1f40b36e0 2022-07-21T01:34:07.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f27fc700 1 -- 172.21.15.3:0/2194470509 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1e800d890 con 0x7fb1f40b36e0 2022-07-21T01:34:07.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f8805700 1 -- 172.21.15.3:0/2194470509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f40b36e0 msgr2=0x7fb1f4141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f8805700 1 --2- 172.21.15.3:0/2194470509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f40b36e0 0x7fb1f4141a80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fb1e8002f70 tx=0x7fb1e8006df0).stop 2022-07-21T01:34:07.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f8805700 1 -- 172.21.15.3:0/2194470509 shutdown_connections 2022-07-21T01:34:07.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f8805700 1 --2- 172.21.15.3:0/2194470509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f40b36e0 0x7fb1f4141a80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f8805700 1 -- 172.21.15.3:0/2194470509 >> 172.21.15.3:0/2194470509 conn(0x7fb1f405f840 msgr2=0x7fb1f405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:07.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f8805700 1 -- 172.21.15.3:0/2194470509 shutdown_connections 2022-07-21T01:34:07.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.341+0000 7fb1f8805700 1 -- 172.21.15.3:0/2194470509 wait complete. 2022-07-21T01:34:07.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 Processor -- start 2022-07-21T01:34:07.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 -- start start 2022-07-21T01:34:07.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1f4064b10 0x7fb1f41512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f40b36e0 0x7fb1f413f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1f4144e60 0x7fb1f40b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41506d0 con 0x7fb1f4064b10 2022-07-21T01:34:07.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41508c0 con 0x7fb1f40b36e0 2022-07-21T01:34:07.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f4150b60 con 0x7fb1f4144e60 2022-07-21T01:34:07.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1f4064b10 0x7fb1f41512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1f4064b10 0x7fb1f41512a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44152/0 (socket says 172.21.15.3:44152) 2022-07-21T01:34:07.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f37fe700 1 -- 172.21.15.3:0/3626638215 learned_addr learned my addr 172.21.15.3:0/3626638215 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:07.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f2ffd700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f40b36e0 0x7fb1f413f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f3fff700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1f4144e60 0x7fb1f40b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f37fe700 1 -- 172.21.15.3:0/3626638215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1f4144e60 msgr2=0x7fb1f40b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f37fe700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1f4144e60 0x7fb1f40b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f37fe700 1 -- 172.21.15.3:0/3626638215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f40b36e0 msgr2=0x7fb1f413f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f37fe700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f40b36e0 0x7fb1f413f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f37fe700 1 -- 172.21.15.3:0/3626638215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e800b180 con 0x7fb1f4064b10 2022-07-21T01:34:07.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f37fe700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1f4064b10 0x7fb1f41512a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fb1e4001110 tx=0x7fb1e4007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f0ff9700 1 -- 172.21.15.3:0/3626638215 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1e4012400 con 0x7fb1f4064b10 2022-07-21T01:34:07.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f0ff9700 1 -- 172.21.15.3:0/3626638215 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e4012560 con 0x7fb1f4064b10 2022-07-21T01:34:07.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f4150e00 con 0x7fb1f4064b10 2022-07-21T01:34:07.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f0ff9700 1 -- 172.21.15.3:0/3626638215 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1e40128c0 con 0x7fb1f4064b10 2022-07-21T01:34:07.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.345+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f4153ca0 con 0x7fb1f4064b10 2022-07-21T01:34:07.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.349+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f40b4af0 con 0x7fb1f4064b10 2022-07-21T01:34:07.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.349+0000 7fb1f0ff9700 1 -- 172.21.15.3:0/3626638215 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb1e4019030 con 0x7fb1f4064b10 2022-07-21T01:34:07.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.349+0000 7fb1f0ff9700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb1d4053aa0 0x7fb1d4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.349+0000 7fb1f0ff9700 1 -- 172.21.15.3:0/3626638215 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb1e406b330 con 0x7fb1f4064b10 2022-07-21T01:34:07.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.349+0000 7fb1f2ffd700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb1d4053aa0 0x7fb1d4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.353+0000 7fb1f2ffd700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb1d4053aa0 0x7fb1d4055f60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb1e801a000 tx=0x7fb1e8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.353+0000 7fb1f0ff9700 1 -- 172.21.15.3:0/3626638215 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb1e40420f0 con 0x7fb1f4064b10 2022-07-21T01:34:07.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.481+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb1f40a7b70 con 0x7fb1f4064b10 2022-07-21T01:34:07.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.481+0000 7fb1f0ff9700 1 -- 172.21.15.3:0/3626638215 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v37) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb1e4045710 con 0x7fb1f4064b10 2022-07-21T01:34:07.484 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:34:07.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb1d4053aa0 msgr2=0x7fb1d4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb1d4053aa0 0x7fb1d4055f60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb1e801a000 tx=0x7fb1e8014000).stop 2022-07-21T01:34:07.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1f4064b10 msgr2=0x7fb1f41512a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1f4064b10 0x7fb1f41512a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fb1e4001110 tx=0x7fb1e4007ae0).stop 2022-07-21T01:34:07.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 shutdown_connections 2022-07-21T01:34:07.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1f4144e60 0x7fb1f40b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb1d4053aa0 0x7fb1d4055f60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f40b36e0 0x7fb1f413f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 --2- 172.21.15.3:0/3626638215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1f4064b10 0x7fb1f41512a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 >> 172.21.15.3:0/3626638215 conn(0x7fb1f405f840 msgr2=0x7fb1f4140360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:07.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 shutdown_connections 2022-07-21T01:34:07.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.485+0000 7fb1f8805700 1 -- 172.21.15.3:0/3626638215 wait complete. 2022-07-21T01:34:07.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:34:07.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.601+0000 7f9c8dd5d700 1 Processor -- start 2022-07-21T01:34:07.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.601+0000 7f9c8dd5d700 1 -- start start 2022-07-21T01:34:07.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c880b6ee0 0x7f9c880b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880b5ed0 0x7f9c880b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c88061d90 con 0x7f9c880619b0 2022-07-21T01:34:07.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c88143fa0 con 0x7f9c880b6ee0 2022-07-21T01:34:07.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88142f60 con 0x7f9c880b5ed0 2022-07-21T01:34:07.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8d55c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880b5ed0 0x7f9c880b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c87fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c880b6ee0 0x7f9c880b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8d55c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880b5ed0 0x7f9c880b5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:51080/0 (socket says 172.21.15.3:51080) 2022-07-21T01:34:07.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8d55c700 1 -- 172.21.15.3:0/694050839 learned_addr learned my addr 172.21.15.3:0/694050839 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:07.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c87fff700 1 -- 172.21.15.3:0/694050839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880b5ed0 msgr2=0x7f9c880b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:07.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c87fff700 1 --2- 172.21.15.3:0/694050839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880b5ed0 0x7f9c880b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c87fff700 1 -- 172.21.15.3:0/694050839 >> v1:172.21.15.143:6789/0 conn(0x7f9c880619b0 legacy=0x7f9c88143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c87fff700 1 -- 172.21.15.3:0/694050839 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c880ba5a0 con 0x7f9c880b6ee0 2022-07-21T01:34:07.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c87fff700 1 --2- 172.21.15.3:0/694050839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c880b6ee0 0x7f9c880b48a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f9c880b6880 tx=0x7f9c700091d0).ready entity=mon.1 client_cookie=5ee5bcf37e1aa0d3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c877fe700 1 -- 172.21.15.3:0/694050839 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c7000b680 con 0x7f9c880b6ee0 2022-07-21T01:34:07.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c877fe700 1 -- 172.21.15.3:0/694050839 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7000b7e0 con 0x7f9c880b6ee0 2022-07-21T01:34:07.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c877fe700 1 -- 172.21.15.3:0/694050839 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c7000b950 con 0x7f9c880b6ee0 2022-07-21T01:34:07.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/694050839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c880b6ee0 msgr2=0x7f9c880b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 --2- 172.21.15.3:0/694050839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c880b6ee0 0x7f9c880b48a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f9c880b6880 tx=0x7f9c700091d0).stop 2022-07-21T01:34:07.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/694050839 shutdown_connections 2022-07-21T01:34:07.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 --2- 172.21.15.3:0/694050839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880b5ed0 0x7f9c880b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 --2- 172.21.15.3:0/694050839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c880b6ee0 0x7f9c880b48a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/694050839 >> 172.21.15.3:0/694050839 conn(0x7f9c8806af80 msgr2=0x7f9c8805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:07.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/694050839 shutdown_connections 2022-07-21T01:34:07.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.605+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/694050839 wait complete. 2022-07-21T01:34:07.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 Processor -- start 2022-07-21T01:34:07.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 -- start start 2022-07-21T01:34:07.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880619b0 0x7f9c880b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c880b5ed0 0x7f9c8813f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c880b6ee0 0x7f9c8813f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88146d70 con 0x7f9c880b5ed0 2022-07-21T01:34:07.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88155ae0 con 0x7f9c880b6ee0 2022-07-21T01:34:07.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88145c90 con 0x7f9c880619b0 2022-07-21T01:34:07.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c87fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c880b5ed0 0x7f9c8813f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8d55c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c880b6ee0 0x7f9c8813f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c87fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c880b5ed0 0x7f9c8813f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44170/0 (socket says 172.21.15.3:44170) 2022-07-21T01:34:07.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c87fff700 1 -- 172.21.15.3:0/2242620451 learned_addr learned my addr 172.21.15.3:0/2242620451 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:07.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8cd5b700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880619b0 0x7f9c880b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c87fff700 1 -- 172.21.15.3:0/2242620451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880619b0 msgr2=0x7f9c880b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c87fff700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880619b0 0x7f9c880b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c87fff700 1 -- 172.21.15.3:0/2242620451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c880b6ee0 msgr2=0x7f9c8813f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c87fff700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c880b6ee0 0x7f9c8813f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c87fff700 1 -- 172.21.15.3:0/2242620451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c70009470 con 0x7f9c880b5ed0 2022-07-21T01:34:07.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c87fff700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c880b5ed0 0x7f9c8813f210 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f9c880b6880 tx=0x7f9c70016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c85ffb700 1 -- 172.21.15.3:0/2242620451 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c7000b680 con 0x7f9c880b5ed0 2022-07-21T01:34:07.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c85ffb700 1 -- 172.21.15.3:0/2242620451 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c70021630 con 0x7f9c880b5ed0 2022-07-21T01:34:07.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c85ffb700 1 -- 172.21.15.3:0/2242620451 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c70021990 con 0x7f9c880b5ed0 2022-07-21T01:34:07.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c881467b0 con 0x7f9c880b5ed0 2022-07-21T01:34:07.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.609+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c88112a30 con 0x7f9c880b5ed0 2022-07-21T01:34:07.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.613+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c880574e0 con 0x7f9c880b5ed0 2022-07-21T01:34:07.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.613+0000 7f9c85ffb700 1 -- 172.21.15.3:0/2242620451 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9c70027020 con 0x7f9c880b5ed0 2022-07-21T01:34:07.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.613+0000 7f9c85ffb700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9c6c053a70 0x7f9c6c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.613+0000 7f9c85ffb700 1 -- 172.21.15.3:0/2242620451 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9c7006d600 con 0x7f9c880b5ed0 2022-07-21T01:34:07.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.613+0000 7f9c8cd5b700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9c6c053a70 0x7f9c6c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.613+0000 7f9c8cd5b700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9c6c053a70 0x7f9c6c055f30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f9c7800d000 tx=0x7f9c78000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.621+0000 7f9c85ffb700 1 -- 172.21.15.3:0/2242620451 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9c700442b0 con 0x7f9c880b5ed0 2022-07-21T01:34:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.745+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c8805db60 con 0x7f9c880b5ed0 2022-07-21T01:34:07.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c85ffb700 1 -- 172.21.15.3:0/2242620451 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9c700478d0 con 0x7f9c880b5ed0 2022-07-21T01:34:07.751 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:34:07.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9c6c053a70 msgr2=0x7f9c6c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9c6c053a70 0x7f9c6c055f30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f9c7800d000 tx=0x7f9c78000bc0).stop 2022-07-21T01:34:07.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c880b5ed0 msgr2=0x7f9c8813f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c880b5ed0 0x7f9c8813f210 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f9c880b6880 tx=0x7f9c70016000).stop 2022-07-21T01:34:07.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 shutdown_connections 2022-07-21T01:34:07.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9c6c053a70 0x7f9c6c055f30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c880b6ee0 0x7f9c8813f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c880b5ed0 0x7f9c8813f210 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 --2- 172.21.15.3:0/2242620451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c880619b0 0x7f9c880b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 >> 172.21.15.3:0/2242620451 conn(0x7f9c8806af80 msgr2=0x7f9c881423e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:07.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 shutdown_connections 2022-07-21T01:34:07.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.749+0000 7f9c8dd5d700 1 -- 172.21.15.3:0/2242620451 wait complete. 2022-07-21T01:34:07.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:34:07.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5f4a3d700 1 Processor -- start 2022-07-21T01:34:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5f4a3d700 1 -- start start 2022-07-21T01:34:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5f4a3d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5f00b2710 0x7fb5f014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5f4a3d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5f00b01a0 0x7fb5f014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5f4a3d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f00a7250 con 0x7fb5f00afd60 2022-07-21T01:34:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5f4a3d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f00a7480 con 0x7fb5f00b2710 2022-07-21T01:34:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5f4a3d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f00a7720 con 0x7fb5f00b01a0 2022-07-21T01:34:07.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ef7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5f00b2710 0x7fb5f014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ef7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5f00b2710 0x7fb5f014afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44174/0 (socket says 172.21.15.3:44174) 2022-07-21T01:34:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ef7fe700 1 -- 172.21.15.3:0/2646588431 learned_addr learned my addr 172.21.15.3:0/2646588431 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5eeffd700 1 --2- 172.21.15.3:0/2646588431 >> v2:172.21.15.143:3300/0 conn(0x7fb5f00b01a0 0x7fb5f014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ef7fe700 1 -- 172.21.15.3:0/2646588431 >> v2:172.21.15.143:3300/0 conn(0x7fb5f00b01a0 msgr2=0x7fb5f014b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ef7fe700 1 --2- 172.21.15.3:0/2646588431 >> v2:172.21.15.143:3300/0 conn(0x7fb5f00b01a0 0x7fb5f014b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ef7fe700 1 -- 172.21.15.3:0/2646588431 >> v1:172.21.15.143:6789/0 conn(0x7fb5f00afd60 legacy=0x7fb5f00a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ef7fe700 1 -- 172.21.15.3:0/2646588431 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f00a79c0 con 0x7fb5f00b2710 2022-07-21T01:34:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ef7fe700 1 --2- 172.21.15.3:0/2646588431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b2710 0x7fb5f014afd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fb5d8002f70 tx=0x7fb5d8006eb0).ready entity=mon.0 client_cookie=6073e3aeaf03c71f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ee7fc700 1 -- 172.21.15.3:0/2646588431 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5d800d610 con 0x7fb5f00b2710 2022-07-21T01:34:07.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ee7fc700 1 -- 172.21.15.3:0/2646588431 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d8000e90 con 0x7fb5f00b2710 2022-07-21T01:34:07.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.873+0000 7fb5ee7fc700 1 -- 172.21.15.3:0/2646588431 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5d800d9a0 con 0x7fb5f00b2710 2022-07-21T01:34:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/2646588431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b2710 msgr2=0x7fb5f014afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 --2- 172.21.15.3:0/2646588431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b2710 0x7fb5f014afd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fb5d8002f70 tx=0x7fb5d8006eb0).stop 2022-07-21T01:34:07.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/2646588431 shutdown_connections 2022-07-21T01:34:07.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 --2- 172.21.15.3:0/2646588431 >> v2:172.21.15.143:3300/0 conn(0x7fb5f00b01a0 0x7fb5f014b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 --2- 172.21.15.3:0/2646588431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b2710 0x7fb5f014afd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/2646588431 >> 172.21.15.3:0/2646588431 conn(0x7fb5f0060310 msgr2=0x7fb5f00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:07.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/2646588431 shutdown_connections 2022-07-21T01:34:07.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/2646588431 wait complete. 2022-07-21T01:34:07.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 Processor -- start 2022-07-21T01:34:07.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.877+0000 7fb5f4a3d700 1 -- start start 2022-07-21T01:34:07.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5f4a3d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f00afd60 0x7fb5f00674c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5f4a3d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b01a0 0x7fb5f00aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5f4a3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5f00b2710 0x7fb5f0159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5f4a3d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f0067d30 con 0x7fb5f00b01a0 2022-07-21T01:34:07.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5f4a3d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f0065f70 con 0x7fb5f00afd60 2022-07-21T01:34:07.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5f4a3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f0067a90 con 0x7fb5f00b2710 2022-07-21T01:34:07.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5eeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b01a0 0x7fb5f00aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5eeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b01a0 0x7fb5f00aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44182/0 (socket says 172.21.15.3:44182) 2022-07-21T01:34:07.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5eeffd700 1 -- 172.21.15.3:0/3938839809 learned_addr learned my addr 172.21.15.3:0/3938839809 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:07.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5ef7fe700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f00afd60 0x7fb5f00674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5effff700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5f00b2710 0x7fb5f0159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5eeffd700 1 -- 172.21.15.3:0/3938839809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5f00b2710 msgr2=0x7fb5f0159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5eeffd700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5f00b2710 0x7fb5f0159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5eeffd700 1 -- 172.21.15.3:0/3938839809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f00afd60 msgr2=0x7fb5f00674c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:07.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5eeffd700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f00afd60 0x7fb5f00674c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:07.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5eeffd700 1 -- 172.21.15.3:0/3938839809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d800b1e0 con 0x7fb5f00b01a0 2022-07-21T01:34:07.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5eeffd700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b01a0 0x7fb5f00aeab0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fb5e000a000 tx=0x7fb5e0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5ecff9700 1 -- 172.21.15.3:0/3938839809 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5e0011780 con 0x7fb5f00b01a0 2022-07-21T01:34:07.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f00ab120 con 0x7fb5f00b01a0 2022-07-21T01:34:07.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f00ab860 con 0x7fb5f00b01a0 2022-07-21T01:34:07.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5ecff9700 1 -- 172.21.15.3:0/3938839809 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e00118e0 con 0x7fb5f00b01a0 2022-07-21T01:34:07.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5ecff9700 1 -- 172.21.15.3:0/3938839809 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5e0011c80 con 0x7fb5f00b01a0 2022-07-21T01:34:07.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.881+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f00574e0 con 0x7fb5f00b01a0 2022-07-21T01:34:07.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.885+0000 7fb5ecff9700 1 -- 172.21.15.3:0/3938839809 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb5e0012420 con 0x7fb5f00b01a0 2022-07-21T01:34:07.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.885+0000 7fb5ecff9700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5d4053b00 0x7fb5d4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:07.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.885+0000 7fb5ecff9700 1 -- 172.21.15.3:0/3938839809 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb5e006bfb0 con 0x7fb5f00b01a0 2022-07-21T01:34:07.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.885+0000 7fb5ef7fe700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5d4053b00 0x7fb5d4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:07.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.885+0000 7fb5ef7fe700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5d4053b00 0x7fb5d4055fc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fb5d801c000 tx=0x7fb5d8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:07.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:07.889+0000 7fb5ecff9700 1 -- 172.21.15.3:0/3938839809 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5e0042c70 con 0x7fb5f00b01a0 2022-07-21T01:34:08.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.013+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb5f0069a70 con 0x7fb5f00b01a0 2022-07-21T01:34:08.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5ecff9700 1 -- 172.21.15.3:0/3938839809 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb5e0046290 con 0x7fb5f00b01a0 2022-07-21T01:34:08.018 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:34:08.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5d4053b00 msgr2=0x7fb5d4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:08.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5d4053b00 0x7fb5d4055fc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fb5d801c000 tx=0x7fb5d8015000).stop 2022-07-21T01:34:08.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b01a0 msgr2=0x7fb5f00aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:08.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b01a0 0x7fb5f00aeab0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fb5e000a000 tx=0x7fb5e0008040).stop 2022-07-21T01:34:08.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 shutdown_connections 2022-07-21T01:34:08.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5f00b2710 0x7fb5f0159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f00b01a0 0x7fb5f00aeab0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5d4053b00 0x7fb5d4055fc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 --2- 172.21.15.3:0/3938839809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f00afd60 0x7fb5f00674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 >> 172.21.15.3:0/3938839809 conn(0x7fb5f0060310 msgr2=0x7fb5f005be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:08.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 shutdown_connections 2022-07-21T01:34:08.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.017+0000 7fb5f4a3d700 1 -- 172.21.15.3:0/3938839809 wait complete. 2022-07-21T01:34:08.032 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:34:08.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:34:08.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca2f4a700 1 Processor -- start 2022-07-21T01:34:08.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca2f4a700 1 -- start start 2022-07-21T01:34:08.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca2f4a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbc9c0b2710 0x7fbc9c14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:08.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca2f4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b01a0 0x7fbc9c14b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:08.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca2f4a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c0a7250 con 0x7fbc9c0afd60 2022-07-21T01:34:08.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca2f4a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c0a7480 con 0x7fbc9c0b2710 2022-07-21T01:34:08.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca2f4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0a7720 con 0x7fbc9c0b01a0 2022-07-21T01:34:08.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca2749700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbc9c0afd60 0x7fbc9c0a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52472/0 (socket says 172.21.15.3:52472) 2022-07-21T01:34:08.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca2749700 1 -- 172.21.15.3:0/579557067 learned_addr learned my addr 172.21.15.3:0/579557067 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:08.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca1f48700 1 --2- 172.21.15.3:0/579557067 >> v2:172.21.15.143:3300/0 conn(0x7fbc9c0b2710 0x7fbc9c14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:08.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.129+0000 7fbca1747700 1 --2- 172.21.15.3:0/579557067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b01a0 0x7fbc9c14b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:08.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca0f46700 1 -- 172.21.15.3:0/579557067 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302916111 0 0) 0x7fbc9c0a7250 con 0x7fbc9c0afd60 2022-07-21T01:34:08.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca0f46700 1 -- 172.21.15.3:0/579557067 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc84003260 con 0x7fbc9c0afd60 2022-07-21T01:34:08.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca1747700 1 -- 172.21.15.3:0/579557067 >> v2:172.21.15.143:3300/0 conn(0x7fbc9c0b2710 msgr2=0x7fbc9c14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:08.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca1747700 1 --2- 172.21.15.3:0/579557067 >> v2:172.21.15.143:3300/0 conn(0x7fbc9c0b2710 0x7fbc9c14afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca1747700 1 -- 172.21.15.3:0/579557067 >> v1:172.21.15.3:6789/0 conn(0x7fbc9c0afd60 legacy=0x7fbc9c0a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:08.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca1747700 1 -- 172.21.15.3:0/579557067 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c0a79c0 con 0x7fbc9c0b01a0 2022-07-21T01:34:08.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca1747700 1 --2- 172.21.15.3:0/579557067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b01a0 0x7fbc9c14b4c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbc98002f70 tx=0x7fbc98006df0).ready entity=mon.2 client_cookie=5aa65997d3f01185 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:08.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca0f46700 1 -- 172.21.15.3:0/579557067 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc9800d4a0 con 0x7fbc9c0b01a0 2022-07-21T01:34:08.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca0f46700 1 -- 172.21.15.3:0/579557067 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc98000e90 con 0x7fbc9c0b01a0 2022-07-21T01:34:08.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca0f46700 1 -- 172.21.15.3:0/579557067 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc9800d7f0 con 0x7fbc9c0b01a0 2022-07-21T01:34:08.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca2f4a700 1 -- 172.21.15.3:0/579557067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b01a0 msgr2=0x7fbc9c14b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca2f4a700 1 --2- 172.21.15.3:0/579557067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b01a0 0x7fbc9c14b4c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbc98002f70 tx=0x7fbc98006df0).stop 2022-07-21T01:34:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca2f4a700 1 -- 172.21.15.3:0/579557067 shutdown_connections 2022-07-21T01:34:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca2f4a700 1 --2- 172.21.15.3:0/579557067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b01a0 0x7fbc9c14b4c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca2f4a700 1 --2- 172.21.15.3:0/579557067 >> v2:172.21.15.143:3300/0 conn(0x7fbc9c0b2710 0x7fbc9c14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca2f4a700 1 -- 172.21.15.3:0/579557067 >> 172.21.15.3:0/579557067 conn(0x7fbc9c060310 msgr2=0x7fbc9c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:08.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca2f4a700 1 -- 172.21.15.3:0/579557067 shutdown_connections 2022-07-21T01:34:08.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.133+0000 7fbca2f4a700 1 -- 172.21.15.3:0/579557067 wait complete. 2022-07-21T01:34:08.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca2f4a700 1 Processor -- start 2022-07-21T01:34:08.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca2f4a700 1 -- start start 2022-07-21T01:34:08.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca2f4a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc9c0afd60 0x7fbc9c0b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:08.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca2f4a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc9c0b01a0 0x7fbc9c159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:08.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca2f4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b2710 0x7fbc9c15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:08.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca2f4a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0aabe0 con 0x7fbc9c0afd60 2022-07-21T01:34:08.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca2f4a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0ac950 con 0x7fbc9c0b01a0 2022-07-21T01:34:08.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca2f4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0aae10 con 0x7fbc9c0b2710 2022-07-21T01:34:08.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca1f48700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc9c0afd60 0x7fbc9c0b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca1f48700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc9c0afd60 0x7fbc9c0b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44194/0 (socket says 172.21.15.3:44194) 2022-07-21T01:34:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca1f48700 1 -- 172.21.15.3:0/3881171954 learned_addr learned my addr 172.21.15.3:0/3881171954 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca2749700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b2710 0x7fbc9c15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca1747700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc9c0b01a0 0x7fbc9c159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca1f48700 1 -- 172.21.15.3:0/3881171954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b2710 msgr2=0x7fbc9c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca1f48700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b2710 0x7fbc9c15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca1f48700 1 -- 172.21.15.3:0/3881171954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc9c0b01a0 msgr2=0x7fbc9c159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca1f48700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc9c0b01a0 0x7fbc9c159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.137+0000 7fbca1f48700 1 -- 172.21.15.3:0/3881171954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9800b180 con 0x7fbc9c0afd60 2022-07-21T01:34:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbca1f48700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc9c0afd60 0x7fbc9c0b51d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fbc8c00c000 tx=0x7fbc8c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbc8affd700 1 -- 172.21.15.3:0/3881171954 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc8c011750 con 0x7fbc9c0afd60 2022-07-21T01:34:08.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c0ab0b0 con 0x7fbc9c0afd60 2022-07-21T01:34:08.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbc8affd700 1 -- 172.21.15.3:0/3881171954 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8c0118b0 con 0x7fbc9c0afd60 2022-07-21T01:34:08.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbc8affd700 1 -- 172.21.15.3:0/3881171954 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc8c011c10 con 0x7fbc9c0afd60 2022-07-21T01:34:08.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c0ac440 con 0x7fbc9c0afd60 2022-07-21T01:34:08.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c0b0880 con 0x7fbc9c0afd60 2022-07-21T01:34:08.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbc8affd700 1 -- 172.21.15.3:0/3881171954 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbc8c011d70 con 0x7fbc9c0afd60 2022-07-21T01:34:08.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbc8affd700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc7c0539e0 0x7fbc7c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:08.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbc8affd700 1 -- 172.21.15.3:0/3881171954 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fbc8c06c580 con 0x7fbc9c0afd60 2022-07-21T01:34:08.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbca1747700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc7c0539e0 0x7fbc7c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:08.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.141+0000 7fbca1747700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc7c0539e0 0x7fbc7c055ea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbc9801a000 tx=0x7fbc98014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:08.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.149+0000 7fbc8affd700 1 -- 172.21.15.3:0/3881171954 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc8c0432a0 con 0x7fbc9c0afd60 2022-07-21T01:34:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.309+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc9c0b5710 con 0x7fbc9c0afd60 2022-07-21T01:34:08.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbc8affd700 1 -- 172.21.15.3:0/3881171954 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2480 (secure 0 0 0) 0x7fbc8c0468c0 con 0x7fbc9c0afd60 2022-07-21T01:34:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:08.315 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:34:03.710770+0000","modified":"2022-07-21T01:34:04.713666+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4896":{"gid":4896,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/24695956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":24695956},{"type":"v1","addr":"172.21.15.3:6837","nonce":24695956}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-21T01:34:08.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc7c0539e0 msgr2=0x7fbc7c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc7c0539e0 0x7fbc7c055ea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbc9801a000 tx=0x7fbc98014000).stop 2022-07-21T01:34:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc9c0afd60 msgr2=0x7fbc9c0b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc9c0afd60 0x7fbc9c0b51d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fbc8c00c000 tx=0x7fbc8c006df0).stop 2022-07-21T01:34:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 shutdown_connections 2022-07-21T01:34:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc9c0b2710 0x7fbc9c15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc7c0539e0 0x7fbc7c055ea0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc9c0b01a0 0x7fbc9c159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 --2- 172.21.15.3:0/3881171954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc9c0afd60 0x7fbc9c0b51d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 >> 172.21.15.3:0/3881171954 conn(0x7fbc9c060310 msgr2=0x7fbc9c149fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 shutdown_connections 2022-07-21T01:34:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:08.313+0000 7fbca2f4a700 1 -- 172.21.15.3:0/3881171954 wait complete. 2022-07-21T01:34:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 12 2022-07-21T01:34:08.631 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:34:03.710770+0000', 'modified': '2022-07-21T01:34:04.713666+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4896}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4896': {'gid': 4896, 'name': 'a', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/24695956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 24695956}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 24695956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:34:08.631 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:34:08.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:34:08.751 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":4896,"want_state":"up:active","state":"up:active","rank_uptime":5.0266739249999999,"mdsmap_epoch":12,"osdmap_epoch":37,"osdmap_epoch_barrier":36,"uptime":11.938163506} 2022-07-21T01:34:08.752 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 4896, "mdsmap_epoch": 12, "osdmap_epoch": 37, "osdmap_epoch_barrier": 36, "rank_uptime": 5.026673925, "state": "up:active", "uptime": 11.938163506, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:34:08.753 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:34:08.753 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:34:08.754 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:34:08.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:34:08.756 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:34:08.766 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:34:08.824 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:34:08.825 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:34:08.835 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:34:08.836 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:34:08.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:34:08.837 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:34:08.843 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:34:08.844 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:34:08.848 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:34:08.864 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:34:08.865 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:34:08.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:34:08.921+0000 7f37df18f080 -1 init, newargv = 0x55e31fd7db80 newargc=9 2022-07-21T01:34:08.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22231]: starting ceph client 2022-07-21T01:34:08.922 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:34:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:34:08.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:34:08.925 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:34:08.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22231]: starting fuse 2022-07-21T01:34:08.931 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:34:08.931 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:34:08.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:34:08.932 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:34:08.933 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:34:08.933 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:34:08.934 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:34:08.934 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:34:08.934 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:34:08.935 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:34:08.935 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:34:08.936 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:34:08.936 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:34:08.936 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:34:08.937 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:34:08.937 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:34:08.938 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:34:08.938 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:34:08.938 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:34:08.939 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:34:08.939 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:34:08.940 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:34:08.940 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:34:08.941 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:34:08.941 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:34:08.941 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:34:09.013 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22231.asok 2022-07-21T01:34:09.018 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22231.asok 2022-07-21T01:34:09.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:34:09.019 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22231.asok status 2022-07-21T01:34:09.105 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:34:09.105 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:34:09.106 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:34:09.106 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:34:09.106 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:34:09.107 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:34:09.107 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:34:09.108 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22231", 2022-07-21T01:34:09.108 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:34:09.109 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:34:09.109 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:34:09.109 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:34:09.109 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 5089, 2022-07-21T01:34:09.109 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:34:09.110 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:34:09.110 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:34:09.110 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 5089 2022-07-21T01:34:09.110 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:34:09.111 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:34:09.111 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:34:09.111 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:34:09.111 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 1327889079 2022-07-21T01:34:09.111 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:34:09.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:34:09.112 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:34:09.112 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:34:09.112 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:34:09.112 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 1327889079 2022-07-21T01:34:09.113 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:34:09.113 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.5089 172.21.15.171:0/1327889079", 2022-07-21T01:34:09.113 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/1327889079", 2022-07-21T01:34:09.113 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:34:09.114 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 12, 2022-07-21T01:34:09.114 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 37, 2022-07-21T01:34:09.114 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:34:09.114 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:34:09.115 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:34:09.115 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:34:09.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:34:09.117 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:34:09.117 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:34:09.125 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:34:09.126 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:34:09.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:34:09.127 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:34:09.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_min_caps_per_client 2022-07-21T01:34:09.298 INFO:teuthology.orchestra.run.smithi003.stdout:{"mds_min_caps_per_client":"100"} 2022-07-21T01:34:09.299 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_min_caps_per_client": "100" } 2022-07-21T01:34:09.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:34:09.300 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:34:09.301 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:34:09.309 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:34:09.311 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:34:09.312 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-21T01:34:09.312 DEBUG:teuthology.orchestra.run.smithi171:> import sys 2022-07-21T01:34:09.313 DEBUG:teuthology.orchestra.run.smithi171:> import time 2022-07-21T01:34:09.313 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:34:09.314 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:34:09.314 DEBUG:teuthology.orchestra.run.smithi171:> n = 1000 2022-07-21T01:34:09.315 DEBUG:teuthology.orchestra.run.smithi171:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-21T01:34:09.315 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:34:09.315 DEBUG:teuthology.orchestra.run.smithi171:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-21T01:34:09.316 DEBUG:teuthology.orchestra.run.smithi171:> os.makedirs(os.path.dirname(abs_path)) 2022-07-21T01:34:09.316 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:34:09.317 DEBUG:teuthology.orchestra.run.smithi171:> for i in range(0, n): 2022-07-21T01:34:09.317 DEBUG:teuthology.orchestra.run.smithi171:> fname = "{0}_{1}".format(abs_path, i) 2022-07-21T01:34:09.318 DEBUG:teuthology.orchestra.run.smithi171:> with open(fname, '"'"'w'"'"') as f: 2022-07-21T01:34:09.319 DEBUG:teuthology.orchestra.run.smithi171:> f.write('"'"'content'"'"') 2022-07-21T01:34:09.319 DEBUG:teuthology.orchestra.run.smithi171:> if True: 2022-07-21T01:34:09.319 DEBUG:teuthology.orchestra.run.smithi171:> f.flush() 2022-07-21T01:34:09.320 DEBUG:teuthology.orchestra.run.smithi171:> os.fsync(f.fileno()) 2022-07-21T01:34:09.320 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:34:20.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:34:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f14990fc700 1 Processor -- start 2022-07-21T01:34:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f14990fc700 1 -- start start 2022-07-21T01:34:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f14990fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f14940b08a0 0x7f1494142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f14990fc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1494143ab0 con 0x7f14940b3260 2022-07-21T01:34:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f14990fc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14940b0c80 con 0x7f14940ad000 2022-07-21T01:34:20.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f14990fc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f149405cd30 con 0x7f14940b08a0 2022-07-21T01:34:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f14937fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f14940b3260 0x7f149405c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52490/0 (socket says 172.21.15.3:52490) 2022-07-21T01:34:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f14937fe700 1 -- 172.21.15.3:0/1156399376 learned_addr learned my addr 172.21.15.3:0/1156399376 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1493fff700 1 --2- 172.21.15.3:0/1156399376 >> v2:172.21.15.3:3300/0 conn(0x7f14940b08a0 0x7f1494142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1492ffd700 1 -- 172.21.15.3:0/1156399376 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482558875 0 0) 0x7f1494143ab0 con 0x7f14940b3260 2022-07-21T01:34:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1492ffd700 1 -- 172.21.15.3:0/1156399376 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1478003260 con 0x7f14940b3260 2022-07-21T01:34:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1493fff700 1 -- 172.21.15.3:0/1156399376 >> v1:172.21.15.143:6789/0 conn(0x7f14940ad000 legacy=0x7f14941433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1493fff700 1 -- 172.21.15.3:0/1156399376 >> v1:172.21.15.3:6789/0 conn(0x7f14940b3260 legacy=0x7f149405c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1493fff700 1 -- 172.21.15.3:0/1156399376 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14940ae810 con 0x7f14940b08a0 2022-07-21T01:34:20.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1493fff700 1 --2- 172.21.15.3:0/1156399376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940b08a0 0x7f1494142eb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f147c002f70 tx=0x7f147c006df0).ready entity=mon.0 client_cookie=bde5f51590e5ea57 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:20.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1492ffd700 1 -- 172.21.15.3:0/1156399376 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f147c010b50 con 0x7f14940b08a0 2022-07-21T01:34:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1492ffd700 1 -- 172.21.15.3:0/1156399376 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147c000e90 con 0x7f14940b08a0 2022-07-21T01:34:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.549+0000 7f1492ffd700 1 -- 172.21.15.3:0/1156399376 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f147c010ea0 con 0x7f14940b08a0 2022-07-21T01:34:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.553+0000 7f14990fc700 1 -- 172.21.15.3:0/1156399376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940b08a0 msgr2=0x7f1494142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.553+0000 7f14990fc700 1 --2- 172.21.15.3:0/1156399376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940b08a0 0x7f1494142eb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f147c002f70 tx=0x7f147c006df0).stop 2022-07-21T01:34:20.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.553+0000 7f14990fc700 1 -- 172.21.15.3:0/1156399376 shutdown_connections 2022-07-21T01:34:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.553+0000 7f14990fc700 1 --2- 172.21.15.3:0/1156399376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940b08a0 0x7f1494142eb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.553+0000 7f14990fc700 1 -- 172.21.15.3:0/1156399376 >> 172.21.15.3:0/1156399376 conn(0x7f14940612d0 msgr2=0x7f1494062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.553+0000 7f14990fc700 1 -- 172.21.15.3:0/1156399376 shutdown_connections 2022-07-21T01:34:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.553+0000 7f14990fc700 1 -- 172.21.15.3:0/1156399376 wait complete. 2022-07-21T01:34:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.553+0000 7f14990fc700 1 Processor -- start 2022-07-21T01:34:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.553+0000 7f14990fc700 1 -- start start 2022-07-21T01:34:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14990fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940ad000 0x7f14940abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14990fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14940b08a0 0x7f1494063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14990fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14940b3260 0x7f14941408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14990fc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1494067e30 con 0x7f14940ad000 2022-07-21T01:34:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14990fc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1494067b90 con 0x7f14940b3260 2022-07-21T01:34:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14990fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14940ac630 con 0x7f14940b08a0 2022-07-21T01:34:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f1493fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940ad000 0x7f14940abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f1493fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940ad000 0x7f14940abbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44214/0 (socket says 172.21.15.3:44214) 2022-07-21T01:34:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f1493fff700 1 -- 172.21.15.3:0/940458295 learned_addr learned my addr 172.21.15.3:0/940458295 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14937fe700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14940b08a0 0x7f1494063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14988fb700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14940b3260 0x7f14941408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f1493fff700 1 -- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14940b08a0 msgr2=0x7f1494063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f1493fff700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14940b08a0 0x7f1494063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f1493fff700 1 -- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14940b3260 msgr2=0x7f14941408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f1493fff700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14940b3260 0x7f14941408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f1493fff700 1 -- 172.21.15.3:0/940458295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c00b180 con 0x7f14940ad000 2022-07-21T01:34:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14988fb700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14940b3260 0x7f14941408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:34:20.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14937fe700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14940b08a0 0x7f1494063d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:34:20.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f1493fff700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940ad000 0x7f14940abbf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f147c002f70 tx=0x7f147c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:20.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14917fa700 1 -- 172.21.15.3:0/940458295 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f147c020070 con 0x7f14940ad000 2022-07-21T01:34:20.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14917fa700 1 -- 172.21.15.3:0/940458295 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147c014070 con 0x7f14940ad000 2022-07-21T01:34:20.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14940ac8d0 con 0x7f14940ad000 2022-07-21T01:34:20.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14917fa700 1 -- 172.21.15.3:0/940458295 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f147c019820 con 0x7f14940ad000 2022-07-21T01:34:20.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1494140de0 con 0x7f14940ad000 2022-07-21T01:34:20.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.557+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14940ab040 con 0x7f14940ad000 2022-07-21T01:34:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.561+0000 7f14917fa700 1 -- 172.21.15.3:0/940458295 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f147c02b070 con 0x7f14940ad000 2022-07-21T01:34:20.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.561+0000 7f14917fa700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f146c053b00 0x7f146c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.561+0000 7f14917fa700 1 -- 172.21.15.3:0/940458295 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f147c070990 con 0x7f14940ad000 2022-07-21T01:34:20.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.561+0000 7f14937fe700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f146c053b00 0x7f146c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.561+0000 7f14937fe700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f146c053b00 0x7f146c055fc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f148c008000 tx=0x7f148c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.565+0000 7f14917fa700 1 -- 172.21.15.3:0/940458295 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f147c047790 con 0x7f14940ad000 2022-07-21T01:34:20.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14940574e0 con 0x7f14940ad000 2022-07-21T01:34:20.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14917fa700 1 -- 172.21.15.3:0/940458295 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2480 (secure 0 0 0) 0x7f147c04ad70 con 0x7f14940ad000 2022-07-21T01:34:20.730 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:20.731 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:34:03.710770+0000","modified":"2022-07-21T01:34:04.713666+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4896":{"gid":4896,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/24695956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":24695956},{"type":"v1","addr":"172.21.15.3:6837","nonce":24695956}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-21T01:34:20.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f146c053b00 msgr2=0x7f146c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f146c053b00 0x7f146c055fc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f148c008000 tx=0x7f148c009040).stop 2022-07-21T01:34:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940ad000 msgr2=0x7f14940abbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940ad000 0x7f14940abbf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f147c002f70 tx=0x7f147c016000).stop 2022-07-21T01:34:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 shutdown_connections 2022-07-21T01:34:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f146c053b00 0x7f146c055fc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14940b3260 0x7f14941408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14940b08a0 0x7f1494063d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 --2- 172.21.15.3:0/940458295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14940ad000 0x7f14940abbf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 >> 172.21.15.3:0/940458295 conn(0x7f14940612d0 msgr2=0x7f14940600c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 shutdown_connections 2022-07-21T01:34:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.729+0000 7f14990fc700 1 -- 172.21.15.3:0/940458295 wait complete. 2022-07-21T01:34:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 12 2022-07-21T01:34:20.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-21T01:34:20.844 INFO:teuthology.orchestra.run.smithi003.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-21T01:34:20.845 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-21T01:34:20.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:34:20.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.941+0000 7f888b418700 1 Processor -- start 2022-07-21T01:34:20.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.941+0000 7f888b418700 1 -- start start 2022-07-21T01:34:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.941+0000 7f888b418700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8884066780 0x7f8884066b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.941+0000 7f888b418700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 0x7f888405ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888405f8e0 con 0x7f8884140e40 2022-07-21T01:34:20.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88840b1630 con 0x7f8884066780 2022-07-21T01:34:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888405fb10 con 0x7f88840b1bc0 2022-07-21T01:34:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888a416700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8884066780 0x7f8884066b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888a416700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8884066780 0x7f8884066b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:49670/0 (socket says 172.21.15.3:49670) 2022-07-21T01:34:20.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f8889c15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 0x7f888405ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888a416700 1 -- 172.21.15.3:0/2629491955 learned_addr learned my addr 172.21.15.3:0/2629491955 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888ac17700 1 --1- 172.21.15.3:0/2629491955 >> v1:172.21.15.143:6789/0 conn(0x7f8884140e40 0x7f888413e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:54620/0 (socket says 172.21.15.3:54620) 2022-07-21T01:34:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888a416700 1 -- 172.21.15.3:0/2629491955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 msgr2=0x7f888405ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888a416700 1 --2- 172.21.15.3:0/2629491955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 0x7f888405ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888a416700 1 -- 172.21.15.3:0/2629491955 >> v1:172.21.15.143:6789/0 conn(0x7f8884140e40 legacy=0x7f888413e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888a416700 1 -- 172.21.15.3:0/2629491955 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88840604a0 con 0x7f8884066780 2022-07-21T01:34:20.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888a416700 1 --2- 172.21.15.3:0/2629491955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884066780 0x7f8884066b60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f887c001490 tx=0x7f8880006eb0).ready entity=mon.1 client_cookie=a3c7dd561e0bb20c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:20.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f8889414700 1 -- 172.21.15.3:0/2629491955 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f888000e540 con 0x7f8884066780 2022-07-21T01:34:20.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f8889414700 1 -- 172.21.15.3:0/2629491955 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880000e90 con 0x7f8884066780 2022-07-21T01:34:20.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f8889414700 1 -- 172.21.15.3:0/2629491955 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f888000e8d0 con 0x7f8884066780 2022-07-21T01:34:20.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 -- 172.21.15.3:0/2629491955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884066780 msgr2=0x7f8884066b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 --2- 172.21.15.3:0/2629491955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884066780 0x7f8884066b60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f887c001490 tx=0x7f8880006eb0).stop 2022-07-21T01:34:20.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 -- 172.21.15.3:0/2629491955 shutdown_connections 2022-07-21T01:34:20.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 --2- 172.21.15.3:0/2629491955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 0x7f888405ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 --2- 172.21.15.3:0/2629491955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884066780 0x7f8884066b60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 -- 172.21.15.3:0/2629491955 >> 172.21.15.3:0/2629491955 conn(0x7f8884063870 msgr2=0x7f8884063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:20.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 -- 172.21.15.3:0/2629491955 shutdown_connections 2022-07-21T01:34:20.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.945+0000 7f888b418700 1 -- 172.21.15.3:0/2629491955 wait complete. 2022-07-21T01:34:20.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 Processor -- start 2022-07-21T01:34:20.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 -- start start 2022-07-21T01:34:20.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8884066780 0x7f88840afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 0x7f8884065710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884140e40 0x7f88840a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88840b0550 con 0x7f8884066780 2022-07-21T01:34:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8884065db0 con 0x7f8884140e40 2022-07-21T01:34:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888409eb50 con 0x7f88840b1bc0 2022-07-21T01:34:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888a416700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8884066780 0x7f88840afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888a416700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8884066780 0x7f88840afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44228/0 (socket says 172.21.15.3:44228) 2022-07-21T01:34:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888a416700 1 -- 172.21.15.3:0/2815285759 learned_addr learned my addr 172.21.15.3:0/2815285759 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f8889c15700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 0x7f8884065710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888ac17700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884140e40 0x7f88840a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888a416700 1 -- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 msgr2=0x7f8884065710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888a416700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 0x7f8884065710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888a416700 1 -- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884140e40 msgr2=0x7f88840a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:20.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888a416700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884140e40 0x7f88840a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:20.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888a416700 1 -- 172.21.15.3:0/2815285759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888000b1e0 con 0x7f8884066780 2022-07-21T01:34:20.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888ac17700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884140e40 0x7f88840a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:34:20.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f8889c15700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 0x7f8884065710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:34:20.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888a416700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8884066780 0x7f88840afc30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f88840a1370 tx=0x7f8880015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:20.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f88737fe700 1 -- 172.21.15.3:0/2815285759 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88800116e0 con 0x7f8884066780 2022-07-21T01:34:20.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f88737fe700 1 -- 172.21.15.3:0/2815285759 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880011840 con 0x7f8884066780 2022-07-21T01:34:20.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888409edf0 con 0x7f8884066780 2022-07-21T01:34:20.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f88737fe700 1 -- 172.21.15.3:0/2815285759 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8880011ba0 con 0x7f8884066780 2022-07-21T01:34:20.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88840a0010 con 0x7f8884066780 2022-07-21T01:34:20.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.949+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88840b18d0 con 0x7f8884066780 2022-07-21T01:34:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.953+0000 7f88737fe700 1 -- 172.21.15.3:0/2815285759 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8880025030 con 0x7f8884066780 2022-07-21T01:34:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.953+0000 7f88737fe700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8864053a10 0x7f8864055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.953+0000 7f88737fe700 1 -- 172.21.15.3:0/2815285759 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f888006d250 con 0x7f8884066780 2022-07-21T01:34:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.953+0000 7f8889c15700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8864053a10 0x7f8864055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.953+0000 7f8889c15700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8864053a10 0x7f8864055ed0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f887400b000 tx=0x7f8874008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:20.953+0000 7f88737fe700 1 -- 172.21.15.3:0/2815285759 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8880044010 con 0x7f8884066780 2022-07-21T01:34:21.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88840574e0 con 0x7f8884066780 2022-07-21T01:34:21.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f88737fe700 1 -- 172.21.15.3:0/2815285759 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2480 (secure 0 0 0) 0x7f8880023020 con 0x7f8884066780 2022-07-21T01:34:21.115 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:21.116 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:34:03.710770+0000","modified":"2022-07-21T01:34:04.713666+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4896":{"gid":4896,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/24695956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":24695956},{"type":"v1","addr":"172.21.15.3:6837","nonce":24695956}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-21T01:34:21.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8864053a10 msgr2=0x7f8864055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8864053a10 0x7f8864055ed0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f887400b000 tx=0x7f8874008040).stop 2022-07-21T01:34:21.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8884066780 msgr2=0x7f88840afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8884066780 0x7f88840afc30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f88840a1370 tx=0x7f8880015000).stop 2022-07-21T01:34:21.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 shutdown_connections 2022-07-21T01:34:21.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8864053a10 0x7f8864055ed0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8884140e40 0x7f88840a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840b1bc0 0x7f8884065710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 --2- 172.21.15.3:0/2815285759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8884066780 0x7f88840afc30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 >> 172.21.15.3:0/2815285759 conn(0x7f8884063870 msgr2=0x7f8884140310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:21.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 shutdown_connections 2022-07-21T01:34:21.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.113+0000 7f888b418700 1 -- 172.21.15.3:0/2815285759 wait complete. 2022-07-21T01:34:21.124 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 12 2022-07-21T01:34:21.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-21T01:34:21.233 INFO:teuthology.orchestra.run.smithi003.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-21T01:34:21.234 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-21T01:34:21.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:34:21.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77b1ccd700 1 Processor -- start 2022-07-21T01:34:21.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77b1ccd700 1 -- start start 2022-07-21T01:34:21.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77b1ccd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f77ac0b1f40 0x7f77ac0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77b1ccd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f77ac0a7250 0x7f77ac1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77b1ccd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac0b04e0 con 0x7f77ac148530 2022-07-21T01:34:21.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77b1ccd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac0a9250 con 0x7f77ac0a7250 2022-07-21T01:34:21.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77b1ccd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac0b0780 con 0x7f77ac0b1f40 2022-07-21T01:34:21.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77abfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f77ac0a7250 0x7f77ac1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77aaffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f77ac148530 0x7f77ac0b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52516/0 (socket says 172.21.15.3:52516) 2022-07-21T01:34:21.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77aaffd700 1 -- 172.21.15.3:0/2008986089 learned_addr learned my addr 172.21.15.3:0/2008986089 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77ab7fe700 1 --2- 172.21.15.3:0/2008986089 >> v2:172.21.15.143:3300/0 conn(0x7f77ac0b1f40 0x7f77ac0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77aa7fc700 1 -- 172.21.15.3:0/2008986089 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374166798 0 0) 0x7f77ac0b04e0 con 0x7f77ac148530 2022-07-21T01:34:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77aa7fc700 1 -- 172.21.15.3:0/2008986089 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7798003260 con 0x7f77ac148530 2022-07-21T01:34:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77abfff700 1 -- 172.21.15.3:0/2008986089 >> v2:172.21.15.143:3300/0 conn(0x7f77ac0b1f40 msgr2=0x7f77ac0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77abfff700 1 --2- 172.21.15.3:0/2008986089 >> v2:172.21.15.143:3300/0 conn(0x7f77ac0b1f40 0x7f77ac0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77abfff700 1 -- 172.21.15.3:0/2008986089 >> v1:172.21.15.3:6789/0 conn(0x7f77ac148530 legacy=0x7f77ac0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77aa7fc700 1 -- 172.21.15.3:0/2008986089 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 960238025 0 0) 0x7f7798003260 con 0x7f77ac148530 2022-07-21T01:34:21.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77abfff700 1 -- 172.21.15.3:0/2008986089 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac0b0a20 con 0x7f77ac0a7250 2022-07-21T01:34:21.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77abfff700 1 --2- 172.21.15.3:0/2008986089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 0x7f77ac1448b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f77ac1478c0 tx=0x7f77a4006df0).ready entity=mon.0 client_cookie=475e1441c590af3c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77aa7fc700 1 -- 172.21.15.3:0/2008986089 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f779c0019e0 con 0x7f77ac0a7250 2022-07-21T01:34:21.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77aa7fc700 1 -- 172.21.15.3:0/2008986089 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a4000e90 con 0x7f77ac0a7250 2022-07-21T01:34:21.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.337+0000 7f77aa7fc700 1 -- 172.21.15.3:0/2008986089 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77a4012460 con 0x7f77ac0a7250 2022-07-21T01:34:21.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- 172.21.15.3:0/2008986089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 msgr2=0x7f77ac1448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 --2- 172.21.15.3:0/2008986089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 0x7f77ac1448b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f77ac1478c0 tx=0x7f77a4006df0).stop 2022-07-21T01:34:21.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- 172.21.15.3:0/2008986089 shutdown_connections 2022-07-21T01:34:21.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 --2- 172.21.15.3:0/2008986089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 0x7f77ac1448b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 --2- 172.21.15.3:0/2008986089 >> v2:172.21.15.143:3300/0 conn(0x7f77ac0b1f40 0x7f77ac0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- 172.21.15.3:0/2008986089 >> 172.21.15.3:0/2008986089 conn(0x7f77ac05f840 msgr2=0x7f77ac05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:21.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- 172.21.15.3:0/2008986089 shutdown_connections 2022-07-21T01:34:21.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- 172.21.15.3:0/2008986089 wait complete. 2022-07-21T01:34:21.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 Processor -- start 2022-07-21T01:34:21.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- start start 2022-07-21T01:34:21.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 0x7f77ac146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac0b1f40 0x7f77ac15a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac148530 0x7f77ac143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac15ad00 con 0x7f77ac0a7250 2022-07-21T01:34:21.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac14a180 con 0x7f77ac0b1f40 2022-07-21T01:34:21.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac14a420 con 0x7f77ac148530 2022-07-21T01:34:21.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77ab7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 0x7f77ac146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77ab7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 0x7f77ac146240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44244/0 (socket says 172.21.15.3:44244) 2022-07-21T01:34:21.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77ab7fe700 1 -- 172.21.15.3:0/3238687698 learned_addr learned my addr 172.21.15.3:0/3238687698 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:21.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77abfff700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac148530 0x7f77ac143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77aaffd700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac0b1f40 0x7f77ac15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77ab7fe700 1 -- 172.21.15.3:0/3238687698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac148530 msgr2=0x7f77ac143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77ab7fe700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac148530 0x7f77ac143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77ab7fe700 1 -- 172.21.15.3:0/3238687698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac0b1f40 msgr2=0x7f77ac15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77ab7fe700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac0b1f40 0x7f77ac15a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77ab7fe700 1 -- 172.21.15.3:0/3238687698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a400b180 con 0x7f77ac0a7250 2022-07-21T01:34:21.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77ab7fe700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 0x7f77ac146240 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f77a000c000 tx=0x7f77a0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77a8ff9700 1 -- 172.21.15.3:0/3238687698 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77a0010780 con 0x7f77ac0a7250 2022-07-21T01:34:21.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77a8ff9700 1 -- 172.21.15.3:0/3238687698 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a00108e0 con 0x7f77ac0a7250 2022-07-21T01:34:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77a8ff9700 1 -- 172.21.15.3:0/3238687698 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77a0010c40 con 0x7f77ac0a7250 2022-07-21T01:34:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.341+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ac14a6c0 con 0x7f77ac0a7250 2022-07-21T01:34:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.345+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac145050 con 0x7f77ac0a7250 2022-07-21T01:34:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.345+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ac0b0cc0 con 0x7f77ac0a7250 2022-07-21T01:34:21.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.345+0000 7f77a8ff9700 1 -- 172.21.15.3:0/3238687698 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f77a001c460 con 0x7f77ac0a7250 2022-07-21T01:34:21.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.345+0000 7f77a8ff9700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f778c053b00 0x7f778c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.345+0000 7f77a8ff9700 1 -- 172.21.15.3:0/3238687698 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f77a006c400 con 0x7f77ac0a7250 2022-07-21T01:34:21.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.345+0000 7f77aaffd700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f778c053b00 0x7f778c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.349+0000 7f77aaffd700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f778c053b00 0x7f778c055fc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f779c00d000 tx=0x7f779c002060).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.353+0000 7f77a8ff9700 1 -- 172.21.15.3:0/3238687698 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f77a0043160 con 0x7f77ac0a7250 2022-07-21T01:34:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.513+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77ac0574e0 con 0x7f77ac0a7250 2022-07-21T01:34:21.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.513+0000 7f77a8ff9700 1 -- 172.21.15.3:0/3238687698 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2480 (secure 0 0 0) 0x7f77a0046780 con 0x7f77ac0a7250 2022-07-21T01:34:21.516 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:34:21.516 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:34:03.710770+0000","modified":"2022-07-21T01:34:04.713666+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4896":{"gid":4896,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/24695956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":24695956},{"type":"v1","addr":"172.21.15.3:6837","nonce":24695956}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-21T01:34:21.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f778c053b00 msgr2=0x7f778c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f778c053b00 0x7f778c055fc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f779c00d000 tx=0x7f779c002060).stop 2022-07-21T01:34:21.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 msgr2=0x7f77ac146240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 0x7f77ac146240 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f77a000c000 tx=0x7f77a0006df0).stop 2022-07-21T01:34:21.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 shutdown_connections 2022-07-21T01:34:21.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac148530 0x7f77ac143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f778c053b00 0x7f778c055fc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac0b1f40 0x7f77ac15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 --2- 172.21.15.3:0/3238687698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77ac0a7250 0x7f77ac146240 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 >> 172.21.15.3:0/3238687698 conn(0x7f77ac05f840 msgr2=0x7f77ac0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:21.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 shutdown_connections 2022-07-21T01:34:21.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.517+0000 7f77b1ccd700 1 -- 172.21.15.3:0/3238687698 wait complete. 2022-07-21T01:34:21.522 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 12 2022-07-21T01:34:21.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 2022-07-21T01:34:21.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.633+0000 7fdaaf0b3700 1 Processor -- start 2022-07-21T01:34:21.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.633+0000 7fdaaf0b3700 1 -- start start 2022-07-21T01:34:21.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa80b5b20 0x7fdaa80b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa81440f0 con 0x7fdaa80b66f0 2022-07-21T01:34:21.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa8060170 con 0x7fdaa805fd90 2022-07-21T01:34:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa80ba840 con 0x7fdaa80b5b20 2022-07-21T01:34:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaa7fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdaa80b66f0 0x7fdaa80b4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52532/0 (socket says 172.21.15.3:52532) 2022-07-21T01:34:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaa7fff700 1 -- 172.21.15.3:0/3426830701 learned_addr learned my addr 172.21.15.3:0/3426830701 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:34:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaad650700 1 --2- 172.21.15.3:0/3426830701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa80b5b20 0x7fdaa80b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaa77fe700 1 -- 172.21.15.3:0/3426830701 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035357862 0 0) 0x7fdaa81440f0 con 0x7fdaa80b66f0 2022-07-21T01:34:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaa77fe700 1 -- 172.21.15.3:0/3426830701 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda94003260 con 0x7fdaa80b66f0 2022-07-21T01:34:21.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaad650700 1 -- 172.21.15.3:0/3426830701 >> v1:172.21.15.143:6789/0 conn(0x7fdaa805fd90 legacy=0x7fdaa80ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaad650700 1 -- 172.21.15.3:0/3426830701 >> v1:172.21.15.3:6789/0 conn(0x7fdaa80b66f0 legacy=0x7fdaa80b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaa77fe700 1 -- 172.21.15.3:0/3426830701 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2446007409 0 0) 0x7fda94003260 con 0x7fdaa80b66f0 2022-07-21T01:34:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaad650700 1 -- 172.21.15.3:0/3426830701 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa8143be0 con 0x7fdaa80b5b20 2022-07-21T01:34:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaad650700 1 --2- 172.21.15.3:0/3426830701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa80b5b20 0x7fdaa80b4340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fda9c001800 tx=0x7fdaa0006df0).ready entity=mon.2 client_cookie=f1f29b1515832f9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaa77fe700 1 -- 172.21.15.3:0/3426830701 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9c001660 con 0x7fdaa80b5b20 2022-07-21T01:34:21.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaa77fe700 1 -- 172.21.15.3:0/3426830701 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa0000e90 con 0x7fdaa80b5b20 2022-07-21T01:34:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaa77fe700 1 -- 172.21.15.3:0/3426830701 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdaa000d4a0 con 0x7fdaa80b5b20 2022-07-21T01:34:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3426830701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa80b5b20 msgr2=0x7fdaa80b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 --2- 172.21.15.3:0/3426830701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa80b5b20 0x7fdaa80b4340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fda9c001800 tx=0x7fdaa0006df0).stop 2022-07-21T01:34:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3426830701 shutdown_connections 2022-07-21T01:34:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 --2- 172.21.15.3:0/3426830701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa80b5b20 0x7fdaa80b4340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3426830701 >> 172.21.15.3:0/3426830701 conn(0x7fdaa8062530 msgr2=0x7fdaa8066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:21.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3426830701 shutdown_connections 2022-07-21T01:34:21.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.637+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3426830701 wait complete. 2022-07-21T01:34:21.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaaf0b3700 1 Processor -- start 2022-07-21T01:34:21.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaaf0b3700 1 -- start start 2022-07-21T01:34:21.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaaf0b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa805fd90 0x7fdaa80b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaaf0b3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaa80b5b20 0x7fdaa8152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaaf0b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa80b66f0 0x7fdaa81562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaaf0b3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa812fb00 con 0x7fdaa80b5b20 2022-07-21T01:34:21.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaaf0b3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa812fd30 con 0x7fdaa80b66f0 2022-07-21T01:34:21.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaaf0b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa812ffd0 con 0x7fdaa805fd90 2022-07-21T01:34:21.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaa80b5b20 0x7fdaa8152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaa80b5b20 0x7fdaa8152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44258/0 (socket says 172.21.15.3:44258) 2022-07-21T01:34:21.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa7fff700 1 -- 172.21.15.3:0/3586556698 learned_addr learned my addr 172.21.15.3:0/3586556698 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:21.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaace4f700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa805fd90 0x7fdaa80b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaad650700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa80b66f0 0x7fdaa81562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa7fff700 1 -- 172.21.15.3:0/3586556698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa805fd90 msgr2=0x7fdaa80b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa805fd90 0x7fdaa80b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa7fff700 1 -- 172.21.15.3:0/3586556698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa80b66f0 msgr2=0x7fdaa81562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa80b66f0 0x7fdaa81562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa7fff700 1 -- 172.21.15.3:0/3586556698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa000b180 con 0x7fdaa80b5b20 2022-07-21T01:34:21.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaad650700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa80b66f0 0x7fdaa81562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:34:21.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaa80b5b20 0x7fdaa8152520 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fda9c0018c0 tx=0x7fda9c001e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.641+0000 7fdaa5ffb700 1 -- 172.21.15.3:0/3586556698 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9c00ce60 con 0x7fdaa80b5b20 2022-07-21T01:34:21.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.645+0000 7fdaa5ffb700 1 -- 172.21.15.3:0/3586556698 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c007170 con 0x7fdaa80b5b20 2022-07-21T01:34:21.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.645+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3586556698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa8155b90 con 0x7fdaa80b5b20 2022-07-21T01:34:21.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.645+0000 7fdaa5ffb700 1 -- 172.21.15.3:0/3586556698 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9c004730 con 0x7fdaa80b5b20 2022-07-21T01:34:21.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.645+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3586556698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa812e0c0 con 0x7fdaa80b5b20 2022-07-21T01:34:21.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.645+0000 7fdaa5ffb700 1 -- 172.21.15.3:0/3586556698 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fda9c01b070 con 0x7fdaa80b5b20 2022-07-21T01:34:21.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.645+0000 7fdaa5ffb700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda88053a70 0x7fda88055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.645+0000 7fdaa5ffb700 1 -- 172.21.15.3:0/3586556698 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fda9c06b610 con 0x7fdaa80b5b20 2022-07-21T01:34:21.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.645+0000 7fdaace4f700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda88053a70 0x7fda88055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda88053a70 0x7fda88055f30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fda9800d000 tx=0x7fda98007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdab930a740 1 Processor -- start 2022-07-21T01:34:21.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdab930a740 1 -- start start 2022-07-21T01:34:21.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdab930a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdab930a740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd43dc0 0xd44240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdab930a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd7de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdab930a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0xcb9010 con 0xd45c10 2022-07-21T01:34:21.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdab930a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0xc99250 con 0xd7da00 2022-07-21T01:34:21.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdab930a740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0xd77740 con 0xd43dc0 2022-07-21T01:34:21.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd438d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44264/0 (socket says 172.21.15.3:44264) 2022-07-21T01:34:21.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 -- 172.21.15.3:0/3781985315 learned_addr learned my addr 172.21.15.3:0/3781985315 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:21.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd43dc0 0xd44240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaad650700 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd7de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 -- 172.21.15.3:0/3781985315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd43dc0 msgr2=0xd44240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:21.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd43dc0 0xd44240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 -- 172.21.15.3:0/3781985315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 msgr2=0xd7de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:21.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd7de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 -- 172.21.15.3:0/3781985315 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xcf1030 con 0xd45c10 2022-07-21T01:34:21.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaad650700 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd7de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:34:21.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaace4f700 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd438d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fdaa8061830 tx=0x7fda98009000).ready entity=mon.0 client_cookie=cafaca2001f02df2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3781985315 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9800e750 con 0xd45c10 2022-07-21T01:34:21.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3781985315 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9800e8b0 con 0xd45c10 2022-07-21T01:34:21.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.649+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3781985315 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9800ec10 con 0xd45c10 2022-07-21T01:34:21.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- 172.21.15.3:0/3781985315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 msgr2=0xd438d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd438d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fdaa8061830 tx=0x7fda98009000).stop 2022-07-21T01:34:21.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- 172.21.15.3:0/3781985315 shutdown_connections 2022-07-21T01:34:21.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd7de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd43dc0 0xd44240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 --2- 172.21.15.3:0/3781985315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd438d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- 172.21.15.3:0/3781985315 >> 172.21.15.3:0/3781985315 conn(0xd46430 msgr2=0xd451a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:21.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- 172.21.15.3:0/3781985315 shutdown_connections 2022-07-21T01:34:21.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- 172.21.15.3:0/3781985315 wait complete. 2022-07-21T01:34:21.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- start start 2022-07-21T01:34:21.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 Processor -- start 2022-07-21T01:34:21.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xd8c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd45c10 0xd8cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 0xd8fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0xd8f7f0 con 0xd43dc0 2022-07-21T01:34:21.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0xd8d110 con 0xd45c10 2022-07-21T01:34:21.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0xd81280 con 0xd7da00 2022-07-21T01:34:21.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 10 client.0 ll_register_callbacks cb 0xd00610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-21T01:34:21.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaace4f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xd8c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaace4f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xd8c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44270/0 (socket says 172.21.15.3:44270) 2022-07-21T01:34:21.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaace4f700 1 -- 172.21.15.3:0/3611351959 learned_addr learned my addr 172.21.15.3:0/3611351959 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:21.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd45c10 0xd8cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaad650700 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 0xd8fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaace4f700 1 -- 172.21.15.3:0/3611351959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 msgr2=0xd8fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:21.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaace4f700 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 0xd8fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaace4f700 1 -- 172.21.15.3:0/3611351959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd45c10 msgr2=0xd8cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:21.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaace4f700 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd45c10 0xd8cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaace4f700 1 -- 172.21.15.3:0/3611351959 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9800f040 con 0xd43dc0 2022-07-21T01:34:21.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdaace4f700 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xd8c730 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fdaa8061830 tx=0x7fda98016000).ready entity=mon.0 client_cookie=196129d18e7ca6be server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fda877fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2022-07-21T01:34:21.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fda877fe700 1 -- 172.21.15.3:0/3611351959 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda98002030 con 0xd43dc0 2022-07-21T01:34:21.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fda877fe700 1 -- 172.21.15.3:0/3611351959 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fda98002190 con 0xd43dc0 2022-07-21T01:34:21.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- 172.21.15.3:0/3611351959 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xcaefc0 con 0xd43dc0 2022-07-21T01:34:21.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fda877fe700 1 -- 172.21.15.3:0/3611351959 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda98002350 con 0xd43dc0 2022-07-21T01:34:21.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 10 client.5124 fetch_fsmap learned FSMap version 12 2022-07-21T01:34:21.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- 172.21.15.3:0/3611351959 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0xc8b9b0 con 0xd43dc0 2022-07-21T01:34:21.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fda877fe700 1 -- 172.21.15.3:0/3611351959 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda98033070 con 0xd43dc0 2022-07-21T01:34:21.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fda877fe700 1 -- 172.21.15.3:0/3611351959 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 12) v1 ==== 1434+0+0 (secure 0 0 0) 0x7fda98001c80 con 0xd43dc0 2022-07-21T01:34:21.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 10 client.5124 fetch_fsmap finished waiting for FSMap version 12 2022-07-21T01:34:21.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 10 client.5124 resolve_mds: resolved ID 'a' to GID 4896 2022-07-21T01:34:21.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 20 client.5124 populate_metadata read hostname 'smithi003' 2022-07-21T01:34:21.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] conn(0xd49770 0xd46e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 4 client.5124 mds_command: new command op to 4896 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-21T01:34:21.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.653+0000 7fdab930a740 1 -- 172.21.15.3:0/3611351959 --> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xca7040 con 0xd49770 2022-07-21T01:34:21.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] conn(0xd49770 0xd46e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fda877fe700 10 client.5124 ms_handle_connect on v2:172.21.15.3:6836/24695956 2022-07-21T01:34:21.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] conn(0xd49770 0xd46e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=95237393406647a9 server_cookie=752e9f0fbae0eb12 in_seq=0 out_seq=0 2022-07-21T01:34:21.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fda877fe700 1 -- 172.21.15.3:0/3611351959 <== mds.0 v2:172.21.15.3:6836/24695956 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0xca7040 con 0xd49770 2022-07-21T01:34:21.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fda877fe700 10 client.5124 handle_command_reply: tid=0 2022-07-21T01:34:21.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 client.5124 shutdown 2022-07-21T01:34:21.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 -- 172.21.15.3:0/3611351959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 msgr2=0xd8c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xd8c730 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fdaa8061830 tx=0x7fda98016000).stop 2022-07-21T01:34:21.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 -- 172.21.15.3:0/3611351959 shutdown_connections 2022-07-21T01:34:21.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 0xd8fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd45c10 0xd8cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] conn(0xd49770 0xd46e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 --2- 172.21.15.3:0/3611351959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xd8c730 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fda877fe700 0 client.5124 ms_handle_reset on v2:172.21.15.3:6836/24695956 2022-07-21T01:34:21.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 -- 172.21.15.3:0/3611351959 >> 172.21.15.3:0/3611351959 conn(0xd46430 msgr2=0xd7de60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:21.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 -- 172.21.15.3:0/3611351959 shutdown_connections 2022-07-21T01:34:21.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 1 -- 172.21.15.3:0/3611351959 wait complete. 2022-07-21T01:34:21.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.657+0000 7fdab930a740 20 client.5124 trim_cache size 0 max 16384 2022-07-21T01:34:21.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 Processor -- start 2022-07-21T01:34:21.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- start start 2022-07-21T01:34:21.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd43dc0 0xd7fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd44ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 0xd453b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0xca7040 con 0xd45c10 2022-07-21T01:34:21.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0xd7bc70 con 0xd43dc0 2022-07-21T01:34:21.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0xd17470 con 0xd7da00 2022-07-21T01:34:21.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaa7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd44ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaa7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd44ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44280/0 (socket says 172.21.15.3:44280) 2022-07-21T01:34:21.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaa7fff700 1 -- 172.21.15.3:0/2589391970 learned_addr learned my addr 172.21.15.3:0/2589391970 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:21.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaace4f700 1 --2- 172.21.15.3:0/2589391970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd43dc0 0xd7fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaad650700 1 --2- 172.21.15.3:0/2589391970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 0xd453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaa7fff700 1 -- 172.21.15.3:0/2589391970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 msgr2=0xd453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:21.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/2589391970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 0xd453b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaa7fff700 1 -- 172.21.15.3:0/2589391970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd43dc0 msgr2=0xd7fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:21.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/2589391970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd43dc0 0xd7fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaa7fff700 1 -- 172.21.15.3:0/2589391970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xca85b0 con 0xd45c10 2022-07-21T01:34:21.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/2589391970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd44ec0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0xd7d9c0 tx=0x7fda9c06b040).ready entity=mon.0 client_cookie=8e52a1cf6cb25203 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fda8dffb700 1 -- 172.21.15.3:0/2589391970 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9c050040 con 0xd45c10 2022-07-21T01:34:21.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fda8dffb700 1 -- 172.21.15.3:0/2589391970 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c069b40 con 0xd45c10 2022-07-21T01:34:21.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fda8dffb700 1 -- 172.21.15.3:0/2589391970 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9c054280 con 0xd45c10 2022-07-21T01:34:21.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- 172.21.15.3:0/2589391970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 msgr2=0xd44ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:34:21.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 --2- 172.21.15.3:0/2589391970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd44ec0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0xd7d9c0 tx=0x7fda9c06b040).stop 2022-07-21T01:34:21.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- 172.21.15.3:0/2589391970 shutdown_connections 2022-07-21T01:34:21.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 --2- 172.21.15.3:0/2589391970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd7da00 0xd453b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 --2- 172.21.15.3:0/2589391970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd45c10 0xd44ec0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 --2- 172.21.15.3:0/2589391970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd43dc0 0xd7fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- 172.21.15.3:0/2589391970 >> 172.21.15.3:0/2589391970 conn(0xd46430 msgr2=0xd4b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:34:21.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- 172.21.15.3:0/2589391970 shutdown_connections 2022-07-21T01:34:21.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- 172.21.15.3:0/2589391970 wait complete. 2022-07-21T01:34:21.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 -- start start 2022-07-21T01:34:21.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.673+0000 7fdab930a740 1 Processor -- start 2022-07-21T01:34:21.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xdd8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd45c10 0xdd94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd46eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0xd84120 con 0xd43dc0 2022-07-21T01:34:21.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0xd89ea0 con 0xd7da00 2022-07-21T01:34:21.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0xd854d0 con 0xd45c10 2022-07-21T01:34:21.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 10 client.0 ll_register_callbacks cb 0xd49be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-21T01:34:21.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaace4f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xdd8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaace4f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xdd8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44284/0 (socket says 172.21.15.3:44284) 2022-07-21T01:34:21.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaace4f700 1 -- 172.21.15.3:0/2582579525 learned_addr learned my addr 172.21.15.3:0/2582579525 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:34:21.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaad650700 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd46eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd45c10 0xdd94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaace4f700 1 -- 172.21.15.3:0/2582579525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd45c10 msgr2=0xdd94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:21.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaace4f700 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd45c10 0xdd94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaace4f700 1 -- 172.21.15.3:0/2582579525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 msgr2=0xd46eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:34:21.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaace4f700 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd46eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:34:21.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaace4f700 1 -- 172.21.15.3:0/2582579525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa812e0c0 con 0xd43dc0 2022-07-21T01:34:21.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaad650700 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd46eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:34:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaace4f700 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xdd8fd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fda98024000 tx=0x7fda98026040).ready entity=mon.0 client_cookie=ca001d1e72f26602 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:34:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fda8cff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2022-07-21T01:34:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fda8cff9700 1 -- 172.21.15.3:0/2582579525 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9800e560 con 0xd43dc0 2022-07-21T01:34:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fda8cff9700 1 -- 172.21.15.3:0/2582579525 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fda9800e6c0 con 0xd43dc0 2022-07-21T01:34:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 -- 172.21.15.3:0/2582579525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda9800e8b0 con 0xd43dc0 2022-07-21T01:34:21.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fda8cff9700 1 -- 172.21.15.3:0/2582579525 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda980224e0 con 0xd43dc0 2022-07-21T01:34:21.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 10 client.5130 fetch_fsmap learned FSMap version 12 2022-07-21T01:34:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 -- 172.21.15.3:0/2582579525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0xcadb90 con 0xd43dc0 2022-07-21T01:34:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fda8cff9700 1 -- 172.21.15.3:0/2582579525 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9800e8b0 con 0xd43dc0 2022-07-21T01:34:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fda8cff9700 1 -- 172.21.15.3:0/2582579525 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 12) v1 ==== 1434+0+0 (secure 0 0 0) 0x7fda9803f030 con 0xd43dc0 2022-07-21T01:34:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 10 client.5130 fetch_fsmap finished waiting for FSMap version 12 2022-07-21T01:34:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 10 client.5130 resolve_mds: resolved ID 'a' to GID 4896 2022-07-21T01:34:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 20 client.5130 populate_metadata read hostname 'smithi003' 2022-07-21T01:34:21.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] conn(0xd473a0 0xd477a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:34:21.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 4 client.5130 mds_command: new command op to 4896 tid=0[{"prefix": "cache drop"}] 2022-07-21T01:34:21.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdab930a740 1 -- 172.21.15.3:0/2582579525 --> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0xb9c590 con 0xd473a0 2022-07-21T01:34:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] conn(0xd473a0 0xd477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:34:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fda8cff9700 10 client.5130 ms_handle_connect on v2:172.21.15.3:6836/24695956 2022-07-21T01:34:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:34:21.677+0000 7fdaa7fff700 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] conn(0xd473a0 0xd477a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=620a2a030a715aad server_cookie=6629e2de773520db in_seq=0 out_seq=0 2022-07-21T01:34:31.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:34:31.302 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:34:31.305 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:34:31.307 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:35:01.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:35:01.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:35:01.364 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:35:01.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:35:06.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fda8cff9700 1 -- 172.21.15.3:0/2582579525 <== mds.0 v2:172.21.15.3:6836/24695956 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+386 (crc 0 0 0) 0xb9c590 con 0xd473a0 2022-07-21T01:35:06.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fda8cff9700 10 client.5130 handle_command_reply: tid=0 2022-07-21T01:35:06.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 client.5130 shutdown 2022-07-21T01:35:06.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 -- 172.21.15.3:0/2582579525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 msgr2=0xdd8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xdd8fd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fda98024000 tx=0x7fda98026040).stop 2022-07-21T01:35:06.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 -- 172.21.15.3:0/2582579525 shutdown_connections 2022-07-21T01:35:06.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd7da00 0xd46eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd45c10 0xdd94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.3:6836/24695956,v1:172.21.15.3:6837/24695956] conn(0xd473a0 0xd477a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 --2- 172.21.15.3:0/2582579525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0xd43dc0 0xdd8fd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 -- 172.21.15.3:0/2582579525 >> 172.21.15.3:0/2582579525 conn(0xd46430 msgr2=0xd4b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:06.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fda8cff9700 0 client.5130 ms_handle_reset on v2:172.21.15.3:6836/24695956 2022-07-21T01:35:06.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 -- 172.21.15.3:0/2582579525 shutdown_connections 2022-07-21T01:35:06.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 1 -- 172.21.15.3:0/2582579525 wait complete. 2022-07-21T01:35:06.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdab930a740 20 client.5130 trim_cache size 0 max 16384 2022-07-21T01:35:06.124 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:35:06.124 INFO:teuthology.orchestra.run.smithi003.stdout: "client_recall": { 2022-07-21T01:35:06.124 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2022-07-21T01:35:06.124 INFO:teuthology.orchestra.run.smithi003.stdout: "message": "(0) Success", 2022-07-21T01:35:06.125 INFO:teuthology.orchestra.run.smithi003.stdout: "recalled": 920 2022-07-21T01:35:06.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.125 INFO:teuthology.orchestra.run.smithi003.stdout: "flush_journal": { 2022-07-21T01:35:06.125 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2022-07-21T01:35:06.126 INFO:teuthology.orchestra.run.smithi003.stdout: "message": "" 2022-07-21T01:35:06.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.126 INFO:teuthology.orchestra.run.smithi003.stdout: "trim_cache": { 2022-07-21T01:35:06.126 INFO:teuthology.orchestra.run.smithi003.stdout: "trimmed": 900 2022-07-21T01:35:06.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.127 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2022-07-21T01:35:06.127 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": { 2022-07-21T01:35:06.127 INFO:teuthology.orchestra.run.smithi003.stdout: "items": 2517, 2022-07-21T01:35:06.127 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes": 431856 2022-07-21T01:35:06.128 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 44.438607359999999 2022-07-21T01:35:06.128 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:35:06.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda88053a70 msgr2=0x7fda88055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda88053a70 0x7fda88055f30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fda9800d000 tx=0x7fda98007f10).stop 2022-07-21T01:35:06.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaa80b5b20 msgr2=0x7fdaa8152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaa80b5b20 0x7fdaa8152520 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fda9c0018c0 tx=0x7fda9c001e10).stop 2022-07-21T01:35:06.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3586556698 shutdown_connections 2022-07-21T01:35:06.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda88053a70 0x7fda88055f30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa80b66f0 0x7fdaa81562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaa80b5b20 0x7fdaa8152520 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 --2- 172.21.15.3:0/3586556698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa805fd90 0x7fdaa80b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3586556698 >> 172.21.15.3:0/3586556698 conn(0x7fdaa8062530 msgr2=0x7fdaa812c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:06.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3586556698 shutdown_connections 2022-07-21T01:35:06.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.117+0000 7fdaaf0b3700 1 -- 172.21.15.3:0/3586556698 wait complete. 2022-07-21T01:35:06.133 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:35:06.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:35:06.150 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:35:06.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:35:06.192 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:35:06.213 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:35:06.214 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:06.215 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:06.215 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:06.224 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:35:06.225 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:35:06.226 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:35:06.226 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:35:06.227 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:06.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22231]: fuse finished with error 0 and tester_r 0 2022-07-21T01:35:06.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:06.244 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:06.245 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:06.293 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:35:06.294 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:35:06.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:35:06.398 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:06.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:06.406 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:06.406 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:35:06.413 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:35:06.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:35:06.415 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:35:06.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:35:06.416 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:35:06.423 INFO:teuthology.orchestra.run.smithi174.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:35:06.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:35:06.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:35:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.533+0000 7fac03156700 1 Processor -- start 2022-07-21T01:35:06.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.533+0000 7fac03156700 1 -- start start 2022-07-21T01:35:06.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.533+0000 7fac03156700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fabfc0b01a0 0x7fabfc0a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:06.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.533+0000 7fac03156700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc0a7250 con 0x7fabfc0b2710 2022-07-21T01:35:06.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.533+0000 7fac03156700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc0a7480 con 0x7fabfc0afd60 2022-07-21T01:35:06.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.533+0000 7fac03156700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabfc0a7720 con 0x7fabfc0b01a0 2022-07-21T01:35:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.533+0000 7fac01953700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fabfc0b01a0 0x7fabfc0a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.533+0000 7fac02154700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fabfc0b2710 0x7fabfc14afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52574/0 (socket says 172.21.15.3:52574) 2022-07-21T01:35:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.533+0000 7fac02154700 1 -- 172.21.15.3:0/3017303968 learned_addr learned my addr 172.21.15.3:0/3017303968 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac01953700 1 -- 172.21.15.3:0/3017303968 >> v1:172.21.15.143:6789/0 conn(0x7fabfc0afd60 legacy=0x7fabfc151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac01152700 1 -- 172.21.15.3:0/3017303968 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799551580 0 0) 0x7fabfc0a7250 con 0x7fabfc0b2710 2022-07-21T01:35:06.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac01953700 1 -- 172.21.15.3:0/3017303968 >> v1:172.21.15.3:6789/0 conn(0x7fabfc0b2710 legacy=0x7fabfc14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac01953700 1 -- 172.21.15.3:0/3017303968 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc0a79c0 con 0x7fabfc0b01a0 2022-07-21T01:35:06.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac01953700 1 --2- 172.21.15.3:0/3017303968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b01a0 0x7fabfc0a8540 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fabec0032f0 tx=0x7fabec0091d0).ready entity=mon.0 client_cookie=34826cadeb512d75 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:06.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac01152700 1 -- 172.21.15.3:0/3017303968 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabec009ea0 con 0x7fabfc0b01a0 2022-07-21T01:35:06.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac01152700 1 -- 172.21.15.3:0/3017303968 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabec00b8a0 con 0x7fabfc0b01a0 2022-07-21T01:35:06.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac01152700 1 -- 172.21.15.3:0/3017303968 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabec00ba10 con 0x7fabfc0b01a0 2022-07-21T01:35:06.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac03156700 1 -- 172.21.15.3:0/3017303968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b01a0 msgr2=0x7fabfc0a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac03156700 1 --2- 172.21.15.3:0/3017303968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b01a0 0x7fabfc0a8540 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fabec0032f0 tx=0x7fabec0091d0).stop 2022-07-21T01:35:06.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac03156700 1 -- 172.21.15.3:0/3017303968 shutdown_connections 2022-07-21T01:35:06.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac03156700 1 --2- 172.21.15.3:0/3017303968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b01a0 0x7fabfc0a8540 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac03156700 1 -- 172.21.15.3:0/3017303968 >> 172.21.15.3:0/3017303968 conn(0x7fabfc060310 msgr2=0x7fabfc0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:06.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac03156700 1 -- 172.21.15.3:0/3017303968 shutdown_connections 2022-07-21T01:35:06.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac03156700 1 -- 172.21.15.3:0/3017303968 wait complete. 2022-07-21T01:35:06.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac03156700 1 Processor -- start 2022-07-21T01:35:06.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.537+0000 7fac03156700 1 -- start start 2022-07-21T01:35:06.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac03156700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0afd60 0x7fabfc064580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:06.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac03156700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b01a0 0x7fabfc0a9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:06.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac03156700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0b2710 0x7fabfc1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:06.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac03156700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0ab470 con 0x7fabfc0afd60 2022-07-21T01:35:06.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac03156700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0ab1d0 con 0x7fabfc0b01a0 2022-07-21T01:35:06.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac03156700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc063e40 con 0x7fabfc0b2710 2022-07-21T01:35:06.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02154700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0afd60 0x7fabfc064580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:06.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02154700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0afd60 0x7fabfc064580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44300/0 (socket says 172.21.15.3:44300) 2022-07-21T01:35:06.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02154700 1 -- 172.21.15.3:0/3461217458 learned_addr learned my addr 172.21.15.3:0/3461217458 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:06.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac01953700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b01a0 0x7fabfc0a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:06.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02955700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0b2710 0x7fabfc1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:06.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02154700 1 -- 172.21.15.3:0/3461217458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0b2710 msgr2=0x7fabfc1567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02154700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0b2710 0x7fabfc1567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02154700 1 -- 172.21.15.3:0/3461217458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b01a0 msgr2=0x7fabfc0a9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02154700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b01a0 0x7fabfc0a9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02154700 1 -- 172.21.15.3:0/3461217458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec009470 con 0x7fabfc0afd60 2022-07-21T01:35:06.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02955700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0b2710 0x7fabfc1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:06.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac02154700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0afd60 0x7fabfc064580 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fabf0000bc0 tx=0x7fabf0007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:06.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fabfb7fe700 1 -- 172.21.15.3:0/3461217458 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf00070d0 con 0x7fabfc0afd60 2022-07-21T01:35:06.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fabfb7fe700 1 -- 172.21.15.3:0/3461217458 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf0007230 con 0x7fabfc0afd60 2022-07-21T01:35:06.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc0640e0 con 0x7fabfc0afd60 2022-07-21T01:35:06.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fabfb7fe700 1 -- 172.21.15.3:0/3461217458 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf0005610 con 0x7fabfc0afd60 2022-07-21T01:35:06.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.541+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc0aacc0 con 0x7fabfc0afd60 2022-07-21T01:35:06.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.545+0000 7fabfb7fe700 1 -- 172.21.15.3:0/3461217458 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fabf001a060 con 0x7fabfc0afd60 2022-07-21T01:35:06.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.545+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc0574e0 con 0x7fabfc0afd60 2022-07-21T01:35:06.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.545+0000 7fabfb7fe700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 0x7fabdc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:06.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.545+0000 7fabfb7fe700 1 -- 172.21.15.3:0/3461217458 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fabf006b850 con 0x7fabfc0afd60 2022-07-21T01:35:06.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.545+0000 7fac01953700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 0x7fabdc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:06.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.545+0000 7fac01953700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 0x7fabdc055f30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fabec01b000 tx=0x7fabec016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:06.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.549+0000 7fabfb7fe700 1 -- 172.21.15.3:0/3461217458 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fabf0042500 con 0x7fabfc0afd60 2022-07-21T01:35:06.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.677+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fabfc0ac740 con 0x7fabfc0afd60 2022-07-21T01:35:06.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.677+0000 7fabfb7fe700 1 -- 172.21.15.3:0/3461217458 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v37) v1 ==== 81+0+34037 (secure 0 0 0) 0x7fabf0018020 con 0x7fabfc0afd60 2022-07-21T01:35:06.680 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:06.680 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:35:06.681 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 37, 2022-07-21T01:35:06.681 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:35:06.681 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:35:06.681 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:34:04.714529+0000", 2022-07-21T01:35:06.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:35:06.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:35:06.682 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:35:06.683 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:35:06.683 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:35:06.684 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:35:06.684 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:35:06.684 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:35:06.684 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:35:06.684 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:35:06.685 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:35:06.685 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:35:06.685 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:35:06.685 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:35:06.686 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:35:06.686 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 6, 2022-07-21T01:35:06.687 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:35:06.687 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:35:06.688 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:35:06.688 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:35:06.689 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:35:06.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.690 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:35:06.690 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:35:06.691 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:35:06.691 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:35:06.691 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:35:06.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:35:06.692 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:35:06.693 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:35:06.693 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:35:06.694 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:35:06.694 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:35:06.695 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:35:06.695 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:35:06.696 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:35:06.696 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:35:06.696 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:35:06.697 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:35:06.697 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:35:06.698 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:35:06.698 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:35:06.699 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:35:06.699 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:35:06.700 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:35:06.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.700 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:35:06.701 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:35:06.701 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:35:06.702 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:35:06.702 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:35:06.703 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:35:06.703 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:35:06.704 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:35:06.704 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:35:06.705 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:35:06.705 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:35:06.705 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:35:06.706 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:35:06.706 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:35:06.708 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:35:06.709 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:35:06.709 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:35:06.709 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:35:06.709 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:35:06.710 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:35:06.710 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:35:06.710 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:35:06.710 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:35:06.711 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:35:06.711 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:35:06.711 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:35:06.711 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:35:06.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.712 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:35:06.712 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:35:06.712 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:35:06.713 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:35:06.713 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:35:06.713 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:35:06.713 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:35:06.714 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:35:06.714 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:35:06.714 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:35:06.714 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:35:06.715 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:35:06.715 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:35:06.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.715 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:35:06.716 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:35:06.716 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.717 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:35:06.717 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:35:06.717 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:35:06.717 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:35:06.718 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:35:06.718 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:35:06.718 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:35:06.718 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:35:06.719 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:35:06.719 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:35:06.719 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:35:06.719 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:35:06.720 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:35:06.720 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:35:06.720 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:35:06.720 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:35:06.721 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:35:06.721 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:35:06.721 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:35:06.721 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:35:06.722 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:35:06.722 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:35:06.722 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:35:06.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.723 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:35:06.723 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:35:06.723 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:35:06.723 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:35:06.724 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:35:06.724 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:35:06.724 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:35:06.724 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:35:06.725 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:35:06.725 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:35:06.725 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:35:06.725 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:35:06.726 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:35:06.726 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:35:06.726 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:35:06.726 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:35:06.727 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:35:06.727 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:35:06.727 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:35:06.727 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:35:06.728 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:35:06.728 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:35:06.728 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:35:06.728 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:35:06.729 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:35:06.729 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:35:06.729 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:35:06.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.730 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:35:06.730 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:35:06.730 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:35:06.730 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:35:06.731 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:35:06.731 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:35:06.731 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:35:06.731 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:35:06.732 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:35:06.732 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:35:06.732 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:35:06.732 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:35:06.733 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:35:06.733 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:35:06.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.734 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 5, 2022-07-21T01:35:06.734 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:35:06.734 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:34:01.732226+0000", 2022-07-21T01:35:06.735 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:35:06.735 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:35:06.735 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:35:06.735 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:35:06.736 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:35:06.736 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:35:06.736 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:35:06.736 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:35:06.737 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:35:06.737 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:35:06.737 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:35:06.737 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:35:06.738 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:35:06.738 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:35:06.738 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:35:06.738 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:35:06.739 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:35:06.739 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:35:06.739 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:35:06.739 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:35:06.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.740 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "36", 2022-07-21T01:35:06.740 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:35:06.740 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:35:06.741 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:35:06.741 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:35:06.741 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:35:06.741 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:35:06.742 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:35:06.742 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:35:06.742 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:35:06.742 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:35:06.742 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:35:06.743 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:35:06.743 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:35:06.743 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:35:06.743 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:35:06.744 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:35:06.744 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:35:06.744 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:35:06.744 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:35:06.745 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:35:06.745 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:35:06.745 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:35:06.745 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:35:06.746 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:35:06.746 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:35:06.746 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:35:06.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.747 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:35:06.747 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:35:06.747 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:35:06.747 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:35:06.748 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:35:06.748 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:35:06.748 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:35:06.748 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:35:06.749 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:35:06.749 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:35:06.749 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:35:06.749 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:35:06.750 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:35:06.750 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:35:06.750 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:35:06.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.751 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:35:06.751 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:35:06.751 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:35:06.751 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.752 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.752 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 6, 2022-07-21T01:35:06.753 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:35:06.753 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:34:02.749428+0000", 2022-07-21T01:35:06.753 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:35:06.753 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:35:06.754 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:35:06.754 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:35:06.754 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:35:06.754 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:35:06.755 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:35:06.755 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:35:06.755 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:35:06.755 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:35:06.756 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:35:06.756 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:35:06.756 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:35:06.756 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:35:06.757 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:35:06.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:35:06.757 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:35:06.757 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:35:06.758 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:35:06.758 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:35:06.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.758 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "37", 2022-07-21T01:35:06.759 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:35:06.759 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:35:06.759 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:35:06.759 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:35:06.760 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:35:06.760 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:35:06.760 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:35:06.760 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:35:06.760 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:35:06.761 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:35:06.761 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:35:06.761 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:35:06.761 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:35:06.762 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:35:06.762 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:35:06.762 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:35:06.762 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:35:06.763 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:35:06.763 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:35:06.763 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:35:06.763 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:35:06.764 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:35:06.764 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:35:06.764 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:35:06.764 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:35:06.765 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:35:06.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.765 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:35:06.765 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:35:06.766 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:35:06.766 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:35:06.766 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:35:06.766 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:35:06.767 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:35:06.767 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:35:06.767 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:35:06.767 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:35:06.767 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:35:06.768 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:35:06.768 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:35:06.768 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:35:06.768 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:35:06.769 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.769 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.769 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.769 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:35:06.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:35:06.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:35:06.770 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:35:06.771 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:06.771 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:06.771 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:06.771 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:06.772 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:06.772 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:06.772 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:35:06.772 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:06.772 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:06.773 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:06.773 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:06.773 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.774 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.774 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:35:06.774 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:06.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.775 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.775 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:35:06.775 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:06.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.776 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.776 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:06.776 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.777 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.777 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:35:06.777 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:06.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.778 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.778 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:35:06.779 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:06.779 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.779 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.780 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:06.780 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.780 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.781 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:35:06.781 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:06.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.781 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.782 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:35:06.782 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:06.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.782 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.783 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:06.783 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.784 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.784 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:35:06.784 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:06.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.785 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.785 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:35:06.785 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:06.785 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.786 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.786 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:35:06.786 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:35:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:35:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:35:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.789 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:35:06.789 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:35:06.789 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:06.790 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:06.790 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:06.790 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:06.790 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:06.791 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:06.791 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:06.791 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:06.791 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:06.792 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:06.792 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:06.792 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.792 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.793 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:35:06.793 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:06.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.794 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.794 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:35:06.794 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:06.794 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.795 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.795 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:06.795 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.796 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.796 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:35:06.796 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:06.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.797 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:35:06.798 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:06.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.798 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.799 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:06.799 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.799 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.799 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:35:06.800 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:06.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.800 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.801 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:35:06.801 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:06.801 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.801 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.802 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:06.802 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.803 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.803 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:35:06.803 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:06.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.804 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.804 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:35:06.804 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:06.804 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.805 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.805 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:35:06.805 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:35:06.806 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:35:06.806 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:35:06.806 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:06.806 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:06.807 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:06.807 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.807 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:35:06.808 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:35:06.808 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:06.808 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:06.808 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:06.809 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:06.809 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:06.809 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:06.809 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:35:06.810 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 34, 2022-07-21T01:35:06.810 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:06.810 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:06.810 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:06.811 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.811 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.811 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:35:06.812 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:06.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.812 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.812 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:35:06.813 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:06.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.813 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.814 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:06.814 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.814 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.815 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:35:06.815 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:06.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.816 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.816 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:35:06.816 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:06.816 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.817 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.817 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:06.817 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.818 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.818 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:35:06.818 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:06.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.819 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.819 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:35:06.819 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:06.820 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.820 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.820 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:06.821 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.821 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.821 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:35:06.822 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:06.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.823 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:35:06.823 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:06.823 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.823 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.824 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:35:06.824 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:35:06.824 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:35:06.824 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:35:06.825 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:06.825 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:06.825 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:06.825 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.826 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:35:06.826 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:35:06.827 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:06.827 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:06.827 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:06.827 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:06.828 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:06.828 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:06.828 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:35:06.828 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:06.828 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:06.829 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:06.829 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:06.829 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.830 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.830 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:35:06.830 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:06.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.831 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.831 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:35:06.831 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:06.832 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.832 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.832 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:06.833 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.833 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.833 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:35:06.833 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:06.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.834 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.834 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:35:06.835 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:06.835 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.835 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.836 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:06.836 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.836 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.837 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:35:06.837 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:06.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.837 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.838 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:35:06.838 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:06.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.838 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.839 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:06.839 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.840 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.840 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:35:06.840 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:06.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.841 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.841 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:35:06.841 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:06.842 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.842 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.842 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:35:06.842 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:35:06.843 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:35:06.843 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:35:06.843 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:06.843 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:06.844 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:06.844 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.845 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:35:06.845 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:35:06.845 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:06.845 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:06.846 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:06.846 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:06.846 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:06.846 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:06.846 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:06.847 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:06.847 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:06.847 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:06.847 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:06.848 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.848 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.848 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:35:06.849 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:06.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.849 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.850 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:35:06.850 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:06.850 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.850 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.851 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:06.851 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.851 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.852 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:35:06.852 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:06.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.853 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.853 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:35:06.853 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:06.853 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.854 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.854 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:06.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.855 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:35:06.855 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:06.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.856 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.856 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:35:06.856 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:06.857 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.857 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.857 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:06.858 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.858 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.858 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:35:06.859 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:06.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.859 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:35:06.860 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:06.860 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.860 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.861 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:35:06.861 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:35:06.861 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:35:06.861 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:35:06.862 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:06.862 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:06.862 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:06.862 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.863 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:35:06.863 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:35:06.864 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:06.864 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:06.864 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:06.864 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:06.865 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:06.865 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:06.865 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:06.865 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:06.865 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:06.866 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:06.866 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:06.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:35:06.867 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:06.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.868 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.868 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:35:06.868 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:06.869 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.869 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.869 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:06.869 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.870 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.870 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:35:06.870 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:06.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.871 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.871 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:35:06.872 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:06.872 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.872 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.873 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:06.873 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.873 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.873 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:35:06.874 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:06.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.874 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.875 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:35:06.875 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:06.875 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.875 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.876 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:06.876 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.877 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.877 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:35:06.877 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:06.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.878 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.878 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:35:06.878 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:06.879 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.879 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.879 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:35:06.879 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:35:06.880 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:35:06.880 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:35:06.880 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:06.880 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:06.881 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:06.881 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.882 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:35:06.882 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:35:06.882 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:06.882 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:06.883 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:06.883 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:06.883 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:06.883 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:06.883 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:06.884 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 34, 2022-07-21T01:35:06.884 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:06.884 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:06.884 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:06.885 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.885 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.885 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:35:06.886 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:06.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.886 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.887 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:35:06.887 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:06.887 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.887 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.888 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:06.888 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.888 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.889 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:35:06.889 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:06.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.890 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.890 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:35:06.890 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:06.890 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.891 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.891 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:06.891 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.892 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.892 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:35:06.892 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:06.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.893 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.893 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:35:06.893 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:06.894 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.894 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.894 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:06.895 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.895 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.895 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:35:06.895 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:06.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.896 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.896 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:35:06.897 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:06.897 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.897 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:35:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:35:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:35:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:35:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:35:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:35:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:35:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:35:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:35:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:35:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:35:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:35:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:35:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:35:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:35:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:35:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:35:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:35:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:35:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:35:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:35:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:35:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:35:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:35:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:35:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:35:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:35:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:35:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:35:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:35:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:35:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:35:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:35:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:35:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:35:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:35:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:35:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:35:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:35:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:35:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:35:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:35:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:35:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:35:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:35:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:35:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:35:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:35:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:35:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:35:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:35:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:35:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:35:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:35:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:35:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:35:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:35:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:35:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:35:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:35:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:35:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:35:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:35:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:35:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:35:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:35:06.946 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:35:06.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 msgr2=0x7fabdc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 0x7fabdc055f30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fabec01b000 tx=0x7fabec016000).stop 2022-07-21T01:35:06.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0afd60 msgr2=0x7fabfc064580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:06.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0afd60 0x7fabfc064580 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fabf0000bc0 tx=0x7fabf0007c90).stop 2022-07-21T01:35:06.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 shutdown_connections 2022-07-21T01:35:06.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0b2710 0x7fabfc1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 0x7fabdc055f30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b01a0 0x7fabfc0a9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 --2- 172.21.15.3:0/3461217458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0afd60 0x7fabfc064580 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:06.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 >> 172.21.15.3:0/3461217458 conn(0x7fabfc060310 msgr2=0x7fabfc149f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:06.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 shutdown_connections 2022-07-21T01:35:06.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:06.681+0000 7fac03156700 1 -- 172.21.15.3:0/3461217458 wait complete. 2022-07-21T01:35:06.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4eee43700 1 Processor -- start 2022-07-21T01:35:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4eee43700 1 -- start start 2022-07-21T01:35:07.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4eee43700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe4e80b2aa0 0x7fe4e80b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4eee43700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe4e8140c10 0x7fe4e80b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4eee43700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e81467f0 con 0x7fe4e80b6510 2022-07-21T01:35:07.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4eee43700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e8062140 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4eee43700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e80656b0 con 0x7fe4e8140c10 2022-07-21T01:35:07.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ed640700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe4e80b2aa0 0x7fe4e80b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ed640700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe4e80b2aa0 0x7fe4e80b0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44308/0 (socket says 172.21.15.3:44308) 2022-07-21T01:35:07.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ed640700 1 -- 172.21.15.3:0/3679043793 learned_addr learned my addr 172.21.15.3:0/3679043793 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:07.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ede41700 1 --1- 172.21.15.3:0/3679043793 >> v1:172.21.15.3:6789/0 conn(0x7fe4e80b6510 0x7fe4e8140500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52590/0 (socket says 172.21.15.3:52590) 2022-07-21T01:35:07.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ee642700 1 --2- 172.21.15.3:0/3679043793 >> v2:172.21.15.143:3300/0 conn(0x7fe4e8140c10 0x7fe4e80b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ed640700 1 -- 172.21.15.3:0/3679043793 >> v2:172.21.15.143:3300/0 conn(0x7fe4e8140c10 msgr2=0x7fe4e80b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:07.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ed640700 1 --2- 172.21.15.3:0/3679043793 >> v2:172.21.15.143:3300/0 conn(0x7fe4e8140c10 0x7fe4e80b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ed640700 1 -- 172.21.15.3:0/3679043793 >> v1:172.21.15.3:6789/0 conn(0x7fe4e80b6510 legacy=0x7fe4e8140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ed640700 1 -- 172.21.15.3:0/3679043793 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e81476a0 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.065+0000 7fe4ed640700 1 --2- 172.21.15.3:0/3679043793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 0x7fe4e80b0e60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe4e4002f70 tx=0x7fe4e4006eb0).ready entity=mon.0 client_cookie=20653c9e2b490433 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:07.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4ece3f700 1 -- 172.21.15.3:0/3679043793 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e400d5c0 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4ece3f700 1 -- 172.21.15.3:0/3679043793 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e4000e90 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4ece3f700 1 -- 172.21.15.3:0/3679043793 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e400d950 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4eee43700 1 -- 172.21.15.3:0/3679043793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 msgr2=0x7fe4e80b0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4eee43700 1 --2- 172.21.15.3:0/3679043793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 0x7fe4e80b0e60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe4e4002f70 tx=0x7fe4e4006eb0).stop 2022-07-21T01:35:07.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4eee43700 1 -- 172.21.15.3:0/3679043793 shutdown_connections 2022-07-21T01:35:07.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4eee43700 1 --2- 172.21.15.3:0/3679043793 >> v2:172.21.15.143:3300/0 conn(0x7fe4e8140c10 0x7fe4e80b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4eee43700 1 --2- 172.21.15.3:0/3679043793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 0x7fe4e80b0e60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4eee43700 1 -- 172.21.15.3:0/3679043793 >> 172.21.15.3:0/3679043793 conn(0x7fe4e8063800 msgr2=0x7fe4e8063be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4eee43700 1 -- 172.21.15.3:0/3679043793 shutdown_connections 2022-07-21T01:35:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.069+0000 7fe4eee43700 1 -- 172.21.15.3:0/3679043793 wait complete. 2022-07-21T01:35:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 Processor -- start 2022-07-21T01:35:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 -- start start 2022-07-21T01:35:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 0x7fe4e81458f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4e80b6510 0x7fe4e80aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8140c10 0x7fe4e8152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e80af140 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e80a4280 con 0x7fe4e8140c10 2022-07-21T01:35:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e80af360 con 0x7fe4e80b6510 2022-07-21T01:35:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ede41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 0x7fe4e81458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ede41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 0x7fe4e81458f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44314/0 (socket says 172.21.15.3:44314) 2022-07-21T01:35:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ede41700 1 -- 172.21.15.3:0/1992053 learned_addr learned my addr 172.21.15.3:0/1992053 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ee642700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8140c10 0x7fe4e8152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ed640700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4e80b6510 0x7fe4e80aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ede41700 1 -- 172.21.15.3:0/1992053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4e80b6510 msgr2=0x7fe4e80aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ede41700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4e80b6510 0x7fe4e80aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ede41700 1 -- 172.21.15.3:0/1992053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8140c10 msgr2=0x7fe4e8152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ede41700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8140c10 0x7fe4e8152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ede41700 1 -- 172.21.15.3:0/1992053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e400b1e0 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ed640700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4e80b6510 0x7fe4e80aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4ede41700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 0x7fe4e81458f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fe4e80634d0 tx=0x7fe4d8007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4d6ffd700 1 -- 172.21.15.3:0/1992053 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d80070d0 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4d6ffd700 1 -- 172.21.15.3:0/1992053 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d8007230 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e80af600 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4d6ffd700 1 -- 172.21.15.3:0/1992053 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d8005550 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.073+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e81526a0 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.077+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e8153370 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.077+0000 7fe4d6ffd700 1 -- 172.21.15.3:0/1992053 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe4d801a070 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.077+0000 7fe4d6ffd700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4c8053950 0x7fe4c8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.077+0000 7fe4d6ffd700 1 -- 172.21.15.3:0/1992053 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe4d806b030 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.077+0000 7fe4ed640700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4c8053950 0x7fe4c8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.077+0000 7fe4ed640700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4c8053950 0x7fe4c8055e10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe4e401a000 tx=0x7fe4e4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:07.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.085+0000 7fe4d6ffd700 1 -- 172.21.15.3:0/1992053 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4e8153370 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.245+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4e8153370 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.245+0000 7fe4d6ffd700 1 -- 172.21.15.3:0/1992053 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2480 (secure 0 0 0) 0x7fe4e8153370 con 0x7fe4e80b2aa0 2022-07-21T01:35:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:07.248 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:34:03.710770+0000","modified":"2022-07-21T01:34:04.713666+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4896},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4896":{"gid":4896,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/24695956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":24695956},{"type":"v1","addr":"172.21.15.3:6837","nonce":24695956}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-21T01:35:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4c8053950 msgr2=0x7fe4c8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4c8053950 0x7fe4c8055e10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe4e401a000 tx=0x7fe4e4015000).stop 2022-07-21T01:35:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 msgr2=0x7fe4e81458f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 0x7fe4e81458f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fe4e80634d0 tx=0x7fe4d8007c40).stop 2022-07-21T01:35:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 shutdown_connections 2022-07-21T01:35:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4c8053950 0x7fe4c8055e10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8140c10 0x7fe4e8152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4e80b6510 0x7fe4e80aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 --2- 172.21.15.3:0/1992053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4e80b2aa0 0x7fe4e81458f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 >> 172.21.15.3:0/1992053 conn(0x7fe4e8063800 msgr2=0x7fe4e813fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 shutdown_connections 2022-07-21T01:35:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.249+0000 7fe4eee43700 1 -- 172.21.15.3:0/1992053 wait complete. 2022-07-21T01:35:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 12 2022-07-21T01:35:07.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:35:07.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.365+0000 7f4ce3b0d700 1 Processor -- start 2022-07-21T01:35:07.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.365+0000 7f4ce3b0d700 1 -- start start 2022-07-21T01:35:07.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4cdc0b1b30 0x7f4cdc067150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc0a9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cdc0ac030 con 0x7f4cdc060410 2022-07-21T01:35:07.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cdc0ab4a0 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc140490 con 0x7f4cdc0a9930 2022-07-21T01:35:07.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce20aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc0a9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce10a8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4cdc0b1b30 0x7f4cdc067150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce20aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc0a9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:51238/0 (socket says 172.21.15.3:51238) 2022-07-21T01:35:07.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce10a8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4cdc0b1b30 0x7f4cdc067150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:49770/0 (socket says 172.21.15.3:49770) 2022-07-21T01:35:07.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce20aa700 1 -- 172.21.15.3:0/3434523526 learned_addr learned my addr 172.21.15.3:0/3434523526 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:07.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce20aa700 1 -- 172.21.15.3:0/3434523526 >> v2:172.21.15.143:3300/0 conn(0x7f4cdc0b1b30 msgr2=0x7f4cdc067150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:07.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce20aa700 1 --2- 172.21.15.3:0/3434523526 >> v2:172.21.15.143:3300/0 conn(0x7f4cdc0b1b30 0x7f4cdc067150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce20aa700 1 -- 172.21.15.3:0/3434523526 >> v1:172.21.15.143:6789/0 conn(0x7f4cdc060410 legacy=0x7f4cdc066a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce20aa700 1 -- 172.21.15.3:0/3434523526 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cdc140730 con 0x7f4cdc0a9930 2022-07-21T01:35:07.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce10a8700 1 --2- 172.21.15.3:0/3434523526 >> v2:172.21.15.143:3300/0 conn(0x7f4cdc0b1b30 0x7f4cdc067150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:07.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce20aa700 1 --2- 172.21.15.3:0/3434523526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc0a9d70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4cd40032f0 tx=0x7f4cd400b530).ready entity=mon.2 client_cookie=c21bd1c458654640 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:07.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce08a7700 1 -- 172.21.15.3:0/3434523526 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cd40153f0 con 0x7f4cdc0a9930 2022-07-21T01:35:07.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce08a7700 1 -- 172.21.15.3:0/3434523526 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd4015550 con 0x7f4cdc0a9930 2022-07-21T01:35:07.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce08a7700 1 -- 172.21.15.3:0/3434523526 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cd40158b0 con 0x7f4cdc0a9930 2022-07-21T01:35:07.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/3434523526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 msgr2=0x7f4cdc0a9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 --2- 172.21.15.3:0/3434523526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc0a9d70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4cd40032f0 tx=0x7f4cd400b530).stop 2022-07-21T01:35:07.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/3434523526 shutdown_connections 2022-07-21T01:35:07.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 --2- 172.21.15.3:0/3434523526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc0a9d70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 --2- 172.21.15.3:0/3434523526 >> v2:172.21.15.143:3300/0 conn(0x7f4cdc0b1b30 0x7f4cdc067150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/3434523526 >> 172.21.15.3:0/3434523526 conn(0x7f4cdc0b5570 msgr2=0x7f4cdc0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:07.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/3434523526 shutdown_connections 2022-07-21T01:35:07.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.369+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/3434523526 wait complete. 2022-07-21T01:35:07.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 Processor -- start 2022-07-21T01:35:07.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 -- start start 2022-07-21T01:35:07.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cdc060410 0x7f4cdc1528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc1412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4cdc0b1b30 0x7f4cdc062fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc0a8c70 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc0668f0 con 0x7f4cdc060410 2022-07-21T01:35:07.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc143c40 con 0x7f4cdc0a9930 2022-07-21T01:35:07.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce20aa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4cdc0b1b30 0x7f4cdc062fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce18a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cdc060410 0x7f4cdc1528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce20aa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4cdc0b1b30 0x7f4cdc062fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44332/0 (socket says 172.21.15.3:44332) 2022-07-21T01:35:07.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce20aa700 1 -- 172.21.15.3:0/4162770210 learned_addr learned my addr 172.21.15.3:0/4162770210 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:07.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce10a8700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc1412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce20aa700 1 -- 172.21.15.3:0/4162770210 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 msgr2=0x7f4cdc1412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce20aa700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc1412d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce20aa700 1 -- 172.21.15.3:0/4162770210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cdc060410 msgr2=0x7f4cdc1528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce20aa700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cdc060410 0x7f4cdc1528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce20aa700 1 -- 172.21.15.3:0/4162770210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd000b180 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce20aa700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4cdc0b1b30 0x7f4cdc062fd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f4cdc066100 tx=0x7f4cd4019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:07.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4cceffd700 1 -- 172.21.15.3:0/4162770210 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cd4009040 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4cceffd700 1 -- 172.21.15.3:0/4162770210 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd401c580 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd400b7d0 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4cceffd700 1 -- 172.21.15.3:0/4162770210 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cd40153f0 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.373+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cdc143040 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.377+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cdc066620 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.377+0000 7f4cceffd700 1 -- 172.21.15.3:0/4162770210 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4cd401c8e0 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.377+0000 7f4cceffd700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4cbc053b00 0x7f4cbc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.377+0000 7f4cceffd700 1 -- 172.21.15.3:0/4162770210 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4cd406dcf0 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.377+0000 7f4ce18a9700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4cbc053b00 0x7f4cbc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.377+0000 7f4ce18a9700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4cbc053b00 0x7f4cbc055fc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f4cd800d000 tx=0x7f4cd8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:07.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.385+0000 7f4cceffd700 1 -- 172.21.15.3:0/4162770210 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4cd40449a0 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.549+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4cdc0a9490 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.642 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:35:07.641+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:35:07.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.657+0000 7f4cceffd700 1 -- 172.21.15.3:0/4162770210 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v13) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4cd4047fc0 con 0x7f4cdc0b1b30 2022-07-21T01:35:07.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.657+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4cbc053b00 msgr2=0x7f4cbc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.657+0000 7f4ce3b0d700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4cbc053b00 0x7f4cbc055fc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f4cd800d000 tx=0x7f4cd8007b90).stop 2022-07-21T01:35:07.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.657+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4cdc0b1b30 msgr2=0x7f4cdc062fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.657+0000 7f4ce3b0d700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4cdc0b1b30 0x7f4cdc062fd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f4cdc066100 tx=0x7f4cd4019000).stop 2022-07-21T01:35:07.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.657+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 shutdown_connections 2022-07-21T01:35:07.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.657+0000 7f4ce3b0d700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4cdc0b1b30 0x7f4cdc062fd0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.661+0000 7f4ce3b0d700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cdc0a9930 0x7f4cdc1412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.661+0000 7f4ce3b0d700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4cbc053b00 0x7f4cbc055fc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.661+0000 7f4ce3b0d700 1 --2- 172.21.15.3:0/4162770210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cdc060410 0x7f4cdc1528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.661+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 >> 172.21.15.3:0/4162770210 conn(0x7f4cdc0b5570 msgr2=0x7f4cdc0a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:07.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.661+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 shutdown_connections 2022-07-21T01:35:07.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.661+0000 7f4ce3b0d700 1 -- 172.21.15.3:0/4162770210 wait complete. 2022-07-21T01:35:07.664 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:35:07.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:07.694 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:35:07.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.769+0000 7f4edd22f700 1 Processor -- start 2022-07-21T01:35:07.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.769+0000 7f4edd22f700 1 -- start start 2022-07-21T01:35:07.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80b1f40 0x7f4ed80b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ed80a7250 0x7f4ed81448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80b04e0 con 0x7f4ed8148530 2022-07-21T01:35:07.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed80a9250 con 0x7f4ed80a7250 2022-07-21T01:35:07.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80b0780 con 0x7f4ed80b1f40 2022-07-21T01:35:07.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80b1f40 0x7f4ed80b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ed80a7250 0x7f4ed81448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed659c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4ed8148530 0x7f4ed80b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:54746/0 (socket says 172.21.15.3:54746) 2022-07-21T01:35:07.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80b1f40 0x7f4ed80b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:51264/0 (socket says 172.21.15.3:51264) 2022-07-21T01:35:07.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ed80a7250 0x7f4ed81448b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:49798/0 (socket says 172.21.15.3:49798) 2022-07-21T01:35:07.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed659c700 1 -- 172.21.15.3:0/2776744316 learned_addr learned my addr 172.21.15.3:0/2776744316 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:07.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed5d9b700 1 -- 172.21.15.3:0/2776744316 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411567913 0 0) 0x7f4ed80b04e0 con 0x7f4ed8148530 2022-07-21T01:35:07.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed5d9b700 1 -- 172.21.15.3:0/2776744316 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb8003260 con 0x7f4ed8148530 2022-07-21T01:35:07.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed6d9d700 1 -- 172.21.15.3:0/2776744316 >> v2:172.21.15.143:3300/0 conn(0x7f4ed80a7250 msgr2=0x7f4ed81448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:07.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed6d9d700 1 --2- 172.21.15.3:0/2776744316 >> v2:172.21.15.143:3300/0 conn(0x7f4ed80a7250 0x7f4ed81448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed6d9d700 1 -- 172.21.15.3:0/2776744316 >> v1:172.21.15.143:6789/0 conn(0x7f4ed8148530 legacy=0x7f4ed80b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed6d9d700 1 -- 172.21.15.3:0/2776744316 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80b0a20 con 0x7f4ed80b1f40 2022-07-21T01:35:07.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed759e700 1 --2- 172.21.15.3:0/2776744316 >> v2:172.21.15.143:3300/0 conn(0x7f4ed80a7250 0x7f4ed81448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:07.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed6d9d700 1 --2- 172.21.15.3:0/2776744316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80b1f40 0x7f4ed80b1160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f4ec80032f0 tx=0x7f4ec80091d0).ready entity=mon.2 client_cookie=c27e487c579bbfd5 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:07.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed5d9b700 1 -- 172.21.15.3:0/2776744316 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ec800b600 con 0x7f4ed80b1f40 2022-07-21T01:35:07.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed5d9b700 1 -- 172.21.15.3:0/2776744316 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec800b760 con 0x7f4ed80b1f40 2022-07-21T01:35:07.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4ed5d9b700 1 -- 172.21.15.3:0/2776744316 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ec800b8d0 con 0x7f4ed80b1f40 2022-07-21T01:35:07.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 -- 172.21.15.3:0/2776744316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80b1f40 msgr2=0x7f4ed80b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 --2- 172.21.15.3:0/2776744316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80b1f40 0x7f4ed80b1160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f4ec80032f0 tx=0x7f4ec80091d0).stop 2022-07-21T01:35:07.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 -- 172.21.15.3:0/2776744316 shutdown_connections 2022-07-21T01:35:07.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 --2- 172.21.15.3:0/2776744316 >> v2:172.21.15.143:3300/0 conn(0x7f4ed80a7250 0x7f4ed81448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 --2- 172.21.15.3:0/2776744316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80b1f40 0x7f4ed80b1160 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 -- 172.21.15.3:0/2776744316 >> 172.21.15.3:0/2776744316 conn(0x7f4ed805f840 msgr2=0x7f4ed805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:07.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.773+0000 7f4edd22f700 1 -- 172.21.15.3:0/2776744316 shutdown_connections 2022-07-21T01:35:07.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.777+0000 7f4edd22f700 1 -- 172.21.15.3:0/2776744316 wait complete. 2022-07-21T01:35:07.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.777+0000 7f4edd22f700 1 Processor -- start 2022-07-21T01:35:07.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.777+0000 7f4edd22f700 1 -- start start 2022-07-21T01:35:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4edd22f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80a7250 0x7f4ed8144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4edd22f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80b1f40 0x7f4ed8156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4edd22f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ed8148530 0x7f4ed8157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4edd22f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed814ac30 con 0x7f4ed8148530 2022-07-21T01:35:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4edd22f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80a9750 con 0x7f4ed80b1f40 2022-07-21T01:35:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4edd22f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80a99f0 con 0x7f4ed80a7250 2022-07-21T01:35:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ed8148530 0x7f4ed8157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ed8148530 0x7f4ed8157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44358/0 (socket says 172.21.15.3:44358) 2022-07-21T01:35:07.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed759e700 1 -- 172.21.15.3:0/71288982 learned_addr learned my addr 172.21.15.3:0/71288982 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:07.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed659c700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80b1f40 0x7f4ed8156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed6d9d700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80a7250 0x7f4ed8144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed759e700 1 -- 172.21.15.3:0/71288982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80a7250 msgr2=0x7f4ed8144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed759e700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80a7250 0x7f4ed8144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed759e700 1 -- 172.21.15.3:0/71288982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80b1f40 msgr2=0x7f4ed8156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed759e700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80b1f40 0x7f4ed8156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed759e700 1 -- 172.21.15.3:0/71288982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed000b180 con 0x7f4ed8148530 2022-07-21T01:35:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed6d9d700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80a7250 0x7f4ed8144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:35:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ed759e700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ed8148530 0x7f4ed8157180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f4ed000c000 tx=0x7f4ed0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:07.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ecffff700 1 -- 172.21.15.3:0/71288982 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ed0010780 con 0x7f4ed8148530 2022-07-21T01:35:07.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ecffff700 1 -- 172.21.15.3:0/71288982 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed00108e0 con 0x7f4ed8148530 2022-07-21T01:35:07.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec8009470 con 0x7f4ed8148530 2022-07-21T01:35:07.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4ecffff700 1 -- 172.21.15.3:0/71288982 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ed0010c40 con 0x7f4ed8148530 2022-07-21T01:35:07.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.781+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed8149d10 con 0x7f4ed8148530 2022-07-21T01:35:07.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.785+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed80b0c00 con 0x7f4ed8148530 2022-07-21T01:35:07.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.785+0000 7f4ecffff700 1 -- 172.21.15.3:0/71288982 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4ed001c460 con 0x7f4ed8148530 2022-07-21T01:35:07.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.785+0000 7f4ecffff700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ebc053b00 0x7f4ebc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:07.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.785+0000 7f4ecffff700 1 -- 172.21.15.3:0/71288982 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f4ed006c7e0 con 0x7f4ed8148530 2022-07-21T01:35:07.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.785+0000 7f4ed6d9d700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ebc053b00 0x7f4ebc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:07.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.785+0000 7f4ed6d9d700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ebc053b00 0x7f4ebc055fc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4ec8017000 tx=0x7f4ec8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:07.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.789+0000 7f4ecffff700 1 -- 172.21.15.3:0/71288982 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ed0043360 con 0x7f4ed8148530 2022-07-21T01:35:07.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.957+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ed80a9e90 con 0x7f4ed8148530 2022-07-21T01:35:07.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.957+0000 7f4ecffff700 1 -- 172.21.15.3:0/71288982 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2128 (secure 0 0 0) 0x7f4ed0046980 con 0x7f4ed8148530 2022-07-21T01:35:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:07.961 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":13,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:34:03.710770+0000","modified":"2022-07-21T01:35:07.642650+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-21T01:35:07.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ebc053b00 msgr2=0x7f4ebc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ebc053b00 0x7f4ebc055fc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4ec8017000 tx=0x7f4ec8016000).stop 2022-07-21T01:35:07.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ed8148530 msgr2=0x7f4ed8157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:07.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ed8148530 0x7f4ed8157180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f4ed000c000 tx=0x7f4ed0006df0).stop 2022-07-21T01:35:07.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 shutdown_connections 2022-07-21T01:35:07.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ed8148530 0x7f4ed8157180 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ebc053b00 0x7f4ebc055fc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80b1f40 0x7f4ed8156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 --2- 172.21.15.3:0/71288982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80a7250 0x7f4ed8144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:07.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 >> 172.21.15.3:0/71288982 conn(0x7f4ed805f840 msgr2=0x7f4ed80a76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:07.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 shutdown_connections 2022-07-21T01:35:07.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:07.961+0000 7f4edd22f700 1 -- 172.21.15.3:0/71288982 wait complete. 2022-07-21T01:35:07.966 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 13 2022-07-21T01:35:07.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:35:08.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9f197700 1 Processor -- start 2022-07-21T01:35:08.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9f197700 1 -- start start 2022-07-21T01:35:08.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9f197700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4d980b4310 0x7f4d98140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9f197700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4d980b4820 0x7f4d980b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9f197700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d98143cf0 con 0x7f4d98146ae0 2022-07-21T01:35:08.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9f197700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d981476a0 con 0x7f4d980b4310 2022-07-21T01:35:08.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9f197700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d98143f90 con 0x7f4d980b4820 2022-07-21T01:35:08.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d97fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4d980b4310 0x7f4d98140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9cf33700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4d98146ae0 0x7f4d981403c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52642/0 (socket says 172.21.15.3:52642) 2022-07-21T01:35:08.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9cf33700 1 -- 172.21.15.3:0/426756317 learned_addr learned my addr 172.21.15.3:0/426756317 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:08.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d97fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4d980b4310 0x7f4d98140ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44364/0 (socket says 172.21.15.3:44364) 2022-07-21T01:35:08.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d9d734700 1 --2- 172.21.15.3:0/426756317 >> v2:172.21.15.143:3300/0 conn(0x7f4d980b4820 0x7f4d980b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d97fff700 1 -- 172.21.15.3:0/426756317 >> v2:172.21.15.143:3300/0 conn(0x7f4d980b4820 msgr2=0x7f4d980b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:08.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d97fff700 1 --2- 172.21.15.3:0/426756317 >> v2:172.21.15.143:3300/0 conn(0x7f4d980b4820 0x7f4d980b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d97fff700 1 -- 172.21.15.3:0/426756317 >> v1:172.21.15.3:6789/0 conn(0x7f4d98146ae0 legacy=0x7f4d981403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.129+0000 7f4d977fe700 1 -- 172.21.15.3:0/426756317 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845660805 0 0) 0x7f4d98143cf0 con 0x7f4d98146ae0 2022-07-21T01:35:08.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d97fff700 1 -- 172.21.15.3:0/426756317 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d980b0e20 con 0x7f4d980b4310 2022-07-21T01:35:08.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d97fff700 1 --2- 172.21.15.3:0/426756317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 0x7f4d98140ad0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f4d88002f70 tx=0x7f4d88006df0).ready entity=mon.0 client_cookie=faaaac16cb6956e8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:08.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d977fe700 1 -- 172.21.15.3:0/426756317 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d8800d570 con 0x7f4d980b4310 2022-07-21T01:35:08.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d977fe700 1 -- 172.21.15.3:0/426756317 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d88000e90 con 0x7f4d980b4310 2022-07-21T01:35:08.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d977fe700 1 -- 172.21.15.3:0/426756317 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d8800d8c0 con 0x7f4d980b4310 2022-07-21T01:35:08.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 -- 172.21.15.3:0/426756317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 msgr2=0x7f4d98140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 --2- 172.21.15.3:0/426756317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 0x7f4d98140ad0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f4d88002f70 tx=0x7f4d88006df0).stop 2022-07-21T01:35:08.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 -- 172.21.15.3:0/426756317 shutdown_connections 2022-07-21T01:35:08.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 --2- 172.21.15.3:0/426756317 >> v2:172.21.15.143:3300/0 conn(0x7f4d980b4820 0x7f4d980b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 --2- 172.21.15.3:0/426756317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 0x7f4d98140ad0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 -- 172.21.15.3:0/426756317 >> 172.21.15.3:0/426756317 conn(0x7f4d9805c620 msgr2=0x7f4d9805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:08.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 -- 172.21.15.3:0/426756317 shutdown_connections 2022-07-21T01:35:08.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 -- 172.21.15.3:0/426756317 wait complete. 2022-07-21T01:35:08.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 Processor -- start 2022-07-21T01:35:08.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.133+0000 7f4d9f197700 1 -- start start 2022-07-21T01:35:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9f197700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 0x7f4d9813b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9f197700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d980b4820 0x7f4d9813b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9f197700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d98146ae0 0x7f4d98152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9f197700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d98146480 con 0x7f4d980b4310 2022-07-21T01:35:08.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9f197700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980b6ce0 con 0x7f4d980b4820 2022-07-21T01:35:08.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9f197700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d98144d50 con 0x7f4d98146ae0 2022-07-21T01:35:08.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9cf33700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 0x7f4d9813b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9cf33700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 0x7f4d9813b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44368/0 (socket says 172.21.15.3:44368) 2022-07-21T01:35:08.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9cf33700 1 -- 172.21.15.3:0/1299448456 learned_addr learned my addr 172.21.15.3:0/1299448456 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:08.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d97fff700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d980b4820 0x7f4d9813b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9d734700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d98146ae0 0x7f4d98152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9cf33700 1 -- 172.21.15.3:0/1299448456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d98146ae0 msgr2=0x7f4d98152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9cf33700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d98146ae0 0x7f4d98152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9cf33700 1 -- 172.21.15.3:0/1299448456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d980b4820 msgr2=0x7f4d9813b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9cf33700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d980b4820 0x7f4d9813b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9cf33700 1 -- 172.21.15.3:0/1299448456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8800b180 con 0x7f4d980b4310 2022-07-21T01:35:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9cf33700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 0x7f4d9813b010 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f4d80000ed0 tx=0x7f4d80007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d95ffb700 1 -- 172.21.15.3:0/1299448456 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d800070d0 con 0x7f4d980b4310 2022-07-21T01:35:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d95ffb700 1 -- 172.21.15.3:0/1299448456 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d80007230 con 0x7f4d980b4310 2022-07-21T01:35:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d95ffb700 1 -- 172.21.15.3:0/1299448456 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d80005610 con 0x7f4d980b4310 2022-07-21T01:35:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d98156300 con 0x7f4d980b4310 2022-07-21T01:35:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d98145660 con 0x7f4d980b4310 2022-07-21T01:35:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d98144ef0 con 0x7f4d980b4310 2022-07-21T01:35:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.137+0000 7f4d95ffb700 1 -- 172.21.15.3:0/1299448456 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4d80010070 con 0x7f4d980b4310 2022-07-21T01:35:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.141+0000 7f4d95ffb700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4d7c053b00 0x7f4d7c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.141+0000 7f4d95ffb700 1 -- 172.21.15.3:0/1299448456 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f4d8006b730 con 0x7f4d980b4310 2022-07-21T01:35:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.141+0000 7f4d97fff700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4d7c053b00 0x7f4d7c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.141+0000 7f4d97fff700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4d7c053b00 0x7f4d7c055fc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f4d8801a000 tx=0x7f4d88014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:08.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.145+0000 7f4d95ffb700 1 -- 172.21.15.3:0/1299448456 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4d800424a0 con 0x7f4d980b4310 2022-07-21T01:35:08.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.305+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4d98145350 con 0x7f4d980b4310 2022-07-21T01:35:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.661+0000 7f4d95ffb700 1 -- 172.21.15.3:0/1299448456 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4d80045ac0 con 0x7f4d980b4310 2022-07-21T01:35:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4d7c053b00 msgr2=0x7f4d7c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4d7c053b00 0x7f4d7c055fc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f4d8801a000 tx=0x7f4d88014000).stop 2022-07-21T01:35:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 msgr2=0x7f4d9813b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 0x7f4d9813b010 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f4d80000ed0 tx=0x7f4d80007ac0).stop 2022-07-21T01:35:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 shutdown_connections 2022-07-21T01:35:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d98146ae0 0x7f4d98152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4d7c053b00 0x7f4d7c055fc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d980b4820 0x7f4d9813b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 --2- 172.21.15.3:0/1299448456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d980b4310 0x7f4d9813b010 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 >> 172.21.15.3:0/1299448456 conn(0x7f4d9805c620 msgr2=0x7f4d98143240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:08.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 shutdown_connections 2022-07-21T01:35:08.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.665+0000 7f4d9f197700 1 -- 172.21.15.3:0/1299448456 wait complete. 2022-07-21T01:35:08.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:35:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b60ab8700 1 Processor -- start 2022-07-21T01:35:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b60ab8700 1 -- start start 2022-07-21T01:35:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b60ab8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 0x7f9b5c0b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b60ab8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9b5c0b4550 0x7f9b5c0b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b60ab8700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5c144240 con 0x7f9b5c061ec0 2022-07-21T01:35:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b60ab8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b5c0b6fe0 con 0x7f9b5c0b4550 2022-07-21T01:35:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b60ab8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c143d00 con 0x7f9b5c0b6710 2022-07-21T01:35:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b59d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9b5c0b4550 0x7f9b5c0b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b5ad9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9b5c061ec0 0x7f9b5c146850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52660/0 (socket says 172.21.15.3:52660) 2022-07-21T01:35:08.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.809+0000 7f9b5ad9d700 1 -- 172.21.15.3:0/1934926523 learned_addr learned my addr 172.21.15.3:0/1934926523 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:08.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b5a59c700 1 --2- 172.21.15.3:0/1934926523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 0x7f9b5c0b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b5959a700 1 -- 172.21.15.3:0/1934926523 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233052640 0 0) 0x7f9b5c144240 con 0x7f9b5c061ec0 2022-07-21T01:35:08.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b5959a700 1 -- 172.21.15.3:0/1934926523 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3c003260 con 0x7f9b5c061ec0 2022-07-21T01:35:08.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b59d9b700 1 -- 172.21.15.3:0/1934926523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 msgr2=0x7f9b5c0b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:08.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b59d9b700 1 --2- 172.21.15.3:0/1934926523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 0x7f9b5c0b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b59d9b700 1 -- 172.21.15.3:0/1934926523 >> v1:172.21.15.3:6789/0 conn(0x7f9b5c061ec0 legacy=0x7f9b5c146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b59d9b700 1 -- 172.21.15.3:0/1934926523 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c143fa0 con 0x7f9b5c0b4550 2022-07-21T01:35:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b59d9b700 1 --2- 172.21.15.3:0/1934926523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 0x7f9b5c0b6af0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f9b44002f70 tx=0x7f9b44006df0).ready entity=mon.0 client_cookie=7bb689b3ab3cb12d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b5959a700 1 -- 172.21.15.3:0/1934926523 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b4400d4a0 con 0x7f9b5c0b4550 2022-07-21T01:35:08.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b5959a700 1 -- 172.21.15.3:0/1934926523 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b44000e90 con 0x7f9b5c0b4550 2022-07-21T01:35:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b5959a700 1 -- 172.21.15.3:0/1934926523 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b4400d7f0 con 0x7f9b5c0b4550 2022-07-21T01:35:08.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1934926523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 msgr2=0x7f9b5c0b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 --2- 172.21.15.3:0/1934926523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 0x7f9b5c0b6af0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f9b44002f70 tx=0x7f9b44006df0).stop 2022-07-21T01:35:08.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1934926523 shutdown_connections 2022-07-21T01:35:08.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 --2- 172.21.15.3:0/1934926523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 0x7f9b5c0b6af0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 --2- 172.21.15.3:0/1934926523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 0x7f9b5c0b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1934926523 >> 172.21.15.3:0/1934926523 conn(0x7f9b5c05c620 msgr2=0x7f9b5c05ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:08.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1934926523 shutdown_connections 2022-07-21T01:35:08.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1934926523 wait complete. 2022-07-21T01:35:08.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 Processor -- start 2022-07-21T01:35:08.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.813+0000 7f9b60ab8700 1 -- start start 2022-07-21T01:35:08.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b60ab8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b5c061ec0 0x7f9b5c1264f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b60ab8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 0x7f9b5c14a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b60ab8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 0x7f9b5c0b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b60ab8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0b38b0 con 0x7f9b5c0b4550 2022-07-21T01:35:08.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b60ab8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0b3b10 con 0x7f9b5c061ec0 2022-07-21T01:35:08.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b60ab8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0b2c80 con 0x7f9b5c0b6710 2022-07-21T01:35:08.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b59d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 0x7f9b5c14a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b59d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 0x7f9b5c14a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44384/0 (socket says 172.21.15.3:44384) 2022-07-21T01:35:08.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b59d9b700 1 -- 172.21.15.3:0/1172296546 learned_addr learned my addr 172.21.15.3:0/1172296546 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:08.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b5a59c700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b5c061ec0 0x7f9b5c1264f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b5ad9d700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 0x7f9b5c0b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b59d9b700 1 -- 172.21.15.3:0/1172296546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 msgr2=0x7f9b5c0b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b59d9b700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 0x7f9b5c0b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b59d9b700 1 -- 172.21.15.3:0/1172296546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b5c061ec0 msgr2=0x7f9b5c1264f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:08.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b59d9b700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b5c061ec0 0x7f9b5c1264f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:08.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b59d9b700 1 -- 172.21.15.3:0/1172296546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4400b180 con 0x7f9b5c0b4550 2022-07-21T01:35:08.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b5ad9d700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 0x7f9b5c0b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:35:08.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b59d9b700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 0x7f9b5c14a000 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f9b44002f70 tx=0x7f9b44014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:08.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b537fe700 1 -- 172.21.15.3:0/1172296546 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b44013ac0 con 0x7f9b5c0b4550 2022-07-21T01:35:08.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b537fe700 1 -- 172.21.15.3:0/1172296546 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b44013c20 con 0x7f9b5c0b4550 2022-07-21T01:35:08.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5c0b2f20 con 0x7f9b5c0b4550 2022-07-21T01:35:08.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b537fe700 1 -- 172.21.15.3:0/1172296546 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b4401e400 con 0x7f9b5c0b4550 2022-07-21T01:35:08.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.817+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5c126b10 con 0x7f9b5c0b4550 2022-07-21T01:35:08.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.821+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5c0b30b0 con 0x7f9b5c0b4550 2022-07-21T01:35:08.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.821+0000 7f9b537fe700 1 -- 172.21.15.3:0/1172296546 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9b4401c070 con 0x7f9b5c0b4550 2022-07-21T01:35:08.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.821+0000 7f9b537fe700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b40053a70 0x7f9b40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:08.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.821+0000 7f9b537fe700 1 -- 172.21.15.3:0/1172296546 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f9b4406cb00 con 0x7f9b5c0b4550 2022-07-21T01:35:08.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.821+0000 7f9b5a59c700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b40053a70 0x7f9b40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:08.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.821+0000 7f9b5a59c700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b40053a70 0x7f9b40055f30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f9b4c00b000 tx=0x7f9b4c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:08.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.829+0000 7f9b537fe700 1 -- 172.21.15.3:0/1172296546 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9b440438b0 con 0x7f9b5c0b4550 2022-07-21T01:35:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:08.957+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9b5c061780 con 0x7f9b5c0b4550 2022-07-21T01:35:09.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.669+0000 7f9b537fe700 1 -- 172.21.15.3:0/1172296546 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v39) v1 ==== 185+0+0 (secure 0 0 0) 0x7f9b44023020 con 0x7f9b5c0b4550 2022-07-21T01:35:09.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b40053a70 msgr2=0x7f9b40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:09.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b40053a70 0x7f9b40055f30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f9b4c00b000 tx=0x7f9b4c006df0).stop 2022-07-21T01:35:09.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 msgr2=0x7f9b5c14a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:09.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 0x7f9b5c14a000 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f9b44002f70 tx=0x7f9b44014000).stop 2022-07-21T01:35:09.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 shutdown_connections 2022-07-21T01:35:09.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b5c0b6710 0x7f9b5c0b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b5c0b4550 0x7f9b5c14a000 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b40053a70 0x7f9b40055f30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 --2- 172.21.15.3:0/1172296546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b5c061ec0 0x7f9b5c1264f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 >> 172.21.15.3:0/1172296546 conn(0x7f9b5c05c620 msgr2=0x7f9b5c128b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 shutdown_connections 2022-07-21T01:35:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.673+0000 7f9b60ab8700 1 -- 172.21.15.3:0/1172296546 wait complete. 2022-07-21T01:35:09.679 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:35:09.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:35:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.805+0000 7fbaf9393700 1 Processor -- start 2022-07-21T01:35:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.805+0000 7fbaf9393700 1 -- start start 2022-07-21T01:35:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbaf40b0630 0x7fbaf40adec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbaf40aa2b0 0x7fbaf40ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40b0a10 0x7fbaf41438a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:09.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf40b2c80 con 0x7fbaf40b0630 2022-07-21T01:35:09.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf40b2900 con 0x7fbaf40aa2b0 2022-07-21T01:35:09.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf40b2eb0 con 0x7fbaf40b0a10 2022-07-21T01:35:09.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf2ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbaf40b0630 0x7fbaf40adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:09.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf2ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbaf40b0630 0x7fbaf40adec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44390/0 (socket says 172.21.15.3:44390) 2022-07-21T01:35:09.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf2ffd700 1 -- 172.21.15.3:0/3975483743 learned_addr learned my addr 172.21.15.3:0/3975483743 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:09.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf37fe700 1 --2- 172.21.15.3:0/3975483743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40b0a10 0x7fbaf41438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:09.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaebfff700 1 --2- 172.21.15.3:0/3975483743 >> v2:172.21.15.143:3300/0 conn(0x7fbaf40aa2b0 0x7fbaf40ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:09.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf2ffd700 1 -- 172.21.15.3:0/3975483743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40b0a10 msgr2=0x7fbaf41438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:09.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf2ffd700 1 --2- 172.21.15.3:0/3975483743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40b0a10 0x7fbaf41438a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf2ffd700 1 -- 172.21.15.3:0/3975483743 >> v2:172.21.15.143:3300/0 conn(0x7fbaf40aa2b0 msgr2=0x7fbaf40ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:09.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf2ffd700 1 --2- 172.21.15.3:0/3975483743 >> v2:172.21.15.143:3300/0 conn(0x7fbaf40aa2b0 0x7fbaf40ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf2ffd700 1 -- 172.21.15.3:0/3975483743 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf40a9d00 con 0x7fbaf40b0630 2022-07-21T01:35:09.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaebfff700 1 --2- 172.21.15.3:0/3975483743 >> v2:172.21.15.143:3300/0 conn(0x7fbaf40aa2b0 0x7fbaf40ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:09.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf2ffd700 1 --2- 172.21.15.3:0/3975483743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0630 0x7fbaf40adec0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fbae4002f70 tx=0x7fbae4006f40).ready entity=mon.0 client_cookie=f09ed78356365f77 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:09.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf27fc700 1 -- 172.21.15.3:0/3975483743 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbae400da80 con 0x7fbaf40b0630 2022-07-21T01:35:09.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf27fc700 1 -- 172.21.15.3:0/3975483743 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae4000e90 con 0x7fbaf40b0630 2022-07-21T01:35:09.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf27fc700 1 -- 172.21.15.3:0/3975483743 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbae400de10 con 0x7fbaf40b0630 2022-07-21T01:35:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 -- 172.21.15.3:0/3975483743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0630 msgr2=0x7fbaf40adec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 --2- 172.21.15.3:0/3975483743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0630 0x7fbaf40adec0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fbae4002f70 tx=0x7fbae4006f40).stop 2022-07-21T01:35:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 -- 172.21.15.3:0/3975483743 shutdown_connections 2022-07-21T01:35:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 --2- 172.21.15.3:0/3975483743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40b0a10 0x7fbaf41438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 --2- 172.21.15.3:0/3975483743 >> v2:172.21.15.143:3300/0 conn(0x7fbaf40aa2b0 0x7fbaf40ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 --2- 172.21.15.3:0/3975483743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0630 0x7fbaf40adec0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 -- 172.21.15.3:0/3975483743 >> 172.21.15.3:0/3975483743 conn(0x7fbaf4060950 msgr2=0x7fbaf4060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:09.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 -- 172.21.15.3:0/3975483743 shutdown_connections 2022-07-21T01:35:09.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.809+0000 7fbaf9393700 1 -- 172.21.15.3:0/3975483743 wait complete. 2022-07-21T01:35:09.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf9393700 1 Processor -- start 2022-07-21T01:35:09.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf9393700 1 -- start start 2022-07-21T01:35:09.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf9393700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40aa2b0 0x7fbaf40671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:09.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf9393700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaf40b0630 0x7fbaf40676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:09.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf9393700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0a10 0x7fbaf4151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:09.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf9393700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf4067bc0 con 0x7fbaf40b0a10 2022-07-21T01:35:09.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf9393700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf4143850 con 0x7fbaf40b0630 2022-07-21T01:35:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf9393700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf40ac380 con 0x7fbaf40aa2b0 2022-07-21T01:35:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0a10 0x7fbaf4151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0a10 0x7fbaf4151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44398/0 (socket says 172.21.15.3:44398) 2022-07-21T01:35:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf37fe700 1 -- 172.21.15.3:0/4197425265 learned_addr learned my addr 172.21.15.3:0/4197425265 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaf2ffd700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40aa2b0 0x7fbaf40671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.813+0000 7fbaebfff700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaf40b0630 0x7fbaf40676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf37fe700 1 -- 172.21.15.3:0/4197425265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40aa2b0 msgr2=0x7fbaf40671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf37fe700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40aa2b0 0x7fbaf40671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf37fe700 1 -- 172.21.15.3:0/4197425265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaf40b0630 msgr2=0x7fbaf40676d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf37fe700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaf40b0630 0x7fbaf40676d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf37fe700 1 -- 172.21.15.3:0/4197425265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae400b460 con 0x7fbaf40b0a10 2022-07-21T01:35:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf37fe700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0a10 0x7fbaf4151c10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fbaf4145200 tx=0x7fbaec0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf0ff9700 1 -- 172.21.15.3:0/4197425265 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaec012600 con 0x7fbaf40b0a10 2022-07-21T01:35:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf0ff9700 1 -- 172.21.15.3:0/4197425265 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec012760 con 0x7fbaf40b0a10 2022-07-21T01:35:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf4063be0 con 0x7fbaf40b0a10 2022-07-21T01:35:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf0ff9700 1 -- 172.21.15.3:0/4197425265 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaec012ac0 con 0x7fbaf40b0a10 2022-07-21T01:35:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf41420a0 con 0x7fbaf40b0a10 2022-07-21T01:35:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf41439e0 con 0x7fbaf40b0a10 2022-07-21T01:35:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf0ff9700 1 -- 172.21.15.3:0/4197425265 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbaec022070 con 0x7fbaf40b0a10 2022-07-21T01:35:09.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf0ff9700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbad8053b30 0x7fbad8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:09.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.817+0000 7fbaf0ff9700 1 -- 172.21.15.3:0/4197425265 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6068+0+0 (secure 0 0 0) 0x7fbaec06c570 con 0x7fbaf40b0a10 2022-07-21T01:35:09.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.821+0000 7fbaf2ffd700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbad8053b30 0x7fbad8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.821+0000 7fbaf2ffd700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbad8053b30 0x7fbad8055ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fbae401e000 tx=0x7fbae4006f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.821+0000 7fbaf0ff9700 1 -- 172.21.15.3:0/4197425265 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbaec0432a0 con 0x7fbaf40b0a10 2022-07-21T01:35:09.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:09.949+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbaf40574e0 con 0x7fbaf40b0a10 2022-07-21T01:35:10.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.681+0000 7fbaf0ff9700 1 -- 172.21.15.3:0/4197425265 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v40) v1 ==== 173+0+0 (secure 0 0 0) 0x7fbaec016070 con 0x7fbaf40b0a10 2022-07-21T01:35:10.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbad8053b30 msgr2=0x7fbad8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:10.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbad8053b30 0x7fbad8055ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fbae401e000 tx=0x7fbae4006f40).stop 2022-07-21T01:35:10.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0a10 msgr2=0x7fbaf4151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:10.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0a10 0x7fbaf4151c10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fbaf4145200 tx=0x7fbaec0091d0).stop 2022-07-21T01:35:10.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 shutdown_connections 2022-07-21T01:35:10.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbaf40b0a10 0x7fbaf4151c10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbad8053b30 0x7fbad8055ff0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaf40b0630 0x7fbaf40676d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 --2- 172.21.15.3:0/4197425265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaf40aa2b0 0x7fbaf40671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 >> 172.21.15.3:0/4197425265 conn(0x7fbaf4060950 msgr2=0x7fbaf40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:10.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.685+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 shutdown_connections 2022-07-21T01:35:10.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.689+0000 7fbaf9393700 1 -- 172.21.15.3:0/4197425265 wait complete. 2022-07-21T01:35:10.693 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:35:10.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:35:10.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 Processor -- start 2022-07-21T01:35:10.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 -- start start 2022-07-21T01:35:10.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f19e40af490 0x7f19e40b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:10.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e40b5a10 con 0x7f19e4067300 2022-07-21T01:35:10.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e40aff00 con 0x7f19e40b5630 2022-07-21T01:35:10.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e40b0e40 con 0x7f19e40af490 2022-07-21T01:35:10.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f19e40af490 0x7f19e40b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:10.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e8fa7700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f19e4067300 0x7f19e4067750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52686/0 (socket says 172.21.15.3:52686) 2022-07-21T01:35:10.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f19e40af490 0x7f19e40b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44406/0 (socket says 172.21.15.3:44406) 2022-07-21T01:35:10.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e8fa7700 1 -- 172.21.15.3:0/1465729091 learned_addr learned my addr 172.21.15.3:0/1465729091 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:10.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e2ffd700 1 -- 172.21.15.3:0/1465729091 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982233838 0 0) 0x7f19e40b5a10 con 0x7f19e4067300 2022-07-21T01:35:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e2ffd700 1 -- 172.21.15.3:0/1465729091 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d0003260 con 0x7f19e4067300 2022-07-21T01:35:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e37fe700 1 -- 172.21.15.3:0/1465729091 >> v1:172.21.15.143:6789/0 conn(0x7f19e40b5630 legacy=0x7f19e41447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e37fe700 1 -- 172.21.15.3:0/1465729091 >> v1:172.21.15.3:6789/0 conn(0x7f19e4067300 legacy=0x7f19e4067750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e37fe700 1 -- 172.21.15.3:0/1465729091 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e40b10e0 con 0x7f19e40af490 2022-07-21T01:35:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e37fe700 1 --2- 172.21.15.3:0/1465729091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e40af490 0x7f19e40b1580 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f19d8002f70 tx=0x7f19d8006df0).ready entity=mon.0 client_cookie=f869db04f9b0af64 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:10.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e2ffd700 1 -- 172.21.15.3:0/1465729091 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19d800d610 con 0x7f19e40af490 2022-07-21T01:35:10.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e2ffd700 1 -- 172.21.15.3:0/1465729091 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8000e90 con 0x7f19e40af490 2022-07-21T01:35:10.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e2ffd700 1 -- 172.21.15.3:0/1465729091 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19d800d960 con 0x7f19e40af490 2022-07-21T01:35:10.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 -- 172.21.15.3:0/1465729091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e40af490 msgr2=0x7f19e40b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:10.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 --2- 172.21.15.3:0/1465729091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e40af490 0x7f19e40b1580 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f19d8002f70 tx=0x7f19d8006df0).stop 2022-07-21T01:35:10.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 -- 172.21.15.3:0/1465729091 shutdown_connections 2022-07-21T01:35:10.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 --2- 172.21.15.3:0/1465729091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e40af490 0x7f19e40b1580 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 -- 172.21.15.3:0/1465729091 >> 172.21.15.3:0/1465729091 conn(0x7f19e40626f0 msgr2=0x7f19e4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 -- 172.21.15.3:0/1465729091 shutdown_connections 2022-07-21T01:35:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.813+0000 7f19e97a8700 1 -- 172.21.15.3:0/1465729091 wait complete. 2022-07-21T01:35:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 Processor -- start 2022-07-21T01:35:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 -- start start 2022-07-21T01:35:10.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e4067300 0x7f19e4066570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:10.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e40af490 0x7f19e40a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:10.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e40b5630 0x7f19e413f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:10.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e40a4810 con 0x7f19e4067300 2022-07-21T01:35:10.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e40a4ab0 con 0x7f19e40af490 2022-07-21T01:35:10.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e40a4d50 con 0x7f19e40b5630 2022-07-21T01:35:10.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e4067300 0x7f19e4066570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:10.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e4067300 0x7f19e4066570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44410/0 (socket says 172.21.15.3:44410) 2022-07-21T01:35:10.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e3fff700 1 -- 172.21.15.3:0/2695607984 learned_addr learned my addr 172.21.15.3:0/2695607984 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:10.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e37fe700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e40af490 0x7f19e40a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:10.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e8fa7700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e40b5630 0x7f19e413f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:10.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e3fff700 1 -- 172.21.15.3:0/2695607984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e40b5630 msgr2=0x7f19e413f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:10.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e3fff700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e40b5630 0x7f19e413f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e3fff700 1 -- 172.21.15.3:0/2695607984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e40af490 msgr2=0x7f19e40a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:10.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e3fff700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e40af490 0x7f19e40a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e3fff700 1 -- 172.21.15.3:0/2695607984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d800b180 con 0x7f19e4067300 2022-07-21T01:35:10.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e3fff700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e4067300 0x7f19e4066570 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f19e4069f60 tx=0x7f19d4007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:10.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e17fa700 1 -- 172.21.15.3:0/2695607984 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19d400ce30 con 0x7f19e4067300 2022-07-21T01:35:10.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e17fa700 1 -- 172.21.15.3:0/2695607984 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d400f470 con 0x7f19e4067300 2022-07-21T01:35:10.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e40a4ff0 con 0x7f19e4067300 2022-07-21T01:35:10.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e17fa700 1 -- 172.21.15.3:0/2695607984 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19d400f7d0 con 0x7f19e4067300 2022-07-21T01:35:10.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.817+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e40a0ff0 con 0x7f19e4067300 2022-07-21T01:35:10.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.821+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e4144ed0 con 0x7f19e4067300 2022-07-21T01:35:10.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.821+0000 7f19e17fa700 1 -- 172.21.15.3:0/2695607984 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f19d4019030 con 0x7f19e4067300 2022-07-21T01:35:10.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.821+0000 7f19e17fa700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19c4053b00 0x7f19c4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:10.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.821+0000 7f19e17fa700 1 -- 172.21.15.3:0/2695607984 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f19d406b380 con 0x7f19e4067300 2022-07-21T01:35:10.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.821+0000 7f19e37fe700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19c4053b00 0x7f19c4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:10.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.821+0000 7f19e37fe700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19c4053b00 0x7f19c4055fc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f19d8018000 tx=0x7f19d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:10.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.829+0000 7f19e17fa700 1 -- 172.21.15.3:0/2695607984 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f19d4042310 con 0x7f19e4067300 2022-07-21T01:35:10.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.953+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f19e40574e0 con 0x7f19e4067300 2022-07-21T01:35:10.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e17fa700 1 -- 172.21.15.3:0/2695607984 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7f19d4045930 con 0x7f19e4067300 2022-07-21T01:35:10.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19c4053b00 msgr2=0x7f19c4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19c4053b00 0x7f19c4055fc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f19d8018000 tx=0x7f19d8014000).stop 2022-07-21T01:35:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e4067300 msgr2=0x7f19e4066570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e4067300 0x7f19e4066570 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f19e4069f60 tx=0x7f19d4007c50).stop 2022-07-21T01:35:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 shutdown_connections 2022-07-21T01:35:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e40b5630 0x7f19e413f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19c4053b00 0x7f19c4055fc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e40af490 0x7f19e40a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 --2- 172.21.15.3:0/2695607984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19e4067300 0x7f19e4066570 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 >> 172.21.15.3:0/2695607984 conn(0x7f19e40626f0 msgr2=0x7f19e4141b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:10.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 shutdown_connections 2022-07-21T01:35:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:10.961+0000 7f19e97a8700 1 -- 172.21.15.3:0/2695607984 wait complete. 2022-07-21T01:35:10.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-07-21T01:35:11.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f315dbf2700 1 Processor -- start 2022-07-21T01:35:11.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f315dbf2700 1 -- start start 2022-07-21T01:35:11.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f315dbf2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31580b1b30 0x7f3158067150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f315dbf2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31580ac030 con 0x7f31580a9930 2022-07-21T01:35:11.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f315dbf2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3158140490 con 0x7f3158060410 2022-07-21T01:35:11.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f315dbf2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31580ab4a0 con 0x7f31580b1b30 2022-07-21T01:35:11.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3157fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f31580a9930 0x7f31580a9dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52702/0 (socket says 172.21.15.3:52702) 2022-07-21T01:35:11.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3157fff700 1 -- 172.21.15.3:0/3131219556 learned_addr learned my addr 172.21.15.3:0/3131219556 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:11.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f314effd700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31580b1b30 0x7f3158067150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038394843 0 0) 0x7f31580ac030 con 0x7f31580a9930 2022-07-21T01:35:11.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3138003260 con 0x7f31580a9930 2022-07-21T01:35:11.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580515994 0 0) 0x7f3158140490 con 0x7f3158060410 2022-07-21T01:35:11.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31580ac030 con 0x7f3158060410 2022-07-21T01:35:11.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1724187034 0 0) 0x7f3138003260 con 0x7f31580a9930 2022-07-21T01:35:11.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31580b1b30 msgr2=0x7f3158067150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:11.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31580b1b30 0x7f3158067150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 >> v1:172.21.15.143:6789/0 conn(0x7f3158060410 legacy=0x7f3158066a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158140730 con 0x7f31580a9930 2022-07-21T01:35:11.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f31500019e0 con 0x7f31580a9930 2022-07-21T01:35:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 >> v1:172.21.15.3:6789/0 conn(0x7f31580a9930 legacy=0x7f31580a9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3138003910 0x7f3138003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31580b1b30 0x7f3138004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.073+0000 7f3156ffd700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31380049f0 0x7f3138004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31580ab4a0 con 0x7f3138003910 2022-07-21T01:35:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313800b7b0 con 0x7f31380049f0 2022-07-21T01:35:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313800b970 con 0x7f31580b1b30 2022-07-21T01:35:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f31577fe700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3138003910 0x7f3138003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f314effd700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31580b1b30 0x7f3138004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f3157fff700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31380049f0 0x7f3138004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f31577fe700 1 -- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31580b1b30 msgr2=0x7f3138004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f31577fe700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31580b1b30 0x7f3138004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f31577fe700 1 -- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31380049f0 msgr2=0x7f3138004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f31577fe700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31380049f0 0x7f3138004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f31577fe700 1 -- 172.21.15.3:0/3131219556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3138003260 con 0x7f3138003910 2022-07-21T01:35:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f3157fff700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31380049f0 0x7f3138004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:35:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f31577fe700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3138003910 0x7f3138003d10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f3148001110 tx=0x7f3148007a00).ready entity=mon.0 client_cookie=e1aabb7f99502af8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3148007d10 con 0x7f3138003910 2022-07-21T01:35:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3148007030 con 0x7f3138003910 2022-07-21T01:35:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f3156ffd700 1 -- 172.21.15.3:0/3131219556 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3148007390 con 0x7f3138003910 2022-07-21T01:35:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f315dbf2700 1 -- 172.21.15.3:0/3131219556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3138003910 msgr2=0x7f3138003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f315dbf2700 1 --2- 172.21.15.3:0/3131219556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3138003910 0x7f3138003d10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f3148001110 tx=0x7f3148007a00).stop 2022-07-21T01:35:11.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f3157fff700 1 -- 172.21.15.3:0/3131219556 reap_dead start 2022-07-21T01:35:11.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f315dbf2700 1 -- 172.21.15.3:0/3131219556 shutdown_connections 2022-07-21T01:35:11.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f315dbf2700 1 -- 172.21.15.3:0/3131219556 >> 172.21.15.3:0/3131219556 conn(0x7f31580b5570 msgr2=0x7f31580b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:11.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f315dbf2700 1 -- 172.21.15.3:0/3131219556 shutdown_connections 2022-07-21T01:35:11.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f315dbf2700 1 -- 172.21.15.3:0/3131219556 wait complete. 2022-07-21T01:35:11.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f315dbf2700 1 Processor -- start 2022-07-21T01:35:11.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.077+0000 7f315dbf2700 1 -- start start 2022-07-21T01:35:11.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f315dbf2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3138003910 0x7f31580ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f315dbf2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31581309e0 0x7f31581354f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f315dbf2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31581359e0 0x7f315812d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f315dbf2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315812f4c0 con 0x7f31581359e0 2022-07-21T01:35:11.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f315dbf2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315812eff0 con 0x7f3138003910 2022-07-21T01:35:11.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f315dbf2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31580673b0 con 0x7f31581309e0 2022-07-21T01:35:11.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f3157fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31581359e0 0x7f315812d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f3157fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31581359e0 0x7f315812d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44432/0 (socket says 172.21.15.3:44432) 2022-07-21T01:35:11.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f3157fff700 1 -- 172.21.15.3:0/2545909803 learned_addr learned my addr 172.21.15.3:0/2545909803 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:11.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f31577fe700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3138003910 0x7f31580ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f314effd700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31581309e0 0x7f31581354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f3157fff700 1 -- 172.21.15.3:0/2545909803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31581309e0 msgr2=0x7f31581354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f3157fff700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31581309e0 0x7f31581354f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f3157fff700 1 -- 172.21.15.3:0/2545909803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3138003910 msgr2=0x7f31580ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f3157fff700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3138003910 0x7f31580ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f3157fff700 1 -- 172.21.15.3:0/2545909803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31580acba0 con 0x7f31581359e0 2022-07-21T01:35:11.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f3157fff700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31581359e0 0x7f315812d670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f3150001bc0 tx=0x7f3150009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:11.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f31557fa700 1 -- 172.21.15.3:0/2545909803 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3150004ab0 con 0x7f31581359e0 2022-07-21T01:35:11.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f31557fa700 1 -- 172.21.15.3:0/2545909803 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3150004c10 con 0x7f31581359e0 2022-07-21T01:35:11.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3158143830 con 0x7f31581359e0 2022-07-21T01:35:11.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f31557fa700 1 -- 172.21.15.3:0/2545909803 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3150007170 con 0x7f31581359e0 2022-07-21T01:35:11.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.081+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315812f220 con 0x7f31581359e0 2022-07-21T01:35:11.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.085+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31580a9490 con 0x7f31581359e0 2022-07-21T01:35:11.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.085+0000 7f31557fa700 1 -- 172.21.15.3:0/2545909803 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f31500072d0 con 0x7f31581359e0 2022-07-21T01:35:11.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.085+0000 7f31557fa700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f313c053950 0x7f313c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.085+0000 7f31557fa700 1 -- 172.21.15.3:0/2545909803 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f315006be60 con 0x7f31581359e0 2022-07-21T01:35:11.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.085+0000 7f31577fe700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f313c053950 0x7f313c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.085+0000 7f31577fe700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f313c053950 0x7f313c055e10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f314801b000 tx=0x7f3148016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:11.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.093+0000 7f31557fa700 1 -- 172.21.15.3:0/2545909803 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3150042d70 con 0x7f31581359e0 2022-07-21T01:35:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.257+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7f31580574e0 con 0x7f31581359e0 2022-07-21T01:35:11.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.685+0000 7f31557fa700 1 -- 172.21.15.3:0/2545909803 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v96) v1 ==== 139+0+0 (secure 0 0 0) 0x7f3150046390 con 0x7f31581359e0 2022-07-21T01:35:11.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f313c053950 msgr2=0x7f313c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f313c053950 0x7f313c055e10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f314801b000 tx=0x7f3148016000).stop 2022-07-21T01:35:11.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31581359e0 msgr2=0x7f315812d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31581359e0 0x7f315812d670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f3150001bc0 tx=0x7f3150009040).stop 2022-07-21T01:35:11.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 shutdown_connections 2022-07-21T01:35:11.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31581359e0 0x7f315812d670 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31581309e0 0x7f31581354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f313c053950 0x7f313c055e10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 --2- 172.21.15.3:0/2545909803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3138003910 0x7f31580ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 >> 172.21.15.3:0/2545909803 conn(0x7f31580b5570 msgr2=0x7f31580afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:11.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 shutdown_connections 2022-07-21T01:35:11.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.689+0000 7f315dbf2700 1 -- 172.21.15.3:0/2545909803 wait complete. 2022-07-21T01:35:11.707 INFO:tasks.cephfs_test_runner:test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-21T01:35:11.709 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-21T01:35:11.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-07-21T01:35:11.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f215489a700 1 Processor -- start 2022-07-21T01:35:11.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f215489a700 1 -- start start 2022-07-21T01:35:11.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f215489a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f21500b07a0 0x7f21500af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f215489a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2150147730 0x7f21500acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f215489a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500b5a10 con 0x7f21500b5630 2022-07-21T01:35:11.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f215489a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2150145470 con 0x7f21500b07a0 2022-07-21T01:35:11.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f215489a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21500ae180 con 0x7f2150147730 2022-07-21T01:35:11.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f214e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f21500b07a0 0x7f21500af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f214e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f21500b07a0 0x7f21500af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:49888/0 (socket says 172.21.15.3:49888) 2022-07-21T01:35:11.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f214e59c700 1 -- 172.21.15.3:0/1973042053 learned_addr learned my addr 172.21.15.3:0/1973042053 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:11.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f214f59e700 1 --2- 172.21.15.3:0/1973042053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2150147730 0x7f21500acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f214e59c700 1 -- 172.21.15.3:0/1973042053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2150147730 msgr2=0x7f21500acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:11.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f214e59c700 1 --2- 172.21.15.3:0/1973042053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2150147730 0x7f21500acdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f214e59c700 1 -- 172.21.15.3:0/1973042053 >> v1:172.21.15.143:6789/0 conn(0x7f21500b5630 legacy=0x7f2150067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f214e59c700 1 -- 172.21.15.3:0/1973042053 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21500ac990 con 0x7f21500b07a0 2022-07-21T01:35:11.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.813+0000 7f214e59c700 1 --2- 172.21.15.3:0/1973042053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21500b07a0 0x7f21500af6e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f21400032f0 tx=0x7f2140009290).ready entity=mon.1 client_cookie=d70ae398bba188f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:11.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f214dd9b700 1 -- 172.21.15.3:0/1973042053 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f214000b870 con 0x7f21500b07a0 2022-07-21T01:35:11.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f214dd9b700 1 -- 172.21.15.3:0/1973042053 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214000b9d0 con 0x7f21500b07a0 2022-07-21T01:35:11.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f214dd9b700 1 -- 172.21.15.3:0/1973042053 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f214000bb40 con 0x7f21500b07a0 2022-07-21T01:35:11.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f215489a700 1 -- 172.21.15.3:0/1973042053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21500b07a0 msgr2=0x7f21500af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f215489a700 1 --2- 172.21.15.3:0/1973042053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21500b07a0 0x7f21500af6e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f21400032f0 tx=0x7f2140009290).stop 2022-07-21T01:35:11.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f215489a700 1 -- 172.21.15.3:0/1973042053 shutdown_connections 2022-07-21T01:35:11.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f215489a700 1 --2- 172.21.15.3:0/1973042053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2150147730 0x7f21500acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f215489a700 1 --2- 172.21.15.3:0/1973042053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21500b07a0 0x7f21500af6e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f215489a700 1 -- 172.21.15.3:0/1973042053 >> 172.21.15.3:0/1973042053 conn(0x7f21500626f0 msgr2=0x7f2150062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:11.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f215489a700 1 -- 172.21.15.3:0/1973042053 shutdown_connections 2022-07-21T01:35:11.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.817+0000 7f215489a700 1 -- 172.21.15.3:0/1973042053 wait complete. 2022-07-21T01:35:11.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 Processor -- start 2022-07-21T01:35:11.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 -- start start 2022-07-21T01:35:11.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21500b07a0 0x7f2150146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21500b5630 0x7f21501572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2150147730 0x7f215015a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21501548b0 con 0x7f21500b5630 2022-07-21T01:35:11.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21500b4770 con 0x7f2150147730 2022-07-21T01:35:11.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21500b4a10 con 0x7f21500b07a0 2022-07-21T01:35:11.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21500b5630 0x7f21501572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21500b5630 0x7f21501572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44446/0 (socket says 172.21.15.3:44446) 2022-07-21T01:35:11.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214e59c700 1 -- 172.21.15.3:0/1300004157 learned_addr learned my addr 172.21.15.3:0/1300004157 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:11.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214f59e700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2150147730 0x7f215015a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214ed9d700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21500b07a0 0x7f2150146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214e59c700 1 -- 172.21.15.3:0/1300004157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21500b07a0 msgr2=0x7f2150146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214e59c700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21500b07a0 0x7f2150146880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214e59c700 1 -- 172.21.15.3:0/1300004157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2150147730 msgr2=0x7f215015a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:11.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214e59c700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2150147730 0x7f215015a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:11.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214e59c700 1 -- 172.21.15.3:0/1300004157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2140009530 con 0x7f21500b5630 2022-07-21T01:35:11.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f214e59c700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21500b5630 0x7f21501572c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f21400032f0 tx=0x7f2140016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:11.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f2147fff700 1 -- 172.21.15.3:0/1300004157 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2140021440 con 0x7f21500b5630 2022-07-21T01:35:11.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f2147fff700 1 -- 172.21.15.3:0/1300004157 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21400215a0 con 0x7f21500b5630 2022-07-21T01:35:11.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2150156b80 con 0x7f21500b5630 2022-07-21T01:35:11.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f2147fff700 1 -- 172.21.15.3:0/1300004157 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2140021900 con 0x7f21500b5630 2022-07-21T01:35:11.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.821+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2150066cb0 con 0x7f21500b5630 2022-07-21T01:35:11.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.825+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21500574e0 con 0x7f21500b5630 2022-07-21T01:35:11.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.825+0000 7f2147fff700 1 -- 172.21.15.3:0/1300004157 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f214000f070 con 0x7f21500b5630 2022-07-21T01:35:11.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.825+0000 7f2147fff700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2134053aa0 0x7f2134055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:11.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.825+0000 7f2147fff700 1 -- 172.21.15.3:0/1300004157 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f214006c310 con 0x7f21500b5630 2022-07-21T01:35:11.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.825+0000 7f214ed9d700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2134053aa0 0x7f2134055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:11.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.825+0000 7f214ed9d700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2134053aa0 0x7f2134055f60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f213800d000 tx=0x7f2138000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:11.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:11.833+0000 7f2147fff700 1 -- 172.21.15.3:0/1300004157 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f21400433b0 con 0x7f21500b5630 2022-07-21T01:35:12.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.001+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7f215005f3d0 con 0x7f21500b5630 2022-07-21T01:35:12.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.685+0000 7f2147fff700 1 -- 172.21.15.3:0/1300004157 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v97) v1 ==== 147+0+0 (secure 0 0 0) 0x7f2140015020 con 0x7f21500b5630 2022-07-21T01:35:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2134053aa0 msgr2=0x7f2134055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2134053aa0 0x7f2134055f60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f213800d000 tx=0x7f2138000bc0).stop 2022-07-21T01:35:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21500b5630 msgr2=0x7f21501572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21500b5630 0x7f21501572c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f21400032f0 tx=0x7f2140016000).stop 2022-07-21T01:35:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 shutdown_connections 2022-07-21T01:35:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2134053aa0 0x7f2134055f60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2150147730 0x7f215015a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21500b5630 0x7f21501572c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 --2- 172.21.15.3:0/1300004157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21500b07a0 0x7f2150146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 >> 172.21.15.3:0/1300004157 conn(0x7f21500626f0 msgr2=0x7f2150067660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:12.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 shutdown_connections 2022-07-21T01:35:12.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.693+0000 7f215489a700 1 -- 172.21.15.3:0/1300004157 wait complete. 2022-07-21T01:35:12.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:35:12.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4aef59e700 1 Processor -- start 2022-07-21T01:35:12.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4aef59e700 1 -- start start 2022-07-21T01:35:12.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4aef59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4af00ad750 0x7f4af0061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4aef59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4aef59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af00aee80 con 0x7f4af00614b0 2022-07-21T01:35:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4aef59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af00b70b0 con 0x7f4af00ad750 2022-07-21T01:35:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4aef59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af00af0b0 con 0x7f4af01405d0 2022-07-21T01:35:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4aeed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4ae5d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4af00ad750 0x7f4af0061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4ae5d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4af00ad750 0x7f4af0061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:49904/0 (socket says 172.21.15.3:49904) 2022-07-21T01:35:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4ae5d9b700 1 -- 172.21.15.3:0/603714693 learned_addr learned my addr 172.21.15.3:0/603714693 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:12.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.817+0000 7f4aeed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0140a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:51374/0 (socket says 172.21.15.3:51374) 2022-07-21T01:35:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aeed9d700 1 -- 172.21.15.3:0/603714693 >> v2:172.21.15.143:3300/0 conn(0x7f4af00ad750 msgr2=0x7f4af0061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aeed9d700 1 --2- 172.21.15.3:0/603714693 >> v2:172.21.15.143:3300/0 conn(0x7f4af00ad750 0x7f4af0061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aeed9d700 1 -- 172.21.15.3:0/603714693 >> v1:172.21.15.143:6789/0 conn(0x7f4af00614b0 legacy=0x7f4af00b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:12.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aeed9d700 1 -- 172.21.15.3:0/603714693 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af00af350 con 0x7f4af01405d0 2022-07-21T01:35:12.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aeed9d700 1 --2- 172.21.15.3:0/603714693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0140a80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4ae80032f0 tx=0x7f4ae80091d0).ready entity=mon.2 client_cookie=fd8d8259349f5097 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:12.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aedd9b700 1 -- 172.21.15.3:0/603714693 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ae800b6c0 con 0x7f4af01405d0 2022-07-21T01:35:12.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aedd9b700 1 -- 172.21.15.3:0/603714693 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae800b820 con 0x7f4af01405d0 2022-07-21T01:35:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aedd9b700 1 -- 172.21.15.3:0/603714693 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ae800b990 con 0x7f4af01405d0 2022-07-21T01:35:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aef59e700 1 -- 172.21.15.3:0/603714693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 msgr2=0x7f4af0140a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aef59e700 1 --2- 172.21.15.3:0/603714693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0140a80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4ae80032f0 tx=0x7f4ae80091d0).stop 2022-07-21T01:35:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aef59e700 1 -- 172.21.15.3:0/603714693 shutdown_connections 2022-07-21T01:35:12.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aef59e700 1 --2- 172.21.15.3:0/603714693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0140a80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aef59e700 1 --2- 172.21.15.3:0/603714693 >> v2:172.21.15.143:3300/0 conn(0x7f4af00ad750 0x7f4af0061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aef59e700 1 -- 172.21.15.3:0/603714693 >> 172.21.15.3:0/603714693 conn(0x7f4af0066f70 msgr2=0x7f4af0067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:12.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aef59e700 1 -- 172.21.15.3:0/603714693 shutdown_connections 2022-07-21T01:35:12.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.821+0000 7f4aef59e700 1 -- 172.21.15.3:0/603714693 wait complete. 2022-07-21T01:35:12.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 Processor -- start 2022-07-21T01:35:12.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 -- start start 2022-07-21T01:35:12.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4af00614b0 0x7f4af014a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:12.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af00ad750 0x7f4af0155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:12.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:12.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af00abee0 con 0x7f4af00ad750 2022-07-21T01:35:12.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0152560 con 0x7f4af00614b0 2022-07-21T01:35:12.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0155ab0 con 0x7f4af01405d0 2022-07-21T01:35:12.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae5d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af00ad750 0x7f4af0155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:12.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aee59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4af00614b0 0x7f4af014a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:12.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae5d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af00ad750 0x7f4af0155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44462/0 (socket says 172.21.15.3:44462) 2022-07-21T01:35:12.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aeed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:12.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae5d9b700 1 -- 172.21.15.3:0/682239825 learned_addr learned my addr 172.21.15.3:0/682239825 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:12.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae5d9b700 1 -- 172.21.15.3:0/682239825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 msgr2=0x7f4af0156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:12.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae5d9b700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae5d9b700 1 -- 172.21.15.3:0/682239825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4af00614b0 msgr2=0x7f4af014a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:12.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae5d9b700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4af00614b0 0x7f4af014a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae5d9b700 1 -- 172.21.15.3:0/682239825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8009470 con 0x7f4af00ad750 2022-07-21T01:35:12.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae5d9b700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af00ad750 0x7f4af0155fa0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f4af0145760 tx=0x7f4ad8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:12.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae7fff700 1 -- 172.21.15.3:0/682239825 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ad800fe60 con 0x7f4af00ad750 2022-07-21T01:35:12.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae7fff700 1 -- 172.21.15.3:0/682239825 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad8013610 con 0x7f4af00ad750 2022-07-21T01:35:12.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0156980 con 0x7f4af00ad750 2022-07-21T01:35:12.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4ae7fff700 1 -- 172.21.15.3:0/682239825 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ad8013970 con 0x7f4af00ad750 2022-07-21T01:35:12.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.825+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af00b1180 con 0x7f4af00ad750 2022-07-21T01:35:12.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.829+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af00af5f0 con 0x7f4af00ad750 2022-07-21T01:35:12.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.829+0000 7f4ae7fff700 1 -- 172.21.15.3:0/682239825 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4ad8023070 con 0x7f4af00ad750 2022-07-21T01:35:12.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.829+0000 7f4ae7fff700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ad40539e0 0x7f4ad4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:12.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.829+0000 7f4ae7fff700 1 -- 172.21.15.3:0/682239825 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f4ad806bf10 con 0x7f4af00ad750 2022-07-21T01:35:12.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.829+0000 7f4aee59c700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ad40539e0 0x7f4ad4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:12.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.829+0000 7f4aee59c700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ad40539e0 0x7f4ad4055ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f4ae000d000 tx=0x7f4ae0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:12.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.833+0000 7f4ae7fff700 1 -- 172.21.15.3:0/682239825 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ad8042fb0 con 0x7f4af00ad750 2022-07-21T01:35:12.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.965+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4af00574e0 con 0x7f4af00ad750 2022-07-21T01:35:12.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.973+0000 7f4ae7fff700 1 -- 172.21.15.3:0/682239825 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4ad80465d0 con 0x7f4af00ad750 2022-07-21T01:35:12.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ad40539e0 msgr2=0x7f4ad4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:12.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ad40539e0 0x7f4ad4055ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f4ae000d000 tx=0x7f4ae0007b90).stop 2022-07-21T01:35:12.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af00ad750 msgr2=0x7f4af0155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:12.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af00ad750 0x7f4af0155fa0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f4af0145760 tx=0x7f4ad8008040).stop 2022-07-21T01:35:12.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 shutdown_connections 2022-07-21T01:35:12.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4af01405d0 0x7f4af0156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af00ad750 0x7f4af0155fa0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ad40539e0 0x7f4ad4055ea0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 --2- 172.21.15.3:0/682239825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4af00614b0 0x7f4af014a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:12.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 >> 172.21.15.3:0/682239825 conn(0x7f4af0066f70 msgr2=0x7f4af00b7960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:12.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 shutdown_connections 2022-07-21T01:35:12.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:12.977+0000 7f4aef59e700 1 -- 172.21.15.3:0/682239825 wait complete. 2022-07-21T01:35:12.995 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:35:12.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:35:13.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:35:13.034 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:35:13.054 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:35:13.076 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:35:13.076 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:13.077 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:13.077 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:13.084 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:35:13.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:35:13.086 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:35:13.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:13.087 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:13.087 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:35:13.095 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:35:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:35:13.096 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:35:13.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:35:13.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.209+0000 7f9f4697b700 1 Processor -- start 2022-07-21T01:35:13.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.209+0000 7f9f4697b700 1 -- start start 2022-07-21T01:35:13.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9f400a9da0 0x7f9f40060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9f401402d0 0x7f9f40140720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f40067030 con 0x7f9f400b31d0 2022-07-21T01:35:13.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f40064a70 con 0x7f9f400a9da0 2022-07-21T01:35:13.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f40061f60 con 0x7f9f401402d0 2022-07-21T01:35:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f45178700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9f400a9da0 0x7f9f40060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f45178700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9f400a9da0 0x7f9f40060950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44470/0 (socket says 172.21.15.3:44470) 2022-07-21T01:35:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f45178700 1 -- 172.21.15.3:0/896790295 learned_addr learned my addr 172.21.15.3:0/896790295 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4617a700 1 --2- 172.21.15.3:0/896790295 >> v2:172.21.15.143:3300/0 conn(0x7f9f401402d0 0x7f9f40140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f45178700 1 -- 172.21.15.3:0/896790295 >> v2:172.21.15.143:3300/0 conn(0x7f9f401402d0 msgr2=0x7f9f40140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f45178700 1 --2- 172.21.15.3:0/896790295 >> v2:172.21.15.143:3300/0 conn(0x7f9f401402d0 0x7f9f40140720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f45178700 1 -- 172.21.15.3:0/896790295 >> v1:172.21.15.143:6789/0 conn(0x7f9f400b31d0 legacy=0x7f9f40066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f45178700 1 -- 172.21.15.3:0/896790295 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f400b35b0 con 0x7f9f400a9da0 2022-07-21T01:35:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f45178700 1 --2- 172.21.15.3:0/896790295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f400a9da0 0x7f9f40060950 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f9f3c0032f0 tx=0x7f9f3c009290).ready entity=mon.0 client_cookie=feb399320752f77f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:13.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f44977700 1 -- 172.21.15.3:0/896790295 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3c00b870 con 0x7f9f400a9da0 2022-07-21T01:35:13.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f44977700 1 -- 172.21.15.3:0/896790295 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c00b9d0 con 0x7f9f400a9da0 2022-07-21T01:35:13.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f44977700 1 -- 172.21.15.3:0/896790295 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3c00bb40 con 0x7f9f400a9da0 2022-07-21T01:35:13.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 -- 172.21.15.3:0/896790295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f400a9da0 msgr2=0x7f9f40060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 --2- 172.21.15.3:0/896790295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f400a9da0 0x7f9f40060950 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f9f3c0032f0 tx=0x7f9f3c009290).stop 2022-07-21T01:35:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 -- 172.21.15.3:0/896790295 shutdown_connections 2022-07-21T01:35:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 --2- 172.21.15.3:0/896790295 >> v2:172.21.15.143:3300/0 conn(0x7f9f401402d0 0x7f9f40140720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 --2- 172.21.15.3:0/896790295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f400a9da0 0x7f9f40060950 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 -- 172.21.15.3:0/896790295 >> 172.21.15.3:0/896790295 conn(0x7f9f40063870 msgr2=0x7f9f40063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 -- 172.21.15.3:0/896790295 shutdown_connections 2022-07-21T01:35:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.213+0000 7f9f4697b700 1 -- 172.21.15.3:0/896790295 wait complete. 2022-07-21T01:35:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4697b700 1 Processor -- start 2022-07-21T01:35:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4697b700 1 -- start start 2022-07-21T01:35:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4697b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f400a9da0 0x7f9f401562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4697b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f400b31d0 0x7f9f40143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4697b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f401402d0 0x7f9f4009edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4697b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400a07d0 con 0x7f9f401402d0 2022-07-21T01:35:13.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4697b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400abfb0 con 0x7f9f400b31d0 2022-07-21T01:35:13.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4697b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400aadc0 con 0x7f9f400a9da0 2022-07-21T01:35:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4617a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f401402d0 0x7f9f4009edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4617a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f401402d0 0x7f9f4009edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44478/0 (socket says 172.21.15.3:44478) 2022-07-21T01:35:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f4617a700 1 -- 172.21.15.3:0/3828329059 learned_addr learned my addr 172.21.15.3:0/3828329059 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f45979700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f400a9da0 0x7f9f401562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.217+0000 7f9f45178700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f400b31d0 0x7f9f40143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f4617a700 1 -- 172.21.15.3:0/3828329059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f400a9da0 msgr2=0x7f9f401562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f4617a700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f400a9da0 0x7f9f401562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f4617a700 1 -- 172.21.15.3:0/3828329059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f400b31d0 msgr2=0x7f9f40143310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f4617a700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f400b31d0 0x7f9f40143310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f4617a700 1 -- 172.21.15.3:0/3828329059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c009530 con 0x7f9f401402d0 2022-07-21T01:35:13.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f4617a700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f401402d0 0x7f9f4009edc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f9f3800c000 tx=0x7f9f38006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f36ffd700 1 -- 172.21.15.3:0/3828329059 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f38011750 con 0x7f9f401402d0 2022-07-21T01:35:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f36ffd700 1 -- 172.21.15.3:0/3828329059 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f380118b0 con 0x7f9f401402d0 2022-07-21T01:35:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f400aaff0 con 0x7f9f401402d0 2022-07-21T01:35:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f36ffd700 1 -- 172.21.15.3:0/3828329059 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f38011c10 con 0x7f9f401402d0 2022-07-21T01:35:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f400ab8e0 con 0x7f9f401402d0 2022-07-21T01:35:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f400ab490 con 0x7f9f401402d0 2022-07-21T01:35:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f36ffd700 1 -- 172.21.15.3:0/3828329059 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9f38011d70 con 0x7f9f401402d0 2022-07-21T01:35:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f36ffd700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f200539e0 0x7f9f20055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f45979700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f200539e0 0x7f9f20055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.221+0000 7f9f36ffd700 1 -- 172.21.15.3:0/3828329059 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f9f3806c050 con 0x7f9f401402d0 2022-07-21T01:35:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.225+0000 7f9f45979700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f200539e0 0x7f9f20055ea0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f9f3000d000 tx=0x7f9f30000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:13.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.233+0000 7f9f36ffd700 1 -- 172.21.15.3:0/3828329059 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f380430f0 con 0x7f9f401402d0 2022-07-21T01:35:13.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.361+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f9f40066af0 con 0x7f9f401402d0 2022-07-21T01:35:13.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.361+0000 7f9f36ffd700 1 -- 172.21.15.3:0/3828329059 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v40) v1 ==== 81+0+29213 (secure 0 0 0) 0x7f9f38046710 con 0x7f9f401402d0 2022-07-21T01:35:13.364 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:13.364 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:35:13.364 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 40, 2022-07-21T01:35:13.365 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:35:13.365 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:35:13.365 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:35:10.668859+0000", 2022-07-21T01:35:13.365 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:35:13.366 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:35:13.366 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:35:13.366 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:35:13.366 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:35:13.367 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:35:13.367 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:35:13.367 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:35:13.367 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:35:13.368 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:35:13.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:35:13.368 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:35:13.368 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:35:13.369 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:35:13.369 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:35:13.369 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 6, 2022-07-21T01:35:13.369 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:35:13.370 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:35:13.370 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:35:13.370 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:35:13.370 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:35:13.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.371 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:35:13.371 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:35:13.371 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:35:13.372 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:35:13.372 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:35:13.372 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:35:13.372 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:35:13.373 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:35:13.373 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:35:13.373 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:35:13.373 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:35:13.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:35:13.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:35:13.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:35:13.375 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:35:13.375 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:35:13.375 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:35:13.376 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:35:13.376 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:35:13.376 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:35:13.376 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:35:13.377 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:35:13.377 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:35:13.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.377 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:35:13.378 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:35:13.378 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:35:13.378 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:35:13.379 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:35:13.379 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:35:13.379 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:35:13.379 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:35:13.380 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:35:13.380 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:35:13.380 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:35:13.380 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:35:13.381 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:35:13.381 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:35:13.381 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:35:13.381 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:35:13.382 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:35:13.382 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:35:13.382 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:35:13.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:35:13.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:35:13.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:35:13.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:35:13.384 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:35:13.384 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:35:13.384 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:35:13.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:35:13.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.385 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:35:13.385 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:35:13.386 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:35:13.386 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:35:13.386 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:35:13.386 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:35:13.387 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:35:13.387 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:35:13.387 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:35:13.388 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:35:13.388 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:35:13.388 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:35:13.388 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:35:13.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.389 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:35:13.389 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:35:13.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.390 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:35:13.391 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:35:13.391 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:35:13.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:35:13.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:35:13.392 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:35:13.393 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:35:13.393 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:35:13.394 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:35:13.395 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:35:13.395 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:35:13.396 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:35:13.396 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:35:13.397 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:35:13.398 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:35:13.398 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:35:13.399 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:35:13.399 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:35:13.400 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:35:13.400 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:35:13.401 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:35:13.402 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:35:13.402 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:35:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:35:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:35:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:35:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:35:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:35:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:35:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:35:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:35:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:35:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:35:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:35:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:35:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:35:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:35:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:35:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:35:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:35:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:35:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:35:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:35:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:35:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:35:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:35:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:35:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:35:13.417 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:35:13.417 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:35:13.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.418 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:35:13.419 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:35:13.419 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:35:13.420 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:35:13.420 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:35:13.421 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:35:13.422 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:35:13.422 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:35:13.422 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:35:13.422 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:35:13.423 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:35:13.423 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:35:13.423 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:35:13.423 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:35:13.424 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.424 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.424 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:35:13.424 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:35:13.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.425 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:35:13.425 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:35:13.425 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:13.426 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:13.426 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:13.426 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:13.427 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:13.427 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:13.428 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:35:13.428 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:13.429 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:13.429 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:13.430 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:13.430 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.431 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.432 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:35:13.432 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:13.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.434 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.434 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:35:13.434 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:13.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.435 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.436 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:13.436 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.437 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:35:13.437 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:13.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.438 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.438 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:35:13.438 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:13.439 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.439 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.439 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:13.440 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.441 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:35:13.441 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:13.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.441 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.442 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:35:13.442 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:13.442 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.442 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.443 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:13.443 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.444 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:35:13.444 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:13.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.445 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.445 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:35:13.445 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:35:13.446 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.446 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.446 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:35:13.447 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:35:13.447 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:35:13.447 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:35:13.447 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:13.448 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:13.448 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:13.448 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:35:13.449 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:35:13.449 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:13.450 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:13.450 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:13.450 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:13.450 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:13.451 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:13.451 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:13.451 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:13.451 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:13.452 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:13.452 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:13.452 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.452 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.453 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:35:13.453 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:13.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.454 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:35:13.454 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:13.454 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.455 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.455 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:13.455 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.456 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:35:13.456 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:13.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.457 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:35:13.458 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:13.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.459 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:13.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:35:13.460 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:13.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.461 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.461 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:35:13.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:13.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.462 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.462 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:13.462 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.463 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.463 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:35:13.463 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:13.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.464 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:35:13.465 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:35:13.465 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.465 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.466 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:35:13.466 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:35:13.466 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:35:13.466 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:35:13.467 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:13.467 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:13.467 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:13.467 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:35:13.469 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:35:13.469 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:13.469 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:13.470 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:13.470 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:13.470 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:13.471 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:13.471 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:35:13.471 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 34, 2022-07-21T01:35:13.471 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:13.472 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:13.472 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:13.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.473 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:35:13.473 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:13.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.474 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:35:13.475 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:13.475 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.475 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.476 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:13.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.477 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:35:13.477 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:13.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:35:13.478 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:13.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.479 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.479 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:13.479 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.480 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.480 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:35:13.480 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:13.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.481 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.481 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:35:13.482 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:13.482 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.482 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.483 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:13.483 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.484 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.484 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:35:13.484 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:13.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.485 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:35:13.486 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:35:13.486 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.486 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.487 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:35:13.487 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:35:13.487 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:35:13.487 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:35:13.488 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:13.488 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:13.488 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:13.489 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.490 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:35:13.490 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:35:13.490 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:13.490 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:13.491 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:13.491 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:13.491 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:13.492 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:13.492 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:35:13.492 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:13.492 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:13.493 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:13.493 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:13.493 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:35:13.494 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:13.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.495 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.496 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:35:13.496 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:13.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.496 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.497 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:13.497 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.498 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.498 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:35:13.498 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:13.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.499 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.499 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:35:13.500 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:13.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.500 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.501 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:13.501 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.502 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.502 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:35:13.502 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:13.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:35:13.504 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:13.504 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.504 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.505 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:13.505 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.506 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:35:13.507 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:13.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:35:13.508 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:35:13.509 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.509 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.509 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:35:13.510 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:35:13.510 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:35:13.510 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:35:13.511 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:13.511 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:13.511 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:13.511 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.512 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:35:13.512 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:35:13.513 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:13.513 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:13.513 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:13.513 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:13.514 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:13.514 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:13.514 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:13.515 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:13.524 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:13.525 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:13.525 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:13.525 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.526 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.526 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:35:13.526 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:13.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.527 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:35:13.528 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:13.528 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.528 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.529 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:13.529 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.530 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.530 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:35:13.530 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:13.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.531 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:35:13.531 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:13.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.532 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.532 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:13.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:35:13.534 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:13.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.535 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:35:13.535 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:13.535 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.535 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.536 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:13.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.537 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.537 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:35:13.537 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:13.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.538 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:35:13.538 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:35:13.539 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.539 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.539 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:35:13.540 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:35:13.540 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:35:13.540 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:35:13.540 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:13.541 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:13.541 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:13.541 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.542 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:35:13.542 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:35:13.542 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:13.543 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:13.543 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:13.543 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:13.543 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:13.544 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:13.544 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:13.544 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:13.544 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:13.545 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:13.545 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:13.545 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.546 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.546 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:35:13.546 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:13.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.547 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.547 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:35:13.547 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:13.548 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.548 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.548 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:13.549 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.549 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.549 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:35:13.550 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:13.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.551 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:35:13.551 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:13.551 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.551 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.552 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:13.552 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.553 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.553 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:35:13.553 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:13.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.554 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:35:13.555 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:13.555 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.555 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.556 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:13.556 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.556 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.557 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:35:13.557 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:13.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.558 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:35:13.558 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:35:13.558 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.559 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.559 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:35:13.559 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:35:13.560 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:35:13.560 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:35:13.560 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:13.560 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:13.560 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:13.561 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.561 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:35:13.562 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:35:13.562 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:13.562 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:13.563 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:13.563 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:13.563 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:13.563 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:13.564 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:13.564 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 34, 2022-07-21T01:35:13.564 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:13.564 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:13.564 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:13.565 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.565 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.565 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:35:13.566 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:13.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.566 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.567 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:35:13.567 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:13.567 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.567 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.568 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:13.568 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.569 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.569 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:35:13.569 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:13.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.570 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.570 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:35:13.570 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:13.571 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.571 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.571 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:13.572 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.572 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:35:13.573 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:13.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.573 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.574 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:35:13.574 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:13.574 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.574 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.575 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:13.575 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.576 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.576 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:35:13.576 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:13.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.577 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.577 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:35:13.577 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:35:13.578 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.578 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.578 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:35:13.579 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:35:13.579 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:35:13.579 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:35:13.579 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:13.579 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:13.580 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:13.580 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.581 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:35:13.581 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:35:13.581 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:35:13.581 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:35:13.582 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:35:13.582 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:35:13.582 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:35:13.582 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:35:13.583 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:35:13.583 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 35, 2022-07-21T01:35:13.583 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:35:13.583 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:35:13.584 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:35:13.584 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.584 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.585 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:35:13.585 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:13.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.585 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.586 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:35:13.586 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:13.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.586 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.587 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:35:13.587 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.588 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.588 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:35:13.588 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:13.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.589 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:35:13.589 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:13.589 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.590 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.590 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:35:13.590 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.591 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.591 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:35:13.591 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:13.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.592 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.592 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:35:13.593 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:13.593 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.593 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.594 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:35:13.594 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:35:13.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.594 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:35:13.594 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:35:13.595 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:13.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.595 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:35:13.596 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:35:13.596 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:35:13.596 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.596 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.597 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:35:13.597 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:35:13.597 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:35:13.598 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:35:13.598 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:35:13.598 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:35:13.598 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:35:13.599 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:13.599 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.599 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:35:13.599 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:35:13.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.600 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:35:13.600 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:13.600 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:13.600 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:13.601 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:13.601 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:13.601 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:35:13.601 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:13.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.602 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:35:13.602 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:13.603 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:13.603 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:13.603 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:13.603 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:13.604 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:35:13.604 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:13.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.605 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:35:13.605 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:13.605 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:13.605 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:13.605 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:13.606 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:13.606 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:35:13.606 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:13.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.607 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:35:13.607 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:13.607 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:13.608 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:13.608 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:13.608 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:13.608 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:35:13.609 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:13.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.609 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:35:13.610 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:13.610 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:13.610 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:13.610 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:13.611 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:13.611 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:35:13.611 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:13.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.612 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:35:13.612 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:13.612 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:13.613 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:13.613 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:13.613 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:13.613 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:35:13.614 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:13.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.615 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:35:13.615 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:13.615 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:13.615 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:13.616 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:13.616 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:13.616 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:35:13.616 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:13.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:13.617 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:35:13.617 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:35:13.618 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:35:13.618 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:35:13.618 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:35:13.618 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:35:13.619 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:35:13.619 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:35:13.619 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.619 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:35:13.620 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:35:13.620 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:35:13.620 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:35:13.620 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:35:13.621 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:35:13.621 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6837/24695956": "2022-07-22T01:35:07.550747+0000", 2022-07-21T01:35:13.621 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6836/24695956": "2022-07-22T01:35:07.550747+0000" 2022-07-21T01:35:13.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.621 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:35:13.622 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:35:13.622 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:35:13.622 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:35:13.622 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:35:13.623 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:35:13.623 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:35:13.623 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:35:13.623 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:35:13.624 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:35:13.624 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:35:13.624 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:35:13.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.625 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:35:13.625 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:35:13.625 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:35:13.625 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:35:13.626 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:35:13.626 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:35:13.626 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:35:13.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:13.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:13.627 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:35:13.627 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:35:13.627 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:35:13.628 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:35:13.628 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:35:13.628 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:35:13.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f200539e0 msgr2=0x7f9f20055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f200539e0 0x7f9f20055ea0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f9f3000d000 tx=0x7f9f30000bc0).stop 2022-07-21T01:35:13.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f401402d0 msgr2=0x7f9f4009edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f401402d0 0x7f9f4009edc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f9f3800c000 tx=0x7f9f38006df0).stop 2022-07-21T01:35:13.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 shutdown_connections 2022-07-21T01:35:13.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f401402d0 0x7f9f4009edc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9f200539e0 0x7f9f20055ea0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f400b31d0 0x7f9f40143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 --2- 172.21.15.3:0/3828329059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f400a9da0 0x7f9f401562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 >> 172.21.15.3:0/3828329059 conn(0x7f9f40063870 msgr2=0x7f9f40140a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:13.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 shutdown_connections 2022-07-21T01:35:13.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.365+0000 7f9f4697b700 1 -- 172.21.15.3:0/3828329059 wait complete. 2022-07-21T01:35:13.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:13.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2759e700 1 Processor -- start 2022-07-21T01:35:13.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2759e700 1 -- start start 2022-07-21T01:35:13.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9e280b2710 0x7f9e2814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e280a8c50 0x7f9e2814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2759e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280a7250 con 0x7f9e280b01a0 2022-07-21T01:35:13.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2759e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e280a7480 con 0x7f9e280a8c50 2022-07-21T01:35:13.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e280a7720 con 0x7f9e280b2710 2022-07-21T01:35:13.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e25d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9e280b01a0 0x7f9e280a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52762/0 (socket says 172.21.15.3:52762) 2022-07-21T01:35:13.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e25d9b700 1 -- 172.21.15.3:0/3304900784 learned_addr learned my addr 172.21.15.3:0/3304900784 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:13.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e26d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e280a8c50 0x7f9e2814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2659c700 1 --2- 172.21.15.3:0/3304900784 >> v2:172.21.15.143:3300/0 conn(0x7f9e280b2710 0x7f9e2814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2559a700 1 -- 172.21.15.3:0/3304900784 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485488567 0 0) 0x7f9e280a7250 con 0x7f9e280b01a0 2022-07-21T01:35:13.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2559a700 1 -- 172.21.15.3:0/3304900784 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c003260 con 0x7f9e280b01a0 2022-07-21T01:35:13.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e2559a700 1 -- 172.21.15.3:0/3304900784 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095534632 0 0) 0x7f9e0c003260 con 0x7f9e280b01a0 2022-07-21T01:35:13.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e26d9d700 1 -- 172.21.15.3:0/3304900784 >> v2:172.21.15.143:3300/0 conn(0x7f9e280b2710 msgr2=0x7f9e2814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:13.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e26d9d700 1 --2- 172.21.15.3:0/3304900784 >> v2:172.21.15.143:3300/0 conn(0x7f9e280b2710 0x7f9e2814afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e26d9d700 1 -- 172.21.15.3:0/3304900784 >> v1:172.21.15.3:6789/0 conn(0x7f9e280b01a0 legacy=0x7f9e280a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e26d9d700 1 -- 172.21.15.3:0/3304900784 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280a79c0 con 0x7f9e280a8c50 2022-07-21T01:35:13.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.745+0000 7f9e26d9d700 1 --2- 172.21.15.3:0/3304900784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280a8c50 0x7f9e2814b4c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f9e18001800 tx=0x7f9e200091d0).ready entity=mon.0 client_cookie=7af818d880724f0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:13.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2559a700 1 -- 172.21.15.3:0/3304900784 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e18001660 con 0x7f9e280a8c50 2022-07-21T01:35:13.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2559a700 1 -- 172.21.15.3:0/3304900784 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2000f510 con 0x7f9e280a8c50 2022-07-21T01:35:13.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2559a700 1 -- 172.21.15.3:0/3304900784 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e2000f680 con 0x7f9e280a8c50 2022-07-21T01:35:13.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 -- 172.21.15.3:0/3304900784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280a8c50 msgr2=0x7f9e2814b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 --2- 172.21.15.3:0/3304900784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280a8c50 0x7f9e2814b4c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f9e18001800 tx=0x7f9e200091d0).stop 2022-07-21T01:35:13.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 -- 172.21.15.3:0/3304900784 shutdown_connections 2022-07-21T01:35:13.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 --2- 172.21.15.3:0/3304900784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280a8c50 0x7f9e2814b4c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 --2- 172.21.15.3:0/3304900784 >> v2:172.21.15.143:3300/0 conn(0x7f9e280b2710 0x7f9e2814afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 -- 172.21.15.3:0/3304900784 >> 172.21.15.3:0/3304900784 conn(0x7f9e28060310 msgr2=0x7f9e280b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:13.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 -- 172.21.15.3:0/3304900784 shutdown_connections 2022-07-21T01:35:13.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 -- 172.21.15.3:0/3304900784 wait complete. 2022-07-21T01:35:13.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 Processor -- start 2022-07-21T01:35:13.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 -- start start 2022-07-21T01:35:13.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280a8c50 0x7f9e2805cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b01a0 0x7f9e28064ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b2710 0x7f9e28064fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280ab440 con 0x7f9e280b2710 2022-07-21T01:35:13.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280aa390 con 0x7f9e280a8c50 2022-07-21T01:35:13.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e28159a00 con 0x7f9e280b01a0 2022-07-21T01:35:13.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e26d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b2710 0x7f9e28064fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e26d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b2710 0x7f9e28064fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44492/0 (socket says 172.21.15.3:44492) 2022-07-21T01:35:13.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e26d9d700 1 -- 172.21.15.3:0/894188067 learned_addr learned my addr 172.21.15.3:0/894188067 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:13.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e2659c700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280a8c50 0x7f9e2805cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e26d9d700 1 -- 172.21.15.3:0/894188067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b01a0 msgr2=0x7f9e28064ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e26d9d700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b01a0 0x7f9e28064ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:35:13.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e26d9d700 1 -- 172.21.15.3:0/894188067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280a8c50 msgr2=0x7f9e2805cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e26d9d700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280a8c50 0x7f9e2805cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.749+0000 7f9e26d9d700 1 -- 172.21.15.3:0/894188067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e20009470 con 0x7f9e280b2710 2022-07-21T01:35:13.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e26d9d700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b2710 0x7f9e28064fb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f9e18001800 tx=0x7f9e20016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:13.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e177fe700 1 -- 172.21.15.3:0/894188067 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e20011a20 con 0x7f9e280b2710 2022-07-21T01:35:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e177fe700 1 -- 172.21.15.3:0/894188067 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e20011b80 con 0x7f9e280b2710 2022-07-21T01:35:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e28063350 con 0x7f9e280b2710 2022-07-21T01:35:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e177fe700 1 -- 172.21.15.3:0/894188067 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e2001f3f0 con 0x7f9e280b2710 2022-07-21T01:35:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2814d600 con 0x7f9e280b2710 2022-07-21T01:35:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e28063190 con 0x7f9e280b2710 2022-07-21T01:35:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e177fe700 1 -- 172.21.15.3:0/894188067 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9e20026030 con 0x7f9e280b2710 2022-07-21T01:35:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e177fe700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e177fe700 1 -- 172.21.15.3:0/894188067 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f9e2006c6d0 con 0x7f9e280b2710 2022-07-21T01:35:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.753+0000 7f9e2659c700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:13.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.757+0000 7f9e2659c700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9e1c00b000 tx=0x7f9e1c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:13.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.761+0000 7f9e177fe700 1 -- 172.21.15.3:0/894188067 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9e200436d0 con 0x7f9e280b2710 2022-07-21T01:35:13.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.925+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e280634e0 con 0x7f9e280b2710 2022-07-21T01:35:13.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e177fe700 1 -- 172.21.15.3:0/894188067 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1557 (secure 0 0 0) 0x7f9e20024090 con 0x7f9e280b2710 2022-07-21T01:35:13.929 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:13.930 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":15,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":5146,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1404999735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1404999735},{"type":"v1","addr":"172.21.15.3:6837","nonce":1404999735}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[]} 2022-07-21T01:35:13.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 msgr2=0x7f9e04055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9e1c00b000 tx=0x7f9e1c006df0).stop 2022-07-21T01:35:13.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b2710 msgr2=0x7f9e28064fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:13.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b2710 0x7f9e28064fb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f9e18001800 tx=0x7f9e20016000).stop 2022-07-21T01:35:13.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 shutdown_connections 2022-07-21T01:35:13.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e280b2710 0x7f9e28064fb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e280b01a0 0x7f9e28064ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:35:13.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e04053b00 0x7f9e04055fc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 --2- 172.21.15.3:0/894188067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e280a8c50 0x7f9e2805cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 >> 172.21.15.3:0/894188067 conn(0x7f9e28060310 msgr2=0x7f9e28149b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 shutdown_connections 2022-07-21T01:35:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:13.929+0000 7f9e2759e700 1 -- 172.21.15.3:0/894188067 wait complete. 2022-07-21T01:35:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 15 2022-07-21T01:35:13.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.049+0000 7fe35545b700 1 Processor -- start 2022-07-21T01:35:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.049+0000 7fe35545b700 1 -- start start 2022-07-21T01:35:14.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.049+0000 7fe35545b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3500b2350 0x7fe3500acd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.049+0000 7fe35545b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3500ad200 con 0x7fe350139e50 2022-07-21T01:35:14.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.049+0000 7fe35545b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe350060ed0 con 0x7fe35013a230 2022-07-21T01:35:14.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.049+0000 7fe35545b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3500b28c0 con 0x7fe3500b2350 2022-07-21T01:35:14.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34effd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe350139e50 0x7fe350143680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52776/0 (socket says 172.21.15.3:52776) 2022-07-21T01:35:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34effd700 1 -- 172.21.15.3:0/2804463611 learned_addr learned my addr 172.21.15.3:0/2804463611 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34e7fc700 1 --2- 172.21.15.3:0/2804463611 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3500b2350 0x7fe3500acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34dffb700 1 -- 172.21.15.3:0/2804463611 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923734844 0 0) 0x7fe340001110 con 0x7fe350139e50 2022-07-21T01:35:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34dffb700 1 -- 172.21.15.3:0/2804463611 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe334003260 con 0x7fe350139e50 2022-07-21T01:35:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34dffb700 1 -- 172.21.15.3:0/2804463611 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913135015 0 0) 0x7fe350060ed0 con 0x7fe35013a230 2022-07-21T01:35:14.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34dffb700 1 -- 172.21.15.3:0/2804463611 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe340001110 con 0x7fe35013a230 2022-07-21T01:35:14.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34e7fc700 1 -- 172.21.15.3:0/2804463611 >> v1:172.21.15.143:6789/0 conn(0x7fe35013a230 legacy=0x7fe3500b6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34e7fc700 1 -- 172.21.15.3:0/2804463611 >> v1:172.21.15.3:6789/0 conn(0x7fe350139e50 legacy=0x7fe350143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34dffb700 1 -- 172.21.15.3:0/2804463611 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1978145617 0 0) 0x7fe334003260 con 0x7fe350139e50 2022-07-21T01:35:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34e7fc700 1 -- 172.21.15.3:0/2804463611 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3500b6e60 con 0x7fe3500b2350 2022-07-21T01:35:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34e7fc700 1 --2- 172.21.15.3:0/2804463611 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3500b2350 0x7fe3500acd10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fe340001990 tx=0x7fe344006df0).ready entity=mon.2 client_cookie=41ae637b3951fc86 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34dffb700 1 -- 172.21.15.3:0/2804463611 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3400017f0 con 0x7fe3500b2350 2022-07-21T01:35:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34dffb700 1 -- 172.21.15.3:0/2804463611 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe344000e90 con 0x7fe3500b2350 2022-07-21T01:35:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe34dffb700 1 -- 172.21.15.3:0/2804463611 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe344012460 con 0x7fe3500b2350 2022-07-21T01:35:14.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe35545b700 1 -- 172.21.15.3:0/2804463611 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3500b2350 msgr2=0x7fe3500acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe35545b700 1 --2- 172.21.15.3:0/2804463611 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3500b2350 0x7fe3500acd10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fe340001990 tx=0x7fe344006df0).stop 2022-07-21T01:35:14.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe35545b700 1 -- 172.21.15.3:0/2804463611 shutdown_connections 2022-07-21T01:35:14.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe35545b700 1 --2- 172.21.15.3:0/2804463611 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3500b2350 0x7fe3500acd10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe35545b700 1 -- 172.21.15.3:0/2804463611 >> 172.21.15.3:0/2804463611 conn(0x7fe3500b3610 msgr2=0x7fe3500b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:14.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe35545b700 1 -- 172.21.15.3:0/2804463611 shutdown_connections 2022-07-21T01:35:14.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.053+0000 7fe35545b700 1 -- 172.21.15.3:0/2804463611 wait complete. 2022-07-21T01:35:14.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 Processor -- start 2022-07-21T01:35:14.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 -- start start 2022-07-21T01:35:14.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3500b2350 0x7fe35013e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe350139e50 0x7fe350150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe35013a230 0x7fe350140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35005f840 con 0x7fe35013a230 2022-07-21T01:35:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35013bcb0 con 0x7fe3500b2350 2022-07-21T01:35:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35005fa70 con 0x7fe350139e50 2022-07-21T01:35:14.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe35013a230 0x7fe350140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe35013a230 0x7fe350140b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44506/0 (socket says 172.21.15.3:44506) 2022-07-21T01:35:14.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34f7fe700 1 -- 172.21.15.3:0/181814731 learned_addr learned my addr 172.21.15.3:0/181814731 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:14.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34e7fc700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe350139e50 0x7fe350150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34effd700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3500b2350 0x7fe35013e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34f7fe700 1 -- 172.21.15.3:0/181814731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe350139e50 msgr2=0x7fe350150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34f7fe700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe350139e50 0x7fe350150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34f7fe700 1 -- 172.21.15.3:0/181814731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3500b2350 msgr2=0x7fe35013e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34f7fe700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3500b2350 0x7fe35013e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34f7fe700 1 -- 172.21.15.3:0/181814731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34400b180 con 0x7fe35013a230 2022-07-21T01:35:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe34f7fe700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe35013a230 0x7fe350140b30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fe348001490 tx=0x7fe348007da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe33ffff700 1 -- 172.21.15.3:0/181814731 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe348002310 con 0x7fe35013a230 2022-07-21T01:35:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe33ffff700 1 -- 172.21.15.3:0/181814731 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348002470 con 0x7fe35013a230 2022-07-21T01:35:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35005fd10 con 0x7fe35013a230 2022-07-21T01:35:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe33ffff700 1 -- 172.21.15.3:0/181814731 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe348001860 con 0x7fe35013a230 2022-07-21T01:35:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.057+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35013be80 con 0x7fe35013a230 2022-07-21T01:35:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.061+0000 7fe33ffff700 1 -- 172.21.15.3:0/181814731 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe348019020 con 0x7fe35013a230 2022-07-21T01:35:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.061+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe350061130 con 0x7fe35013a230 2022-07-21T01:35:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.061+0000 7fe33ffff700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c053a70 0x7fe32c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.061+0000 7fe33ffff700 1 -- 172.21.15.3:0/181814731 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fe34806bcf0 con 0x7fe35013a230 2022-07-21T01:35:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.061+0000 7fe34effd700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c053a70 0x7fe32c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.061+0000 7fe34effd700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c053a70 0x7fe32c055f30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fe34000d000 tx=0x7fe3400018f0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.069+0000 7fe33ffff700 1 -- 172.21.15.3:0/181814731 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe348042c00 con 0x7fe35013a230 2022-07-21T01:35:14.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.229+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3500574e0 con 0x7fe35013a230 2022-07-21T01:35:14.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.229+0000 7fe33ffff700 1 -- 172.21.15.3:0/181814731 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1557 (secure 0 0 0) 0x7fe348046220 con 0x7fe35013a230 2022-07-21T01:35:14.232 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:14.232 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":15,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4679,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4263408266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4263408266},{"type":"v1","addr":"172.21.15.143:6833","nonce":4263408266}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":4893,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3448004703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3448004703},{"type":"v1","addr":"172.21.15.3:6835","nonce":3448004703}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":9},{"gid":5146,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1404999735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1404999735},{"type":"v1","addr":"172.21.15.3:6837","nonce":1404999735}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[]} 2022-07-21T01:35:14.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c053a70 msgr2=0x7fe32c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c053a70 0x7fe32c055f30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fe34000d000 tx=0x7fe3400018f0).stop 2022-07-21T01:35:14.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe35013a230 msgr2=0x7fe350140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe35013a230 0x7fe350140b30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fe348001490 tx=0x7fe348007da0).stop 2022-07-21T01:35:14.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 shutdown_connections 2022-07-21T01:35:14.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe35013a230 0x7fe350140b30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe350139e50 0x7fe350150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c053a70 0x7fe32c055f30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 --2- 172.21.15.3:0/181814731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3500b2350 0x7fe35013e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 >> 172.21.15.3:0/181814731 conn(0x7fe3500b3610 msgr2=0x7fe3501392e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:14.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 shutdown_connections 2022-07-21T01:35:14.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.233+0000 7fe35545b700 1 -- 172.21.15.3:0/181814731 wait complete. 2022-07-21T01:35:14.237 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 15 2022-07-21T01:35:14.245 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:35:14.245 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:35:14.246 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:35:14.246 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:35:14.247 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:35:14.247 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:35:14.247 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:35:14.248 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:35:14.248 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:35:14.248 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:35:14.249 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:35:14.249 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:35:14.264 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:35:14.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:35:14.266 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:35:14.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:35:14.268 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:35:14.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:35:14.272 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:35:14.273 DEBUG:teuthology.parallel:result is None 2022-07-21T01:35:14.316 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:35:14.317 DEBUG:teuthology.parallel:result is None 2022-07-21T01:35:14.319 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:35:14.319 DEBUG:teuthology.parallel:result is None 2022-07-21T01:35:14.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:35:14.350 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:35:14.388 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:35:14.402 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:35:14.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec7e19b700 1 Processor -- start 2022-07-21T01:35:14.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec7e19b700 1 -- start start 2022-07-21T01:35:14.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec7e19b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fec780b19d0 0x7fec78142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec7e19b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 0x7fec78142f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec7e19b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec780b37b0 con 0x7fec780b33d0 2022-07-21T01:35:14.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec7e19b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec78143880 con 0x7fec780b19d0 2022-07-21T01:35:14.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec7e19b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec78065e30 con 0x7fec780b1000 2022-07-21T01:35:14.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec76ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fec780b19d0 0x7fec78142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec76ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fec780b19d0 0x7fec78142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44534/0 (socket says 172.21.15.3:44534) 2022-07-21T01:35:14.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec76ffd700 1 -- 172.21.15.3:0/753907479 learned_addr learned my addr 172.21.15.3:0/753907479 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:14.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec77fff700 1 --2- 172.21.15.3:0/753907479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 0x7fec78142f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec76ffd700 1 -- 172.21.15.3:0/753907479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 msgr2=0x7fec78142f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:14.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec76ffd700 1 --2- 172.21.15.3:0/753907479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 0x7fec78142f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec76ffd700 1 -- 172.21.15.3:0/753907479 >> v1:172.21.15.143:6789/0 conn(0x7fec780b33d0 legacy=0x7fec78066810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec76ffd700 1 -- 172.21.15.3:0/753907479 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec780660d0 con 0x7fec780b19d0 2022-07-21T01:35:14.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec76ffd700 1 --2- 172.21.15.3:0/753907479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b19d0 0x7fec78142a20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fec6c002f70 tx=0x7fec6c006eb0).ready entity=mon.0 client_cookie=1bdc5d7d518a9d02 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:14.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec767fc700 1 -- 172.21.15.3:0/753907479 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec6c00d5c0 con 0x7fec780b19d0 2022-07-21T01:35:14.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec767fc700 1 -- 172.21.15.3:0/753907479 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c000e90 con 0x7fec780b19d0 2022-07-21T01:35:14.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.433+0000 7fec767fc700 1 -- 172.21.15.3:0/753907479 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec6c00d950 con 0x7fec780b19d0 2022-07-21T01:35:14.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 -- 172.21.15.3:0/753907479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b19d0 msgr2=0x7fec78142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 --2- 172.21.15.3:0/753907479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b19d0 0x7fec78142a20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fec6c002f70 tx=0x7fec6c006eb0).stop 2022-07-21T01:35:14.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 -- 172.21.15.3:0/753907479 shutdown_connections 2022-07-21T01:35:14.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 --2- 172.21.15.3:0/753907479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 0x7fec78142f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 --2- 172.21.15.3:0/753907479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b19d0 0x7fec78142a20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 -- 172.21.15.3:0/753907479 >> 172.21.15.3:0/753907479 conn(0x7fec7805c620 msgr2=0x7fec7805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:14.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 -- 172.21.15.3:0/753907479 shutdown_connections 2022-07-21T01:35:14.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 -- 172.21.15.3:0/753907479 wait complete. 2022-07-21T01:35:14.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 Processor -- start 2022-07-21T01:35:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.437+0000 7fec7e19b700 1 -- start start 2022-07-21T01:35:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec7e19b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 0x7fec7813ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec7e19b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780b19d0 0x7fec780ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec7e19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b33d0 0x7fec780ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec7e19b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780acac0 con 0x7fec780b33d0 2022-07-21T01:35:14.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec7e19b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780ad4f0 con 0x7fec780b19d0 2022-07-21T01:35:14.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec7e19b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec780b06c0 con 0x7fec780b1000 2022-07-21T01:35:14.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec77fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b33d0 0x7fec780ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec76ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780b19d0 0x7fec780ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec77fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b33d0 0x7fec780ac5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44544/0 (socket says 172.21.15.3:44544) 2022-07-21T01:35:14.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec77fff700 1 -- 172.21.15.3:0/3206826566 learned_addr learned my addr 172.21.15.3:0/3206826566 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:14.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec777fe700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 0x7fec7813ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec77fff700 1 -- 172.21.15.3:0/3206826566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 msgr2=0x7fec7813ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec77fff700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 0x7fec7813ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec77fff700 1 -- 172.21.15.3:0/3206826566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780b19d0 msgr2=0x7fec780ac0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:14.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec77fff700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780b19d0 0x7fec780ac0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:14.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec77fff700 1 -- 172.21.15.3:0/3206826566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c00b1e0 con 0x7fec780b33d0 2022-07-21T01:35:14.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec77fff700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b33d0 0x7fec780ac5d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fec7000a000 tx=0x7fec70008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:14.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec74ff9700 1 -- 172.21.15.3:0/3206826566 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec70013620 con 0x7fec780b33d0 2022-07-21T01:35:14.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec74ff9700 1 -- 172.21.15.3:0/3206826566 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70013780 con 0x7fec780b33d0 2022-07-21T01:35:14.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec780b0960 con 0x7fec780b33d0 2022-07-21T01:35:14.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec74ff9700 1 -- 172.21.15.3:0/3206826566 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec70013ae0 con 0x7fec780b33d0 2022-07-21T01:35:14.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec780ae400 con 0x7fec780b33d0 2022-07-21T01:35:14.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec780ae5c0 con 0x7fec780b33d0 2022-07-21T01:35:14.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec74ff9700 1 -- 172.21.15.3:0/3206826566 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fec70013c40 con 0x7fec780b33d0 2022-07-21T01:35:14.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.441+0000 7fec74ff9700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fec58053aa0 0x7fec58055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:14.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.445+0000 7fec74ff9700 1 -- 172.21.15.3:0/3206826566 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fec7006bed0 con 0x7fec780b33d0 2022-07-21T01:35:14.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.445+0000 7fec777fe700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fec58053aa0 0x7fec58055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:14.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.445+0000 7fec777fe700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fec58053aa0 0x7fec58055f60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fec6800d000 tx=0x7fec68007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:14.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.453+0000 7fec74ff9700 1 -- 172.21.15.3:0/3206826566 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fec70042e00 con 0x7fec780b33d0 2022-07-21T01:35:14.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:14.585+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fec780574e0 con 0x7fec780b33d0 2022-07-21T01:35:15.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.437+0000 7fec74ff9700 1 -- 172.21.15.3:0/3206826566 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v43) v1 ==== 97+0+0 (secure 0 0 0) 0x7fec70046420 con 0x7fec780b33d0 2022-07-21T01:35:15.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fec58053aa0 msgr2=0x7fec58055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fec58053aa0 0x7fec58055f60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fec6800d000 tx=0x7fec68007b90).stop 2022-07-21T01:35:15.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b33d0 msgr2=0x7fec780ac5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b33d0 0x7fec780ac5d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fec7000a000 tx=0x7fec70008040).stop 2022-07-21T01:35:15.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 shutdown_connections 2022-07-21T01:35:15.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec780b33d0 0x7fec780ac5d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fec58053aa0 0x7fec58055f60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780b19d0 0x7fec780ac0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 --2- 172.21.15.3:0/3206826566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec780b1000 0x7fec7813ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 >> 172.21.15.3:0/3206826566 conn(0x7fec7805c620 msgr2=0x7fec78141d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:15.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 shutdown_connections 2022-07-21T01:35:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.441+0000 7fec7e19b700 1 -- 172.21.15.3:0/3206826566 wait complete. 2022-07-21T01:35:15.446 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:35:15.460 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:35:15.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:35:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 Processor -- start 2022-07-21T01:35:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 -- start start 2022-07-21T01:35:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa0cc0b1160 0x7fa0cc1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc0b04e0 con 0x7fa0cc148530 2022-07-21T01:35:15.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc0b0780 con 0x7fa0cc0b1f40 2022-07-21T01:35:15.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0cc0a9250 con 0x7fa0cc0b1160 2022-07-21T01:35:15.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d10fa700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa0cc0b1160 0x7fa0cc1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0cbfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa0cc148530 0x7fa0cc0a7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52832/0 (socket says 172.21.15.3:52832) 2022-07-21T01:35:15.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0cbfff700 1 -- 172.21.15.3:0/718957593 learned_addr learned my addr 172.21.15.3:0/718957593 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:15.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d10fa700 1 -- 172.21.15.3:0/718957593 >> v1:172.21.15.143:6789/0 conn(0x7fa0cc0b1f40 legacy=0x7fa0cc0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d10fa700 1 -- 172.21.15.3:0/718957593 >> v1:172.21.15.3:6789/0 conn(0x7fa0cc148530 legacy=0x7fa0cc0a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0cb7fe700 1 -- 172.21.15.3:0/718957593 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322323990 0 0) 0x7fa0cc0b04e0 con 0x7fa0cc148530 2022-07-21T01:35:15.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d10fa700 1 -- 172.21.15.3:0/718957593 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc0b0a20 con 0x7fa0cc0b1160 2022-07-21T01:35:15.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d10fa700 1 --2- 172.21.15.3:0/718957593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 0x7fa0cc1448b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fa0cc1478c0 tx=0x7fa0c4006df0).ready entity=mon.0 client_cookie=9efd33a606a19042 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:15.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0cb7fe700 1 -- 172.21.15.3:0/718957593 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0c400d610 con 0x7fa0cc0b1160 2022-07-21T01:35:15.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0cb7fe700 1 -- 172.21.15.3:0/718957593 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c4000e90 con 0x7fa0cc0b1160 2022-07-21T01:35:15.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0cb7fe700 1 -- 172.21.15.3:0/718957593 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0c400d960 con 0x7fa0cc0b1160 2022-07-21T01:35:15.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/718957593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 msgr2=0x7fa0cc1448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 --2- 172.21.15.3:0/718957593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 0x7fa0cc1448b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fa0cc1478c0 tx=0x7fa0c4006df0).stop 2022-07-21T01:35:15.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/718957593 shutdown_connections 2022-07-21T01:35:15.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 --2- 172.21.15.3:0/718957593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 0x7fa0cc1448b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/718957593 >> 172.21.15.3:0/718957593 conn(0x7fa0cc05f840 msgr2=0x7fa0cc05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/718957593 shutdown_connections 2022-07-21T01:35:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.569+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/718957593 wait complete. 2022-07-21T01:35:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 Processor -- start 2022-07-21T01:35:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 -- start start 2022-07-21T01:35:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 0x7fa0cc14a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0cc0b1f40 0x7fa0cc14b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0cc148530 0x7fa0cc0ab390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc14af00 con 0x7fa0cc0b1160 2022-07-21T01:35:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc0af050 con 0x7fa0cc148530 2022-07-21T01:35:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc0af2f0 con 0x7fa0cc0b1f40 2022-07-21T01:35:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d08f9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 0x7fa0cc14a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d08f9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 0x7fa0cc14a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44556/0 (socket says 172.21.15.3:44556) 2022-07-21T01:35:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d08f9700 1 -- 172.21.15.3:0/4038260071 learned_addr learned my addr 172.21.15.3:0/4038260071 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:15.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d10fa700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0cc148530 0x7fa0cc0ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0cbfff700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0cc0b1f40 0x7fa0cc14b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d08f9700 1 -- 172.21.15.3:0/4038260071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0cc0b1f40 msgr2=0x7fa0cc14b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d08f9700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0cc0b1f40 0x7fa0cc14b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d08f9700 1 -- 172.21.15.3:0/4038260071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0cc148530 msgr2=0x7fa0cc0ab390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d08f9700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0cc148530 0x7fa0cc0ab390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d08f9700 1 -- 172.21.15.3:0/4038260071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c400b180 con 0x7fa0cc0b1160 2022-07-21T01:35:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d10fa700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0cc148530 0x7fa0cc0ab390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d08f9700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 0x7fa0cc14a1a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fa0cc0b3300 tx=0x7fa0bc0014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:15.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0c9ffb700 1 -- 172.21.15.3:0/4038260071 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0bc00e4a0 con 0x7fa0cc0b1160 2022-07-21T01:35:15.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0c9ffb700 1 -- 172.21.15.3:0/4038260071 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc00e600 con 0x7fa0cc0b1160 2022-07-21T01:35:15.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc0af590 con 0x7fa0cc0b1160 2022-07-21T01:35:15.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0c9ffb700 1 -- 172.21.15.3:0/4038260071 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0bc00e960 con 0x7fa0cc0b1160 2022-07-21T01:35:15.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.573+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc0abf00 con 0x7fa0cc0b1160 2022-07-21T01:35:15.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.577+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0cc14bd60 con 0x7fa0cc0b1160 2022-07-21T01:35:15.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.577+0000 7fa0c9ffb700 1 -- 172.21.15.3:0/4038260071 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa0bc019030 con 0x7fa0cc0b1160 2022-07-21T01:35:15.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.577+0000 7fa0c9ffb700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ac0539e0 0x7fa0ac055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.577+0000 7fa0c9ffb700 1 -- 172.21.15.3:0/4038260071 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa0bc06b4a0 con 0x7fa0cc0b1160 2022-07-21T01:35:15.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.577+0000 7fa0cbfff700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ac0539e0 0x7fa0ac055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.577+0000 7fa0cbfff700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ac0539e0 0x7fa0ac055ea0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa0c000d000 tx=0x7fa0c0007c00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:15.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.585+0000 7fa0c9ffb700 1 -- 172.21.15.3:0/4038260071 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa0cc14bd60 con 0x7fa0cc0b1160 2022-07-21T01:35:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa0cc14bd60 con 0x7fa0cc0b1160 2022-07-21T01:35:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0c9ffb700 1 -- 172.21.15.3:0/4038260071 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fa0cc14bd60 con 0x7fa0cc0b1160 2022-07-21T01:35:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:15.751 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:35:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:35:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:35:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:35:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:35:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:35:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:35:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:35:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:35:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:35:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:35:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:35:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:35:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:35:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:35:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:35:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:35:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:35:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:35:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:35:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:35:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:35:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:35:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:35:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:35:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:35:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:35:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:35:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:35:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:35:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:35:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:35:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:35:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:35:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:35:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:35:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:35:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:35:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:35:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:35:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:35:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:35:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:35:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:35:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:35:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:35:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:35:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:35:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:35:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:35:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:35:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:35:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:35:15.789 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:35:15.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ac0539e0 msgr2=0x7fa0ac055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ac0539e0 0x7fa0ac055ea0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa0c000d000 tx=0x7fa0c0007c00).stop 2022-07-21T01:35:15.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 msgr2=0x7fa0cc14a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 0x7fa0cc14a1a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fa0cc0b3300 tx=0x7fa0bc0014a0).stop 2022-07-21T01:35:15.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 shutdown_connections 2022-07-21T01:35:15.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0ac0539e0 0x7fa0ac055ea0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0cc148530 0x7fa0cc0ab390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0cc0b1f40 0x7fa0cc14b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 --2- 172.21.15.3:0/4038260071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0cc0b1160 0x7fa0cc14a1a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 >> 172.21.15.3:0/4038260071 conn(0x7fa0cc05f840 msgr2=0x7fa0cc0b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 shutdown_connections 2022-07-21T01:35:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.749+0000 7fa0d2b5d700 1 -- 172.21.15.3:0/4038260071 wait complete. 2022-07-21T01:35:15.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:35:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f4334700 1 Processor -- start 2022-07-21T01:35:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f4334700 1 -- start start 2022-07-21T01:35:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f4334700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f13ec13a230 0x7f13ec0acd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f4334700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec0ad200 con 0x7f13ec139e50 2022-07-21T01:35:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f4334700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec060ed0 con 0x7f13ec0b2350 2022-07-21T01:35:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f4334700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ec0b28c0 con 0x7f13ec13a230 2022-07-21T01:35:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f20d0700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f13ec139e50 0x7f13ec143680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52844/0 (socket says 172.21.15.3:52844) 2022-07-21T01:35:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f20d0700 1 -- 172.21.15.3:0/236365978 learned_addr learned my addr 172.21.15.3:0/236365978 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f28d1700 1 --2- 172.21.15.3:0/236365978 >> v2:172.21.15.143:3300/0 conn(0x7f13ec13a230 0x7f13ec0acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f10ce700 1 -- 172.21.15.3:0/236365978 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278033693 0 0) 0x7f13e8001110 con 0x7f13ec139e50 2022-07-21T01:35:15.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f10ce700 1 -- 172.21.15.3:0/236365978 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d4003260 con 0x7f13ec139e50 2022-07-21T01:35:15.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f28d1700 1 -- 172.21.15.3:0/236365978 >> v1:172.21.15.143:6789/0 conn(0x7f13ec0b2350 legacy=0x7f13ec0b6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f28d1700 1 -- 172.21.15.3:0/236365978 >> v1:172.21.15.3:6789/0 conn(0x7f13ec139e50 legacy=0x7f13ec143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f10ce700 1 -- 172.21.15.3:0/236365978 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859152179 0 0) 0x7f13ec060ed0 con 0x7f13ec0b2350 2022-07-21T01:35:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f28d1700 1 -- 172.21.15.3:0/236365978 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec0b6e60 con 0x7f13ec13a230 2022-07-21T01:35:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f28d1700 1 --2- 172.21.15.3:0/236365978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec13a230 0x7f13ec0acd10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f13e40032f0 tx=0x7f13e40091d0).ready entity=mon.1 client_cookie=b9ecd06d6d1f140e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f10ce700 1 -- 172.21.15.3:0/236365978 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13e400b6c0 con 0x7f13ec13a230 2022-07-21T01:35:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f10ce700 1 -- 172.21.15.3:0/236365978 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e400b820 con 0x7f13ec13a230 2022-07-21T01:35:15.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.901+0000 7f13f10ce700 1 -- 172.21.15.3:0/236365978 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13e400b990 con 0x7f13ec13a230 2022-07-21T01:35:15.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 -- 172.21.15.3:0/236365978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec13a230 msgr2=0x7f13ec0acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 --2- 172.21.15.3:0/236365978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec13a230 0x7f13ec0acd10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f13e40032f0 tx=0x7f13e40091d0).stop 2022-07-21T01:35:15.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 -- 172.21.15.3:0/236365978 shutdown_connections 2022-07-21T01:35:15.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 --2- 172.21.15.3:0/236365978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec13a230 0x7f13ec0acd10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 -- 172.21.15.3:0/236365978 >> 172.21.15.3:0/236365978 conn(0x7f13ec0b3610 msgr2=0x7f13ec0b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:15.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 -- 172.21.15.3:0/236365978 shutdown_connections 2022-07-21T01:35:15.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 -- 172.21.15.3:0/236365978 wait complete. 2022-07-21T01:35:15.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 Processor -- start 2022-07-21T01:35:15.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 -- start start 2022-07-21T01:35:15.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13ec0b2350 0x7f13ec13aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec139e50 0x7f13ec05fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13ec13a230 0x7f13ec154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec13e3f0 con 0x7f13ec13a230 2022-07-21T01:35:15.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec13b080 con 0x7f13ec139e50 2022-07-21T01:35:15.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f4334700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec1539b0 con 0x7f13ec0b2350 2022-07-21T01:35:15.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f28d1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13ec13a230 0x7f13ec154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f28d1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13ec13a230 0x7f13ec154390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44574/0 (socket says 172.21.15.3:44574) 2022-07-21T01:35:15.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f28d1700 1 -- 172.21.15.3:0/3020609521 learned_addr learned my addr 172.21.15.3:0/3020609521 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:15.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f20d0700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13ec0b2350 0x7f13ec13aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f18cf700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec139e50 0x7f13ec05fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f28d1700 1 -- 172.21.15.3:0/3020609521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13ec0b2350 msgr2=0x7f13ec13aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f28d1700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13ec0b2350 0x7f13ec13aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f28d1700 1 -- 172.21.15.3:0/3020609521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec139e50 msgr2=0x7f13ec05fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:15.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f28d1700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec139e50 0x7f13ec05fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:15.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f28d1700 1 -- 172.21.15.3:0/3020609521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e4009470 con 0x7f13ec13a230 2022-07-21T01:35:15.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.905+0000 7f13f18cf700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec139e50 0x7f13ec05fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:15.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13f28d1700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13ec13a230 0x7f13ec154390 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f13e40032f0 tx=0x7f13e4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:15.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13db7fe700 1 -- 172.21.15.3:0/3020609521 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13e400b6c0 con 0x7f13ec13a230 2022-07-21T01:35:15.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13db7fe700 1 -- 172.21.15.3:0/3020609521 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e40107a0 con 0x7f13ec13a230 2022-07-21T01:35:15.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec153c50 con 0x7f13ec13a230 2022-07-21T01:35:15.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13db7fe700 1 -- 172.21.15.3:0/3020609521 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13e4010b00 con 0x7f13ec13a230 2022-07-21T01:35:15.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec13cd60 con 0x7f13ec13a230 2022-07-21T01:35:15.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec13d2f0 con 0x7f13ec13a230 2022-07-21T01:35:15.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13db7fe700 1 -- 172.21.15.3:0/3020609521 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f13e400f070 con 0x7f13ec13a230 2022-07-21T01:35:15.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13db7fe700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13cc053b00 0x7f13cc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:15.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13db7fe700 1 -- 172.21.15.3:0/3020609521 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f13e406d100 con 0x7f13ec13a230 2022-07-21T01:35:15.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13f20d0700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13cc053b00 0x7f13cc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:15.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.909+0000 7f13f20d0700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13cc053b00 0x7f13cc055fc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f13e800b000 tx=0x7f13e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:15.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:15.917+0000 7f13db7fe700 1 -- 172.21.15.3:0/3020609521 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f13e4044200 con 0x7f13ec13a230 2022-07-21T01:35:16.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.045+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13ec061130 con 0x7f13ec13a230 2022-07-21T01:35:16.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.045+0000 7f13db7fe700 1 -- 172.21.15.3:0/3020609521 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+13414 (secure 0 0 0) 0x7f13e4015090 con 0x7f13ec13a230 2022-07-21T01:35:16.047 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:16.048 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":43,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:35:15.428965+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:35:16.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13cc053b00 msgr2=0x7f13cc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13cc053b00 0x7f13cc055fc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f13e800b000 tx=0x7f13e8009040).stop 2022-07-21T01:35:16.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13ec13a230 msgr2=0x7f13ec154390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13ec13a230 0x7f13ec154390 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f13e40032f0 tx=0x7f13e4016000).stop 2022-07-21T01:35:16.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 shutdown_connections 2022-07-21T01:35:16.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13ec13a230 0x7f13ec154390 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f13cc053b00 0x7f13cc055fc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13ec139e50 0x7f13ec05fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 --2- 172.21.15.3:0/3020609521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13ec0b2350 0x7f13ec13aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 >> 172.21.15.3:0/3020609521 conn(0x7f13ec0b3610 msgr2=0x7f13ec141710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:16.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 shutdown_connections 2022-07-21T01:35:16.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.049+0000 7f13f4334700 1 -- 172.21.15.3:0/3020609521 wait complete. 2022-07-21T01:35:16.062 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:35:16.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:35:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a1b52700 1 Processor -- start 2022-07-21T01:35:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a1b52700 1 -- start start 2022-07-21T01:35:16.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a1b52700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa29c0b07a0 0x7fa29c0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a1b52700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c147730 0x7fa29c0acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a1b52700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c0b5a10 con 0x7fa29c0b5630 2022-07-21T01:35:16.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a1b52700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29c145470 con 0x7fa29c0b07a0 2022-07-21T01:35:16.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a1b52700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0ae180 con 0x7fa29c147730 2022-07-21T01:35:16.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a0b50700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa29c0b5630 0x7fa29c067730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52858/0 (socket says 172.21.15.3:52858) 2022-07-21T01:35:16.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a0b50700 1 -- 172.21.15.3:0/989355272 learned_addr learned my addr 172.21.15.3:0/989355272 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:16.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa293fff700 1 --2- 172.21.15.3:0/989355272 >> v2:172.21.15.143:3300/0 conn(0x7fa29c0b07a0 0x7fa29c0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a1351700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c147730 0x7fa29c0acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520995859 0 0) 0x7fa29c0b5a10 con 0x7fa29c0b5630 2022-07-21T01:35:16.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa288003260 con 0x7fa29c0b5630 2022-07-21T01:35:16.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 732395035 0 0) 0x7fa288003260 con 0x7fa29c0b5630 2022-07-21T01:35:16.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c147730 msgr2=0x7fa29c0acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:16.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c147730 0x7fa29c0acdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 >> v2:172.21.15.143:3300/0 conn(0x7fa29c0b07a0 msgr2=0x7fa29c0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:16.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 --2- 172.21.15.3:0/989355272 >> v2:172.21.15.143:3300/0 conn(0x7fa29c0b07a0 0x7fa29c0af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c0ac990 con 0x7fa29c0b5630 2022-07-21T01:35:16.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fa298001660 con 0x7fa29c0b5630 2022-07-21T01:35:16.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 >> v1:172.21.15.3:6789/0 conn(0x7fa29c0b5630 legacy=0x7fa29c067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa288003750 0x7fa288003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c147730 0x7fa288004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2880049f0 0x7fa288004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c145470 con 0x7fa288003750 2022-07-21T01:35:16.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0ae180 con 0x7fa2880049f0 2022-07-21T01:35:16.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28800b750 con 0x7fa29c147730 2022-07-21T01:35:16.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a0b50700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa288003750 0x7fa288003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa293fff700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c147730 0x7fa288004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a0b50700 1 -- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c147730 msgr2=0x7fa288004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:16.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a0b50700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c147730 0x7fa288004500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a0b50700 1 -- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2880049f0 msgr2=0x7fa288004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:16.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a0b50700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2880049f0 0x7fa288004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:35:16.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a0b50700 1 -- 172.21.15.3:0/989355272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288003260 con 0x7fa288003750 2022-07-21T01:35:16.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2a0b50700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa288003750 0x7fa288003b90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fa298001370 tx=0x7fa298001e50).ready entity=mon.0 client_cookie=90361b695fad4877 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2980070d0 con 0x7fa288003750 2022-07-21T01:35:16.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa298007230 con 0x7fa288003750 2022-07-21T01:35:16.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.177+0000 7fa2937fe700 1 -- 172.21.15.3:0/989355272 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa29800c9c0 con 0x7fa288003750 2022-07-21T01:35:16.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.181+0000 7fa2a1b52700 1 -- 172.21.15.3:0/989355272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa288003750 msgr2=0x7fa288003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.181+0000 7fa2a1b52700 1 --2- 172.21.15.3:0/989355272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa288003750 0x7fa288003b90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fa298001370 tx=0x7fa298001e50).stop 2022-07-21T01:35:16.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.181+0000 7fa2a1351700 1 -- 172.21.15.3:0/989355272 reap_dead start 2022-07-21T01:35:16.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.181+0000 7fa2a1b52700 1 -- 172.21.15.3:0/989355272 shutdown_connections 2022-07-21T01:35:16.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.181+0000 7fa2a1b52700 1 -- 172.21.15.3:0/989355272 >> 172.21.15.3:0/989355272 conn(0x7fa29c0626f0 msgr2=0x7fa29c062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:16.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.181+0000 7fa2a1b52700 1 -- 172.21.15.3:0/989355272 shutdown_connections 2022-07-21T01:35:16.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.181+0000 7fa2a1b52700 1 -- 172.21.15.3:0/989355272 wait complete. 2022-07-21T01:35:16.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.181+0000 7fa2a1b52700 1 Processor -- start 2022-07-21T01:35:16.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.181+0000 7fa2a1b52700 1 -- start start 2022-07-21T01:35:16.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a1b52700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa288003750 0x7fa29c0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a1b52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa29c0b4da0 0x7fa29c13e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a1b52700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c13e7c0 0x7fa29c13ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a1b52700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c13f160 con 0x7fa29c0b4da0 2022-07-21T01:35:16.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a1b52700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c146510 con 0x7fa288003750 2022-07-21T01:35:16.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a1b52700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0b4940 con 0x7fa29c13e7c0 2022-07-21T01:35:16.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa293fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa29c0b4da0 0x7fa29c13e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa293fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa29c0b4da0 0x7fa29c13e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44590/0 (socket says 172.21.15.3:44590) 2022-07-21T01:35:16.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa293fff700 1 -- 172.21.15.3:0/1713904069 learned_addr learned my addr 172.21.15.3:0/1713904069 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:16.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a0b50700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa288003750 0x7fa29c0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a1351700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c13e7c0 0x7fa29c13ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa293fff700 1 -- 172.21.15.3:0/1713904069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c13e7c0 msgr2=0x7fa29c13ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa293fff700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c13e7c0 0x7fa29c13ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa293fff700 1 -- 172.21.15.3:0/1713904069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa288003750 msgr2=0x7fa29c0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa293fff700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa288003750 0x7fa29c0b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa293fff700 1 -- 172.21.15.3:0/1713904069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c1456c0 con 0x7fa29c0b4da0 2022-07-21T01:35:16.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa293fff700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa29c0b4da0 0x7fa29c13e2d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa29c0b51c0 tx=0x7fa28c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa291ffb700 1 -- 172.21.15.3:0/1713904069 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa28c017070 con 0x7fa29c0b4da0 2022-07-21T01:35:16.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa291ffb700 1 -- 172.21.15.3:0/1713904069 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28c000e90 con 0x7fa29c0b4da0 2022-07-21T01:35:16.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c145960 con 0x7fa29c0b4da0 2022-07-21T01:35:16.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa291ffb700 1 -- 172.21.15.3:0/1713904069 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa28c012950 con 0x7fa29c0b4da0 2022-07-21T01:35:16.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.185+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c143eb0 con 0x7fa29c0b4da0 2022-07-21T01:35:16.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.189+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c0574e0 con 0x7fa29c0b4da0 2022-07-21T01:35:16.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.189+0000 7fa291ffb700 1 -- 172.21.15.3:0/1713904069 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa28c012ab0 con 0x7fa29c0b4da0 2022-07-21T01:35:16.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.189+0000 7fa291ffb700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa27c053b30 0x7fa27c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.189+0000 7fa291ffb700 1 -- 172.21.15.3:0/1713904069 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa28c06bb90 con 0x7fa29c0b4da0 2022-07-21T01:35:16.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.189+0000 7fa2a0b50700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa27c053b30 0x7fa27c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.189+0000 7fa2a0b50700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa27c053b30 0x7fa27c055ff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa29801a000 tx=0x7fa298015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.193+0000 7fa291ffb700 1 -- 172.21.15.3:0/1713904069 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa28c042c90 con 0x7fa29c0b4da0 2022-07-21T01:35:16.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.321+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa29c05f3d0 con 0x7fa29c0b4da0 2022-07-21T01:35:16.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.321+0000 7fa291ffb700 1 -- 172.21.15.3:0/1713904069 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v43) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa28c0462b0 con 0x7fa29c0b4da0 2022-07-21T01:35:16.324 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:35:16.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa27c053b30 msgr2=0x7fa27c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa27c053b30 0x7fa27c055ff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa29801a000 tx=0x7fa298015000).stop 2022-07-21T01:35:16.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa29c0b4da0 msgr2=0x7fa29c13e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa29c0b4da0 0x7fa29c13e2d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa29c0b51c0 tx=0x7fa28c008040).stop 2022-07-21T01:35:16.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 shutdown_connections 2022-07-21T01:35:16.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa29c13e7c0 0x7fa29c13ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa29c0b4da0 0x7fa29c13e2d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa27c053b30 0x7fa27c055ff0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 --2- 172.21.15.3:0/1713904069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa288003750 0x7fa29c0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 >> 172.21.15.3:0/1713904069 conn(0x7fa29c0626f0 msgr2=0x7fa29c0acd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:16.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 shutdown_connections 2022-07-21T01:35:16.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.325+0000 7fa2a1b52700 1 -- 172.21.15.3:0/1713904069 wait complete. 2022-07-21T01:35:16.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:35:16.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.457+0000 7fa40a13b700 1 Processor -- start 2022-07-21T01:35:16.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.457+0000 7fa40a13b700 1 -- start start 2022-07-21T01:35:16.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa40413d9b0 0x7fa4040609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4040b0e60 0x7fa404060eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa404066af0 con 0x7fa4040b6510 2022-07-21T01:35:16.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa404066d90 con 0x7fa4040b0e60 2022-07-21T01:35:16.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa404067030 con 0x7fa40413d9b0 2022-07-21T01:35:16.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa409139700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa4040b6510 0x7fa40413d2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52878/0 (socket says 172.21.15.3:52878) 2022-07-21T01:35:16.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40993a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4040b0e60 0x7fa404060eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa409139700 1 -- 172.21.15.3:0/2180521300 learned_addr learned my addr 172.21.15.3:0/2180521300 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:16.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa408938700 1 --2- 172.21.15.3:0/2180521300 >> v2:172.21.15.143:3300/0 conn(0x7fa40413d9b0 0x7fa4040609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa3f3fff700 1 -- 172.21.15.3:0/2180521300 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216478514 0 0) 0x7fa404066af0 con 0x7fa4040b6510 2022-07-21T01:35:16.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa3f3fff700 1 -- 172.21.15.3:0/2180521300 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ec003260 con 0x7fa4040b6510 2022-07-21T01:35:16.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40993a700 1 -- 172.21.15.3:0/2180521300 >> v2:172.21.15.143:3300/0 conn(0x7fa40413d9b0 msgr2=0x7fa4040609c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:16.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40993a700 1 --2- 172.21.15.3:0/2180521300 >> v2:172.21.15.143:3300/0 conn(0x7fa40413d9b0 0x7fa4040609c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40993a700 1 -- 172.21.15.3:0/2180521300 >> v1:172.21.15.3:6789/0 conn(0x7fa4040b6510 legacy=0x7fa40413d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40993a700 1 -- 172.21.15.3:0/2180521300 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4040672d0 con 0x7fa4040b0e60 2022-07-21T01:35:16.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40993a700 1 --2- 172.21.15.3:0/2180521300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b0e60 0x7fa404060eb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fa3fc0032f0 tx=0x7fa3fc0091d0).ready entity=mon.0 client_cookie=898e3fd58dcb0cdb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa3f3fff700 1 -- 172.21.15.3:0/2180521300 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3fc00b680 con 0x7fa4040b0e60 2022-07-21T01:35:16.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa3f3fff700 1 -- 172.21.15.3:0/2180521300 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3fc00b7e0 con 0x7fa4040b0e60 2022-07-21T01:35:16.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa3f3fff700 1 -- 172.21.15.3:0/2180521300 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3fc00b950 con 0x7fa4040b0e60 2022-07-21T01:35:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 -- 172.21.15.3:0/2180521300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b0e60 msgr2=0x7fa404060eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 --2- 172.21.15.3:0/2180521300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b0e60 0x7fa404060eb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fa3fc0032f0 tx=0x7fa3fc0091d0).stop 2022-07-21T01:35:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 -- 172.21.15.3:0/2180521300 shutdown_connections 2022-07-21T01:35:16.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 --2- 172.21.15.3:0/2180521300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b0e60 0x7fa404060eb0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 --2- 172.21.15.3:0/2180521300 >> v2:172.21.15.143:3300/0 conn(0x7fa40413d9b0 0x7fa4040609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 -- 172.21.15.3:0/2180521300 >> 172.21.15.3:0/2180521300 conn(0x7fa404063800 msgr2=0x7fa404063be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:16.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 -- 172.21.15.3:0/2180521300 shutdown_connections 2022-07-21T01:35:16.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.461+0000 7fa40a13b700 1 -- 172.21.15.3:0/2180521300 wait complete. 2022-07-21T01:35:16.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 Processor -- start 2022-07-21T01:35:16.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 -- start start 2022-07-21T01:35:16.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4040b0e60 0x7fa40409ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b6510 0x7fa4040a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa40413d9b0 0x7fa4040a9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4040aa1c0 con 0x7fa4040b6510 2022-07-21T01:35:16.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40414fbf0 con 0x7fa40413d9b0 2022-07-21T01:35:16.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4040ad710 con 0x7fa4040b0e60 2022-07-21T01:35:16.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa408938700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b6510 0x7fa4040a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa408938700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b6510 0x7fa4040a97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44604/0 (socket says 172.21.15.3:44604) 2022-07-21T01:35:16.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa408938700 1 -- 172.21.15.3:0/226673082 learned_addr learned my addr 172.21.15.3:0/226673082 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:16.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa409139700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4040b0e60 0x7fa40409ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40993a700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa40413d9b0 0x7fa4040a9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa408938700 1 -- 172.21.15.3:0/226673082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4040b0e60 msgr2=0x7fa40409ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa408938700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4040b0e60 0x7fa40409ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa408938700 1 -- 172.21.15.3:0/226673082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa40413d9b0 msgr2=0x7fa4040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa408938700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa40413d9b0 0x7fa4040a9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa408938700 1 -- 172.21.15.3:0/226673082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc009470 con 0x7fa4040b6510 2022-07-21T01:35:16.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40993a700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa40413d9b0 0x7fa4040a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:16.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa408938700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b6510 0x7fa4040a97e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa3f400a000 tx=0x7fa3f4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa3f27fc700 1 -- 172.21.15.3:0/226673082 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3f40099f0 con 0x7fa4040b6510 2022-07-21T01:35:16.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa3f27fc700 1 -- 172.21.15.3:0/226673082 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4009b50 con 0x7fa4040b6510 2022-07-21T01:35:16.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4040adbb0 con 0x7fa4040b6510 2022-07-21T01:35:16.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa3f27fc700 1 -- 172.21.15.3:0/226673082 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3f401a460 con 0x7fa4040b6510 2022-07-21T01:35:16.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.465+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4040aca70 con 0x7fa4040b6510 2022-07-21T01:35:16.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.469+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40414fa60 con 0x7fa4040b6510 2022-07-21T01:35:16.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.469+0000 7fa3f27fc700 1 -- 172.21.15.3:0/226673082 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa3f4021060 con 0x7fa4040b6510 2022-07-21T01:35:16.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.469+0000 7fa3f27fc700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3e4053aa0 0x7fa3e4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.469+0000 7fa3f27fc700 1 -- 172.21.15.3:0/226673082 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa3f406df00 con 0x7fa4040b6510 2022-07-21T01:35:16.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.469+0000 7fa409139700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3e4053aa0 0x7fa3e4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.469+0000 7fa409139700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3e4053aa0 0x7fa3e4055f60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fa40000d000 tx=0x7fa400009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.477+0000 7fa3f27fc700 1 -- 172.21.15.3:0/226673082 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa3f4045000 con 0x7fa4040b6510 2022-07-21T01:35:16.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.605+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa40414fdf0 con 0x7fa4040b6510 2022-07-21T01:35:16.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa3f27fc700 1 -- 172.21.15.3:0/226673082 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v43) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa3f4048620 con 0x7fa4040b6510 2022-07-21T01:35:16.609 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:35:16.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3e4053aa0 msgr2=0x7fa3e4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3e4053aa0 0x7fa3e4055f60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fa40000d000 tx=0x7fa400009040).stop 2022-07-21T01:35:16.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b6510 msgr2=0x7fa4040a97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b6510 0x7fa4040a97e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa3f400a000 tx=0x7fa3f4008040).stop 2022-07-21T01:35:16.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 shutdown_connections 2022-07-21T01:35:16.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3e4053aa0 0x7fa3e4055f60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa40413d9b0 0x7fa4040a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4040b6510 0x7fa4040a97e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 --2- 172.21.15.3:0/226673082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4040b0e60 0x7fa40409ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 >> 172.21.15.3:0/226673082 conn(0x7fa404063800 msgr2=0x7fa4040b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:16.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 shutdown_connections 2022-07-21T01:35:16.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.609+0000 7fa40a13b700 1 -- 172.21.15.3:0/226673082 wait complete. 2022-07-21T01:35:16.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:35:16.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 Processor -- start 2022-07-21T01:35:16.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 -- start start 2022-07-21T01:35:16.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fadcc0b1f40 0x7fadcc0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fadcc148530 0x7fadcc0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc0b04e0 con 0x7fadcc0a7250 2022-07-21T01:35:16.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadcc0a9250 con 0x7fadcc148530 2022-07-21T01:35:16.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadcc0b0780 con 0x7fadcc0b1f40 2022-07-21T01:35:16.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcbfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fadcc148530 0x7fadcc0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcbfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fadcc148530 0x7fadcc0b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44612/0 (socket says 172.21.15.3:44612) 2022-07-21T01:35:16.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcbfff700 1 -- 172.21.15.3:0/1496242368 learned_addr learned my addr 172.21.15.3:0/1496242368 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:16.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd0cfe700 1 --2- 172.21.15.3:0/1496242368 >> v2:172.21.15.143:3300/0 conn(0x7fadcc0b1f40 0x7fadcc0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcbfff700 1 -- 172.21.15.3:0/1496242368 >> v2:172.21.15.143:3300/0 conn(0x7fadcc0b1f40 msgr2=0x7fadcc0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:16.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcbfff700 1 --2- 172.21.15.3:0/1496242368 >> v2:172.21.15.143:3300/0 conn(0x7fadcc0b1f40 0x7fadcc0b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcbfff700 1 -- 172.21.15.3:0/1496242368 >> v1:172.21.15.3:6789/0 conn(0x7fadcc0a7250 legacy=0x7fadcc1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcbfff700 1 -- 172.21.15.3:0/1496242368 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc0b0a20 con 0x7fadcc148530 2022-07-21T01:35:16.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcbfff700 1 --2- 172.21.15.3:0/1496242368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc148530 0x7fadcc0b2320 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fadcc1478c0 tx=0x7fadc4009290).ready entity=mon.0 client_cookie=1a4c21148eef6a0d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcb7fe700 1 -- 172.21.15.3:0/1496242368 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadc400b860 con 0x7fadcc148530 2022-07-21T01:35:16.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcb7fe700 1 -- 172.21.15.3:0/1496242368 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc400b9c0 con 0x7fadcc148530 2022-07-21T01:35:16.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadcb7fe700 1 -- 172.21.15.3:0/1496242368 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadc400bb30 con 0x7fadcc148530 2022-07-21T01:35:16.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 -- 172.21.15.3:0/1496242368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc148530 msgr2=0x7fadcc0b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 --2- 172.21.15.3:0/1496242368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc148530 0x7fadcc0b2320 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fadcc1478c0 tx=0x7fadc4009290).stop 2022-07-21T01:35:16.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 -- 172.21.15.3:0/1496242368 shutdown_connections 2022-07-21T01:35:16.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 --2- 172.21.15.3:0/1496242368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc148530 0x7fadcc0b2320 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 --2- 172.21.15.3:0/1496242368 >> v2:172.21.15.143:3300/0 conn(0x7fadcc0b1f40 0x7fadcc0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 -- 172.21.15.3:0/1496242368 >> 172.21.15.3:0/1496242368 conn(0x7fadcc05f840 msgr2=0x7fadcc05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:16.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 -- 172.21.15.3:0/1496242368 shutdown_connections 2022-07-21T01:35:16.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.725+0000 7fadd2f62700 1 -- 172.21.15.3:0/1496242368 wait complete. 2022-07-21T01:35:16.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 Processor -- start 2022-07-21T01:35:16.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 -- start start 2022-07-21T01:35:16.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc0a7250 0x7fadcc14b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc0b1f40 0x7fadcc156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc148530 0x7fadcc157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc14b2c0 con 0x7fadcc0b1f40 2022-07-21T01:35:16.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc0aa0e0 con 0x7fadcc148530 2022-07-21T01:35:16.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc0aa380 con 0x7fadcc0a7250 2022-07-21T01:35:16.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadcbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc0b1f40 0x7fadcc156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadcbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc0b1f40 0x7fadcc156e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44620/0 (socket says 172.21.15.3:44620) 2022-07-21T01:35:16.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadcbfff700 1 -- 172.21.15.3:0/1667462777 learned_addr learned my addr 172.21.15.3:0/1667462777 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:16.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd0cfe700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc0a7250 0x7fadcc14b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd14ff700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc148530 0x7fadcc157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadcbfff700 1 -- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc0a7250 msgr2=0x7fadcc14b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadcbfff700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc0a7250 0x7fadcc14b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadcbfff700 1 -- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc148530 msgr2=0x7fadcc157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadcbfff700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc148530 0x7fadcc157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadcbfff700 1 -- 172.21.15.3:0/1667462777 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc4009530 con 0x7fadcc0b1f40 2022-07-21T01:35:16.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd0cfe700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc0a7250 0x7fadcc14b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:35:16.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd14ff700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc148530 0x7fadcc157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:35:16.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadcbfff700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc0b1f40 0x7fadcc156e20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fadcc1478c0 tx=0x7fadc4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadc9ffb700 1 -- 172.21.15.3:0/1667462777 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadc401f3f0 con 0x7fadcc0b1f40 2022-07-21T01:35:16.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadc9ffb700 1 -- 172.21.15.3:0/1667462777 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc401f550 con 0x7fadcc0b1f40 2022-07-21T01:35:16.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadcc0aa620 con 0x7fadcc0b1f40 2022-07-21T01:35:16.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadc9ffb700 1 -- 172.21.15.3:0/1667462777 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadc401f8b0 con 0x7fadcc0b1f40 2022-07-21T01:35:16.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.729+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadcc0aae80 con 0x7fadcc0b1f40 2022-07-21T01:35:16.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.733+0000 7fadc9ffb700 1 -- 172.21.15.3:0/1667462777 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fadc4026030 con 0x7fadcc0b1f40 2022-07-21T01:35:16.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.733+0000 7fadc9ffb700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fada4053b30 0x7fada4055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.733+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadcc0b0cc0 con 0x7fadcc0b1f40 2022-07-21T01:35:16.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.733+0000 7fadc9ffb700 1 -- 172.21.15.3:0/1667462777 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fadc406c8a0 con 0x7fadcc0b1f40 2022-07-21T01:35:16.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.733+0000 7fadd0cfe700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fada4053b30 0x7fada4055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.733+0000 7fadd0cfe700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fada4053b30 0x7fada4055ff0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fadbc008000 tx=0x7fadbc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.737+0000 7fadc9ffb700 1 -- 172.21.15.3:0/1667462777 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fadc4024020 con 0x7fadcc0b1f40 2022-07-21T01:35:16.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fadcc0574e0 con 0x7fadcc0b1f40 2022-07-21T01:35:16.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadc9ffb700 1 -- 172.21.15.3:0/1667462777 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v43) v1 ==== 84+0+143 (secure 0 0 0) 0x7fadc4024030 con 0x7fadcc0b1f40 2022-07-21T01:35:16.865 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:16.866 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:35:16.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:16.866 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:35:16.866 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:35:16.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:16.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:35:16.867 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:35:16.867 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:35:16.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:16.868 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:35:16.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fada4053b30 msgr2=0x7fada4055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fada4053b30 0x7fada4055ff0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fadbc008000 tx=0x7fadbc008040).stop 2022-07-21T01:35:16.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc0b1f40 msgr2=0x7fadcc156e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc0b1f40 0x7fadcc156e20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fadcc1478c0 tx=0x7fadc4016000).stop 2022-07-21T01:35:16.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 shutdown_connections 2022-07-21T01:35:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fada4053b30 0x7fada4055ff0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadcc148530 0x7fadcc157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadcc0b1f40 0x7fadcc156e20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 --2- 172.21.15.3:0/1667462777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadcc0a7250 0x7fadcc14b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 >> 172.21.15.3:0/1667462777 conn(0x7fadcc05f840 msgr2=0x7fadcc144bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 shutdown_connections 2022-07-21T01:35:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.865+0000 7fadd2f62700 1 -- 172.21.15.3:0/1667462777 wait complete. 2022-07-21T01:35:16.878 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:35:16.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:35:16.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574d1fc700 1 Processor -- start 2022-07-21T01:35:16.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574d1fc700 1 -- start start 2022-07-21T01:35:16.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574d1fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f574805fd90 0x7f57480b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574d1fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f57480b66f0 0x7f57480ba130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574d1fc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57481440f0 con 0x7f57480b5b20 2022-07-21T01:35:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574d1fc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5748060170 con 0x7f57480b66f0 2022-07-21T01:35:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574d1fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5748143be0 con 0x7f574805fd90 2022-07-21T01:35:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f573ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f57480b66f0 0x7f57480ba130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f573ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f57480b66f0 0x7f57480ba130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44628/0 (socket says 172.21.15.3:44628) 2022-07-21T01:35:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f573ffff700 1 -- 172.21.15.3:0/4254935155 learned_addr learned my addr 172.21.15.3:0/4254935155 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f5746d9d700 1 --2- 172.21.15.3:0/4254935155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f574805fd90 0x7f57480b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:16.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574659c700 1 -- 172.21.15.3:0/4254935155 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765129116 0 0) 0x7f57481440f0 con 0x7f57480b5b20 2022-07-21T01:35:16.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f573ffff700 1 -- 172.21.15.3:0/4254935155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f574805fd90 msgr2=0x7f57480b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:16.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f573ffff700 1 --2- 172.21.15.3:0/4254935155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f574805fd90 0x7f57480b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:16.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f573ffff700 1 -- 172.21.15.3:0/4254935155 >> v1:172.21.15.143:6789/0 conn(0x7f57480b5b20 legacy=0x7f57480b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:16.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f573ffff700 1 -- 172.21.15.3:0/4254935155 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5748143e50 con 0x7f57480b66f0 2022-07-21T01:35:16.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f573ffff700 1 --2- 172.21.15.3:0/4254935155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 0x7f57480ba130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f5730002f70 tx=0x7f5730006eb0).ready entity=mon.0 client_cookie=1d5ca9626776bdbe server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:16.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574659c700 1 -- 172.21.15.3:0/4254935155 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f573000d570 con 0x7f57480b66f0 2022-07-21T01:35:17.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574659c700 1 -- 172.21.15.3:0/4254935155 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5730000e90 con 0x7f57480b66f0 2022-07-21T01:35:17.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.993+0000 7f574659c700 1 -- 172.21.15.3:0/4254935155 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f573000d8c0 con 0x7f57480b66f0 2022-07-21T01:35:17.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 -- 172.21.15.3:0/4254935155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 msgr2=0x7f57480ba130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:17.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 --2- 172.21.15.3:0/4254935155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 0x7f57480ba130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f5730002f70 tx=0x7f5730006eb0).stop 2022-07-21T01:35:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 -- 172.21.15.3:0/4254935155 shutdown_connections 2022-07-21T01:35:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 --2- 172.21.15.3:0/4254935155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 0x7f57480ba130 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 --2- 172.21.15.3:0/4254935155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f574805fd90 0x7f57480b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 -- 172.21.15.3:0/4254935155 >> 172.21.15.3:0/4254935155 conn(0x7f5748062530 msgr2=0x7f5748066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 -- 172.21.15.3:0/4254935155 shutdown_connections 2022-07-21T01:35:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 -- 172.21.15.3:0/4254935155 wait complete. 2022-07-21T01:35:17.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 Processor -- start 2022-07-21T01:35:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:16.997+0000 7f574d1fc700 1 -- start start 2022-07-21T01:35:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574d1fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f574805fd90 0x7f57480b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574d1fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57480b5b20 0x7f574813fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574d1fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 0x7f574812d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574d1fc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574812f9f0 con 0x7f57480b66f0 2022-07-21T01:35:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574d1fc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574812fc90 con 0x7f574805fd90 2022-07-21T01:35:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574d1fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574812ff30 con 0x7f57480b5b20 2022-07-21T01:35:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 0x7f574812d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 0x7f574812d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44636/0 (socket says 172.21.15.3:44636) 2022-07-21T01:35:17.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574759e700 1 -- 172.21.15.3:0/1482466718 learned_addr learned my addr 172.21.15.3:0/1482466718 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:17.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f5746d9d700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f574805fd90 0x7f57480b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:17.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f573ffff700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57480b5b20 0x7f574813fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:17.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574759e700 1 -- 172.21.15.3:0/1482466718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57480b5b20 msgr2=0x7f574813fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:17.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574759e700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57480b5b20 0x7f574813fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:17.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574759e700 1 -- 172.21.15.3:0/1482466718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f574805fd90 msgr2=0x7f57480b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:17.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574759e700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f574805fd90 0x7f57480b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:17.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574759e700 1 -- 172.21.15.3:0/1482466718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573000b1e0 con 0x7f57480b66f0 2022-07-21T01:35:17.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574759e700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 0x7f574812d6d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5740000f40 tx=0x7f5740007d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:17.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f5744d99700 1 -- 172.21.15.3:0/1482466718 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f574000f470 con 0x7f57480b66f0 2022-07-21T01:35:17.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f5744d99700 1 -- 172.21.15.3:0/1482466718 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574000f5d0 con 0x7f57480b66f0 2022-07-21T01:35:17.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57481301d0 con 0x7f57480b66f0 2022-07-21T01:35:17.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f5744d99700 1 -- 172.21.15.3:0/1482466718 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f574000f930 con 0x7f57480b66f0 2022-07-21T01:35:17.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574813ff70 con 0x7f57480b66f0 2022-07-21T01:35:17.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.001+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57480b99f0 con 0x7f57480b66f0 2022-07-21T01:35:17.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.005+0000 7f5744d99700 1 -- 172.21.15.3:0/1482466718 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5740010070 con 0x7f57480b66f0 2022-07-21T01:35:17.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.005+0000 7f5744d99700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f572c053b30 0x7f572c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:17.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.005+0000 7f5744d99700 1 -- 172.21.15.3:0/1482466718 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f574006b130 con 0x7f57480b66f0 2022-07-21T01:35:17.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.005+0000 7f5746d9d700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f572c053b30 0x7f572c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:17.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.005+0000 7f5746d9d700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f572c053b30 0x7f572c055ff0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f573800b000 tx=0x7f5738006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:17.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.013+0000 7f5744d99700 1 -- 172.21.15.3:0/1482466718 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f57400420a0 con 0x7f57480b66f0 2022-07-21T01:35:17.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.133+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f57480574e0 con 0x7f57480b66f0 2022-07-21T01:35:18.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:17.997+0000 7f5744d99700 1 -- 172.21.15.3:0/1482466718 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v44) v1 ==== 133+0+0 (secure 0 0 0) 0x7f57400456c0 con 0x7f57480b66f0 2022-07-21T01:35:18.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f572c053b30 msgr2=0x7f572c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f572c053b30 0x7f572c055ff0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f573800b000 tx=0x7f5738006df0).stop 2022-07-21T01:35:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 msgr2=0x7f574812d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 0x7f574812d6d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5740000f40 tx=0x7f5740007d60).stop 2022-07-21T01:35:18.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 shutdown_connections 2022-07-21T01:35:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57480b66f0 0x7f574812d6d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57480b5b20 0x7f574813fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f572c053b30 0x7f572c055ff0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 --2- 172.21.15.3:0/1482466718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f574805fd90 0x7f57480b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 >> 172.21.15.3:0/1482466718 conn(0x7f5748062530 msgr2=0x7f574812c270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:18.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 shutdown_connections 2022-07-21T01:35:18.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.005+0000 7f574d1fc700 1 -- 172.21.15.3:0/1482466718 wait complete. 2022-07-21T01:35:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:35:18.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:35:18.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9eac7700 1 Processor -- start 2022-07-21T01:35:18.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9eac7700 1 -- start start 2022-07-21T01:35:18.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9eac7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbb980af550 0x7fbb980b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9eac7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb98060330 0x7fbb981445e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9eac7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb98144ad0 0x7fbb980a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9eac7700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb980b15e0 con 0x7fbb98060330 2022-07-21T01:35:18.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9eac7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb980aa630 con 0x7fbb980af550 2022-07-21T01:35:18.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9eac7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9805f840 con 0x7fbb98144ad0 2022-07-21T01:35:18.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9d2c4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb98060330 0x7fbb981445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9d2c4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbb98060330 0x7fbb981445e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44642/0 (socket says 172.21.15.3:44642) 2022-07-21T01:35:18.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9dac5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbb980af550 0x7fbb980b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9d2c4700 1 -- 172.21.15.3:0/2830429530 learned_addr learned my addr 172.21.15.3:0/2830429530 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:18.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9d2c4700 1 -- 172.21.15.3:0/2830429530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb98144ad0 msgr2=0x7fbb980a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:18.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9d2c4700 1 --2- 172.21.15.3:0/2830429530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb98144ad0 0x7fbb980a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:35:18.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9d2c4700 1 -- 172.21.15.3:0/2830429530 >> v2:172.21.15.143:3300/0 conn(0x7fbb980af550 msgr2=0x7fbb980b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:18.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9d2c4700 1 --2- 172.21.15.3:0/2830429530 >> v2:172.21.15.143:3300/0 conn(0x7fbb980af550 0x7fbb980b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9d2c4700 1 -- 172.21.15.3:0/2830429530 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9805fd10 con 0x7fbb98060330 2022-07-21T01:35:18.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9dac5700 1 --2- 172.21.15.3:0/2830429530 >> v2:172.21.15.143:3300/0 conn(0x7fbb980af550 0x7fbb980b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:18.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.133+0000 7fbb9d2c4700 1 --2- 172.21.15.3:0/2830429530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98060330 0x7fbb981445e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fbb88002f70 tx=0x7fbb88006eb0).ready entity=mon.0 client_cookie=6c5fe4b665c337aa server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9cac3700 1 -- 172.21.15.3:0/2830429530 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb8800d570 con 0x7fbb98060330 2022-07-21T01:35:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9cac3700 1 -- 172.21.15.3:0/2830429530 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb88000e90 con 0x7fbb98060330 2022-07-21T01:35:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9cac3700 1 -- 172.21.15.3:0/2830429530 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb8800d900 con 0x7fbb98060330 2022-07-21T01:35:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9eac7700 1 -- 172.21.15.3:0/2830429530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98060330 msgr2=0x7fbb981445e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/2830429530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98060330 0x7fbb981445e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fbb88002f70 tx=0x7fbb88006eb0).stop 2022-07-21T01:35:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9eac7700 1 -- 172.21.15.3:0/2830429530 shutdown_connections 2022-07-21T01:35:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/2830429530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb98144ad0 0x7fbb980a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:35:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/2830429530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98060330 0x7fbb981445e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/2830429530 >> v2:172.21.15.143:3300/0 conn(0x7fbb980af550 0x7fbb980b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9eac7700 1 -- 172.21.15.3:0/2830429530 >> 172.21.15.3:0/2830429530 conn(0x7fbb980612d0 msgr2=0x7fbb98062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9eac7700 1 -- 172.21.15.3:0/2830429530 shutdown_connections 2022-07-21T01:35:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.137+0000 7fbb9eac7700 1 -- 172.21.15.3:0/2830429530 wait complete. 2022-07-21T01:35:18.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 Processor -- start 2022-07-21T01:35:18.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 -- start start 2022-07-21T01:35:18.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb98060330 0x7fbb9809ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb980af550 0x7fbb9814cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98144ad0 0x7fbb9814d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9813fec0 con 0x7fbb98144ad0 2022-07-21T01:35:18.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb98150960 con 0x7fbb98060330 2022-07-21T01:35:18.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb98150c00 con 0x7fbb980af550 2022-07-21T01:35:18.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9e2c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98144ad0 0x7fbb9814d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9e2c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98144ad0 0x7fbb9814d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44648/0 (socket says 172.21.15.3:44648) 2022-07-21T01:35:18.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9e2c6700 1 -- 172.21.15.3:0/1414233697 learned_addr learned my addr 172.21.15.3:0/1414233697 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:18.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9d2c4700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb980af550 0x7fbb9814cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9dac5700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb98060330 0x7fbb9809ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9e2c6700 1 -- 172.21.15.3:0/1414233697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb980af550 msgr2=0x7fbb9814cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9e2c6700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb980af550 0x7fbb9814cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9e2c6700 1 -- 172.21.15.3:0/1414233697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb98060330 msgr2=0x7fbb9809ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9e2c6700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb98060330 0x7fbb9809ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9e2c6700 1 -- 172.21.15.3:0/1414233697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9400b180 con 0x7fbb98144ad0 2022-07-21T01:35:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9e2c6700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98144ad0 0x7fbb9814d440 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fbb900032f0 tx=0x7fbb90009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb86ffd700 1 -- 172.21.15.3:0/1414233697 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb9000d580 con 0x7fbb98144ad0 2022-07-21T01:35:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb86ffd700 1 -- 172.21.15.3:0/1414233697 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9000d6e0 con 0x7fbb98144ad0 2022-07-21T01:35:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8800b1e0 con 0x7fbb98144ad0 2022-07-21T01:35:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb86ffd700 1 -- 172.21.15.3:0/1414233697 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb9000da40 con 0x7fbb98144ad0 2022-07-21T01:35:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.141+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb98063ad0 con 0x7fbb98144ad0 2022-07-21T01:35:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.145+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb980574e0 con 0x7fbb98144ad0 2022-07-21T01:35:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.145+0000 7fbb86ffd700 1 -- 172.21.15.3:0/1414233697 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbb9000dba0 con 0x7fbb98144ad0 2022-07-21T01:35:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.145+0000 7fbb86ffd700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb78053980 0x7fbb78055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.145+0000 7fbb86ffd700 1 -- 172.21.15.3:0/1414233697 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5954+0+0 (secure 0 0 0) 0x7fbb9006c2d0 con 0x7fbb98144ad0 2022-07-21T01:35:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.145+0000 7fbb9dac5700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb78053980 0x7fbb78055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.145+0000 7fbb9dac5700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb78053980 0x7fbb78055e40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fbb9400b000 tx=0x7fbb94008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.149+0000 7fbb86ffd700 1 -- 172.21.15.3:0/1414233697 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb900431c0 con 0x7fbb98144ad0 2022-07-21T01:35:18.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.281+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fbb98069a70 con 0x7fbb98144ad0 2022-07-21T01:35:18.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.437+0000 7fbb86ffd700 1 -- 172.21.15.3:0/1414233697 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v45) v1 ==== 125+0+0 (secure 0 0 0) 0x7fbb900467e0 con 0x7fbb98144ad0 2022-07-21T01:35:18.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb78053980 msgr2=0x7fbb78055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb78053980 0x7fbb78055e40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fbb9400b000 tx=0x7fbb94008040).stop 2022-07-21T01:35:18.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98144ad0 msgr2=0x7fbb9814d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98144ad0 0x7fbb9814d440 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fbb900032f0 tx=0x7fbb90009220).stop 2022-07-21T01:35:18.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 shutdown_connections 2022-07-21T01:35:18.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb98144ad0 0x7fbb9814d440 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb980af550 0x7fbb9814cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbb78053980 0x7fbb78055e40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 --2- 172.21.15.3:0/1414233697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb98060330 0x7fbb9809ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 >> 172.21.15.3:0/1414233697 conn(0x7fbb980612d0 msgr2=0x7fbb98144390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:18.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 shutdown_connections 2022-07-21T01:35:18.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.441+0000 7fbb9eac7700 1 -- 172.21.15.3:0/1414233697 wait complete. 2022-07-21T01:35:18.446 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:35:18.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:35:18.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.561+0000 7fde5ee45700 1 Processor -- start 2022-07-21T01:35:18.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.561+0000 7fde5ee45700 1 -- start start 2022-07-21T01:35:18.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ee45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde5805c960 0x7fde580b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ee45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805d240 0x7fde5813f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ee45700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde5813fa10 con 0x7fde580acfc0 2022-07-21T01:35:18.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ee45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde58067f00 con 0x7fde5805c960 2022-07-21T01:35:18.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ee45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5805cd40 con 0x7fde5805d240 2022-07-21T01:35:18.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5e644700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fde580acfc0 0x7fde580aee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52938/0 (socket says 172.21.15.3:52938) 2022-07-21T01:35:18.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5e644700 1 -- 172.21.15.3:0/1937471588 learned_addr learned my addr 172.21.15.3:0/1937471588 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:18.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5d642700 1 --2- 172.21.15.3:0/1937471588 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805d240 0x7fde5813f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5de43700 1 --2- 172.21.15.3:0/1937471588 >> v2:172.21.15.143:3300/0 conn(0x7fde5805c960 0x7fde580b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ce41700 1 -- 172.21.15.3:0/1937471588 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262845721 0 0) 0x7fde50001490 con 0x7fde580acfc0 2022-07-21T01:35:18.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ce41700 1 -- 172.21.15.3:0/1937471588 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde40004370 con 0x7fde580acfc0 2022-07-21T01:35:18.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5d642700 1 -- 172.21.15.3:0/1937471588 >> v2:172.21.15.143:3300/0 conn(0x7fde5805c960 msgr2=0x7fde580b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:18.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5d642700 1 --2- 172.21.15.3:0/1937471588 >> v2:172.21.15.143:3300/0 conn(0x7fde5805c960 0x7fde580b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5d642700 1 -- 172.21.15.3:0/1937471588 >> v1:172.21.15.3:6789/0 conn(0x7fde580acfc0 legacy=0x7fde580aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5d642700 1 -- 172.21.15.3:0/1937471588 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580b1920 con 0x7fde5805d240 2022-07-21T01:35:18.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5de43700 1 --2- 172.21.15.3:0/1937471588 >> v2:172.21.15.143:3300/0 conn(0x7fde5805c960 0x7fde580b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:18.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5d642700 1 --2- 172.21.15.3:0/1937471588 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805d240 0x7fde5813f520 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fde4c002f70 tx=0x7fde4c006df0).ready entity=mon.2 client_cookie=c01a8f421eff849e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:18.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ce41700 1 -- 172.21.15.3:0/1937471588 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde4c00d4a0 con 0x7fde5805d240 2022-07-21T01:35:18.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ce41700 1 -- 172.21.15.3:0/1937471588 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4c000e90 con 0x7fde5805d240 2022-07-21T01:35:18.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.565+0000 7fde5ce41700 1 -- 172.21.15.3:0/1937471588 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde4c00d7f0 con 0x7fde5805d240 2022-07-21T01:35:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 -- 172.21.15.3:0/1937471588 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805d240 msgr2=0x7fde5813f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 --2- 172.21.15.3:0/1937471588 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805d240 0x7fde5813f520 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fde4c002f70 tx=0x7fde4c006df0).stop 2022-07-21T01:35:18.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 -- 172.21.15.3:0/1937471588 shutdown_connections 2022-07-21T01:35:18.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 --2- 172.21.15.3:0/1937471588 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805d240 0x7fde5813f520 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 --2- 172.21.15.3:0/1937471588 >> v2:172.21.15.143:3300/0 conn(0x7fde5805c960 0x7fde580b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 -- 172.21.15.3:0/1937471588 >> 172.21.15.3:0/1937471588 conn(0x7fde58060310 msgr2=0x7fde580b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 -- 172.21.15.3:0/1937471588 shutdown_connections 2022-07-21T01:35:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 -- 172.21.15.3:0/1937471588 wait complete. 2022-07-21T01:35:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 Processor -- start 2022-07-21T01:35:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.569+0000 7fde5ee45700 1 -- start start 2022-07-21T01:35:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5ee45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805c960 0x7fde5814e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5ee45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5805d240 0x7fde580a9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5ee45700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde580acfc0 0x7fde58142b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5ee45700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde580ab830 con 0x7fde580acfc0 2022-07-21T01:35:18.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5ee45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde581416f0 con 0x7fde5805d240 2022-07-21T01:35:18.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5ee45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5814ea90 con 0x7fde5805c960 2022-07-21T01:35:18.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5e644700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde580acfc0 0x7fde58142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5e644700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde580acfc0 0x7fde58142b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44664/0 (socket says 172.21.15.3:44664) 2022-07-21T01:35:18.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5e644700 1 -- 172.21.15.3:0/4259001301 learned_addr learned my addr 172.21.15.3:0/4259001301 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:18.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5d642700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5805d240 0x7fde580a9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5de43700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805c960 0x7fde5814e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5e644700 1 -- 172.21.15.3:0/4259001301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805c960 msgr2=0x7fde5814e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5e644700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805c960 0x7fde5814e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5e644700 1 -- 172.21.15.3:0/4259001301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5805d240 msgr2=0x7fde580a9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5e644700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5805d240 0x7fde580a9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5e644700 1 -- 172.21.15.3:0/4259001301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5400b180 con 0x7fde580acfc0 2022-07-21T01:35:18.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5e644700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde580acfc0 0x7fde58142b50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fde50001620 tx=0x7fde50007c60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:18.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde4affd700 1 -- 172.21.15.3:0/4259001301 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde50005ba0 con 0x7fde580acfc0 2022-07-21T01:35:18.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde4affd700 1 -- 172.21.15.3:0/4259001301 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde50005d00 con 0x7fde580acfc0 2022-07-21T01:35:18.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c00b180 con 0x7fde580acfc0 2022-07-21T01:35:18.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde4affd700 1 -- 172.21.15.3:0/4259001301 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde500072c0 con 0x7fde580acfc0 2022-07-21T01:35:18.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.573+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5814efd0 con 0x7fde580acfc0 2022-07-21T01:35:18.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.577+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde580574e0 con 0x7fde580acfc0 2022-07-21T01:35:18.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.577+0000 7fde4affd700 1 -- 172.21.15.3:0/4259001301 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fde50010070 con 0x7fde580acfc0 2022-07-21T01:35:18.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.577+0000 7fde4affd700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde38053a70 0x7fde38055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.577+0000 7fde4affd700 1 -- 172.21.15.3:0/4259001301 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6304+0+0 (secure 0 0 0) 0x7fde5006b8b0 con 0x7fde580acfc0 2022-07-21T01:35:18.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.577+0000 7fde5de43700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde38053a70 0x7fde38055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.577+0000 7fde5de43700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde38053a70 0x7fde38055f30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fde5400b000 tx=0x7fde54006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:18.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.585+0000 7fde4affd700 1 -- 172.21.15.3:0/4259001301 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fde500425e0 con 0x7fde580acfc0 2022-07-21T01:35:18.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.749+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fde5813fa10 con 0x7fde580acfc0 2022-07-21T01:35:18.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.761+0000 7fde4affd700 1 -- 172.21.15.3:0/4259001301 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 7 and data pool 8 v17) v1 ==== 187+0+0 (secure 0 0 0) 0x7fde5813fa10 con 0x7fde580acfc0 2022-07-21T01:35:18.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde38053a70 msgr2=0x7fde38055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde38053a70 0x7fde38055f30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fde5400b000 tx=0x7fde54006df0).stop 2022-07-21T01:35:18.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde580acfc0 msgr2=0x7fde58142b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde580acfc0 0x7fde58142b50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fde50001620 tx=0x7fde50007c60).stop 2022-07-21T01:35:18.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 shutdown_connections 2022-07-21T01:35:18.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde580acfc0 0x7fde58142b50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde38053a70 0x7fde38055f30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5805d240 0x7fde580a9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 --2- 172.21.15.3:0/4259001301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5805c960 0x7fde5814e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 >> 172.21.15.3:0/4259001301 conn(0x7fde58060310 msgr2=0x7fde580b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:18.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 shutdown_connections 2022-07-21T01:35:18.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.765+0000 7fde5ee45700 1 -- 172.21.15.3:0/4259001301 wait complete. 2022-07-21T01:35:18.770 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 7 and data pool 8 2022-07-21T01:35:18.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:35:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.877+0000 7f64d359e700 1 Processor -- start 2022-07-21T01:35:18.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.877+0000 7f64d359e700 1 -- start start 2022-07-21T01:35:18.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f64d40b6420 0x7f64d4061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f64d40b41f0 0x7f64d4061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d4061fe0 0x7f64d41465c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d40b60e0 con 0x7f64d40b41f0 2022-07-21T01:35:18.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d40b2100 con 0x7f64d40b6420 2022-07-21T01:35:18.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40b23a0 con 0x7f64d4061fe0 2022-07-21T01:35:18.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d1d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f64d40b41f0 0x7f64d4061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d1d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f64d40b41f0 0x7f64d4061af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44682/0 (socket says 172.21.15.3:44682) 2022-07-21T01:35:18.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d1d9b700 1 -- 172.21.15.3:0/2306022822 learned_addr learned my addr 172.21.15.3:0/2306022822 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:18.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d2d9d700 1 --2- 172.21.15.3:0/2306022822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d4061fe0 0x7f64d41465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d259c700 1 --2- 172.21.15.3:0/2306022822 >> v2:172.21.15.143:3300/0 conn(0x7f64d40b6420 0x7f64d4061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d1d9b700 1 -- 172.21.15.3:0/2306022822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d4061fe0 msgr2=0x7f64d41465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:18.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d1d9b700 1 --2- 172.21.15.3:0/2306022822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d4061fe0 0x7f64d41465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d1d9b700 1 -- 172.21.15.3:0/2306022822 >> v2:172.21.15.143:3300/0 conn(0x7f64d40b6420 msgr2=0x7f64d4061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:18.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d1d9b700 1 --2- 172.21.15.3:0/2306022822 >> v2:172.21.15.143:3300/0 conn(0x7f64d40b6420 0x7f64d4061600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d1d9b700 1 -- 172.21.15.3:0/2306022822 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d40b2640 con 0x7f64d40b41f0 2022-07-21T01:35:18.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d1d9b700 1 --2- 172.21.15.3:0/2306022822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d40b41f0 0x7f64d4061af0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f64bc002f70 tx=0x7f64bc006eb0).ready entity=mon.0 client_cookie=9ef743fe7b4a658 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:18.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d159a700 1 -- 172.21.15.3:0/2306022822 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64bc00d570 con 0x7f64d40b41f0 2022-07-21T01:35:18.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d159a700 1 -- 172.21.15.3:0/2306022822 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64bc000e90 con 0x7f64d40b41f0 2022-07-21T01:35:18.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d159a700 1 -- 172.21.15.3:0/2306022822 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64bc00d900 con 0x7f64d40b41f0 2022-07-21T01:35:18.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 -- 172.21.15.3:0/2306022822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d40b41f0 msgr2=0x7f64d4061af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 --2- 172.21.15.3:0/2306022822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d40b41f0 0x7f64d4061af0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f64bc002f70 tx=0x7f64bc006eb0).stop 2022-07-21T01:35:18.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 -- 172.21.15.3:0/2306022822 shutdown_connections 2022-07-21T01:35:18.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 --2- 172.21.15.3:0/2306022822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d4061fe0 0x7f64d41465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 --2- 172.21.15.3:0/2306022822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d40b41f0 0x7f64d4061af0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 --2- 172.21.15.3:0/2306022822 >> v2:172.21.15.143:3300/0 conn(0x7f64d40b6420 0x7f64d4061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 -- 172.21.15.3:0/2306022822 >> 172.21.15.3:0/2306022822 conn(0x7f64d405c620 msgr2=0x7f64d405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:18.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 -- 172.21.15.3:0/2306022822 shutdown_connections 2022-07-21T01:35:18.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.881+0000 7f64d359e700 1 -- 172.21.15.3:0/2306022822 wait complete. 2022-07-21T01:35:18.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 Processor -- start 2022-07-21T01:35:18.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 -- start start 2022-07-21T01:35:18.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d4061fe0 0x7f64d4143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d40b41f0 0x7f64d4141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64d40b6420 0x7f64d4142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4142bf0 con 0x7f64d4061fe0 2022-07-21T01:35:18.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4142880 con 0x7f64d40b6420 2022-07-21T01:35:18.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4152c40 con 0x7f64d40b41f0 2022-07-21T01:35:18.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d4061fe0 0x7f64d4143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d4061fe0 0x7f64d4143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44684/0 (socket says 172.21.15.3:44684) 2022-07-21T01:35:18.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d259c700 1 -- 172.21.15.3:0/251626041 learned_addr learned my addr 172.21.15.3:0/251626041 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:18.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d1d9b700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d40b41f0 0x7f64d4141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d2d9d700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64d40b6420 0x7f64d4142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d259c700 1 -- 172.21.15.3:0/251626041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d40b41f0 msgr2=0x7f64d4141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d259c700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d40b41f0 0x7f64d4141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d259c700 1 -- 172.21.15.3:0/251626041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64d40b6420 msgr2=0x7f64d4142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:18.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d259c700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64d40b6420 0x7f64d4142390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:18.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d259c700 1 -- 172.21.15.3:0/251626041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc00b1e0 con 0x7f64d4061fe0 2022-07-21T01:35:18.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d259c700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d4061fe0 0x7f64d4143c30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f64c400c000 tx=0x7f64c40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:18.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64cb7fe700 1 -- 172.21.15.3:0/251626041 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64c400c6a0 con 0x7f64d4061fe0 2022-07-21T01:35:18.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64cb7fe700 1 -- 172.21.15.3:0/251626041 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c400c800 con 0x7f64d4061fe0 2022-07-21T01:35:18.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d4152ee0 con 0x7f64d4061fe0 2022-07-21T01:35:18.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64cb7fe700 1 -- 172.21.15.3:0/251626041 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64c400cb60 con 0x7f64d4061fe0 2022-07-21T01:35:18.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.885+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d4153420 con 0x7f64d4061fe0 2022-07-21T01:35:18.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.889+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d40b28e0 con 0x7f64d4061fe0 2022-07-21T01:35:18.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.889+0000 7f64cb7fe700 1 -- 172.21.15.3:0/251626041 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f64c4022070 con 0x7f64d4061fe0 2022-07-21T01:35:18.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.889+0000 7f64cb7fe700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64b80539e0 0x7f64b8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:18.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.889+0000 7f64cb7fe700 1 -- 172.21.15.3:0/251626041 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f64c406c860 con 0x7f64d4061fe0 2022-07-21T01:35:18.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.889+0000 7f64d1d9b700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64b80539e0 0x7f64b8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:18.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.889+0000 7f64d1d9b700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64b80539e0 0x7f64b8055ea0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f64bc01c000 tx=0x7f64bc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:18.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:18.897+0000 7f64cb7fe700 1 -- 172.21.15.3:0/251626041 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f64c4043510 con 0x7f64d4061fe0 2022-07-21T01:35:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.029+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64d40574e0 con 0x7f64d4061fe0 2022-07-21T01:35:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.029+0000 7f64cb7fe700 1 -- 172.21.15.3:0/251626041 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16297 (secure 0 0 0) 0x7f64c4046b30 con 0x7f64d4061fe0 2022-07-21T01:35:19.032 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:19.033 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":46,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:35:18.753222+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:35:17.190459+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-21T01:35:18.336558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:35:19.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64b80539e0 msgr2=0x7f64b8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64b80539e0 0x7f64b8055ea0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f64bc01c000 tx=0x7f64bc015000).stop 2022-07-21T01:35:19.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d4061fe0 msgr2=0x7f64d4143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d4061fe0 0x7f64d4143c30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f64c400c000 tx=0x7f64c40091d0).stop 2022-07-21T01:35:19.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 shutdown_connections 2022-07-21T01:35:19.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64b80539e0 0x7f64b8055ea0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64d40b6420 0x7f64d4142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64d40b41f0 0x7f64d4141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 --2- 172.21.15.3:0/251626041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64d4061fe0 0x7f64d4143c30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 >> 172.21.15.3:0/251626041 conn(0x7f64d405c620 msgr2=0x7f64d40b73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:19.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 shutdown_connections 2022-07-21T01:35:19.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.033+0000 7f64d359e700 1 -- 172.21.15.3:0/251626041 wait complete. 2022-07-21T01:35:19.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:35:19.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.145+0000 7f1eeeeee700 1 Processor -- start 2022-07-21T01:35:19.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.145+0000 7f1eeeeee700 1 -- start start 2022-07-21T01:35:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.145+0000 7f1eeeeee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b2710 0x7f1ee814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.145+0000 7f1eeeeee700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1ee80b01a0 0x7f1ee814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.145+0000 7f1eeeeee700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee80a7250 con 0x7f1ee80afd60 2022-07-21T01:35:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.145+0000 7f1eeeeee700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee80a7480 con 0x7f1ee80b01a0 2022-07-21T01:35:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.145+0000 7f1eeeeee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee80a7720 con 0x7f1ee80b2710 2022-07-21T01:35:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eedeec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b2710 0x7f1ee814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eed6eb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1ee80b01a0 0x7f1ee814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eed6eb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1ee80b01a0 0x7f1ee814b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:50142/0 (socket says 172.21.15.3:50142) 2022-07-21T01:35:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eedeec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b2710 0x7f1ee814afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:51608/0 (socket says 172.21.15.3:51608) 2022-07-21T01:35:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eee6ed700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1ee80afd60 0x7f1ee80a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:55092/0 (socket says 172.21.15.3:55092) 2022-07-21T01:35:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eed6eb700 1 -- 172.21.15.3:0/1865285366 learned_addr learned my addr 172.21.15.3:0/1865285366 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eed6eb700 1 -- 172.21.15.3:0/1865285366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b2710 msgr2=0x7f1ee814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eed6eb700 1 --2- 172.21.15.3:0/1865285366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b2710 0x7f1ee814afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eed6eb700 1 -- 172.21.15.3:0/1865285366 >> v1:172.21.15.143:6789/0 conn(0x7f1ee80afd60 legacy=0x7f1ee80a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeceea700 1 -- 172.21.15.3:0/1865285366 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832556252 0 0) 0x7f1ee80a7250 con 0x7f1ee80afd60 2022-07-21T01:35:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eed6eb700 1 -- 172.21.15.3:0/1865285366 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee80a79c0 con 0x7f1ee80b01a0 2022-07-21T01:35:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eedeec700 1 --2- 172.21.15.3:0/1865285366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b2710 0x7f1ee814afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eed6eb700 1 --2- 172.21.15.3:0/1865285366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b01a0 0x7f1ee814b4c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f1ed8002f70 tx=0x7f1ed8006eb0).ready entity=mon.1 client_cookie=fea9df968a88be02 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeceea700 1 -- 172.21.15.3:0/1865285366 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed800d570 con 0x7f1ee80b01a0 2022-07-21T01:35:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeceea700 1 -- 172.21.15.3:0/1865285366 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed8000e90 con 0x7f1ee80b01a0 2022-07-21T01:35:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeceea700 1 -- 172.21.15.3:0/1865285366 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed800d8c0 con 0x7f1ee80b01a0 2022-07-21T01:35:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 -- 172.21.15.3:0/1865285366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b01a0 msgr2=0x7f1ee814b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 --2- 172.21.15.3:0/1865285366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b01a0 0x7f1ee814b4c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f1ed8002f70 tx=0x7f1ed8006eb0).stop 2022-07-21T01:35:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 -- 172.21.15.3:0/1865285366 shutdown_connections 2022-07-21T01:35:19.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 --2- 172.21.15.3:0/1865285366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b01a0 0x7f1ee814b4c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 --2- 172.21.15.3:0/1865285366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b2710 0x7f1ee814afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 -- 172.21.15.3:0/1865285366 >> 172.21.15.3:0/1865285366 conn(0x7f1ee8060310 msgr2=0x7f1ee80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:19.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 -- 172.21.15.3:0/1865285366 shutdown_connections 2022-07-21T01:35:19.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 -- 172.21.15.3:0/1865285366 wait complete. 2022-07-21T01:35:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 Processor -- start 2022-07-21T01:35:19.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.149+0000 7f1eeeeee700 1 -- start start 2022-07-21T01:35:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eeeeee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80afd60 0x7f1ee8144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eeeeee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b01a0 0x7f1ee8156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eeeeee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee80b2710 0x7f1ee8156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eeeeee700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee814cda0 con 0x7f1ee80b2710 2022-07-21T01:35:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eeeeee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee814c660 con 0x7f1ee80b01a0 2022-07-21T01:35:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eeeeee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee8142880 con 0x7f1ee80afd60 2022-07-21T01:35:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eee6ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee80b2710 0x7f1ee8156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eee6ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee80b2710 0x7f1ee8156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44700/0 (socket says 172.21.15.3:44700) 2022-07-21T01:35:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eee6ed700 1 -- 172.21.15.3:0/3592467902 learned_addr learned my addr 172.21.15.3:0/3592467902 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eedeec700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80afd60 0x7f1ee8144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eed6eb700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b01a0 0x7f1ee8156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eee6ed700 1 -- 172.21.15.3:0/3592467902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80afd60 msgr2=0x7f1ee8144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eee6ed700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80afd60 0x7f1ee8144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eee6ed700 1 -- 172.21.15.3:0/3592467902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b01a0 msgr2=0x7f1ee8156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eee6ed700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b01a0 0x7f1ee8156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eee6ed700 1 -- 172.21.15.3:0/3592467902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed0009470 con 0x7f1ee80b2710 2022-07-21T01:35:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eee6ed700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee80b2710 0x7f1ee8156a00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f1ee0000ed0 tx=0x7f1ee0007a30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1ee6ffd700 1 -- 172.21.15.3:0/3592467902 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ee00070d0 con 0x7f1ee80b2710 2022-07-21T01:35:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1ee6ffd700 1 -- 172.21.15.3:0/3592467902 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee0007230 con 0x7f1ee80b2710 2022-07-21T01:35:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed800b1e0 con 0x7f1ee80b2710 2022-07-21T01:35:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1ee6ffd700 1 -- 172.21.15.3:0/3592467902 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ee0005740 con 0x7f1ee80b2710 2022-07-21T01:35:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee80a9da0 con 0x7f1ee80b2710 2022-07-21T01:35:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.153+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee80a9f60 con 0x7f1ee80b2710 2022-07-21T01:35:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.157+0000 7f1ee6ffd700 1 -- 172.21.15.3:0/3592467902 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ee0010070 con 0x7f1ee80b2710 2022-07-21T01:35:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.157+0000 7f1ee6ffd700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ecc053a70 0x7f1ecc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.157+0000 7f1ee6ffd700 1 -- 172.21.15.3:0/3592467902 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1ee006b780 con 0x7f1ee80b2710 2022-07-21T01:35:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.157+0000 7f1eedeec700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ecc053a70 0x7f1ecc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.157+0000 7f1eedeec700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ecc053a70 0x7f1ecc055f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1ed000b000 tx=0x7f1ed00091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.161+0000 7f1ee6ffd700 1 -- 172.21.15.3:0/3592467902 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ee0042430 con 0x7f1ee80b2710 2022-07-21T01:35:19.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.289+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ee80574e0 con 0x7f1ee80b2710 2022-07-21T01:35:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.289+0000 7f1ee6ffd700 1 -- 172.21.15.3:0/3592467902 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16297 (secure 0 0 0) 0x7f1ee0045a50 con 0x7f1ee80b2710 2022-07-21T01:35:19.292 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:19.292 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":46,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:35:18.753222+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:35:17.190459+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-21T01:35:18.336558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:35:19.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ecc053a70 msgr2=0x7f1ecc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ecc053a70 0x7f1ecc055f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1ed000b000 tx=0x7f1ed00091d0).stop 2022-07-21T01:35:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee80b2710 msgr2=0x7f1ee8156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee80b2710 0x7f1ee8156a00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f1ee0000ed0 tx=0x7f1ee0007a30).stop 2022-07-21T01:35:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 shutdown_connections 2022-07-21T01:35:19.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee80b2710 0x7f1ee8156a00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ecc053a70 0x7f1ecc055f30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b01a0 0x7f1ee8156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 --2- 172.21.15.3:0/3592467902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80afd60 0x7f1ee8144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 >> 172.21.15.3:0/3592467902 conn(0x7f1ee8060310 msgr2=0x7f1ee8144c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:19.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 shutdown_connections 2022-07-21T01:35:19.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.293+0000 7f1eeeeee700 1 -- 172.21.15.3:0/3592467902 wait complete. 2022-07-21T01:35:19.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:35:19.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.409+0000 7fe012b3c700 1 Processor -- start 2022-07-21T01:35:19.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.409+0000 7fe012b3c700 1 -- start start 2022-07-21T01:35:19.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe00c05fd90 0x7fe00c0b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c0b66f0 0x7fe00c0ba130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c1440f0 con 0x7fe00c0b5b20 2022-07-21T01:35:19.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c060170 con 0x7fe00c05fd90 2022-07-21T01:35:19.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c143be0 con 0x7fe00c0b66f0 2022-07-21T01:35:19.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe0108d8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe00c05fd90 0x7fe00c0b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe0108d8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe00c05fd90 0x7fe00c0b4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44708/0 (socket says 172.21.15.3:44708) 2022-07-21T01:35:19.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe0108d8700 1 -- 172.21.15.3:0/1879778791 learned_addr learned my addr 172.21.15.3:0/1879778791 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:19.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe00bfff700 1 --2- 172.21.15.3:0/1879778791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c0b66f0 0x7fe00c0ba130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe0108d8700 1 -- 172.21.15.3:0/1879778791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c0b66f0 msgr2=0x7fe00c0ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:19.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe0108d8700 1 --2- 172.21.15.3:0/1879778791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c0b66f0 0x7fe00c0ba130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe0108d8700 1 -- 172.21.15.3:0/1879778791 >> v1:172.21.15.143:6789/0 conn(0x7fe00c0b5b20 legacy=0x7fe00c0b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe0108d8700 1 -- 172.21.15.3:0/1879778791 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c143e50 con 0x7fe00c05fd90 2022-07-21T01:35:19.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe0108d8700 1 --2- 172.21.15.3:0/1879778791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c05fd90 0x7fe00c0b4340 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fdffc002f70 tx=0x7fdffc006eb0).ready entity=mon.0 client_cookie=ca1163bf4f14f1f8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:19.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe00b7fe700 1 -- 172.21.15.3:0/1879778791 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdffc00d570 con 0x7fe00c05fd90 2022-07-21T01:35:19.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe00b7fe700 1 -- 172.21.15.3:0/1879778791 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdffc000e90 con 0x7fe00c05fd90 2022-07-21T01:35:19.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe00b7fe700 1 -- 172.21.15.3:0/1879778791 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdffc00d900 con 0x7fe00c05fd90 2022-07-21T01:35:19.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 -- 172.21.15.3:0/1879778791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c05fd90 msgr2=0x7fe00c0b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 --2- 172.21.15.3:0/1879778791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c05fd90 0x7fe00c0b4340 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fdffc002f70 tx=0x7fdffc006eb0).stop 2022-07-21T01:35:19.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 -- 172.21.15.3:0/1879778791 shutdown_connections 2022-07-21T01:35:19.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 --2- 172.21.15.3:0/1879778791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c0b66f0 0x7fe00c0ba130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 --2- 172.21.15.3:0/1879778791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c05fd90 0x7fe00c0b4340 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 -- 172.21.15.3:0/1879778791 >> 172.21.15.3:0/1879778791 conn(0x7fe00c062530 msgr2=0x7fe00c066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:19.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 -- 172.21.15.3:0/1879778791 shutdown_connections 2022-07-21T01:35:19.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.413+0000 7fe012b3c700 1 -- 172.21.15.3:0/1879778791 wait complete. 2022-07-21T01:35:19.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.417+0000 7fe012b3c700 1 Processor -- start 2022-07-21T01:35:19.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.417+0000 7fe012b3c700 1 -- start start 2022-07-21T01:35:19.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.417+0000 7fe012b3c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c05fd90 0x7fe00c141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.417+0000 7fe012b3c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c0b5b20 0x7fe00c152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.417+0000 7fe012b3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe00c0b66f0 0x7fe00c130a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.417+0000 7fe012b3c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c12efb0 con 0x7fe00c0b5b20 2022-07-21T01:35:19.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.417+0000 7fe012b3c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c1566e0 con 0x7fe00c0b66f0 2022-07-21T01:35:19.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.417+0000 7fe012b3c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c13fd10 con 0x7fe00c05fd90 2022-07-21T01:35:19.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe00bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c0b5b20 0x7fe00c152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe00bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c0b5b20 0x7fe00c152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44714/0 (socket says 172.21.15.3:44714) 2022-07-21T01:35:19.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe00bfff700 1 -- 172.21.15.3:0/3784143354 learned_addr learned my addr 172.21.15.3:0/3784143354 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:19.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe0108d8700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c05fd90 0x7fe00c141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe0110d9700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe00c0b66f0 0x7fe00c130a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe00bfff700 1 -- 172.21.15.3:0/3784143354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c05fd90 msgr2=0x7fe00c141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe00bfff700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c05fd90 0x7fe00c141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe00bfff700 1 -- 172.21.15.3:0/3784143354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe00c0b66f0 msgr2=0x7fe00c130a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe00bfff700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe00c0b66f0 0x7fe00c130a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe00bfff700 1 -- 172.21.15.3:0/3784143354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc00b1e0 con 0x7fe00c0b5b20 2022-07-21T01:35:19.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe00bfff700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c0b5b20 0x7fe00c152660 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fdff400c000 tx=0x7fdff4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:19.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe009ffb700 1 -- 172.21.15.3:0/3784143354 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdff4010780 con 0x7fe00c0b5b20 2022-07-21T01:35:19.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe009ffb700 1 -- 172.21.15.3:0/3784143354 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff40108e0 con 0x7fe00c0b5b20 2022-07-21T01:35:19.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00c156910 con 0x7fe00c0b5b20 2022-07-21T01:35:19.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe009ffb700 1 -- 172.21.15.3:0/3784143354 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdff4010c40 con 0x7fe00c0b5b20 2022-07-21T01:35:19.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c12d450 con 0x7fe00c0b5b20 2022-07-21T01:35:19.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe009ffb700 1 -- 172.21.15.3:0/3784143354 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdff401c460 con 0x7fe00c0b5b20 2022-07-21T01:35:19.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe009ffb700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdff00539e0 0x7fdff0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe009ffb700 1 -- 172.21.15.3:0/3784143354 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdff406c3a0 con 0x7fe00c0b5b20 2022-07-21T01:35:19.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe0108d8700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdff00539e0 0x7fdff0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.421+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c156aa0 con 0x7fe00c0b5b20 2022-07-21T01:35:19.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.425+0000 7fe0108d8700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdff00539e0 0x7fdff0055ea0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdffc01c000 tx=0x7fdffc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:19.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.433+0000 7fe009ffb700 1 -- 172.21.15.3:0/3784143354 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdff4043100 con 0x7fe00c0b5b20 2022-07-21T01:35:19.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.597+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe00c0b99f0 con 0x7fe00c0b5b20 2022-07-21T01:35:19.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.769+0000 7fe009ffb700 1 -- 172.21.15.3:0/3784143354 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdff4046720 con 0x7fe00c0b5b20 2022-07-21T01:35:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdff00539e0 msgr2=0x7fdff0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdff00539e0 0x7fdff0055ea0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdffc01c000 tx=0x7fdffc015000).stop 2022-07-21T01:35:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c0b5b20 msgr2=0x7fe00c152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c0b5b20 0x7fe00c152660 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fdff400c000 tx=0x7fdff4006df0).stop 2022-07-21T01:35:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 shutdown_connections 2022-07-21T01:35:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdff00539e0 0x7fdff0055ea0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe00c0b66f0 0x7fe00c130a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe00c0b5b20 0x7fe00c152660 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 --2- 172.21.15.3:0/3784143354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe00c05fd90 0x7fe00c141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 >> 172.21.15.3:0/3784143354 conn(0x7fe00c062530 msgr2=0x7fe00c0b8210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:19.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 shutdown_connections 2022-07-21T01:35:19.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.773+0000 7fe012b3c700 1 -- 172.21.15.3:0/3784143354 wait complete. 2022-07-21T01:35:19.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:19.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d962cc700 1 Processor -- start 2022-07-21T01:35:19.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d962cc700 1 -- start start 2022-07-21T01:35:19.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d962cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b4550 0x7f5d900b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d962cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5d900b6af0 0x7f5d9012bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d962cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d90144240 con 0x7f5d900b6710 2022-07-21T01:35:19.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d962cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d90143d00 con 0x7f5d900b6af0 2022-07-21T01:35:19.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d962cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d90143fa0 con 0x7f5d900b4550 2022-07-21T01:35:19.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d94869700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5d900b6af0 0x7f5d9012bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d8f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b4550 0x7f5d900b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d8f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b4550 0x7f5d900b5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:51638/0 (socket says 172.21.15.3:51638) 2022-07-21T01:35:19.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.889+0000 7f5d8f7fe700 1 -- 172.21.15.3:0/376339406 learned_addr learned my addr 172.21.15.3:0/376339406 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:19.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d8f7fe700 1 -- 172.21.15.3:0/376339406 >> v2:172.21.15.143:3300/0 conn(0x7f5d900b6af0 msgr2=0x7f5d9012bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:19.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d8f7fe700 1 --2- 172.21.15.3:0/376339406 >> v2:172.21.15.143:3300/0 conn(0x7f5d900b6af0 0x7f5d9012bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d8f7fe700 1 -- 172.21.15.3:0/376339406 >> v1:172.21.15.143:6789/0 conn(0x7f5d900b6710 legacy=0x7f5d90061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d8f7fe700 1 -- 172.21.15.3:0/376339406 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d90061780 con 0x7f5d900b4550 2022-07-21T01:35:19.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d8f7fe700 1 --2- 172.21.15.3:0/376339406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b4550 0x7f5d900b5d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f5d78002f70 tx=0x7f5d78006eb0).ready entity=mon.2 client_cookie=feda46715d2848dc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:19.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d8effd700 1 -- 172.21.15.3:0/376339406 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d7800d570 con 0x7f5d900b4550 2022-07-21T01:35:19.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d8effd700 1 -- 172.21.15.3:0/376339406 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78000e90 con 0x7f5d900b4550 2022-07-21T01:35:19.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d8effd700 1 -- 172.21.15.3:0/376339406 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d7800d900 con 0x7f5d900b4550 2022-07-21T01:35:19.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 -- 172.21.15.3:0/376339406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b4550 msgr2=0x7f5d900b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 --2- 172.21.15.3:0/376339406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b4550 0x7f5d900b5d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f5d78002f70 tx=0x7f5d78006eb0).stop 2022-07-21T01:35:19.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 -- 172.21.15.3:0/376339406 shutdown_connections 2022-07-21T01:35:19.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 --2- 172.21.15.3:0/376339406 >> v2:172.21.15.143:3300/0 conn(0x7f5d900b6af0 0x7f5d9012bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 --2- 172.21.15.3:0/376339406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b4550 0x7f5d900b5d80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 -- 172.21.15.3:0/376339406 >> 172.21.15.3:0/376339406 conn(0x7f5d9005c620 msgr2=0x7f5d9005ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:19.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 -- 172.21.15.3:0/376339406 shutdown_connections 2022-07-21T01:35:19.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 -- 172.21.15.3:0/376339406 wait complete. 2022-07-21T01:35:19.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 Processor -- start 2022-07-21T01:35:19.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.893+0000 7f5d962cc700 1 -- start start 2022-07-21T01:35:19.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d962cc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d900b4550 0x7f5d900b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d962cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d900b6710 0x7f5d90156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d962cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b6af0 0x7f5d90129910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d962cc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9012b200 con 0x7f5d900b4550 2022-07-21T01:35:19.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d962cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d900ba660 con 0x7f5d900b6710 2022-07-21T01:35:19.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d962cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d90155c60 con 0x7f5d900b6af0 2022-07-21T01:35:19.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d900b4550 0x7f5d900b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d900b6710 0x7f5d90156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d94869700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b6af0 0x7f5d90129910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d900b4550 0x7f5d900b2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44728/0 (socket says 172.21.15.3:44728) 2022-07-21T01:35:19.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8ffff700 1 -- 172.21.15.3:0/3894802790 learned_addr learned my addr 172.21.15.3:0/3894802790 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:19.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8ffff700 1 -- 172.21.15.3:0/3894802790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b6af0 msgr2=0x7f5d90129910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8ffff700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b6af0 0x7f5d90129910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8ffff700 1 -- 172.21.15.3:0/3894802790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d900b6710 msgr2=0x7f5d90156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:19.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8ffff700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d900b6710 0x7f5d90156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:19.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8ffff700 1 -- 172.21.15.3:0/3894802790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7800b1e0 con 0x7f5d900b4550 2022-07-21T01:35:19.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8ffff700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d900b4550 0x7f5d900b2900 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5d90061100 tx=0x7f5d80007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:19.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8d7fa700 1 -- 172.21.15.3:0/3894802790 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d80012400 con 0x7f5d900b4550 2022-07-21T01:35:19.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8d7fa700 1 -- 172.21.15.3:0/3894802790 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d80012560 con 0x7f5d900b4550 2022-07-21T01:35:19.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d90155f00 con 0x7f5d900b4550 2022-07-21T01:35:19.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d8d7fa700 1 -- 172.21.15.3:0/3894802790 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d800128c0 con 0x7f5d900b4550 2022-07-21T01:35:19.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.897+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d90142900 con 0x7f5d900b4550 2022-07-21T01:35:19.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.901+0000 7f5d8d7fa700 1 -- 172.21.15.3:0/3894802790 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5d80019030 con 0x7f5d900b4550 2022-07-21T01:35:19.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.901+0000 7f5d8d7fa700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5d74053aa0 0x7f5d74055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:19.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.901+0000 7f5d8d7fa700 1 -- 172.21.15.3:0/3894802790 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5d8006b350 con 0x7f5d900b4550 2022-07-21T01:35:19.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.901+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d90129fa0 con 0x7f5d900b4550 2022-07-21T01:35:19.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.901+0000 7f5d8f7fe700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5d74053aa0 0x7f5d74055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:19.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.901+0000 7f5d8f7fe700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5d74053aa0 0x7f5d74055f60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f5d7801c000 tx=0x7f5d78015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:19.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:19.905+0000 7f5d8d7fa700 1 -- 172.21.15.3:0/3894802790 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d800420b0 con 0x7f5d900b4550 2022-07-21T01:35:20.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.069+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d90061a20 con 0x7f5d900b4550 2022-07-21T01:35:20.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d8d7fa700 1 -- 172.21.15.3:0/3894802790 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2488 (secure 0 0 0) 0x7f5d800456d0 con 0x7f5d900b4550 2022-07-21T01:35:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:20.075 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:35:18.753732+0000","modified":"2022-07-21T01:35:19.758646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/2892507714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2892507714},{"type":"v1","addr":"172.21.15.3:6837","nonce":2892507714}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-21T01:35:20.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5d74053aa0 msgr2=0x7f5d74055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5d74053aa0 0x7f5d74055f60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f5d7801c000 tx=0x7f5d78015000).stop 2022-07-21T01:35:20.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d900b4550 msgr2=0x7f5d900b2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d900b4550 0x7f5d900b2900 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5d90061100 tx=0x7f5d80007c50).stop 2022-07-21T01:35:20.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 shutdown_connections 2022-07-21T01:35:20.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d900b6af0 0x7f5d90129910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5d74053aa0 0x7f5d74055f60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d900b6710 0x7f5d90156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 --2- 172.21.15.3:0/3894802790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d900b4550 0x7f5d900b2900 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 >> 172.21.15.3:0/3894802790 conn(0x7f5d9005c620 msgr2=0x7f5d90128510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:20.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 shutdown_connections 2022-07-21T01:35:20.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.073+0000 7f5d962cc700 1 -- 172.21.15.3:0/3894802790 wait complete. 2022-07-21T01:35:20.083 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 18 2022-07-21T01:35:20.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:35:20.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516ad6d700 1 Processor -- start 2022-07-21T01:35:20.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516ad6d700 1 -- start start 2022-07-21T01:35:20.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516ad6d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640b08a0 0x7f5164142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516ad6d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f51640b3260 0x7f51641433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516ad6d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f51640ad000 0x7f516405f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516ad6d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51640b0c80 con 0x7f51640b3260 2022-07-21T01:35:20.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516ad6d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51640ae810 con 0x7f51640ad000 2022-07-21T01:35:20.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516ad6d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51640aeab0 con 0x7f51640b08a0 2022-07-21T01:35:20.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516956a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f51640b3260 0x7f51641433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516956a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f51640b3260 0x7f51641433a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44736/0 (socket says 172.21.15.3:44736) 2022-07-21T01:35:20.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516956a700 1 -- 172.21.15.3:0/402941627 learned_addr learned my addr 172.21.15.3:0/402941627 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:20.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f5169d6b700 1 --2- 172.21.15.3:0/402941627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640b08a0 0x7f5164142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516a56c700 1 --2- 172.21.15.3:0/402941627 >> v2:172.21.15.143:3300/0 conn(0x7f51640ad000 0x7f516405f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516956a700 1 -- 172.21.15.3:0/402941627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640b08a0 msgr2=0x7f5164142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:20.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516956a700 1 --2- 172.21.15.3:0/402941627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640b08a0 0x7f5164142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516956a700 1 -- 172.21.15.3:0/402941627 >> v2:172.21.15.143:3300/0 conn(0x7f51640ad000 msgr2=0x7f516405f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:20.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516956a700 1 --2- 172.21.15.3:0/402941627 >> v2:172.21.15.143:3300/0 conn(0x7f51640ad000 0x7f516405f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516956a700 1 -- 172.21.15.3:0/402941627 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51640aed50 con 0x7f51640b3260 2022-07-21T01:35:20.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516a56c700 1 --2- 172.21.15.3:0/402941627 >> v2:172.21.15.143:3300/0 conn(0x7f51640ad000 0x7f516405f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:20.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f516956a700 1 --2- 172.21.15.3:0/402941627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 0x7f51641433a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f5158002f70 tx=0x7f5158006eb0).ready entity=mon.0 client_cookie=47e3f09cec0534b9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:20.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f5168d69700 1 -- 172.21.15.3:0/402941627 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f515800f970 con 0x7f51640b3260 2022-07-21T01:35:20.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f5168d69700 1 -- 172.21.15.3:0/402941627 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5158000e90 con 0x7f51640b3260 2022-07-21T01:35:20.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.185+0000 7f5168d69700 1 -- 172.21.15.3:0/402941627 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f515800fd00 con 0x7f51640b3260 2022-07-21T01:35:20.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 -- 172.21.15.3:0/402941627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 msgr2=0x7f51641433a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 --2- 172.21.15.3:0/402941627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 0x7f51641433a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f5158002f70 tx=0x7f5158006eb0).stop 2022-07-21T01:35:20.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 -- 172.21.15.3:0/402941627 shutdown_connections 2022-07-21T01:35:20.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 --2- 172.21.15.3:0/402941627 >> v2:172.21.15.143:3300/0 conn(0x7f51640ad000 0x7f516405f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 --2- 172.21.15.3:0/402941627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 0x7f51641433a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 --2- 172.21.15.3:0/402941627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640b08a0 0x7f5164142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 -- 172.21.15.3:0/402941627 >> 172.21.15.3:0/402941627 conn(0x7f51640612d0 msgr2=0x7f5164062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:20.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 -- 172.21.15.3:0/402941627 shutdown_connections 2022-07-21T01:35:20.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 -- 172.21.15.3:0/402941627 wait complete. 2022-07-21T01:35:20.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 Processor -- start 2022-07-21T01:35:20.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.189+0000 7f516ad6d700 1 -- start start 2022-07-21T01:35:20.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516ad6d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640ad000 0x7f5164067500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516ad6d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51640b08a0 0x7f5164065560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516ad6d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 0x7f5164065a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516ad6d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5164065f40 con 0x7f51640b3260 2022-07-21T01:35:20.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516ad6d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5164144d70 con 0x7f51640b08a0 2022-07-21T01:35:20.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516ad6d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5164144040 con 0x7f51640ad000 2022-07-21T01:35:20.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516a56c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 0x7f5164065a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516a56c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 0x7f5164065a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44744/0 (socket says 172.21.15.3:44744) 2022-07-21T01:35:20.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516a56c700 1 -- 172.21.15.3:0/155170111 learned_addr learned my addr 172.21.15.3:0/155170111 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:20.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516956a700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51640b08a0 0x7f5164065560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f5169d6b700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640ad000 0x7f5164067500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516a56c700 1 -- 172.21.15.3:0/155170111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640ad000 msgr2=0x7f5164067500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516a56c700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640ad000 0x7f5164067500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516a56c700 1 -- 172.21.15.3:0/155170111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51640b08a0 msgr2=0x7f5164065560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516a56c700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51640b08a0 0x7f5164065560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516a56c700 1 -- 172.21.15.3:0/155170111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515800b1e0 con 0x7f51640b3260 2022-07-21T01:35:20.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516a56c700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 0x7f5164065a50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f515c00a000 tx=0x7f515c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:20.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f5162ffd700 1 -- 172.21.15.3:0/155170111 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f515c011780 con 0x7f51640b3260 2022-07-21T01:35:20.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f5162ffd700 1 -- 172.21.15.3:0/155170111 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c0118e0 con 0x7f51640b3260 2022-07-21T01:35:20.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5164144fd0 con 0x7f51640b3260 2022-07-21T01:35:20.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f5162ffd700 1 -- 172.21.15.3:0/155170111 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f515c011c40 con 0x7f51640b3260 2022-07-21T01:35:20.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5164144740 con 0x7f51640b3260 2022-07-21T01:35:20.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5164066a50 con 0x7f51640b3260 2022-07-21T01:35:20.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.193+0000 7f5162ffd700 1 -- 172.21.15.3:0/155170111 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f515c01c460 con 0x7f51640b3260 2022-07-21T01:35:20.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.197+0000 7f5162ffd700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5144053950 0x7f5144055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.197+0000 7f5162ffd700 1 -- 172.21.15.3:0/155170111 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f515c06c400 con 0x7f51640b3260 2022-07-21T01:35:20.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.197+0000 7f5169d6b700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5144053950 0x7f5144055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.197+0000 7f5169d6b700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5144053950 0x7f5144055e10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f515400b000 tx=0x7f51540091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:20.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.201+0000 7f5162ffd700 1 -- 172.21.15.3:0/155170111 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f515c043160 con 0x7f51640b3260 2022-07-21T01:35:20.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.325+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51640574e0 con 0x7f51640b3260 2022-07-21T01:35:20.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.325+0000 7f5162ffd700 1 -- 172.21.15.3:0/155170111 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16284 (secure 0 0 0) 0x7f515c046780 con 0x7f51640b3260 2022-07-21T01:35:20.327 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:20.327 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":47,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:35:19.757419+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:35:17.190459+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-21T01:35:18.336558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:35:20.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5144053950 msgr2=0x7f5144055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5144053950 0x7f5144055e10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f515400b000 tx=0x7f51540091d0).stop 2022-07-21T01:35:20.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 msgr2=0x7f5164065a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 0x7f5164065a50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f515c00a000 tx=0x7f515c008040).stop 2022-07-21T01:35:20.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 shutdown_connections 2022-07-21T01:35:20.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51640b3260 0x7f5164065a50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5144053950 0x7f5144055e10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51640b08a0 0x7f5164065560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 --2- 172.21.15.3:0/155170111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51640ad000 0x7f5164067500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 >> 172.21.15.3:0/155170111 conn(0x7f51640612d0 msgr2=0x7f5164142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 shutdown_connections 2022-07-21T01:35:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.329+0000 7f516ad6d700 1 -- 172.21.15.3:0/155170111 wait complete. 2022-07-21T01:35:20.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:35:20.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70f4c71700 1 Processor -- start 2022-07-21T01:35:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70f4c71700 1 -- start start 2022-07-21T01:35:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70f4c71700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70f00b1f40 0x7f70f00b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70f4c71700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f70f00a7250 0x7f70f01448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70f4c71700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f00b04e0 con 0x7f70f0148530 2022-07-21T01:35:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70f4c71700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f00a9250 con 0x7f70f00a7250 2022-07-21T01:35:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70f4c71700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f00b0780 con 0x7f70f00b1f40 2022-07-21T01:35:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70eed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f70f00a7250 0x7f70f01448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70edd9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f70f0148530 0x7f70f00b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53030/0 (socket says 172.21.15.3:53030) 2022-07-21T01:35:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70eed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f70f00a7250 0x7f70f01448b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44752/0 (socket says 172.21.15.3:44752) 2022-07-21T01:35:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70edd9b700 1 -- 172.21.15.3:0/2592012116 learned_addr learned my addr 172.21.15.3:0/2592012116 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70ee59c700 1 --2- 172.21.15.3:0/2592012116 >> v2:172.21.15.143:3300/0 conn(0x7f70f00b1f40 0x7f70f00b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70eed9d700 1 -- 172.21.15.3:0/2592012116 >> v2:172.21.15.143:3300/0 conn(0x7f70f00b1f40 msgr2=0x7f70f00b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70eed9d700 1 --2- 172.21.15.3:0/2592012116 >> v2:172.21.15.143:3300/0 conn(0x7f70f00b1f40 0x7f70f00b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70eed9d700 1 -- 172.21.15.3:0/2592012116 >> v1:172.21.15.3:6789/0 conn(0x7f70f0148530 legacy=0x7f70f00b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70eed9d700 1 -- 172.21.15.3:0/2592012116 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f00b0a20 con 0x7f70f00a7250 2022-07-21T01:35:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70eed9d700 1 --2- 172.21.15.3:0/2592012116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00a7250 0x7f70f01448b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f70f01478c0 tx=0x7f70e80091d0).ready entity=mon.0 client_cookie=8f397c1ad1220f43 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70ed59a700 1 -- 172.21.15.3:0/2592012116 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e800b680 con 0x7f70f00a7250 2022-07-21T01:35:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70ed59a700 1 -- 172.21.15.3:0/2592012116 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e800b7e0 con 0x7f70f00a7250 2022-07-21T01:35:20.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.453+0000 7f70ed59a700 1 -- 172.21.15.3:0/2592012116 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e800b950 con 0x7f70f00a7250 2022-07-21T01:35:20.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 -- 172.21.15.3:0/2592012116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00a7250 msgr2=0x7f70f01448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 --2- 172.21.15.3:0/2592012116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00a7250 0x7f70f01448b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f70f01478c0 tx=0x7f70e80091d0).stop 2022-07-21T01:35:20.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 -- 172.21.15.3:0/2592012116 shutdown_connections 2022-07-21T01:35:20.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 --2- 172.21.15.3:0/2592012116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00a7250 0x7f70f01448b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 --2- 172.21.15.3:0/2592012116 >> v2:172.21.15.143:3300/0 conn(0x7f70f00b1f40 0x7f70f00b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 -- 172.21.15.3:0/2592012116 >> 172.21.15.3:0/2592012116 conn(0x7f70f005f840 msgr2=0x7f70f005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:20.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 -- 172.21.15.3:0/2592012116 shutdown_connections 2022-07-21T01:35:20.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 -- 172.21.15.3:0/2592012116 wait complete. 2022-07-21T01:35:20.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 Processor -- start 2022-07-21T01:35:20.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 -- start start 2022-07-21T01:35:20.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70f00a7250 0x7f70f00aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00b1f40 0x7f70f00adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70f0148530 0x7f70f00aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f01499a0 con 0x7f70f00b1f40 2022-07-21T01:35:20.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f00aa290 con 0x7f70f00a7250 2022-07-21T01:35:20.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70f4c71700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f00aa480 con 0x7f70f0148530 2022-07-21T01:35:20.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70edd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00b1f40 0x7f70f00adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70edd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00b1f40 0x7f70f00adfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44756/0 (socket says 172.21.15.3:44756) 2022-07-21T01:35:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70edd9b700 1 -- 172.21.15.3:0/170476381 learned_addr learned my addr 172.21.15.3:0/170476381 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70eed9d700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70f0148530 0x7f70f00aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.457+0000 7f70ee59c700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70f00a7250 0x7f70f00aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70edd9b700 1 -- 172.21.15.3:0/170476381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70f0148530 msgr2=0x7f70f00aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70edd9b700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70f0148530 0x7f70f00aba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70edd9b700 1 -- 172.21.15.3:0/170476381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70f00a7250 msgr2=0x7f70f00aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70edd9b700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70f00a7250 0x7f70f00aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70edd9b700 1 -- 172.21.15.3:0/170476381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e8009470 con 0x7f70f00b1f40 2022-07-21T01:35:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70edd9b700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00b1f40 0x7f70f00adfc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f70f00acff0 tx=0x7f70e4007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70df7fe700 1 -- 172.21.15.3:0/170476381 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e4001d70 con 0x7f70f00b1f40 2022-07-21T01:35:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70df7fe700 1 -- 172.21.15.3:0/170476381 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e4007110 con 0x7f70f00b1f40 2022-07-21T01:35:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f01577d0 con 0x7f70f00b1f40 2022-07-21T01:35:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70df7fe700 1 -- 172.21.15.3:0/170476381 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e4007470 con 0x7f70f00b1f40 2022-07-21T01:35:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f0148cc0 con 0x7f70f00b1f40 2022-07-21T01:35:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f00b0cc0 con 0x7f70f00b1f40 2022-07-21T01:35:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70df7fe700 1 -- 172.21.15.3:0/170476381 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f70e401b070 con 0x7f70f00b1f40 2022-07-21T01:35:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70df7fe700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f70cc053a70 0x7f70cc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70df7fe700 1 -- 172.21.15.3:0/170476381 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f70e406b3b0 con 0x7f70f00b1f40 2022-07-21T01:35:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70ee59c700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f70cc053a70 0x7f70cc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.461+0000 7f70ee59c700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f70cc053a70 0x7f70cc055f30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f70e000b000 tx=0x7f70e0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.469+0000 7f70df7fe700 1 -- 172.21.15.3:0/170476381 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f70e4042170 con 0x7f70f00b1f40 2022-07-21T01:35:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.633+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f70f00aa640 con 0x7f70f00b1f40 2022-07-21T01:35:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70df7fe700 1 -- 172.21.15.3:0/170476381 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7f70e4045790 con 0x7f70f00b1f40 2022-07-21T01:35:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f70cc053a70 msgr2=0x7f70cc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f70cc053a70 0x7f70cc055f30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f70e000b000 tx=0x7f70e0006df0).stop 2022-07-21T01:35:20.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00b1f40 msgr2=0x7f70f00adfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00b1f40 0x7f70f00adfc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f70f00acff0 tx=0x7f70e4007c30).stop 2022-07-21T01:35:20.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 shutdown_connections 2022-07-21T01:35:20.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70f0148530 0x7f70f00aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f70f00b1f40 0x7f70f00adfc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f70cc053a70 0x7f70cc055f30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 --2- 172.21.15.3:0/170476381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70f00a7250 0x7f70f00aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 >> 172.21.15.3:0/170476381 conn(0x7f70f005f840 msgr2=0x7f70f00b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:20.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 shutdown_connections 2022-07-21T01:35:20.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.641+0000 7f70f4c71700 1 -- 172.21.15.3:0/170476381 wait complete. 2022-07-21T01:35:20.651 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:35:20.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:35:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f75a1212700 1 Processor -- start 2022-07-21T01:35:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f75a1212700 1 -- start start 2022-07-21T01:35:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f75a1212700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f759c0b6710 0x7f759c0b7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f75a1212700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f759c0b5cd0 0x7f759c1481d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f75a1212700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c0b6af0 con 0x7f759c0b0de0 2022-07-21T01:35:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f75a1212700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759c147640 con 0x7f759c0b5cd0 2022-07-21T01:35:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f75a1212700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759c0b6250 con 0x7f759c0b6710 2022-07-21T01:35:20.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f759c0b5cd0 0x7f759c1481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f759c0b5cd0 0x7f759c1481d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44762/0 (socket says 172.21.15.3:44762) 2022-07-21T01:35:20.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759a59c700 1 -- 172.21.15.3:0/1437693794 learned_addr learned my addr 172.21.15.3:0/1437693794 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:20.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759ad9d700 1 --2- 172.21.15.3:0/1437693794 >> v2:172.21.15.143:3300/0 conn(0x7f759c0b6710 0x7f759c0b7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759a59c700 1 -- 172.21.15.3:0/1437693794 >> v2:172.21.15.143:3300/0 conn(0x7f759c0b6710 msgr2=0x7f759c0b7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759a59c700 1 --2- 172.21.15.3:0/1437693794 >> v2:172.21.15.143:3300/0 conn(0x7f759c0b6710 0x7f759c0b7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759a59c700 1 -- 172.21.15.3:0/1437693794 >> v1:172.21.15.143:6789/0 conn(0x7f759c0b0de0 legacy=0x7f759c144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759a59c700 1 -- 172.21.15.3:0/1437693794 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c147830 con 0x7f759c0b5cd0 2022-07-21T01:35:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759ad9d700 1 --2- 172.21.15.3:0/1437693794 >> v2:172.21.15.143:3300/0 conn(0x7f759c0b6710 0x7f759c0b7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.769+0000 7f759a59c700 1 --2- 172.21.15.3:0/1437693794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 0x7f759c1481d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f759c0b6390 tx=0x7f7590006eb0).ready entity=mon.0 client_cookie=61365cca6b33dc59 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f7599d9b700 1 -- 172.21.15.3:0/1437693794 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f759000d610 con 0x7f759c0b5cd0 2022-07-21T01:35:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f7599d9b700 1 -- 172.21.15.3:0/1437693794 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590000e90 con 0x7f759c0b5cd0 2022-07-21T01:35:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f7599d9b700 1 -- 172.21.15.3:0/1437693794 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f759000d960 con 0x7f759c0b5cd0 2022-07-21T01:35:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 -- 172.21.15.3:0/1437693794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 msgr2=0x7f759c1481d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 --2- 172.21.15.3:0/1437693794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 0x7f759c1481d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f759c0b6390 tx=0x7f7590006eb0).stop 2022-07-21T01:35:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 -- 172.21.15.3:0/1437693794 shutdown_connections 2022-07-21T01:35:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 --2- 172.21.15.3:0/1437693794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 0x7f759c1481d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 --2- 172.21.15.3:0/1437693794 >> v2:172.21.15.143:3300/0 conn(0x7f759c0b6710 0x7f759c0b7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 -- 172.21.15.3:0/1437693794 >> 172.21.15.3:0/1437693794 conn(0x7f759c05c620 msgr2=0x7f759c05ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 -- 172.21.15.3:0/1437693794 shutdown_connections 2022-07-21T01:35:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 -- 172.21.15.3:0/1437693794 wait complete. 2022-07-21T01:35:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 Processor -- start 2022-07-21T01:35:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 -- start start 2022-07-21T01:35:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f759c0b0de0 0x7f759c156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 0x7f759c062cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f759c0b6710 0x7f759c14b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c0621b0 con 0x7f759c0b5cd0 2022-07-21T01:35:20.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c063f10 con 0x7f759c0b0de0 2022-07-21T01:35:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.777+0000 7f75a1212700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759c156d20 con 0x7f759c0b6710 2022-07-21T01:35:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 0x7f759c062cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 0x7f759c062cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44770/0 (socket says 172.21.15.3:44770) 2022-07-21T01:35:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759a59c700 1 -- 172.21.15.3:0/1931742719 learned_addr learned my addr 172.21.15.3:0/1931742719 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759ad9d700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f759c0b0de0 0x7f759c156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759b59e700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f759c0b6710 0x7f759c14b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759a59c700 1 -- 172.21.15.3:0/1931742719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f759c0b6710 msgr2=0x7f759c14b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759a59c700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f759c0b6710 0x7f759c14b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759a59c700 1 -- 172.21.15.3:0/1931742719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f759c0b0de0 msgr2=0x7f759c156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759a59c700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f759c0b0de0 0x7f759c156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759a59c700 1 -- 172.21.15.3:0/1931742719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759000b1e0 con 0x7f759c0b5cd0 2022-07-21T01:35:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759a59c700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 0x7f759c062cb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f759c0b6390 tx=0x7f7590015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f758bfff700 1 -- 172.21.15.3:0/1931742719 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7590021070 con 0x7f759c0b5cd0 2022-07-21T01:35:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f758bfff700 1 -- 172.21.15.3:0/1931742719 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590002f70 con 0x7f759c0b5cd0 2022-07-21T01:35:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c0625b0 con 0x7f759c0b5cd0 2022-07-21T01:35:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f758bfff700 1 -- 172.21.15.3:0/1931742719 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7590011d70 con 0x7f759c0b5cd0 2022-07-21T01:35:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c14a410 con 0x7f759c0b5cd0 2022-07-21T01:35:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759c0574e0 con 0x7f759c0b5cd0 2022-07-21T01:35:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f758bfff700 1 -- 172.21.15.3:0/1931742719 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f759000f070 con 0x7f759c0b5cd0 2022-07-21T01:35:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f758bfff700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7578053a70 0x7f7578055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f758bfff700 1 -- 172.21.15.3:0/1931742719 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f759006d3d0 con 0x7f759c0b5cd0 2022-07-21T01:35:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.781+0000 7f759ad9d700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7578053a70 0x7f7578055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.785+0000 7f759ad9d700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7578053a70 0x7f7578055f30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f758c00c000 tx=0x7f758c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.789+0000 7f758bfff700 1 -- 172.21.15.3:0/1931742719 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7590044130 con 0x7f759c0b5cd0 2022-07-21T01:35:20.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.949+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f759c066b60 con 0x7f759c0b5cd0 2022-07-21T01:35:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.957+0000 7f758bfff700 1 -- 172.21.15.3:0/1931742719 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7590047750 con 0x7f759c0b5cd0 2022-07-21T01:35:20.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7578053a70 msgr2=0x7f7578055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7578053a70 0x7f7578055f30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f758c00c000 tx=0x7f758c008040).stop 2022-07-21T01:35:20.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 msgr2=0x7f759c062cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 0x7f759c062cb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f759c0b6390 tx=0x7f7590015000).stop 2022-07-21T01:35:20.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 shutdown_connections 2022-07-21T01:35:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f759c0b6710 0x7f759c14b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f759c0b5cd0 0x7f759c062cb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7578053a70 0x7f7578055f30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 --2- 172.21.15.3:0/1931742719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f759c0b0de0 0x7f759c156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 >> 172.21.15.3:0/1931742719 conn(0x7f759c05c620 msgr2=0x7f759c143f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:20.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 shutdown_connections 2022-07-21T01:35:20.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:20.961+0000 7f75a1212700 1 -- 172.21.15.3:0/1931742719 wait complete. 2022-07-21T01:35:20.969 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:35:20.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:21.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 Processor -- start 2022-07-21T01:35:21.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 -- start start 2022-07-21T01:35:21.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0614b0 0x7f1a0c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1a0c0ad750 0x7f1a0c0b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0c0aee80 con 0x7f1a0c1405d0 2022-07-21T01:35:21.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a0c0b70b0 con 0x7f1a0c0ad750 2022-07-21T01:35:21.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c0af0b0 con 0x7f1a0c0614b0 2022-07-21T01:35:21.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a03fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1a0c0ad750 0x7f1a0c0b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a03fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1a0c0ad750 0x7f1a0c0b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44776/0 (socket says 172.21.15.3:44776) 2022-07-21T01:35:21.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a03fff700 1 -- 172.21.15.3:0/4087657066 learned_addr learned my addr 172.21.15.3:0/4087657066 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:21.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0affd700 1 --2- 172.21.15.3:0/4087657066 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0614b0 0x7f1a0c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a03fff700 1 -- 172.21.15.3:0/4087657066 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0614b0 msgr2=0x7f1a0c061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:21.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/4087657066 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782465820 0 0) 0x7f1a0c0aee80 con 0x7f1a0c1405d0 2022-07-21T01:35:21.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a03fff700 1 --2- 172.21.15.3:0/4087657066 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0614b0 0x7f1a0c061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a03fff700 1 -- 172.21.15.3:0/4087657066 >> v1:172.21.15.3:6789/0 conn(0x7f1a0c1405d0 legacy=0x7f1a0c1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a03fff700 1 -- 172.21.15.3:0/4087657066 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c0af350 con 0x7f1a0c0ad750 2022-07-21T01:35:21.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0affd700 1 --2- 172.21.15.3:0/4087657066 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0614b0 0x7f1a0c061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:35:21.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a03fff700 1 --2- 172.21.15.3:0/4087657066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c0ad750 0x7f1a0c0b9ff0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f19fc002f70 tx=0x7f19fc006eb0).ready entity=mon.0 client_cookie=34d576ce8180bcaf server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:21.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/4087657066 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19fc00d5c0 con 0x7f1a0c0ad750 2022-07-21T01:35:21.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/4087657066 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19fc000e90 con 0x7f1a0c0ad750 2022-07-21T01:35:21.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/4087657066 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19fc00d910 con 0x7f1a0c0ad750 2022-07-21T01:35:21.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 -- 172.21.15.3:0/4087657066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c0ad750 msgr2=0x7f1a0c0b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 --2- 172.21.15.3:0/4087657066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c0ad750 0x7f1a0c0b9ff0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f19fc002f70 tx=0x7f19fc006eb0).stop 2022-07-21T01:35:21.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 -- 172.21.15.3:0/4087657066 shutdown_connections 2022-07-21T01:35:21.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 --2- 172.21.15.3:0/4087657066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c0ad750 0x7f1a0c0b9ff0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 --2- 172.21.15.3:0/4087657066 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0614b0 0x7f1a0c061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 -- 172.21.15.3:0/4087657066 >> 172.21.15.3:0/4087657066 conn(0x7f1a0c066f70 msgr2=0x7f1a0c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:21.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 -- 172.21.15.3:0/4087657066 shutdown_connections 2022-07-21T01:35:21.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.081+0000 7f1a0bfff700 1 -- 172.21.15.3:0/4087657066 wait complete. 2022-07-21T01:35:21.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 Processor -- start 2022-07-21T01:35:21.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 -- start start 2022-07-21T01:35:21.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a0c0614b0 0x7f1a0c1483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0ad750 0x7f1a0c1499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c1405d0 0x7f1a0c0b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c0b4110 con 0x7f1a0c1405d0 2022-07-21T01:35:21.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c148de0 con 0x7f1a0c0614b0 2022-07-21T01:35:21.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c1534d0 con 0x7f1a0c0ad750 2022-07-21T01:35:21.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c1405d0 0x7f1a0c0b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c1405d0 0x7f1a0c0b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44786/0 (socket says 172.21.15.3:44786) 2022-07-21T01:35:21.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0b7fe700 1 -- 172.21.15.3:0/2241230704 learned_addr learned my addr 172.21.15.3:0/2241230704 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:21.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0affd700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a0c0614b0 0x7f1a0c1483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a03fff700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0ad750 0x7f1a0c1499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0b7fe700 1 -- 172.21.15.3:0/2241230704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0ad750 msgr2=0x7f1a0c1499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0b7fe700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0ad750 0x7f1a0c1499f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0b7fe700 1 -- 172.21.15.3:0/2241230704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a0c0614b0 msgr2=0x7f1a0c1483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0b7fe700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a0c0614b0 0x7f1a0c1483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0b7fe700 1 -- 172.21.15.3:0/2241230704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc00b1e0 con 0x7f1a0c1405d0 2022-07-21T01:35:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0b7fe700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c1405d0 0x7f1a0c0b0d00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f1a0c148110 tx=0x7f1a0400c800).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:21.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a08ff9700 1 -- 172.21.15.3:0/2241230704 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a04006aa0 con 0x7f1a0c1405d0 2022-07-21T01:35:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a08ff9700 1 -- 172.21.15.3:0/2241230704 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a04006c00 con 0x7f1a0c1405d0 2022-07-21T01:35:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0c149010 con 0x7f1a0c1405d0 2022-07-21T01:35:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a08ff9700 1 -- 172.21.15.3:0/2241230704 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a04007030 con 0x7f1a0c1405d0 2022-07-21T01:35:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.085+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0c0ad150 con 0x7f1a0c1405d0 2022-07-21T01:35:21.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.089+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0c0af5f0 con 0x7f1a0c1405d0 2022-07-21T01:35:21.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.089+0000 7f1a08ff9700 1 -- 172.21.15.3:0/2241230704 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1a04010030 con 0x7f1a0c1405d0 2022-07-21T01:35:21.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.089+0000 7f1a08ff9700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19f0053950 0x7f19f0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.089+0000 7f1a08ff9700 1 -- 172.21.15.3:0/2241230704 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1a0406bfe0 con 0x7f1a0c1405d0 2022-07-21T01:35:21.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.089+0000 7f1a0affd700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19f0053950 0x7f19f0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.089+0000 7f1a0affd700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19f0053950 0x7f19f0055e10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f19f400d000 tx=0x7f19f4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:21.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.097+0000 7f1a08ff9700 1 -- 172.21.15.3:0/2241230704 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a04042c10 con 0x7f1a0c1405d0 2022-07-21T01:35:21.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.257+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a0c0574e0 con 0x7f1a0c1405d0 2022-07-21T01:35:21.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.257+0000 7f1a08ff9700 1 -- 172.21.15.3:0/2241230704 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2488 (secure 0 0 0) 0x7f1a04046230 con 0x7f1a0c1405d0 2022-07-21T01:35:21.260 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:21.260 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:35:18.753732+0000","modified":"2022-07-21T01:35:19.758646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/2892507714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2892507714},{"type":"v1","addr":"172.21.15.3:6837","nonce":2892507714}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-21T01:35:21.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19f0053950 msgr2=0x7f19f0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19f0053950 0x7f19f0055e10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f19f400d000 tx=0x7f19f4008040).stop 2022-07-21T01:35:21.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c1405d0 msgr2=0x7f1a0c0b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c1405d0 0x7f1a0c0b0d00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f1a0c148110 tx=0x7f1a0400c800).stop 2022-07-21T01:35:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 shutdown_connections 2022-07-21T01:35:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a0c1405d0 0x7f1a0c0b0d00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a0c0ad750 0x7f1a0c1499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f19f0053950 0x7f19f0055e10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 --2- 172.21.15.3:0/2241230704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a0c0614b0 0x7f1a0c1483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 >> 172.21.15.3:0/2241230704 conn(0x7f1a0c066f70 msgr2=0x7f1a0c1467e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 shutdown_connections 2022-07-21T01:35:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.261+0000 7f1a0bfff700 1 -- 172.21.15.3:0/2241230704 wait complete. 2022-07-21T01:35:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 18 2022-07-21T01:35:21.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:35:18.753732+0000', 'modified': '2022-07-21T01:35:19.758646+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'a', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/2892507714', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 2892507714}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 2892507714}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:35:21.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:35:21.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:35:21.697 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":5241,"want_state":"up:active","state":"up:active","rank_uptime":2.930439104,"mdsmap_epoch":18,"osdmap_epoch":48,"osdmap_epoch_barrier":47,"uptime":7.2838539420000004} 2022-07-21T01:35:21.698 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 5241, "mdsmap_epoch": 18, "osdmap_epoch": 48, "osdmap_epoch_barrier": 47, "rank_uptime": 2.930439104, "state": "up:active", "uptime": 7.283853942, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:35:21.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:35:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e89da700 1 Processor -- start 2022-07-21T01:35:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e89da700 1 -- start start 2022-07-21T01:35:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e89da700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86e40b7840 0x7f86e4060be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e89da700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86e40b2930 0x7f86e40610d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e89da700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e40671e0 con 0x7f86e4146720 2022-07-21T01:35:21.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e89da700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e4065540 con 0x7f86e40b7840 2022-07-21T01:35:21.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e89da700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e40b7c20 con 0x7f86e40b2930 2022-07-21T01:35:21.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e2d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86e40b7840 0x7f86e4060be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e259c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86e40b2930 0x7f86e40610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e259c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86e40b2930 0x7f86e40610d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:50238/0 (socket says 172.21.15.3:50238) 2022-07-21T01:35:21.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.805+0000 7f86e259c700 1 -- 172.21.15.3:0/1817821593 learned_addr learned my addr 172.21.15.3:0/1817821593 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:21.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e2d9d700 1 -- 172.21.15.3:0/1817821593 >> v2:172.21.15.143:3300/0 conn(0x7f86e40b2930 msgr2=0x7f86e40610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:21.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e2d9d700 1 --2- 172.21.15.3:0/1817821593 >> v2:172.21.15.143:3300/0 conn(0x7f86e40b2930 0x7f86e40610d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e2d9d700 1 -- 172.21.15.3:0/1817821593 >> v1:172.21.15.143:6789/0 conn(0x7f86e4146720 legacy=0x7f86e4146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e1d9b700 1 -- 172.21.15.3:0/1817821593 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593755878 0 0) 0x7f86e40671e0 con 0x7f86e4146720 2022-07-21T01:35:21.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e2d9d700 1 -- 172.21.15.3:0/1817821593 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e4066ce0 con 0x7f86e40b7840 2022-07-21T01:35:21.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e259c700 1 --2- 172.21.15.3:0/1817821593 >> v2:172.21.15.143:3300/0 conn(0x7f86e40b2930 0x7f86e40610d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:21.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e2d9d700 1 --2- 172.21.15.3:0/1817821593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 0x7f86e4060be0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f86cc002f70 tx=0x7f86cc006df0).ready entity=mon.0 client_cookie=91b56ff9f6c1ad45 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:21.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e1d9b700 1 -- 172.21.15.3:0/1817821593 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86cc00d4a0 con 0x7f86e40b7840 2022-07-21T01:35:21.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e1d9b700 1 -- 172.21.15.3:0/1817821593 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc000e90 con 0x7f86e40b7840 2022-07-21T01:35:21.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e1d9b700 1 -- 172.21.15.3:0/1817821593 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86cc00d7f0 con 0x7f86e40b7840 2022-07-21T01:35:21.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 -- 172.21.15.3:0/1817821593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 msgr2=0x7f86e4060be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 --2- 172.21.15.3:0/1817821593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 0x7f86e4060be0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f86cc002f70 tx=0x7f86cc006df0).stop 2022-07-21T01:35:21.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 -- 172.21.15.3:0/1817821593 shutdown_connections 2022-07-21T01:35:21.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 --2- 172.21.15.3:0/1817821593 >> v2:172.21.15.143:3300/0 conn(0x7f86e40b2930 0x7f86e40610d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 --2- 172.21.15.3:0/1817821593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 0x7f86e4060be0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 -- 172.21.15.3:0/1817821593 >> 172.21.15.3:0/1817821593 conn(0x7f86e405f840 msgr2=0x7f86e405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 -- 172.21.15.3:0/1817821593 shutdown_connections 2022-07-21T01:35:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 -- 172.21.15.3:0/1817821593 wait complete. 2022-07-21T01:35:21.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 Processor -- start 2022-07-21T01:35:21.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.809+0000 7f86e89da700 1 -- start start 2022-07-21T01:35:21.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e89da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86e40b2930 0x7f86e4142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e89da700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 0x7f86e41559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e89da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86e4146720 0x7f86e4155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e89da700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40abb60 con 0x7f86e40b7840 2022-07-21T01:35:21.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e89da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40b1500 con 0x7f86e4146720 2022-07-21T01:35:21.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e89da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e413fa30 con 0x7f86e40b2930 2022-07-21T01:35:21.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 0x7f86e41559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 0x7f86e41559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44798/0 (socket says 172.21.15.3:44798) 2022-07-21T01:35:21.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e259c700 1 -- 172.21.15.3:0/4043982886 learned_addr learned my addr 172.21.15.3:0/4043982886 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:21.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e359e700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86e4146720 0x7f86e4155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e2d9d700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86e40b2930 0x7f86e4142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e259c700 1 -- 172.21.15.3:0/4043982886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86e40b2930 msgr2=0x7f86e4142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e259c700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86e40b2930 0x7f86e4142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e259c700 1 -- 172.21.15.3:0/4043982886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86e4146720 msgr2=0x7f86e4155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e259c700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86e4146720 0x7f86e4155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e259c700 1 -- 172.21.15.3:0/4043982886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc009530 con 0x7f86e40b7840 2022-07-21T01:35:21.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e259c700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 0x7f86e41559e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f86e40b1720 tx=0x7f86dc009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:21.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86dbfff700 1 -- 172.21.15.3:0/4043982886 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86dc00c9d0 con 0x7f86e40b7840 2022-07-21T01:35:21.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86dbfff700 1 -- 172.21.15.3:0/4043982886 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc00cb30 con 0x7f86e40b7840 2022-07-21T01:35:21.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86cc00b120 con 0x7f86e40b7840 2022-07-21T01:35:21.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86dbfff700 1 -- 172.21.15.3:0/4043982886 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86dc00ce90 con 0x7f86e40b7840 2022-07-21T01:35:21.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e40ad090 con 0x7f86e40b7840 2022-07-21T01:35:21.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e41447e0 con 0x7f86e40b7840 2022-07-21T01:35:21.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.813+0000 7f86dbfff700 1 -- 172.21.15.3:0/4043982886 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f86dc022070 con 0x7f86e40b7840 2022-07-21T01:35:21.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.817+0000 7f86dbfff700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86c8053b00 0x7f86c8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:21.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.817+0000 7f86e2d9d700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86c8053b00 0x7f86c8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:21.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.817+0000 7f86dbfff700 1 -- 172.21.15.3:0/4043982886 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f86dc06c970 con 0x7f86e40b7840 2022-07-21T01:35:21.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.817+0000 7f86e2d9d700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86c8053b00 0x7f86c8055fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f86cc018000 tx=0x7f86cc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:21.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.821+0000 7f86dbfff700 1 -- 172.21.15.3:0/4043982886 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f86dc043730 con 0x7f86e40b7840 2022-07-21T01:35:21.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.945+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86e40574e0 con 0x7f86e40b7840 2022-07-21T01:35:21.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.945+0000 7f86dbfff700 1 -- 172.21.15.3:0/4043982886 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16271 (secure 0 0 0) 0x7f86dc046d50 con 0x7f86e40b7840 2022-07-21T01:35:21.948 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:21.949 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":48,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:35:20.769681+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:35:17.190459+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-21T01:35:18.336558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:35:21.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86c8053b00 msgr2=0x7f86c8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86c8053b00 0x7f86c8055fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f86cc018000 tx=0x7f86cc014000).stop 2022-07-21T01:35:21.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 msgr2=0x7f86e41559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:21.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 0x7f86e41559e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f86e40b1720 tx=0x7f86dc009290).stop 2022-07-21T01:35:21.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 shutdown_connections 2022-07-21T01:35:21.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86c8053b00 0x7f86c8055fc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86e4146720 0x7f86e4155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e40b7840 0x7f86e41559e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 --2- 172.21.15.3:0/4043982886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86e40b2930 0x7f86e4142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:21.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 >> 172.21.15.3:0/4043982886 conn(0x7f86e405f840 msgr2=0x7f86e40b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:21.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 shutdown_connections 2022-07-21T01:35:21.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:21.949+0000 7f86e89da700 1 -- 172.21.15.3:0/4043982886 wait complete. 2022-07-21T01:35:21.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:35:17.190459+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:35:18.336558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:35:21.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:35:22.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 Processor -- start 2022-07-21T01:35:22.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 -- start start 2022-07-21T01:35:22.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee100b66f0 0x7fee100b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee101440f0 con 0x7fee100b5b20 2022-07-21T01:35:22.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee10060170 con 0x7fee1005fd90 2022-07-21T01:35:22.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee100ba840 con 0x7fee100b66f0 2022-07-21T01:35:22.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee151db700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee100b66f0 0x7fee100b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee161dd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fee100b5b20 0x7fee100b4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53086/0 (socket says 172.21.15.3:53086) 2022-07-21T01:35:22.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee161dd700 1 -- 172.21.15.3:0/867985348 learned_addr learned my addr 172.21.15.3:0/867985348 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:22.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee151db700 1 -- 172.21.15.3:0/867985348 >> v1:172.21.15.143:6789/0 conn(0x7fee1005fd90 legacy=0x7fee100ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee151db700 1 -- 172.21.15.3:0/867985348 >> v1:172.21.15.3:6789/0 conn(0x7fee100b5b20 legacy=0x7fee100b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee151db700 1 -- 172.21.15.3:0/867985348 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10143be0 con 0x7fee100b66f0 2022-07-21T01:35:22.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee151db700 1 --2- 172.21.15.3:0/867985348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b66f0 0x7fee100b4340 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fee00002f70 tx=0x7fee00006df0).ready entity=mon.0 client_cookie=285ac3051c089b23 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee149da700 1 -- 172.21.15.3:0/867985348 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee0000d610 con 0x7fee100b66f0 2022-07-21T01:35:22.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee149da700 1 -- 172.21.15.3:0/867985348 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee00000e90 con 0x7fee100b66f0 2022-07-21T01:35:22.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee149da700 1 -- 172.21.15.3:0/867985348 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee0000d960 con 0x7fee100b66f0 2022-07-21T01:35:22.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 -- 172.21.15.3:0/867985348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b66f0 msgr2=0x7fee100b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 --2- 172.21.15.3:0/867985348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b66f0 0x7fee100b4340 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fee00002f70 tx=0x7fee00006df0).stop 2022-07-21T01:35:22.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 -- 172.21.15.3:0/867985348 shutdown_connections 2022-07-21T01:35:22.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 --2- 172.21.15.3:0/867985348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b66f0 0x7fee100b4340 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.081+0000 7fee17c40700 1 -- 172.21.15.3:0/867985348 >> 172.21.15.3:0/867985348 conn(0x7fee10062530 msgr2=0x7fee10066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:22.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 -- 172.21.15.3:0/867985348 shutdown_connections 2022-07-21T01:35:22.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 -- 172.21.15.3:0/867985348 wait complete. 2022-07-21T01:35:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 Processor -- start 2022-07-21T01:35:22.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 -- start start 2022-07-21T01:35:22.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee1005fd90 0x7fee1012ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b5b20 0x7fee101526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee100b66f0 0x7fee100b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee101536b0 con 0x7fee100b5b20 2022-07-21T01:35:22.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1012f240 con 0x7fee1005fd90 2022-07-21T01:35:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1012f4d0 con 0x7fee100b66f0 2022-07-21T01:35:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee151db700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b5b20 0x7fee101526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee151db700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b5b20 0x7fee101526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44812/0 (socket says 172.21.15.3:44812) 2022-07-21T01:35:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee151db700 1 -- 172.21.15.3:0/5317408 learned_addr learned my addr 172.21.15.3:0/5317408 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:22.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee161dd700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee100b66f0 0x7fee100b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee159dc700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee1005fd90 0x7fee1012ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee151db700 1 -- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee100b66f0 msgr2=0x7fee100b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee151db700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee100b66f0 0x7fee100b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee151db700 1 -- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee1005fd90 msgr2=0x7fee1012ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee151db700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee1005fd90 0x7fee1012ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee151db700 1 -- 172.21.15.3:0/5317408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0000b180 con 0x7fee100b5b20 2022-07-21T01:35:22.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee161dd700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee100b66f0 0x7fee100b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:35:22.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee159dc700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee1005fd90 0x7fee1012ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:35:22.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee151db700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b5b20 0x7fee101526e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fee00002f70 tx=0x7fee00015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fedfeffd700 1 -- 172.21.15.3:0/5317408 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee0000d990 con 0x7fee100b5b20 2022-07-21T01:35:22.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fedfeffd700 1 -- 172.21.15.3:0/5317408 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee00011ce0 con 0x7fee100b5b20 2022-07-21T01:35:22.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee10155e30 con 0x7fee100b5b20 2022-07-21T01:35:22.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fedfeffd700 1 -- 172.21.15.3:0/5317408 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee0001e650 con 0x7fee100b5b20 2022-07-21T01:35:22.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.085+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee10152bd0 con 0x7fee100b5b20 2022-07-21T01:35:22.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.089+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee10143e50 con 0x7fee100b5b20 2022-07-21T01:35:22.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.089+0000 7fedfeffd700 1 -- 172.21.15.3:0/5317408 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fee00025030 con 0x7fee100b5b20 2022-07-21T01:35:22.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.089+0000 7fedfeffd700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedf0053b00 0x7fedf0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.089+0000 7fedfeffd700 1 -- 172.21.15.3:0/5317408 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fee0006d4b0 con 0x7fee100b5b20 2022-07-21T01:35:22.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.089+0000 7fee159dc700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedf0053b00 0x7fedf0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.089+0000 7fee159dc700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedf0053b00 0x7fedf0055fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fee0c008000 tx=0x7fee0c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.097+0000 7fedfeffd700 1 -- 172.21.15.3:0/5317408 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fee00044270 con 0x7fee100b5b20 2022-07-21T01:35:22.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.221+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fee100574e0 con 0x7fee100b5b20 2022-07-21T01:35:22.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fedfeffd700 1 -- 172.21.15.3:0/5317408 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v48) v1 ==== 110+0+10 (secure 0 0 0) 0x7fee000230b0 con 0x7fee100b5b20 2022-07-21T01:35:22.227 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:35:22.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedf0053b00 msgr2=0x7fedf0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedf0053b00 0x7fedf0055fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fee0c008000 tx=0x7fee0c009040).stop 2022-07-21T01:35:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b5b20 msgr2=0x7fee101526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b5b20 0x7fee101526e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fee00002f70 tx=0x7fee00015000).stop 2022-07-21T01:35:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 shutdown_connections 2022-07-21T01:35:22.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee100b66f0 0x7fee100b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee100b5b20 0x7fee101526e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedf0053b00 0x7fedf0055fc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 --2- 172.21.15.3:0/5317408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee1005fd90 0x7fee1012ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 >> 172.21.15.3:0/5317408 conn(0x7fee10062530 msgr2=0x7fee10066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 shutdown_connections 2022-07-21T01:35:22.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.225+0000 7fee17c40700 1 -- 172.21.15.3:0/5317408 wait complete. 2022-07-21T01:35:22.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:35:22.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7998b8700 1 Processor -- start 2022-07-21T01:35:22.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7998b8700 1 -- start start 2022-07-21T01:35:22.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7998b8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe7940b2710 0x7fe79414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7998b8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7940b01a0 0x7fe79414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7998b8700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940a7250 con 0x7fe7940afd60 2022-07-21T01:35:22.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7998b8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7940a7480 con 0x7fe7940b2710 2022-07-21T01:35:22.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7998b8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940a7720 con 0x7fe7940b01a0 2022-07-21T01:35:22.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7990b7700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe7940afd60 0x7fe7940a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53100/0 (socket says 172.21.15.3:53100) 2022-07-21T01:35:22.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7990b7700 1 -- 172.21.15.3:0/2506420666 learned_addr learned my addr 172.21.15.3:0/2506420666 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:22.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7988b6700 1 --2- 172.21.15.3:0/2506420666 >> v2:172.21.15.143:3300/0 conn(0x7fe7940b2710 0x7fe79414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe793fff700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7940b01a0 0x7fe79414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316663627 0 0) 0x7fe7940a7250 con 0x7fe7940afd60 2022-07-21T01:35:22.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe780003260 con 0x7fe7940afd60 2022-07-21T01:35:22.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4209668622 0 0) 0x7fe780003260 con 0x7fe7940afd60 2022-07-21T01:35:22.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7940b01a0 msgr2=0x7fe79414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:22.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7940b01a0 0x7fe79414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 >> v2:172.21.15.143:3300/0 conn(0x7fe7940b2710 msgr2=0x7fe79414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:22.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 --2- 172.21.15.3:0/2506420666 >> v2:172.21.15.143:3300/0 conn(0x7fe7940b2710 0x7fe79414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940a79c0 con 0x7fe7940afd60 2022-07-21T01:35:22.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fe78c0019e0 con 0x7fe7940afd60 2022-07-21T01:35:22.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 >> v1:172.21.15.3:6789/0 conn(0x7fe7940afd60 legacy=0x7fe7940a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe780003750 0x7fe780003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe780004500 0x7fe780004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe780004e70 0x7fe78000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940a7480 con 0x7fe780003750 2022-07-21T01:35:22.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940a7720 con 0x7fe780004500 2022-07-21T01:35:22.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78000bc40 con 0x7fe780004e70 2022-07-21T01:35:22.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe793fff700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7940b01a0 0x7fe79414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:22.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7988b6700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe780003750 0x7fe780003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe7990b7700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe780004e70 0x7fe78000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.349+0000 7fe793fff700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe780004500 0x7fe780004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7988b6700 1 -- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe780004e70 msgr2=0x7fe78000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:22.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7988b6700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe780004e70 0x7fe78000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7988b6700 1 -- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe780004500 msgr2=0x7fe780004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:22.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7988b6700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe780004500 0x7fe780004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7988b6700 1 -- 172.21.15.3:0/2506420666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe780003260 con 0x7fe780003750 2022-07-21T01:35:22.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7988b6700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe780003750 0x7fe780003b70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fe780004470 tx=0x7fe788006df0).ready entity=mon.0 client_cookie=d35f6b0fed618bee server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe78800e830 con 0x7fe780003750 2022-07-21T01:35:22.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78800be90 con 0x7fe780003750 2022-07-21T01:35:22.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7937fe700 1 -- 172.21.15.3:0/2506420666 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe78800eb80 con 0x7fe780003750 2022-07-21T01:35:22.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7998b8700 1 -- 172.21.15.3:0/2506420666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe780003750 msgr2=0x7fe780003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7998b8700 1 --2- 172.21.15.3:0/2506420666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe780003750 0x7fe780003b70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fe780004470 tx=0x7fe788006df0).stop 2022-07-21T01:35:22.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7990b7700 1 -- 172.21.15.3:0/2506420666 reap_dead start 2022-07-21T01:35:22.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7998b8700 1 -- 172.21.15.3:0/2506420666 shutdown_connections 2022-07-21T01:35:22.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7998b8700 1 -- 172.21.15.3:0/2506420666 >> 172.21.15.3:0/2506420666 conn(0x7fe794060310 msgr2=0x7fe7940b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:22.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7998b8700 1 -- 172.21.15.3:0/2506420666 shutdown_connections 2022-07-21T01:35:22.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7998b8700 1 -- 172.21.15.3:0/2506420666 wait complete. 2022-07-21T01:35:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7998b8700 1 Processor -- start 2022-07-21T01:35:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.353+0000 7fe7998b8700 1 -- start start 2022-07-21T01:35:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7998b8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe780003750 0x7fe7940aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7998b8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7940ab210 0x7fe794065120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7998b8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe794065610 0x7fe794159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7998b8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe794063c80 con 0x7fe7940ab210 2022-07-21T01:35:22.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7998b8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe794065a90 con 0x7fe780003750 2022-07-21T01:35:22.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7998b8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940abc10 con 0x7fe794065610 2022-07-21T01:35:22.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe793fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7940ab210 0x7fe794065120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe793fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7940ab210 0x7fe794065120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44832/0 (socket says 172.21.15.3:44832) 2022-07-21T01:35:22.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe793fff700 1 -- 172.21.15.3:0/3484085638 learned_addr learned my addr 172.21.15.3:0/3484085638 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:22.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7988b6700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe780003750 0x7fe7940aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7990b7700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe794065610 0x7fe794159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe793fff700 1 -- 172.21.15.3:0/3484085638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe794065610 msgr2=0x7fe794159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe793fff700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe794065610 0x7fe794159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe793fff700 1 -- 172.21.15.3:0/3484085638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe780003750 msgr2=0x7fe7940aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe793fff700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe780003750 0x7fe7940aeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe793fff700 1 -- 172.21.15.3:0/3484085638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940ac550 con 0x7fe7940ab210 2022-07-21T01:35:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7990b7700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe794065610 0x7fe794159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe793fff700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7940ab210 0x7fe794065120 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fe794065bd0 tx=0x7fe784008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe791ffb700 1 -- 172.21.15.3:0/3484085638 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe784012800 con 0x7fe7940ab210 2022-07-21T01:35:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe791ffb700 1 -- 172.21.15.3:0/3484085638 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe784012960 con 0x7fe7940ab210 2022-07-21T01:35:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7940ac7f0 con 0x7fe7940ab210 2022-07-21T01:35:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe791ffb700 1 -- 172.21.15.3:0/3484085638 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe784012cc0 con 0x7fe7940ab210 2022-07-21T01:35:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7940aae30 con 0x7fe7940ab210 2022-07-21T01:35:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.357+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7940574e0 con 0x7fe7940ab210 2022-07-21T01:35:22.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.361+0000 7fe791ffb700 1 -- 172.21.15.3:0/3484085638 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe784014470 con 0x7fe7940ab210 2022-07-21T01:35:22.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.361+0000 7fe791ffb700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe774053a70 0x7fe774055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.361+0000 7fe791ffb700 1 -- 172.21.15.3:0/3484085638 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe78406cae0 con 0x7fe7940ab210 2022-07-21T01:35:22.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.361+0000 7fe7988b6700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe774053a70 0x7fe774055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.361+0000 7fe7988b6700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe774053a70 0x7fe774055f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe78801d000 tx=0x7fe788019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.365+0000 7fe791ffb700 1 -- 172.21.15.3:0/3484085638 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe784043840 con 0x7fe7940ab210 2022-07-21T01:35:22.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.493+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe794069a70 con 0x7fe7940ab210 2022-07-21T01:35:22.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe791ffb700 1 -- 172.21.15.3:0/3484085638 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v48) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe784046e60 con 0x7fe7940ab210 2022-07-21T01:35:22.499 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:35:22.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe774053a70 msgr2=0x7fe774055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe774053a70 0x7fe774055f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe78801d000 tx=0x7fe788019000).stop 2022-07-21T01:35:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7940ab210 msgr2=0x7fe794065120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7940ab210 0x7fe794065120 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fe794065bd0 tx=0x7fe784008040).stop 2022-07-21T01:35:22.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 shutdown_connections 2022-07-21T01:35:22.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe794065610 0x7fe794159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7940ab210 0x7fe794065120 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe774053a70 0x7fe774055f30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 --2- 172.21.15.3:0/3484085638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe780003750 0x7fe7940aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.497+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 >> 172.21.15.3:0/3484085638 conn(0x7fe794060310 msgr2=0x7fe7940ad3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.501+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 shutdown_connections 2022-07-21T01:35:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.501+0000 7fe7998b8700 1 -- 172.21.15.3:0/3484085638 wait complete. 2022-07-21T01:35:22.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:35:22.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.621+0000 7f50bbef0700 1 Processor -- start 2022-07-21T01:35:22.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 -- start start 2022-07-21T01:35:22.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b40b7950 0x7f50b40b2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b40609f0 con 0x7f50b4140a50 2022-07-21T01:35:22.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b4140ec0 con 0x7f50b4140610 2022-07-21T01:35:22.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4060c50 con 0x7f50b40b7950 2022-07-21T01:35:22.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50ba48d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f50b4140a50 0x7f50b4143a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53120/0 (socket says 172.21.15.3:53120) 2022-07-21T01:35:22.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50ba48d700 1 -- 172.21.15.3:0/2201736027 learned_addr learned my addr 172.21.15.3:0/2201736027 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:22.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b9c8c700 1 --2- 172.21.15.3:0/2201736027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b40b7950 0x7f50b40b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b8c8a700 1 -- 172.21.15.3:0/2201736027 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194815314 0 0) 0x7f50b40609f0 con 0x7f50b4140a50 2022-07-21T01:35:22.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b8c8a700 1 -- 172.21.15.3:0/2201736027 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f509c003260 con 0x7f50b4140a50 2022-07-21T01:35:22.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b8c8a700 1 -- 172.21.15.3:0/2201736027 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165137263 0 0) 0x7f50b4140ec0 con 0x7f50b4140610 2022-07-21T01:35:22.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b8c8a700 1 -- 172.21.15.3:0/2201736027 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b40609f0 con 0x7f50b4140610 2022-07-21T01:35:22.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b8c8a700 1 -- 172.21.15.3:0/2201736027 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2854793262 0 0) 0x7f509c003260 con 0x7f50b4140a50 2022-07-21T01:35:22.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b9c8c700 1 -- 172.21.15.3:0/2201736027 >> v1:172.21.15.143:6789/0 conn(0x7f50b4140610 legacy=0x7f50b41432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b9c8c700 1 -- 172.21.15.3:0/2201736027 >> v1:172.21.15.3:6789/0 conn(0x7f50b4140a50 legacy=0x7f50b4143a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b9c8c700 1 -- 172.21.15.3:0/2201736027 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4060ef0 con 0x7f50b40b7950 2022-07-21T01:35:22.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b9c8c700 1 --2- 172.21.15.3:0/2201736027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b40b7950 0x7f50b40b2390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f50ac001490 tx=0x7f50b0006df0).ready entity=mon.2 client_cookie=fd1881c9b383b824 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b8c8a700 1 -- 172.21.15.3:0/2201736027 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50ac0019e0 con 0x7f50b40b7950 2022-07-21T01:35:22.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b8c8a700 1 -- 172.21.15.3:0/2201736027 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0000e90 con 0x7f50b40b7950 2022-07-21T01:35:22.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50b8c8a700 1 -- 172.21.15.3:0/2201736027 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50b0012460 con 0x7f50b40b7950 2022-07-21T01:35:22.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 -- 172.21.15.3:0/2201736027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b40b7950 msgr2=0x7f50b40b2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 --2- 172.21.15.3:0/2201736027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b40b7950 0x7f50b40b2390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f50ac001490 tx=0x7f50b0006df0).stop 2022-07-21T01:35:22.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 -- 172.21.15.3:0/2201736027 shutdown_connections 2022-07-21T01:35:22.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 --2- 172.21.15.3:0/2201736027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b40b7950 0x7f50b40b2390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.625+0000 7f50bbef0700 1 -- 172.21.15.3:0/2201736027 >> 172.21.15.3:0/2201736027 conn(0x7f50b4062b60 msgr2=0x7f50b4062f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:22.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 -- 172.21.15.3:0/2201736027 shutdown_connections 2022-07-21T01:35:22.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 -- 172.21.15.3:0/2201736027 wait complete. 2022-07-21T01:35:22.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 Processor -- start 2022-07-21T01:35:22.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 -- start start 2022-07-21T01:35:22.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b40b7950 0x7f50b40b6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b4140610 0x7f50b413fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b4140a50 0x7f50b41400f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4145120 con 0x7f50b40b7950 2022-07-21T01:35:22.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b41453c0 con 0x7f50b4140a50 2022-07-21T01:35:22.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40acc00 con 0x7f50b4140610 2022-07-21T01:35:22.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b9c8c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b40b7950 0x7f50b40b6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b9c8c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b40b7950 0x7f50b40b6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44844/0 (socket says 172.21.15.3:44844) 2022-07-21T01:35:22.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b9c8c700 1 -- 172.21.15.3:0/3727317670 learned_addr learned my addr 172.21.15.3:0/3727317670 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:22.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50ba48d700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b4140a50 0x7f50b41400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b948b700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b4140610 0x7f50b413fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b9c8c700 1 -- 172.21.15.3:0/3727317670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b4140610 msgr2=0x7f50b413fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b9c8c700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b4140610 0x7f50b413fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b9c8c700 1 -- 172.21.15.3:0/3727317670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b4140a50 msgr2=0x7f50b41400f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b9c8c700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b4140a50 0x7f50b41400f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b9c8c700 1 -- 172.21.15.3:0/3727317670 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b000b180 con 0x7f50b40b7950 2022-07-21T01:35:22.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50b9c8c700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b40b7950 0x7f50b40b6680 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f50ac001490 tx=0x7f50b0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50a2ffd700 1 -- 172.21.15.3:0/3727317670 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50b000da20 con 0x7f50b40b7950 2022-07-21T01:35:22.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50a2ffd700 1 -- 172.21.15.3:0/3727317670 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b000db80 con 0x7f50b40b7950 2022-07-21T01:35:22.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b4145660 con 0x7f50b40b7950 2022-07-21T01:35:22.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50a2ffd700 1 -- 172.21.15.3:0/3727317670 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50b001e400 con 0x7f50b40b7950 2022-07-21T01:35:22.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.629+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b4142c10 con 0x7f50b40b7950 2022-07-21T01:35:22.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.633+0000 7f50a2ffd700 1 -- 172.21.15.3:0/3727317670 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f50b0025030 con 0x7f50b40b7950 2022-07-21T01:35:22.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.633+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b41448d0 con 0x7f50b40b7950 2022-07-21T01:35:22.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.633+0000 7f50a2ffd700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5094053a70 0x7f5094055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.633+0000 7f50a2ffd700 1 -- 172.21.15.3:0/3727317670 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f50b006ce20 con 0x7f50b40b7950 2022-07-21T01:35:22.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.633+0000 7f50b948b700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5094053a70 0x7f5094055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.633+0000 7f50b948b700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5094053a70 0x7f5094055f30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f50a400b000 tx=0x7f50a4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.641+0000 7f50a2ffd700 1 -- 172.21.15.3:0/3727317670 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f50b41448d0 con 0x7f50b40b7950 2022-07-21T01:35:22.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.765+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f50b405f480 con 0x7f50b40b7950 2022-07-21T01:35:22.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.765+0000 7f50a2ffd700 1 -- 172.21.15.3:0/3727317670 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) v1 ==== 104+0+10 (secure 0 0 0) 0x7f50b405f480 con 0x7f50b40b7950 2022-07-21T01:35:22.768 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:35:22.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5094053a70 msgr2=0x7f5094055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5094053a70 0x7f5094055f30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f50a400b000 tx=0x7f50a4009040).stop 2022-07-21T01:35:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b40b7950 msgr2=0x7f50b40b6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b40b7950 0x7f50b40b6680 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f50ac001490 tx=0x7f50b0014000).stop 2022-07-21T01:35:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 shutdown_connections 2022-07-21T01:35:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5094053a70 0x7f5094055f30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b4140a50 0x7f50b41400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50b4140610 0x7f50b413fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 --2- 172.21.15.3:0/3727317670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b40b7950 0x7f50b40b6680 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 >> 172.21.15.3:0/3727317670 conn(0x7f50b4062b60 msgr2=0x7f50b40b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:22.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 shutdown_connections 2022-07-21T01:35:22.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.769+0000 7f50bbef0700 1 -- 172.21.15.3:0/3727317670 wait complete. 2022-07-21T01:35:22.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:35:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.889+0000 7f8425fa4700 1 Processor -- start 2022-07-21T01:35:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.889+0000 7f8425fa4700 1 -- start start 2022-07-21T01:35:22.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.889+0000 7f8425fa4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8420066dc0 0x7f842014eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.889+0000 7f8425fa4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84200654c0 con 0x7f84200b3190 2022-07-21T01:35:22.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.889+0000 7f8425fa4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84200b3570 con 0x7f84200b2730 2022-07-21T01:35:22.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.889+0000 7f8425fa4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8420062190 con 0x7f8420066dc0 2022-07-21T01:35:22.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.889+0000 7f8424fa2700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f84200b3190 0x7f8420060ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53134/0 (socket says 172.21.15.3:53134) 2022-07-21T01:35:22.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.889+0000 7f8424fa2700 1 -- 172.21.15.3:0/4221275669 learned_addr learned my addr 172.21.15.3:0/4221275669 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:22.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f84257a3700 1 --2- 172.21.15.3:0/4221275669 >> v2:172.21.15.143:3300/0 conn(0x7f8420066dc0 0x7f842014eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f841f7fe700 1 -- 172.21.15.3:0/4221275669 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390367553 0 0) 0x7f84200654c0 con 0x7f84200b3190 2022-07-21T01:35:22.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f841f7fe700 1 -- 172.21.15.3:0/4221275669 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8400003260 con 0x7f84200b3190 2022-07-21T01:35:22.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f84257a3700 1 -- 172.21.15.3:0/4221275669 >> v1:172.21.15.143:6789/0 conn(0x7f84200b2730 legacy=0x7f84200666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f84257a3700 1 -- 172.21.15.3:0/4221275669 >> v1:172.21.15.3:6789/0 conn(0x7f84200b3190 legacy=0x7f8420060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f84257a3700 1 -- 172.21.15.3:0/4221275669 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842005fbf0 con 0x7f8420066dc0 2022-07-21T01:35:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f84257a3700 1 --2- 172.21.15.3:0/4221275669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8420066dc0 0x7f842014eac0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f84180032f0 tx=0x7f84180091d0).ready entity=mon.1 client_cookie=f9e8f2467153ee5c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f841f7fe700 1 -- 172.21.15.3:0/4221275669 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f841800b710 con 0x7f8420066dc0 2022-07-21T01:35:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f841f7fe700 1 -- 172.21.15.3:0/4221275669 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841800b870 con 0x7f8420066dc0 2022-07-21T01:35:22.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f841f7fe700 1 -- 172.21.15.3:0/4221275669 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f841800b9e0 con 0x7f8420066dc0 2022-07-21T01:35:22.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f8425fa4700 1 -- 172.21.15.3:0/4221275669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8420066dc0 msgr2=0x7f842014eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f8425fa4700 1 --2- 172.21.15.3:0/4221275669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8420066dc0 0x7f842014eac0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f84180032f0 tx=0x7f84180091d0).stop 2022-07-21T01:35:22.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f8425fa4700 1 -- 172.21.15.3:0/4221275669 shutdown_connections 2022-07-21T01:35:22.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f8425fa4700 1 --2- 172.21.15.3:0/4221275669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8420066dc0 0x7f842014eac0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f8425fa4700 1 -- 172.21.15.3:0/4221275669 >> 172.21.15.3:0/4221275669 conn(0x7f8420063870 msgr2=0x7f8420063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:22.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f8425fa4700 1 -- 172.21.15.3:0/4221275669 shutdown_connections 2022-07-21T01:35:22.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.893+0000 7f8425fa4700 1 -- 172.21.15.3:0/4221275669 wait complete. 2022-07-21T01:35:22.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 Processor -- start 2022-07-21T01:35:22.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 -- start start 2022-07-21T01:35:22.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8420066dc0 0x7f84201524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84200b2730 0x7f84200b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84200b3190 0x7f8420156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84201468c0 con 0x7f84200b2730 2022-07-21T01:35:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8420155c00 con 0x7f84200b3190 2022-07-21T01:35:22.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8420155ea0 con 0x7f8420066dc0 2022-07-21T01:35:22.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84200b2730 0x7f84200b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84200b2730 0x7f84200b5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44858/0 (socket says 172.21.15.3:44858) 2022-07-21T01:35:22.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841ffff700 1 -- 172.21.15.3:0/3274869842 learned_addr learned my addr 172.21.15.3:0/3274869842 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:22.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f84257a3700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84200b3190 0x7f8420156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8424fa2700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8420066dc0 0x7f84201524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841ffff700 1 -- 172.21.15.3:0/3274869842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8420066dc0 msgr2=0x7f84201524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841ffff700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8420066dc0 0x7f84201524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841ffff700 1 -- 172.21.15.3:0/3274869842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84200b3190 msgr2=0x7f8420156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:22.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841ffff700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84200b3190 0x7f8420156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:22.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841ffff700 1 -- 172.21.15.3:0/3274869842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8418009470 con 0x7f84200b2730 2022-07-21T01:35:22.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8424fa2700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8420066dc0 0x7f84201524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841ffff700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84200b2730 0x7f84200b5ee0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f8420068010 tx=0x7f8408007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841dffb700 1 -- 172.21.15.3:0/3274869842 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8408012400 con 0x7f84200b2730 2022-07-21T01:35:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841dffb700 1 -- 172.21.15.3:0/3274869842 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8408012560 con 0x7f84200b2730 2022-07-21T01:35:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84201529e0 con 0x7f84200b2730 2022-07-21T01:35:22.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f841dffb700 1 -- 172.21.15.3:0/3274869842 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84080128c0 con 0x7f84200b2730 2022-07-21T01:35:22.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.897+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8420156830 con 0x7f84200b2730 2022-07-21T01:35:22.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.901+0000 7f841dffb700 1 -- 172.21.15.3:0/3274869842 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8408019030 con 0x7f84200b2730 2022-07-21T01:35:22.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.901+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8420146ac0 con 0x7f84200b2730 2022-07-21T01:35:22.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.901+0000 7f841dffb700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8404053a70 0x7f8404055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:22.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.901+0000 7f8424fa2700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8404053a70 0x7f8404055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:22.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.901+0000 7f841dffb700 1 -- 172.21.15.3:0/3274869842 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f840806b570 con 0x7f84200b2730 2022-07-21T01:35:22.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.901+0000 7f8424fa2700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8404053a70 0x7f8404055f30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f841000b000 tx=0x7f8410009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:22.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:22.905+0000 7f841dffb700 1 -- 172.21.15.3:0/3274869842 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8408042220 con 0x7f84200b2730 2022-07-21T01:35:23.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.033+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f842005fe90 con 0x7f84200b2730 2022-07-21T01:35:23.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.033+0000 7f841dffb700 1 -- 172.21.15.3:0/3274869842 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8408045840 con 0x7f84200b2730 2022-07-21T01:35:23.036 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:35:23.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8404053a70 msgr2=0x7f8404055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:23.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8404053a70 0x7f8404055f30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f841000b000 tx=0x7f8410009040).stop 2022-07-21T01:35:23.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84200b2730 msgr2=0x7f84200b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:23.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84200b2730 0x7f84200b5ee0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f8420068010 tx=0x7f8408007c20).stop 2022-07-21T01:35:23.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 shutdown_connections 2022-07-21T01:35:23.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8404053a70 0x7f8404055f30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84200b3190 0x7f8420156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84200b2730 0x7f84200b5ee0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 --2- 172.21.15.3:0/3274869842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8420066dc0 0x7f84201524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 >> 172.21.15.3:0/3274869842 conn(0x7f8420063870 msgr2=0x7f84201430d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:23.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 shutdown_connections 2022-07-21T01:35:23.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.037+0000 7f8425fa4700 1 -- 172.21.15.3:0/3274869842 wait complete. 2022-07-21T01:35:23.049 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:35:23.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:23.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070359e700 1 Processor -- start 2022-07-21T01:35:23.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070359e700 1 -- start start 2022-07-21T01:35:23.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f07040614b0 0x7f0704061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:23.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f07041405d0 0x7f0704140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:23.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070359e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07040aee80 con 0x7f07040ad750 2022-07-21T01:35:23.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070359e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07040b70b0 con 0x7f07040614b0 2022-07-21T01:35:23.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07040af0b0 con 0x7f07041405d0 2022-07-21T01:35:23.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f0701d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f07040ad750 0x7f07040b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53146/0 (socket says 172.21.15.3:53146) 2022-07-21T01:35:23.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f0701d9b700 1 -- 172.21.15.3:0/1204653763 learned_addr learned my addr 172.21.15.3:0/1204653763 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:23.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070259c700 1 --2- 172.21.15.3:0/1204653763 >> v2:172.21.15.3:3300/0 conn(0x7f07040614b0 0x7f0704061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:23.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070159a700 1 -- 172.21.15.3:0/1204653763 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558163107 0 0) 0x7f07040aee80 con 0x7f07040ad750 2022-07-21T01:35:23.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070159a700 1 -- 172.21.15.3:0/1204653763 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e8003260 con 0x7f07040ad750 2022-07-21T01:35:23.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070259c700 1 -- 172.21.15.3:0/1204653763 >> v2:172.21.15.143:3300/0 conn(0x7f07041405d0 msgr2=0x7f0704140a60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2022-07-21T01:35:23.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070259c700 1 --2- 172.21.15.3:0/1204653763 >> v2:172.21.15.143:3300/0 conn(0x7f07041405d0 0x7f0704140a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:35:23.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070259c700 1 -- 172.21.15.3:0/1204653763 >> v1:172.21.15.3:6789/0 conn(0x7f07040ad750 legacy=0x7f07040b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:23.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070259c700 1 -- 172.21.15.3:0/1204653763 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040af350 con 0x7f07040614b0 2022-07-21T01:35:23.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.153+0000 7f070259c700 1 --2- 172.21.15.3:0/1204653763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 0x7f0704061890 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f07040aed30 tx=0x7f06f8006df0).ready entity=mon.0 client_cookie=cb6ab63de81dc911 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:23.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070159a700 1 -- 172.21.15.3:0/1204653763 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06f800d5c0 con 0x7f07040614b0 2022-07-21T01:35:23.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070159a700 1 -- 172.21.15.3:0/1204653763 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f8000e90 con 0x7f07040614b0 2022-07-21T01:35:23.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070159a700 1 -- 172.21.15.3:0/1204653763 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06f800d950 con 0x7f07040614b0 2022-07-21T01:35:23.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 -- 172.21.15.3:0/1204653763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 msgr2=0x7f0704061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:23.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 --2- 172.21.15.3:0/1204653763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 0x7f0704061890 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f07040aed30 tx=0x7f06f8006df0).stop 2022-07-21T01:35:23.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 -- 172.21.15.3:0/1204653763 shutdown_connections 2022-07-21T01:35:23.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 --2- 172.21.15.3:0/1204653763 >> v2:172.21.15.143:3300/0 conn(0x7f07041405d0 0x7f0704140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:35:23.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 --2- 172.21.15.3:0/1204653763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 0x7f0704061890 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 -- 172.21.15.3:0/1204653763 >> 172.21.15.3:0/1204653763 conn(0x7f0704066f70 msgr2=0x7f0704067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:23.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 -- 172.21.15.3:0/1204653763 shutdown_connections 2022-07-21T01:35:23.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 -- 172.21.15.3:0/1204653763 wait complete. 2022-07-21T01:35:23.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 Processor -- start 2022-07-21T01:35:23.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 -- start start 2022-07-21T01:35:23.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 0x7f0704152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:23.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07040ad750 0x7f0704152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:23.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07041405d0 0x7f07040a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:23.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0704153050 con 0x7f07040614b0 2022-07-21T01:35:23.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07041532b0 con 0x7f07040ad750 2022-07-21T01:35:23.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07040b3360 con 0x7f07041405d0 2022-07-21T01:35:23.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 0x7f0704152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:23.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 0x7f0704152530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44874/0 (socket says 172.21.15.3:44874) 2022-07-21T01:35:23.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.157+0000 7f070259c700 1 -- 172.21.15.3:0/314820431 learned_addr learned my addr 172.21.15.3:0/314820431 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:23.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f0701d9b700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07040ad750 0x7f0704152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:23.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f0702d9d700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07041405d0 0x7f07040a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:23.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f070259c700 1 -- 172.21.15.3:0/314820431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07041405d0 msgr2=0x7f07040a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:23.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f070259c700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07041405d0 0x7f07040a1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f070259c700 1 -- 172.21.15.3:0/314820431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07040ad750 msgr2=0x7f0704152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:23.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f070259c700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07040ad750 0x7f0704152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f070259c700 1 -- 172.21.15.3:0/314820431 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f800b180 con 0x7f07040614b0 2022-07-21T01:35:23.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f070259c700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 0x7f0704152530 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f07040aed30 tx=0x7f06f8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:23.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f06f37fe700 1 -- 172.21.15.3:0/314820431 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06f8016070 con 0x7f07040614b0 2022-07-21T01:35:23.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f06f37fe700 1 -- 172.21.15.3:0/314820431 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f8002f70 con 0x7f07040614b0 2022-07-21T01:35:23.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07040b3800 con 0x7f07040614b0 2022-07-21T01:35:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f06f37fe700 1 -- 172.21.15.3:0/314820431 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06f8013de0 con 0x7f07040614b0 2022-07-21T01:35:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07040a22f0 con 0x7f07040614b0 2022-07-21T01:35:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07041465d0 con 0x7f07040614b0 2022-07-21T01:35:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f06f37fe700 1 -- 172.21.15.3:0/314820431 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f06f8025030 con 0x7f07040614b0 2022-07-21T01:35:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f06f37fe700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06e0053a70 0x7f06e0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f06f37fe700 1 -- 172.21.15.3:0/314820431 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f06f806cfd0 con 0x7f07040614b0 2022-07-21T01:35:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f0701d9b700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06e0053a70 0x7f06e0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:23.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.161+0000 7f0701d9b700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06e0053a70 0x7f06e0055f30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f06f400d000 tx=0x7f06f4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:23.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.169+0000 7f06f37fe700 1 -- 172.21.15.3:0/314820431 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f06f8043d90 con 0x7f07040614b0 2022-07-21T01:35:23.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.333+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07040af5f0 con 0x7f07040614b0 2022-07-21T01:35:23.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.333+0000 7f06f37fe700 1 -- 172.21.15.3:0/314820431 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2488 (secure 0 0 0) 0x7f06f8023070 con 0x7f07040614b0 2022-07-21T01:35:23.335 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:23.336 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:35:18.753732+0000","modified":"2022-07-21T01:35:19.758646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/2892507714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2892507714},{"type":"v1","addr":"172.21.15.3:6837","nonce":2892507714}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-21T01:35:23.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06e0053a70 msgr2=0x7f06e0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:23.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06e0053a70 0x7f06e0055f30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f06f400d000 tx=0x7f06f4009040).stop 2022-07-21T01:35:23.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 msgr2=0x7f0704152530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:23.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 0x7f0704152530 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f07040aed30 tx=0x7f06f8014000).stop 2022-07-21T01:35:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 shutdown_connections 2022-07-21T01:35:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07041405d0 0x7f07040a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06e0053a70 0x7f06e0055f30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07040ad750 0x7f0704152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 --2- 172.21.15.3:0/314820431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07040614b0 0x7f0704152530 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:23.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 >> 172.21.15.3:0/314820431 conn(0x7f0704066f70 msgr2=0x7f070405cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:23.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 shutdown_connections 2022-07-21T01:35:23.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:23.337+0000 7f070359e700 1 -- 172.21.15.3:0/314820431 wait complete. 2022-07-21T01:35:23.341 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 18 2022-07-21T01:35:23.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:35:18.753732+0000', 'modified': '2022-07-21T01:35:19.758646+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'a', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/2892507714', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 2892507714}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 2892507714}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:35:23.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:35:23.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:35:23.775 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":5241,"want_state":"up:active","state":"up:active","rank_uptime":5.0082236120000001,"mdsmap_epoch":18,"osdmap_epoch":48,"osdmap_epoch_barrier":47,"uptime":9.3616360959999998} 2022-07-21T01:35:23.775 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 5241, "mdsmap_epoch": 18, "osdmap_epoch": 48, "osdmap_epoch_barrier": 47, "rank_uptime": 5.008223612, "state": "up:active", "uptime": 9.361636096, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:35:23.776 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:35:23.777 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:35:23.778 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:35:23.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:23.779 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:23.788 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:35:23.806 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:35:23.807 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:35:23.817 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:35:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:35:23.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:23.820 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:35:23.825 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:35:23.826 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:23.868 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:35:23.886 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:35:23.887 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:35:23.896 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:35:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:35:23.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:23.899 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:35:23.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:35:23.945+0000 7f1a4250d080 -1 init, newargv = 0x5616d8a6f070 newargc=9ceph-fuse[22332 2022-07-21T01:35:23.945 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:]: starting ceph client 2022-07-21T01:35:23.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22332]: starting fuse 2022-07-21T01:35:25.051 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:35:25.071 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:35:25.071 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:35:25.085 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:35:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:35:25.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:25.089 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:35:25.136 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:35:25.137 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:35:25.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:25.138 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:35:25.138 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:35:25.139 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:35:25.139 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:35:25.140 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:35:25.140 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:25.141 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:35:25.141 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:35:25.142 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:35:25.142 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:25.142 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:35:25.143 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:35:25.143 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:35:25.144 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:35:25.145 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:25.145 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:35:25.146 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:35:25.146 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:35:25.147 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:35:25.147 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:35:25.148 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:25.148 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:35:25.148 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:35:25.193 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22332.asok 2022-07-21T01:35:25.198 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22332.asok 2022-07-21T01:35:25.198 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:25.199 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22332.asok status 2022-07-21T01:35:25.329 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:35:25.329 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:35:25.330 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:35:25.330 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:35:25.330 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:35:25.330 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:35:25.331 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:35:25.331 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22332", 2022-07-21T01:35:25.331 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:35:25.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:35:25.332 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:35:25.332 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:35:25.332 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 5024, 2022-07-21T01:35:25.332 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:35:25.333 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:35:25.333 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:35:25.333 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 5024 2022-07-21T01:35:25.333 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:35:25.334 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:35:25.334 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:35:25.334 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:35:25.334 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2267923326 2022-07-21T01:35:25.335 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:35:25.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:35:25.335 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:35:25.335 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:35:25.336 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:35:25.336 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2267923326 2022-07-21T01:35:25.336 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:35:25.336 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.5024 172.21.15.171:0/2267923326", 2022-07-21T01:35:25.336 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/2267923326", 2022-07-21T01:35:25.337 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:35:25.337 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 18, 2022-07-21T01:35:25.337 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 48, 2022-07-21T01:35:25.337 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:35:25.338 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:35:25.338 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:35:25.338 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:35:25.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:25.340 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:25.341 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:25.349 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:35:25.350 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:35:25.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:25.351 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:25.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:25.370 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:25.371 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:25.418 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:35:25.419 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:35:25.419 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-21T01:35:25.420 DEBUG:teuthology.orchestra.run.smithi171:> import sys 2022-07-21T01:35:25.420 DEBUG:teuthology.orchestra.run.smithi171:> import time 2022-07-21T01:35:25.421 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:35:25.421 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:25.421 DEBUG:teuthology.orchestra.run.smithi171:> n = 1000 2022-07-21T01:35:25.422 DEBUG:teuthology.orchestra.run.smithi171:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-21T01:35:25.422 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:25.423 DEBUG:teuthology.orchestra.run.smithi171:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-21T01:35:25.423 DEBUG:teuthology.orchestra.run.smithi171:> os.makedirs(os.path.dirname(abs_path)) 2022-07-21T01:35:25.423 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:25.424 DEBUG:teuthology.orchestra.run.smithi171:> for i in range(0, n): 2022-07-21T01:35:25.424 DEBUG:teuthology.orchestra.run.smithi171:> fname = "{0}_{1}".format(abs_path, i) 2022-07-21T01:35:25.425 DEBUG:teuthology.orchestra.run.smithi171:> with open(fname, '"'"'w'"'"') as f: 2022-07-21T01:35:25.425 DEBUG:teuthology.orchestra.run.smithi171:> f.write('"'"'content'"'"') 2022-07-21T01:35:25.425 DEBUG:teuthology.orchestra.run.smithi171:> if True: 2022-07-21T01:35:25.426 DEBUG:teuthology.orchestra.run.smithi171:> f.flush() 2022-07-21T01:35:25.426 DEBUG:teuthology.orchestra.run.smithi171:> os.fsync(f.fileno()) 2022-07-21T01:35:25.427 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:35:31.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:35:31.465 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:35:31.469 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:35:31.472 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:35:36.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:36.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66ce733700 1 Processor -- start 2022-07-21T01:35:36.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66ce733700 1 -- start start 2022-07-21T01:35:36.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66ce733700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f66c80614b0 0x7f66c8061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:36.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66ce733700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f66c80ad750 0x7f66c80b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:36.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66ce733700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c80aee80 con 0x7f66c81405d0 2022-07-21T01:35:36.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66ce733700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c80b70b0 con 0x7f66c80614b0 2022-07-21T01:35:36.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66ce733700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c80af0b0 con 0x7f66c80ad750 2022-07-21T01:35:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66cd731700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f66c80614b0 0x7f66c8061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66cd731700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f66c80614b0 0x7f66c8061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44882/0 (socket says 172.21.15.3:44882) 2022-07-21T01:35:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66cd731700 1 -- 172.21.15.3:0/2430203101 learned_addr learned my addr 172.21.15.3:0/2430203101 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66ccf30700 1 --2- 172.21.15.3:0/2430203101 >> v2:172.21.15.143:3300/0 conn(0x7f66c80ad750 0x7f66c80b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66cd731700 1 -- 172.21.15.3:0/2430203101 >> v2:172.21.15.143:3300/0 conn(0x7f66c80ad750 msgr2=0x7f66c80b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66cd731700 1 --2- 172.21.15.3:0/2430203101 >> v2:172.21.15.143:3300/0 conn(0x7f66c80ad750 0x7f66c80b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66cd731700 1 -- 172.21.15.3:0/2430203101 >> v1:172.21.15.143:6789/0 conn(0x7f66c81405d0 legacy=0x7f66c81466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66cd731700 1 -- 172.21.15.3:0/2430203101 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c80af350 con 0x7f66c80614b0 2022-07-21T01:35:36.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66ccf30700 1 --2- 172.21.15.3:0/2430203101 >> v2:172.21.15.143:3300/0 conn(0x7f66c80ad750 0x7f66c80b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:36.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66cd731700 1 --2- 172.21.15.3:0/2430203101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 0x7f66c8061890 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f66c80aed30 tx=0x7f66c4006eb0).ready entity=mon.0 client_cookie=ad79841f53c15da server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:36.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66bbfff700 1 -- 172.21.15.3:0/2430203101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66c400d610 con 0x7f66c80614b0 2022-07-21T01:35:36.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66bbfff700 1 -- 172.21.15.3:0/2430203101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c4000e90 con 0x7f66c80614b0 2022-07-21T01:35:36.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.625+0000 7f66bbfff700 1 -- 172.21.15.3:0/2430203101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66c400d960 con 0x7f66c80614b0 2022-07-21T01:35:36.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 -- 172.21.15.3:0/2430203101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 msgr2=0x7f66c8061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:36.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 --2- 172.21.15.3:0/2430203101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 0x7f66c8061890 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f66c80aed30 tx=0x7f66c4006eb0).stop 2022-07-21T01:35:36.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 -- 172.21.15.3:0/2430203101 shutdown_connections 2022-07-21T01:35:36.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 --2- 172.21.15.3:0/2430203101 >> v2:172.21.15.143:3300/0 conn(0x7f66c80ad750 0x7f66c80b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:36.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 --2- 172.21.15.3:0/2430203101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 0x7f66c8061890 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:36.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 -- 172.21.15.3:0/2430203101 >> 172.21.15.3:0/2430203101 conn(0x7f66c8066f70 msgr2=0x7f66c8067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:36.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 -- 172.21.15.3:0/2430203101 shutdown_connections 2022-07-21T01:35:36.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 -- 172.21.15.3:0/2430203101 wait complete. 2022-07-21T01:35:36.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 Processor -- start 2022-07-21T01:35:36.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 -- start start 2022-07-21T01:35:36.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 0x7f66c80affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:36.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c80ad750 0x7f66c80b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:36.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c81405d0 0x7f66c8143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:36.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c80b38a0 con 0x7f66c80614b0 2022-07-21T01:35:36.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c80acc50 con 0x7f66c81405d0 2022-07-21T01:35:36.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ce733700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c81566a0 con 0x7f66c80ad750 2022-07-21T01:35:36.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66cd731700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 0x7f66c80affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:36.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66cd731700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 0x7f66c80affb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44888/0 (socket says 172.21.15.3:44888) 2022-07-21T01:35:36.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66cd731700 1 -- 172.21.15.3:0/1806541888 learned_addr learned my addr 172.21.15.3:0/1806541888 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:36.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66ccf30700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c80ad750 0x7f66c80b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:36.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66cdf32700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c81405d0 0x7f66c8143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:36.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66cd731700 1 -- 172.21.15.3:0/1806541888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c80ad750 msgr2=0x7f66c80b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:36.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66cd731700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c80ad750 0x7f66c80b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:36.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66cd731700 1 -- 172.21.15.3:0/1806541888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c81405d0 msgr2=0x7f66c8143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:36.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.629+0000 7f66cd731700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c81405d0 0x7f66c8143590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:36.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66cd731700 1 -- 172.21.15.3:0/1806541888 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c400b1e0 con 0x7f66c80614b0 2022-07-21T01:35:36.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66cdf32700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c81405d0 0x7f66c8143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:36.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66cd731700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 0x7f66c80affb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f66c80aed30 tx=0x7f66c4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:36.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66baffd700 1 -- 172.21.15.3:0/1806541888 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66c40068b0 con 0x7f66c80614b0 2022-07-21T01:35:36.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66baffd700 1 -- 172.21.15.3:0/1806541888 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c4002f50 con 0x7f66c80614b0 2022-07-21T01:35:36.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c8156940 con 0x7f66c80614b0 2022-07-21T01:35:36.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66baffd700 1 -- 172.21.15.3:0/1806541888 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66c4011c30 con 0x7f66c80614b0 2022-07-21T01:35:36.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c81429d0 con 0x7f66c80614b0 2022-07-21T01:35:36.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c80b3610 con 0x7f66c80614b0 2022-07-21T01:35:36.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.633+0000 7f66baffd700 1 -- 172.21.15.3:0/1806541888 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f66c4025030 con 0x7f66c80614b0 2022-07-21T01:35:36.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.637+0000 7f66baffd700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66a4053950 0x7f66a4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:36.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.637+0000 7f66baffd700 1 -- 172.21.15.3:0/1806541888 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f66c406d1e0 con 0x7f66c80614b0 2022-07-21T01:35:36.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.637+0000 7f66ccf30700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66a4053950 0x7f66a4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:36.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.637+0000 7f66ccf30700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66a4053950 0x7f66a4055e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f66bc00d000 tx=0x7f66bc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:36.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.641+0000 7f66baffd700 1 -- 172.21.15.3:0/1806541888 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f66c4043fa0 con 0x7f66c80614b0 2022-07-21T01:35:36.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.809+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66c80af5f0 con 0x7f66c80614b0 2022-07-21T01:35:36.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.809+0000 7f66baffd700 1 -- 172.21.15.3:0/1806541888 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2488 (secure 0 0 0) 0x7f66c4023090 con 0x7f66c80614b0 2022-07-21T01:35:36.810 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:36.811 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:35:18.753732+0000","modified":"2022-07-21T01:35:19.758646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/2892507714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2892507714},{"type":"v1","addr":"172.21.15.3:6837","nonce":2892507714}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-21T01:35:36.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66a4053950 msgr2=0x7f66a4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:36.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66a4053950 0x7f66a4055e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f66bc00d000 tx=0x7f66bc008040).stop 2022-07-21T01:35:36.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 msgr2=0x7f66c80affb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:36.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 0x7f66c80affb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f66c80aed30 tx=0x7f66c4015000).stop 2022-07-21T01:35:36.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 shutdown_connections 2022-07-21T01:35:36.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66a4053950 0x7f66a4055e10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:36.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c81405d0 0x7f66c8143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:36.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c80ad750 0x7f66c80b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:36.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 --2- 172.21.15.3:0/1806541888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66c80614b0 0x7f66c80affb0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:36.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 >> 172.21.15.3:0/1806541888 conn(0x7f66c8066f70 msgr2=0x7f66c813fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:36.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 shutdown_connections 2022-07-21T01:35:36.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:36.813+0000 7f66ce733700 1 -- 172.21.15.3:0/1806541888 wait complete. 2022-07-21T01:35:36.816 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 18 2022-07-21T01:35:36.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-21T01:35:36.931 INFO:teuthology.orchestra.run.smithi003.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-21T01:35:36.932 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-21T01:35:36.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:37.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e2d772700 1 Processor -- start 2022-07-21T01:35:37.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e2d772700 1 -- start start 2022-07-21T01:35:37.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e2d772700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0e280b4550 0x7f0e280b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e2d772700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e28144240 con 0x7f0e280b6710 2022-07-21T01:35:37.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e2d772700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e28143d00 con 0x7f0e280b6af0 2022-07-21T01:35:37.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e2d772700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e28143fa0 con 0x7f0e280b4550 2022-07-21T01:35:37.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e26ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0e280b6710 0x7f0e28061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53176/0 (socket says 172.21.15.3:53176) 2022-07-21T01:35:37.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e26ffd700 1 -- 172.21.15.3:0/1520059002 learned_addr learned my addr 172.21.15.3:0/1520059002 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:37.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581420809 0 0) 0x7f0e28144240 con 0x7f0e280b6710 2022-07-21T01:35:37.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e267fc700 1 --2- 172.21.15.3:0/1520059002 >> v2:172.21.15.143:3300/0 conn(0x7f0e280b4550 0x7f0e280b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e0c003260 con 0x7f0e280b6710 2022-07-21T01:35:37.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2262531599 0 0) 0x7f0e0c003260 con 0x7f0e280b6710 2022-07-21T01:35:37.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 >> v2:172.21.15.143:3300/0 conn(0x7f0e280b4550 msgr2=0x7f0e280b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 --2- 172.21.15.3:0/1520059002 >> v2:172.21.15.143:3300/0 conn(0x7f0e280b4550 0x7f0e280b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 >> v1:172.21.15.143:6789/0 conn(0x7f0e280b6af0 legacy=0x7f0e2812bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28061780 con 0x7f0e280b6710 2022-07-21T01:35:37.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f0e200019e0 con 0x7f0e280b6710 2022-07-21T01:35:37.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 >> v1:172.21.15.3:6789/0 conn(0x7f0e280b6710 legacy=0x7f0e28061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e0c003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0c004500 0x7f0e0c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e0c004e70 0x7f0e0c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.029+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e28143fa0 con 0x7f0e0c003780 2022-07-21T01:35:37.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c00bd20 con 0x7f0e0c004500 2022-07-21T01:35:37.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c00bee0 con 0x7f0e0c004e70 2022-07-21T01:35:37.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e26ffd700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e0c003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e267fc700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0c004500 0x7f0e0c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e277fe700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e0c004e70 0x7f0e0c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e26ffd700 1 -- 172.21.15.3:0/1520059002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e0c004e70 msgr2=0x7f0e0c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e26ffd700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e0c004e70 0x7f0e0c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e26ffd700 1 -- 172.21.15.3:0/1520059002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0c004500 msgr2=0x7f0e0c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e26ffd700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0c004500 0x7f0e0c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e277fe700 1 -- 172.21.15.3:0/1520059002 reap_dead start 2022-07-21T01:35:37.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e26ffd700 1 -- 172.21.15.3:0/1520059002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c003260 con 0x7f0e0c003780 2022-07-21T01:35:37.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e26ffd700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e0c003ba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f0e200016f0 tx=0x7f0e28061910).ready entity=mon.0 client_cookie=1b1d376f49d07232 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e2000cc70 con 0x7f0e0c003780 2022-07-21T01:35:37.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2000cdd0 con 0x7f0e0c003780 2022-07-21T01:35:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e25ffb700 1 -- 172.21.15.3:0/1520059002 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e20005b80 con 0x7f0e0c003780 2022-07-21T01:35:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e2d772700 1 -- 172.21.15.3:0/1520059002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 msgr2=0x7f0e0c003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e2d772700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e0c003ba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f0e200016f0 tx=0x7f0e28061910).stop 2022-07-21T01:35:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e2d772700 1 -- 172.21.15.3:0/1520059002 shutdown_connections 2022-07-21T01:35:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e2d772700 1 --2- 172.21.15.3:0/1520059002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e0c003ba0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e2d772700 1 -- 172.21.15.3:0/1520059002 >> 172.21.15.3:0/1520059002 conn(0x7f0e2805c620 msgr2=0x7f0e2805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e2d772700 1 -- 172.21.15.3:0/1520059002 shutdown_connections 2022-07-21T01:35:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e2d772700 1 -- 172.21.15.3:0/1520059002 wait complete. 2022-07-21T01:35:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e2d772700 1 Processor -- start 2022-07-21T01:35:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.033+0000 7f0e2d772700 1 -- start start 2022-07-21T01:35:37.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e2d772700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e280b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e2d772700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28128f60 0x7f0e280ba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e2d772700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e28155d70 0x7f0e281561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e2d772700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e281293c0 con 0x7f0e0c003780 2022-07-21T01:35:37.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e2d772700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e281559b0 con 0x7f0e28128f60 2022-07-21T01:35:37.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e2d772700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e28144680 con 0x7f0e28155d70 2022-07-21T01:35:37.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e26ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e280b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e26ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e280b8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44908/0 (socket says 172.21.15.3:44908) 2022-07-21T01:35:37.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e26ffd700 1 -- 172.21.15.3:0/2493706892 learned_addr learned my addr 172.21.15.3:0/2493706892 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:37.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e277fe700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e28155d70 0x7f0e281561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e267fc700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28128f60 0x7f0e280ba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e26ffd700 1 -- 172.21.15.3:0/2493706892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e28155d70 msgr2=0x7f0e281561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e26ffd700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e28155d70 0x7f0e281561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e26ffd700 1 -- 172.21.15.3:0/2493706892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28128f60 msgr2=0x7f0e280ba1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e26ffd700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28128f60 0x7f0e280ba1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e26ffd700 1 -- 172.21.15.3:0/2493706892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2812a040 con 0x7f0e0c003780 2022-07-21T01:35:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e26ffd700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e280b8a10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f0e28126da0 tx=0x7f0e20013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e17fff700 1 -- 172.21.15.3:0/2493706892 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e20017020 con 0x7f0e0c003780 2022-07-21T01:35:37.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e17fff700 1 -- 172.21.15.3:0/2493706892 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e20004ca0 con 0x7f0e0c003780 2022-07-21T01:35:37.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2812c0f0 con 0x7f0e0c003780 2022-07-21T01:35:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e17fff700 1 -- 172.21.15.3:0/2493706892 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e2000cc70 con 0x7f0e0c003780 2022-07-21T01:35:37.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e280ba6c0 con 0x7f0e0c003780 2022-07-21T01:35:37.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2812bd10 con 0x7f0e0c003780 2022-07-21T01:35:37.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.037+0000 7f0e17fff700 1 -- 172.21.15.3:0/2493706892 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e20010070 con 0x7f0e0c003780 2022-07-21T01:35:37.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.041+0000 7f0e17fff700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e04053b30 0x7f0e04055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.041+0000 7f0e17fff700 1 -- 172.21.15.3:0/2493706892 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0e2001a040 con 0x7f0e0c003780 2022-07-21T01:35:37.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.041+0000 7f0e267fc700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e04053b30 0x7f0e04055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.041+0000 7f0e267fc700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e04053b30 0x7f0e04055ff0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f0e1c00b000 tx=0x7f0e1c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.045+0000 7f0e17fff700 1 -- 172.21.15.3:0/2493706892 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e20043b00 con 0x7f0e0c003780 2022-07-21T01:35:37.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.209+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e28061a20 con 0x7f0e0c003780 2022-07-21T01:35:37.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.209+0000 7f0e17fff700 1 -- 172.21.15.3:0/2493706892 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2488 (secure 0 0 0) 0x7f0e20047120 con 0x7f0e0c003780 2022-07-21T01:35:37.210 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:37.210 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:35:18.753732+0000","modified":"2022-07-21T01:35:19.758646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/2892507714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2892507714},{"type":"v1","addr":"172.21.15.3:6837","nonce":2892507714}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-21T01:35:37.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e04053b30 msgr2=0x7f0e04055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e04053b30 0x7f0e04055ff0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f0e1c00b000 tx=0x7f0e1c006df0).stop 2022-07-21T01:35:37.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 msgr2=0x7f0e280b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e280b8a10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f0e28126da0 tx=0x7f0e20013000).stop 2022-07-21T01:35:37.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 shutdown_connections 2022-07-21T01:35:37.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e28155d70 0x7f0e281561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e04053b30 0x7f0e04055ff0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28128f60 0x7f0e280ba1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 --2- 172.21.15.3:0/2493706892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c003780 0x7f0e280b8a10 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 >> 172.21.15.3:0/2493706892 conn(0x7f0e2805c620 msgr2=0x7f0e28126fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 shutdown_connections 2022-07-21T01:35:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.213+0000 7f0e2d772700 1 -- 172.21.15.3:0/2493706892 wait complete. 2022-07-21T01:35:37.216 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 18 2022-07-21T01:35:37.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-21T01:35:37.329 INFO:teuthology.orchestra.run.smithi003.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-21T01:35:37.330 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-21T01:35:37.331 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:35:37.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 Processor -- start 2022-07-21T01:35:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 -- start start 2022-07-21T01:35:37.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00aa040 0x7f25a00b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f25a00b32c0 0x7f25a00af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a00aa420 con 0x7f25a014b6f0 2022-07-21T01:35:37.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a00ae490 con 0x7f25a00b32c0 2022-07-21T01:35:37.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00af6f0 con 0x7f25a00aa040 2022-07-21T01:35:37.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f25a00b32c0 0x7f25a00af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f25a00b32c0 0x7f25a00af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44918/0 (socket says 172.21.15.3:44918) 2022-07-21T01:35:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259ffff700 1 -- 172.21.15.3:0/608376989 learned_addr learned my addr 172.21.15.3:0/608376989 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a48ab700 1 --2- 172.21.15.3:0/608376989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00aa040 0x7f25a00b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259ffff700 1 -- 172.21.15.3:0/608376989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00aa040 msgr2=0x7f25a00b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259ffff700 1 --2- 172.21.15.3:0/608376989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00aa040 0x7f25a00b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259ffff700 1 -- 172.21.15.3:0/608376989 >> v1:172.21.15.3:6789/0 conn(0x7f25a014b6f0 legacy=0x7f25a005f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259ffff700 1 -- 172.21.15.3:0/608376989 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0148300 con 0x7f25a00b32c0 2022-07-21T01:35:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a48ab700 1 --2- 172.21.15.3:0/608376989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00aa040 0x7f25a00b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:35:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259ffff700 1 --2- 172.21.15.3:0/608376989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00b32c0 0x7f25a00af200 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f25a00adb20 tx=0x7f2594006eb0).ready entity=mon.0 client_cookie=def4ded7c0d5f19 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259f7fe700 1 -- 172.21.15.3:0/608376989 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f259400d570 con 0x7f25a00b32c0 2022-07-21T01:35:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259f7fe700 1 -- 172.21.15.3:0/608376989 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2594000e90 con 0x7f25a00b32c0 2022-07-21T01:35:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f259f7fe700 1 -- 172.21.15.3:0/608376989 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f259400d8c0 con 0x7f25a00b32c0 2022-07-21T01:35:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 -- 172.21.15.3:0/608376989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00b32c0 msgr2=0x7f25a00af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 --2- 172.21.15.3:0/608376989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00b32c0 0x7f25a00af200 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f25a00adb20 tx=0x7f2594006eb0).stop 2022-07-21T01:35:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 -- 172.21.15.3:0/608376989 shutdown_connections 2022-07-21T01:35:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 --2- 172.21.15.3:0/608376989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00b32c0 0x7f25a00af200 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 --2- 172.21.15.3:0/608376989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00aa040 0x7f25a00b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 -- 172.21.15.3:0/608376989 >> 172.21.15.3:0/608376989 conn(0x7f25a0060950 msgr2=0x7f25a0060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:37.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 -- 172.21.15.3:0/608376989 shutdown_connections 2022-07-21T01:35:37.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.457+0000 7f25a6b0f700 1 -- 172.21.15.3:0/608376989 wait complete. 2022-07-21T01:35:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.461+0000 7f25a6b0f700 1 Processor -- start 2022-07-21T01:35:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.461+0000 7f25a6b0f700 1 -- start start 2022-07-21T01:35:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a6b0f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00aa040 0x7f25a00b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a6b0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00b32c0 0x7f25a013de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a6b0f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a014b6f0 0x7f25a013e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a6b0f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a0156810 con 0x7f25a00aa040 2022-07-21T01:35:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a6b0f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a014ac20 con 0x7f25a014b6f0 2022-07-21T01:35:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a6b0f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00b1b60 con 0x7f25a00b32c0 2022-07-21T01:35:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a48ab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00aa040 0x7f25a00b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a48ab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00aa040 0x7f25a00b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44922/0 (socket says 172.21.15.3:44922) 2022-07-21T01:35:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a48ab700 1 -- 172.21.15.3:0/452935830 learned_addr learned my addr 172.21.15.3:0/452935830 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:37.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f259ffff700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00b32c0 0x7f25a013de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a50ac700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a014b6f0 0x7f25a013e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a48ab700 1 -- 172.21.15.3:0/452935830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00b32c0 msgr2=0x7f25a013de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a48ab700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00b32c0 0x7f25a013de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a48ab700 1 -- 172.21.15.3:0/452935830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a014b6f0 msgr2=0x7f25a013e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a48ab700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a014b6f0 0x7f25a013e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a48ab700 1 -- 172.21.15.3:0/452935830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259400b1e0 con 0x7f25a00aa040 2022-07-21T01:35:37.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a48ab700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00aa040 0x7f25a00b1170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f2590007000 tx=0x7f2590008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f259dffb700 1 -- 172.21.15.3:0/452935830 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2590010580 con 0x7f25a00aa040 2022-07-21T01:35:37.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f259dffb700 1 -- 172.21.15.3:0/452935830 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25900106e0 con 0x7f25a00aa040 2022-07-21T01:35:37.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a00b1e00 con 0x7f25a00aa040 2022-07-21T01:35:37.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f259dffb700 1 -- 172.21.15.3:0/452935830 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2590010a40 con 0x7f25a00aa040 2022-07-21T01:35:37.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a00b1660 con 0x7f25a00aa040 2022-07-21T01:35:37.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a00b1f90 con 0x7f25a00aa040 2022-07-21T01:35:37.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.465+0000 7f259dffb700 1 -- 172.21.15.3:0/452935830 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2590010ba0 con 0x7f25a00aa040 2022-07-21T01:35:37.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.469+0000 7f259dffb700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2580053950 0x7f2580055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.469+0000 7f259dffb700 1 -- 172.21.15.3:0/452935830 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f259006c490 con 0x7f25a00aa040 2022-07-21T01:35:37.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.469+0000 7f259ffff700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2580053950 0x7f2580055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.469+0000 7f259ffff700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2580053950 0x7f2580055e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f259401c000 tx=0x7f2594015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.473+0000 7f259dffb700 1 -- 172.21.15.3:0/452935830 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f25900431b0 con 0x7f25a00aa040 2022-07-21T01:35:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a01485a0 con 0x7f25a00aa040 2022-07-21T01:35:37.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f259dffb700 1 -- 172.21.15.3:0/452935830 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2488 (secure 0 0 0) 0x7f25900467d0 con 0x7f25a00aa040 2022-07-21T01:35:37.639 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:37.640 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:35:18.753732+0000","modified":"2022-07-21T01:35:19.758646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/2892507714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2892507714},{"type":"v1","addr":"172.21.15.3:6837","nonce":2892507714}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-21T01:35:37.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2580053950 msgr2=0x7f2580055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2580053950 0x7f2580055e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f259401c000 tx=0x7f2594015000).stop 2022-07-21T01:35:37.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00aa040 msgr2=0x7f25a00b1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00aa040 0x7f25a00b1170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f2590007000 tx=0x7f2590008040).stop 2022-07-21T01:35:37.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 shutdown_connections 2022-07-21T01:35:37.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2580053950 0x7f2580055e10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a014b6f0 0x7f25a013e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a00b32c0 0x7f25a013de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 --2- 172.21.15.3:0/452935830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25a00aa040 0x7f25a00b1170 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.637+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 >> 172.21.15.3:0/452935830 conn(0x7f25a0060950 msgr2=0x7f25a00b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:37.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.641+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 shutdown_connections 2022-07-21T01:35:37.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.641+0000 7f25a6b0f700 1 -- 172.21.15.3:0/452935830 wait complete. 2022-07-21T01:35:37.644 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 18 2022-07-21T01:35:37.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 2022-07-21T01:35:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.757+0000 7fc38b59e700 1 Processor -- start 2022-07-21T01:35:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.757+0000 7fc38b59e700 1 -- start start 2022-07-21T01:35:37.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.757+0000 7fc38b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc38c060eb0 0x7fc38c05f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.757+0000 7fc38b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c143e20 0x7fc38c12c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0b3dd0 con 0x7fc38c0b2340 2022-07-21T01:35:37.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc38c063f80 con 0x7fc38c060eb0 2022-07-21T01:35:37.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0b2720 con 0x7fc38c143e20 2022-07-21T01:35:37.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc389d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc38c060eb0 0x7fc38c05f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38ad9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c143e20 0x7fc38c12c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc389d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc38c060eb0 0x7fc38c05f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:50380/0 (socket says 172.21.15.3:50380) 2022-07-21T01:35:37.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc389d9b700 1 -- 172.21.15.3:0/564271741 learned_addr learned my addr 172.21.15.3:0/564271741 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:37.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38a59c700 1 --1- 172.21.15.3:0/564271741 >> v1:172.21.15.143:6789/0 conn(0x7fc38c0b2340 0x7fc38c0b36c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:55326/0 (socket says 172.21.15.3:55326) 2022-07-21T01:35:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc389d9b700 1 -- 172.21.15.3:0/564271741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c143e20 msgr2=0x7fc38c12c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc389d9b700 1 --2- 172.21.15.3:0/564271741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c143e20 0x7fc38c12c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc389d9b700 1 -- 172.21.15.3:0/564271741 >> v1:172.21.15.143:6789/0 conn(0x7fc38c0b2340 legacy=0x7fc38c0b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc389d9b700 1 -- 172.21.15.3:0/564271741 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c0b2fc0 con 0x7fc38c060eb0 2022-07-21T01:35:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc389d9b700 1 --2- 172.21.15.3:0/564271741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc38c060eb0 0x7fc38c05f840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc3740032f0 tx=0x7fc374009290).ready entity=mon.1 client_cookie=e0518a069136f19c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38959a700 1 -- 172.21.15.3:0/564271741 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37400b8a0 con 0x7fc38c060eb0 2022-07-21T01:35:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38959a700 1 -- 172.21.15.3:0/564271741 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37400ba00 con 0x7fc38c060eb0 2022-07-21T01:35:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38959a700 1 -- 172.21.15.3:0/564271741 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37400bb70 con 0x7fc38c060eb0 2022-07-21T01:35:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 -- 172.21.15.3:0/564271741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc38c060eb0 msgr2=0x7fc38c05f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 --2- 172.21.15.3:0/564271741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc38c060eb0 0x7fc38c05f840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc3740032f0 tx=0x7fc374009290).stop 2022-07-21T01:35:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 -- 172.21.15.3:0/564271741 shutdown_connections 2022-07-21T01:35:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 --2- 172.21.15.3:0/564271741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c143e20 0x7fc38c12c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 --2- 172.21.15.3:0/564271741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc38c060eb0 0x7fc38c05f840 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 -- 172.21.15.3:0/564271741 >> 172.21.15.3:0/564271741 conn(0x7fc38c05c620 msgr2=0x7fc38c05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 -- 172.21.15.3:0/564271741 shutdown_connections 2022-07-21T01:35:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.761+0000 7fc38b59e700 1 -- 172.21.15.3:0/564271741 wait complete. 2022-07-21T01:35:37.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38b59e700 1 Processor -- start 2022-07-21T01:35:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38b59e700 1 -- start start 2022-07-21T01:35:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c060eb0 0x7fc38c0b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc38c0b2340 0x7fc38c145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc38c143e20 0x7fc38c0ad990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c147650 con 0x7fc38c0b2340 2022-07-21T01:35:37.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0b01b0 con 0x7fc38c143e20 2022-07-21T01:35:37.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c146f80 con 0x7fc38c060eb0 2022-07-21T01:35:37.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc389d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc38c0b2340 0x7fc38c145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc389d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc38c0b2340 0x7fc38c145a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44938/0 (socket says 172.21.15.3:44938) 2022-07-21T01:35:37.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc389d9b700 1 -- 172.21.15.3:0/2378991237 learned_addr learned my addr 172.21.15.3:0/2378991237 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:37.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38a59c700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c060eb0 0x7fc38c0b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.765+0000 7fc38ad9d700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc38c143e20 0x7fc38c0ad990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc389d9b700 1 -- 172.21.15.3:0/2378991237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c060eb0 msgr2=0x7fc38c0b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc389d9b700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c060eb0 0x7fc38c0b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc389d9b700 1 -- 172.21.15.3:0/2378991237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc38c143e20 msgr2=0x7fc38c0ad990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc389d9b700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc38c143e20 0x7fc38c0ad990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc389d9b700 1 -- 172.21.15.3:0/2378991237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374009530 con 0x7fc38c0b2340 2022-07-21T01:35:37.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc389d9b700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc38c0b2340 0x7fc38c145a50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fc38c147790 tx=0x7fc374016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc3837fe700 1 -- 172.21.15.3:0/2378991237 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37400b8a0 con 0x7fc38c0b2340 2022-07-21T01:35:37.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc3837fe700 1 -- 172.21.15.3:0/2378991237 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc374021480 con 0x7fc38c0b2340 2022-07-21T01:35:37.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc38b59e700 1 -- 172.21.15.3:0/2378991237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c0b03e0 con 0x7fc38c0b2340 2022-07-21T01:35:37.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc3837fe700 1 -- 172.21.15.3:0/2378991237 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3740217e0 con 0x7fc38c0b2340 2022-07-21T01:35:37.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.769+0000 7fc38b59e700 1 -- 172.21.15.3:0/2378991237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c147220 con 0x7fc38c0b2340 2022-07-21T01:35:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc3837fe700 1 -- 172.21.15.3:0/2378991237 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc37400f070 con 0x7fc38c0b2340 2022-07-21T01:35:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc3837fe700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc370053a70 0x7fc370055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc3837fe700 1 -- 172.21.15.3:0/2378991237 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc37406d1c0 con 0x7fc38c0b2340 2022-07-21T01:35:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc38a59c700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc370053a70 0x7fc370055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc38a59c700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc370053a70 0x7fc370055f30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fc37c00d000 tx=0x7fc37c007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc39b24b740 1 Processor -- start 2022-07-21T01:35:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc39b24b740 1 -- start start 2022-07-21T01:35:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e12f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 0x1e13870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 0x1e117b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc39b24b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d50010 con 0x1e18bc0 2022-07-21T01:35:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc39b24b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d30250 con 0x1e11330 2022-07-21T01:35:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc39b24b740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1dde170 con 0x1e13410 2022-07-21T01:35:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc38a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e12f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc38a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e12f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44944/0 (socket says 172.21.15.3:44944) 2022-07-21T01:35:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc38a59c700 1 -- 172.21.15.3:0/2735725488 learned_addr learned my addr 172.21.15.3:0/2735725488 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.773+0000 7fc389d9b700 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 0x1e13870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38ad9d700 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 0x1e117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38a59c700 1 -- 172.21.15.3:0/2735725488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 msgr2=0x1e13870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38a59c700 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 0x1e13870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38a59c700 1 -- 172.21.15.3:0/2735725488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 msgr2=0x1e117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38a59c700 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 0x1e117b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38a59c700 1 -- 172.21.15.3:0/2735725488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1d88030 con 0x1e18bc0 2022-07-21T01:35:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc389d9b700 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 0x1e13870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38a59c700 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e12f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x1de0240 tx=0x7fc37c00a000).ready entity=mon.0 client_cookie=fb21cf8864fcaab1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38b59e700 1 -- 172.21.15.3:0/2735725488 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37c01c070 con 0x1e18bc0 2022-07-21T01:35:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38b59e700 1 -- 172.21.15.3:0/2735725488 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c022610 con 0x1e18bc0 2022-07-21T01:35:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc38b59e700 1 -- 172.21.15.3:0/2735725488 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37c022970 con 0x1e18bc0 2022-07-21T01:35:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 -- 172.21.15.3:0/2735725488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 msgr2=0x1e12f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e12f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x1de0240 tx=0x7fc37c00a000).stop 2022-07-21T01:35:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 -- 172.21.15.3:0/2735725488 shutdown_connections 2022-07-21T01:35:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 0x1e117b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 0x1e13870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2735725488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e12f20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 -- 172.21.15.3:0/2735725488 >> 172.21.15.3:0/2735725488 conn(0x1ddd540 msgr2=0x1e18150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 -- 172.21.15.3:0/2735725488 shutdown_connections 2022-07-21T01:35:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 -- 172.21.15.3:0/2735725488 wait complete. 2022-07-21T01:35:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 -- start start 2022-07-21T01:35:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.777+0000 7fc39b24b740 1 Processor -- start 2022-07-21T01:35:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e0a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 0x1e0aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 0x1e0df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dd9c70 con 0x1e11330 2022-07-21T01:35:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dd97a0 con 0x1e13410 2022-07-21T01:35:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e272e0 con 0x1e18bc0 2022-07-21T01:35:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e0a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 10 client.0 ll_register_callbacks cb 0x1d97610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-21T01:35:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e0a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44950/0 (socket says 172.21.15.3:44950) 2022-07-21T01:35:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38a59c700 1 -- 172.21.15.3:0/2713930946 learned_addr learned my addr 172.21.15.3:0/2713930946 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38ad9d700 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 0x1e0df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc389d9b700 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 0x1e0aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38a59c700 1 -- 172.21.15.3:0/2713930946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 msgr2=0x1e0df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38a59c700 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 0x1e0df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38a59c700 1 -- 172.21.15.3:0/2713930946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 msgr2=0x1e0aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38a59c700 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 0x1e0aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38a59c700 1 -- 172.21.15.3:0/2713930946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c147220 con 0x1e11330 2022-07-21T01:35:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc38a59c700 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e0a550 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x1de0240 tx=0x7fc37c013000).ready entity=mon.0 client_cookie=d979eb244095159e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc3697fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2022-07-21T01:35:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc3697fa700 1 -- 172.21.15.3:0/2713930946 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37c007850 con 0x1e11330 2022-07-21T01:35:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc3697fa700 1 -- 172.21.15.3:0/2713930946 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc37c022420 con 0x1e11330 2022-07-21T01:35:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 -- 172.21.15.3:0/2713930946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1d45fc0 con 0x1e11330 2022-07-21T01:35:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc3697fa700 1 -- 172.21.15.3:0/2713930946 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc37c0133f0 con 0x1e11330 2022-07-21T01:35:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 10 client.5427 fetch_fsmap learned FSMap version 18 2022-07-21T01:35:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 -- 172.21.15.3:0/2713930946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x7fc38c0b3dd0 con 0x1e11330 2022-07-21T01:35:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc3697fa700 1 -- 172.21.15.3:0/2713930946 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1d45fc0 con 0x1e11330 2022-07-21T01:35:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc3697fa700 1 -- 172.21.15.3:0/2713930946 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 18) v1 ==== 1434+0+0 (secure 0 0 0) 0x7fc37c020030 con 0x1e11330 2022-07-21T01:35:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 10 client.5427 fetch_fsmap finished waiting for FSMap version 18 2022-07-21T01:35:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 10 client.5427 resolve_mds: resolved ID 'a' to GID 5241 2022-07-21T01:35:37.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 20 client.5427 populate_metadata read hostname 'smithi003' 2022-07-21T01:35:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] conn(0x1e0b230 0x1e0fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 4 client.5427 mds_command: new command op to 5241 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-21T01:35:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.781+0000 7fc39b24b740 1 -- 172.21.15.3:0/2713930946 --> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1d3e040 con 0x1e0b230 2022-07-21T01:35:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc389d9b700 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] conn(0x1e0b230 0x1e0fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc3697fa700 10 client.5427 ms_handle_connect on v2:172.21.15.3:6836/2892507714 2022-07-21T01:35:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc389d9b700 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] conn(0x1e0b230 0x1e0fc20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=41c169e17b1849bb server_cookie=abd465fedb99b628 in_seq=0 out_seq=0 2022-07-21T01:35:37.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc3697fa700 1 -- 172.21.15.3:0/2713930946 <== mds.0 v2:172.21.15.3:6836/2892507714 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x1d3e040 con 0x1e0b230 2022-07-21T01:35:37.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc3697fa700 10 client.5427 handle_command_reply: tid=0 2022-07-21T01:35:37.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 client.5427 shutdown 2022-07-21T01:35:37.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 -- 172.21.15.3:0/2713930946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 msgr2=0x1e0a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e0a550 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x1de0240 tx=0x7fc37c013000).stop 2022-07-21T01:35:37.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 -- 172.21.15.3:0/2713930946 shutdown_connections 2022-07-21T01:35:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] conn(0x1e0b230 0x1e0fc20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 0x1e0df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 0x1e0aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 --2- 172.21.15.3:0/2713930946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e0a550 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc3697fa700 0 client.5427 ms_handle_reset on v2:172.21.15.3:6836/2892507714 2022-07-21T01:35:37.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 -- 172.21.15.3:0/2713930946 >> 172.21.15.3:0/2713930946 conn(0x1ddd540 msgr2=0x1dde300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:37.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 -- 172.21.15.3:0/2713930946 shutdown_connections 2022-07-21T01:35:37.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 1 -- 172.21.15.3:0/2713930946 wait complete. 2022-07-21T01:35:37.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.785+0000 7fc39b24b740 20 client.5427 trim_cache size 0 max 16384 2022-07-21T01:35:37.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 Processor -- start 2022-07-21T01:35:37.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 -- start start 2022-07-21T01:35:37.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 0x1e0ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 0x1e11710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e0a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e16f10 con 0x1e18bc0 2022-07-21T01:35:37.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d3e040 con 0x1e11330 2022-07-21T01:35:37.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1dddfa0 con 0x1e13410 2022-07-21T01:35:37.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e0a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e0a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44962/0 (socket says 172.21.15.3:44962) 2022-07-21T01:35:37.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38ad9d700 1 -- 172.21.15.3:0/579168814 learned_addr learned my addr 172.21.15.3:0/579168814 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:37.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc389d9b700 1 --2- 172.21.15.3:0/579168814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 0x1e11710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38a59c700 1 --2- 172.21.15.3:0/579168814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 0x1e0ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38ad9d700 1 -- 172.21.15.3:0/579168814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 msgr2=0x1e11710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38ad9d700 1 --2- 172.21.15.3:0/579168814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 0x1e11710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38ad9d700 1 -- 172.21.15.3:0/579168814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 msgr2=0x1e0ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38ad9d700 1 --2- 172.21.15.3:0/579168814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 0x1e0ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38ad9d700 1 -- 172.21.15.3:0/579168814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1d229b0 con 0x1e18bc0 2022-07-21T01:35:37.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc38ad9d700 1 --2- 172.21.15.3:0/579168814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e0a0c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x1dddcd0 tx=0x7fc384006df0).ready entity=mon.0 client_cookie=9a3bb61623728fc1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc36b7fe700 1 -- 172.21.15.3:0/579168814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc38401a070 con 0x1e18bc0 2022-07-21T01:35:37.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc36b7fe700 1 -- 172.21.15.3:0/579168814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc384015630 con 0x1e18bc0 2022-07-21T01:35:37.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc36b7fe700 1 -- 172.21.15.3:0/579168814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc384015990 con 0x1e18bc0 2022-07-21T01:35:37.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 -- 172.21.15.3:0/579168814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 msgr2=0x1e0a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:37.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 --2- 172.21.15.3:0/579168814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e0a0c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x1dddcd0 tx=0x7fc384006df0).stop 2022-07-21T01:35:37.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 -- 172.21.15.3:0/579168814 shutdown_connections 2022-07-21T01:35:37.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 --2- 172.21.15.3:0/579168814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e18bc0 0x1e0a0c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 --2- 172.21.15.3:0/579168814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e13410 0x1e11710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 --2- 172.21.15.3:0/579168814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e11330 0x1e0ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 -- 172.21.15.3:0/579168814 >> 172.21.15.3:0/579168814 conn(0x1ddd540 msgr2=0x1dde2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 -- 172.21.15.3:0/579168814 shutdown_connections 2022-07-21T01:35:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.801+0000 7fc39b24b740 1 -- 172.21.15.3:0/579168814 wait complete. 2022-07-21T01:35:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 -- start start 2022-07-21T01:35:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 Processor -- start 2022-07-21T01:35:37.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e70500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 0x1e709f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 0x1e7a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e0b740 con 0x1e11330 2022-07-21T01:35:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e0dd10 con 0x1e13410 2022-07-21T01:35:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e767f0 con 0x1e18bc0 2022-07-21T01:35:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 10 client.0 ll_register_callbacks cb 0x1e0f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-21T01:35:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e70500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e70500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44964/0 (socket says 172.21.15.3:44964) 2022-07-21T01:35:37.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38a59c700 1 -- 172.21.15.3:0/39831561 learned_addr learned my addr 172.21.15.3:0/39831561 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:37.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc389d9b700 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 0x1e709f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38ad9d700 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 0x1e7a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38a59c700 1 -- 172.21.15.3:0/39831561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 msgr2=0x1e7a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38a59c700 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 0x1e7a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38a59c700 1 -- 172.21.15.3:0/39831561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 msgr2=0x1e709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38a59c700 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 0x1e709f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38a59c700 1 -- 172.21.15.3:0/39831561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38400b770 con 0x1e11330 2022-07-21T01:35:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc38a59c700 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e70500 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fc37c025040 tx=0x7fc37c02a000).ready entity=mon.0 client_cookie=ca34c217f295c2d7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:37.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc36a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2022-07-21T01:35:37.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc36a7fc700 1 -- 172.21.15.3:0/39831561 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37c0224a0 con 0x1e11330 2022-07-21T01:35:37.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc36a7fc700 1 -- 172.21.15.3:0/39831561 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc37c039da0 con 0x1e11330 2022-07-21T01:35:37.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 -- 172.21.15.3:0/39831561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc37c022610 con 0x1e11330 2022-07-21T01:35:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc36a7fc700 1 -- 172.21.15.3:0/39831561 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc37c013600 con 0x1e11330 2022-07-21T01:35:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 10 client.5433 fetch_fsmap learned FSMap version 18 2022-07-21T01:35:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc39b24b740 1 -- 172.21.15.3:0/39831561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x1d3f5b0 con 0x1e11330 2022-07-21T01:35:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc36a7fc700 1 -- 172.21.15.3:0/39831561 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c022610 con 0x1e11330 2022-07-21T01:35:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.805+0000 7fc36a7fc700 1 -- 172.21.15.3:0/39831561 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 18) v1 ==== 1434+0+0 (secure 0 0 0) 0x7fc37c020030 con 0x1e11330 2022-07-21T01:35:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.809+0000 7fc39b24b740 10 client.5433 fetch_fsmap finished waiting for FSMap version 18 2022-07-21T01:35:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.809+0000 7fc39b24b740 10 client.5433 resolve_mds: resolved ID 'a' to GID 5241 2022-07-21T01:35:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.809+0000 7fc39b24b740 20 client.5433 populate_metadata read hostname 'smithi003' 2022-07-21T01:35:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.809+0000 7fc39b24b740 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] conn(0x1e0a0c0 0x1e7a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.809+0000 7fc39b24b740 4 client.5433 mds_command: new command op to 5241 tid=0[{"prefix": "cache drop"}] 2022-07-21T01:35:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.809+0000 7fc39b24b740 1 -- 172.21.15.3:0/39831561 --> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0x1c33590 con 0x1e0a0c0 2022-07-21T01:35:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.809+0000 7fc389d9b700 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] conn(0x1e0a0c0 0x1e7a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.809+0000 7fc36a7fc700 10 client.5433 ms_handle_connect on v2:172.21.15.3:6836/2892507714 2022-07-21T01:35:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:37.809+0000 7fc389d9b700 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] conn(0x1e0a0c0 0x1e7a630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=9ffa993a7adbaa0d server_cookie=e98f5ab95ee0e7d5 in_seq=0 out_seq=0 2022-07-21T01:35:49.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc36a7fc700 1 -- 172.21.15.3:0/39831561 <== mds.0 v2:172.21.15.3:6836/2892507714 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+380 (crc 0 0 0) 0x1c33590 con 0x1e0a0c0 2022-07-21T01:35:49.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc36a7fc700 10 client.5433 handle_command_reply: tid=0 2022-07-21T01:35:49.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 client.5433 shutdown 2022-07-21T01:35:49.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 -- 172.21.15.3:0/39831561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 msgr2=0x1e70500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e70500 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fc37c025040 tx=0x7fc37c02a000).stop 2022-07-21T01:35:49.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 -- 172.21.15.3:0/39831561 shutdown_connections 2022-07-21T01:35:49.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.3:6836/2892507714,v1:172.21.15.3:6837/2892507714] conn(0x1e0a0c0 0x1e7a630 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e18bc0 0x1e7a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc36a7fc700 0 client.5433 ms_handle_reset on v2:172.21.15.3:6836/2892507714 2022-07-21T01:35:49.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e13410 0x1e709f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 --2- 172.21.15.3:0/39831561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1e11330 0x1e70500 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 -- 172.21.15.3:0/39831561 >> 172.21.15.3:0/39831561 conn(0x1ddd540 msgr2=0x1e17280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:49.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 -- 172.21.15.3:0/39831561 shutdown_connections 2022-07-21T01:35:49.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 1 -- 172.21.15.3:0/39831561 wait complete. 2022-07-21T01:35:49.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc39b24b740 20 client.5433 trim_cache size 0 max 16384 2022-07-21T01:35:49.015 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:35:49.015 INFO:teuthology.orchestra.run.smithi003.stdout: "client_recall": { 2022-07-21T01:35:49.016 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2022-07-21T01:35:49.016 INFO:teuthology.orchestra.run.smithi003.stdout: "message": "(0) Success", 2022-07-21T01:35:49.017 INFO:teuthology.orchestra.run.smithi003.stdout: "recalled": 220 2022-07-21T01:35:49.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:49.018 INFO:teuthology.orchestra.run.smithi003.stdout: "flush_journal": { 2022-07-21T01:35:49.018 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2022-07-21T01:35:49.018 INFO:teuthology.orchestra.run.smithi003.stdout: "message": "" 2022-07-21T01:35:49.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:49.019 INFO:teuthology.orchestra.run.smithi003.stdout: "trim_cache": { 2022-07-21T01:35:49.019 INFO:teuthology.orchestra.run.smithi003.stdout: "trimmed": 0 2022-07-21T01:35:49.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:49.019 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2022-07-21T01:35:49.020 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": { 2022-07-21T01:35:49.020 INFO:teuthology.orchestra.run.smithi003.stdout: "items": 19983, 2022-07-21T01:35:49.021 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes": 3099232 2022-07-21T01:35:49.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:35:49.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:35:49.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 11.199628227 2022-07-21T01:35:49.023 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:35:49.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 -- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc370053a70 msgr2=0x7fc370055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc370053a70 0x7fc370055f30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fc37c00d000 tx=0x7fc37c007b80).stop 2022-07-21T01:35:49.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 -- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc38c0b2340 msgr2=0x7fc38c145a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc38c0b2340 0x7fc38c145a50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fc38c147790 tx=0x7fc374016000).stop 2022-07-21T01:35:49.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 -- 172.21.15.3:0/2378991237 shutdown_connections 2022-07-21T01:35:49.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc370053a70 0x7fc370055f30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc38c143e20 0x7fc38c0ad990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc38c0b2340 0x7fc38c145a50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 --2- 172.21.15.3:0/2378991237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc38c060eb0 0x7fc38c0b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 -- 172.21.15.3:0/2378991237 >> 172.21.15.3:0/2378991237 conn(0x7fc38c05c620 msgr2=0x7fc38c0b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:49.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 -- 172.21.15.3:0/2378991237 shutdown_connections 2022-07-21T01:35:49.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.009+0000 7fc38b59e700 1 -- 172.21.15.3:0/2378991237 wait complete. 2022-07-21T01:35:49.028 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-21T01:35:49.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:49.029 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:49.030 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:49.038 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-21T01:35:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:35:49.040 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-21T01:35:49.041 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:35:49.041 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:35:49.041 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:49.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:49.102 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:49.103 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:49.109 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:35:49.111 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:35:49.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:35:49.113 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:49.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:35:49.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 Processor -- start 2022-07-21T01:35:49.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 -- start start 2022-07-21T01:35:49.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b6710 0x7ff5800b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff5800b4550 0x7ff5800b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff580144240 con 0x7ff580061ec0 2022-07-21T01:35:49.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5800b6fe0 con 0x7ff5800b4550 2022-07-21T01:35:49.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff580143d00 con 0x7ff5800b6710 2022-07-21T01:35:49.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58598c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff5800b4550 0x7ff5800b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58618d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b6710 0x7ff5800b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58598c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff5800b4550 0x7ff5800b6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:50422/0 (socket says 172.21.15.3:50422) 2022-07-21T01:35:49.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58598c700 1 -- 172.21.15.3:0/481413622 learned_addr learned my addr 172.21.15.3:0/481413622 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:49.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58618d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b6710 0x7ff5800b5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:51892/0 (socket says 172.21.15.3:51892) 2022-07-21T01:35:49.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58618d700 1 -- 172.21.15.3:0/481413622 >> v2:172.21.15.143:3300/0 conn(0x7ff5800b4550 msgr2=0x7ff5800b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:49.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58618d700 1 --2- 172.21.15.3:0/481413622 >> v2:172.21.15.143:3300/0 conn(0x7ff5800b4550 0x7ff5800b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58618d700 1 -- 172.21.15.3:0/481413622 >> v1:172.21.15.143:6789/0 conn(0x7ff580061ec0 legacy=0x7ff580146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58518b700 1 -- 172.21.15.3:0/481413622 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181434023 0 0) 0x7ff580144240 con 0x7ff580061ec0 2022-07-21T01:35:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58618d700 1 -- 172.21.15.3:0/481413622 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff580143fa0 con 0x7ff5800b6710 2022-07-21T01:35:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58598c700 1 --2- 172.21.15.3:0/481413622 >> v2:172.21.15.143:3300/0 conn(0x7ff5800b4550 0x7ff5800b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:49.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58618d700 1 --2- 172.21.15.3:0/481413622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b6710 0x7ff5800b5d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff57c002f70 tx=0x7ff57c006df0).ready entity=mon.2 client_cookie=9fcbae0a6af21dfd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:49.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58518b700 1 -- 172.21.15.3:0/481413622 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff57c00d570 con 0x7ff5800b6710 2022-07-21T01:35:49.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58518b700 1 -- 172.21.15.3:0/481413622 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c000e90 con 0x7ff5800b6710 2022-07-21T01:35:49.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff58518b700 1 -- 172.21.15.3:0/481413622 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff57c00d8c0 con 0x7ff5800b6710 2022-07-21T01:35:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 -- 172.21.15.3:0/481413622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b6710 msgr2=0x7ff5800b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 --2- 172.21.15.3:0/481413622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b6710 0x7ff5800b5d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff57c002f70 tx=0x7ff57c006df0).stop 2022-07-21T01:35:49.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 -- 172.21.15.3:0/481413622 shutdown_connections 2022-07-21T01:35:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 --2- 172.21.15.3:0/481413622 >> v2:172.21.15.143:3300/0 conn(0x7ff5800b4550 0x7ff5800b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 --2- 172.21.15.3:0/481413622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b6710 0x7ff5800b5d80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 -- 172.21.15.3:0/481413622 >> 172.21.15.3:0/481413622 conn(0x7ff58005c620 msgr2=0x7ff58005ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 -- 172.21.15.3:0/481413622 shutdown_connections 2022-07-21T01:35:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.269+0000 7ff5883f1700 1 -- 172.21.15.3:0/481413622 wait complete. 2022-07-21T01:35:49.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 Processor -- start 2022-07-21T01:35:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 -- start start 2022-07-21T01:35:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff580061ec0 0x7ff58012ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b4550 0x7ff5800b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5800b6710 0x7ff5800b9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58012c930 con 0x7ff5800b6710 2022-07-21T01:35:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58012c030 con 0x7ff580061ec0 2022-07-21T01:35:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800b96f0 con 0x7ff5800b4550 2022-07-21T01:35:49.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58698e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5800b6710 0x7ff5800b9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58698e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5800b6710 0x7ff5800b9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44984/0 (socket says 172.21.15.3:44984) 2022-07-21T01:35:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58698e700 1 -- 172.21.15.3:0/3015785850 learned_addr learned my addr 172.21.15.3:0/3015785850 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58598c700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b4550 0x7ff5800b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58618d700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff580061ec0 0x7ff58012ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58698e700 1 -- 172.21.15.3:0/3015785850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b4550 msgr2=0x7ff5800b8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58698e700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b4550 0x7ff5800b8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58698e700 1 -- 172.21.15.3:0/3015785850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff580061ec0 msgr2=0x7ff58012ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58698e700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff580061ec0 0x7ff58012ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58698e700 1 -- 172.21.15.3:0/3015785850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57400b1e0 con 0x7ff5800b6710 2022-07-21T01:35:49.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff58698e700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5800b6710 0x7ff5800b9200 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7ff578000f40 tx=0x7ff578007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:49.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5737fe700 1 -- 172.21.15.3:0/3015785850 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff578006aa0 con 0x7ff5800b6710 2022-07-21T01:35:49.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5737fe700 1 -- 172.21.15.3:0/3015785850 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff578006c00 con 0x7ff5800b6710 2022-07-21T01:35:49.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c00b180 con 0x7ff5800b6710 2022-07-21T01:35:49.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5737fe700 1 -- 172.21.15.3:0/3015785850 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff578007030 con 0x7ff5800b6710 2022-07-21T01:35:49.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.273+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58012c690 con 0x7ff5800b6710 2022-07-21T01:35:49.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.277+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff580061780 con 0x7ff5800b6710 2022-07-21T01:35:49.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.277+0000 7ff5737fe700 1 -- 172.21.15.3:0/3015785850 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff578010030 con 0x7ff5800b6710 2022-07-21T01:35:49.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.277+0000 7ff5737fe700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff5600539e0 0x7ff560055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.277+0000 7ff5737fe700 1 -- 172.21.15.3:0/3015785850 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff57806c060 con 0x7ff5800b6710 2022-07-21T01:35:49.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.277+0000 7ff58618d700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff5600539e0 0x7ff560055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.277+0000 7ff58618d700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff5600539e0 0x7ff560055ea0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ff57c01a000 tx=0x7ff57c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:49.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.285+0000 7ff5737fe700 1 -- 172.21.15.3:0/3015785850 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff578042d10 con 0x7ff5800b6710 2022-07-21T01:35:49.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.413+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5800574e0 con 0x7ff5800b6710 2022-07-21T01:35:49.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.417+0000 7ff5737fe700 1 -- 172.21.15.3:0/3015785850 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16271 (secure 0 0 0) 0x7ff578046330 con 0x7ff5800b6710 2022-07-21T01:35:49.418 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:49.418 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":48,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:35:20.769681+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:35:17.190459+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-21T01:35:18.336558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:35:49.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.417+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff5600539e0 msgr2=0x7ff560055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.417+0000 7ff5883f1700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff5600539e0 0x7ff560055ea0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ff57c01a000 tx=0x7ff57c014000).stop 2022-07-21T01:35:49.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.417+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5800b6710 msgr2=0x7ff5800b9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.417+0000 7ff5883f1700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5800b6710 0x7ff5800b9200 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7ff578000f40 tx=0x7ff578007f90).stop 2022-07-21T01:35:49.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.421+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 shutdown_connections 2022-07-21T01:35:49.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.421+0000 7ff5883f1700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5800b6710 0x7ff5800b9200 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.421+0000 7ff5883f1700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5800b4550 0x7ff5800b8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.421+0000 7ff5883f1700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff5600539e0 0x7ff560055ea0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.421+0000 7ff5883f1700 1 --2- 172.21.15.3:0/3015785850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff580061ec0 0x7ff58012ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.421+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 >> 172.21.15.3:0/3015785850 conn(0x7ff58005c620 msgr2=0x7ff5801289b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.421+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 shutdown_connections 2022-07-21T01:35:49.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.421+0000 7ff5883f1700 1 -- 172.21.15.3:0/3015785850 wait complete. 2022-07-21T01:35:49.432 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:35:17.190459+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:35:18.336558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:35:49.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:35:49.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.541+0000 7ff79a366700 1 Processor -- start 2022-07-21T01:35:49.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.541+0000 7ff79a366700 1 -- start start 2022-07-21T01:35:49.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.541+0000 7ff79a366700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff7940b1940 0x7ff794062050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.541+0000 7ff79a366700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7940b10d0 con 0x7ff79413e7d0 2022-07-21T01:35:49.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.541+0000 7ff79a366700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff794063e40 con 0x7ff794063a60 2022-07-21T01:35:49.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.541+0000 7ff79a366700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff794060200 con 0x7ff7940b1940 2022-07-21T01:35:49.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.541+0000 7ff798b63700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff79413e7d0 0x7ff7941414c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53270/0 (socket says 172.21.15.3:53270) 2022-07-21T01:35:49.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.541+0000 7ff798b63700 1 -- 172.21.15.3:0/96444778 learned_addr learned my addr 172.21.15.3:0/96444778 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:49.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff799b65700 1 --2- 172.21.15.3:0/96444778 >> v2:172.21.15.143:3300/0 conn(0x7ff7940b1940 0x7ff794062050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff78bfff700 1 -- 172.21.15.3:0/96444778 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340935132 0 0) 0x7ff7940b10d0 con 0x7ff79413e7d0 2022-07-21T01:35:49.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff78bfff700 1 -- 172.21.15.3:0/96444778 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff778003260 con 0x7ff79413e7d0 2022-07-21T01:35:49.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff78bfff700 1 -- 172.21.15.3:0/96444778 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422483514 0 0) 0x7ff794063e40 con 0x7ff794063a60 2022-07-21T01:35:49.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff78bfff700 1 -- 172.21.15.3:0/96444778 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7940b10d0 con 0x7ff794063a60 2022-07-21T01:35:49.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff799b65700 1 -- 172.21.15.3:0/96444778 >> v1:172.21.15.143:6789/0 conn(0x7ff794063a60 legacy=0x7ff79413e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff799b65700 1 -- 172.21.15.3:0/96444778 >> v1:172.21.15.3:6789/0 conn(0x7ff79413e7d0 legacy=0x7ff7941414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff799b65700 1 -- 172.21.15.3:0/96444778 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7940b1300 con 0x7ff7940b1940 2022-07-21T01:35:49.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff799b65700 1 --2- 172.21.15.3:0/96444778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7940b1940 0x7ff794062050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff78c0032f0 tx=0x7ff78c0091d0).ready entity=mon.1 client_cookie=305c4da7b66bddd2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:49.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff78bfff700 1 -- 172.21.15.3:0/96444778 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff78c00b600 con 0x7ff7940b1940 2022-07-21T01:35:49.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff78bfff700 1 -- 172.21.15.3:0/96444778 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c00b760 con 0x7ff7940b1940 2022-07-21T01:35:49.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff78bfff700 1 -- 172.21.15.3:0/96444778 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff78c00b8d0 con 0x7ff7940b1940 2022-07-21T01:35:49.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff79a366700 1 -- 172.21.15.3:0/96444778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7940b1940 msgr2=0x7ff794062050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff79a366700 1 --2- 172.21.15.3:0/96444778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7940b1940 0x7ff794062050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff78c0032f0 tx=0x7ff78c0091d0).stop 2022-07-21T01:35:49.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff79a366700 1 -- 172.21.15.3:0/96444778 shutdown_connections 2022-07-21T01:35:49.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff79a366700 1 --2- 172.21.15.3:0/96444778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7940b1940 0x7ff794062050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff79a366700 1 -- 172.21.15.3:0/96444778 >> 172.21.15.3:0/96444778 conn(0x7ff79405c620 msgr2=0x7ff79405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:49.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff79a366700 1 -- 172.21.15.3:0/96444778 shutdown_connections 2022-07-21T01:35:49.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.545+0000 7ff79a366700 1 -- 172.21.15.3:0/96444778 wait complete. 2022-07-21T01:35:49.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 Processor -- start 2022-07-21T01:35:49.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 -- start start 2022-07-21T01:35:49.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff794063a60 0x7ff79413c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7940b1940 0x7ff7940ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79413e7d0 0x7ff7940b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7940ae650 con 0x7ff7940b1940 2022-07-21T01:35:49.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7940b4b40 con 0x7ff79413e7d0 2022-07-21T01:35:49.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7940b5c70 con 0x7ff794063a60 2022-07-21T01:35:49.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff798b63700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7940b1940 0x7ff7940ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff798b63700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7940b1940 0x7ff7940ae160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44994/0 (socket says 172.21.15.3:44994) 2022-07-21T01:35:49.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff798b63700 1 -- 172.21.15.3:0/1571634328 learned_addr learned my addr 172.21.15.3:0/1571634328 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:49.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff799b65700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79413e7d0 0x7ff7940b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff799364700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff794063a60 0x7ff79413c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff798b63700 1 -- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff794063a60 msgr2=0x7ff79413c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff798b63700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff794063a60 0x7ff79413c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff798b63700 1 -- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79413e7d0 msgr2=0x7ff7940b4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff798b63700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79413e7d0 0x7ff7940b4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff798b63700 1 -- 172.21.15.3:0/1571634328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c009470 con 0x7ff7940b1940 2022-07-21T01:35:49.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff799b65700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79413e7d0 0x7ff7940b4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:49.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff799364700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff794063a60 0x7ff79413c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:49.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff798b63700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7940b1940 0x7ff7940ae160 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ff790001110 tx=0x7ff790007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:49.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff78a7fc700 1 -- 172.21.15.3:0/1571634328 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7900070d0 con 0x7ff7940b1940 2022-07-21T01:35:49.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff78a7fc700 1 -- 172.21.15.3:0/1571634328 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff790007230 con 0x7ff7940b1940 2022-07-21T01:35:49.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7940b5f10 con 0x7ff7940b1940 2022-07-21T01:35:49.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff78a7fc700 1 -- 172.21.15.3:0/1571634328 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff790005610 con 0x7ff7940b1940 2022-07-21T01:35:49.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.549+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7940b2e60 con 0x7ff7940b1940 2022-07-21T01:35:49.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.553+0000 7ff78a7fc700 1 -- 172.21.15.3:0/1571634328 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff79001a060 con 0x7ff7940b1940 2022-07-21T01:35:49.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.553+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7940b15a0 con 0x7ff7940b1940 2022-07-21T01:35:49.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.553+0000 7ff78a7fc700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff77c053ac0 0x7ff77c055f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.553+0000 7ff78a7fc700 1 -- 172.21.15.3:0/1571634328 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff79006ba70 con 0x7ff7940b1940 2022-07-21T01:35:49.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.553+0000 7ff799364700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff77c053ac0 0x7ff77c055f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.553+0000 7ff799364700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff77c053ac0 0x7ff77c055f80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff78400b000 tx=0x7ff784009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:49.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.561+0000 7ff78a7fc700 1 -- 172.21.15.3:0/1571634328 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff79006de60 con 0x7ff7940b1940 2022-07-21T01:35:49.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.681+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff7940574e0 con 0x7ff7940b1940 2022-07-21T01:35:49.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.681+0000 7ff78a7fc700 1 -- 172.21.15.3:0/1571634328 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v48) v1 ==== 110+0+10 (secure 0 0 0) 0x7ff790018020 con 0x7ff7940b1940 2022-07-21T01:35:49.683 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:35:49.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff77c053ac0 msgr2=0x7ff77c055f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff77c053ac0 0x7ff77c055f80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff78400b000 tx=0x7ff784009040).stop 2022-07-21T01:35:49.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7940b1940 msgr2=0x7ff7940ae160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7940b1940 0x7ff7940ae160 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ff790001110 tx=0x7ff790007af0).stop 2022-07-21T01:35:49.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 shutdown_connections 2022-07-21T01:35:49.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff77c053ac0 0x7ff77c055f80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79413e7d0 0x7ff7940b4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7940b1940 0x7ff7940ae160 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 --2- 172.21.15.3:0/1571634328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff794063a60 0x7ff79413c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 >> 172.21.15.3:0/1571634328 conn(0x7ff79405c620 msgr2=0x7ff7940b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:49.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 shutdown_connections 2022-07-21T01:35:49.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.685+0000 7ff79a366700 1 -- 172.21.15.3:0/1571634328 wait complete. 2022-07-21T01:35:49.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:35:49.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5ceebee700 1 Processor -- start 2022-07-21T01:35:49.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5ceebee700 1 -- start start 2022-07-21T01:35:49.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5ceebee700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5ce80614b0 0x7f5ce8061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5ceebee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5ceebee700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80aee80 con 0x7f5ce80ad750 2022-07-21T01:35:49.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5ceebee700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce80b70b0 con 0x7f5ce80614b0 2022-07-21T01:35:49.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5ceebee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80af0b0 con 0x7f5ce81405d0 2022-07-21T01:35:49.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cee3ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cedbec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5ce80614b0 0x7f5ce8061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cedbec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5ce80614b0 0x7f5ce8061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:50450/0 (socket says 172.21.15.3:50450) 2022-07-21T01:35:49.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cee3ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8140a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:51922/0 (socket says 172.21.15.3:51922) 2022-07-21T01:35:49.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cedbec700 1 -- 172.21.15.3:0/1857103528 learned_addr learned my addr 172.21.15.3:0/1857103528 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:49.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cedbec700 1 -- 172.21.15.3:0/1857103528 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 msgr2=0x7f5ce8140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:49.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cedbec700 1 --2- 172.21.15.3:0/1857103528 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8140a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cedbec700 1 -- 172.21.15.3:0/1857103528 >> v1:172.21.15.143:6789/0 conn(0x7f5ce80ad750 legacy=0x7f5ce80b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cedbec700 1 -- 172.21.15.3:0/1857103528 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce80af350 con 0x7f5ce80614b0 2022-07-21T01:35:49.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.797+0000 7f5cee3ed700 1 --2- 172.21.15.3:0/1857103528 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:35:49.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5cedbec700 1 --2- 172.21.15.3:0/1857103528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce80614b0 0x7f5ce8061890 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f5ce80aed30 tx=0x7f5cd8006eb0).ready entity=mon.1 client_cookie=b971132a77811531 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:49.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5cecbea700 1 -- 172.21.15.3:0/1857103528 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cd800d5c0 con 0x7f5ce80614b0 2022-07-21T01:35:49.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5cecbea700 1 -- 172.21.15.3:0/1857103528 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd8000e90 con 0x7f5ce80614b0 2022-07-21T01:35:49.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5cecbea700 1 -- 172.21.15.3:0/1857103528 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cd800d950 con 0x7f5ce80614b0 2022-07-21T01:35:49.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 -- 172.21.15.3:0/1857103528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce80614b0 msgr2=0x7f5ce8061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 --2- 172.21.15.3:0/1857103528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce80614b0 0x7f5ce8061890 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f5ce80aed30 tx=0x7f5cd8006eb0).stop 2022-07-21T01:35:49.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 -- 172.21.15.3:0/1857103528 shutdown_connections 2022-07-21T01:35:49.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 --2- 172.21.15.3:0/1857103528 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 --2- 172.21.15.3:0/1857103528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce80614b0 0x7f5ce8061890 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 -- 172.21.15.3:0/1857103528 >> 172.21.15.3:0/1857103528 conn(0x7f5ce8066f70 msgr2=0x7f5ce8067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:49.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 -- 172.21.15.3:0/1857103528 shutdown_connections 2022-07-21T01:35:49.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 -- 172.21.15.3:0/1857103528 wait complete. 2022-07-21T01:35:49.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 Processor -- start 2022-07-21T01:35:49.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.801+0000 7f5ceebee700 1 -- start start 2022-07-21T01:35:49.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ceebee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce80614b0 0x7f5ce80b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ceebee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce80ad750 0x7f5ce81415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ceebee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ceebee700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8141fb0 con 0x7f5ce80ad750 2022-07-21T01:35:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ceebee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80a9eb0 con 0x7f5ce80614b0 2022-07-21T01:35:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ceebee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce81576f0 con 0x7f5ce81405d0 2022-07-21T01:35:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ced3eb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce80ad750 0x7f5ce81415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ced3eb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce80ad750 0x7f5ce81415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45010/0 (socket says 172.21.15.3:45010) 2022-07-21T01:35:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ced3eb700 1 -- 172.21.15.3:0/766988707 learned_addr learned my addr 172.21.15.3:0/766988707 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5cedbec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce80614b0 0x7f5ce80b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5cee3ed700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ced3eb700 1 -- 172.21.15.3:0/766988707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 msgr2=0x7f5ce8141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ced3eb700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ced3eb700 1 -- 172.21.15.3:0/766988707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce80614b0 msgr2=0x7f5ce80b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ced3eb700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce80614b0 0x7f5ce80b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ced3eb700 1 -- 172.21.15.3:0/766988707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce0009470 con 0x7f5ce80ad750 2022-07-21T01:35:49.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ced3eb700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce80ad750 0x7f5ce81415d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f5ce80af970 tx=0x7f5ce4007bf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:49.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5cd6ffd700 1 -- 172.21.15.3:0/766988707 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ce4012400 con 0x7f5ce80ad750 2022-07-21T01:35:49.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5cd6ffd700 1 -- 172.21.15.3:0/766988707 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce4012560 con 0x7f5ce80ad750 2022-07-21T01:35:49.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5cd6ffd700 1 -- 172.21.15.3:0/766988707 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ce40128c0 con 0x7f5ce80ad750 2022-07-21T01:35:49.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd800b1e0 con 0x7f5ce80ad750 2022-07-21T01:35:49.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce8157340 con 0x7f5ce80ad750 2022-07-21T01:35:49.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.805+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce80b1700 con 0x7f5ce80ad750 2022-07-21T01:35:49.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.809+0000 7f5cd6ffd700 1 -- 172.21.15.3:0/766988707 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5ce4019030 con 0x7f5ce80ad750 2022-07-21T01:35:49.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.809+0000 7f5cd6ffd700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5cc8053a70 0x7f5cc8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:49.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.809+0000 7f5cd6ffd700 1 -- 172.21.15.3:0/766988707 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5ce406b3d0 con 0x7f5ce80ad750 2022-07-21T01:35:49.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.809+0000 7f5cedbec700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5cc8053a70 0x7f5cc8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:49.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.809+0000 7f5cedbec700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5cc8053a70 0x7f5cc8055f30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5cd801c000 tx=0x7f5cd8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:49.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.817+0000 7f5cd6ffd700 1 -- 172.21.15.3:0/766988707 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5ce80b1700 con 0x7f5ce80ad750 2022-07-21T01:35:49.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.941+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5ce80b1700 con 0x7f5ce80ad750 2022-07-21T01:35:49.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.941+0000 7f5cd6ffd700 1 -- 172.21.15.3:0/766988707 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v48) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5ce80b1700 con 0x7f5ce80ad750 2022-07-21T01:35:49.942 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:35:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.941+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5cc8053a70 msgr2=0x7f5cc8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.941+0000 7f5ceebee700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5cc8053a70 0x7f5cc8055f30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5cd801c000 tx=0x7f5cd8015000).stop 2022-07-21T01:35:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.941+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce80ad750 msgr2=0x7f5ce81415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.941+0000 7f5ceebee700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce80ad750 0x7f5ce81415d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f5ce80af970 tx=0x7f5ce4007bf0).stop 2022-07-21T01:35:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.945+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 shutdown_connections 2022-07-21T01:35:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.945+0000 7f5ceebee700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce81405d0 0x7f5ce8141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.945+0000 7f5ceebee700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce80ad750 0x7f5ce81415d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.945+0000 7f5ceebee700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5cc8053a70 0x7f5cc8055f30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.945+0000 7f5ceebee700 1 --2- 172.21.15.3:0/766988707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce80614b0 0x7f5ce80b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.945+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 >> 172.21.15.3:0/766988707 conn(0x7f5ce8066f70 msgr2=0x7f5ce81444c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.945+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 shutdown_connections 2022-07-21T01:35:49.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:49.945+0000 7f5ceebee700 1 -- 172.21.15.3:0/766988707 wait complete. 2022-07-21T01:35:49.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:35:50.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc895112700 1 Processor -- start 2022-07-21T01:35:50.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc895112700 1 -- start start 2022-07-21T01:35:50.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc895112700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc8900b07a0 0x7fc8900af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc895112700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc890147730 0x7fc8900acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc895112700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc890147b80 con 0x7fc8900b5630 2022-07-21T01:35:50.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc895112700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc890145470 con 0x7fc890147730 2022-07-21T01:35:50.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc895112700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8900ae180 con 0x7fc8900b07a0 2022-07-21T01:35:50.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc894911700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc890147730 0x7fc8900acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc894911700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc890147730 0x7fc8900acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45018/0 (socket says 172.21.15.3:45018) 2022-07-21T01:35:50.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc88ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc8900b5630 0x7fc890067730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53296/0 (socket says 172.21.15.3:53296) 2022-07-21T01:35:50.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.065+0000 7fc894911700 1 -- 172.21.15.3:0/907420410 learned_addr learned my addr 172.21.15.3:0/907420410 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:50.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc88f7fe700 1 --2- 172.21.15.3:0/907420410 >> v2:172.21.15.143:3300/0 conn(0x7fc8900b07a0 0x7fc8900af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc894911700 1 -- 172.21.15.3:0/907420410 >> v2:172.21.15.143:3300/0 conn(0x7fc8900b07a0 msgr2=0x7fc8900af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:50.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc894911700 1 --2- 172.21.15.3:0/907420410 >> v2:172.21.15.143:3300/0 conn(0x7fc8900b07a0 0x7fc8900af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc894911700 1 -- 172.21.15.3:0/907420410 >> v1:172.21.15.3:6789/0 conn(0x7fc8900b5630 legacy=0x7fc890067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc894911700 1 -- 172.21.15.3:0/907420410 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8900b5a10 con 0x7fc890147730 2022-07-21T01:35:50.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc88effd700 1 -- 172.21.15.3:0/907420410 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252383376 0 0) 0x7fc890147b80 con 0x7fc8900b5630 2022-07-21T01:35:50.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc894911700 1 --2- 172.21.15.3:0/907420410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc890147730 0x7fc8900acdf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fc888002f70 tx=0x7fc888006eb0).ready entity=mon.0 client_cookie=6de6b8b9553a0b19 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:50.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc88effd700 1 -- 172.21.15.3:0/907420410 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc88800d5c0 con 0x7fc890147730 2022-07-21T01:35:50.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc88effd700 1 -- 172.21.15.3:0/907420410 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc888000e90 con 0x7fc890147730 2022-07-21T01:35:50.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc88effd700 1 -- 172.21.15.3:0/907420410 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc88800d910 con 0x7fc890147730 2022-07-21T01:35:50.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 -- 172.21.15.3:0/907420410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc890147730 msgr2=0x7fc8900acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 --2- 172.21.15.3:0/907420410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc890147730 0x7fc8900acdf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fc888002f70 tx=0x7fc888006eb0).stop 2022-07-21T01:35:50.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 -- 172.21.15.3:0/907420410 shutdown_connections 2022-07-21T01:35:50.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 --2- 172.21.15.3:0/907420410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc890147730 0x7fc8900acdf0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 --2- 172.21.15.3:0/907420410 >> v2:172.21.15.143:3300/0 conn(0x7fc8900b07a0 0x7fc8900af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 -- 172.21.15.3:0/907420410 >> 172.21.15.3:0/907420410 conn(0x7fc8900626f0 msgr2=0x7fc890062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:50.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 -- 172.21.15.3:0/907420410 shutdown_connections 2022-07-21T01:35:50.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 -- 172.21.15.3:0/907420410 wait complete. 2022-07-21T01:35:50.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 Processor -- start 2022-07-21T01:35:50.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.069+0000 7fc895112700 1 -- start start 2022-07-21T01:35:50.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc895112700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8900b07a0 0x7fc8901458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc895112700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8900b5630 0x7fc89013f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc895112700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc890147730 0x7fc89013f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc895112700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc890140e20 con 0x7fc8900b07a0 2022-07-21T01:35:50.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc895112700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901463f0 con 0x7fc890147730 2022-07-21T01:35:50.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc895112700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc890140a50 con 0x7fc8900b5630 2022-07-21T01:35:50.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8900b07a0 0x7fc8901458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8900b07a0 0x7fc8901458d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45022/0 (socket says 172.21.15.3:45022) 2022-07-21T01:35:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88ffff700 1 -- 172.21.15.3:0/3240110085 learned_addr learned my addr 172.21.15.3:0/3240110085 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:50.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc894911700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc890147730 0x7fc89013f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88f7fe700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8900b5630 0x7fc89013f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88ffff700 1 -- 172.21.15.3:0/3240110085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8900b5630 msgr2=0x7fc89013f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88ffff700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8900b5630 0x7fc89013f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88ffff700 1 -- 172.21.15.3:0/3240110085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc890147730 msgr2=0x7fc89013f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88ffff700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc890147730 0x7fc89013f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88ffff700 1 -- 172.21.15.3:0/3240110085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88800b1e0 con 0x7fc8900b07a0 2022-07-21T01:35:50.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88ffff700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8900b07a0 0x7fc8901458d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fc884000ed0 tx=0x7fc884007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88d7fa700 1 -- 172.21.15.3:0/3240110085 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc88400ce40 con 0x7fc8900b07a0 2022-07-21T01:35:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88d7fa700 1 -- 172.21.15.3:0/3240110085 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8840072e0 con 0x7fc8900b07a0 2022-07-21T01:35:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8901455b0 con 0x7fc8900b07a0 2022-07-21T01:35:50.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc88d7fa700 1 -- 172.21.15.3:0/3240110085 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc884005740 con 0x7fc8900b07a0 2022-07-21T01:35:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.073+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8900b3c80 con 0x7fc8900b07a0 2022-07-21T01:35:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.077+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc890140c10 con 0x7fc8900b07a0 2022-07-21T01:35:50.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.077+0000 7fc88d7fa700 1 -- 172.21.15.3:0/3240110085 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc88401a060 con 0x7fc8900b07a0 2022-07-21T01:35:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.077+0000 7fc88d7fa700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc870053a10 0x7fc870055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.077+0000 7fc88d7fa700 1 -- 172.21.15.3:0/3240110085 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc88406b550 con 0x7fc8900b07a0 2022-07-21T01:35:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.077+0000 7fc88f7fe700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc870053a10 0x7fc870055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.077+0000 7fc88f7fe700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc870053a10 0x7fc870055ed0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fc88000b000 tx=0x7fc880008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:50.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.085+0000 7fc88d7fa700 1 -- 172.21.15.3:0/3240110085 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc884042200 con 0x7fc8900b07a0 2022-07-21T01:35:50.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.209+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc8901408b0 con 0x7fc8900b07a0 2022-07-21T01:35:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.209+0000 7fc88d7fa700 1 -- 172.21.15.3:0/3240110085 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc8901408b0 con 0x7fc8900b07a0 2022-07-21T01:35:50.212 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:35:50.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc870053a10 msgr2=0x7fc870055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc870053a10 0x7fc870055ed0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fc88000b000 tx=0x7fc880008040).stop 2022-07-21T01:35:50.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8900b07a0 msgr2=0x7fc8901458d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8900b07a0 0x7fc8901458d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fc884000ed0 tx=0x7fc884007ac0).stop 2022-07-21T01:35:50.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 shutdown_connections 2022-07-21T01:35:50.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc870053a10 0x7fc870055ed0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc890147730 0x7fc89013f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8900b5630 0x7fc89013f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 --2- 172.21.15.3:0/3240110085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8900b07a0 0x7fc8901458d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 >> 172.21.15.3:0/3240110085 conn(0x7fc8900626f0 msgr2=0x7fc8900acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:50.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 shutdown_connections 2022-07-21T01:35:50.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.213+0000 7fc895112700 1 -- 172.21.15.3:0/3240110085 wait complete. 2022-07-21T01:35:50.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:35:50.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638f014700 1 Processor -- start 2022-07-21T01:35:50.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638f014700 1 -- start start 2022-07-21T01:35:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638f014700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f63880b6420 0x7f6388061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638f014700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63880b60e0 con 0x7f6388142ff0 2022-07-21T01:35:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638f014700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63880b2100 con 0x7f63880b41f0 2022-07-21T01:35:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638f014700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63880b23a0 con 0x7f63880b6420 2022-07-21T01:35:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638e813700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6388142ff0 0x7f6388143480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53312/0 (socket says 172.21.15.3:53312) 2022-07-21T01:35:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638e813700 1 -- 172.21.15.3:0/442649853 learned_addr learned my addr 172.21.15.3:0/442649853 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638e012700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f63880b6420 0x7f6388061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.333+0000 7f638d010700 1 -- 172.21.15.3:0/442649853 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820433805 0 0) 0x7f6378001490 con 0x7f6388142ff0 2022-07-21T01:35:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638d010700 1 -- 172.21.15.3:0/442649853 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6374003260 con 0x7f6388142ff0 2022-07-21T01:35:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638e012700 1 -- 172.21.15.3:0/442649853 >> v1:172.21.15.143:6789/0 conn(0x7f63880b41f0 legacy=0x7f6388061af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638d010700 1 -- 172.21.15.3:0/442649853 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830869485 0 0) 0x7f63880b2100 con 0x7f63880b41f0 2022-07-21T01:35:50.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638e012700 1 -- 172.21.15.3:0/442649853 >> v1:172.21.15.3:6789/0 conn(0x7f6388142ff0 legacy=0x7f6388143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638e012700 1 -- 172.21.15.3:0/442649853 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63880b2640 con 0x7f63880b6420 2022-07-21T01:35:50.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638e012700 1 --2- 172.21.15.3:0/442649853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63880b6420 0x7f6388061600 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f6380002f70 tx=0x7f6380006df0).ready entity=mon.0 client_cookie=241c4312667e418a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:50.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638d010700 1 -- 172.21.15.3:0/442649853 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f638000d610 con 0x7f63880b6420 2022-07-21T01:35:50.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638d010700 1 -- 172.21.15.3:0/442649853 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6380000e90 con 0x7f63880b6420 2022-07-21T01:35:50.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638d010700 1 -- 172.21.15.3:0/442649853 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f638000d960 con 0x7f63880b6420 2022-07-21T01:35:50.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638f014700 1 -- 172.21.15.3:0/442649853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63880b6420 msgr2=0x7f6388061600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638f014700 1 --2- 172.21.15.3:0/442649853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63880b6420 0x7f6388061600 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f6380002f70 tx=0x7f6380006df0).stop 2022-07-21T01:35:50.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638f014700 1 -- 172.21.15.3:0/442649853 shutdown_connections 2022-07-21T01:35:50.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638f014700 1 --2- 172.21.15.3:0/442649853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63880b6420 0x7f6388061600 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638f014700 1 -- 172.21.15.3:0/442649853 >> 172.21.15.3:0/442649853 conn(0x7f638805c620 msgr2=0x7f638805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:50.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638f014700 1 -- 172.21.15.3:0/442649853 shutdown_connections 2022-07-21T01:35:50.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638f014700 1 -- 172.21.15.3:0/442649853 wait complete. 2022-07-21T01:35:50.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638f014700 1 Processor -- start 2022-07-21T01:35:50.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.337+0000 7f638f014700 1 -- start start 2022-07-21T01:35:50.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638f014700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63880b41f0 0x7f6388142340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638f014700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63880b6420 0x7f638813f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638f014700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6388142ff0 0x7f638813f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638f014700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638813fb20 con 0x7f6388142ff0 2022-07-21T01:35:50.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638f014700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388146060 con 0x7f63880b41f0 2022-07-21T01:35:50.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638f014700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638813ffc0 con 0x7f63880b6420 2022-07-21T01:35:50.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e813700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6388142ff0 0x7f638813f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e813700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6388142ff0 0x7f638813f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45038/0 (socket says 172.21.15.3:45038) 2022-07-21T01:35:50.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e813700 1 -- 172.21.15.3:0/1135660505 learned_addr learned my addr 172.21.15.3:0/1135660505 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:50.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638d811700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63880b6420 0x7f638813f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e012700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63880b41f0 0x7f6388142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e813700 1 -- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63880b6420 msgr2=0x7f638813f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e813700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63880b6420 0x7f638813f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e813700 1 -- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63880b41f0 msgr2=0x7f6388142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e813700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63880b41f0 0x7f6388142340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e813700 1 -- 172.21.15.3:0/1135660505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638000b180 con 0x7f6388142ff0 2022-07-21T01:35:50.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e012700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63880b41f0 0x7f6388142340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:50.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638d811700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63880b6420 0x7f638813f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:35:50.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638e813700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6388142ff0 0x7f638813f630 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6378001620 tx=0x7f6378007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:50.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f637f7fe700 1 -- 172.21.15.3:0/1135660505 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6378005ba0 con 0x7f6388142ff0 2022-07-21T01:35:50.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6388140260 con 0x7f6388142ff0 2022-07-21T01:35:50.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f637f7fe700 1 -- 172.21.15.3:0/1135660505 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6378005d00 con 0x7f6388142ff0 2022-07-21T01:35:50.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f637f7fe700 1 -- 172.21.15.3:0/1135660505 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63780072c0 con 0x7f6388142ff0 2022-07-21T01:35:50.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63880afb70 con 0x7f6388142ff0 2022-07-21T01:35:50.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.341+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638813fe20 con 0x7f6388142ff0 2022-07-21T01:35:50.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.345+0000 7f637f7fe700 1 -- 172.21.15.3:0/1135660505 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6378010070 con 0x7f6388142ff0 2022-07-21T01:35:50.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.345+0000 7f637f7fe700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6368053b30 0x7f6368055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.345+0000 7f637f7fe700 1 -- 172.21.15.3:0/1135660505 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f637806b810 con 0x7f6388142ff0 2022-07-21T01:35:50.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.345+0000 7f638e012700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6368053b30 0x7f6368055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.345+0000 7f638e012700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6368053b30 0x7f6368055ff0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6380018000 tx=0x7f6380014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:50.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.349+0000 7f637f7fe700 1 -- 172.21.15.3:0/1135660505 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f638813fe20 con 0x7f6388142ff0 2022-07-21T01:35:50.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.477+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63880b60e0 con 0x7f6388142ff0 2022-07-21T01:35:50.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.477+0000 7f637f7fe700 1 -- 172.21.15.3:0/1135660505 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) v1 ==== 100+0+10 (secure 0 0 0) 0x7f63880b60e0 con 0x7f6388142ff0 2022-07-21T01:35:50.480 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:35:50.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6368053b30 msgr2=0x7f6368055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6368053b30 0x7f6368055ff0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6380018000 tx=0x7f6380014000).stop 2022-07-21T01:35:50.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6388142ff0 msgr2=0x7f638813f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6388142ff0 0x7f638813f630 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6378001620 tx=0x7f6378007c20).stop 2022-07-21T01:35:50.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 shutdown_connections 2022-07-21T01:35:50.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6388142ff0 0x7f638813f630 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63880b6420 0x7f638813f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6368053b30 0x7f6368055ff0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 --2- 172.21.15.3:0/1135660505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63880b41f0 0x7f6388142340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 >> 172.21.15.3:0/1135660505 conn(0x7f638805c620 msgr2=0x7f63880ba0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:50.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 shutdown_connections 2022-07-21T01:35:50.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.481+0000 7f638f014700 1 -- 172.21.15.3:0/1135660505 wait complete. 2022-07-21T01:35:50.493 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:35:50.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:35:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 Processor -- start 2022-07-21T01:35:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 -- start start 2022-07-21T01:35:50.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fae200a9320 0x7fae201466e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae200b0d10 0x7fae20146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae200b12d0 con 0x7fae200b0560 2022-07-21T01:35:50.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae200a9700 con 0x7fae200a9320 2022-07-21T01:35:50.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae200b1500 con 0x7fae200b0d10 2022-07-21T01:35:50.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae25938700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fae200a9320 0x7fae201466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae26139700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fae200b0560 0x7fae20140350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53326/0 (socket says 172.21.15.3:53326) 2022-07-21T01:35:50.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae26139700 1 -- 172.21.15.3:0/2892251729 learned_addr learned my addr 172.21.15.3:0/2892251729 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:35:50.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae25137700 1 --2- 172.21.15.3:0/2892251729 >> v2:172.21.15.143:3300/0 conn(0x7fae200b0d10 0x7fae20146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae24936700 1 -- 172.21.15.3:0/2892251729 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206774996 0 0) 0x7fae200b12d0 con 0x7fae200b0560 2022-07-21T01:35:50.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae24936700 1 -- 172.21.15.3:0/2892251729 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae08003260 con 0x7fae200b0560 2022-07-21T01:35:50.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae25938700 1 -- 172.21.15.3:0/2892251729 >> v2:172.21.15.143:3300/0 conn(0x7fae200b0d10 msgr2=0x7fae20146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:35:50.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae25938700 1 --2- 172.21.15.3:0/2892251729 >> v2:172.21.15.143:3300/0 conn(0x7fae200b0d10 0x7fae20146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae25938700 1 -- 172.21.15.3:0/2892251729 >> v1:172.21.15.3:6789/0 conn(0x7fae200b0560 legacy=0x7fae20140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae25938700 1 -- 172.21.15.3:0/2892251729 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae200b17a0 con 0x7fae200a9320 2022-07-21T01:35:50.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae25938700 1 --2- 172.21.15.3:0/2892251729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 0x7fae201466e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fae1c002f70 tx=0x7fae1c006df0).ready entity=mon.0 client_cookie=ae6a573755615ca4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:50.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae24936700 1 -- 172.21.15.3:0/2892251729 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae1c00d4f0 con 0x7fae200a9320 2022-07-21T01:35:50.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae24936700 1 -- 172.21.15.3:0/2892251729 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1c000e90 con 0x7fae200a9320 2022-07-21T01:35:50.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae24936700 1 -- 172.21.15.3:0/2892251729 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae1c00d840 con 0x7fae200a9320 2022-07-21T01:35:50.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 -- 172.21.15.3:0/2892251729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 msgr2=0x7fae201466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 --2- 172.21.15.3:0/2892251729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 0x7fae201466e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fae1c002f70 tx=0x7fae1c006df0).stop 2022-07-21T01:35:50.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 -- 172.21.15.3:0/2892251729 shutdown_connections 2022-07-21T01:35:50.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 --2- 172.21.15.3:0/2892251729 >> v2:172.21.15.143:3300/0 conn(0x7fae200b0d10 0x7fae20146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 --2- 172.21.15.3:0/2892251729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 0x7fae201466e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 -- 172.21.15.3:0/2892251729 >> 172.21.15.3:0/2892251729 conn(0x7fae200612d0 msgr2=0x7fae20062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:50.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 -- 172.21.15.3:0/2892251729 shutdown_connections 2022-07-21T01:35:50.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.601+0000 7fae2693a700 1 -- 172.21.15.3:0/2892251729 wait complete. 2022-07-21T01:35:50.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 Processor -- start 2022-07-21T01:35:50.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 -- start start 2022-07-21T01:35:50.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 0x7fae2013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae200b0560 0x7fae200ab290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae200b0d10 0x7fae20156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae201558c0 con 0x7fae200a9320 2022-07-21T01:35:50.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae20156720 con 0x7fae200b0d10 2022-07-21T01:35:50.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae20155af0 con 0x7fae200b0560 2022-07-21T01:35:50.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25938700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 0x7fae2013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25938700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 0x7fae2013f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45048/0 (socket says 172.21.15.3:45048) 2022-07-21T01:35:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25938700 1 -- 172.21.15.3:0/2729343650 learned_addr learned my addr 172.21.15.3:0/2729343650 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:35:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25137700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae200b0560 0x7fae200ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae26139700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae200b0d10 0x7fae20156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25938700 1 -- 172.21.15.3:0/2729343650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae200b0560 msgr2=0x7fae200ab290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25938700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae200b0560 0x7fae200ab290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25938700 1 -- 172.21.15.3:0/2729343650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae200b0d10 msgr2=0x7fae20156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25938700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae200b0d10 0x7fae20156230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25938700 1 -- 172.21.15.3:0/2729343650 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c00b180 con 0x7fae200a9320 2022-07-21T01:35:50.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae25938700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 0x7fae2013f1a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fae1c002f70 tx=0x7fae1c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:50.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae0effd700 1 -- 172.21.15.3:0/2729343650 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae1c013e90 con 0x7fae200a9320 2022-07-21T01:35:50.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae0effd700 1 -- 172.21.15.3:0/2729343650 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1c00de60 con 0x7fae200a9320 2022-07-21T01:35:50.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae20155d90 con 0x7fae200a9320 2022-07-21T01:35:50.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae0effd700 1 -- 172.21.15.3:0/2729343650 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae1c01e5f0 con 0x7fae200a9320 2022-07-21T01:35:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.605+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2014fdc0 con 0x7fae200a9320 2022-07-21T01:35:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.609+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae200574e0 con 0x7fae200a9320 2022-07-21T01:35:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.609+0000 7fae0effd700 1 -- 172.21.15.3:0/2729343650 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fae1c025030 con 0x7fae200a9320 2022-07-21T01:35:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.609+0000 7fae0effd700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae00053b00 0x7fae00055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:35:50.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.609+0000 7fae25137700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae00053b00 0x7fae00055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:35:50.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.609+0000 7fae0effd700 1 -- 172.21.15.3:0/2729343650 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fae1c06cd20 con 0x7fae200a9320 2022-07-21T01:35:50.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.609+0000 7fae25137700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae00053b00 0x7fae00055fc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fae1000b000 tx=0x7fae10006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:35:50.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.617+0000 7fae0effd700 1 -- 172.21.15.3:0/2729343650 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fae1c043a10 con 0x7fae200a9320 2022-07-21T01:35:50.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae200aca20 con 0x7fae200a9320 2022-07-21T01:35:50.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae0effd700 1 -- 172.21.15.3:0/2729343650 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2488 (secure 0 0 0) 0x7fae1c023070 con 0x7fae200a9320 2022-07-21T01:35:50.792 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:35:50.792 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:35:18.753732+0000","modified":"2022-07-21T01:35:19.758646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/2892507714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2892507714},{"type":"v1","addr":"172.21.15.3:6837","nonce":2892507714}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-21T01:35:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae00053b00 msgr2=0x7fae00055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae00053b00 0x7fae00055fc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fae1000b000 tx=0x7fae10006df0).stop 2022-07-21T01:35:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 msgr2=0x7fae2013f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:35:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 0x7fae2013f1a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fae1c002f70 tx=0x7fae1c014000).stop 2022-07-21T01:35:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 shutdown_connections 2022-07-21T01:35:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae00053b00 0x7fae00055fc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae200b0d10 0x7fae20156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae200b0560 0x7fae200ab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 --2- 172.21.15.3:0/2729343650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae200a9320 0x7fae2013f1a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:35:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 >> 172.21.15.3:0/2729343650 conn(0x7fae200612d0 msgr2=0x7fae2013ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:35:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 shutdown_connections 2022-07-21T01:35:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:35:50.781+0000 7fae2693a700 1 -- 172.21.15.3:0/2729343650 wait complete. 2022-07-21T01:35:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 18 2022-07-21T01:35:51.100 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:35:18.753732+0000', 'modified': '2022-07-21T01:35:19.758646+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'a', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/2892507714', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 2892507714}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 2892507714}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:35:51.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:35:51.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:35:51.221 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":5241,"want_state":"up:active","state":"up:active","rank_uptime":32.455550477000003,"mdsmap_epoch":18,"osdmap_epoch":48,"osdmap_epoch_barrier":47,"uptime":36.808968268000001} 2022-07-21T01:35:51.222 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 5241, "mdsmap_epoch": 18, "osdmap_epoch": 48, "osdmap_epoch_barrier": 47, "rank_uptime": 32.455550477, "state": "up:active", "uptime": 36.808968268, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:35:51.222 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:35:51.223 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:35:51.223 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:35:51.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:51.225 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:51.233 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:35:51.249 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:35:51.249 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:35:51.303 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:35:51.304 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:35:51.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:51.307 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:35:51.313 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:35:51.313 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:51.356 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:35:51.374 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:35:51.375 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:35:51.388 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:35:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:35:51.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:51.391 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:35:51.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:35:51.441+0000 7f71eadfe080 -1 init, newargv = 0x562b42fac900 newargc=9 2022-07-21T01:35:51.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22413]: starting ceph client 2022-07-21T01:35:51.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22413]: starting fuse 2022-07-21T01:35:52.543 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:35:52.562 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:35:52.563 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:35:52.621 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:35:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:35:52.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:52.623 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:35:52.628 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:35:52.629 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:35:52.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:52.630 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:35:52.630 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:35:52.631 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:35:52.631 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:35:52.632 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:35:52.632 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:52.632 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:35:52.633 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:35:52.633 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:35:52.634 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:52.634 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:35:52.635 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:35:52.635 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:35:52.636 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:35:52.636 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:52.636 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:35:52.637 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:35:52.637 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:35:52.638 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:35:52.638 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:35:52.639 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:35:52.639 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:35:52.640 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:35:52.681 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22413.asok 2022-07-21T01:35:52.686 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22413.asok 2022-07-21T01:35:52.686 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:52.687 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22413.asok status 2022-07-21T01:35:52.774 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:35:52.775 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:35:52.775 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:35:52.775 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:35:52.775 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:35:52.776 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:35:52.776 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:35:52.776 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22413", 2022-07-21T01:35:52.776 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:35:52.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:35:52.777 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:35:52.778 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:35:52.778 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 5475, 2022-07-21T01:35:52.778 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:35:52.778 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:35:52.779 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:35:52.779 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 5475 2022-07-21T01:35:52.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:35:52.779 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:35:52.780 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:35:52.780 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:35:52.780 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 4267700484 2022-07-21T01:35:52.780 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:35:52.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:35:52.781 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:35:52.781 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:35:52.782 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:35:52.782 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 4267700484 2022-07-21T01:35:52.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:35:52.782 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.5475 172.21.15.171:0/4267700484", 2022-07-21T01:35:52.783 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/4267700484", 2022-07-21T01:35:52.783 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:35:52.783 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 18, 2022-07-21T01:35:52.783 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 48, 2022-07-21T01:35:52.784 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:35:52.784 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:35:52.784 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:35:52.784 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:35:52.786 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:52.787 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:35:52.787 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:35:52.806 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:35:52.806 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:35:52.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:35:52.807 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:01.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:36:01.549 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:36:01.552 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:36:01.555 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:36:31.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:36:31.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:36:31.629 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:36:31.633 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:36:39.410 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:36:39.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:36:39.429 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:36:39.452 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:36:39.474 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:36:39.496 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:36:39.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:39.497 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:36:39.497 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:39.506 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:36:39.507 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:36:39.508 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:36:39.508 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:36:39.509 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:39.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22413]: fuse finished with error 0 and tester_r 0 2022-07-21T01:36:39.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:39.568 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:36:39.569 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:39.576 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:36:39.578 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:36:39.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:36:39.696 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:39.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:39.745 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:36:39.746 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:36:39.754 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:36:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:36:39.757 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:36:39.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:36:39.759 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:36:39.766 INFO:teuthology.orchestra.run.smithi174.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:36:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:36:39.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:36:39.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.885+0000 7f4165db9700 1 Processor -- start 2022-07-21T01:36:39.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.885+0000 7f4165db9700 1 -- start start 2022-07-21T01:36:39.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f416005c960 0x7f41600b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:39.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416013f520 0x7f41600aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:39.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416005cd40 con 0x7f416005d240 2022-07-21T01:36:39.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41600ad6d0 con 0x7f416005c960 2022-07-21T01:36:39.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4160067f00 con 0x7f416013f520 2022-07-21T01:36:39.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4164db7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f416005c960 0x7f41600b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:39.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4164db7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f416005c960 0x7f41600b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45060/0 (socket says 172.21.15.3:45060) 2022-07-21T01:36:39.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4164db7700 1 -- 172.21.15.3:0/1325908937 learned_addr learned my addr 172.21.15.3:0/1325908937 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:39.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f41655b8700 1 --2- 172.21.15.3:0/1325908937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416013f520 0x7f41600aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:39.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4164db7700 1 -- 172.21.15.3:0/1325908937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416013f520 msgr2=0x7f41600aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:39.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4164db7700 1 --2- 172.21.15.3:0/1325908937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416013f520 0x7f41600aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:39.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4164db7700 1 -- 172.21.15.3:0/1325908937 >> v1:172.21.15.143:6789/0 conn(0x7f416005d240 legacy=0x7f41600acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:39.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4164db7700 1 -- 172.21.15.3:0/1325908937 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41600b1920 con 0x7f416005c960 2022-07-21T01:36:39.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4164db7700 1 --2- 172.21.15.3:0/1325908937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 0x7f41600b1280 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f4154002f70 tx=0x7f4154006f40).ready entity=mon.0 client_cookie=676f6a78e9b26452 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:39.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f415f7fe700 1 -- 172.21.15.3:0/1325908937 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f415400da80 con 0x7f416005c960 2022-07-21T01:36:39.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f415f7fe700 1 -- 172.21.15.3:0/1325908937 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4154000e90 con 0x7f416005c960 2022-07-21T01:36:39.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f415f7fe700 1 -- 172.21.15.3:0/1325908937 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f415400de10 con 0x7f416005c960 2022-07-21T01:36:39.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 -- 172.21.15.3:0/1325908937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 msgr2=0x7f41600b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:39.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 --2- 172.21.15.3:0/1325908937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 0x7f41600b1280 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f4154002f70 tx=0x7f4154006f40).stop 2022-07-21T01:36:39.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 -- 172.21.15.3:0/1325908937 shutdown_connections 2022-07-21T01:36:39.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 --2- 172.21.15.3:0/1325908937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416013f520 0x7f41600aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:39.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 --2- 172.21.15.3:0/1325908937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 0x7f41600b1280 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:39.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 -- 172.21.15.3:0/1325908937 >> 172.21.15.3:0/1325908937 conn(0x7f4160060310 msgr2=0x7f41600b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:39.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 -- 172.21.15.3:0/1325908937 shutdown_connections 2022-07-21T01:36:39.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.889+0000 7f4165db9700 1 -- 172.21.15.3:0/1325908937 wait complete. 2022-07-21T01:36:39.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 Processor -- start 2022-07-21T01:36:39.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 -- start start 2022-07-21T01:36:39.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 0x7f41601423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:39.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416005d240 0x7f416014ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:39.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f416013f520 0x7f41600ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:39.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416014e120 con 0x7f416005c960 2022-07-21T01:36:39.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41600abae0 con 0x7f416013f520 2022-07-21T01:36:39.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416014e310 con 0x7f416005d240 2022-07-21T01:36:39.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4164db7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 0x7f41601423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:39.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4164db7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 0x7f41601423d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45062/0 (socket says 172.21.15.3:45062) 2022-07-21T01:36:39.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4164db7700 1 -- 172.21.15.3:0/2216092252 learned_addr learned my addr 172.21.15.3:0/2216092252 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:39.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f415ffff700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416005d240 0x7f416014ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:39.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f41655b8700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f416013f520 0x7f41600ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:39.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4164db7700 1 -- 172.21.15.3:0/2216092252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416005d240 msgr2=0x7f416014ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:39.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4164db7700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416005d240 0x7f416014ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:39.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4164db7700 1 -- 172.21.15.3:0/2216092252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f416013f520 msgr2=0x7f41600ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:39.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4164db7700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f416013f520 0x7f41600ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:39.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4164db7700 1 -- 172.21.15.3:0/2216092252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415400b460 con 0x7f416005c960 2022-07-21T01:36:39.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f415ffff700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416005d240 0x7f416014ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:39.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4164db7700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 0x7f41601423d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f41600b4fa0 tx=0x7f416005d6b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:39.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f415dffb700 1 -- 172.21.15.3:0/2216092252 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4154010e80 con 0x7f416005c960 2022-07-21T01:36:39.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f415dffb700 1 -- 172.21.15.3:0/2216092252 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415400dde0 con 0x7f416005c960 2022-07-21T01:36:39.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416014e5b0 con 0x7f416005c960 2022-07-21T01:36:39.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f415dffb700 1 -- 172.21.15.3:0/2216092252 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4154018800 con 0x7f416005c960 2022-07-21T01:36:39.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.893+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41600aafe0 con 0x7f416005c960 2022-07-21T01:36:39.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.897+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41600b1bc0 con 0x7f416005c960 2022-07-21T01:36:39.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.897+0000 7f415dffb700 1 -- 172.21.15.3:0/2216092252 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4154021020 con 0x7f416005c960 2022-07-21T01:36:39.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.897+0000 7f415dffb700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4140053980 0x7f4140055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:39.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.897+0000 7f415dffb700 1 -- 172.21.15.3:0/2216092252 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f415406fe20 con 0x7f416005c960 2022-07-21T01:36:39.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.897+0000 7f415ffff700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4140053980 0x7f4140055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:39.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.897+0000 7f415ffff700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4140053980 0x7f4140055e40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f415000b000 tx=0x7f4150009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:39.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:39.901+0000 7f415dffb700 1 -- 172.21.15.3:0/2216092252 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4154046a50 con 0x7f416005c960 2022-07-21T01:36:40.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.029+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f41600574e0 con 0x7f416005c960 2022-07-21T01:36:40.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f415dffb700 1 -- 172.21.15.3:0/2216092252 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v48) v1 ==== 81+0+34037 (secure 0 0 0) 0x7f415404a070 con 0x7f416005c960 2022-07-21T01:36:40.034 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:40.035 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:36:40.035 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 48, 2022-07-21T01:36:40.036 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:36:40.036 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:36:40.037 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:35:20.769681+0000", 2022-07-21T01:36:40.037 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:36:40.038 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:36:40.039 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:36:40.039 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:36:40.039 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:36:40.040 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:36:40.040 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:36:40.041 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:36:40.041 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:36:40.042 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:36:40.042 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:36:40.043 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:36:40.043 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:36:40.044 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:36:40.044 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:36:40.045 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 8, 2022-07-21T01:36:40.046 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:36:40.046 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:36:40.047 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:36:40.047 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:36:40.047 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:36:40.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.048 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:36:40.049 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:36:40.049 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:36:40.050 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:36:40.050 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:36:40.051 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:36:40.051 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:36:40.051 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:36:40.052 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:36:40.052 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:36:40.053 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:36:40.053 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:36:40.054 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:36:40.054 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:36:40.055 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:36:40.055 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:36:40.056 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:36:40.056 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:36:40.056 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:36:40.057 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:36:40.057 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:36:40.058 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:36:40.058 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:36:40.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.059 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:36:40.060 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:36:40.060 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:36:40.060 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:36:40.061 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:36:40.061 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:36:40.062 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:36:40.062 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:36:40.063 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:36:40.063 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:36:40.063 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:36:40.064 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:36:40.064 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:36:40.065 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:36:40.065 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:36:40.066 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:36:40.066 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:36:40.066 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:36:40.067 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:36:40.067 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:36:40.068 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:36:40.068 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:36:40.069 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:36:40.069 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:36:40.070 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:36:40.070 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:36:40.070 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:36:40.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.071 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:36:40.072 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:36:40.072 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:36:40.073 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:36:40.073 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:36:40.073 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:36:40.074 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:36:40.074 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:36:40.075 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:36:40.075 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:36:40.076 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:36:40.076 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:36:40.076 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:36:40.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.077 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:36:40.078 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:36:40.078 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.080 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:36:40.080 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:36:40.080 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:36:40.081 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:36:40.081 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:36:40.082 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:36:40.082 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:36:40.083 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:36:40.083 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:36:40.083 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:36:40.084 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:36:40.084 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:36:40.085 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:36:40.085 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:36:40.086 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:36:40.086 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:36:40.087 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:36:40.087 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:36:40.087 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:36:40.088 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:36:40.088 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:36:40.089 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:36:40.089 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:36:40.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.090 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:36:40.091 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:36:40.091 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:36:40.091 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:36:40.092 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:36:40.092 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:36:40.093 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:36:40.093 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:36:40.094 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:36:40.094 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:36:40.094 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:36:40.095 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:36:40.095 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:36:40.096 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:36:40.096 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:36:40.097 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:36:40.097 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:36:40.097 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:36:40.098 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:36:40.098 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:36:40.099 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:36:40.099 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:36:40.100 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:36:40.100 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:36:40.100 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:36:40.101 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:36:40.101 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:36:40.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.102 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:36:40.103 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:36:40.103 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:36:40.103 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:36:40.104 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:36:40.104 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:36:40.105 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:36:40.105 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:36:40.106 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:36:40.106 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:36:40.106 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:36:40.107 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:36:40.107 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:36:40.108 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:36:40.108 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.109 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 7, 2022-07-21T01:36:40.110 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:36:40.110 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:35:17.190459+0000", 2022-07-21T01:36:40.111 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:36:40.111 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:36:40.111 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:36:40.112 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:36:40.112 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:36:40.112 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:36:40.112 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:36:40.113 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:36:40.113 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:36:40.113 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:36:40.113 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:36:40.113 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:36:40.114 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:36:40.114 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:36:40.114 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:36:40.115 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:36:40.115 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:36:40.115 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:36:40.115 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:36:40.116 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:36:40.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.116 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "47", 2022-07-21T01:36:40.117 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:36:40.117 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:36:40.117 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:36:40.117 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:36:40.118 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:36:40.118 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:36:40.118 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:36:40.118 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:36:40.119 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:36:40.119 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:36:40.119 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:36:40.119 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:36:40.120 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:36:40.120 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:36:40.120 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:36:40.120 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:36:40.121 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:36:40.121 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:36:40.121 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:36:40.122 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:36:40.122 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:36:40.122 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:36:40.122 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:36:40.123 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:36:40.123 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:36:40.123 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:36:40.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.124 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:36:40.124 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:36:40.124 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:36:40.124 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:36:40.125 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:36:40.125 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:36:40.125 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:36:40.125 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:36:40.126 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:36:40.126 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:36:40.126 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:36:40.126 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:36:40.127 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:36:40.127 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:36:40.127 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:36:40.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.128 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:36:40.128 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:36:40.128 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:36:40.129 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.129 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.130 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 8, 2022-07-21T01:36:40.130 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:36:40.130 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:35:18.336558+0000", 2022-07-21T01:36:40.131 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:36:40.131 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:36:40.131 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:36:40.131 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:36:40.132 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:36:40.132 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:36:40.132 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:36:40.132 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:36:40.133 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:36:40.133 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:36:40.133 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:36:40.134 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:36:40.134 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:36:40.134 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:36:40.135 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:36:40.135 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:36:40.135 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:36:40.135 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:36:40.136 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:36:40.136 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:36:40.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.136 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "48", 2022-07-21T01:36:40.137 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:36:40.137 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:36:40.137 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:36:40.137 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:36:40.138 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:36:40.138 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:36:40.138 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:36:40.138 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:36:40.139 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:36:40.139 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:36:40.139 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:36:40.139 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:36:40.140 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:36:40.140 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:36:40.140 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:36:40.140 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:36:40.141 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:36:40.141 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:36:40.141 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:36:40.141 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:36:40.141 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:36:40.142 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:36:40.142 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:36:40.142 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:36:40.142 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:36:40.143 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:36:40.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.143 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:36:40.143 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:36:40.143 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:36:40.144 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:36:40.144 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:36:40.144 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:36:40.144 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:36:40.145 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:36:40.145 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:36:40.145 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:36:40.146 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:36:40.146 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:36:40.146 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:36:40.146 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:36:40.146 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:36:40.147 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.147 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.147 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.147 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:36:40.148 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:36:40.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.148 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:36:40.148 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:36:40.148 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:40.149 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:40.149 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:40.149 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:40.149 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:40.150 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:40.150 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:36:40.150 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:40.150 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:40.151 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:40.151 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:40.151 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.151 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.152 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:36:40.152 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:40.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.153 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.153 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:36:40.153 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:40.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.153 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.154 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:40.154 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.155 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.155 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:36:40.155 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:40.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.156 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.156 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:36:40.156 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:40.156 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:40.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:40.159 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.160 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.160 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:40.160 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:40.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:36:40.162 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:36:40.163 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:40.164 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:40.165 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:40.165 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.166 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:36:40.166 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:36:40.166 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:40.166 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:40.166 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:40.167 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:40.167 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:40.167 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:40.167 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:40.168 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 46, 2022-07-21T01:36:40.168 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:40.168 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:40.168 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:40.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.169 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:36:40.169 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:40.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.170 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.170 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:36:40.171 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:40.171 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.171 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.171 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:40.172 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.172 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.172 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:36:40.173 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:40.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.173 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.173 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:36:40.174 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:40.174 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.174 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.175 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:40.175 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.175 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.175 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:36:40.176 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:40.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.176 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.177 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:36:40.177 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:40.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.178 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:40.178 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.179 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:36:40.179 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:40.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.180 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:36:40.180 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:40.180 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.180 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.181 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:36:40.181 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:36:40.181 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:36:40.182 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:36:40.182 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:40.182 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:40.182 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:40.182 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.183 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:36:40.183 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:36:40.184 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:40.184 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:40.184 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:40.184 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:40.184 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:40.185 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:40.185 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:36:40.185 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 34, 2022-07-21T01:36:40.185 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:40.186 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:40.186 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:40.186 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.186 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.187 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:36:40.187 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:40.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.188 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.188 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:36:40.188 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:40.188 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.188 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.189 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:40.189 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.190 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.190 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:36:40.190 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:40.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.191 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:36:40.191 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:40.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.192 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.192 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:40.193 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.193 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.193 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:36:40.194 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:40.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.194 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.194 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:36:40.195 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:40.195 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.195 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.196 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:40.196 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.196 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.196 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:36:40.197 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:40.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.198 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:36:40.198 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:40.198 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.198 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.199 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:36:40.199 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:36:40.199 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:36:40.199 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:36:40.200 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:40.200 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:40.200 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:40.200 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.201 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:36:40.201 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:36:40.201 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:40.202 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:40.202 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:40.202 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:40.202 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:40.203 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:40.203 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:36:40.203 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:40.203 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:40.203 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:40.204 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:40.204 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.204 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.204 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:36:40.205 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:40.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.205 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.206 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:36:40.206 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:40.206 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.206 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.207 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:40.207 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.207 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.208 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:36:40.208 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:40.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.208 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.209 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:36:40.209 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:40.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.209 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.210 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:40.210 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.211 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.211 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:36:40.211 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:40.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.212 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.212 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:36:40.212 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:40.212 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.212 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.213 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:40.213 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.214 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:36:40.214 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:40.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.215 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.215 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:36:40.215 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:40.216 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.216 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.216 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:36:40.217 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:36:40.217 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:36:40.217 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:36:40.217 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:40.217 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:40.218 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:40.218 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.219 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:36:40.219 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:36:40.219 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:40.219 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:40.219 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:40.220 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:40.220 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:40.220 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:40.220 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:40.221 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:40.221 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:40.221 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:40.221 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:40.221 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.222 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.222 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:36:40.222 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:40.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.223 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.223 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:36:40.223 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:40.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.224 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.224 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:40.225 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.225 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.225 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:36:40.225 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:40.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.226 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.226 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:36:40.227 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:40.227 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.227 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.228 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:40.228 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.228 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.228 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:36:40.229 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:40.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.229 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.230 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:36:40.230 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:40.230 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.230 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.231 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:40.231 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.231 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.232 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:36:40.232 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:40.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.232 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.233 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:36:40.233 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:40.233 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.233 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.234 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:36:40.234 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:36:40.234 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:36:40.235 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:36:40.235 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:40.235 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:40.235 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:40.235 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.236 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:36:40.236 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:36:40.237 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:40.237 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:40.237 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:40.237 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:40.238 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:40.238 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:40.238 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:40.238 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:40.238 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:40.239 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:40.239 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:40.239 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.240 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.240 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:36:40.240 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:40.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.241 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.241 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:36:40.241 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:40.241 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.242 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.242 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:40.242 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.243 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.243 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:36:40.243 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:40.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.244 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.245 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:36:40.245 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:40.245 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.245 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.246 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:40.246 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.247 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.247 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:36:40.247 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:40.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.248 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.248 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:36:40.249 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:40.249 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.249 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.250 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:40.250 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.251 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.251 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:36:40.251 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:40.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.252 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.252 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:36:40.253 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:40.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.253 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.254 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:36:40.255 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:36:40.255 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:36:40.255 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:36:40.255 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:40.256 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:40.256 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:40.256 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.257 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:36:40.257 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:36:40.258 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:40.258 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:40.258 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:40.258 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:40.259 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:40.259 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:40.259 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:40.259 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:40.260 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:40.260 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:40.260 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:40.260 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.261 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.261 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:36:40.261 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:40.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.262 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.262 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:36:40.262 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:40.263 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.263 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.263 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:40.264 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.264 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.264 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:36:40.264 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:40.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.265 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.265 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:36:40.266 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:40.266 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.266 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.267 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:40.267 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.268 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.268 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:36:40.268 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:40.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.269 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.270 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:36:40.270 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:40.270 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.270 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.271 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:40.271 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.272 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.272 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:36:40.273 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:40.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.274 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.274 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:36:40.274 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:40.274 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.275 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.275 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:36:40.275 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:36:40.276 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:36:40.276 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:36:40.276 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:40.276 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:40.277 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:40.277 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.278 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:36:40.278 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:36:40.278 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:40.278 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:40.279 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:40.279 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:40.279 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:40.279 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:40.280 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:40.280 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 44, 2022-07-21T01:36:40.280 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:40.280 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:40.280 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:40.281 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.281 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.282 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:36:40.282 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:40.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.283 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.283 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:36:40.283 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:40.284 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.284 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.284 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:40.285 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.285 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.285 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:36:40.285 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:40.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.286 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.286 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:36:40.287 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:40.287 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.287 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.288 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:40.288 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.288 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.288 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:36:40.289 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:40.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.289 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.290 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:36:40.290 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:40.290 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.290 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.291 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:40.291 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:40.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.291 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:40.292 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:36:40.292 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:40.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.292 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:40.293 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:36:40.293 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:40.293 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.293 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.294 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:36:40.294 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:36:40.294 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:36:40.295 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:36:40.295 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:40.295 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:40.295 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:40.295 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:40.296 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.296 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:36:40.296 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:36:40.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.297 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:36:40.297 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:40.297 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:40.297 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:40.297 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:40.298 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:40.298 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:36:40.298 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:40.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.299 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:36:40.299 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:40.299 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:40.300 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:40.300 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:40.300 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:40.300 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:36:40.301 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:40.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.301 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:36:40.302 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:40.302 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:40.302 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:40.302 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:40.303 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:40.303 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:36:40.303 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:40.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.304 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:36:40.304 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:40.304 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:40.304 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:40.305 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:40.305 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:40.305 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:36:40.305 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:40.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.306 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:36:40.306 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:40.307 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:40.307 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:40.307 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:40.307 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:40.308 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:36:40.308 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:40.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.308 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:36:40.309 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:40.309 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:40.309 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:40.309 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:40.310 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:40.310 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:36:40.310 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:40.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.311 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:36:40.311 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:40.311 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:40.311 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:40.312 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:40.312 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:40.312 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:36:40.312 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:40.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:40.313 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:36:40.313 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:40.313 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:40.314 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:40.314 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:40.314 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:40.314 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:36:40.315 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:40.315 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.315 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:36:40.315 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:36:40.315 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:36:40.316 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:36:40.316 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:36:40.316 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:36:40.316 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:36:40.317 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:36:40.317 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:36:40.317 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:36:40.317 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:36:40.317 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:36:40.318 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:36:40.318 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:36:40.318 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:36:40.318 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:36:40.319 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:36:40.319 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:36:40.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.319 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:36:40.320 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:36:40.320 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:36:40.320 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:36:40.320 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:36:40.320 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:36:40.321 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:36:40.321 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:40.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:40.321 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:36:40.322 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:36:40.322 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:36:40.322 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:36:40.322 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:36:40.323 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:36:40.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4140053980 msgr2=0x7f4140055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4140053980 0x7f4140055e40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f415000b000 tx=0x7f4150009040).stop 2022-07-21T01:36:40.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 msgr2=0x7f41601423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 0x7f41601423d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f41600b4fa0 tx=0x7f416005d6b0).stop 2022-07-21T01:36:40.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 shutdown_connections 2022-07-21T01:36:40.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4140053980 0x7f4140055e40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f416013f520 0x7f41600ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f416005d240 0x7f416014ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 --2- 172.21.15.3:0/2216092252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f416005c960 0x7f41601423d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 >> 172.21.15.3:0/2216092252 conn(0x7f4160060310 msgr2=0x7f416013f970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:40.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 shutdown_connections 2022-07-21T01:36:40.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.033+0000 7f4165db9700 1 -- 172.21.15.3:0/2216092252 wait complete. 2022-07-21T01:36:40.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:36:40.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9a74700 1 Processor -- start 2022-07-21T01:36:40.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9a74700 1 -- start start 2022-07-21T01:36:40.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9a74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40614b0 0x7f82c4061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9a74700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82c40ad750 0x7f82c40b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9a74700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f82c41405d0 0x7f82c4140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9a74700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c40aee80 con 0x7f82c41405d0 2022-07-21T01:36:40.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9a74700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c40b70b0 con 0x7f82c40ad750 2022-07-21T01:36:40.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9a74700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40af0b0 con 0x7f82c40614b0 2022-07-21T01:36:40.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9273700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f82c41405d0 0x7f82c4140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9273700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f82c41405d0 0x7f82c4140a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45074/0 (socket says 172.21.15.3:45074) 2022-07-21T01:36:40.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.437+0000 7f82c9273700 1 -- 172.21.15.3:0/1947820417 learned_addr learned my addr 172.21.15.3:0/1947820417 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:40.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c8a72700 1 --2- 172.21.15.3:0/1947820417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40614b0 0x7f82c4061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c3fff700 1 --2- 172.21.15.3:0/1947820417 >> v2:172.21.15.143:3300/0 conn(0x7f82c40ad750 0x7f82c40b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9273700 1 -- 172.21.15.3:0/1947820417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40614b0 msgr2=0x7f82c4061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:40.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9273700 1 --2- 172.21.15.3:0/1947820417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40614b0 0x7f82c4061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9273700 1 -- 172.21.15.3:0/1947820417 >> v2:172.21.15.143:3300/0 conn(0x7f82c40ad750 msgr2=0x7f82c40b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:40.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9273700 1 --2- 172.21.15.3:0/1947820417 >> v2:172.21.15.143:3300/0 conn(0x7f82c40ad750 0x7f82c40b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9273700 1 -- 172.21.15.3:0/1947820417 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c40af350 con 0x7f82c41405d0 2022-07-21T01:36:40.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c3fff700 1 --2- 172.21.15.3:0/1947820417 >> v2:172.21.15.143:3300/0 conn(0x7f82c40ad750 0x7f82c40b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:40.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c8a72700 1 --2- 172.21.15.3:0/1947820417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40614b0 0x7f82c4061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:40.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9273700 1 --2- 172.21.15.3:0/1947820417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 0x7f82c4140a80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f82bc0032f0 tx=0x7f82bc009290).ready entity=mon.0 client_cookie=6126cb436f3bb643 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:40.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c37fe700 1 -- 172.21.15.3:0/1947820417 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82bc00b870 con 0x7f82c41405d0 2022-07-21T01:36:40.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c37fe700 1 -- 172.21.15.3:0/1947820417 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc00b9d0 con 0x7f82c41405d0 2022-07-21T01:36:40.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c37fe700 1 -- 172.21.15.3:0/1947820417 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82bc00bb40 con 0x7f82c41405d0 2022-07-21T01:36:40.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 -- 172.21.15.3:0/1947820417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 msgr2=0x7f82c4140a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 --2- 172.21.15.3:0/1947820417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 0x7f82c4140a80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f82bc0032f0 tx=0x7f82bc009290).stop 2022-07-21T01:36:40.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 -- 172.21.15.3:0/1947820417 shutdown_connections 2022-07-21T01:36:40.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 --2- 172.21.15.3:0/1947820417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 0x7f82c4140a80 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 --2- 172.21.15.3:0/1947820417 >> v2:172.21.15.143:3300/0 conn(0x7f82c40ad750 0x7f82c40b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 --2- 172.21.15.3:0/1947820417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40614b0 0x7f82c4061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 -- 172.21.15.3:0/1947820417 >> 172.21.15.3:0/1947820417 conn(0x7f82c4066f70 msgr2=0x7f82c4067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:40.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 -- 172.21.15.3:0/1947820417 shutdown_connections 2022-07-21T01:36:40.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 -- 172.21.15.3:0/1947820417 wait complete. 2022-07-21T01:36:40.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 Processor -- start 2022-07-21T01:36:40.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 -- start start 2022-07-21T01:36:40.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82c40614b0 0x7f82c41522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40ad750 0x7f82c4141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 0x7f82c409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c41563a0 con 0x7f82c41405d0 2022-07-21T01:36:40.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c4156640 con 0x7f82c40614b0 2022-07-21T01:36:40.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.441+0000 7f82c9a74700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40b0880 con 0x7f82c40ad750 2022-07-21T01:36:40.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9273700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 0x7f82c409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9273700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 0x7f82c409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45080/0 (socket says 172.21.15.3:45080) 2022-07-21T01:36:40.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9273700 1 -- 172.21.15.3:0/3325595104 learned_addr learned my addr 172.21.15.3:0/3325595104 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:40.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c3fff700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40ad750 0x7f82c4141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c8a72700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82c40614b0 0x7f82c41522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9273700 1 -- 172.21.15.3:0/3325595104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40ad750 msgr2=0x7f82c4141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9273700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40ad750 0x7f82c4141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9273700 1 -- 172.21.15.3:0/3325595104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82c40614b0 msgr2=0x7f82c41522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9273700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82c40614b0 0x7f82c41522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9273700 1 -- 172.21.15.3:0/3325595104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc009530 con 0x7f82c41405d0 2022-07-21T01:36:40.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9273700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 0x7f82c409ea40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f82bc0032f0 tx=0x7f82bc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:40.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c1ffb700 1 -- 172.21.15.3:0/3325595104 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82bc00b870 con 0x7f82c41405d0 2022-07-21T01:36:40.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c1ffb700 1 -- 172.21.15.3:0/3325595104 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc020420 con 0x7f82c41405d0 2022-07-21T01:36:40.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c40b0b20 con 0x7f82c41405d0 2022-07-21T01:36:40.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c1ffb700 1 -- 172.21.15.3:0/3325595104 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82bc020780 con 0x7f82c41405d0 2022-07-21T01:36:40.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c40b0260 con 0x7f82c41405d0 2022-07-21T01:36:40.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c40b0cb0 con 0x7f82c41405d0 2022-07-21T01:36:40.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c1ffb700 1 -- 172.21.15.3:0/3325595104 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f82bc027070 con 0x7f82c41405d0 2022-07-21T01:36:40.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c1ffb700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82a4053950 0x7f82a4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c1ffb700 1 -- 172.21.15.3:0/3325595104 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f82bc06c7c0 con 0x7f82c41405d0 2022-07-21T01:36:40.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.445+0000 7f82c8a72700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82a4053950 0x7f82a4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.449+0000 7f82c8a72700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82a4053950 0x7f82a4055e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f82b800c000 tx=0x7f82b8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:40.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.453+0000 7f82c1ffb700 1 -- 172.21.15.3:0/3325595104 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f82bc043580 con 0x7f82c41405d0 2022-07-21T01:36:40.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.613+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82c40af5f0 con 0x7f82c41405d0 2022-07-21T01:36:40.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.613+0000 7f82c1ffb700 1 -- 172.21.15.3:0/3325595104 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2488 (secure 0 0 0) 0x7f82bc015020 con 0x7f82c41405d0 2022-07-21T01:36:40.615 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:40.615 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:35:18.753732+0000","modified":"2022-07-21T01:35:19.758646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/2892507714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2892507714},{"type":"v1","addr":"172.21.15.3:6837","nonce":2892507714}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-21T01:36:40.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82a4053950 msgr2=0x7f82a4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82a4053950 0x7f82a4055e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f82b800c000 tx=0x7f82b8008040).stop 2022-07-21T01:36:40.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 msgr2=0x7f82c409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 0x7f82c409ea40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f82bc0032f0 tx=0x7f82bc016000).stop 2022-07-21T01:36:40.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 shutdown_connections 2022-07-21T01:36:40.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c41405d0 0x7f82c409ea40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82c40ad750 0x7f82c4141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82a4053950 0x7f82a4055e10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 --2- 172.21.15.3:0/3325595104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82c40614b0 0x7f82c41522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 >> 172.21.15.3:0/3325595104 conn(0x7f82c4066f70 msgr2=0x7f82c41467e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:40.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 shutdown_connections 2022-07-21T01:36:40.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.617+0000 7f82c9a74700 1 -- 172.21.15.3:0/3325595104 wait complete. 2022-07-21T01:36:40.620 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 18 2022-07-21T01:36:40.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:36:40.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 Processor -- start 2022-07-21T01:36:40.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 -- start start 2022-07-21T01:36:40.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f36080619b0 0x7f36080b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080b6ee0 0x7f3608143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f36080b5ed0 0x7f36080b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3608142f60 con 0x7f36080619b0 2022-07-21T01:36:40.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3608061d90 con 0x7f36080b5ed0 2022-07-21T01:36:40.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36080ba5a0 con 0x7f36080b6ee0 2022-07-21T01:36:40.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e159700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f36080619b0 0x7f36080b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e159700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f36080619b0 0x7f36080b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45084/0 (socket says 172.21.15.3:45084) 2022-07-21T01:36:40.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e159700 1 -- 172.21.15.3:0/1723697771 learned_addr learned my addr 172.21.15.3:0/1723697771 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:40.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360d958700 1 --2- 172.21.15.3:0/1723697771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080b6ee0 0x7f3608143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e95a700 1 --2- 172.21.15.3:0/1723697771 >> v2:172.21.15.143:3300/0 conn(0x7f36080b5ed0 0x7f36080b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e159700 1 -- 172.21.15.3:0/1723697771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080b6ee0 msgr2=0x7f3608143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:40.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e159700 1 --2- 172.21.15.3:0/1723697771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080b6ee0 0x7f3608143890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e159700 1 -- 172.21.15.3:0/1723697771 >> v2:172.21.15.143:3300/0 conn(0x7f36080b5ed0 msgr2=0x7f36080b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:40.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e159700 1 --2- 172.21.15.3:0/1723697771 >> v2:172.21.15.143:3300/0 conn(0x7f36080b5ed0 0x7f36080b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e159700 1 -- 172.21.15.3:0/1723697771 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3608143150 con 0x7f36080619b0 2022-07-21T01:36:40.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360e159700 1 --2- 172.21.15.3:0/1723697771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080619b0 0x7f36080b48a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f3604002f70 tx=0x7f3604006eb0).ready entity=mon.0 client_cookie=9ebcef6d1124ac86 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:40.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360d157700 1 -- 172.21.15.3:0/1723697771 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f360400d570 con 0x7f36080619b0 2022-07-21T01:36:40.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360d157700 1 -- 172.21.15.3:0/1723697771 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3604000e90 con 0x7f36080619b0 2022-07-21T01:36:40.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360d157700 1 -- 172.21.15.3:0/1723697771 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f360400d900 con 0x7f36080619b0 2022-07-21T01:36:40.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 -- 172.21.15.3:0/1723697771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080619b0 msgr2=0x7f36080b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 --2- 172.21.15.3:0/1723697771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080619b0 0x7f36080b48a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f3604002f70 tx=0x7f3604006eb0).stop 2022-07-21T01:36:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 -- 172.21.15.3:0/1723697771 shutdown_connections 2022-07-21T01:36:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 --2- 172.21.15.3:0/1723697771 >> v2:172.21.15.143:3300/0 conn(0x7f36080b5ed0 0x7f36080b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 --2- 172.21.15.3:0/1723697771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080b6ee0 0x7f3608143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 --2- 172.21.15.3:0/1723697771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080619b0 0x7f36080b48a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 -- 172.21.15.3:0/1723697771 >> 172.21.15.3:0/1723697771 conn(0x7f360806af80 msgr2=0x7f360805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:40.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.729+0000 7f360f15b700 1 -- 172.21.15.3:0/1723697771 shutdown_connections 2022-07-21T01:36:40.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360f15b700 1 -- 172.21.15.3:0/1723697771 wait complete. 2022-07-21T01:36:40.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360f15b700 1 Processor -- start 2022-07-21T01:36:40.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360f15b700 1 -- start start 2022-07-21T01:36:40.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360f15b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080619b0 0x7f36080b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360f15b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36080b5ed0 0x7f3608144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360f15b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080b6ee0 0x7f3608152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360f15b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3608152510 con 0x7f36080b6ee0 2022-07-21T01:36:40.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360f15b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3608144c50 con 0x7f36080b5ed0 2022-07-21T01:36:40.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360f15b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3608144530 con 0x7f36080619b0 2022-07-21T01:36:40.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e95a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080b6ee0 0x7f3608152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e95a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080b6ee0 0x7f3608152e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45094/0 (socket says 172.21.15.3:45094) 2022-07-21T01:36:40.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e95a700 1 -- 172.21.15.3:0/2768612744 learned_addr learned my addr 172.21.15.3:0/2768612744 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:40.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360d958700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36080b5ed0 0x7f3608144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e159700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080619b0 0x7f36080b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e95a700 1 -- 172.21.15.3:0/2768612744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080619b0 msgr2=0x7f36080b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e95a700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080619b0 0x7f36080b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e95a700 1 -- 172.21.15.3:0/2768612744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36080b5ed0 msgr2=0x7f3608144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:40.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e95a700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36080b5ed0 0x7f3608144760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:40.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e95a700 1 -- 172.21.15.3:0/2768612744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360400b1e0 con 0x7f36080b6ee0 2022-07-21T01:36:40.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.733+0000 7f360e95a700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080b6ee0 0x7f3608152e40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f36080afbc0 tx=0x7f3600008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:40.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f35f77fe700 1 -- 172.21.15.3:0/2768612744 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3600013620 con 0x7f36080b6ee0 2022-07-21T01:36:40.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f35f77fe700 1 -- 172.21.15.3:0/2768612744 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600013780 con 0x7f36080b6ee0 2022-07-21T01:36:40.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36080af8f0 con 0x7f36080b6ee0 2022-07-21T01:36:40.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f35f77fe700 1 -- 172.21.15.3:0/2768612744 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3600013ae0 con 0x7f36080b6ee0 2022-07-21T01:36:40.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36081529a0 con 0x7f36080b6ee0 2022-07-21T01:36:40.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3608152b60 con 0x7f36080b6ee0 2022-07-21T01:36:40.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f35f77fe700 1 -- 172.21.15.3:0/2768612744 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3600013c40 con 0x7f36080b6ee0 2022-07-21T01:36:40.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f35f77fe700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35e8053a10 0x7f35e8055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:40.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f35f77fe700 1 -- 172.21.15.3:0/2768612744 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f360006bd40 con 0x7f36080b6ee0 2022-07-21T01:36:40.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f360e159700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35e8053a10 0x7f35e8055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:40.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.737+0000 7f360e159700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35e8053a10 0x7f35e8055ed0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f360401c000 tx=0x7f3604015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:40.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.749+0000 7f35f77fe700 1 -- 172.21.15.3:0/2768612744 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3600042aa0 con 0x7f36080b6ee0 2022-07-21T01:36:40.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:40.909+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f36080574e0 con 0x7f36080b6ee0 2022-07-21T01:36:41.679 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:36:41.677+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:36:41.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.693+0000 7f35f77fe700 1 -- 172.21.15.3:0/2768612744 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) v1 ==== 161+0+0 (secure 0 0 0) 0x7f36000460c0 con 0x7f36080b6ee0 2022-07-21T01:36:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35e8053a10 msgr2=0x7f35e8055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35e8053a10 0x7f35e8055ed0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f360401c000 tx=0x7f3604015000).stop 2022-07-21T01:36:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080b6ee0 msgr2=0x7f3608152e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080b6ee0 0x7f3608152e40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f36080afbc0 tx=0x7f3600008040).stop 2022-07-21T01:36:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 shutdown_connections 2022-07-21T01:36:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36080b6ee0 0x7f3608152e40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35e8053a10 0x7f35e8055ed0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36080b5ed0 0x7f3608144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 --2- 172.21.15.3:0/2768612744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36080619b0 0x7f36080b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:41.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 >> 172.21.15.3:0/2768612744 conn(0x7f360806af80 msgr2=0x7f3608140e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:41.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 shutdown_connections 2022-07-21T01:36:41.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.697+0000 7f360f15b700 1 -- 172.21.15.3:0/2768612744 wait complete. 2022-07-21T01:36:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:36:41.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:36:41.729 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:36:41.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe265790700 1 Processor -- start 2022-07-21T01:36:41.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe265790700 1 -- start start 2022-07-21T01:36:41.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe265790700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe2600b2930 0x7fe260147600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:41.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe265790700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2600b1ea0 0x7fe26014b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:41.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe265790700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2600b2d10 con 0x7fe26005c620 2022-07-21T01:36:41.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe265790700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe260145340 con 0x7fe2600b2930 2022-07-21T01:36:41.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe265790700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26014ac60 con 0x7fe2600b1ea0 2022-07-21T01:36:41.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe2600b2930 0x7fe260147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:41.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe264f8f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe26005c620 0x7fe2600a8d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53394/0 (socket says 172.21.15.3:53394) 2022-07-21T01:36:41.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe264f8f700 1 -- 172.21.15.3:0/863584392 learned_addr learned my addr 172.21.15.3:0/863584392 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:41.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25f7fe700 1 --2- 172.21.15.3:0/863584392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2600b1ea0 0x7fe26014b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:41.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25ffff700 1 -- 172.21.15.3:0/863584392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2600b1ea0 msgr2=0x7fe26014b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:41.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25ffff700 1 --2- 172.21.15.3:0/863584392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2600b1ea0 0x7fe26014b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:41.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25ffff700 1 -- 172.21.15.3:0/863584392 >> v1:172.21.15.3:6789/0 conn(0x7fe26005c620 legacy=0x7fe2600a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:41.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25effd700 1 -- 172.21.15.3:0/863584392 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531855089 0 0) 0x7fe2600b2d10 con 0x7fe26005c620 2022-07-21T01:36:41.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25ffff700 1 -- 172.21.15.3:0/863584392 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2600b1050 con 0x7fe2600b2930 2022-07-21T01:36:41.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25ffff700 1 --2- 172.21.15.3:0/863584392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b2930 0x7fe260147600 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fe250002f70 tx=0x7fe250006df0).ready entity=mon.0 client_cookie=198d08398dfa28fb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:41.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25effd700 1 -- 172.21.15.3:0/863584392 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe25000d570 con 0x7fe2600b2930 2022-07-21T01:36:41.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25effd700 1 -- 172.21.15.3:0/863584392 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe250000e90 con 0x7fe2600b2930 2022-07-21T01:36:41.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.817+0000 7fe25effd700 1 -- 172.21.15.3:0/863584392 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe25000d8c0 con 0x7fe2600b2930 2022-07-21T01:36:41.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 -- 172.21.15.3:0/863584392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b2930 msgr2=0x7fe260147600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:41.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 --2- 172.21.15.3:0/863584392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b2930 0x7fe260147600 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fe250002f70 tx=0x7fe250006df0).stop 2022-07-21T01:36:41.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 -- 172.21.15.3:0/863584392 shutdown_connections 2022-07-21T01:36:41.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 --2- 172.21.15.3:0/863584392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2600b1ea0 0x7fe26014b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:41.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 --2- 172.21.15.3:0/863584392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b2930 0x7fe260147600 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:41.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 -- 172.21.15.3:0/863584392 >> 172.21.15.3:0/863584392 conn(0x7fe2600612d0 msgr2=0x7fe260062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:41.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 -- 172.21.15.3:0/863584392 shutdown_connections 2022-07-21T01:36:41.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 -- 172.21.15.3:0/863584392 wait complete. 2022-07-21T01:36:41.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 Processor -- start 2022-07-21T01:36:41.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.821+0000 7fe265790700 1 -- start start 2022-07-21T01:36:41.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe265790700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe26005c620 0x7fe260149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:41.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe265790700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b1ea0 0x7fe260144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:41.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe265790700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2600b2930 0x7fe260144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:41.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe265790700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe260063fb0 con 0x7fe2600b1ea0 2022-07-21T01:36:41.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe265790700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe260146ce0 con 0x7fe2600b2930 2022-07-21T01:36:41.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe265790700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe260063d10 con 0x7fe26005c620 2022-07-21T01:36:41.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b1ea0 0x7fe260144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:41.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b1ea0 0x7fe260144640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45118/0 (socket says 172.21.15.3:45118) 2022-07-21T01:36:41.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25f7fe700 1 -- 172.21.15.3:0/3157927580 learned_addr learned my addr 172.21.15.3:0/3157927580 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:41.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25ffff700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe26005c620 0x7fe260149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:41.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe264f8f700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2600b2930 0x7fe260144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:41.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25f7fe700 1 -- 172.21.15.3:0/3157927580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe26005c620 msgr2=0x7fe260149880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:41.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25f7fe700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe26005c620 0x7fe260149880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:41.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25f7fe700 1 -- 172.21.15.3:0/3157927580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2600b2930 msgr2=0x7fe260144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:41.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25f7fe700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2600b2930 0x7fe260144b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:41.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25f7fe700 1 -- 172.21.15.3:0/3157927580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25000b180 con 0x7fe2600b1ea0 2022-07-21T01:36:41.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25f7fe700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b1ea0 0x7fe260144640 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fe25400a000 tx=0x7fe254008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:41.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25d7fa700 1 -- 172.21.15.3:0/3157927580 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe254011780 con 0x7fe2600b1ea0 2022-07-21T01:36:41.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25d7fa700 1 -- 172.21.15.3:0/3157927580 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2540118e0 con 0x7fe2600b1ea0 2022-07-21T01:36:41.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe260145020 con 0x7fe2600b1ea0 2022-07-21T01:36:41.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25d7fa700 1 -- 172.21.15.3:0/3157927580 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe254011c40 con 0x7fe2600b1ea0 2022-07-21T01:36:41.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2601476a0 con 0x7fe2600b1ea0 2022-07-21T01:36:41.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.825+0000 7fe25d7fa700 1 -- 172.21.15.3:0/3157927580 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe25401c460 con 0x7fe2600b1ea0 2022-07-21T01:36:41.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.829+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2600574e0 con 0x7fe2600b1ea0 2022-07-21T01:36:41.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.829+0000 7fe25d7fa700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe240053b00 0x7fe240055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:41.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.829+0000 7fe25d7fa700 1 -- 172.21.15.3:0/3157927580 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fe25406c290 con 0x7fe2600b1ea0 2022-07-21T01:36:41.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.829+0000 7fe25ffff700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe240053b00 0x7fe240055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:41.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.829+0000 7fe25ffff700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe240053b00 0x7fe240055fc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fe25001c000 tx=0x7fe250015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:41.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:41.837+0000 7fe25d7fa700 1 -- 172.21.15.3:0/3157927580 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe254042e90 con 0x7fe2600b1ea0 2022-07-21T01:36:42.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.001+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe260069a70 con 0x7fe2600b1ea0 2022-07-21T01:36:42.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.001+0000 7fe25d7fa700 1 -- 172.21.15.3:0/3157927580 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+2130 (secure 0 0 0) 0x7fe2540464b0 con 0x7fe2600b1ea0 2022-07-21T01:36:42.003 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:42.003 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":19,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:35:18.753732+0000","modified":"2022-07-21T01:36:41.680543+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-21T01:36:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe240053b00 msgr2=0x7fe240055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe240053b00 0x7fe240055fc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fe25001c000 tx=0x7fe250015000).stop 2022-07-21T01:36:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b1ea0 msgr2=0x7fe260144640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b1ea0 0x7fe260144640 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fe25400a000 tx=0x7fe254008040).stop 2022-07-21T01:36:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 shutdown_connections 2022-07-21T01:36:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe240053b00 0x7fe240055fc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2600b2930 0x7fe260144b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2600b1ea0 0x7fe260144640 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 --2- 172.21.15.3:0/3157927580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe26005c620 0x7fe260149880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 >> 172.21.15.3:0/3157927580 conn(0x7fe2600612d0 msgr2=0x7fe2600b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 shutdown_connections 2022-07-21T01:36:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.005+0000 7fe265790700 1 -- 172.21.15.3:0/3157927580 wait complete. 2022-07-21T01:36:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 19 2022-07-21T01:36:42.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:36:42.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82b59e700 1 Processor -- start 2022-07-21T01:36:42.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82b59e700 1 -- start start 2022-07-21T01:36:42.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb82c0b65c0 0x7fb82c061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0b2100 con 0x7fb82c147d80 2022-07-21T01:36:42.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82c0b1630 con 0x7fb82c0b65c0 2022-07-21T01:36:42.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c061fe0 con 0x7fb82c0b41f0 2022-07-21T01:36:42.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82a59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb82c0b65c0 0x7fb82c061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb829d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82ad9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb82c147d80 0x7fb82c144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:55524/0 (socket says 172.21.15.3:55524) 2022-07-21T01:36:42.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82ad9d700 1 -- 172.21.15.3:0/925570616 learned_addr learned my addr 172.21.15.3:0/925570616 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:42.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb829d9b700 1 -- 172.21.15.3:0/925570616 >> v2:172.21.15.143:3300/0 conn(0x7fb82c0b65c0 msgr2=0x7fb82c061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:42.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb829d9b700 1 --2- 172.21.15.3:0/925570616 >> v2:172.21.15.143:3300/0 conn(0x7fb82c0b65c0 0x7fb82c061600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb829d9b700 1 -- 172.21.15.3:0/925570616 >> v1:172.21.15.143:6789/0 conn(0x7fb82c147d80 legacy=0x7fb82c144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb829d9b700 1 -- 172.21.15.3:0/925570616 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c0b69a0 con 0x7fb82c0b41f0 2022-07-21T01:36:42.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb82a59c700 1 --2- 172.21.15.3:0/925570616 >> v2:172.21.15.143:3300/0 conn(0x7fb82c0b65c0 0x7fb82c061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:42.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.129+0000 7fb829d9b700 1 --2- 172.21.15.3:0/925570616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c061af0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fb820002f70 tx=0x7fb820006df0).ready entity=mon.2 client_cookie=beba3c496eb1da93 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:42.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82959a700 1 -- 172.21.15.3:0/925570616 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb82000d570 con 0x7fb82c0b41f0 2022-07-21T01:36:42.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82959a700 1 -- 172.21.15.3:0/925570616 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb820000e90 con 0x7fb82c0b41f0 2022-07-21T01:36:42.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82959a700 1 -- 172.21.15.3:0/925570616 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb82000d8c0 con 0x7fb82c0b41f0 2022-07-21T01:36:42.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 -- 172.21.15.3:0/925570616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 msgr2=0x7fb82c061af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 --2- 172.21.15.3:0/925570616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c061af0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fb820002f70 tx=0x7fb820006df0).stop 2022-07-21T01:36:42.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 -- 172.21.15.3:0/925570616 shutdown_connections 2022-07-21T01:36:42.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 --2- 172.21.15.3:0/925570616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c061af0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 --2- 172.21.15.3:0/925570616 >> v2:172.21.15.143:3300/0 conn(0x7fb82c0b65c0 0x7fb82c061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 -- 172.21.15.3:0/925570616 >> 172.21.15.3:0/925570616 conn(0x7fb82c05c620 msgr2=0x7fb82c05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:42.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 -- 172.21.15.3:0/925570616 shutdown_connections 2022-07-21T01:36:42.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 -- 172.21.15.3:0/925570616 wait complete. 2022-07-21T01:36:42.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 Processor -- start 2022-07-21T01:36:42.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 -- start start 2022-07-21T01:36:42.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb82c0b65c0 0x7fb82c064a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb82c147d80 0x7fb82c064ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0641e0 con 0x7fb82c0b65c0 2022-07-21T01:36:42.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c149fc0 con 0x7fb82c147d80 2022-07-21T01:36:42.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c14a260 con 0x7fb82c0b41f0 2022-07-21T01:36:42.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb829d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb82c0b65c0 0x7fb82c064a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82a59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb829d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb82c0b65c0 0x7fb82c064a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45132/0 (socket says 172.21.15.3:45132) 2022-07-21T01:36:42.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb829d9b700 1 -- 172.21.15.3:0/4122301646 learned_addr learned my addr 172.21.15.3:0/4122301646 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:42.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82ad9d700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb82c147d80 0x7fb82c064ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82a59c700 1 -- 172.21.15.3:0/4122301646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb82c147d80 msgr2=0x7fb82c064ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82a59c700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb82c147d80 0x7fb82c064ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82a59c700 1 -- 172.21.15.3:0/4122301646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb82c0b65c0 msgr2=0x7fb82c064a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82a59c700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb82c0b65c0 0x7fb82c064a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.133+0000 7fb82a59c700 1 -- 172.21.15.3:0/4122301646 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c00b180 con 0x7fb82c0b41f0 2022-07-21T01:36:42.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb82a59c700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c146120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb82c0b8d20 tx=0x7fb81c006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:42.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb81b7fe700 1 -- 172.21.15.3:0/4122301646 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb81c010780 con 0x7fb82c0b41f0 2022-07-21T01:36:42.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82000b180 con 0x7fb82c0b41f0 2022-07-21T01:36:42.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb81b7fe700 1 -- 172.21.15.3:0/4122301646 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c0108e0 con 0x7fb82c0b41f0 2022-07-21T01:36:42.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb81b7fe700 1 -- 172.21.15.3:0/4122301646 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb81c010c40 con 0x7fb82c0b41f0 2022-07-21T01:36:42.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c14b410 con 0x7fb82c0b41f0 2022-07-21T01:36:42.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82c14afd0 con 0x7fb82c0b41f0 2022-07-21T01:36:42.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb81b7fe700 1 -- 172.21.15.3:0/4122301646 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb81c01c460 con 0x7fb82c0b41f0 2022-07-21T01:36:42.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb81b7fe700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8080539e0 0x7fb808055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb81b7fe700 1 -- 172.21.15.3:0/4122301646 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fb81c06c6b0 con 0x7fb82c0b41f0 2022-07-21T01:36:42.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.137+0000 7fb829d9b700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8080539e0 0x7fb808055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.141+0000 7fb829d9b700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8080539e0 0x7fb808055ea0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb82001a000 tx=0x7fb820014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:42.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.149+0000 7fb81b7fe700 1 -- 172.21.15.3:0/4122301646 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb82c14afd0 con 0x7fb82c0b41f0 2022-07-21T01:36:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.309+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb82c14afd0 con 0x7fb82c0b41f0 2022-07-21T01:36:42.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.697+0000 7fb81b7fe700 1 -- 172.21.15.3:0/4122301646 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb82c14afd0 con 0x7fb82c0b41f0 2022-07-21T01:36:42.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8080539e0 msgr2=0x7fb808055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8080539e0 0x7fb808055ea0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb82001a000 tx=0x7fb820014000).stop 2022-07-21T01:36:42.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 msgr2=0x7fb82c146120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c146120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb82c0b8d20 tx=0x7fb81c006df0).stop 2022-07-21T01:36:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 shutdown_connections 2022-07-21T01:36:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8080539e0 0x7fb808055ea0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb82c147d80 0x7fb82c064ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb82c0b65c0 0x7fb82c064a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 --2- 172.21.15.3:0/4122301646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb82c0b41f0 0x7fb82c146120 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 >> 172.21.15.3:0/4122301646 conn(0x7fb82c05c620 msgr2=0x7fb82c0b5390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 shutdown_connections 2022-07-21T01:36:42.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.701+0000 7fb82b59e700 1 -- 172.21.15.3:0/4122301646 wait complete. 2022-07-21T01:36:42.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:36:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.833+0000 7fa980d3b700 1 Processor -- start 2022-07-21T01:36:42.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.833+0000 7fa980d3b700 1 -- start start 2022-07-21T01:36:42.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa980d3b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c0a9da0 0x7fa97c060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa980d3b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97c067030 con 0x7fa97c0b31d0 2022-07-21T01:36:42.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa980d3b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97c0b35b0 con 0x7fa97c1402d0 2022-07-21T01:36:42.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa980d3b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c064a70 con 0x7fa97c0a9da0 2022-07-21T01:36:42.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97b7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa97c0b31d0 0x7fa97c066020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53420/0 (socket says 172.21.15.3:53420) 2022-07-21T01:36:42.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97b7fe700 1 -- 172.21.15.3:0/355495327 learned_addr learned my addr 172.21.15.3:0/355495327 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:42.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97affd700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c0a9da0 0x7fa97c060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811348715 0 0) 0x7fa97c067030 con 0x7fa97c0b31d0 2022-07-21T01:36:42.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa968003260 con 0x7fa97c0b31d0 2022-07-21T01:36:42.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4119079312 0 0) 0x7fa968003260 con 0x7fa97c0b31d0 2022-07-21T01:36:42.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c0a9da0 msgr2=0x7fa97c060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:42.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c0a9da0 0x7fa97c060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 >> v1:172.21.15.143:6789/0 conn(0x7fa97c1402d0 legacy=0x7fa97c140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c066af0 con 0x7fa97c0b31d0 2022-07-21T01:36:42.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fa96c001660 con 0x7fa97c0b31d0 2022-07-21T01:36:42.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 >> v1:172.21.15.3:6789/0 conn(0x7fa97c0b31d0 legacy=0x7fa97c066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c0a9da0 0x7fa968003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa968003c70 0x7fa968004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9680049f0 0x7fa968004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96800b750 con 0x7fa9680049f0 2022-07-21T01:36:42.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96800b890 con 0x7fa968003c70 2022-07-21T01:36:42.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96800ba50 con 0x7fa97c0a9da0 2022-07-21T01:36:42.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97bfff700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9680049f0 0x7fa968004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97affd700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa968003c70 0x7fa968004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97b7fe700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c0a9da0 0x7fa968003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97bfff700 1 -- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c0a9da0 msgr2=0x7fa968003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:42.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97bfff700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c0a9da0 0x7fa968003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97bfff700 1 -- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa968003c70 msgr2=0x7fa968004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:42.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97bfff700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa968003c70 0x7fa968004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97bfff700 1 -- 172.21.15.3:0/355495327 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa968003260 con 0x7fa9680049f0 2022-07-21T01:36:42.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97bfff700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9680049f0 0x7fa968004e70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fa9740013c0 tx=0x7fa974007d80).ready entity=mon.0 client_cookie=683a2f2be6f79ee server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:42.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa974006890 con 0x7fa9680049f0 2022-07-21T01:36:42.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9740069f0 con 0x7fa9680049f0 2022-07-21T01:36:42.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.837+0000 7fa97a7fc700 1 -- 172.21.15.3:0/355495327 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa974006d50 con 0x7fa9680049f0 2022-07-21T01:36:42.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.841+0000 7fa980d3b700 1 -- 172.21.15.3:0/355495327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9680049f0 msgr2=0x7fa968004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.841+0000 7fa980d3b700 1 --2- 172.21.15.3:0/355495327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9680049f0 0x7fa968004e70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fa9740013c0 tx=0x7fa974007d80).stop 2022-07-21T01:36:42.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.841+0000 7fa97bfff700 1 -- 172.21.15.3:0/355495327 reap_dead start 2022-07-21T01:36:42.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.841+0000 7fa980d3b700 1 -- 172.21.15.3:0/355495327 shutdown_connections 2022-07-21T01:36:42.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.841+0000 7fa980d3b700 1 -- 172.21.15.3:0/355495327 >> 172.21.15.3:0/355495327 conn(0x7fa97c063870 msgr2=0x7fa97c063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:42.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.841+0000 7fa980d3b700 1 -- 172.21.15.3:0/355495327 shutdown_connections 2022-07-21T01:36:42.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.841+0000 7fa980d3b700 1 -- 172.21.15.3:0/355495327 wait complete. 2022-07-21T01:36:42.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.841+0000 7fa980d3b700 1 Processor -- start 2022-07-21T01:36:42.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.841+0000 7fa980d3b700 1 -- start start 2022-07-21T01:36:42.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa980d3b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa97c144be0 0x7fa97c1519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa980d3b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa97c0b24a0 0x7fa97c151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa980d3b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c13ee50 0x7fa97c13f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa980d3b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c0b2bd0 con 0x7fa97c144be0 2022-07-21T01:36:42.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa980d3b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c0b1e30 con 0x7fa97c0b24a0 2022-07-21T01:36:42.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa980d3b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c151500 con 0x7fa97c13ee50 2022-07-21T01:36:42.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa97c144be0 0x7fa97c1519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa97c144be0 0x7fa97c1519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45152/0 (socket says 172.21.15.3:45152) 2022-07-21T01:36:42.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97b7fe700 1 -- 172.21.15.3:0/684267929 learned_addr learned my addr 172.21.15.3:0/684267929 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:42.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97bfff700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c13ee50 0x7fa97c13f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97affd700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa97c0b24a0 0x7fa97c151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97b7fe700 1 -- 172.21.15.3:0/684267929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c13ee50 msgr2=0x7fa97c13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97b7fe700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c13ee50 0x7fa97c13f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97b7fe700 1 -- 172.21.15.3:0/684267929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa97c0b24a0 msgr2=0x7fa97c151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:42.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97b7fe700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa97c0b24a0 0x7fa97c151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:42.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97b7fe700 1 -- 172.21.15.3:0/684267929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c0ab620 con 0x7fa97c144be0 2022-07-21T01:36:42.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa97b7fe700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa97c144be0 0x7fa97c1519a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fa96c00d000 tx=0x7fa97c065990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:42.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa978ff9700 1 -- 172.21.15.3:0/684267929 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c0116b0 con 0x7fa97c144be0 2022-07-21T01:36:42.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa978ff9700 1 -- 172.21.15.3:0/684267929 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c011810 con 0x7fa97c144be0 2022-07-21T01:36:42.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa978ff9700 1 -- 172.21.15.3:0/684267929 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c011b70 con 0x7fa97c144be0 2022-07-21T01:36:42.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c0b28c0 con 0x7fa97c144be0 2022-07-21T01:36:42.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.845+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c144fc0 con 0x7fa97c144be0 2022-07-21T01:36:42.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.849+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c0b1ff0 con 0x7fa97c144be0 2022-07-21T01:36:42.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.849+0000 7fa978ff9700 1 -- 172.21.15.3:0/684267929 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa96c011cd0 con 0x7fa97c144be0 2022-07-21T01:36:42.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.849+0000 7fa978ff9700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa95c053a70 0x7fa95c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:42.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.849+0000 7fa978ff9700 1 -- 172.21.15.3:0/684267929 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fa96c06bb70 con 0x7fa97c144be0 2022-07-21T01:36:42.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.849+0000 7fa97affd700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa95c053a70 0x7fa95c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:42.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.849+0000 7fa97affd700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa95c053a70 0x7fa95c055f30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa97000b000 tx=0x7fa970006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:42.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.857+0000 7fa978ff9700 1 -- 172.21.15.3:0/684267929 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa96c0428e0 con 0x7fa97c144be0 2022-07-21T01:36:42.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:42.985+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa97c1516c0 con 0x7fa97c144be0 2022-07-21T01:36:43.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.705+0000 7fa978ff9700 1 -- 172.21.15.3:0/684267929 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v50) v1 ==== 185+0+0 (secure 0 0 0) 0x7fa96c019020 con 0x7fa97c144be0 2022-07-21T01:36:43.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa95c053a70 msgr2=0x7fa95c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:43.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa95c053a70 0x7fa95c055f30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa97000b000 tx=0x7fa970006df0).stop 2022-07-21T01:36:43.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa97c144be0 msgr2=0x7fa97c1519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:43.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa97c144be0 0x7fa97c1519a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fa96c00d000 tx=0x7fa97c065990).stop 2022-07-21T01:36:43.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 shutdown_connections 2022-07-21T01:36:43.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa97c13ee50 0x7fa97c13f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:43.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa95c053a70 0x7fa95c055f30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:43.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa97c0b24a0 0x7fa97c151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:43.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 --2- 172.21.15.3:0/684267929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa97c144be0 0x7fa97c1519a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:43.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.709+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 >> 172.21.15.3:0/684267929 conn(0x7fa97c063870 msgr2=0x7fa97c13ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:43.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.713+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 shutdown_connections 2022-07-21T01:36:43.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.713+0000 7fa980d3b700 1 -- 172.21.15.3:0/684267929 wait complete. 2022-07-21T01:36:43.716 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:36:43.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:36:43.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.837+0000 7fddb89e9700 1 Processor -- start 2022-07-21T01:36:43.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.837+0000 7fddb89e9700 1 -- start start 2022-07-21T01:36:43.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb89e9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fddb40ad670 0x7fddb40b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:43.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb89e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddb413fa40 0x7fddb413fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:43.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb89e9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb40b0c40 con 0x7fddb40af120 2022-07-21T01:36:43.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb89e9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb40b0ee0 con 0x7fddb40ad670 2022-07-21T01:36:43.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb89e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb40ae9e0 con 0x7fddb413fa40 2022-07-21T01:36:43.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fddb40ad670 0x7fddb40b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:43.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fddb40ad670 0x7fddb40b0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45160/0 (socket says 172.21.15.3:45160) 2022-07-21T01:36:43.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb37fe700 1 -- 172.21.15.3:0/1035252777 learned_addr learned my addr 172.21.15.3:0/1035252777 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:43.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb3fff700 1 --2- 172.21.15.3:0/1035252777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddb413fa40 0x7fddb413fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:43.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb37fe700 1 -- 172.21.15.3:0/1035252777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddb413fa40 msgr2=0x7fddb413fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:43.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb37fe700 1 --2- 172.21.15.3:0/1035252777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddb413fa40 0x7fddb413fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:43.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb37fe700 1 -- 172.21.15.3:0/1035252777 >> v1:172.21.15.143:6789/0 conn(0x7fddb40af120 legacy=0x7fddb40b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:43.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb37fe700 1 -- 172.21.15.3:0/1035252777 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb40aec80 con 0x7fddb40ad670 2022-07-21T01:36:43.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb27fc700 1 -- 172.21.15.3:0/1035252777 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362017437 0 0) 0x7fddb40b0c40 con 0x7fddb40af120 2022-07-21T01:36:43.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb37fe700 1 --2- 172.21.15.3:0/1035252777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb40ad670 0x7fddb40b0570 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fddb40ae3c0 tx=0x7fdda8006eb0).ready entity=mon.0 client_cookie=1e4e4787d13ba60d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:43.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb27fc700 1 -- 172.21.15.3:0/1035252777 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdda800d5c0 con 0x7fddb40ad670 2022-07-21T01:36:43.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb27fc700 1 -- 172.21.15.3:0/1035252777 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda8000e90 con 0x7fddb40ad670 2022-07-21T01:36:43.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb27fc700 1 -- 172.21.15.3:0/1035252777 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdda800d950 con 0x7fddb40ad670 2022-07-21T01:36:43.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb89e9700 1 -- 172.21.15.3:0/1035252777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb40ad670 msgr2=0x7fddb40b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:43.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.841+0000 7fddb89e9700 1 --2- 172.21.15.3:0/1035252777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb40ad670 0x7fddb40b0570 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fddb40ae3c0 tx=0x7fdda8006eb0).stop 2022-07-21T01:36:43.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.845+0000 7fddb89e9700 1 -- 172.21.15.3:0/1035252777 shutdown_connections 2022-07-21T01:36:43.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.845+0000 7fddb89e9700 1 --2- 172.21.15.3:0/1035252777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddb413fa40 0x7fddb413fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:43.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.845+0000 7fddb89e9700 1 --2- 172.21.15.3:0/1035252777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb40ad670 0x7fddb40b0570 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:43.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.845+0000 7fddb89e9700 1 -- 172.21.15.3:0/1035252777 >> 172.21.15.3:0/1035252777 conn(0x7fddb4061360 msgr2=0x7fddb4062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:43.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.845+0000 7fddb89e9700 1 -- 172.21.15.3:0/1035252777 shutdown_connections 2022-07-21T01:36:43.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.845+0000 7fddb89e9700 1 -- 172.21.15.3:0/1035252777 wait complete. 2022-07-21T01:36:43.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.845+0000 7fddb89e9700 1 Processor -- start 2022-07-21T01:36:43.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.845+0000 7fddb89e9700 1 -- start start 2022-07-21T01:36:43.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb89e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddb40ad670 0x7fddb4141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:43.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb89e9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddb40af120 0x7fddb40674b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:43.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb89e9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb413fa40 0x7fddb40679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb89e9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb40ab980 con 0x7fddb413fa40 2022-07-21T01:36:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb89e9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb4063dc0 con 0x7fddb40af120 2022-07-21T01:36:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb89e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb4064060 con 0x7fddb40ad670 2022-07-21T01:36:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb413fa40 0x7fddb40679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb413fa40 0x7fddb40679a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45170/0 (socket says 172.21.15.3:45170) 2022-07-21T01:36:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb3fff700 1 -- 172.21.15.3:0/1828736782 learned_addr learned my addr 172.21.15.3:0/1828736782 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb3fff700 1 -- 172.21.15.3:0/1828736782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddb40ad670 msgr2=0x7fddb4141c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-07-21T01:36:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb3fff700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddb40ad670 0x7fddb4141c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:36:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb3fff700 1 -- 172.21.15.3:0/1828736782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddb40af120 msgr2=0x7fddb40674b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb3fff700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddb40af120 0x7fddb40674b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:36:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.849+0000 7fddb3fff700 1 -- 172.21.15.3:0/1828736782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda800b1e0 con 0x7fddb413fa40 2022-07-21T01:36:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb3fff700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb413fa40 0x7fddb40679a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fddac006e00 tx=0x7fddac0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb0ff9700 1 -- 172.21.15.3:0/1828736782 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddac00c7f0 con 0x7fddb413fa40 2022-07-21T01:36:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb0ff9700 1 -- 172.21.15.3:0/1828736782 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac00c950 con 0x7fddb413fa40 2022-07-21T01:36:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb4142110 con 0x7fddb413fa40 2022-07-21T01:36:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb0ff9700 1 -- 172.21.15.3:0/1828736782 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddac00ccb0 con 0x7fddb413fa40 2022-07-21T01:36:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb4066050 con 0x7fddb413fa40 2022-07-21T01:36:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb4067e90 con 0x7fddb413fa40 2022-07-21T01:36:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb0ff9700 1 -- 172.21.15.3:0/1828736782 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fddac022070 con 0x7fddb413fa40 2022-07-21T01:36:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb0ff9700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdd94053980 0x7fdd94055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb0ff9700 1 -- 172.21.15.3:0/1828736782 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6068+0+0 (secure 0 0 0) 0x7fddac06c5f0 con 0x7fddb413fa40 2022-07-21T01:36:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb37fe700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdd94053980 0x7fdd94055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.853+0000 7fddb37fe700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdd94053980 0x7fdd94055e40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fddb40ae3c0 tx=0x7fdda8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.857+0000 7fddb0ff9700 1 -- 172.21.15.3:0/1828736782 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fddac043510 con 0x7fddb413fa40 2022-07-21T01:36:43.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:43.977+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fddb4064220 con 0x7fddb413fa40 2022-07-21T01:36:44.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.713+0000 7fddb0ff9700 1 -- 172.21.15.3:0/1828736782 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v51) v1 ==== 173+0+0 (secure 0 0 0) 0x7fddac016020 con 0x7fddb413fa40 2022-07-21T01:36:44.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdd94053980 msgr2=0x7fdd94055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:44.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdd94053980 0x7fdd94055e40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fddb40ae3c0 tx=0x7fdda8015000).stop 2022-07-21T01:36:44.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb413fa40 msgr2=0x7fddb40679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:44.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb413fa40 0x7fddb40679a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fddac006e00 tx=0x7fddac0091d0).stop 2022-07-21T01:36:44.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 shutdown_connections 2022-07-21T01:36:44.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddb413fa40 0x7fddb40679a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdd94053980 0x7fdd94055e40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddb40af120 0x7fddb40674b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:36:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 --2- 172.21.15.3:0/1828736782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddb40ad670 0x7fddb4141c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:36:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 >> 172.21.15.3:0/1828736782 conn(0x7fddb4061360 msgr2=0x7fddb413fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 shutdown_connections 2022-07-21T01:36:44.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.717+0000 7fddb89e9700 1 -- 172.21.15.3:0/1828736782 wait complete. 2022-07-21T01:36:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:36:44.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:36:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.841+0000 7f3115451700 1 Processor -- start 2022-07-21T01:36:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.841+0000 7f3115451700 1 -- start start 2022-07-21T01:36:44.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31100b07a0 0x7f31100af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:44.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3110147730 0x7f31100acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:44.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3110147b80 con 0x7f31100b5630 2022-07-21T01:36:44.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3110145470 con 0x7f3110147730 2022-07-21T01:36:44.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31100ae180 con 0x7f31100b07a0 2022-07-21T01:36:44.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3114c50700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3110147730 0x7f31100acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:44.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3114c50700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3110147730 0x7f31100acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45178/0 (socket says 172.21.15.3:45178) 2022-07-21T01:36:44.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3114c50700 1 -- 172.21.15.3:0/3473200066 learned_addr learned my addr 172.21.15.3:0/3473200066 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:44.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f310f7fe700 1 --2- 172.21.15.3:0/3473200066 >> v2:172.21.15.143:3300/0 conn(0x7f31100b07a0 0x7f31100af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:44.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3114c50700 1 -- 172.21.15.3:0/3473200066 >> v2:172.21.15.143:3300/0 conn(0x7f31100b07a0 msgr2=0x7f31100af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:44.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3114c50700 1 --2- 172.21.15.3:0/3473200066 >> v2:172.21.15.143:3300/0 conn(0x7f31100b07a0 0x7f31100af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3114c50700 1 -- 172.21.15.3:0/3473200066 >> v1:172.21.15.143:6789/0 conn(0x7f31100b5630 legacy=0x7f3110067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:44.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3114c50700 1 -- 172.21.15.3:0/3473200066 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100b5a10 con 0x7f3110147730 2022-07-21T01:36:44.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f310f7fe700 1 --2- 172.21.15.3:0/3473200066 >> v2:172.21.15.143:3300/0 conn(0x7f31100b07a0 0x7f31100af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:36:44.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3114c50700 1 --2- 172.21.15.3:0/3473200066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 0x7f31100acdf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f3108002f70 tx=0x7f3108006eb0).ready entity=mon.0 client_cookie=53c0e0f8e8e1cf7c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:44.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f310effd700 1 -- 172.21.15.3:0/3473200066 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f310800d610 con 0x7f3110147730 2022-07-21T01:36:44.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f310effd700 1 -- 172.21.15.3:0/3473200066 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3108000e90 con 0x7f3110147730 2022-07-21T01:36:44.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f310effd700 1 -- 172.21.15.3:0/3473200066 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f310800d9a0 con 0x7f3110147730 2022-07-21T01:36:44.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 -- 172.21.15.3:0/3473200066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 msgr2=0x7f31100acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:44.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 --2- 172.21.15.3:0/3473200066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 0x7f31100acdf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f3108002f70 tx=0x7f3108006eb0).stop 2022-07-21T01:36:44.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 -- 172.21.15.3:0/3473200066 shutdown_connections 2022-07-21T01:36:44.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 --2- 172.21.15.3:0/3473200066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 0x7f31100acdf0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 --2- 172.21.15.3:0/3473200066 >> v2:172.21.15.143:3300/0 conn(0x7f31100b07a0 0x7f31100af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 -- 172.21.15.3:0/3473200066 >> 172.21.15.3:0/3473200066 conn(0x7f31100626f0 msgr2=0x7f3110062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:44.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 -- 172.21.15.3:0/3473200066 shutdown_connections 2022-07-21T01:36:44.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 -- 172.21.15.3:0/3473200066 wait complete. 2022-07-21T01:36:44.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 Processor -- start 2022-07-21T01:36:44.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.845+0000 7f3115451700 1 -- start start 2022-07-21T01:36:44.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3115451700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31100b07a0 0x7f3110143740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:44.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3115451700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31100b5630 0x7f31101466a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:44.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3115451700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 0x7f3110066ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:44.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3115451700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3110142360 con 0x7f3110147730 2022-07-21T01:36:44.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3115451700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100b2fe0 con 0x7f31100b5630 2022-07-21T01:36:44.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3115451700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3110145d00 con 0x7f31100b07a0 2022-07-21T01:36:44.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3114c50700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 0x7f3110066ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:44.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f310f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31100b5630 0x7f31101466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:44.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f310ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31100b07a0 0x7f3110143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:44.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3114c50700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 0x7f3110066ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45184/0 (socket says 172.21.15.3:45184) 2022-07-21T01:36:44.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3114c50700 1 -- 172.21.15.3:0/2471915268 learned_addr learned my addr 172.21.15.3:0/2471915268 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:44.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3114c50700 1 -- 172.21.15.3:0/2471915268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31100b07a0 msgr2=0x7f3110143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:44.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3114c50700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31100b07a0 0x7f3110143740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3114c50700 1 -- 172.21.15.3:0/2471915268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31100b5630 msgr2=0x7f31101466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:44.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3114c50700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31100b5630 0x7f31101466a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3114c50700 1 -- 172.21.15.3:0/2471915268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310800b1e0 con 0x7f3110147730 2022-07-21T01:36:44.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3114c50700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 0x7f3110066ce0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f3108002f70 tx=0x7f3108015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:44.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f310d7fa700 1 -- 172.21.15.3:0/2471915268 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3108011860 con 0x7f3110147730 2022-07-21T01:36:44.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f310d7fa700 1 -- 172.21.15.3:0/2471915268 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31080119c0 con 0x7f3110147730 2022-07-21T01:36:44.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f310d7fa700 1 -- 172.21.15.3:0/2471915268 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3108011d20 con 0x7f3110147730 2022-07-21T01:36:44.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31100b3210 con 0x7f3110147730 2022-07-21T01:36:44.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31100671d0 con 0x7f3110147730 2022-07-21T01:36:44.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31100574e0 con 0x7f3110147730 2022-07-21T01:36:44.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.849+0000 7f310d7fa700 1 -- 172.21.15.3:0/2471915268 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3108025030 con 0x7f3110147730 2022-07-21T01:36:44.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.853+0000 7f310d7fa700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f30f0053b00 0x7f30f0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:44.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.853+0000 7f310d7fa700 1 -- 172.21.15.3:0/2471915268 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f310806cec0 con 0x7f3110147730 2022-07-21T01:36:44.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.853+0000 7f310ffff700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f30f0053b00 0x7f30f0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:44.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.853+0000 7f310ffff700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f30f0053b00 0x7f30f0055fc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f310400d000 tx=0x7f3104009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:44.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.861+0000 7f310d7fa700 1 -- 172.21.15.3:0/2471915268 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3108043fe0 con 0x7f3110147730 2022-07-21T01:36:44.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.985+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f311005f3d0 con 0x7f3110147730 2022-07-21T01:36:44.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f310d7fa700 1 -- 172.21.15.3:0/2471915268 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3108047580 con 0x7f3110147730 2022-07-21T01:36:44.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f30f0053b00 msgr2=0x7f30f0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:44.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f30f0053b00 0x7f30f0055fc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f310400d000 tx=0x7f3104009040).stop 2022-07-21T01:36:44.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 msgr2=0x7f3110066ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:44.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 0x7f3110066ce0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f3108002f70 tx=0x7f3108015000).stop 2022-07-21T01:36:44.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 shutdown_connections 2022-07-21T01:36:44.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3110147730 0x7f3110066ce0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f30f0053b00 0x7f30f0055fc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31100b5630 0x7f31101466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 --2- 172.21.15.3:0/2471915268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31100b07a0 0x7f3110143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:44.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 >> 172.21.15.3:0/2471915268 conn(0x7f31100626f0 msgr2=0x7f3110144810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:44.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 shutdown_connections 2022-07-21T01:36:44.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:44.993+0000 7f3115451700 1 -- 172.21.15.3:0/2471915268 wait complete. 2022-07-21T01:36:45.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-07-21T01:36:45.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544d44f700 1 Processor -- start 2022-07-21T01:36:45.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544d44f700 1 -- start start 2022-07-21T01:36:45.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544d44f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5448064b10 0x7f5448144e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544d44f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5448141a80 0x7f5448066ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544d44f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5448064ef0 con 0x7f54480b36e0 2022-07-21T01:36:45.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544d44f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f544813ebf0 con 0x7f5448141a80 2022-07-21T01:36:45.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544d44f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5448066690 con 0x7f5448064b10 2022-07-21T01:36:45.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544cc4e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5448141a80 0x7f5448066ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544cc4e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5448141a80 0x7f5448066ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45192/0 (socket says 172.21.15.3:45192) 2022-07-21T01:36:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544cc4e700 1 -- 172.21.15.3:0/1376415362 learned_addr learned my addr 172.21.15.3:0/1376415362 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f5447fff700 1 --2- 172.21.15.3:0/1376415362 >> v2:172.21.15.143:3300/0 conn(0x7f5448064b10 0x7f5448144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544cc4e700 1 -- 172.21.15.3:0/1376415362 >> v2:172.21.15.143:3300/0 conn(0x7f5448064b10 msgr2=0x7f5448144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:45.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544cc4e700 1 --2- 172.21.15.3:0/1376415362 >> v2:172.21.15.143:3300/0 conn(0x7f5448064b10 0x7f5448144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544cc4e700 1 -- 172.21.15.3:0/1376415362 >> v1:172.21.15.143:6789/0 conn(0x7f54480b36e0 legacy=0x7f5448141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544cc4e700 1 -- 172.21.15.3:0/1376415362 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54480b4850 con 0x7f5448141a80 2022-07-21T01:36:45.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f544cc4e700 1 --2- 172.21.15.3:0/1376415362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448141a80 0x7f5448066ac0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f5440002f70 tx=0x7f5440006eb0).ready entity=mon.0 client_cookie=6f434417954e838 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:45.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f5446ffd700 1 -- 172.21.15.3:0/1376415362 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f544000d5c0 con 0x7f5448141a80 2022-07-21T01:36:45.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f5446ffd700 1 -- 172.21.15.3:0/1376415362 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5440000e90 con 0x7f5448141a80 2022-07-21T01:36:45.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.113+0000 7f5446ffd700 1 -- 172.21.15.3:0/1376415362 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f544000d950 con 0x7f5448141a80 2022-07-21T01:36:45.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- 172.21.15.3:0/1376415362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448141a80 msgr2=0x7f5448066ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 --2- 172.21.15.3:0/1376415362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448141a80 0x7f5448066ac0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f5440002f70 tx=0x7f5440006eb0).stop 2022-07-21T01:36:45.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- 172.21.15.3:0/1376415362 shutdown_connections 2022-07-21T01:36:45.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 --2- 172.21.15.3:0/1376415362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448141a80 0x7f5448066ac0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 --2- 172.21.15.3:0/1376415362 >> v2:172.21.15.143:3300/0 conn(0x7f5448064b10 0x7f5448144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- 172.21.15.3:0/1376415362 >> 172.21.15.3:0/1376415362 conn(0x7f544805f840 msgr2=0x7f544805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:45.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- 172.21.15.3:0/1376415362 shutdown_connections 2022-07-21T01:36:45.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- 172.21.15.3:0/1376415362 wait complete. 2022-07-21T01:36:45.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 Processor -- start 2022-07-21T01:36:45.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- start start 2022-07-21T01:36:45.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448064b10 0x7f54480b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54480b36e0 0x7f54480ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5448141a80 0x7f5448150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544813e8d0 con 0x7f5448064b10 2022-07-21T01:36:45.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5448153800 con 0x7f54480b36e0 2022-07-21T01:36:45.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5448153a60 con 0x7f5448141a80 2022-07-21T01:36:45.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f5447fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448064b10 0x7f54480b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f5447fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448064b10 0x7f54480b7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45194/0 (socket says 172.21.15.3:45194) 2022-07-21T01:36:45.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f5447fff700 1 -- 172.21.15.3:0/2825828946 learned_addr learned my addr 172.21.15.3:0/2825828946 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:45.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544cc4e700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5448141a80 0x7f5448150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f54477fe700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54480b36e0 0x7f54480ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f5447fff700 1 -- 172.21.15.3:0/2825828946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5448141a80 msgr2=0x7f5448150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f5447fff700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5448141a80 0x7f5448150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f5447fff700 1 -- 172.21.15.3:0/2825828946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54480b36e0 msgr2=0x7f54480ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f5447fff700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54480b36e0 0x7f54480ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f5447fff700 1 -- 172.21.15.3:0/2825828946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544000b1e0 con 0x7f5448064b10 2022-07-21T01:36:45.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f5447fff700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448064b10 0x7f54480b7e10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f543c00c000 tx=0x7f543c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:45.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f54457fa700 1 -- 172.21.15.3:0/2825828946 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f543c010780 con 0x7f5448064b10 2022-07-21T01:36:45.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f54457fa700 1 -- 172.21.15.3:0/2825828946 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c0108e0 con 0x7f5448064b10 2022-07-21T01:36:45.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5448153d00 con 0x7f5448064b10 2022-07-21T01:36:45.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f54457fa700 1 -- 172.21.15.3:0/2825828946 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f543c010c40 con 0x7f5448064b10 2022-07-21T01:36:45.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.117+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5448154440 con 0x7f5448064b10 2022-07-21T01:36:45.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.121+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5448153e90 con 0x7f5448064b10 2022-07-21T01:36:45.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.121+0000 7f54457fa700 1 -- 172.21.15.3:0/2825828946 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f543c01c460 con 0x7f5448064b10 2022-07-21T01:36:45.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.121+0000 7f54457fa700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54280539e0 0x7f5428055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.121+0000 7f54457fa700 1 -- 172.21.15.3:0/2825828946 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f543c06c2b0 con 0x7f5448064b10 2022-07-21T01:36:45.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.121+0000 7f54477fe700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54280539e0 0x7f5428055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.121+0000 7f54477fe700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54280539e0 0x7f5428055ea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f543800b000 tx=0x7f5438009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:45.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.125+0000 7f54457fa700 1 -- 172.21.15.3:0/2825828946 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f543c0432f0 con 0x7f5448064b10 2022-07-21T01:36:45.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.297+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7f54480b4af0 con 0x7f5448064b10 2022-07-21T01:36:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.721+0000 7f54457fa700 1 -- 172.21.15.3:0/2825828946 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v155) v1 ==== 144+0+0 (secure 0 0 0) 0x7f543c046910 con 0x7f5448064b10 2022-07-21T01:36:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54280539e0 msgr2=0x7f5428055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54280539e0 0x7f5428055ea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f543800b000 tx=0x7f5438009040).stop 2022-07-21T01:36:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448064b10 msgr2=0x7f54480b7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448064b10 0x7f54480b7e10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f543c00c000 tx=0x7f543c006df0).stop 2022-07-21T01:36:45.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 shutdown_connections 2022-07-21T01:36:45.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5448141a80 0x7f5448150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54280539e0 0x7f5428055ea0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54480b36e0 0x7f54480ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 --2- 172.21.15.3:0/2825828946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5448064b10 0x7f54480b7e10 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 >> 172.21.15.3:0/2825828946 conn(0x7f544805f840 msgr2=0x7f54480b5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:45.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 shutdown_connections 2022-07-21T01:36:45.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.729+0000 7f544d44f700 1 -- 172.21.15.3:0/2825828946 wait complete. 2022-07-21T01:36:45.743 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-21T01:36:45.743 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-21T01:36:45.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-07-21T01:36:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faafcd00700 1 Processor -- start 2022-07-21T01:36:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faafcd00700 1 -- start start 2022-07-21T01:36:45.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faafcd00700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80a7840 0x7faaf8066c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faafcd00700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faaf80b01f0 0x7faaf805ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faafcd00700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf80a7c20 con 0x7faaf80b1b00 2022-07-21T01:36:45.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faafcd00700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf81477b0 con 0x7faaf80b01f0 2022-07-21T01:36:45.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faafcd00700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8147a50 con 0x7faaf80a7840 2022-07-21T01:36:45.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf6d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faaf80b01f0 0x7faaf805ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf6d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faaf80b01f0 0x7faaf805ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45206/0 (socket says 172.21.15.3:45206) 2022-07-21T01:36:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf6d9d700 1 -- 172.21.15.3:0/1506486412 learned_addr learned my addr 172.21.15.3:0/1506486412 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf659c700 1 --2- 172.21.15.3:0/1506486412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80a7840 0x7faaf8066c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf559a700 1 -- 172.21.15.3:0/1506486412 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631232176 0 0) 0x7faaf80a7c20 con 0x7faaf80b1b00 2022-07-21T01:36:45.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf6d9d700 1 -- 172.21.15.3:0/1506486412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80a7840 msgr2=0x7faaf8066c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:45.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf6d9d700 1 --2- 172.21.15.3:0/1506486412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80a7840 0x7faaf8066c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf6d9d700 1 -- 172.21.15.3:0/1506486412 >> v1:172.21.15.143:6789/0 conn(0x7faaf80b1b00 legacy=0x7faaf805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf6d9d700 1 -- 172.21.15.3:0/1506486412 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf8147cf0 con 0x7faaf80b01f0 2022-07-21T01:36:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf6d9d700 1 --2- 172.21.15.3:0/1506486412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80b01f0 0x7faaf805ff50 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7faaf0002f70 tx=0x7faaf0006eb0).ready entity=mon.0 client_cookie=d71682c1316e3d8d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf559a700 1 -- 172.21.15.3:0/1506486412 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaf000d570 con 0x7faaf80b01f0 2022-07-21T01:36:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf559a700 1 -- 172.21.15.3:0/1506486412 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf0000e90 con 0x7faaf80b01f0 2022-07-21T01:36:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faaf559a700 1 -- 172.21.15.3:0/1506486412 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaf000d8c0 con 0x7faaf80b01f0 2022-07-21T01:36:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faafcd00700 1 -- 172.21.15.3:0/1506486412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80b01f0 msgr2=0x7faaf805ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.849+0000 7faafcd00700 1 --2- 172.21.15.3:0/1506486412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80b01f0 0x7faaf805ff50 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7faaf0002f70 tx=0x7faaf0006eb0).stop 2022-07-21T01:36:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 -- 172.21.15.3:0/1506486412 shutdown_connections 2022-07-21T01:36:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 --2- 172.21.15.3:0/1506486412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80b01f0 0x7faaf805ff50 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 --2- 172.21.15.3:0/1506486412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80a7840 0x7faaf8066c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 -- 172.21.15.3:0/1506486412 >> 172.21.15.3:0/1506486412 conn(0x7faaf80b5570 msgr2=0x7faaf80b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 -- 172.21.15.3:0/1506486412 shutdown_connections 2022-07-21T01:36:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 -- 172.21.15.3:0/1506486412 wait complete. 2022-07-21T01:36:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 Processor -- start 2022-07-21T01:36:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 -- start start 2022-07-21T01:36:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80a7840 0x7faaf80667a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80b01f0 0x7faaf8066c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaf80b1b00 0x7faaf80a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80ad6a0 con 0x7faaf80a7840 2022-07-21T01:36:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8157520 con 0x7faaf80b1b00 2022-07-21T01:36:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faafcd00700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80ad8d0 con 0x7faaf80b01f0 2022-07-21T01:36:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faaf659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80a7840 0x7faaf80667a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faaf659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80a7840 0x7faaf80667a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45208/0 (socket says 172.21.15.3:45208) 2022-07-21T01:36:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faaf659c700 1 -- 172.21.15.3:0/880556405 learned_addr learned my addr 172.21.15.3:0/880556405 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faaf6d9d700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaf80b1b00 0x7faaf80a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.853+0000 7faaf5d9b700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80b01f0 0x7faaf8066c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaf659c700 1 -- 172.21.15.3:0/880556405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80b01f0 msgr2=0x7faaf8066c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaf659c700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80b01f0 0x7faaf8066c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaf659c700 1 -- 172.21.15.3:0/880556405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaf80b1b00 msgr2=0x7faaf80a9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaf659c700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaf80b1b00 0x7faaf80a9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaf659c700 1 -- 172.21.15.3:0/880556405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf000b1e0 con 0x7faaf80a7840 2022-07-21T01:36:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaf659c700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80a7840 0x7faaf80667a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7faae000c000 tx=0x7faae0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaef7fe700 1 -- 172.21.15.3:0/880556405 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faae0011750 con 0x7faaf80a7840 2022-07-21T01:36:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaef7fe700 1 -- 172.21.15.3:0/880556405 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae00118b0 con 0x7faaf80a7840 2022-07-21T01:36:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf80adb70 con 0x7faaf80a7840 2022-07-21T01:36:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaef7fe700 1 -- 172.21.15.3:0/880556405 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faae0011c10 con 0x7faaf80a7840 2022-07-21T01:36:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf80ae010 con 0x7faaf80a7840 2022-07-21T01:36:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf80574e0 con 0x7faaf80a7840 2022-07-21T01:36:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaef7fe700 1 -- 172.21.15.3:0/880556405 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faae0011d70 con 0x7faaf80a7840 2022-07-21T01:36:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaef7fe700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faadc0539e0 0x7faadc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.857+0000 7faaef7fe700 1 -- 172.21.15.3:0/880556405 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5686+0+0 (secure 0 0 0) 0x7faae006c130 con 0x7faaf80a7840 2022-07-21T01:36:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.861+0000 7faaf5d9b700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faadc0539e0 0x7faadc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.861+0000 7faaf5d9b700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faadc0539e0 0x7faadc055ea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7faae800d000 tx=0x7faae8007d80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:45.865+0000 7faaef7fe700 1 -- 172.21.15.3:0/880556405 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faae0043170 con 0x7faaf80a7840 2022-07-21T01:36:46.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.033+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7faaf8069a70 con 0x7faaf80a7840 2022-07-21T01:36:46.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.725+0000 7faaef7fe700 1 -- 172.21.15.3:0/880556405 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v156) v1 ==== 155+0+0 (secure 0 0 0) 0x7faae0046790 con 0x7faaf80a7840 2022-07-21T01:36:46.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faadc0539e0 msgr2=0x7faadc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:46.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faadc0539e0 0x7faadc055ea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7faae800d000 tx=0x7faae8007d80).stop 2022-07-21T01:36:46.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80a7840 msgr2=0x7faaf80667a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:46.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80a7840 0x7faaf80667a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7faae000c000 tx=0x7faae0006df0).stop 2022-07-21T01:36:46.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 shutdown_connections 2022-07-21T01:36:46.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faadc0539e0 0x7faadc055ea0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:46.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaf80b1b00 0x7faaf80a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf80b01f0 0x7faaf8066c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 --2- 172.21.15.3:0/880556405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf80a7840 0x7faaf80667a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 >> 172.21.15.3:0/880556405 conn(0x7faaf80b5570 msgr2=0x7faaf81448d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 shutdown_connections 2022-07-21T01:36:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.733+0000 7faafcd00700 1 -- 172.21.15.3:0/880556405 wait complete. 2022-07-21T01:36:46.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:36:46.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.861+0000 7f2d925d8700 1 Processor -- start 2022-07-21T01:36:46.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.861+0000 7f2d925d8700 1 -- start start 2022-07-21T01:36:46.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2d8c146ae0 0x7f2d8c1403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:46.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c0b4310 0x7f2d8c1408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:46.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c143cf0 con 0x7f2d8c0b4820 2022-07-21T01:36:46.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8c1476a0 con 0x7f2d8c146ae0 2022-07-21T01:36:46.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c143f90 con 0x7f2d8c0b4310 2022-07-21T01:36:46.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c0b4310 0x7f2d8c1408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c0b4310 0x7f2d8c1408b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:52136/0 (socket says 172.21.15.3:52136) 2022-07-21T01:36:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d90b75700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f2d8c0b4820 0x7f2d8c0b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:55612/0 (socket says 172.21.15.3:55612) 2022-07-21T01:36:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2d8c146ae0 0x7f2d8c1403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8b7fe700 1 -- 172.21.15.3:0/3834869739 learned_addr learned my addr 172.21.15.3:0/3834869739 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8b7fe700 1 -- 172.21.15.3:0/3834869739 >> v2:172.21.15.143:3300/0 conn(0x7f2d8c146ae0 msgr2=0x7f2d8c1403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8b7fe700 1 --2- 172.21.15.3:0/3834869739 >> v2:172.21.15.143:3300/0 conn(0x7f2d8c146ae0 0x7f2d8c1403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8b7fe700 1 -- 172.21.15.3:0/3834869739 >> v1:172.21.15.143:6789/0 conn(0x7f2d8c0b4820 legacy=0x7f2d8c0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8b7fe700 1 -- 172.21.15.3:0/3834869739 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0b0e20 con 0x7f2d8c0b4310 2022-07-21T01:36:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8affd700 1 -- 172.21.15.3:0/3834869739 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806195766 0 0) 0x7f2d8c143cf0 con 0x7f2d8c0b4820 2022-07-21T01:36:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8b7fe700 1 --2- 172.21.15.3:0/3834869739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c0b4310 0x7f2d8c1408b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f2d80002f70 tx=0x7f2d80006eb0).ready entity=mon.2 client_cookie=454d8c30b728a534 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8affd700 1 -- 172.21.15.3:0/3834869739 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8000d570 con 0x7f2d8c0b4310 2022-07-21T01:36:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8affd700 1 -- 172.21.15.3:0/3834869739 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d80000e90 con 0x7f2d8c0b4310 2022-07-21T01:36:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d8affd700 1 -- 172.21.15.3:0/3834869739 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8000d8c0 con 0x7f2d8c0b4310 2022-07-21T01:36:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 -- 172.21.15.3:0/3834869739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c0b4310 msgr2=0x7f2d8c1408b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 --2- 172.21.15.3:0/3834869739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c0b4310 0x7f2d8c1408b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f2d80002f70 tx=0x7f2d80006eb0).stop 2022-07-21T01:36:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 -- 172.21.15.3:0/3834869739 shutdown_connections 2022-07-21T01:36:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 --2- 172.21.15.3:0/3834869739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c0b4310 0x7f2d8c1408b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 --2- 172.21.15.3:0/3834869739 >> v2:172.21.15.143:3300/0 conn(0x7f2d8c146ae0 0x7f2d8c1403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:46.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 -- 172.21.15.3:0/3834869739 >> 172.21.15.3:0/3834869739 conn(0x7f2d8c05c620 msgr2=0x7f2d8c05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:46.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 -- 172.21.15.3:0/3834869739 shutdown_connections 2022-07-21T01:36:46.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.865+0000 7f2d925d8700 1 -- 172.21.15.3:0/3834869739 wait complete. 2022-07-21T01:36:46.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d925d8700 1 Processor -- start 2022-07-21T01:36:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d925d8700 1 -- start start 2022-07-21T01:36:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d925d8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d8c0b4310 0x7f2d8c0b8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d925d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d8c0b4820 0x7f2d8c142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d925d8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c146ae0 0x7f2d8c0ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:46.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d925d8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c153420 con 0x7f2d8c0b4310 2022-07-21T01:36:46.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d925d8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c1434c0 con 0x7f2d8c0b4820 2022-07-21T01:36:46.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d925d8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c1436e0 con 0x7f2d8c146ae0 2022-07-21T01:36:46.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d8c0b4310 0x7f2d8c0b8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:46.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d8c0b4310 0x7f2d8c0b8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45222/0 (socket says 172.21.15.3:45222) 2022-07-21T01:36:46.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8bfff700 1 -- 172.21.15.3:0/3631497618 learned_addr learned my addr 172.21.15.3:0/3631497618 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:46.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8b7fe700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d8c0b4820 0x7f2d8c142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:46.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d90b75700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c146ae0 0x7f2d8c0ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:46.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8bfff700 1 -- 172.21.15.3:0/3631497618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c146ae0 msgr2=0x7f2d8c0ad780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:46.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8bfff700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c146ae0 0x7f2d8c0ad780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:46.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8bfff700 1 -- 172.21.15.3:0/3631497618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d8c0b4820 msgr2=0x7f2d8c142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:46.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8bfff700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d8c0b4820 0x7f2d8c142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:46.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8bfff700 1 -- 172.21.15.3:0/3631497618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8000b1e0 con 0x7f2d8c0b4310 2022-07-21T01:36:46.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.869+0000 7f2d8bfff700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d8c0b4310 0x7f2d8c0b8820 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2d8c143600 tx=0x7f2d7c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:46.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.873+0000 7f2d897fa700 1 -- 172.21.15.3:0/3631497618 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d7c010780 con 0x7f2d8c0b4310 2022-07-21T01:36:46.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.873+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c0ae360 con 0x7f2d8c0b4310 2022-07-21T01:36:46.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.873+0000 7f2d897fa700 1 -- 172.21.15.3:0/3631497618 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c0108e0 con 0x7f2d8c0b4310 2022-07-21T01:36:46.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.873+0000 7f2d897fa700 1 -- 172.21.15.3:0/3631497618 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d7c010c40 con 0x7f2d8c0b4310 2022-07-21T01:36:46.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.873+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c1523b0 con 0x7f2d8c0b4310 2022-07-21T01:36:46.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.873+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c142a50 con 0x7f2d8c0b4310 2022-07-21T01:36:46.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.873+0000 7f2d897fa700 1 -- 172.21.15.3:0/3631497618 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2d7c01c460 con 0x7f2d8c0b4310 2022-07-21T01:36:46.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.877+0000 7f2d897fa700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d6c053950 0x7f2d6c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:46.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.877+0000 7f2d897fa700 1 -- 172.21.15.3:0/3631497618 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f2d7c06ca40 con 0x7f2d8c0b4310 2022-07-21T01:36:46.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.877+0000 7f2d8b7fe700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d6c053950 0x7f2d6c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:46.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.877+0000 7f2d8b7fe700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d6c053950 0x7f2d6c055e10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f2d8001c000 tx=0x7f2d80015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:46.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:46.881+0000 7f2d897fa700 1 -- 172.21.15.3:0/3631497618 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d7c043a80 con 0x7f2d8c0b4310 2022-07-21T01:36:47.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.009+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2d8c142a50 con 0x7f2d8c0b4310 2022-07-21T01:36:47.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.017+0000 7f2d897fa700 1 -- 172.21.15.3:0/3631497618 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2d8c142a50 con 0x7f2d8c0b4310 2022-07-21T01:36:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d6c053950 msgr2=0x7f2d6c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d6c053950 0x7f2d6c055e10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f2d8001c000 tx=0x7f2d80015000).stop 2022-07-21T01:36:47.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d8c0b4310 msgr2=0x7f2d8c0b8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d8c0b4310 0x7f2d8c0b8820 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2d8c143600 tx=0x7f2d7c006df0).stop 2022-07-21T01:36:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 shutdown_connections 2022-07-21T01:36:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d8c146ae0 0x7f2d8c0ad780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d6c053950 0x7f2d6c055e10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d8c0b4820 0x7f2d8c142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 --2- 172.21.15.3:0/3631497618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d8c0b4310 0x7f2d8c0b8820 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 >> 172.21.15.3:0/3631497618 conn(0x7f2d8c05c620 msgr2=0x7f2d8c0b61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:47.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 shutdown_connections 2022-07-21T01:36:47.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.021+0000 7f2d925d8700 1 -- 172.21.15.3:0/3631497618 wait complete. 2022-07-21T01:36:47.033 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:36:47.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:36:47.054 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:36:47.076 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:36:47.097 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:36:47.119 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:36:47.120 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:47.120 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:36:47.121 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:47.128 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:36:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:36:47.129 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:36:47.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:47.131 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:36:47.131 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:36:47.138 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:36:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:36:47.140 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:36:47.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:36:47.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda77fff700 1 Processor -- start 2022-07-21T01:36:47.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda77fff700 1 -- start start 2022-07-21T01:36:47.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda77fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda780614b0 0x7fda78061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda77fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda780aee80 con 0x7fda780ad750 2022-07-21T01:36:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda77fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda780b70b0 con 0x7fda781405d0 2022-07-21T01:36:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda77fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda780af0b0 con 0x7fda780614b0 2022-07-21T01:36:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda767fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fda780ad750 0x7fda780b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53512/0 (socket says 172.21.15.3:53512) 2022-07-21T01:36:47.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda767fc700 1 -- 172.21.15.3:0/1188321097 learned_addr learned my addr 172.21.15.3:0/1188321097 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:47.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda76ffd700 1 --2- 172.21.15.3:0/1188321097 >> v2:172.21.15.143:3300/0 conn(0x7fda780614b0 0x7fda78061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675698707 0 0) 0x7fda780aee80 con 0x7fda780ad750 2022-07-21T01:36:47.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda58003260 con 0x7fda780ad750 2022-07-21T01:36:47.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1478508695 0 0) 0x7fda58003260 con 0x7fda780ad750 2022-07-21T01:36:47.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 >> v2:172.21.15.143:3300/0 conn(0x7fda780614b0 msgr2=0x7fda78061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:47.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 --2- 172.21.15.3:0/1188321097 >> v2:172.21.15.143:3300/0 conn(0x7fda780614b0 0x7fda78061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 >> v1:172.21.15.143:6789/0 conn(0x7fda781405d0 legacy=0x7fda781466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda780af350 con 0x7fda780ad750 2022-07-21T01:36:47.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fda60001660 con 0x7fda780ad750 2022-07-21T01:36:47.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 >> v1:172.21.15.3:6789/0 conn(0x7fda780ad750 legacy=0x7fda780b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.245+0000 7fda75ffb700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda58003750 0x7fda58003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda75ffb700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda58004500 0x7fda58004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda75ffb700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda5800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda780af0b0 con 0x7fda58004e70 2022-07-21T01:36:47.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5800bd20 con 0x7fda58004500 2022-07-21T01:36:47.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5800bee0 con 0x7fda58003750 2022-07-21T01:36:47.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda777fe700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda5800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda767fc700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda58004500 0x7fda58004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda76ffd700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda58003750 0x7fda58003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda777fe700 1 -- 172.21.15.3:0/1188321097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda58003750 msgr2=0x7fda58003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:47.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda777fe700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda58003750 0x7fda58003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda777fe700 1 -- 172.21.15.3:0/1188321097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda58004500 msgr2=0x7fda58004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda777fe700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda58004500 0x7fda58004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda777fe700 1 -- 172.21.15.3:0/1188321097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda58003260 con 0x7fda58004e70 2022-07-21T01:36:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda76ffd700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda58003750 0x7fda58003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda777fe700 1 -- 172.21.15.3:0/1188321097 reap_dead start 2022-07-21T01:36:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda777fe700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda5800b750 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fda780af970 tx=0x7fda70007a00).ready entity=mon.0 client_cookie=cef6b31a90f9b9f1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:47.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda70007d10 con 0x7fda58004e70 2022-07-21T01:36:47.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda70007030 con 0x7fda58004e70 2022-07-21T01:36:47.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda75ffb700 1 -- 172.21.15.3:0/1188321097 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda70007390 con 0x7fda58004e70 2022-07-21T01:36:47.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda77fff700 1 -- 172.21.15.3:0/1188321097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 msgr2=0x7fda5800b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda77fff700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda5800b750 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fda780af970 tx=0x7fda70007a00).stop 2022-07-21T01:36:47.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda77fff700 1 -- 172.21.15.3:0/1188321097 shutdown_connections 2022-07-21T01:36:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda77fff700 1 --2- 172.21.15.3:0/1188321097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda5800b750 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda77fff700 1 -- 172.21.15.3:0/1188321097 >> 172.21.15.3:0/1188321097 conn(0x7fda78066f70 msgr2=0x7fda78067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:47.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda77fff700 1 -- 172.21.15.3:0/1188321097 shutdown_connections 2022-07-21T01:36:47.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda77fff700 1 -- 172.21.15.3:0/1188321097 wait complete. 2022-07-21T01:36:47.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda77fff700 1 Processor -- start 2022-07-21T01:36:47.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.249+0000 7fda77fff700 1 -- start start 2022-07-21T01:36:47.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda77fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda78142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda77fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda7809ea40 0x7fda78156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda77fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda78142fc0 0x7fda78143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda77fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7809f3f0 con 0x7fda58004e70 2022-07-21T01:36:47.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda77fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7809eec0 con 0x7fda7809ea40 2022-07-21T01:36:47.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda77fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7809f150 con 0x7fda78142fc0 2022-07-21T01:36:47.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda76ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda78142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda76ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda78142040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45242/0 (socket says 172.21.15.3:45242) 2022-07-21T01:36:47.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda76ffd700 1 -- 172.21.15.3:0/3846928823 learned_addr learned my addr 172.21.15.3:0/3846928823 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:47.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda777fe700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda78142fc0 0x7fda78143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda767fc700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda7809ea40 0x7fda78156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda76ffd700 1 -- 172.21.15.3:0/3846928823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda78142fc0 msgr2=0x7fda78143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda76ffd700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda78142fc0 0x7fda78143470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda76ffd700 1 -- 172.21.15.3:0/3846928823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda7809ea40 msgr2=0x7fda78156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda76ffd700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda7809ea40 0x7fda78156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda76ffd700 1 -- 172.21.15.3:0/3846928823 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda78155ee0 con 0x7fda58004e70 2022-07-21T01:36:47.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda76ffd700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda78142040 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fda680032b0 tx=0x7fda68008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:47.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda6ffff700 1 -- 172.21.15.3:0/3846928823 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda680136f0 con 0x7fda58004e70 2022-07-21T01:36:47.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda78156180 con 0x7fda58004e70 2022-07-21T01:36:47.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda780b0ce0 con 0x7fda58004e70 2022-07-21T01:36:47.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda6ffff700 1 -- 172.21.15.3:0/3846928823 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda68013850 con 0x7fda58004e70 2022-07-21T01:36:47.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda6ffff700 1 -- 172.21.15.3:0/3846928823 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda68013bb0 con 0x7fda58004e70 2022-07-21T01:36:47.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda78156310 con 0x7fda58004e70 2022-07-21T01:36:47.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda6ffff700 1 -- 172.21.15.3:0/3846928823 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fda68012040 con 0x7fda58004e70 2022-07-21T01:36:47.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda6ffff700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda5c053b30 0x7fda5c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.253+0000 7fda6ffff700 1 -- 172.21.15.3:0/3846928823 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fda6806c730 con 0x7fda58004e70 2022-07-21T01:36:47.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.257+0000 7fda767fc700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda5c053b30 0x7fda5c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.257+0000 7fda767fc700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda5c053b30 0x7fda5c055ff0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fda60005000 tx=0x7fda780af4e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:47.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.261+0000 7fda6ffff700 1 -- 172.21.15.3:0/3846928823 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fda680435e0 con 0x7fda58004e70 2022-07-21T01:36:47.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.389+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fda780af5f0 con 0x7fda58004e70 2022-07-21T01:36:47.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.389+0000 7fda6ffff700 1 -- 172.21.15.3:0/3846928823 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v51) v1 ==== 81+0+29217 (secure 0 0 0) 0x7fda68046c00 con 0x7fda58004e70 2022-07-21T01:36:47.391 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:47.392 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:36:47.392 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 51, 2022-07-21T01:36:47.392 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:36:47.392 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:36:47.393 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:36:44.702974+0000", 2022-07-21T01:36:47.393 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:36:47.393 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:36:47.393 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:36:47.394 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:36:47.394 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:36:47.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:36:47.394 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:36:47.395 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:36:47.395 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:36:47.395 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:36:47.395 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:36:47.396 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:36:47.396 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:36:47.396 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:36:47.396 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:36:47.396 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 8, 2022-07-21T01:36:47.397 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:36:47.397 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:36:47.397 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:36:47.397 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:36:47.398 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:36:47.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.398 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:36:47.398 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:36:47.399 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:36:47.399 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:36:47.399 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:36:47.399 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:36:47.399 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:36:47.400 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:36:47.400 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:36:47.400 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:36:47.400 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:36:47.401 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:36:47.401 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:36:47.401 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:36:47.401 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:36:47.402 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:36:47.402 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:36:47.402 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:36:47.402 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:36:47.403 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:36:47.403 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:36:47.403 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:36:47.403 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:36:47.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.404 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:36:47.404 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:36:47.404 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:36:47.404 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:36:47.405 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:36:47.405 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:36:47.405 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:36:47.405 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:36:47.406 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:36:47.406 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:36:47.406 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:36:47.406 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:36:47.406 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:36:47.407 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:36:47.407 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:36:47.407 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:36:47.407 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:36:47.408 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:36:47.408 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:36:47.408 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:36:47.408 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:36:47.408 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:36:47.409 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:36:47.409 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:36:47.409 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:36:47.409 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:36:47.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:36:47.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.410 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:36:47.410 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:36:47.411 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:36:47.411 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:36:47.411 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:36:47.411 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:36:47.412 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:36:47.412 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:36:47.412 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:36:47.412 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:36:47.413 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:36:47.413 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:36:47.413 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:36:47.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.413 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:36:47.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:36:47.414 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.415 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:36:47.415 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:36:47.415 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:36:47.415 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:36:47.416 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:36:47.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:36:47.416 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:36:47.416 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:36:47.417 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:36:47.417 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:36:47.417 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:36:47.417 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:36:47.418 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:36:47.418 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:36:47.418 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:36:47.418 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:36:47.418 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:36:47.419 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:36:47.419 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:36:47.419 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:36:47.419 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:36:47.420 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:36:47.420 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:36:47.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.420 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:36:47.421 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:36:47.421 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:36:47.421 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:36:47.421 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:36:47.421 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:36:47.422 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:36:47.422 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:36:47.422 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:36:47.423 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:36:47.423 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:36:47.423 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:36:47.423 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:36:47.424 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:36:47.424 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:36:47.424 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:36:47.424 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:36:47.425 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:36:47.425 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:36:47.425 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:36:47.425 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:36:47.426 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:36:47.426 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:36:47.426 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:36:47.427 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:36:47.427 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:36:47.427 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:36:47.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.428 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:36:47.428 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:36:47.428 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:36:47.428 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:36:47.429 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:36:47.429 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:36:47.429 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:36:47.429 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:36:47.430 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:36:47.430 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:36:47.430 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:36:47.431 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:36:47.431 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:36:47.431 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:36:47.432 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.432 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.433 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:36:47.433 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:36:47.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.434 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:36:47.435 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:36:47.435 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:47.436 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:47.436 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:47.437 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:47.437 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:47.438 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:47.438 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:36:47.439 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:47.439 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:47.439 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:47.440 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:47.440 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.441 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.442 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:36:47.442 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:47.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.444 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:36:47.445 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:47.445 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.446 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.447 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:47.447 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.448 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.449 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:36:47.449 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:47.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.450 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:36:47.450 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:47.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.451 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.451 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:47.452 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.452 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.452 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:36:47.453 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:47.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.454 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:36:47.455 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:47.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.456 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.457 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:47.457 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.458 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:36:47.459 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:47.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:36:47.460 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:36:47.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.461 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.461 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:36:47.462 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:36:47.462 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:36:47.462 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:36:47.462 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:47.462 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:36:47.465 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:36:47.466 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:47.466 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:47.467 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:47.467 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:47.468 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:47.468 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:47.468 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:47.469 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 46, 2022-07-21T01:36:47.469 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:47.470 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:47.470 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:47.470 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.471 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.471 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:36:47.471 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:47.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:36:47.473 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:47.473 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.474 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.474 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:47.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:36:47.477 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:47.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.479 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:36:47.479 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:47.479 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.480 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.480 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:47.480 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.481 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.482 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:36:47.482 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:47.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.483 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.483 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:36:47.484 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:47.484 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.484 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.485 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:47.485 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.486 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:36:47.486 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:47.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.487 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:36:47.487 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:36:47.487 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.487 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:36:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:36:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:36:47.489 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:36:47.489 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:47.489 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:47.489 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:47.490 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.490 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:36:47.491 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:36:47.491 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:47.491 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:47.491 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:47.491 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:47.492 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:47.492 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:47.492 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:36:47.492 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 34, 2022-07-21T01:36:47.493 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:47.493 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:47.493 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:47.493 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:36:47.494 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:47.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.495 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.495 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:36:47.495 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:47.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.496 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.496 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:47.496 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.497 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.497 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:36:47.497 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:47.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.498 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.498 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:36:47.499 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:47.499 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.499 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.500 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:47.500 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.500 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.500 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:36:47.501 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:47.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.501 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.502 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:36:47.502 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:47.502 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.502 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.503 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:47.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:36:47.504 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:47.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.505 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.505 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:36:47.505 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:36:47.505 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.506 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.506 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:36:47.506 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:36:47.506 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:36:47.507 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:36:47.507 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:47.507 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:47.507 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:47.508 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.508 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:36:47.508 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:36:47.509 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:47.509 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:47.509 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:47.509 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:47.510 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:47.510 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:47.510 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:36:47.510 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:47.511 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:47.511 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:47.511 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:47.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.512 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.512 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:36:47.512 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:47.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.513 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.513 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:36:47.513 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:47.514 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.514 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.514 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:47.514 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.515 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.515 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:36:47.515 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:47.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.516 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.516 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:36:47.517 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:47.517 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.517 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.518 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:47.518 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.518 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.518 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:36:47.519 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:47.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.519 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.520 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:36:47.520 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:47.520 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.520 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.521 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:47.521 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.521 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.522 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:36:47.522 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:47.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.523 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.523 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:36:47.523 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:36:47.523 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.523 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.524 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:36:47.524 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:36:47.524 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:36:47.525 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:36:47.525 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:47.525 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:47.525 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:47.526 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.526 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:36:47.526 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:36:47.527 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:47.527 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:47.527 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:47.527 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:47.528 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:47.528 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:47.528 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:47.528 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:47.529 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:47.529 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:47.529 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:47.529 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.530 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.530 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:36:47.530 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:47.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.531 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:36:47.531 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:47.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.532 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.532 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:47.532 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:36:47.533 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:47.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:36:47.535 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:47.535 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.535 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.536 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:47.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.536 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:36:47.537 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:47.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.537 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.538 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:36:47.538 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:47.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.538 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.539 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:47.539 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.539 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.540 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:36:47.540 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:47.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.541 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.541 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:36:47.541 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:36:47.541 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.541 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.542 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:36:47.542 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:36:47.542 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:36:47.543 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:36:47.543 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:47.543 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:47.543 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:47.544 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.544 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:36:47.544 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:36:47.545 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:47.545 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:47.545 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:47.545 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:47.546 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:47.546 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:47.546 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:47.546 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:47.547 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:47.547 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:47.547 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:47.547 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.548 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:36:47.548 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:47.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.549 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.549 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:36:47.549 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:47.550 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.550 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.550 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:47.550 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.551 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.551 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:36:47.551 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:47.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.552 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.552 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:36:47.553 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:47.553 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.553 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.553 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:47.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.554 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:36:47.555 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:47.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.555 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.556 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:36:47.556 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:47.556 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.556 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.557 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:47.557 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.558 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:36:47.558 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:47.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.559 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:36:47.559 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:36:47.559 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.559 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.560 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:36:47.560 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:36:47.560 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:36:47.560 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:36:47.561 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:47.561 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:47.561 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:47.561 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.562 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:36:47.562 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:36:47.563 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:47.563 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:47.563 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:47.563 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:47.564 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:47.564 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:47.564 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:47.564 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 45, 2022-07-21T01:36:47.565 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:47.565 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:47.565 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:47.565 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.566 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.566 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:36:47.566 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:47.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.567 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.567 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:36:47.567 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:47.568 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.568 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.569 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:47.569 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.570 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.570 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:36:47.570 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:47.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.571 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.571 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:36:47.571 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:47.571 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.572 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.572 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:47.572 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.573 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.573 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:36:47.574 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:47.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.575 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:36:47.575 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:47.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.576 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.576 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:47.576 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.577 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.577 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:36:47.577 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:47.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.578 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:36:47.579 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:36:47.579 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.579 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:36:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:36:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:36:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:36:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:36:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:36:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:36:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:36:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:36:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:36:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:36:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:36:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:36:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 44, 2022-07-21T01:36:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:36:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:36:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:36:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:36:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:36:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:36:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:36:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:36:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:36:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:36:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:36:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:36:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:36:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:36:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:36:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:36:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:36:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:36:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:36:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:36:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:36:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:36:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:36:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:36:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:36:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:36:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:36:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:36:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:47.604 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:36:47.604 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:47.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.604 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:36:47.605 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:47.605 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:47.605 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:47.605 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:47.606 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:47.606 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:36:47.606 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:47.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.607 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:36:47.607 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:47.608 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:47.608 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:47.608 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:47.609 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:47.609 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:36:47.609 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:47.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.610 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:36:47.611 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:47.611 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:47.611 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:47.611 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:47.612 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:47.612 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:36:47.612 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:47.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.614 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:36:47.614 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:47.615 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:47.615 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:47.616 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:47.616 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:47.617 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:36:47.617 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:47.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.619 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:36:47.619 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:47.620 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:47.620 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:47.621 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:47.621 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:47.622 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:36:47.622 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:47.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.623 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:36:47.624 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:47.624 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:47.625 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:47.625 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:47.626 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:47.626 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:36:47.627 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:47.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:47.628 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:36:47.629 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:36:47.629 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:36:47.630 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:36:47.630 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:36:47.631 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:36:47.631 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:36:47.632 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:36:47.632 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.633 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:36:47.633 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:36:47.633 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:36:47.634 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:36:47.634 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:36:47.635 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:36:47.635 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6837/2892507714": "2022-07-22T01:36:40.913360+0000", 2022-07-21T01:36:47.636 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6836/2892507714": "2022-07-22T01:36:40.913360+0000" 2022-07-21T01:36:47.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.637 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:36:47.637 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:36:47.637 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:36:47.638 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:36:47.638 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:36:47.638 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:36:47.638 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:36:47.639 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:36:47.639 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:36:47.639 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:36:47.640 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:36:47.640 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:36:47.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.640 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:36:47.641 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:36:47.641 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:36:47.641 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:36:47.642 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:36:47.642 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:36:47.642 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:36:47.642 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:47.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:47.643 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:36:47.643 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:36:47.643 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:36:47.644 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:36:47.644 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:36:47.644 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:36:47.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda5c053b30 msgr2=0x7fda5c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda5c053b30 0x7fda5c055ff0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fda60005000 tx=0x7fda780af4e0).stop 2022-07-21T01:36:47.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 msgr2=0x7fda78142040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda78142040 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fda680032b0 tx=0x7fda68008040).stop 2022-07-21T01:36:47.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 shutdown_connections 2022-07-21T01:36:47.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda78142fc0 0x7fda78143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda5c053b30 0x7fda5c055ff0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda7809ea40 0x7fda78156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 --2- 172.21.15.3:0/3846928823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda58004e70 0x7fda78142040 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 >> 172.21.15.3:0/3846928823 conn(0x7fda78066f70 msgr2=0x7fda780b8980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:47.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 shutdown_connections 2022-07-21T01:36:47.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.393+0000 7fda77fff700 1 -- 172.21.15.3:0/3846928823 wait complete. 2022-07-21T01:36:47.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:36:47.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.761+0000 7f17b3fff700 1 Processor -- start 2022-07-21T01:36:47.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.761+0000 7f17b3fff700 1 -- start start 2022-07-21T01:36:47.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f17b40a92c0 0x7f17b4140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b3fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b40af930 con 0x7f17b40b0a70 2022-07-21T01:36:47.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b3fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b40b0f80 con 0x7f17b405c620 2022-07-21T01:36:47.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b3fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17b41413e0 con 0x7f17b40a92c0 2022-07-21T01:36:47.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b27fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f17b40b0a70 0x7f17b40b2df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53532/0 (socket says 172.21.15.3:53532) 2022-07-21T01:36:47.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b27fc700 1 -- 172.21.15.3:0/961364169 learned_addr learned my addr 172.21.15.3:0/961364169 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:47.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b2ffd700 1 --2- 172.21.15.3:0/961364169 >> v2:172.21.15.143:3300/0 conn(0x7f17b40a92c0 0x7f17b4140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b1ffb700 1 -- 172.21.15.3:0/961364169 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421339867 0 0) 0x7f17b40af930 con 0x7f17b40b0a70 2022-07-21T01:36:47.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b1ffb700 1 -- 172.21.15.3:0/961364169 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1798003260 con 0x7f17b40b0a70 2022-07-21T01:36:47.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b2ffd700 1 -- 172.21.15.3:0/961364169 >> v1:172.21.15.143:6789/0 conn(0x7f17b405c620 legacy=0x7f17b40af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b1ffb700 1 -- 172.21.15.3:0/961364169 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2674276890 0 0) 0x7f1798003260 con 0x7f17b40b0a70 2022-07-21T01:36:47.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b2ffd700 1 -- 172.21.15.3:0/961364169 >> v1:172.21.15.3:6789/0 conn(0x7f17b40b0a70 legacy=0x7f17b40b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b2ffd700 1 -- 172.21.15.3:0/961364169 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b40b26b0 con 0x7f17b40a92c0 2022-07-21T01:36:47.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b2ffd700 1 --2- 172.21.15.3:0/961364169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40a92c0 0x7f17b4140ef0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f17a8001800 tx=0x7f17a4006df0).ready entity=mon.1 client_cookie=98701d24ef54c39a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:47.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b1ffb700 1 -- 172.21.15.3:0/961364169 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17a8001660 con 0x7f17b40a92c0 2022-07-21T01:36:47.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b1ffb700 1 -- 172.21.15.3:0/961364169 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a4000e90 con 0x7f17b40a92c0 2022-07-21T01:36:47.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.765+0000 7f17b1ffb700 1 -- 172.21.15.3:0/961364169 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17a4011400 con 0x7f17b40a92c0 2022-07-21T01:36:47.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 -- 172.21.15.3:0/961364169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40a92c0 msgr2=0x7f17b4140ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 --2- 172.21.15.3:0/961364169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40a92c0 0x7f17b4140ef0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f17a8001800 tx=0x7f17a4006df0).stop 2022-07-21T01:36:47.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 -- 172.21.15.3:0/961364169 shutdown_connections 2022-07-21T01:36:47.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 --2- 172.21.15.3:0/961364169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40a92c0 0x7f17b4140ef0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 -- 172.21.15.3:0/961364169 >> 172.21.15.3:0/961364169 conn(0x7f17b4061360 msgr2=0x7f17b4062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:47.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 -- 172.21.15.3:0/961364169 shutdown_connections 2022-07-21T01:36:47.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 -- 172.21.15.3:0/961364169 wait complete. 2022-07-21T01:36:47.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 Processor -- start 2022-07-21T01:36:47.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 -- start start 2022-07-21T01:36:47.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17b405c620 0x7f17b41513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40a92c0 0x7f17b4141860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40b0a70 0x7f17b4063ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b4067b90 con 0x7f17b405c620 2022-07-21T01:36:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b4150810 con 0x7f17b40b0a70 2022-07-21T01:36:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b3fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b4150a00 con 0x7f17b40a92c0 2022-07-21T01:36:47.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17b405c620 0x7f17b41513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17b405c620 0x7f17b41513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45256/0 (socket says 172.21.15.3:45256) 2022-07-21T01:36:47.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b2ffd700 1 -- 172.21.15.3:0/183676145 learned_addr learned my addr 172.21.15.3:0/183676145 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:47.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b37fe700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40b0a70 0x7f17b4063ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.769+0000 7f17b27fc700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40a92c0 0x7f17b4141860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b2ffd700 1 -- 172.21.15.3:0/183676145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40a92c0 msgr2=0x7f17b4141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b2ffd700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40a92c0 0x7f17b4141860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b2ffd700 1 -- 172.21.15.3:0/183676145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40b0a70 msgr2=0x7f17b4063ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b2ffd700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40b0a70 0x7f17b4063ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b2ffd700 1 -- 172.21.15.3:0/183676145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a400b180 con 0x7f17b405c620 2022-07-21T01:36:47.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b37fe700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40b0a70 0x7f17b4063ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:47.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b2ffd700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17b405c620 0x7f17b41513e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f17a8001800 tx=0x7f17b4061c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:47.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17a3fff700 1 -- 172.21.15.3:0/183676145 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17a400d950 con 0x7f17b405c620 2022-07-21T01:36:47.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17a3fff700 1 -- 172.21.15.3:0/183676145 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a400dab0 con 0x7f17b405c620 2022-07-21T01:36:47.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b4150ca0 con 0x7f17b405c620 2022-07-21T01:36:47.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17a3fff700 1 -- 172.21.15.3:0/183676145 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17a400de10 con 0x7f17b405c620 2022-07-21T01:36:47.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b40b65f0 con 0x7f17b405c620 2022-07-21T01:36:47.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b40b2950 con 0x7f17b405c620 2022-07-21T01:36:47.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17a3fff700 1 -- 172.21.15.3:0/183676145 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f17a4024030 con 0x7f17b405c620 2022-07-21T01:36:47.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17a3fff700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f17900539e0 0x7f1790055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:47.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17a3fff700 1 -- 172.21.15.3:0/183676145 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f17a406c7b0 con 0x7f17b405c620 2022-07-21T01:36:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b27fc700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f17900539e0 0x7f1790055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.773+0000 7f17b27fc700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f17900539e0 0x7f1790055ea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f17a800b000 tx=0x7f17a8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:47.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.781+0000 7f17a3fff700 1 -- 172.21.15.3:0/183676145 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f17a4043890 con 0x7f17b405c620 2022-07-21T01:36:47.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.945+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17b40574e0 con 0x7f17b405c620 2022-07-21T01:36:47.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.945+0000 7f17a3fff700 1 -- 172.21.15.3:0/183676145 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+1556 (secure 0 0 0) 0x7f17a4046e70 con 0x7f17b405c620 2022-07-21T01:36:47.947 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:47.948 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/476418120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":476418120},{"type":"v1","addr":"172.21.15.3:6837","nonce":476418120}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":20}],"filesystems":[]} 2022-07-21T01:36:47.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f17900539e0 msgr2=0x7f1790055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f17900539e0 0x7f1790055ea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f17a800b000 tx=0x7f17a8000bc0).stop 2022-07-21T01:36:47.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17b405c620 msgr2=0x7f17b41513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:47.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17b405c620 0x7f17b41513e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f17a8001800 tx=0x7f17b4061c30).stop 2022-07-21T01:36:47.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 shutdown_connections 2022-07-21T01:36:47.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f17900539e0 0x7f1790055ea0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b40b0a70 0x7f17b4063ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40a92c0 0x7f17b4141860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 --2- 172.21.15.3:0/183676145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17b405c620 0x7f17b41513e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:47.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 >> 172.21.15.3:0/183676145 conn(0x7f17b4061360 msgr2=0x7f17b413e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:47.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 shutdown_connections 2022-07-21T01:36:47.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:47.949+0000 7f17b3fff700 1 -- 172.21.15.3:0/183676145 wait complete. 2022-07-21T01:36:47.955 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 20 2022-07-21T01:36:47.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:36:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1bfff700 1 Processor -- start 2022-07-21T01:36:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1bfff700 1 -- start start 2022-07-21T01:36:48.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2f1c0b0560 0x7f2f1c146df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1bfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1c0b12d0 con 0x7f2f1c0b0d10 2022-07-21T01:36:48.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1bfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1c0a9700 con 0x7f2f1c0a9320 2022-07-21T01:36:48.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1bfff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f1c0b1500 con 0x7f2f1c0b0560 2022-07-21T01:36:48.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2f1c0b0560 0x7f2f1c146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2f1c0b0560 0x7f2f1c146df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45268/0 (socket says 172.21.15.3:45268) 2022-07-21T01:36:48.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1b7fe700 1 -- 172.21.15.3:0/630208364 learned_addr learned my addr 172.21.15.3:0/630208364 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:48.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1a7fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2f1c0b0d10 0x7f2f1c140350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53544/0 (socket says 172.21.15.3:53544) 2022-07-21T01:36:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1b7fe700 1 -- 172.21.15.3:0/630208364 >> v1:172.21.15.143:6789/0 conn(0x7f2f1c0a9320 legacy=0x7f2f1c1466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1b7fe700 1 -- 172.21.15.3:0/630208364 >> v1:172.21.15.3:6789/0 conn(0x7f2f1c0b0d10 legacy=0x7f2f1c140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1b7fe700 1 -- 172.21.15.3:0/630208364 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c0b17a0 con 0x7f2f1c0b0560 2022-07-21T01:36:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f1b7fe700 1 --2- 172.21.15.3:0/630208364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0b0560 0x7f2f1c146df0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f2f14002f70 tx=0x7f2f14006eb0).ready entity=mon.0 client_cookie=77920f29cdfca309 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f19ffb700 1 -- 172.21.15.3:0/630208364 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f1400d660 con 0x7f2f1c0b0560 2022-07-21T01:36:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f19ffb700 1 -- 172.21.15.3:0/630208364 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f14000e90 con 0x7f2f1c0b0560 2022-07-21T01:36:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.065+0000 7f2f19ffb700 1 -- 172.21.15.3:0/630208364 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f1400d9f0 con 0x7f2f1c0b0560 2022-07-21T01:36:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 -- 172.21.15.3:0/630208364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0b0560 msgr2=0x7f2f1c146df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 --2- 172.21.15.3:0/630208364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0b0560 0x7f2f1c146df0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f2f14002f70 tx=0x7f2f14006eb0).stop 2022-07-21T01:36:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 -- 172.21.15.3:0/630208364 shutdown_connections 2022-07-21T01:36:48.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 --2- 172.21.15.3:0/630208364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0b0560 0x7f2f1c146df0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 -- 172.21.15.3:0/630208364 >> 172.21.15.3:0/630208364 conn(0x7f2f1c0612d0 msgr2=0x7f2f1c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:48.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 -- 172.21.15.3:0/630208364 shutdown_connections 2022-07-21T01:36:48.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 -- 172.21.15.3:0/630208364 wait complete. 2022-07-21T01:36:48.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 Processor -- start 2022-07-21T01:36:48.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 -- start start 2022-07-21T01:36:48.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0a9320 0x7f2f1c13b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f1c0b0560 0x7f2f1c0aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f1c0b0d10 0x7f2f1c0ac3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c13f340 con 0x7f2f1c0a9320 2022-07-21T01:36:48.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c13fc10 con 0x7f2f1c0b0d10 2022-07-21T01:36:48.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c13fe40 con 0x7f2f1c0b0560 2022-07-21T01:36:48.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0a9320 0x7f2f1c13b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0a9320 0x7f2f1c13b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45270/0 (socket says 172.21.15.3:45270) 2022-07-21T01:36:48.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1affd700 1 -- 172.21.15.3:0/2474326735 learned_addr learned my addr 172.21.15.3:0/2474326735 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:48.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1a7fc700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f1c0b0560 0x7f2f1c0aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1b7fe700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f1c0b0d10 0x7f2f1c0ac3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1affd700 1 -- 172.21.15.3:0/2474326735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f1c0b0560 msgr2=0x7f2f1c0aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.069+0000 7f2f1affd700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f1c0b0560 0x7f2f1c0aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f1affd700 1 -- 172.21.15.3:0/2474326735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f1c0b0d10 msgr2=0x7f2f1c0ac3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f1affd700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f1c0b0d10 0x7f2f1c0ac3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f1affd700 1 -- 172.21.15.3:0/2474326735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1400b1e0 con 0x7f2f1c0a9320 2022-07-21T01:36:48.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f1b7fe700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f1c0b0d10 0x7f2f1c0ac3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:48.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f1affd700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0a9320 0x7f2f1c13b7f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f2f1c13e4d0 tx=0x7f2f04007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:48.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f13fff700 1 -- 172.21.15.3:0/2474326735 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f04007e70 con 0x7f2f1c0a9320 2022-07-21T01:36:48.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f13fff700 1 -- 172.21.15.3:0/2474326735 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f04007030 con 0x7f2f1c0a9320 2022-07-21T01:36:48.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1c13ed10 con 0x7f2f1c0a9320 2022-07-21T01:36:48.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f13fff700 1 -- 172.21.15.3:0/2474326735 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f04007390 con 0x7f2f1c0a9320 2022-07-21T01:36:48.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1c0b38d0 con 0x7f2f1c0a9320 2022-07-21T01:36:48.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1c0b3a90 con 0x7f2f1c0a9320 2022-07-21T01:36:48.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f13fff700 1 -- 172.21.15.3:0/2474326735 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2f04010030 con 0x7f2f1c0a9320 2022-07-21T01:36:48.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f13fff700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2f00053aa0 0x7f2f00055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f1a7fc700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2f00053aa0 0x7f2f00055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.073+0000 7f2f13fff700 1 -- 172.21.15.3:0/2474326735 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f2f0406b980 con 0x7f2f1c0a9320 2022-07-21T01:36:48.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.077+0000 7f2f1a7fc700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2f00053aa0 0x7f2f00055f60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f2f0c00d000 tx=0x7f2f0c007f90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:48.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.077+0000 7f2f13fff700 1 -- 172.21.15.3:0/2474326735 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2f04042910 con 0x7f2f1c0a9320 2022-07-21T01:36:48.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.241+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f1c0574e0 con 0x7f2f1c0a9320 2022-07-21T01:36:48.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.241+0000 7f2f13fff700 1 -- 172.21.15.3:0/2474326735 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+1556 (secure 0 0 0) 0x7f2f04045f30 con 0x7f2f1c0a9320 2022-07-21T01:36:48.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:48.243 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4913,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3078238516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3078238516},{"type":"v1","addr":"172.21.15.143:6833","nonce":3078238516}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5236,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/1401213671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1401213671},{"type":"v1","addr":"172.21.15.3:6835","nonce":1401213671}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":16},{"gid":5499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/476418120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":476418120},{"type":"v1","addr":"172.21.15.3:6837","nonce":476418120}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":20}],"filesystems":[]} 2022-07-21T01:36:48.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2f00053aa0 msgr2=0x7f2f00055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2f00053aa0 0x7f2f00055f60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f2f0c00d000 tx=0x7f2f0c007f90).stop 2022-07-21T01:36:48.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0a9320 msgr2=0x7f2f1c13b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0a9320 0x7f2f1c13b7f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f2f1c13e4d0 tx=0x7f2f04007c00).stop 2022-07-21T01:36:48.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 shutdown_connections 2022-07-21T01:36:48.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2f00053aa0 0x7f2f00055f60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f1c0b0d10 0x7f2f1c0ac3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f1c0b0560 0x7f2f1c0aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 --2- 172.21.15.3:0/2474326735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f1c0a9320 0x7f2f1c13b7f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 >> 172.21.15.3:0/2474326735 conn(0x7f2f1c0612d0 msgr2=0x7f2f1c13ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:48.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 shutdown_connections 2022-07-21T01:36:48.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.245+0000 7f2f1bfff700 1 -- 172.21.15.3:0/2474326735 wait complete. 2022-07-21T01:36:48.248 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 20 2022-07-21T01:36:48.255 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:36:48.256 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:36:48.256 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:36:48.257 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:36:48.257 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:36:48.257 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:36:48.258 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:36:48.258 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:36:48.258 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:36:48.258 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:36:48.259 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:36:48.259 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:36:48.268 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:36:48.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:36:48.270 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:36:48.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:36:48.273 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:36:48.274 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:36:48.277 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:36:48.277 DEBUG:teuthology.parallel:result is None 2022-07-21T01:36:48.312 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:36:48.313 DEBUG:teuthology.parallel:result is None 2022-07-21T01:36:48.314 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:36:48.314 DEBUG:teuthology.parallel:result is None 2022-07-21T01:36:48.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:36:48.351 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:36:48.391 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:36:48.407 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:36:48.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda2785700 1 Processor -- start 2022-07-21T01:36:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda2785700 1 -- start start 2022-07-21T01:36:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda2785700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fed9c0ae3c0 0x7fed9c1406a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda2785700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c0afa50 con 0x7fed9c0b2050 2022-07-21T01:36:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda2785700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c05ff60 con 0x7fed9c05fb80 2022-07-21T01:36:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda2785700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed9c140b90 con 0x7fed9c0ae3c0 2022-07-21T01:36:48.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda0f82700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fed9c0b2050 0x7fed9c13cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53584/0 (socket says 172.21.15.3:53584) 2022-07-21T01:36:48.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda0f82700 1 -- 172.21.15.3:0/4293001306 learned_addr learned my addr 172.21.15.3:0/4293001306 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:48.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda1f84700 1 --2- 172.21.15.3:0/4293001306 >> v2:172.21.15.143:3300/0 conn(0x7fed9c0ae3c0 0x7fed9c1406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda1f84700 1 -- 172.21.15.3:0/4293001306 >> v1:172.21.15.143:6789/0 conn(0x7fed9c05fb80 legacy=0x7fed9c13ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda1f84700 1 -- 172.21.15.3:0/4293001306 >> v1:172.21.15.3:6789/0 conn(0x7fed9c0b2050 legacy=0x7fed9c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda1f84700 1 -- 172.21.15.3:0/4293001306 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c0ad1c0 con 0x7fed9c0ae3c0 2022-07-21T01:36:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.413+0000 7feda1f84700 1 --2- 172.21.15.3:0/4293001306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c0ae3c0 0x7fed9c1406a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fed90002f70 tx=0x7fed90006df0).ready entity=mon.1 client_cookie=85c8ad2a5ca0e0e9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:48.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7fed9bfff700 1 -- 172.21.15.3:0/4293001306 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed9000d610 con 0x7fed9c0ae3c0 2022-07-21T01:36:48.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7fed9bfff700 1 -- 172.21.15.3:0/4293001306 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed90000e90 con 0x7fed9c0ae3c0 2022-07-21T01:36:48.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7fed9bfff700 1 -- 172.21.15.3:0/4293001306 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed9000d960 con 0x7fed9c0ae3c0 2022-07-21T01:36:48.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7feda2785700 1 -- 172.21.15.3:0/4293001306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c0ae3c0 msgr2=0x7fed9c1406a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7feda2785700 1 --2- 172.21.15.3:0/4293001306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c0ae3c0 0x7fed9c1406a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fed90002f70 tx=0x7fed90006df0).stop 2022-07-21T01:36:48.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7feda2785700 1 -- 172.21.15.3:0/4293001306 shutdown_connections 2022-07-21T01:36:48.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7feda2785700 1 --2- 172.21.15.3:0/4293001306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c0ae3c0 0x7fed9c1406a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7feda2785700 1 -- 172.21.15.3:0/4293001306 >> 172.21.15.3:0/4293001306 conn(0x7fed9c0613c0 msgr2=0x7fed9c062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:48.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7feda2785700 1 -- 172.21.15.3:0/4293001306 shutdown_connections 2022-07-21T01:36:48.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7feda2785700 1 -- 172.21.15.3:0/4293001306 wait complete. 2022-07-21T01:36:48.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7feda2785700 1 Processor -- start 2022-07-21T01:36:48.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.421+0000 7feda2785700 1 -- start start 2022-07-21T01:36:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda2785700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c05fb80 0x7fed9c134f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda2785700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed9c0ae3c0 0x7fed9c156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda2785700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed9c0b2050 0x7fed9c156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda2785700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c133400 con 0x7fed9c0b2050 2022-07-21T01:36:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda2785700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c13a2a0 con 0x7fed9c05fb80 2022-07-21T01:36:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda2785700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c136660 con 0x7fed9c0ae3c0 2022-07-21T01:36:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1f84700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed9c0b2050 0x7fed9c156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1f84700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed9c0b2050 0x7fed9c156640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45312/0 (socket says 172.21.15.3:45312) 2022-07-21T01:36:48.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1f84700 1 -- 172.21.15.3:0/27801249 learned_addr learned my addr 172.21.15.3:0/27801249 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:48.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1783700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c05fb80 0x7fed9c134f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda0f82700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed9c0ae3c0 0x7fed9c156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1f84700 1 -- 172.21.15.3:0/27801249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed9c0ae3c0 msgr2=0x7fed9c156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1f84700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed9c0ae3c0 0x7fed9c156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1f84700 1 -- 172.21.15.3:0/27801249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c05fb80 msgr2=0x7fed9c134f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:48.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1f84700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c05fb80 0x7fed9c134f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1f84700 1 -- 172.21.15.3:0/27801249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9000b180 con 0x7fed9c0b2050 2022-07-21T01:36:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1783700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c05fb80 0x7fed9c134f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda1f84700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed9c0b2050 0x7fed9c156640 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fed90002f70 tx=0x7fed90015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7fed9a7fc700 1 -- 172.21.15.3:0/27801249 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed9000d990 con 0x7fed9c0b2050 2022-07-21T01:36:48.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7fed9a7fc700 1 -- 172.21.15.3:0/27801249 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed90011b80 con 0x7fed9c0b2050 2022-07-21T01:36:48.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7fed9a7fc700 1 -- 172.21.15.3:0/27801249 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed9001e460 con 0x7fed9c0b2050 2022-07-21T01:36:48.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9c136d00 con 0x7fed9c0b2050 2022-07-21T01:36:48.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9c156b30 con 0x7fed9c0b2050 2022-07-21T01:36:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9c136b60 con 0x7fed9c0b2050 2022-07-21T01:36:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.425+0000 7fed9a7fc700 1 -- 172.21.15.3:0/27801249 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fed90025030 con 0x7fed9c0b2050 2022-07-21T01:36:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.429+0000 7fed9a7fc700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed7c053a70 0x7fed7c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.429+0000 7fed9a7fc700 1 -- 172.21.15.3:0/27801249 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fed9006d060 con 0x7fed9c0b2050 2022-07-21T01:36:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.429+0000 7feda1783700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed7c053a70 0x7fed7c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.429+0000 7feda1783700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed7c053a70 0x7fed7c055f30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fed9400d000 tx=0x7fed94009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.433+0000 7fed9a7fc700 1 -- 172.21.15.3:0/27801249 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fed9c136b60 con 0x7fed9c0b2050 2022-07-21T01:36:48.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:48.569+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fed9c136b60 con 0x7fed9c0b2050 2022-07-21T01:36:49.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.429+0000 7fed9a7fc700 1 -- 172.21.15.3:0/27801249 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v54) v1 ==== 97+0+0 (secure 0 0 0) 0x7fed9c136b60 con 0x7fed9c0b2050 2022-07-21T01:36:49.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed7c053a70 msgr2=0x7fed7c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed7c053a70 0x7fed7c055f30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fed9400d000 tx=0x7fed94009040).stop 2022-07-21T01:36:49.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed9c0b2050 msgr2=0x7fed9c156640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed9c0b2050 0x7fed9c156640 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fed90002f70 tx=0x7fed90015000).stop 2022-07-21T01:36:49.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 shutdown_connections 2022-07-21T01:36:49.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed9c0b2050 0x7fed9c156640 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed9c0ae3c0 0x7fed9c156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed7c053a70 0x7fed7c055f30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 --2- 172.21.15.3:0/27801249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed9c05fb80 0x7fed9c134f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 >> 172.21.15.3:0/27801249 conn(0x7fed9c0613c0 msgr2=0x7fed9c0b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:49.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 shutdown_connections 2022-07-21T01:36:49.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.433+0000 7feda2785700 1 -- 172.21.15.3:0/27801249 wait complete. 2022-07-21T01:36:49.437 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:36:49.449 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:36:49.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:36:49.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 Processor -- start 2022-07-21T01:36:49.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 -- start start 2022-07-21T01:36:49.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9018148530 0x7f90180b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90180a7250 0x7f90181448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90180b04e0 con 0x7f90180b1f40 2022-07-21T01:36:49.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90180a9250 con 0x7f9018148530 2022-07-21T01:36:49.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90180b0780 con 0x7f90180a7250 2022-07-21T01:36:49.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901c97b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f90180b1f40 0x7f90180b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:55716/0 (socket says 172.21.15.3:55716) 2022-07-21T01:36:49.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901c97b700 1 -- 172.21.15.3:0/2780502222 learned_addr learned my addr 172.21.15.3:0/2780502222 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:49.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f9017fff700 1 --2- 172.21.15.3:0/2780502222 >> v2:172.21.15.3:3300/0 conn(0x7f9018148530 0x7f90180b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901d17c700 1 --2- 172.21.15.3:0/2780502222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90180a7250 0x7f90181448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f90177fe700 1 -- 172.21.15.3:0/2780502222 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538185482 0 0) 0x7f90180b04e0 con 0x7f90180b1f40 2022-07-21T01:36:49.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f9017fff700 1 -- 172.21.15.3:0/2780502222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90180a7250 msgr2=0x7f90181448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:49.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f9017fff700 1 --2- 172.21.15.3:0/2780502222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90180a7250 0x7f90181448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f9017fff700 1 -- 172.21.15.3:0/2780502222 >> v1:172.21.15.143:6789/0 conn(0x7f90180b1f40 legacy=0x7f90180b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f9017fff700 1 -- 172.21.15.3:0/2780502222 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90180b0a20 con 0x7f9018148530 2022-07-21T01:36:49.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f9017fff700 1 --2- 172.21.15.3:0/2780502222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9018148530 0x7f90180b1160 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f90181478c0 tx=0x7f900c006df0).ready entity=mon.0 client_cookie=87f5c2a78fa851ec server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:49.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f90177fe700 1 -- 172.21.15.3:0/2780502222 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f900c00d570 con 0x7f9018148530 2022-07-21T01:36:49.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f90177fe700 1 -- 172.21.15.3:0/2780502222 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c000e90 con 0x7f9018148530 2022-07-21T01:36:49.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f90177fe700 1 -- 172.21.15.3:0/2780502222 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f900c00d8c0 con 0x7f9018148530 2022-07-21T01:36:49.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 -- 172.21.15.3:0/2780502222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9018148530 msgr2=0x7f90180b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 --2- 172.21.15.3:0/2780502222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9018148530 0x7f90180b1160 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f90181478c0 tx=0x7f900c006df0).stop 2022-07-21T01:36:49.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 -- 172.21.15.3:0/2780502222 shutdown_connections 2022-07-21T01:36:49.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 --2- 172.21.15.3:0/2780502222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90180a7250 0x7f90181448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 --2- 172.21.15.3:0/2780502222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9018148530 0x7f90180b1160 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 -- 172.21.15.3:0/2780502222 >> 172.21.15.3:0/2780502222 conn(0x7f901805f840 msgr2=0x7f901805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:49.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 -- 172.21.15.3:0/2780502222 shutdown_connections 2022-07-21T01:36:49.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.561+0000 7f901ebdf700 1 -- 172.21.15.3:0/2780502222 wait complete. 2022-07-21T01:36:49.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901ebdf700 1 Processor -- start 2022-07-21T01:36:49.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901ebdf700 1 -- start start 2022-07-21T01:36:49.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901ebdf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90180a7250 0x7f901814a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901ebdf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90180b1f40 0x7f90180ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901ebdf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9018148530 0x7f90180ab920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901ebdf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90180a9cb0 con 0x7f90180a7250 2022-07-21T01:36:49.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901ebdf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901814b7a0 con 0x7f90180b1f40 2022-07-21T01:36:49.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901ebdf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901814ba40 con 0x7f9018148530 2022-07-21T01:36:49.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901c97b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90180a7250 0x7f901814a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901c97b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90180a7250 0x7f901814a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45326/0 (socket says 172.21.15.3:45326) 2022-07-21T01:36:49.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901c97b700 1 -- 172.21.15.3:0/2258078723 learned_addr learned my addr 172.21.15.3:0/2258078723 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:49.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901d17c700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9018148530 0x7f90180ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f9017fff700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90180b1f40 0x7f90180ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901c97b700 1 -- 172.21.15.3:0/2258078723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9018148530 msgr2=0x7f90180ab920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901c97b700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9018148530 0x7f90180ab920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901c97b700 1 -- 172.21.15.3:0/2258078723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90180b1f40 msgr2=0x7f90180ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901c97b700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90180b1f40 0x7f90180ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901c97b700 1 -- 172.21.15.3:0/2258078723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c00b180 con 0x7f90180a7250 2022-07-21T01:36:49.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901c97b700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90180a7250 0x7f901814a500 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f9010001250 tx=0x7f9010007e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:49.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f9015ffb700 1 -- 172.21.15.3:0/2258078723 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9010005ba0 con 0x7f90180a7250 2022-07-21T01:36:49.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f9015ffb700 1 -- 172.21.15.3:0/2258078723 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9010005d00 con 0x7f90180a7250 2022-07-21T01:36:49.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9018149ea0 con 0x7f90180a7250 2022-07-21T01:36:49.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.565+0000 7f9015ffb700 1 -- 172.21.15.3:0/2258078723 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90100072c0 con 0x7f90180a7250 2022-07-21T01:36:49.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.569+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90180a95d0 con 0x7f90180a7250 2022-07-21T01:36:49.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.569+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901814ad20 con 0x7f90180a7250 2022-07-21T01:36:49.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.569+0000 7f9015ffb700 1 -- 172.21.15.3:0/2258078723 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9010010070 con 0x7f90180a7250 2022-07-21T01:36:49.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.569+0000 7f9015ffb700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ff8053b00 0x7f8ff8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.569+0000 7f9015ffb700 1 -- 172.21.15.3:0/2258078723 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f901006b910 con 0x7f90180a7250 2022-07-21T01:36:49.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.569+0000 7f9017fff700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ff8053b00 0x7f8ff8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.573+0000 7f9017fff700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ff8053b00 0x7f8ff8055fc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f900c01a000 tx=0x7f900c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:49.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.581+0000 7f9015ffb700 1 -- 172.21.15.3:0/2258078723 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9010042880 con 0x7f90180a7250 2022-07-21T01:36:49.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.749+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f90180ab1d0 con 0x7f90180a7250 2022-07-21T01:36:49.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.749+0000 7f9015ffb700 1 -- 172.21.15.3:0/2258078723 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f9010045ea0 con 0x7f90180a7250 2022-07-21T01:36:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:49.751 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:36:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:36:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:36:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:36:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:36:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:36:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:36:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:36:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:36:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:36:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:36:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:36:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:36:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:36:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:36:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:36:49.772 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:36:49.772 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.772 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:36:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:36:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:36:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:36:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:36:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:36:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:36:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:36:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:36:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:36:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:36:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:36:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:36:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:36:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:36:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:36:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:36:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:36:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:36:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:36:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:36:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:36:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.792 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:36:49.792 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:36:49.792 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:36:49.792 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.793 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.794 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:36:49.794 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:36:49.794 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.794 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:36:49.795 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:36:49.795 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.795 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:49.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:49.796 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:36:49.796 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:36:49.797 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:36:49.797 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:36:49.797 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:36:49.798 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:36:49.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:49.798 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:36:49.799 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:36:49.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ff8053b00 msgr2=0x7f8ff8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ff8053b00 0x7f8ff8055fc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f900c01a000 tx=0x7f900c014000).stop 2022-07-21T01:36:49.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90180a7250 msgr2=0x7f901814a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90180a7250 0x7f901814a500 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f9010001250 tx=0x7f9010007e40).stop 2022-07-21T01:36:49.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 shutdown_connections 2022-07-21T01:36:49.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9018148530 0x7f90180ab920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ff8053b00 0x7f8ff8055fc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90180b1f40 0x7f90180ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 --2- 172.21.15.3:0/2258078723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90180a7250 0x7f901814a500 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 >> 172.21.15.3:0/2258078723 conn(0x7f901805f840 msgr2=0x7f9018144890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:49.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 shutdown_connections 2022-07-21T01:36:49.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.753+0000 7f901ebdf700 1 -- 172.21.15.3:0/2258078723 wait complete. 2022-07-21T01:36:49.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:36:49.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fffff700 1 Processor -- start 2022-07-21T01:36:49.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fffff700 1 -- start start 2022-07-21T01:36:49.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b01a0 0x7f06000a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06000afd60 0x7f06000a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fffff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06000a7250 con 0x7f06000b2710 2022-07-21T01:36:49.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06000a7480 con 0x7f06000afd60 2022-07-21T01:36:49.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06000a7720 con 0x7f06000b01a0 2022-07-21T01:36:49.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05feffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f06000b2710 0x7f060014afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53616/0 (socket says 172.21.15.3:53616) 2022-07-21T01:36:49.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05feffd700 1 -- 172.21.15.3:0/4281877437 learned_addr learned my addr 172.21.15.3:0/4281877437 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:49.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fe7fc700 1 --2- 172.21.15.3:0/4281877437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b01a0 0x7f06000a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05ff7fe700 1 --2- 172.21.15.3:0/4281877437 >> v2:172.21.15.143:3300/0 conn(0x7f06000afd60 0x7f06000a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fdffb700 1 -- 172.21.15.3:0/4281877437 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218525037 0 0) 0x7f06000a7250 con 0x7f06000b2710 2022-07-21T01:36:49.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fdffb700 1 -- 172.21.15.3:0/4281877437 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e4003260 con 0x7f06000b2710 2022-07-21T01:36:49.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fe7fc700 1 -- 172.21.15.3:0/4281877437 >> v2:172.21.15.143:3300/0 conn(0x7f06000afd60 msgr2=0x7f06000a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:49.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fe7fc700 1 --2- 172.21.15.3:0/4281877437 >> v2:172.21.15.143:3300/0 conn(0x7f06000afd60 0x7f06000a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fe7fc700 1 -- 172.21.15.3:0/4281877437 >> v1:172.21.15.3:6789/0 conn(0x7f06000b2710 legacy=0x7f060014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.913+0000 7f05fe7fc700 1 -- 172.21.15.3:0/4281877437 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000a79c0 con 0x7f06000b01a0 2022-07-21T01:36:49.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fe7fc700 1 --2- 172.21.15.3:0/4281877437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b01a0 0x7f06000a8540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f05f00032f0 tx=0x7f05f00091d0).ready entity=mon.2 client_cookie=bc00022bd91ff2c0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:49.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fdffb700 1 -- 172.21.15.3:0/4281877437 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05f000b600 con 0x7f06000b01a0 2022-07-21T01:36:49.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fdffb700 1 -- 172.21.15.3:0/4281877437 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f000b760 con 0x7f06000b01a0 2022-07-21T01:36:49.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fdffb700 1 -- 172.21.15.3:0/4281877437 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05f000b8d0 con 0x7f06000b01a0 2022-07-21T01:36:49.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fffff700 1 -- 172.21.15.3:0/4281877437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b01a0 msgr2=0x7f06000a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fffff700 1 --2- 172.21.15.3:0/4281877437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b01a0 0x7f06000a8540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f05f00032f0 tx=0x7f05f00091d0).stop 2022-07-21T01:36:49.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fffff700 1 -- 172.21.15.3:0/4281877437 shutdown_connections 2022-07-21T01:36:49.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fffff700 1 --2- 172.21.15.3:0/4281877437 >> v2:172.21.15.143:3300/0 conn(0x7f06000afd60 0x7f06000a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fffff700 1 --2- 172.21.15.3:0/4281877437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b01a0 0x7f06000a8540 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fffff700 1 -- 172.21.15.3:0/4281877437 >> 172.21.15.3:0/4281877437 conn(0x7f0600060310 msgr2=0x7f06000b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:49.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fffff700 1 -- 172.21.15.3:0/4281877437 shutdown_connections 2022-07-21T01:36:49.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.917+0000 7f05fffff700 1 -- 172.21.15.3:0/4281877437 wait complete. 2022-07-21T01:36:49.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fffff700 1 Processor -- start 2022-07-21T01:36:49.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fffff700 1 -- start start 2022-07-21T01:36:49.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06000afd60 0x7f0600156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06000b01a0 0x7f06000b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b2710 0x7f06000b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06000b59f0 con 0x7f06000b01a0 2022-07-21T01:36:49.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060014c4d0 con 0x7f06000afd60 2022-07-21T01:36:49.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0600149ae0 con 0x7f06000b2710 2022-07-21T01:36:49.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fe7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06000b01a0 0x7f06000b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fe7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06000b01a0 0x7f06000b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45342/0 (socket says 172.21.15.3:45342) 2022-07-21T01:36:49.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fe7fc700 1 -- 172.21.15.3:0/2827729646 learned_addr learned my addr 172.21.15.3:0/2827729646 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:49.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05feffd700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06000afd60 0x7f0600156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05ff7fe700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b2710 0x7f06000b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fe7fc700 1 -- 172.21.15.3:0/2827729646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b2710 msgr2=0x7f06000b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fe7fc700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b2710 0x7f06000b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fe7fc700 1 -- 172.21.15.3:0/2827729646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06000afd60 msgr2=0x7f0600156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:49.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fe7fc700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06000afd60 0x7f0600156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:49.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fe7fc700 1 -- 172.21.15.3:0/2827729646 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0009470 con 0x7f06000b01a0 2022-07-21T01:36:49.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05ff7fe700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b2710 0x7f06000b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:49.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fe7fc700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06000b01a0 0x7f06000b45e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f06000b5d50 tx=0x7f05f0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:49.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05effff700 1 -- 172.21.15.3:0/2827729646 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05f000b8d0 con 0x7f06000b01a0 2022-07-21T01:36:49.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05effff700 1 -- 172.21.15.3:0/2827729646 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f001f3f0 con 0x7f06000b01a0 2022-07-21T01:36:49.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060014c230 con 0x7f06000b01a0 2022-07-21T01:36:49.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.921+0000 7f05effff700 1 -- 172.21.15.3:0/2827729646 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05f000b600 con 0x7f06000b01a0 2022-07-21T01:36:49.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.925+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060014bf20 con 0x7f06000b01a0 2022-07-21T01:36:49.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.925+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0600157130 con 0x7f06000b01a0 2022-07-21T01:36:49.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.925+0000 7f05effff700 1 -- 172.21.15.3:0/2827729646 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f05f0026030 con 0x7f06000b01a0 2022-07-21T01:36:49.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.925+0000 7f05effff700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05dc053a70 0x7f05dc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:49.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.925+0000 7f05feffd700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05dc053a70 0x7f05dc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:49.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.925+0000 7f05effff700 1 -- 172.21.15.3:0/2827729646 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f05f006caa0 con 0x7f06000b01a0 2022-07-21T01:36:49.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.925+0000 7f05feffd700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05dc053a70 0x7f05dc055f30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f05f400d000 tx=0x7f05f4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:49.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:49.933+0000 7f05effff700 1 -- 172.21.15.3:0/2827729646 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05f0043a90 con 0x7f06000b01a0 2022-07-21T01:36:50.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.057+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06000574e0 con 0x7f06000b01a0 2022-07-21T01:36:50.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.061+0000 7f05effff700 1 -- 172.21.15.3:0/2827729646 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+13414 (secure 0 0 0) 0x7f05f0015090 con 0x7f06000b01a0 2022-07-21T01:36:50.061 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:50.062 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":54,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:36:49.422947+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:36:50.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.061+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05dc053a70 msgr2=0x7f05dc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.061+0000 7f05fffff700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05dc053a70 0x7f05dc055f30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f05f400d000 tx=0x7f05f4009040).stop 2022-07-21T01:36:50.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.061+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06000b01a0 msgr2=0x7f06000b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.061+0000 7f05fffff700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06000b01a0 0x7f06000b45e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f06000b5d50 tx=0x7f05f0016000).stop 2022-07-21T01:36:50.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.065+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 shutdown_connections 2022-07-21T01:36:50.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.065+0000 7f05fffff700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06000b2710 0x7f06000b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.065+0000 7f05fffff700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06000b01a0 0x7f06000b45e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.065+0000 7f05fffff700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05dc053a70 0x7f05dc055f30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.065+0000 7f05fffff700 1 --2- 172.21.15.3:0/2827729646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06000afd60 0x7f0600156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.065+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 >> 172.21.15.3:0/2827729646 conn(0x7f0600060310 msgr2=0x7f060014cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:50.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.065+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 shutdown_connections 2022-07-21T01:36:50.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.065+0000 7f05fffff700 1 -- 172.21.15.3:0/2827729646 wait complete. 2022-07-21T01:36:50.075 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:36:50.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:36:50.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.181+0000 7f4c9f18c700 1 Processor -- start 2022-07-21T01:36:50.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.181+0000 7f4c9f18c700 1 -- start start 2022-07-21T01:36:50.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c980b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4c980b6ee0 0x7f4c98143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c98061d90 con 0x7f4c980b5ed0 2022-07-21T01:36:50.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c98142f60 con 0x7f4c980b6ee0 2022-07-21T01:36:50.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980ba5a0 con 0x7f4c980619b0 2022-07-21T01:36:50.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9e18a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c980b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9d989700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4c980b6ee0 0x7f4c98143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9e98b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4c980b5ed0 0x7f4c980b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:55748/0 (socket says 172.21.15.3:55748) 2022-07-21T01:36:50.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9e98b700 1 -- 172.21.15.3:0/2198235902 learned_addr learned my addr 172.21.15.3:0/2198235902 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:50.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9e18a700 1 -- 172.21.15.3:0/2198235902 >> v2:172.21.15.143:3300/0 conn(0x7f4c980b6ee0 msgr2=0x7f4c98143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:50.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9e18a700 1 --2- 172.21.15.3:0/2198235902 >> v2:172.21.15.143:3300/0 conn(0x7f4c980b6ee0 0x7f4c98143890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9e18a700 1 -- 172.21.15.3:0/2198235902 >> v1:172.21.15.143:6789/0 conn(0x7f4c980b5ed0 legacy=0x7f4c980b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9e18a700 1 -- 172.21.15.3:0/2198235902 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c98143150 con 0x7f4c980619b0 2022-07-21T01:36:50.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9e18a700 1 --2- 172.21.15.3:0/2198235902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c980b48a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4c8c002f70 tx=0x7f4c8c006df0).ready entity=mon.2 client_cookie=9962c619e1eefa65 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9d188700 1 -- 172.21.15.3:0/2198235902 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c8c00d570 con 0x7f4c980619b0 2022-07-21T01:36:50.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9d188700 1 -- 172.21.15.3:0/2198235902 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8c000e90 con 0x7f4c980619b0 2022-07-21T01:36:50.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9d188700 1 -- 172.21.15.3:0/2198235902 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c8c00d8c0 con 0x7f4c980619b0 2022-07-21T01:36:50.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 -- 172.21.15.3:0/2198235902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 msgr2=0x7f4c980b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 --2- 172.21.15.3:0/2198235902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c980b48a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4c8c002f70 tx=0x7f4c8c006df0).stop 2022-07-21T01:36:50.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 -- 172.21.15.3:0/2198235902 shutdown_connections 2022-07-21T01:36:50.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 --2- 172.21.15.3:0/2198235902 >> v2:172.21.15.143:3300/0 conn(0x7f4c980b6ee0 0x7f4c98143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 --2- 172.21.15.3:0/2198235902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c980b48a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 -- 172.21.15.3:0/2198235902 >> 172.21.15.3:0/2198235902 conn(0x7f4c9806af80 msgr2=0x7f4c9805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:50.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 -- 172.21.15.3:0/2198235902 shutdown_connections 2022-07-21T01:36:50.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.185+0000 7f4c9f18c700 1 -- 172.21.15.3:0/2198235902 wait complete. 2022-07-21T01:36:50.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9f18c700 1 Processor -- start 2022-07-21T01:36:50.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9f18c700 1 -- start start 2022-07-21T01:36:50.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9f18c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c98112bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9f18c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c980b5ed0 0x7f4c98154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9f18c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c980b6ee0 0x7f4c98155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9f18c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980ba250 con 0x7f4c980b5ed0 2022-07-21T01:36:50.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9f18c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c98153980 con 0x7f4c980b6ee0 2022-07-21T01:36:50.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9f18c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981524b0 con 0x7f4c980619b0 2022-07-21T01:36:50.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9d989700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c980b5ed0 0x7f4c98154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9d989700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c980b5ed0 0x7f4c98154fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45354/0 (socket says 172.21.15.3:45354) 2022-07-21T01:36:50.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9d989700 1 -- 172.21.15.3:0/1459732563 learned_addr learned my addr 172.21.15.3:0/1459732563 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:50.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9e18a700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c98112bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9e98b700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c980b6ee0 0x7f4c98155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9e18a700 1 -- 172.21.15.3:0/1459732563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c980b6ee0 msgr2=0x7f4c98155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9e18a700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c980b6ee0 0x7f4c98155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9e18a700 1 -- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c980b5ed0 msgr2=0x7f4c98154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9e18a700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c980b5ed0 0x7f4c98154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9e18a700 1 -- 172.21.15.3:0/1459732563 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c00b180 con 0x7f4c980619b0 2022-07-21T01:36:50.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9d989700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c980b5ed0 0x7f4c98154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:50.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.189+0000 7f4c9e18a700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c98112bd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4c8c002f70 tx=0x7f4c8c014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c8b7fe700 1 -- 172.21.15.3:0/1459732563 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c8c016070 con 0x7f4c980619b0 2022-07-21T01:36:50.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c8b7fe700 1 -- 172.21.15.3:0/1459732563 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8c00d8f0 con 0x7f4c980619b0 2022-07-21T01:36:50.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c980b05a0 con 0x7f4c980619b0 2022-07-21T01:36:50.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c8b7fe700 1 -- 172.21.15.3:0/1459732563 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c8c013de0 con 0x7f4c980619b0 2022-07-21T01:36:50.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c980b0ae0 con 0x7f4c980619b0 2022-07-21T01:36:50.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c98152600 con 0x7f4c980619b0 2022-07-21T01:36:50.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c8b7fe700 1 -- 172.21.15.3:0/1459732563 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4c8c025030 con 0x7f4c980619b0 2022-07-21T01:36:50.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c8b7fe700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c78053950 0x7f4c78055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c9d989700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c78053950 0x7f4c78055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.193+0000 7f4c8b7fe700 1 -- 172.21.15.3:0/1459732563 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f4c8c06cc10 con 0x7f4c980619b0 2022-07-21T01:36:50.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.197+0000 7f4c9d989700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c78053950 0x7f4c78055e10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f4c9400b000 tx=0x7f4c94006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.201+0000 7f4c8b7fe700 1 -- 172.21.15.3:0/1459732563 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4c98152600 con 0x7f4c980619b0 2022-07-21T01:36:50.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.333+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4c98152600 con 0x7f4c980619b0 2022-07-21T01:36:50.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.333+0000 7f4c8b7fe700 1 -- 172.21.15.3:0/1459732563 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v54) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4c98152600 con 0x7f4c980619b0 2022-07-21T01:36:50.335 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:36:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c78053950 msgr2=0x7f4c78055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c78053950 0x7f4c78055e10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f4c9400b000 tx=0x7f4c94006df0).stop 2022-07-21T01:36:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 msgr2=0x7f4c98112bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c98112bd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4c8c002f70 tx=0x7f4c8c014000).stop 2022-07-21T01:36:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 shutdown_connections 2022-07-21T01:36:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c78053950 0x7f4c78055e10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c980b6ee0 0x7f4c98155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c980b5ed0 0x7f4c98154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 --2- 172.21.15.3:0/1459732563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c980619b0 0x7f4c98112bd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 >> 172.21.15.3:0/1459732563 conn(0x7f4c9806af80 msgr2=0x7f4c981404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:50.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 shutdown_connections 2022-07-21T01:36:50.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.337+0000 7f4c9f18c700 1 -- 172.21.15.3:0/1459732563 wait complete. 2022-07-21T01:36:50.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:36:50.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 Processor -- start 2022-07-21T01:36:50.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 -- start start 2022-07-21T01:36:50.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f15040b2050 0x7f15040b1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15040b05b0 con 0x7f15040b0140 2022-07-21T01:36:50.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15040a7960 con 0x7f15040b2430 2022-07-21T01:36:50.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15040b07e0 con 0x7f15040b2050 2022-07-21T01:36:50.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f1508f33700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f15040b0140 0x7f15040a7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53642/0 (socket says 172.21.15.3:53642) 2022-07-21T01:36:50.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f1508f33700 1 -- 172.21.15.3:0/4212297675 learned_addr learned my addr 172.21.15.3:0/4212297675 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:50.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f1509734700 1 --2- 172.21.15.3:0/4212297675 >> v2:172.21.15.143:3300/0 conn(0x7f15040b2050 0x7f15040b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f14fbfff700 1 -- 172.21.15.3:0/4212297675 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819894244 0 0) 0x7f15040b05b0 con 0x7f15040b0140 2022-07-21T01:36:50.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f14fbfff700 1 -- 172.21.15.3:0/4212297675 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e8003260 con 0x7f15040b0140 2022-07-21T01:36:50.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f1509734700 1 -- 172.21.15.3:0/4212297675 >> v1:172.21.15.143:6789/0 conn(0x7f15040b2430 legacy=0x7f15041413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f1509734700 1 -- 172.21.15.3:0/4212297675 >> v1:172.21.15.3:6789/0 conn(0x7f15040b0140 legacy=0x7f15040a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f1509734700 1 -- 172.21.15.3:0/4212297675 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15040b0a80 con 0x7f15040b2050 2022-07-21T01:36:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f1509734700 1 --2- 172.21.15.3:0/4212297675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 0x7f15040b1460 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f15000032f0 tx=0x7f15000091d0).ready entity=mon.1 client_cookie=c1a6e93ee39dbeb1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f14fbfff700 1 -- 172.21.15.3:0/4212297675 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f150000b6c0 con 0x7f15040b2050 2022-07-21T01:36:50.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f14fbfff700 1 -- 172.21.15.3:0/4212297675 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150000b820 con 0x7f15040b2050 2022-07-21T01:36:50.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f14fbfff700 1 -- 172.21.15.3:0/4212297675 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f150000b990 con 0x7f15040b2050 2022-07-21T01:36:50.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 -- 172.21.15.3:0/4212297675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 msgr2=0x7f15040b1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 --2- 172.21.15.3:0/4212297675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 0x7f15040b1460 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f15000032f0 tx=0x7f15000091d0).stop 2022-07-21T01:36:50.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 -- 172.21.15.3:0/4212297675 shutdown_connections 2022-07-21T01:36:50.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 --2- 172.21.15.3:0/4212297675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 0x7f15040b1460 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 -- 172.21.15.3:0/4212297675 >> 172.21.15.3:0/4212297675 conn(0x7f150405f840 msgr2=0x7f150405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:50.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.453+0000 7f150b998700 1 -- 172.21.15.3:0/4212297675 shutdown_connections 2022-07-21T01:36:50.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f150b998700 1 -- 172.21.15.3:0/4212297675 wait complete. 2022-07-21T01:36:50.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f150b998700 1 Processor -- start 2022-07-21T01:36:50.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f150b998700 1 -- start start 2022-07-21T01:36:50.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f150b998700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040b0140 0x7f15040aa660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f150b998700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 0x7f1504140420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f150b998700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15040b2430 0x7f15041437e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f150b998700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1504145090 con 0x7f15040b0140 2022-07-21T01:36:50.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f150b998700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15040a9960 con 0x7f15040b2050 2022-07-21T01:36:50.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f150b998700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15041430e0 con 0x7f15040b2430 2022-07-21T01:36:50.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1509734700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040b0140 0x7f15040aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1509734700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040b0140 0x7f15040aa660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45366/0 (socket says 172.21.15.3:45366) 2022-07-21T01:36:50.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1509734700 1 -- 172.21.15.3:0/4172942178 learned_addr learned my addr 172.21.15.3:0/4172942178 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:50.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1508f33700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 0x7f1504140420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1509734700 1 -- 172.21.15.3:0/4172942178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15040b2430 msgr2=0x7f15041437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1509734700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15040b2430 0x7f15041437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:36:50.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1509734700 1 -- 172.21.15.3:0/4172942178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 msgr2=0x7f1504140420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1509734700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 0x7f1504140420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1509734700 1 -- 172.21.15.3:0/4172942178 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1500009470 con 0x7f15040b0140 2022-07-21T01:36:50.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1508f33700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 0x7f1504140420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:36:50.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.457+0000 7f1509734700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040b0140 0x7f15040aa660 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f15000032f0 tx=0x7f1500016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f14fa7fc700 1 -- 172.21.15.3:0/4172942178 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f150000b6c0 con 0x7f15040b0140 2022-07-21T01:36:50.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f14fa7fc700 1 -- 172.21.15.3:0/4172942178 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15000107a0 con 0x7f15040b0140 2022-07-21T01:36:50.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15040a96c0 con 0x7f15040b0140 2022-07-21T01:36:50.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f14fa7fc700 1 -- 172.21.15.3:0/4172942178 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1500010b00 con 0x7f15040b0140 2022-07-21T01:36:50.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15040b44b0 con 0x7f15040b0140 2022-07-21T01:36:50.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1504143540 con 0x7f15040b0140 2022-07-21T01:36:50.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f14fa7fc700 1 -- 172.21.15.3:0/4172942178 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1500027020 con 0x7f15040b0140 2022-07-21T01:36:50.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f14fa7fc700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f14ec053a70 0x7f14ec055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f14fa7fc700 1 -- 172.21.15.3:0/4172942178 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f150006d380 con 0x7f15040b0140 2022-07-21T01:36:50.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f1508f33700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f14ec053a70 0x7f14ec055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.461+0000 7f1508f33700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f14ec053a70 0x7f14ec055f30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f14f4008000 tx=0x7f14f4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.469+0000 7f14fa7fc700 1 -- 172.21.15.3:0/4172942178 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1504143540 con 0x7f15040b0140 2022-07-21T01:36:50.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.593+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f15040b0d20 con 0x7f15040b0140 2022-07-21T01:36:50.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.593+0000 7f14fa7fc700 1 -- 172.21.15.3:0/4172942178 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v54) v1 ==== 92+0+10 (secure 0 0 0) 0x7f15000150a0 con 0x7f15040b0140 2022-07-21T01:36:50.597 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:36:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f14ec053a70 msgr2=0x7f14ec055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f14ec053a70 0x7f14ec055f30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f14f4008000 tx=0x7f14f4009040).stop 2022-07-21T01:36:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040b0140 msgr2=0x7f15040aa660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040b0140 0x7f15040aa660 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f15000032f0 tx=0x7f1500016000).stop 2022-07-21T01:36:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 shutdown_connections 2022-07-21T01:36:50.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15040b2430 0x7f15041437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:36:50.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f14ec053a70 0x7f14ec055f30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15040b2050 0x7f1504140420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 --2- 172.21.15.3:0/4172942178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15040b0140 0x7f15040aa660 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 >> 172.21.15.3:0/4172942178 conn(0x7f150405f840 msgr2=0x7f1504146af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:50.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 shutdown_connections 2022-07-21T01:36:50.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.597+0000 7f150b998700 1 -- 172.21.15.3:0/4172942178 wait complete. 2022-07-21T01:36:50.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:36:50.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9e7f4700 1 Processor -- start 2022-07-21T01:36:50.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9e7f4700 1 -- start start 2022-07-21T01:36:50.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9e7f4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4f980b2710 0x7f4f981401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9e7f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980b0270 0x7f4f981406d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9e7f4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f98067a10 con 0x7f4f980ac880 2022-07-21T01:36:50.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9e7f4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f980ad740 con 0x7f4f980b2710 2022-07-21T01:36:50.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9e7f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f98067cb0 con 0x7f4f980b0270 2022-07-21T01:36:50.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9d7f2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4f980b2710 0x7f4f981401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9dff3700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4f980ac880 0x7f4f98143000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:55774/0 (socket says 172.21.15.3:55774) 2022-07-21T01:36:50.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9dff3700 1 -- 172.21.15.3:0/2398782304 learned_addr learned my addr 172.21.15.3:0/2398782304 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:50.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9d7f2700 1 --2- 172.21.15.3:0/2398782304 >> v2:172.21.15.3:3300/0 conn(0x7f4f980b2710 0x7f4f981401e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45376/0 (socket says 172.21.15.3:45376) 2022-07-21T01:36:50.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9cff1700 1 --2- 172.21.15.3:0/2398782304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980b0270 0x7f4f981406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f8ffff700 1 -- 172.21.15.3:0/2398782304 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375260677 0 0) 0x7f4f98067a10 con 0x7f4f980ac880 2022-07-21T01:36:50.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f8ffff700 1 -- 172.21.15.3:0/2398782304 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f7c003260 con 0x7f4f980ac880 2022-07-21T01:36:50.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9d7f2700 1 -- 172.21.15.3:0/2398782304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980b0270 msgr2=0x7f4f981406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:50.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9d7f2700 1 --2- 172.21.15.3:0/2398782304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980b0270 0x7f4f981406d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9d7f2700 1 -- 172.21.15.3:0/2398782304 >> v1:172.21.15.143:6789/0 conn(0x7f4f980ac880 legacy=0x7f4f98143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9d7f2700 1 -- 172.21.15.3:0/2398782304 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f98147160 con 0x7f4f980b2710 2022-07-21T01:36:50.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.709+0000 7f4f9cff1700 1 --2- 172.21.15.3:0/2398782304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980b0270 0x7f4f981406d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:50.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9d7f2700 1 --2- 172.21.15.3:0/2398782304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 0x7f4f981401e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f4f88002f70 tx=0x7f4f88006df0).ready entity=mon.0 client_cookie=b120d51000878981 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f8ffff700 1 -- 172.21.15.3:0/2398782304 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f8800d4a0 con 0x7f4f980b2710 2022-07-21T01:36:50.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f8ffff700 1 -- 172.21.15.3:0/2398782304 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f88000e90 con 0x7f4f980b2710 2022-07-21T01:36:50.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f8ffff700 1 -- 172.21.15.3:0/2398782304 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f8800d7f0 con 0x7f4f980b2710 2022-07-21T01:36:50.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/2398782304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 msgr2=0x7f4f981401e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 --2- 172.21.15.3:0/2398782304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 0x7f4f981401e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f4f88002f70 tx=0x7f4f88006df0).stop 2022-07-21T01:36:50.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/2398782304 shutdown_connections 2022-07-21T01:36:50.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 --2- 172.21.15.3:0/2398782304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980b0270 0x7f4f981406d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 --2- 172.21.15.3:0/2398782304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 0x7f4f981401e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/2398782304 >> 172.21.15.3:0/2398782304 conn(0x7f4f98060310 msgr2=0x7f4f980b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:50.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/2398782304 shutdown_connections 2022-07-21T01:36:50.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/2398782304 wait complete. 2022-07-21T01:36:50.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 Processor -- start 2022-07-21T01:36:50.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.713+0000 7f4f9e7f4700 1 -- start start 2022-07-21T01:36:50.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9e7f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980ac880 0x7f4f981527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9e7f4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f980b0270 0x7f4f9809f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9e7f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 0x7f4f9809fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9e7f4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f98141670 con 0x7f4f980b2710 2022-07-21T01:36:50.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9e7f4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9813d8e0 con 0x7f4f980b0270 2022-07-21T01:36:50.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9e7f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f981418a0 con 0x7f4f980ac880 2022-07-21T01:36:50.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9dff3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 0x7f4f9809fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9dff3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 0x7f4f9809fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45384/0 (socket says 172.21.15.3:45384) 2022-07-21T01:36:50.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9dff3700 1 -- 172.21.15.3:0/908059549 learned_addr learned my addr 172.21.15.3:0/908059549 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:50.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9d7f2700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980ac880 0x7f4f981527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9cff1700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f980b0270 0x7f4f9809f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9dff3700 1 -- 172.21.15.3:0/908059549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980ac880 msgr2=0x7f4f981527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9dff3700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980ac880 0x7f4f981527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9dff3700 1 -- 172.21.15.3:0/908059549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f980b0270 msgr2=0x7f4f9809f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9dff3700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f980b0270 0x7f4f9809f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9dff3700 1 -- 172.21.15.3:0/908059549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f94009470 con 0x7f4f980b2710 2022-07-21T01:36:50.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9d7f2700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980ac880 0x7f4f981527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:36:50.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9dff3700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 0x7f4f9809fcb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f4f90001110 tx=0x7f4f90007a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f8e7fc700 1 -- 172.21.15.3:0/908059549 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f900070d0 con 0x7f4f980b2710 2022-07-21T01:36:50.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f8e7fc700 1 -- 172.21.15.3:0/908059549 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f90007230 con 0x7f4f980b2710 2022-07-21T01:36:50.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f8e7fc700 1 -- 172.21.15.3:0/908059549 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f90005610 con 0x7f4f980b2710 2022-07-21T01:36:50.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8800b180 con 0x7f4f980b2710 2022-07-21T01:36:50.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f980a8ac0 con 0x7f4f980b2710 2022-07-21T01:36:50.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.717+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f980ac310 con 0x7f4f980b2710 2022-07-21T01:36:50.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.721+0000 7f4f8e7fc700 1 -- 172.21.15.3:0/908059549 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4f90010070 con 0x7f4f980b2710 2022-07-21T01:36:50.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.721+0000 7f4f8e7fc700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f80053a70 0x7f4f80055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.721+0000 7f4f8e7fc700 1 -- 172.21.15.3:0/908059549 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f4f9006b4d0 con 0x7f4f980b2710 2022-07-21T01:36:50.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.721+0000 7f4f9d7f2700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f80053a70 0x7f4f80055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.721+0000 7f4f9d7f2700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f80053a70 0x7f4f80055f30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f4f88015000 tx=0x7f4f88014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.729+0000 7f4f8e7fc700 1 -- 172.21.15.3:0/908059549 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f900424c0 con 0x7f4f980b2710 2022-07-21T01:36:50.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.857+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4f98143970 con 0x7f4f980b2710 2022-07-21T01:36:50.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.857+0000 7f4f8e7fc700 1 -- 172.21.15.3:0/908059549 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) v1 ==== 84+0+143 (secure 0 0 0) 0x7f4f90045ae0 con 0x7f4f980b2710 2022-07-21T01:36:50.858 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:50.858 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:36:50.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:50.859 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:36:50.859 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:36:50.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:36:50.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:36:50.860 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:36:50.860 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:36:50.860 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:36:50.861 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:36:50.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f80053a70 msgr2=0x7f4f80055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f80053a70 0x7f4f80055f30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f4f88015000 tx=0x7f4f88014000).stop 2022-07-21T01:36:50.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 msgr2=0x7f4f9809fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 0x7f4f9809fcb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f4f90001110 tx=0x7f4f90007a40).stop 2022-07-21T01:36:50.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 shutdown_connections 2022-07-21T01:36:50.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f980b2710 0x7f4f9809fcb0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f80053a70 0x7f4f80055f30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f980b0270 0x7f4f9809f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 --2- 172.21.15.3:0/908059549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f980ac880 0x7f4f981527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 >> 172.21.15.3:0/908059549 conn(0x7f4f98060310 msgr2=0x7f4f980b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:50.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 shutdown_connections 2022-07-21T01:36:50.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.861+0000 7f4f9e7f4700 1 -- 172.21.15.3:0/908059549 wait complete. 2022-07-21T01:36:50.871 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:36:50.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:36:50.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.981+0000 7f6fcf59e700 1 Processor -- start 2022-07-21T01:36:50.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.981+0000 7f6fcf59e700 1 -- start start 2022-07-21T01:36:50.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6fd00619b0 0x7f6fd00b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd0061d90 con 0x7f6fd00b6ee0 2022-07-21T01:36:50.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00ba5a0 con 0x7f6fd00b5ed0 2022-07-21T01:36:50.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd0143fa0 con 0x7f6fd00619b0 2022-07-21T01:36:50.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fce59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6fd00619b0 0x7f6fd00b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fc7fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6fd00b6ee0 0x7f6fd0143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53670/0 (socket says 172.21.15.3:53670) 2022-07-21T01:36:50.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fc7fff700 1 -- 172.21.15.3:0/2512256792 learned_addr learned my addr 172.21.15.3:0/2512256792 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:50.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fce59c700 1 -- 172.21.15.3:0/2512256792 >> v1:172.21.15.143:6789/0 conn(0x7f6fd00b5ed0 legacy=0x7f6fd00b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fce59c700 1 -- 172.21.15.3:0/2512256792 >> v1:172.21.15.3:6789/0 conn(0x7f6fd00b6ee0 legacy=0x7f6fd0143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fce59c700 1 -- 172.21.15.3:0/2512256792 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0143150 con 0x7f6fd00619b0 2022-07-21T01:36:50.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fce59c700 1 --2- 172.21.15.3:0/2512256792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00619b0 0x7f6fd00b48a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f6fc0002f70 tx=0x7f6fc0006df0).ready entity=mon.0 client_cookie=d2f25d94fff5b4f0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcdd9b700 1 -- 172.21.15.3:0/2512256792 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fc000d610 con 0x7f6fd00619b0 2022-07-21T01:36:50.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcdd9b700 1 -- 172.21.15.3:0/2512256792 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc0000e90 con 0x7f6fd00619b0 2022-07-21T01:36:50.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcdd9b700 1 -- 172.21.15.3:0/2512256792 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fc000d960 con 0x7f6fd00619b0 2022-07-21T01:36:50.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 -- 172.21.15.3:0/2512256792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00619b0 msgr2=0x7f6fd00b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 --2- 172.21.15.3:0/2512256792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00619b0 0x7f6fd00b48a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f6fc0002f70 tx=0x7f6fc0006df0).stop 2022-07-21T01:36:50.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 -- 172.21.15.3:0/2512256792 shutdown_connections 2022-07-21T01:36:50.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 --2- 172.21.15.3:0/2512256792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00619b0 0x7f6fd00b48a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 -- 172.21.15.3:0/2512256792 >> 172.21.15.3:0/2512256792 conn(0x7f6fd006af80 msgr2=0x7f6fd005fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:50.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 -- 172.21.15.3:0/2512256792 shutdown_connections 2022-07-21T01:36:50.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.985+0000 7f6fcf59e700 1 -- 172.21.15.3:0/2512256792 wait complete. 2022-07-21T01:36:50.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 Processor -- start 2022-07-21T01:36:50.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 -- start start 2022-07-21T01:36:50.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd00619b0 0x7f6fd00afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00b5ed0 0x7f6fd0112a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd00b6ee0 0x7f6fd0144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:50.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0146e40 con 0x7f6fd00b5ed0 2022-07-21T01:36:50.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0115540 con 0x7f6fd00619b0 2022-07-21T01:36:50.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0145fe0 con 0x7f6fd00b6ee0 2022-07-21T01:36:50.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00b5ed0 0x7f6fd0112a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00b5ed0 0x7f6fd0112a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45396/0 (socket says 172.21.15.3:45396) 2022-07-21T01:36:50.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc7fff700 1 -- 172.21.15.3:0/1896999044 learned_addr learned my addr 172.21.15.3:0/1896999044 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:50.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fce59c700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd00619b0 0x7f6fd00afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fced9d700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd00b6ee0 0x7f6fd0144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:50.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc7fff700 1 -- 172.21.15.3:0/1896999044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd00b6ee0 msgr2=0x7f6fd0144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc7fff700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd00b6ee0 0x7f6fd0144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc7fff700 1 -- 172.21.15.3:0/1896999044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd00619b0 msgr2=0x7f6fd00afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:50.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc7fff700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd00619b0 0x7f6fd00afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:50.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc7fff700 1 -- 172.21.15.3:0/1896999044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc000b180 con 0x7f6fd00b5ed0 2022-07-21T01:36:50.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc7fff700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00b5ed0 0x7f6fd0112a30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f6fd00b41b0 tx=0x7f6fb8007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:50.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc77fe700 1 -- 172.21.15.3:0/1896999044 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fb80070d0 con 0x7f6fd00b5ed0 2022-07-21T01:36:50.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc77fe700 1 -- 172.21.15.3:0/1896999044 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb8007230 con 0x7f6fd00b5ed0 2022-07-21T01:36:50.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd01458e0 con 0x7f6fd00b5ed0 2022-07-21T01:36:50.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fc77fe700 1 -- 172.21.15.3:0/1896999044 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fb8005610 con 0x7f6fd00b5ed0 2022-07-21T01:36:50.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.989+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd0147030 con 0x7f6fd00b5ed0 2022-07-21T01:36:51.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.993+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd00574e0 con 0x7f6fd00b5ed0 2022-07-21T01:36:51.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.993+0000 7f6fc77fe700 1 -- 172.21.15.3:0/1896999044 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6fb8010070 con 0x7f6fd00b5ed0 2022-07-21T01:36:51.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.993+0000 7f6fc77fe700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6fb4053b00 0x7f6fb4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:51.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.993+0000 7f6fc77fe700 1 -- 172.21.15.3:0/1896999044 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f6fb806b380 con 0x7f6fd00b5ed0 2022-07-21T01:36:51.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.993+0000 7f6fce59c700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6fb4053b00 0x7f6fb4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:51.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.993+0000 7f6fce59c700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6fb4053b00 0x7f6fb4055fc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6fc001c000 tx=0x7f6fc0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:51.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:50.997+0000 7f6fc77fe700 1 -- 172.21.15.3:0/1896999044 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6fb8042370 con 0x7f6fd00b5ed0 2022-07-21T01:36:51.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:51.125+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f6fd005db60 con 0x7f6fd00b5ed0 2022-07-21T01:36:52.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.041+0000 7f6fc77fe700 1 -- 172.21.15.3:0/1896999044 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v55) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6fb8045990 con 0x7f6fd00b5ed0 2022-07-21T01:36:52.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6fb4053b00 msgr2=0x7f6fb4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6fb4053b00 0x7f6fb4055fc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6fc001c000 tx=0x7f6fc0015000).stop 2022-07-21T01:36:52.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00b5ed0 msgr2=0x7f6fd0112a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00b5ed0 0x7f6fd0112a30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f6fd00b41b0 tx=0x7f6fb8007c10).stop 2022-07-21T01:36:52.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 shutdown_connections 2022-07-21T01:36:52.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd00b6ee0 0x7f6fd0144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fd00b5ed0 0x7f6fd0112a30 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6fb4053b00 0x7f6fb4055fc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 --2- 172.21.15.3:0/1896999044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd00619b0 0x7f6fd00afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 >> 172.21.15.3:0/1896999044 conn(0x7f6fd006af80 msgr2=0x7f6fd013f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:52.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 shutdown_connections 2022-07-21T01:36:52.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.045+0000 7f6fcf59e700 1 -- 172.21.15.3:0/1896999044 wait complete. 2022-07-21T01:36:52.049 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:36:52.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:36:52.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db9f0e700 1 Processor -- start 2022-07-21T01:36:52.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db9f0e700 1 -- start start 2022-07-21T01:36:52.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db9f0e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8db40b2710 0x7f8db414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db9f0e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db40a7250 con 0x7f8db40b01a0 2022-07-21T01:36:52.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db9f0e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db40a7480 con 0x7f8db40a8c50 2022-07-21T01:36:52.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db9f0e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db40a7720 con 0x7f8db40b2710 2022-07-21T01:36:52.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db8f0c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8db40b2710 0x7f8db414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db8f0c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8db40b2710 0x7f8db414afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45404/0 (socket says 172.21.15.3:45404) 2022-07-21T01:36:52.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db3fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8db40b01a0 0x7f8db40a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53682/0 (socket says 172.21.15.3:53682) 2022-07-21T01:36:52.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db8f0c700 1 -- 172.21.15.3:0/1071353554 learned_addr learned my addr 172.21.15.3:0/1071353554 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:52.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db8f0c700 1 -- 172.21.15.3:0/1071353554 >> v1:172.21.15.143:6789/0 conn(0x7f8db40a8c50 legacy=0x7f8db4151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db8f0c700 1 -- 172.21.15.3:0/1071353554 >> v1:172.21.15.3:6789/0 conn(0x7f8db40b01a0 legacy=0x7f8db40a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db8f0c700 1 -- 172.21.15.3:0/1071353554 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db40a79c0 con 0x7f8db40b2710 2022-07-21T01:36:52.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db8f0c700 1 --2- 172.21.15.3:0/1071353554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40b2710 0x7f8db414afd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f8da8002f70 tx=0x7f8da8006eb0).ready entity=mon.0 client_cookie=dd773f3e89d8b739 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:52.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db37fe700 1 -- 172.21.15.3:0/1071353554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8da800d660 con 0x7f8db40b2710 2022-07-21T01:36:52.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db37fe700 1 -- 172.21.15.3:0/1071353554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da8000e90 con 0x7f8db40b2710 2022-07-21T01:36:52.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.169+0000 7f8db37fe700 1 -- 172.21.15.3:0/1071353554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8da800d9f0 con 0x7f8db40b2710 2022-07-21T01:36:52.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 -- 172.21.15.3:0/1071353554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40b2710 msgr2=0x7f8db414afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 --2- 172.21.15.3:0/1071353554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40b2710 0x7f8db414afd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f8da8002f70 tx=0x7f8da8006eb0).stop 2022-07-21T01:36:52.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 -- 172.21.15.3:0/1071353554 shutdown_connections 2022-07-21T01:36:52.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 --2- 172.21.15.3:0/1071353554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40b2710 0x7f8db414afd0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 -- 172.21.15.3:0/1071353554 >> 172.21.15.3:0/1071353554 conn(0x7f8db4060310 msgr2=0x7f8db40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:52.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 -- 172.21.15.3:0/1071353554 shutdown_connections 2022-07-21T01:36:52.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 -- 172.21.15.3:0/1071353554 wait complete. 2022-07-21T01:36:52.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 Processor -- start 2022-07-21T01:36:52.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 -- start start 2022-07-21T01:36:52.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.173+0000 7f8db9f0e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40a8c50 0x7f8db4066110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db9f0e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8db40b01a0 0x7f8db40ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db9f0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8db40b2710 0x7f8db4063e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db9f0e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40655c0 con 0x7f8db40a8c50 2022-07-21T01:36:52.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db9f0e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40ac570 con 0x7f8db40b2710 2022-07-21T01:36:52.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db9f0e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4157060 con 0x7f8db40b01a0 2022-07-21T01:36:52.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db8f0c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40a8c50 0x7f8db4066110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db8f0c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40a8c50 0x7f8db4066110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45408/0 (socket says 172.21.15.3:45408) 2022-07-21T01:36:52.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db8f0c700 1 -- 172.21.15.3:0/3959430039 learned_addr learned my addr 172.21.15.3:0/3959430039 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db3fff700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8db40b01a0 0x7f8db40ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db970d700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8db40b2710 0x7f8db4063e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db8f0c700 1 -- 172.21.15.3:0/3959430039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8db40b01a0 msgr2=0x7f8db40ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db8f0c700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8db40b01a0 0x7f8db40ab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db8f0c700 1 -- 172.21.15.3:0/3959430039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8db40b2710 msgr2=0x7f8db4063e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db8f0c700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8db40b2710 0x7f8db4063e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db8f0c700 1 -- 172.21.15.3:0/3959430039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da800b1e0 con 0x7f8db40a8c50 2022-07-21T01:36:52.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db8f0c700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40a8c50 0x7f8db4066110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f8db41571f0 tx=0x7f8da8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:52.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db1ffb700 1 -- 172.21.15.3:0/3959430039 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8da800d9e0 con 0x7f8db40a8c50 2022-07-21T01:36:52.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db1ffb700 1 -- 172.21.15.3:0/3959430039 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da8011b40 con 0x7f8db40a8c50 2022-07-21T01:36:52.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db40ac2d0 con 0x7f8db40a8c50 2022-07-21T01:36:52.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db1ffb700 1 -- 172.21.15.3:0/3959430039 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8da8011ea0 con 0x7f8db40a8c50 2022-07-21T01:36:52.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db4063930 con 0x7f8db40a8c50 2022-07-21T01:36:52.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.177+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db4066800 con 0x7f8db40a8c50 2022-07-21T01:36:52.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.181+0000 7f8db1ffb700 1 -- 172.21.15.3:0/3959430039 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8da8025030 con 0x7f8db40a8c50 2022-07-21T01:36:52.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.181+0000 7f8db1ffb700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8d94053b30 0x7f8d94055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.181+0000 7f8db1ffb700 1 -- 172.21.15.3:0/3959430039 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f8da806d240 con 0x7f8db40a8c50 2022-07-21T01:36:52.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.181+0000 7f8db3fff700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8d94053b30 0x7f8d94055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.181+0000 7f8db3fff700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8d94053b30 0x7f8d94055ff0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f8da400d000 tx=0x7f8da4007f90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:52.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.185+0000 7f8db1ffb700 1 -- 172.21.15.3:0/3959430039 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8da8044130 con 0x7f8db40a8c50 2022-07-21T01:36:52.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.313+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f8db4157230 con 0x7f8db40a8c50 2022-07-21T01:36:52.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.373+0000 7f8db1ffb700 1 -- 172.21.15.3:0/3959430039 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v56) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8da80230a0 con 0x7f8db40a8c50 2022-07-21T01:36:52.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.377+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8d94053b30 msgr2=0x7f8d94055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.377+0000 7f8db9f0e700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8d94053b30 0x7f8d94055ff0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f8da400d000 tx=0x7f8da4007f90).stop 2022-07-21T01:36:52.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.377+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40a8c50 msgr2=0x7f8db4066110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.377+0000 7f8db9f0e700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40a8c50 0x7f8db4066110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f8db41571f0 tx=0x7f8da8015000).stop 2022-07-21T01:36:52.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.381+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 shutdown_connections 2022-07-21T01:36:52.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.381+0000 7f8db9f0e700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8d94053b30 0x7f8d94055ff0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.381+0000 7f8db9f0e700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8db40b2710 0x7f8db4063e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.381+0000 7f8db9f0e700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8db40b01a0 0x7f8db40ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.381+0000 7f8db9f0e700 1 --2- 172.21.15.3:0/3959430039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8db40a8c50 0x7f8db4066110 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.381+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 >> 172.21.15.3:0/3959430039 conn(0x7f8db4060310 msgr2=0x7f8db40676e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:52.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.381+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 shutdown_connections 2022-07-21T01:36:52.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.381+0000 7f8db9f0e700 1 -- 172.21.15.3:0/3959430039 wait complete. 2022-07-21T01:36:52.383 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:36:52.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:36:52.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f501b59e700 1 Processor -- start 2022-07-21T01:36:52.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f501b59e700 1 -- start start 2022-07-21T01:36:52.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f501b59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f501c0614b0 0x7f501c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f501b59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c0aee80 con 0x7f501c0ad750 2022-07-21T01:36:52.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f501b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c0b70b0 con 0x7f501c1405d0 2022-07-21T01:36:52.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f501b59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501c0af0b0 con 0x7f501c0614b0 2022-07-21T01:36:52.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f501a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f501c0614b0 0x7f501c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f5019d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f501c0ad750 0x7f501c0b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53698/0 (socket says 172.21.15.3:53698) 2022-07-21T01:36:52.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f5019d9b700 1 -- 172.21.15.3:0/1092963241 learned_addr learned my addr 172.21.15.3:0/1092963241 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:52.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f501959a700 1 -- 172.21.15.3:0/1092963241 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914365421 0 0) 0x7f501c0aee80 con 0x7f501c0ad750 2022-07-21T01:36:52.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.493+0000 7f501959a700 1 -- 172.21.15.3:0/1092963241 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5000003260 con 0x7f501c0ad750 2022-07-21T01:36:52.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501959a700 1 -- 172.21.15.3:0/1092963241 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498478685 0 0) 0x7f501c0b70b0 con 0x7f501c1405d0 2022-07-21T01:36:52.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501a59c700 1 -- 172.21.15.3:0/1092963241 >> v1:172.21.15.143:6789/0 conn(0x7f501c1405d0 legacy=0x7f501c1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501a59c700 1 -- 172.21.15.3:0/1092963241 >> v1:172.21.15.3:6789/0 conn(0x7f501c0ad750 legacy=0x7f501c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501a59c700 1 -- 172.21.15.3:0/1092963241 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c0af350 con 0x7f501c0614b0 2022-07-21T01:36:52.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501a59c700 1 --2- 172.21.15.3:0/1092963241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0614b0 0x7f501c061890 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f501c0aed30 tx=0x7f5010006df0).ready entity=mon.0 client_cookie=53bc977736074b89 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:52.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501959a700 1 -- 172.21.15.3:0/1092963241 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f501000d610 con 0x7f501c0614b0 2022-07-21T01:36:52.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501959a700 1 -- 172.21.15.3:0/1092963241 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5010000e90 con 0x7f501c0614b0 2022-07-21T01:36:52.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501959a700 1 -- 172.21.15.3:0/1092963241 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f501000d960 con 0x7f501c0614b0 2022-07-21T01:36:52.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501b59e700 1 -- 172.21.15.3:0/1092963241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0614b0 msgr2=0x7f501c061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501b59e700 1 --2- 172.21.15.3:0/1092963241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0614b0 0x7f501c061890 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f501c0aed30 tx=0x7f5010006df0).stop 2022-07-21T01:36:52.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501b59e700 1 -- 172.21.15.3:0/1092963241 shutdown_connections 2022-07-21T01:36:52.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501b59e700 1 --2- 172.21.15.3:0/1092963241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0614b0 0x7f501c061890 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501b59e700 1 -- 172.21.15.3:0/1092963241 >> 172.21.15.3:0/1092963241 conn(0x7f501c066f70 msgr2=0x7f501c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:52.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501b59e700 1 -- 172.21.15.3:0/1092963241 shutdown_connections 2022-07-21T01:36:52.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501b59e700 1 -- 172.21.15.3:0/1092963241 wait complete. 2022-07-21T01:36:52.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501b59e700 1 Processor -- start 2022-07-21T01:36:52.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.497+0000 7f501b59e700 1 -- start start 2022-07-21T01:36:52.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f501c0614b0 0x7f501c0b8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0ad750 0x7f501c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f501c1405d0 0x7f501c09ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c0b1340 con 0x7f501c0ad750 2022-07-21T01:36:52.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c0b28a0 con 0x7f501c0614b0 2022-07-21T01:36:52.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501c0b2b40 con 0x7f501c1405d0 2022-07-21T01:36:52.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f5019d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0ad750 0x7f501c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f5019d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0ad750 0x7f501c09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45424/0 (socket says 172.21.15.3:45424) 2022-07-21T01:36:52.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f5019d9b700 1 -- 172.21.15.3:0/611298923 learned_addr learned my addr 172.21.15.3:0/611298923 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:52.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501a59c700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f501c0614b0 0x7f501c0b8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501ad9d700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f501c1405d0 0x7f501c09ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f5019d9b700 1 -- 172.21.15.3:0/611298923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f501c1405d0 msgr2=0x7f501c09ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f5019d9b700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f501c1405d0 0x7f501c09ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f5019d9b700 1 -- 172.21.15.3:0/611298923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f501c0614b0 msgr2=0x7f501c0b8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f5019d9b700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f501c0614b0 0x7f501c0b8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f5019d9b700 1 -- 172.21.15.3:0/611298923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501000b180 con 0x7f501c0ad750 2022-07-21T01:36:52.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f5019d9b700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0ad750 0x7f501c09ea40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f500c001110 tx=0x7f500c007a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:52.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f500b7fe700 1 -- 172.21.15.3:0/611298923 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f500c0070d0 con 0x7f501c0ad750 2022-07-21T01:36:52.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f500b7fe700 1 -- 172.21.15.3:0/611298923 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c007230 con 0x7f501c0ad750 2022-07-21T01:36:52.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501c0b10a0 con 0x7f501c0ad750 2022-07-21T01:36:52.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f500b7fe700 1 -- 172.21.15.3:0/611298923 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f500c005610 con 0x7f501c0ad750 2022-07-21T01:36:52.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501c0b0af0 con 0x7f501c0ad750 2022-07-21T01:36:52.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501c0af5f0 con 0x7f501c0ad750 2022-07-21T01:36:52.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f500b7fe700 1 -- 172.21.15.3:0/611298923 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f500c01a070 con 0x7f501c0ad750 2022-07-21T01:36:52.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f500b7fe700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ff8053b00 0x7f4ff8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f500b7fe700 1 -- 172.21.15.3:0/611298923 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f500c06b2c0 con 0x7f501c0ad750 2022-07-21T01:36:52.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.501+0000 7f501a59c700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ff8053b00 0x7f4ff8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.505+0000 7f501a59c700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ff8053b00 0x7f4ff8055fc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f5010018000 tx=0x7f5010014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:52.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.509+0000 7f500b7fe700 1 -- 172.21.15.3:0/611298923 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f500c042060 con 0x7f501c0ad750 2022-07-21T01:36:52.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.673+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f501c0574e0 con 0x7f501c0ad750 2022-07-21T01:36:52.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.685+0000 7f500b7fe700 1 -- 172.21.15.3:0/611298923 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 9 and data pool 10 v22) v1 ==== 188+0+0 (secure 0 0 0) 0x7f500c045680 con 0x7f501c0ad750 2022-07-21T01:36:52.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ff8053b00 msgr2=0x7f4ff8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ff8053b00 0x7f4ff8055fc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f5010018000 tx=0x7f5010014000).stop 2022-07-21T01:36:52.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0ad750 msgr2=0x7f501c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0ad750 0x7f501c09ea40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f500c001110 tx=0x7f500c007a40).stop 2022-07-21T01:36:52.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 shutdown_connections 2022-07-21T01:36:52.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f501c1405d0 0x7f501c09ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f501c0ad750 0x7f501c09ea40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4ff8053b00 0x7f4ff8055fc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 --2- 172.21.15.3:0/611298923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f501c0614b0 0x7f501c0b8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 >> 172.21.15.3:0/611298923 conn(0x7f501c066f70 msgr2=0x7f501c145220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:52.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 shutdown_connections 2022-07-21T01:36:52.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.689+0000 7f501b59e700 1 -- 172.21.15.3:0/611298923 wait complete. 2022-07-21T01:36:52.695 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 9 and data pool 10 2022-07-21T01:36:52.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:36:52.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 Processor -- start 2022-07-21T01:36:52.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 -- start start 2022-07-21T01:36:52.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f72400b2710 0x7f724014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72400b01a0 0x7f724014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72400a7250 con 0x7f72400afd60 2022-07-21T01:36:52.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72400a7480 con 0x7f72400b01a0 2022-07-21T01:36:52.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72400a7720 con 0x7f72400b2710 2022-07-21T01:36:52.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723e7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72400b01a0 0x7f724014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f72400afd60 0x7f72400a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53724/0 (socket says 172.21.15.3:53724) 2022-07-21T01:36:52.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723f7fe700 1 -- 172.21.15.3:0/2122002977 learned_addr learned my addr 172.21.15.3:0/2122002977 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:52.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723effd700 1 --2- 172.21.15.3:0/2122002977 >> v2:172.21.15.143:3300/0 conn(0x7f72400b2710 0x7f724014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723e7fc700 1 -- 172.21.15.3:0/2122002977 >> v2:172.21.15.143:3300/0 conn(0x7f72400b2710 msgr2=0x7f724014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:52.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723e7fc700 1 --2- 172.21.15.3:0/2122002977 >> v2:172.21.15.143:3300/0 conn(0x7f72400b2710 0x7f724014afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723e7fc700 1 -- 172.21.15.3:0/2122002977 >> v1:172.21.15.3:6789/0 conn(0x7f72400afd60 legacy=0x7f72400a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723dffb700 1 -- 172.21.15.3:0/2122002977 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664603471 0 0) 0x7f72400a7250 con 0x7f72400afd60 2022-07-21T01:36:52.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723e7fc700 1 -- 172.21.15.3:0/2122002977 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72400a79c0 con 0x7f72400b01a0 2022-07-21T01:36:52.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723e7fc700 1 --2- 172.21.15.3:0/2122002977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 0x7f724014b4c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f7230002f70 tx=0x7f7230006df0).ready entity=mon.0 client_cookie=326911bc56734cff server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:52.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723dffb700 1 -- 172.21.15.3:0/2122002977 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f723000d570 con 0x7f72400b01a0 2022-07-21T01:36:52.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723dffb700 1 -- 172.21.15.3:0/2122002977 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7230000e90 con 0x7f72400b01a0 2022-07-21T01:36:52.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723dffb700 1 -- 172.21.15.3:0/2122002977 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f723000d8c0 con 0x7f72400b01a0 2022-07-21T01:36:52.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 -- 172.21.15.3:0/2122002977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 msgr2=0x7f724014b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 --2- 172.21.15.3:0/2122002977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 0x7f724014b4c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f7230002f70 tx=0x7f7230006df0).stop 2022-07-21T01:36:52.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 -- 172.21.15.3:0/2122002977 shutdown_connections 2022-07-21T01:36:52.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 --2- 172.21.15.3:0/2122002977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 0x7f724014b4c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 --2- 172.21.15.3:0/2122002977 >> v2:172.21.15.143:3300/0 conn(0x7f72400b2710 0x7f724014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 -- 172.21.15.3:0/2122002977 >> 172.21.15.3:0/2122002977 conn(0x7f7240060310 msgr2=0x7f72400b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:52.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 -- 172.21.15.3:0/2122002977 shutdown_connections 2022-07-21T01:36:52.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.801+0000 7f723ffff700 1 -- 172.21.15.3:0/2122002977 wait complete. 2022-07-21T01:36:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 Processor -- start 2022-07-21T01:36:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 -- start start 2022-07-21T01:36:52.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400afd60 0x7f72400ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 0x7f7240156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72400b2710 0x7f7240156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724014adc0 con 0x7f72400b01a0 2022-07-21T01:36:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724014b060 con 0x7f72400afd60 2022-07-21T01:36:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724014b300 con 0x7f72400b2710 2022-07-21T01:36:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 0x7f7240156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 0x7f7240156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45446/0 (socket says 172.21.15.3:45446) 2022-07-21T01:36:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723e7fc700 1 -- 172.21.15.3:0/3360538746 learned_addr learned my addr 172.21.15.3:0/3360538746 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723f7fe700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72400b2710 0x7f7240156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723effd700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400afd60 0x7f72400ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723e7fc700 1 -- 172.21.15.3:0/3360538746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72400b2710 msgr2=0x7f7240156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723e7fc700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72400b2710 0x7f7240156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723e7fc700 1 -- 172.21.15.3:0/3360538746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400afd60 msgr2=0x7f72400ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723e7fc700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400afd60 0x7f72400ac130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723e7fc700 1 -- 172.21.15.3:0/3360538746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723000b180 con 0x7f72400b01a0 2022-07-21T01:36:52.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723e7fc700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 0x7f7240156680 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f7230002f70 tx=0x7f7230014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:52.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f722ffff700 1 -- 172.21.15.3:0/3360538746 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f723000dea0 con 0x7f72400b01a0 2022-07-21T01:36:52.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f722ffff700 1 -- 172.21.15.3:0/3360538746 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7230013de0 con 0x7f72400b01a0 2022-07-21T01:36:52.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724015a970 con 0x7f72400b01a0 2022-07-21T01:36:52.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f722ffff700 1 -- 172.21.15.3:0/3360538746 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f723001e5f0 con 0x7f72400b01a0 2022-07-21T01:36:52.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724015a4c0 con 0x7f72400b01a0 2022-07-21T01:36:52.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.805+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724014b4c0 con 0x7f72400b01a0 2022-07-21T01:36:52.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.809+0000 7f722ffff700 1 -- 172.21.15.3:0/3360538746 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7230025030 con 0x7f72400b01a0 2022-07-21T01:36:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.809+0000 7f722ffff700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f721c0539e0 0x7f721c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.809+0000 7f722ffff700 1 -- 172.21.15.3:0/3360538746 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f723006ce20 con 0x7f72400b01a0 2022-07-21T01:36:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.809+0000 7f723effd700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f721c0539e0 0x7f721c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.809+0000 7f723effd700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f721c0539e0 0x7f721c055ea0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f723400b000 tx=0x7f7234006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.813+0000 7f722ffff700 1 -- 172.21.15.3:0/3360538746 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7230043ad0 con 0x7f72400b01a0 2022-07-21T01:36:52.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.941+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72400574e0 con 0x7f72400b01a0 2022-07-21T01:36:52.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.941+0000 7f722ffff700 1 -- 172.21.15.3:0/3360538746 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16286 (secure 0 0 0) 0x7f7230023070 con 0x7f72400b01a0 2022-07-21T01:36:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:52.944 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":57,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:36:52.675623+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:36:51.181015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-21T01:36:52.366227+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:36:52.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f721c0539e0 msgr2=0x7f721c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f721c0539e0 0x7f721c055ea0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f723400b000 tx=0x7f7234006df0).stop 2022-07-21T01:36:52.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 msgr2=0x7f7240156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:52.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 0x7f7240156680 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f7230002f70 tx=0x7f7230014000).stop 2022-07-21T01:36:52.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 shutdown_connections 2022-07-21T01:36:52.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72400b2710 0x7f7240156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72400b01a0 0x7f7240156680 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f721c0539e0 0x7f721c055ea0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 --2- 172.21.15.3:0/3360538746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400afd60 0x7f72400ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:52.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 >> 172.21.15.3:0/3360538746 conn(0x7f7240060310 msgr2=0x7f72400675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:52.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 shutdown_connections 2022-07-21T01:36:52.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:52.945+0000 7f723ffff700 1 -- 172.21.15.3:0/3360538746 wait complete. 2022-07-21T01:36:52.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:36:53.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e1a26c700 1 Processor -- start 2022-07-21T01:36:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e1a26c700 1 -- start start 2022-07-21T01:36:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e1a26c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140b68f0 0x7f2e1413ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e1a26c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2e140aa210 0x7f2e1413d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e1a26c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2e1405ff40 0x7f2e14146970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e1a26c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1413d860 con 0x7f2e1405ff40 2022-07-21T01:36:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e1a26c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e140ac6e0 con 0x7f2e140aa210 2022-07-21T01:36:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e1a26c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140ac980 con 0x7f2e140b68f0 2022-07-21T01:36:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e18809700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2e1405ff40 0x7f2e14146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e18809700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2e1405ff40 0x7f2e14146970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45456/0 (socket says 172.21.15.3:45456) 2022-07-21T01:36:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e18809700 1 -- 172.21.15.3:0/1138306101 learned_addr learned my addr 172.21.15.3:0/1138306101 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:53.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e0bfff700 1 --2- 172.21.15.3:0/1138306101 >> v2:172.21.15.143:3300/0 conn(0x7f2e140aa210 0x7f2e1413d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e13fff700 1 --2- 172.21.15.3:0/1138306101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140b68f0 0x7f2e1413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e18809700 1 -- 172.21.15.3:0/1138306101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140b68f0 msgr2=0x7f2e1413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e18809700 1 --2- 172.21.15.3:0/1138306101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140b68f0 0x7f2e1413ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e18809700 1 -- 172.21.15.3:0/1138306101 >> v2:172.21.15.143:3300/0 conn(0x7f2e140aa210 msgr2=0x7f2e1413d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:53.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e18809700 1 --2- 172.21.15.3:0/1138306101 >> v2:172.21.15.143:3300/0 conn(0x7f2e140aa210 0x7f2e1413d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e18809700 1 -- 172.21.15.3:0/1138306101 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e140acc20 con 0x7f2e1405ff40 2022-07-21T01:36:53.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e13fff700 1 --2- 172.21.15.3:0/1138306101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140b68f0 0x7f2e1413ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:53.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.061+0000 7f2e18809700 1 --2- 172.21.15.3:0/1138306101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e1405ff40 0x7f2e14146970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f2e0c0032f0 tx=0x7f2e0c009290).ready entity=mon.0 client_cookie=db7c9677330616ac server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:53.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e137fe700 1 -- 172.21.15.3:0/1138306101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e0c00b870 con 0x7f2e1405ff40 2022-07-21T01:36:53.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e137fe700 1 -- 172.21.15.3:0/1138306101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c00b9d0 con 0x7f2e1405ff40 2022-07-21T01:36:53.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e137fe700 1 -- 172.21.15.3:0/1138306101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e0c00bb40 con 0x7f2e1405ff40 2022-07-21T01:36:53.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1138306101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e1405ff40 msgr2=0x7f2e14146970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1138306101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e1405ff40 0x7f2e14146970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f2e0c0032f0 tx=0x7f2e0c009290).stop 2022-07-21T01:36:53.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1138306101 shutdown_connections 2022-07-21T01:36:53.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1138306101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e1405ff40 0x7f2e14146970 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1138306101 >> v2:172.21.15.143:3300/0 conn(0x7f2e140aa210 0x7f2e1413d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1138306101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140b68f0 0x7f2e1413ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1138306101 >> 172.21.15.3:0/1138306101 conn(0x7f2e140b3610 msgr2=0x7f2e140b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:53.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1138306101 shutdown_connections 2022-07-21T01:36:53.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1138306101 wait complete. 2022-07-21T01:36:53.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 Processor -- start 2022-07-21T01:36:53.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 -- start start 2022-07-21T01:36:53.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e1405ff40 0x7f2e14155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140aa210 0x7f2e14155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e140b68f0 0x7f2e141564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140b53d0 con 0x7f2e140b68f0 2022-07-21T01:36:53.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e141569c0 con 0x7f2e1405ff40 2022-07-21T01:36:53.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e1a26c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14144d00 con 0x7f2e140aa210 2022-07-21T01:36:53.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.065+0000 7f2e18809700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e140b68f0 0x7f2e141564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e18809700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e140b68f0 0x7f2e141564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45462/0 (socket says 172.21.15.3:45462) 2022-07-21T01:36:53.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e18809700 1 -- 172.21.15.3:0/1340464721 learned_addr learned my addr 172.21.15.3:0/1340464721 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:53.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e0bfff700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140aa210 0x7f2e14155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e13fff700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e1405ff40 0x7f2e14155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e18809700 1 -- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140aa210 msgr2=0x7f2e14155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e18809700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140aa210 0x7f2e14155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e18809700 1 -- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e1405ff40 msgr2=0x7f2e14155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e18809700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e1405ff40 0x7f2e14155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e18809700 1 -- 172.21.15.3:0/1340464721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c009530 con 0x7f2e140b68f0 2022-07-21T01:36:53.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e0bfff700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140aa210 0x7f2e14155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:53.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e13fff700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e1405ff40 0x7f2e14155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:53.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e18809700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e140b68f0 0x7f2e141564d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f2e0c0032f0 tx=0x7f2e0c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:53.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e11ffb700 1 -- 172.21.15.3:0/1340464721 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e0c00b870 con 0x7f2e140b68f0 2022-07-21T01:36:53.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e11ffb700 1 -- 172.21.15.3:0/1340464721 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c01f410 con 0x7f2e140b68f0 2022-07-21T01:36:53.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e14144fa0 con 0x7f2e140b68f0 2022-07-21T01:36:53.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e11ffb700 1 -- 172.21.15.3:0/1340464721 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e0c01f770 con 0x7f2e140b68f0 2022-07-21T01:36:53.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e141454e0 con 0x7f2e140b68f0 2022-07-21T01:36:53.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e11ffb700 1 -- 172.21.15.3:0/1340464721 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2e0c00f070 con 0x7f2e140b68f0 2022-07-21T01:36:53.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e140acec0 con 0x7f2e140b68f0 2022-07-21T01:36:53.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e11ffb700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df8053a10 0x7f2df8055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e11ffb700 1 -- 172.21.15.3:0/1340464721 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2e0c06cb70 con 0x7f2e140b68f0 2022-07-21T01:36:53.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.069+0000 7f2e13fff700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df8053a10 0x7f2df8055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.073+0000 7f2e13fff700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df8053a10 0x7f2df8055ed0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f2e0400c000 tx=0x7f2e04008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:53.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.081+0000 7f2e11ffb700 1 -- 172.21.15.3:0/1340464721 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2e0c043970 con 0x7f2e140b68f0 2022-07-21T01:36:53.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.209+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e140574e0 con 0x7f2e140b68f0 2022-07-21T01:36:53.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.209+0000 7f2e11ffb700 1 -- 172.21.15.3:0/1340464721 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16286 (secure 0 0 0) 0x7f2e0c030fb0 con 0x7f2e140b68f0 2022-07-21T01:36:53.228 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:53.228 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":57,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:36:52.675623+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:36:51.181015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-21T01:36:52.366227+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:36:53.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df8053a10 msgr2=0x7f2df8055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df8053a10 0x7f2df8055ed0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f2e0400c000 tx=0x7f2e04008040).stop 2022-07-21T01:36:53.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e140b68f0 msgr2=0x7f2e141564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e140b68f0 0x7f2e141564d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f2e0c0032f0 tx=0x7f2e0c016000).stop 2022-07-21T01:36:53.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 shutdown_connections 2022-07-21T01:36:53.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e140b68f0 0x7f2e141564d0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e140aa210 0x7f2e14155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df8053a10 0x7f2df8055ed0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 --2- 172.21.15.3:0/1340464721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e1405ff40 0x7f2e14155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 >> 172.21.15.3:0/1340464721 conn(0x7f2e140b3610 msgr2=0x7f2e1413be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:53.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 shutdown_connections 2022-07-21T01:36:53.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.213+0000 7f2e1a26c700 1 -- 172.21.15.3:0/1340464721 wait complete. 2022-07-21T01:36:53.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:36:53.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.345+0000 7f0934088700 1 Processor -- start 2022-07-21T01:36:53.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.345+0000 7f0934088700 1 -- start start 2022-07-21T01:36:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c064a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f092c144530 0x7f092c066f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c065530 con 0x7f092c0b3a30 2022-07-21T01:36:53.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f092c066160 con 0x7f092c144530 2022-07-21T01:36:53.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092c0b3260 con 0x7f092c13e5f0 2022-07-21T01:36:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0931e24700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c064a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0932625700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f092c144530 0x7f092c066f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0931623700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f092c0b3a30 0x7f092c0b76c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:55866/0 (socket says 172.21.15.3:55866) 2022-07-21T01:36:53.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0932625700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f092c144530 0x7f092c066f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:50918/0 (socket says 172.21.15.3:50918) 2022-07-21T01:36:53.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0931623700 1 -- 172.21.15.3:0/873059403 learned_addr learned my addr 172.21.15.3:0/873059403 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0931e24700 1 -- 172.21.15.3:0/873059403 >> v2:172.21.15.143:3300/0 conn(0x7f092c144530 msgr2=0x7f092c066f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:53.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0931e24700 1 --2- 172.21.15.3:0/873059403 >> v2:172.21.15.143:3300/0 conn(0x7f092c144530 0x7f092c066f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0931e24700 1 -- 172.21.15.3:0/873059403 >> v1:172.21.15.143:6789/0 conn(0x7f092c0b3a30 legacy=0x7f092c0b76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0931e24700 1 -- 172.21.15.3:0/873059403 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c0b7dd0 con 0x7f092c13e5f0 2022-07-21T01:36:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0931e24700 1 --2- 172.21.15.3:0/873059403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c064a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f091c0032f0 tx=0x7f091c0091d0).ready entity=mon.2 client_cookie=ac353afe849606e6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0930e22700 1 -- 172.21.15.3:0/873059403 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f091c00b6c0 con 0x7f092c13e5f0 2022-07-21T01:36:53.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0930e22700 1 -- 172.21.15.3:0/873059403 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091c00b820 con 0x7f092c13e5f0 2022-07-21T01:36:53.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0930e22700 1 -- 172.21.15.3:0/873059403 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f091c00b990 con 0x7f092c13e5f0 2022-07-21T01:36:53.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 -- 172.21.15.3:0/873059403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 msgr2=0x7f092c064a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 --2- 172.21.15.3:0/873059403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c064a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f091c0032f0 tx=0x7f091c0091d0).stop 2022-07-21T01:36:53.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 -- 172.21.15.3:0/873059403 shutdown_connections 2022-07-21T01:36:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 --2- 172.21.15.3:0/873059403 >> v2:172.21.15.143:3300/0 conn(0x7f092c144530 0x7f092c066f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 --2- 172.21.15.3:0/873059403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c064a00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 -- 172.21.15.3:0/873059403 >> 172.21.15.3:0/873059403 conn(0x7f092c062b60 msgr2=0x7f092c062f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:53.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 -- 172.21.15.3:0/873059403 shutdown_connections 2022-07-21T01:36:53.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.349+0000 7f0934088700 1 -- 172.21.15.3:0/873059403 wait complete. 2022-07-21T01:36:53.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0934088700 1 Processor -- start 2022-07-21T01:36:53.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0934088700 1 -- start start 2022-07-21T01:36:53.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0934088700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c0b3a30 0x7f092c0b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0934088700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0934088700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c144530 0x7f092c1530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0934088700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c0b19e0 con 0x7f092c144530 2022-07-21T01:36:53.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0934088700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c0b1790 con 0x7f092c0b3a30 2022-07-21T01:36:53.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0934088700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092c0b14a0 con 0x7f092c13e5f0 2022-07-21T01:36:53.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0932625700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c144530 0x7f092c1530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931e24700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c0b3a30 0x7f092c0b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931623700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931e24700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c0b3a30 0x7f092c0b0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:50922/0 (socket says 172.21.15.3:50922) 2022-07-21T01:36:53.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931e24700 1 -- 172.21.15.3:0/2225704242 learned_addr learned my addr 172.21.15.3:0/2225704242 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:53.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0932625700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c144530 0x7f092c1530a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45478/0 (socket says 172.21.15.3:45478) 2022-07-21T01:36:53.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931e24700 1 -- 172.21.15.3:0/2225704242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 msgr2=0x7f092c152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931e24700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931e24700 1 -- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c144530 msgr2=0x7f092c1530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931e24700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c144530 0x7f092c1530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931e24700 1 -- 172.21.15.3:0/2225704242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c009470 con 0x7f092c0b3a30 2022-07-21T01:36:53.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931623700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:53.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.353+0000 7f0931e24700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c0b3a30 0x7f092c0b0df0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f091c0032f0 tx=0x7f091c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:53.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f092affd700 1 -- 172.21.15.3:0/2225704242 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f091c00b6c0 con 0x7f092c0b3a30 2022-07-21T01:36:53.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f092affd700 1 -- 172.21.15.3:0/2225704242 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091c021630 con 0x7f092c0b3a30 2022-07-21T01:36:53.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c140730 con 0x7f092c0b3a30 2022-07-21T01:36:53.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f092affd700 1 -- 172.21.15.3:0/2225704242 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f091c021990 con 0x7f092c0b3a30 2022-07-21T01:36:53.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092c062490 con 0x7f092c0b3a30 2022-07-21T01:36:53.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092c0af660 con 0x7f092c0b3a30 2022-07-21T01:36:53.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f092affd700 1 -- 172.21.15.3:0/2225704242 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f091c029070 con 0x7f092c0b3a30 2022-07-21T01:36:53.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f092affd700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f090c053950 0x7f090c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f092affd700 1 -- 172.21.15.3:0/2225704242 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f091c06d120 con 0x7f092c0b3a30 2022-07-21T01:36:53.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f0931623700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f090c053950 0x7f090c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.357+0000 7f0931623700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f090c053950 0x7f090c055e10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f092000d000 tx=0x7f0920009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:53.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.369+0000 7f092affd700 1 -- 172.21.15.3:0/2225704242 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f091c043e40 con 0x7f092c0b3a30 2022-07-21T01:36:53.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.537+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f092c1404d0 con 0x7f092c0b3a30 2022-07-21T01:36:53.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.685+0000 7f092affd700 1 -- 172.21.15.3:0/2225704242 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7f091c047460 con 0x7f092c0b3a30 2022-07-21T01:36:53.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f090c053950 msgr2=0x7f090c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f090c053950 0x7f090c055e10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f092000d000 tx=0x7f0920009040).stop 2022-07-21T01:36:53.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c0b3a30 msgr2=0x7f092c0b0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c0b3a30 0x7f092c0b0df0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f091c0032f0 tx=0x7f091c016000).stop 2022-07-21T01:36:53.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 shutdown_connections 2022-07-21T01:36:53.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c144530 0x7f092c1530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f092c13e5f0 0x7f092c152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f090c053950 0x7f090c055e10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 --2- 172.21.15.3:0/2225704242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c0b3a30 0x7f092c0b0df0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 >> 172.21.15.3:0/2225704242 conn(0x7f092c062b60 msgr2=0x7f092c142640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:53.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 shutdown_connections 2022-07-21T01:36:53.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.689+0000 7f0934088700 1 -- 172.21.15.3:0/2225704242 wait complete. 2022-07-21T01:36:53.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:36:53.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7048516700 1 Processor -- start 2022-07-21T01:36:53.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7048516700 1 -- start start 2022-07-21T01:36:53.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7048516700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7040060290 0x7f7040066610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7048516700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7040066b00 0x7f704014b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7048516700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7040060670 con 0x7f70400b38d0 2022-07-21T01:36:53.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7048516700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7040065c30 con 0x7f7040066b00 2022-07-21T01:36:53.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7048516700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7040065ed0 con 0x7f7040060290 2022-07-21T01:36:53.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7046ab3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7040066b00 0x7f704014b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7046ab3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7040066b00 0x7f704014b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45486/0 (socket says 172.21.15.3:45486) 2022-07-21T01:36:53.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7046ab3700 1 -- 172.21.15.3:0/489141922 learned_addr learned my addr 172.21.15.3:0/489141922 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:53.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f70462b2700 1 --2- 172.21.15.3:0/489141922 >> v2:172.21.15.143:3300/0 conn(0x7f7040060290 0x7f7040066610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7046ab3700 1 -- 172.21.15.3:0/489141922 >> v2:172.21.15.143:3300/0 conn(0x7f7040060290 msgr2=0x7f7040066610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:53.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7046ab3700 1 --2- 172.21.15.3:0/489141922 >> v2:172.21.15.143:3300/0 conn(0x7f7040060290 0x7f7040066610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7046ab3700 1 -- 172.21.15.3:0/489141922 >> v1:172.21.15.143:6789/0 conn(0x7f70400b38d0 legacy=0x7f704014ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7046ab3700 1 -- 172.21.15.3:0/489141922 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040066170 con 0x7f7040066b00 2022-07-21T01:36:53.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f7046ab3700 1 --2- 172.21.15.3:0/489141922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 0x7f704014b530 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f70380032f0 tx=0x7f7038009290).ready entity=mon.0 client_cookie=695c68523e19d205 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:53.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f70452b0700 1 -- 172.21.15.3:0/489141922 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f703800b870 con 0x7f7040066b00 2022-07-21T01:36:53.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f70452b0700 1 -- 172.21.15.3:0/489141922 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703800b9d0 con 0x7f7040066b00 2022-07-21T01:36:53.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.817+0000 7f70452b0700 1 -- 172.21.15.3:0/489141922 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f703800bb40 con 0x7f7040066b00 2022-07-21T01:36:53.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 -- 172.21.15.3:0/489141922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 msgr2=0x7f704014b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 --2- 172.21.15.3:0/489141922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 0x7f704014b530 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f70380032f0 tx=0x7f7038009290).stop 2022-07-21T01:36:53.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 -- 172.21.15.3:0/489141922 shutdown_connections 2022-07-21T01:36:53.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 --2- 172.21.15.3:0/489141922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 0x7f704014b530 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 --2- 172.21.15.3:0/489141922 >> v2:172.21.15.143:3300/0 conn(0x7f7040060290 0x7f7040066610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 -- 172.21.15.3:0/489141922 >> 172.21.15.3:0/489141922 conn(0x7f704005d0c0 msgr2=0x7f704005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:53.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 -- 172.21.15.3:0/489141922 shutdown_connections 2022-07-21T01:36:53.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 -- 172.21.15.3:0/489141922 wait complete. 2022-07-21T01:36:53.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 Processor -- start 2022-07-21T01:36:53.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 -- start start 2022-07-21T01:36:53.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7040060290 0x7f70400a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 0x7f7040156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70400b38d0 0x7f7040156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70400a0020 con 0x7f7040066b00 2022-07-21T01:36:53.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70400a0250 con 0x7f70400b38d0 2022-07-21T01:36:53.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7048516700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70400a1380 con 0x7f7040060290 2022-07-21T01:36:53.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7045ab1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 0x7f7040156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f70462b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7040060290 0x7f70400a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7046ab3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70400b38d0 0x7f7040156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7045ab1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 0x7f7040156890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45490/0 (socket says 172.21.15.3:45490) 2022-07-21T01:36:53.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.821+0000 7f7045ab1700 1 -- 172.21.15.3:0/2133381779 learned_addr learned my addr 172.21.15.3:0/2133381779 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:53.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f70462b2700 1 -- 172.21.15.3:0/2133381779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70400b38d0 msgr2=0x7f7040156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f70462b2700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70400b38d0 0x7f7040156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f70462b2700 1 -- 172.21.15.3:0/2133381779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 msgr2=0x7f7040156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:53.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f70462b2700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 0x7f7040156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:53.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f70462b2700 1 -- 172.21.15.3:0/2133381779 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038009530 con 0x7f7040060290 2022-07-21T01:36:53.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f70462b2700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7040060290 0x7f70400a2bf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f703000c000 tx=0x7f7030006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:53.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f702f7fe700 1 -- 172.21.15.3:0/2133381779 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7030010780 con 0x7f7040060290 2022-07-21T01:36:53.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f702f7fe700 1 -- 172.21.15.3:0/2133381779 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70300108e0 con 0x7f7040060290 2022-07-21T01:36:53.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f702f7fe700 1 -- 172.21.15.3:0/2133381779 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7030010c40 con 0x7f7040060290 2022-07-21T01:36:53.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704009eb60 con 0x7f7040060290 2022-07-21T01:36:53.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70400adf20 con 0x7f7040060290 2022-07-21T01:36:53.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70400574e0 con 0x7f7040060290 2022-07-21T01:36:53.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f702f7fe700 1 -- 172.21.15.3:0/2133381779 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f703001b410 con 0x7f7040060290 2022-07-21T01:36:53.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f702f7fe700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7020053b00 0x7f7020055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:53.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f702f7fe700 1 -- 172.21.15.3:0/2133381779 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f703006c0d0 con 0x7f7040060290 2022-07-21T01:36:53.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.825+0000 7f7045ab1700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7020053b00 0x7f7020055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:53.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.829+0000 7f7045ab1700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7020053b00 0x7f7020055fc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f703c00d000 tx=0x7f703c007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:53.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:53.837+0000 7f702f7fe700 1 -- 172.21.15.3:0/2133381779 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7030042e30 con 0x7f7040060290 2022-07-21T01:36:54.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.001+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7040069a70 con 0x7f7040060290 2022-07-21T01:36:54.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.001+0000 7f702f7fe700 1 -- 172.21.15.3:0/2133381779 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f7030046450 con 0x7f7040060290 2022-07-21T01:36:54.002 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:54.002 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:36:52.675976+0000","modified":"2022-07-21T01:36:53.679085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/3381448192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3381448192},{"type":"v1","addr":"172.21.15.3:6837","nonce":3381448192}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-21T01:36:54.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7020053b00 msgr2=0x7f7020055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7020053b00 0x7f7020055fc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f703c00d000 tx=0x7f703c007b70).stop 2022-07-21T01:36:54.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7040060290 msgr2=0x7f70400a2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7040060290 0x7f70400a2bf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f703000c000 tx=0x7f7030006df0).stop 2022-07-21T01:36:54.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 shutdown_connections 2022-07-21T01:36:54.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7020053b00 0x7f7020055fc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70400b38d0 0x7f7040156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7040066b00 0x7f7040156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 --2- 172.21.15.3:0/2133381779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7040060290 0x7f70400a2bf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 >> 172.21.15.3:0/2133381779 conn(0x7f704005d0c0 msgr2=0x7f70400b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:54.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 shutdown_connections 2022-07-21T01:36:54.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.005+0000 7f7048516700 1 -- 172.21.15.3:0/2133381779 wait complete. 2022-07-21T01:36:54.008 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2022-07-21T01:36:54.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:36:54.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 Processor -- start 2022-07-21T01:36:54.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 -- start start 2022-07-21T01:36:54.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5c940b2710 0x7f5c9414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5c940b01a0 0x7f5c9414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940afd60 0x7f5c940a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c940a7250 con 0x7f5c940b2710 2022-07-21T01:36:54.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c940a7480 con 0x7f5c940b01a0 2022-07-21T01:36:54.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940a7720 con 0x7f5c940afd60 2022-07-21T01:36:54.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c98c0c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5c940b2710 0x7f5c9414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c98c0c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5c940b2710 0x7f5c9414afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45496/0 (socket says 172.21.15.3:45496) 2022-07-21T01:36:54.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c98c0c700 1 -- 172.21.15.3:0/3769764513 learned_addr learned my addr 172.21.15.3:0/3769764513 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:54.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c93fff700 1 --2- 172.21.15.3:0/3769764513 >> v2:172.21.15.143:3300/0 conn(0x7f5c940b01a0 0x7f5c9414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c9940d700 1 --2- 172.21.15.3:0/3769764513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940afd60 0x7f5c940a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c98c0c700 1 -- 172.21.15.3:0/3769764513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940afd60 msgr2=0x7f5c940a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:54.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c98c0c700 1 --2- 172.21.15.3:0/3769764513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940afd60 0x7f5c940a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c98c0c700 1 -- 172.21.15.3:0/3769764513 >> v2:172.21.15.143:3300/0 conn(0x7f5c940b01a0 msgr2=0x7f5c9414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:54.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c98c0c700 1 --2- 172.21.15.3:0/3769764513 >> v2:172.21.15.143:3300/0 conn(0x7f5c940b01a0 0x7f5c9414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c98c0c700 1 -- 172.21.15.3:0/3769764513 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c940a79c0 con 0x7f5c940b2710 2022-07-21T01:36:54.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c93fff700 1 --2- 172.21.15.3:0/3769764513 >> v2:172.21.15.143:3300/0 conn(0x7f5c940b01a0 0x7f5c9414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:54.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c98c0c700 1 --2- 172.21.15.3:0/3769764513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b2710 0x7f5c9414afd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f5c7c002f70 tx=0x7f5c7c006eb0).ready entity=mon.0 client_cookie=957d8a9bf43a49fa server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:54.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c937fe700 1 -- 172.21.15.3:0/3769764513 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c7c00d570 con 0x7f5c940b2710 2022-07-21T01:36:54.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c937fe700 1 -- 172.21.15.3:0/3769764513 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7c000e90 con 0x7f5c940b2710 2022-07-21T01:36:54.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c937fe700 1 -- 172.21.15.3:0/3769764513 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c7c00d900 con 0x7f5c940b2710 2022-07-21T01:36:54.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 -- 172.21.15.3:0/3769764513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b2710 msgr2=0x7f5c9414afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/3769764513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b2710 0x7f5c9414afd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f5c7c002f70 tx=0x7f5c7c006eb0).stop 2022-07-21T01:36:54.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 -- 172.21.15.3:0/3769764513 shutdown_connections 2022-07-21T01:36:54.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/3769764513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940afd60 0x7f5c940a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/3769764513 >> v2:172.21.15.143:3300/0 conn(0x7f5c940b01a0 0x7f5c9414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/3769764513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b2710 0x7f5c9414afd0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 -- 172.21.15.3:0/3769764513 >> 172.21.15.3:0/3769764513 conn(0x7f5c94060310 msgr2=0x7f5c940b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:54.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 -- 172.21.15.3:0/3769764513 shutdown_connections 2022-07-21T01:36:54.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.125+0000 7f5c99c0e700 1 -- 172.21.15.3:0/3769764513 wait complete. 2022-07-21T01:36:54.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 Processor -- start 2022-07-21T01:36:54.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 -- start start 2022-07-21T01:36:54.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c940afd60 0x7f5c940aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b01a0 0x7f5c940aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940b2710 0x7f5c940aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c94144710 con 0x7f5c940b01a0 2022-07-21T01:36:54.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c941449b0 con 0x7f5c940afd60 2022-07-21T01:36:54.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940a92d0 con 0x7f5c940b2710 2022-07-21T01:36:54.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c93fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b01a0 0x7f5c940aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c93fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b01a0 0x7f5c940aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45504/0 (socket says 172.21.15.3:45504) 2022-07-21T01:36:54.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c93fff700 1 -- 172.21.15.3:0/2433333017 learned_addr learned my addr 172.21.15.3:0/2433333017 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:54.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c98c0c700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c940afd60 0x7f5c940aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c9940d700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940b2710 0x7f5c940aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c93fff700 1 -- 172.21.15.3:0/2433333017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940b2710 msgr2=0x7f5c940aadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c93fff700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940b2710 0x7f5c940aadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c93fff700 1 -- 172.21.15.3:0/2433333017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c940afd60 msgr2=0x7f5c940aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c93fff700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c940afd60 0x7f5c940aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c93fff700 1 -- 172.21.15.3:0/2433333017 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c00b1e0 con 0x7f5c940b01a0 2022-07-21T01:36:54.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c9940d700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940b2710 0x7f5c940aadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:54.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c93fff700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b01a0 0x7f5c940aa8c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f5c8400a000 tx=0x7f5c84008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:54.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c91ffb700 1 -- 172.21.15.3:0/2433333017 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c84011780 con 0x7f5c940b01a0 2022-07-21T01:36:54.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c91ffb700 1 -- 172.21.15.3:0/2433333017 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c840118e0 con 0x7f5c940b01a0 2022-07-21T01:36:54.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c940a9570 con 0x7f5c940b01a0 2022-07-21T01:36:54.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c91ffb700 1 -- 172.21.15.3:0/2433333017 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c84011c40 con 0x7f5c940b01a0 2022-07-21T01:36:54.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.129+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c940aa1c0 con 0x7f5c940b01a0 2022-07-21T01:36:54.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.133+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c940aa4a0 con 0x7f5c940b01a0 2022-07-21T01:36:54.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.133+0000 7f5c91ffb700 1 -- 172.21.15.3:0/2433333017 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5c8401c460 con 0x7f5c940b01a0 2022-07-21T01:36:54.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.133+0000 7f5c91ffb700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c6c0539e0 0x7f5c6c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.133+0000 7f5c91ffb700 1 -- 172.21.15.3:0/2433333017 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5c8406c560 con 0x7f5c940b01a0 2022-07-21T01:36:54.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.133+0000 7f5c98c0c700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c6c0539e0 0x7f5c6c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.133+0000 7f5c98c0c700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c6c0539e0 0x7f5c6c055ea0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5c7c01a000 tx=0x7f5c7c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:54.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.141+0000 7f5c91ffb700 1 -- 172.21.15.3:0/2433333017 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5c840432c0 con 0x7f5c940b01a0 2022-07-21T01:36:54.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.269+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c940b6190 con 0x7f5c940b01a0 2022-07-21T01:36:54.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.269+0000 7f5c91ffb700 1 -- 172.21.15.3:0/2433333017 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16273 (secure 0 0 0) 0x7f5c840468e0 con 0x7f5c940b01a0 2022-07-21T01:36:54.270 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:54.270 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":58,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:36:53.679840+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:36:51.181015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-21T01:36:52.366227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:36:54.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c6c0539e0 msgr2=0x7f5c6c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c6c0539e0 0x7f5c6c055ea0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5c7c01a000 tx=0x7f5c7c015000).stop 2022-07-21T01:36:54.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b01a0 msgr2=0x7f5c940aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b01a0 0x7f5c940aa8c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f5c8400a000 tx=0x7f5c84008040).stop 2022-07-21T01:36:54.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 shutdown_connections 2022-07-21T01:36:54.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c940b2710 0x7f5c940aadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c940b01a0 0x7f5c940aa8c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5c6c0539e0 0x7f5c6c055ea0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 --2- 172.21.15.3:0/2433333017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c940afd60 0x7f5c940aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 >> 172.21.15.3:0/2433333017 conn(0x7f5c94060310 msgr2=0x7f5c94148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:54.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 shutdown_connections 2022-07-21T01:36:54.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.273+0000 7f5c99c0e700 1 -- 172.21.15.3:0/2433333017 wait complete. 2022-07-21T01:36:54.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:36:54.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 Processor -- start 2022-07-21T01:36:54.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 -- start start 2022-07-21T01:36:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc17c0b2790 0x7fc17c05c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc17c0b1fb0 0x7fc17c143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17c1471b0 con 0x7fc17c143d60 2022-07-21T01:36:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c0ad5d0 con 0x7fc17c0b1fb0 2022-07-21T01:36:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c144140 con 0x7fc17c0b2790 2022-07-21T01:36:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc17c0b1fb0 0x7fc17c143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc17c0b1fb0 0x7fc17c143870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45512/0 (socket says 172.21.15.3:45512) 2022-07-21T01:36:54.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17b7fe700 1 -- 172.21.15.3:0/64480344 learned_addr learned my addr 172.21.15.3:0/64480344 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:54.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17bfff700 1 --2- 172.21.15.3:0/64480344 >> v2:172.21.15.143:3300/0 conn(0x7fc17c0b2790 0x7fc17c05c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17b7fe700 1 -- 172.21.15.3:0/64480344 >> v2:172.21.15.143:3300/0 conn(0x7fc17c0b2790 msgr2=0x7fc17c05c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:54.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17b7fe700 1 --2- 172.21.15.3:0/64480344 >> v2:172.21.15.143:3300/0 conn(0x7fc17c0b2790 0x7fc17c05c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17b7fe700 1 -- 172.21.15.3:0/64480344 >> v1:172.21.15.143:6789/0 conn(0x7fc17c143d60 legacy=0x7fc17c13cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17b7fe700 1 -- 172.21.15.3:0/64480344 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c147450 con 0x7fc17c0b1fb0 2022-07-21T01:36:54.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17bfff700 1 --2- 172.21.15.3:0/64480344 >> v2:172.21.15.143:3300/0 conn(0x7fc17c0b2790 0x7fc17c05c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:54.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17b7fe700 1 --2- 172.21.15.3:0/64480344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c0b1fb0 0x7fc17c143870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fc170002f70 tx=0x7fc170006eb0).ready entity=mon.0 client_cookie=dedf68d6bf3dc840 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:54.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17affd700 1 -- 172.21.15.3:0/64480344 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc17000d610 con 0x7fc17c0b1fb0 2022-07-21T01:36:54.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17affd700 1 -- 172.21.15.3:0/64480344 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc170000e90 con 0x7fc17c0b1fb0 2022-07-21T01:36:54.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc17affd700 1 -- 172.21.15.3:0/64480344 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc17000d9a0 con 0x7fc17c0b1fb0 2022-07-21T01:36:54.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 -- 172.21.15.3:0/64480344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c0b1fb0 msgr2=0x7fc17c143870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 --2- 172.21.15.3:0/64480344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c0b1fb0 0x7fc17c143870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fc170002f70 tx=0x7fc170006eb0).stop 2022-07-21T01:36:54.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 -- 172.21.15.3:0/64480344 shutdown_connections 2022-07-21T01:36:54.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 --2- 172.21.15.3:0/64480344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c0b1fb0 0x7fc17c143870 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 --2- 172.21.15.3:0/64480344 >> v2:172.21.15.143:3300/0 conn(0x7fc17c0b2790 0x7fc17c05c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 -- 172.21.15.3:0/64480344 >> 172.21.15.3:0/64480344 conn(0x7fc17c061360 msgr2=0x7fc17c062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:54.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 -- 172.21.15.3:0/64480344 shutdown_connections 2022-07-21T01:36:54.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.405+0000 7fc18103e700 1 -- 172.21.15.3:0/64480344 wait complete. 2022-07-21T01:36:54.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 Processor -- start 2022-07-21T01:36:54.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 -- start start 2022-07-21T01:36:54.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17c0b1fb0 0x7fc17c152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc17c0b2790 0x7fc17c152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c143d60 0x7fc17c13fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c0aecf0 con 0x7fc17c143d60 2022-07-21T01:36:54.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c145cc0 con 0x7fc17c0b1fb0 2022-07-21T01:36:54.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c0aeee0 con 0x7fc17c0b2790 2022-07-21T01:36:54.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18083d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c143d60 0x7fc17c13fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc17b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc17c0b2790 0x7fc17c152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc17bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17c0b1fb0 0x7fc17c152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18083d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c143d60 0x7fc17c13fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45520/0 (socket says 172.21.15.3:45520) 2022-07-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18083d700 1 -- 172.21.15.3:0/3419128846 learned_addr learned my addr 172.21.15.3:0/3419128846 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18083d700 1 -- 172.21.15.3:0/3419128846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc17c0b2790 msgr2=0x7fc17c152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18083d700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc17c0b2790 0x7fc17c152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18083d700 1 -- 172.21.15.3:0/3419128846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17c0b1fb0 msgr2=0x7fc17c152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18083d700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17c0b1fb0 0x7fc17c152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18083d700 1 -- 172.21.15.3:0/3419128846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17000b1e0 con 0x7fc17c143d60 2022-07-21T01:36:54.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18083d700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c143d60 0x7fc17c13fee0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fc17c0ae200 tx=0x7fc1740013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:54.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc1797fa700 1 -- 172.21.15.3:0/3419128846 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc174006aa0 con 0x7fc17c143d60 2022-07-21T01:36:54.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc1797fa700 1 -- 172.21.15.3:0/3419128846 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc174006c00 con 0x7fc17c143d60 2022-07-21T01:36:54.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c1566f0 con 0x7fc17c143d60 2022-07-21T01:36:54.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc1797fa700 1 -- 172.21.15.3:0/3419128846 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc174007030 con 0x7fc17c143d60 2022-07-21T01:36:54.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.409+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c0b0050 con 0x7fc17c143d60 2022-07-21T01:36:54.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.413+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c143130 con 0x7fc17c143d60 2022-07-21T01:36:54.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.413+0000 7fc1797fa700 1 -- 172.21.15.3:0/3419128846 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc174010030 con 0x7fc17c143d60 2022-07-21T01:36:54.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.413+0000 7fc1797fa700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc15c053980 0x7fc15c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.413+0000 7fc1797fa700 1 -- 172.21.15.3:0/3419128846 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc17406bf30 con 0x7fc17c143d60 2022-07-21T01:36:54.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.413+0000 7fc17bfff700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc15c053980 0x7fc15c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.413+0000 7fc17bfff700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc15c053980 0x7fc15c055e40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fc16c00c000 tx=0x7fc16c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:54.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.425+0000 7fc1797fa700 1 -- 172.21.15.3:0/3419128846 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc174042b80 con 0x7fc17c143d60 2022-07-21T01:36:54.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.589+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc17c0574e0 con 0x7fc17c143d60 2022-07-21T01:36:54.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.597+0000 7fc1797fa700 1 -- 172.21.15.3:0/3419128846 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v19) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc1740461a0 con 0x7fc17c143d60 2022-07-21T01:36:54.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc15c053980 msgr2=0x7fc15c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc15c053980 0x7fc15c055e40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fc16c00c000 tx=0x7fc16c008040).stop 2022-07-21T01:36:54.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c143d60 msgr2=0x7fc17c13fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c143d60 0x7fc17c13fee0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fc17c0ae200 tx=0x7fc1740013a0).stop 2022-07-21T01:36:54.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 shutdown_connections 2022-07-21T01:36:54.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc17c143d60 0x7fc17c13fee0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc17c0b2790 0x7fc17c152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc15c053980 0x7fc15c055e40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 --2- 172.21.15.3:0/3419128846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17c0b1fb0 0x7fc17c152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 >> 172.21.15.3:0/3419128846 conn(0x7fc17c061360 msgr2=0x7fc17c05c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:54.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 shutdown_connections 2022-07-21T01:36:54.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.601+0000 7fc18103e700 1 -- 172.21.15.3:0/3419128846 wait complete. 2022-07-21T01:36:54.608 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:36:54.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:36:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.733+0000 7f35c08c5700 1 Processor -- start 2022-07-21T01:36:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.733+0000 7f35c08c5700 1 -- start start 2022-07-21T01:36:54.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b4550 0x7f35bc0b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f35bc0b6af0 0x7f35bc12bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc144240 con 0x7f35bc0b6710 2022-07-21T01:36:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc143d00 con 0x7f35bc0b6af0 2022-07-21T01:36:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc143fa0 con 0x7f35bc0b4550 2022-07-21T01:36:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35bad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f35bc0b6af0 0x7f35bc12bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35ba59c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f35bc0b6710 0x7f35bc061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53804/0 (socket says 172.21.15.3:53804) 2022-07-21T01:36:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35ba59c700 1 -- 172.21.15.3:0/2567309579 learned_addr learned my addr 172.21.15.3:0/2567309579 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:54.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35b1d9b700 1 --2- 172.21.15.3:0/2567309579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b4550 0x7f35bc0b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35bad9d700 1 -- 172.21.15.3:0/2567309579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b4550 msgr2=0x7f35bc0b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:54.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35bad9d700 1 --2- 172.21.15.3:0/2567309579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b4550 0x7f35bc0b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35bad9d700 1 -- 172.21.15.3:0/2567309579 >> v1:172.21.15.3:6789/0 conn(0x7f35bc0b6710 legacy=0x7f35bc061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35bad9d700 1 -- 172.21.15.3:0/2567309579 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc061780 con 0x7f35bc0b6af0 2022-07-21T01:36:54.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35bad9d700 1 --2- 172.21.15.3:0/2567309579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b6af0 0x7f35bc12bb90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f35b40032f0 tx=0x7f35b40091d0).ready entity=mon.0 client_cookie=db387bde45ba1b2c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:54.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35b9d9b700 1 -- 172.21.15.3:0/2567309579 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35b400b6c0 con 0x7f35bc0b6af0 2022-07-21T01:36:54.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35b9d9b700 1 -- 172.21.15.3:0/2567309579 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b400b820 con 0x7f35bc0b6af0 2022-07-21T01:36:54.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35b9d9b700 1 -- 172.21.15.3:0/2567309579 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35b400b990 con 0x7f35bc0b6af0 2022-07-21T01:36:54.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 -- 172.21.15.3:0/2567309579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b6af0 msgr2=0x7f35bc12bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 --2- 172.21.15.3:0/2567309579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b6af0 0x7f35bc12bb90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f35b40032f0 tx=0x7f35b40091d0).stop 2022-07-21T01:36:54.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 -- 172.21.15.3:0/2567309579 shutdown_connections 2022-07-21T01:36:54.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 --2- 172.21.15.3:0/2567309579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b6af0 0x7f35bc12bb90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 --2- 172.21.15.3:0/2567309579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b4550 0x7f35bc0b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 -- 172.21.15.3:0/2567309579 >> 172.21.15.3:0/2567309579 conn(0x7f35bc05c620 msgr2=0x7f35bc05ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:54.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 -- 172.21.15.3:0/2567309579 shutdown_connections 2022-07-21T01:36:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.737+0000 7f35c08c5700 1 -- 172.21.15.3:0/2567309579 wait complete. 2022-07-21T01:36:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35c08c5700 1 Processor -- start 2022-07-21T01:36:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35c08c5700 1 -- start start 2022-07-21T01:36:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35c08c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b4550 0x7f35bc12a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35c08c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc0b6710 0x7f35bc12aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35c08c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b6af0 0x7f35bc0afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35c08c5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc141240 con 0x7f35bc0b4550 2022-07-21T01:36:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35c08c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0ae210 con 0x7f35bc0b6710 2022-07-21T01:36:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35c08c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc0ae440 con 0x7f35bc0b6af0 2022-07-21T01:36:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35ba59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b4550 0x7f35bc12a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35ba59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b4550 0x7f35bc12a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45530/0 (socket says 172.21.15.3:45530) 2022-07-21T01:36:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35ba59c700 1 -- 172.21.15.3:0/125921760 learned_addr learned my addr 172.21.15.3:0/125921760 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:54.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35b1d9b700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc0b6710 0x7f35bc12aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35bad9d700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b6af0 0x7f35bc0afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35ba59c700 1 -- 172.21.15.3:0/125921760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b6af0 msgr2=0x7f35bc0afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35ba59c700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b6af0 0x7f35bc0afd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35ba59c700 1 -- 172.21.15.3:0/125921760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc0b6710 msgr2=0x7f35bc12aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35ba59c700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc0b6710 0x7f35bc12aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35ba59c700 1 -- 172.21.15.3:0/125921760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b4009470 con 0x7f35bc0b4550 2022-07-21T01:36:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.741+0000 7f35b1d9b700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc0b6710 0x7f35bc12aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35ba59c700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b4550 0x7f35bc12a580 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f35bc141380 tx=0x7f35ac007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:54.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35b3fff700 1 -- 172.21.15.3:0/125921760 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35ac0070d0 con 0x7f35bc0b4550 2022-07-21T01:36:54.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35b3fff700 1 -- 172.21.15.3:0/125921760 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac007230 con 0x7f35bc0b4550 2022-07-21T01:36:54.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc0b3b20 con 0x7f35bc0b4550 2022-07-21T01:36:54.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35b3fff700 1 -- 172.21.15.3:0/125921760 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35ac005610 con 0x7f35bc0b4550 2022-07-21T01:36:54.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc12b6e0 con 0x7f35bc0b4550 2022-07-21T01:36:54.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc061a20 con 0x7f35bc0b4550 2022-07-21T01:36:54.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35b3fff700 1 -- 172.21.15.3:0/125921760 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f35ac010070 con 0x7f35bc0b4550 2022-07-21T01:36:54.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35b3fff700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35a0053b00 0x7f35a0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:54.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35b3fff700 1 -- 172.21.15.3:0/125921760 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f35ac06b5e0 con 0x7f35bc0b4550 2022-07-21T01:36:54.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.745+0000 7f35b1d9b700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35a0053b00 0x7f35a0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:54.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.749+0000 7f35b1d9b700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35a0053b00 0x7f35a0055fc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f35a400b000 tx=0x7f35a4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:54.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.753+0000 7f35b3fff700 1 -- 172.21.15.3:0/125921760 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35ac0423a0 con 0x7f35bc0b4550 2022-07-21T01:36:54.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.917+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f35bc0574e0 con 0x7f35bc0b4550 2022-07-21T01:36:55.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35b3fff700 1 -- 172.21.15.3:0/125921760 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v20) v1 ==== 184+0+0 (secure 0 0 0) 0x7f35ac018070 con 0x7f35bc0b4550 2022-07-21T01:36:55.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35a0053b00 msgr2=0x7f35a0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35a0053b00 0x7f35a0055fc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f35a400b000 tx=0x7f35a4006df0).stop 2022-07-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b4550 msgr2=0x7f35bc12a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b4550 0x7f35bc12a580 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f35bc141380 tx=0x7f35ac007c00).stop 2022-07-21T01:36:55.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 shutdown_connections 2022-07-21T01:36:55.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35bc0b6af0 0x7f35bc0afd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35a0053b00 0x7f35a0055fc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc0b6710 0x7f35bc12aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 --2- 172.21.15.3:0/125921760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0b4550 0x7f35bc12a580 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 >> 172.21.15.3:0/125921760 conn(0x7f35bc05c620 msgr2=0x7f35bc1432f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:55.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 shutdown_connections 2022-07-21T01:36:55.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:54.925+0000 7f35c08c5700 1 -- 172.21.15.3:0/125921760 wait complete. 2022-07-21T01:36:55.004 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:36:55.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:36:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.113+0000 7f4aa279a700 1 Processor -- start 2022-07-21T01:36:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.113+0000 7f4aa279a700 1 -- start start 2022-07-21T01:36:55.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4a9c0614b0 0x7f4a9c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c1405d0 0x7f4a9c140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9c0aee80 con 0x7f4a9c0ad750 2022-07-21T01:36:55.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9c0b70b0 con 0x7f4a9c0614b0 2022-07-21T01:36:55.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0af0b0 con 0x7f4a9c1405d0 2022-07-21T01:36:55.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1798700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4a9c0614b0 0x7f4a9c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1798700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4a9c0614b0 0x7f4a9c061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45538/0 (socket says 172.21.15.3:45538) 2022-07-21T01:36:55.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1798700 1 -- 172.21.15.3:0/2601435549 learned_addr learned my addr 172.21.15.3:0/2601435549 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:55.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1f99700 1 --2- 172.21.15.3:0/2601435549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c1405d0 0x7f4a9c140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4a8bfff700 1 -- 172.21.15.3:0/2601435549 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187176700 0 0) 0x7f4a9c0aee80 con 0x7f4a9c0ad750 2022-07-21T01:36:55.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4a8bfff700 1 -- 172.21.15.3:0/2601435549 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a84003260 con 0x7f4a9c0ad750 2022-07-21T01:36:55.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1798700 1 -- 172.21.15.3:0/2601435549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c1405d0 msgr2=0x7f4a9c140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:55.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1798700 1 --2- 172.21.15.3:0/2601435549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c1405d0 0x7f4a9c140a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1798700 1 -- 172.21.15.3:0/2601435549 >> v1:172.21.15.143:6789/0 conn(0x7f4a9c0ad750 legacy=0x7f4a9c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1798700 1 -- 172.21.15.3:0/2601435549 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c0af350 con 0x7f4a9c0614b0 2022-07-21T01:36:55.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1f99700 1 --2- 172.21.15.3:0/2601435549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c1405d0 0x7f4a9c140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:55.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa1798700 1 --2- 172.21.15.3:0/2601435549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c0614b0 0x7f4a9c061890 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f4a9c0aed30 tx=0x7f4a8c006eb0).ready entity=mon.0 client_cookie=f71c3d36618064dd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:55.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4a8bfff700 1 -- 172.21.15.3:0/2601435549 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a8c00d5c0 con 0x7f4a9c0614b0 2022-07-21T01:36:55.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4a8bfff700 1 -- 172.21.15.3:0/2601435549 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8c000e90 con 0x7f4a9c0614b0 2022-07-21T01:36:55.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4a8bfff700 1 -- 172.21.15.3:0/2601435549 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a8c00d950 con 0x7f4a9c0614b0 2022-07-21T01:36:55.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 -- 172.21.15.3:0/2601435549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c0614b0 msgr2=0x7f4a9c061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 --2- 172.21.15.3:0/2601435549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c0614b0 0x7f4a9c061890 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f4a9c0aed30 tx=0x7f4a8c006eb0).stop 2022-07-21T01:36:55.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 -- 172.21.15.3:0/2601435549 shutdown_connections 2022-07-21T01:36:55.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 --2- 172.21.15.3:0/2601435549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c1405d0 0x7f4a9c140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 --2- 172.21.15.3:0/2601435549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c0614b0 0x7f4a9c061890 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 -- 172.21.15.3:0/2601435549 >> 172.21.15.3:0/2601435549 conn(0x7f4a9c066f70 msgr2=0x7f4a9c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:55.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 -- 172.21.15.3:0/2601435549 shutdown_connections 2022-07-21T01:36:55.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.117+0000 7f4aa279a700 1 -- 172.21.15.3:0/2601435549 wait complete. 2022-07-21T01:36:55.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa279a700 1 Processor -- start 2022-07-21T01:36:55.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa279a700 1 -- start start 2022-07-21T01:36:55.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa279a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c0614b0 0x7f4a9c156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa279a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a9c0ad750 0x7f4a9c05ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa279a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c1405d0 0x7f4a9c141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa279a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c155a10 con 0x7f4a9c1405d0 2022-07-21T01:36:55.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa279a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c155c40 con 0x7f4a9c0ad750 2022-07-21T01:36:55.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa279a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c155ee0 con 0x7f4a9c0614b0 2022-07-21T01:36:55.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1f99700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c1405d0 0x7f4a9c141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1f99700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c1405d0 0x7f4a9c141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45548/0 (socket says 172.21.15.3:45548) 2022-07-21T01:36:55.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa0f97700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a9c0ad750 0x7f4a9c05ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1f99700 1 -- 172.21.15.3:0/3946384143 learned_addr learned my addr 172.21.15.3:0/3946384143 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:55.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1798700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c0614b0 0x7f4a9c156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1f99700 1 -- 172.21.15.3:0/3946384143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c0614b0 msgr2=0x7f4a9c156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1f99700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c0614b0 0x7f4a9c156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1f99700 1 -- 172.21.15.3:0/3946384143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a9c0ad750 msgr2=0x7f4a9c05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1f99700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a9c0ad750 0x7f4a9c05ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1f99700 1 -- 172.21.15.3:0/3946384143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a94009470 con 0x7f4a9c1405d0 2022-07-21T01:36:55.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa1798700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c0614b0 0x7f4a9c156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:55.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.121+0000 7f4aa0f97700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a9c0ad750 0x7f4a9c05ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:55.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4aa1f99700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c1405d0 0x7f4a9c141090 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f4a9400c000 tx=0x7f4a940091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:55.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4a8a7fc700 1 -- 172.21.15.3:0/3946384143 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a9400ee60 con 0x7f4a9c1405d0 2022-07-21T01:36:55.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4a8a7fc700 1 -- 172.21.15.3:0/3946384143 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9400c850 con 0x7f4a9c1405d0 2022-07-21T01:36:55.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8c00b1e0 con 0x7f4a9c1405d0 2022-07-21T01:36:55.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4a8a7fc700 1 -- 172.21.15.3:0/3946384143 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a9400cbb0 con 0x7f4a9c1405d0 2022-07-21T01:36:55.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c0ad170 con 0x7f4a9c1405d0 2022-07-21T01:36:55.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c0af5f0 con 0x7f4a9c1405d0 2022-07-21T01:36:55.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4a8a7fc700 1 -- 172.21.15.3:0/3946384143 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4a94022070 con 0x7f4a9c1405d0 2022-07-21T01:36:55.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4a8a7fc700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4a7c0539e0 0x7f4a7c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4a8a7fc700 1 -- 172.21.15.3:0/3946384143 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4a9406c6f0 con 0x7f4a9c1405d0 2022-07-21T01:36:55.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.125+0000 7f4aa1798700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4a7c0539e0 0x7f4a7c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.129+0000 7f4aa1798700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4a7c0539e0 0x7f4a7c055ea0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f4a8c01c000 tx=0x7f4a8c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:55.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.133+0000 7f4a8a7fc700 1 -- 172.21.15.3:0/3946384143 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4a940434b0 con 0x7f4a9c1405d0 2022-07-21T01:36:55.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.297+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a9c0574e0 con 0x7f4a9c1405d0 2022-07-21T01:36:55.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.297+0000 7f4a8a7fc700 1 -- 172.21.15.3:0/3946384143 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f4a94016090 con 0x7f4a9c1405d0 2022-07-21T01:36:55.298 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:55.298 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:36:52.675976+0000","modified":"2022-07-21T01:36:53.679085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/3381448192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3381448192},{"type":"v1","addr":"172.21.15.3:6837","nonce":3381448192}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-21T01:36:55.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4a7c0539e0 msgr2=0x7f4a7c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4a7c0539e0 0x7f4a7c055ea0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f4a8c01c000 tx=0x7f4a8c015000).stop 2022-07-21T01:36:55.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c1405d0 msgr2=0x7f4a9c141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c1405d0 0x7f4a9c141090 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f4a9400c000 tx=0x7f4a940091d0).stop 2022-07-21T01:36:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 shutdown_connections 2022-07-21T01:36:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a9c1405d0 0x7f4a9c141090 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4a7c0539e0 0x7f4a7c055ea0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a9c0ad750 0x7f4a9c05ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 --2- 172.21.15.3:0/3946384143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a9c0614b0 0x7f4a9c156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 >> 172.21.15.3:0/3946384143 conn(0x7f4a9c066f70 msgr2=0x7f4a9c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:55.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 shutdown_connections 2022-07-21T01:36:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.301+0000 7f4aa279a700 1 -- 172.21.15.3:0/3946384143 wait complete. 2022-07-21T01:36:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2022-07-21T01:36:55.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:36:52.675976+0000', 'modified': '2022-07-21T01:36:53.679085+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5586}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5586': {'gid': 5586, 'name': 'a', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/3381448192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 3381448192}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 3381448192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:36:55.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:36:55.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:36:55.737 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":5586,"want_state":"up:active","state":"up:active","rank_uptime":3.050043171,"mdsmap_epoch":23,"osdmap_epoch":58,"osdmap_epoch_barrier":57,"uptime":7.3186153709999999} 2022-07-21T01:36:55.737 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 5586, "mdsmap_epoch": 23, "osdmap_epoch": 58, "osdmap_epoch_barrier": 57, "rank_uptime": 3.050043171, "state": "up:active", "uptime": 7.318615371, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:36:55.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:36:55.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0cc8d0700 1 Processor -- start 2022-07-21T01:36:55.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0cc8d0700 1 -- start start 2022-07-21T01:36:55.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0cc8d0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa0c80a9da0 0x7fa0c8060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0cc8d0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c8067030 con 0x7fa0c80b31d0 2022-07-21T01:36:55.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0cc8d0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c80b35b0 con 0x7fa0c81402d0 2022-07-21T01:36:55.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0cc8d0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c8064a70 con 0x7fa0c80a9da0 2022-07-21T01:36:55.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c77fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa0c80b31d0 0x7fa0c8066020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53832/0 (socket says 172.21.15.3:53832) 2022-07-21T01:36:55.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c77fe700 1 -- 172.21.15.3:0/1801657878 learned_addr learned my addr 172.21.15.3:0/1801657878 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:55.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c6ffd700 1 --2- 172.21.15.3:0/1801657878 >> v2:172.21.15.143:3300/0 conn(0x7fa0c80a9da0 0x7fa0c8060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c67fc700 1 -- 172.21.15.3:0/1801657878 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567149229 0 0) 0x7fa0c8067030 con 0x7fa0c80b31d0 2022-07-21T01:36:55.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c67fc700 1 -- 172.21.15.3:0/1801657878 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ac003260 con 0x7fa0c80b31d0 2022-07-21T01:36:55.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c67fc700 1 -- 172.21.15.3:0/1801657878 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714898434 0 0) 0x7fa0c80b35b0 con 0x7fa0c81402d0 2022-07-21T01:36:55.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c67fc700 1 -- 172.21.15.3:0/1801657878 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c8067030 con 0x7fa0c81402d0 2022-07-21T01:36:55.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c6ffd700 1 -- 172.21.15.3:0/1801657878 >> v1:172.21.15.143:6789/0 conn(0x7fa0c81402d0 legacy=0x7fa0c8140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c6ffd700 1 -- 172.21.15.3:0/1801657878 >> v1:172.21.15.3:6789/0 conn(0x7fa0c80b31d0 legacy=0x7fa0c8066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c6ffd700 1 -- 172.21.15.3:0/1801657878 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c8066af0 con 0x7fa0c80a9da0 2022-07-21T01:36:55.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c6ffd700 1 --2- 172.21.15.3:0/1801657878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c80a9da0 0x7fa0c8060950 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa0b80032f0 tx=0x7fa0b80091d0).ready entity=mon.1 client_cookie=55fc33f5697f0443 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:55.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c67fc700 1 -- 172.21.15.3:0/1801657878 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0b800b600 con 0x7fa0c80a9da0 2022-07-21T01:36:55.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c67fc700 1 -- 172.21.15.3:0/1801657878 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b800b760 con 0x7fa0c80a9da0 2022-07-21T01:36:55.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.841+0000 7fa0c67fc700 1 -- 172.21.15.3:0/1801657878 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0b800b8d0 con 0x7fa0c80a9da0 2022-07-21T01:36:55.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1801657878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c80a9da0 msgr2=0x7fa0c8060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 --2- 172.21.15.3:0/1801657878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c80a9da0 0x7fa0c8060950 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa0b80032f0 tx=0x7fa0b80091d0).stop 2022-07-21T01:36:55.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1801657878 shutdown_connections 2022-07-21T01:36:55.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 --2- 172.21.15.3:0/1801657878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c80a9da0 0x7fa0c8060950 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1801657878 >> 172.21.15.3:0/1801657878 conn(0x7fa0c8063870 msgr2=0x7fa0c8063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:55.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1801657878 shutdown_connections 2022-07-21T01:36:55.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1801657878 wait complete. 2022-07-21T01:36:55.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 Processor -- start 2022-07-21T01:36:55.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 -- start start 2022-07-21T01:36:55.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0c80a9da0 0x7fa0c80ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c80b31d0 0x7fa0c814ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0c81402d0 0x7fa0c8153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c8140de0 con 0x7fa0c80a9da0 2022-07-21T01:36:55.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c8141080 con 0x7fa0c80b31d0 2022-07-21T01:36:55.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0cc8d0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c8153500 con 0x7fa0c81402d0 2022-07-21T01:36:55.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0c80a9da0 0x7fa0c80ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0c80a9da0 0x7fa0c80ac8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45558/0 (socket says 172.21.15.3:45558) 2022-07-21T01:36:55.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c77fe700 1 -- 172.21.15.3:0/1793914747 learned_addr learned my addr 172.21.15.3:0/1793914747 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:55.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c6ffd700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c80b31d0 0x7fa0c814ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c7fff700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0c81402d0 0x7fa0c8153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c77fe700 1 -- 172.21.15.3:0/1793914747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0c81402d0 msgr2=0x7fa0c8153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c77fe700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0c81402d0 0x7fa0c8153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c77fe700 1 -- 172.21.15.3:0/1793914747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c80b31d0 msgr2=0x7fa0c814ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c77fe700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c80b31d0 0x7fa0c814ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c77fe700 1 -- 172.21.15.3:0/1793914747 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b8009470 con 0x7fa0c80a9da0 2022-07-21T01:36:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.845+0000 7fa0c77fe700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0c80a9da0 0x7fa0c80ac8a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fa0b00011d0 tx=0x7fa0b0007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:55.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0c4ff9700 1 -- 172.21.15.3:0/1793914747 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0b00070d0 con 0x7fa0c80a9da0 2022-07-21T01:36:55.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0c4ff9700 1 -- 172.21.15.3:0/1793914747 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b0007230 con 0x7fa0c80a9da0 2022-07-21T01:36:55.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0c4ff9700 1 -- 172.21.15.3:0/1793914747 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0b0005610 con 0x7fa0c80a9da0 2022-07-21T01:36:55.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c81537a0 con 0x7fa0c80a9da0 2022-07-21T01:36:55.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c8140a60 con 0x7fa0c80a9da0 2022-07-21T01:36:55.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c805f440 con 0x7fa0c80a9da0 2022-07-21T01:36:55.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0c4ff9700 1 -- 172.21.15.3:0/1793914747 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa0b0010070 con 0x7fa0c80a9da0 2022-07-21T01:36:55.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0c4ff9700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0a0053aa0 0x7fa0a0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:55.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0c4ff9700 1 -- 172.21.15.3:0/1793914747 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa0b006b6a0 con 0x7fa0c80a9da0 2022-07-21T01:36:55.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.849+0000 7fa0c6ffd700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0a0053aa0 0x7fa0a0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:55.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.853+0000 7fa0c6ffd700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0a0053aa0 0x7fa0a0055f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fa0b8019000 tx=0x7fa0b8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:55.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.857+0000 7fa0c4ff9700 1 -- 172.21.15.3:0/1793914747 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa0b0042460 con 0x7fa0c80a9da0 2022-07-21T01:36:55.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.985+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0c8153930 con 0x7fa0c80a9da0 2022-07-21T01:36:56.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.985+0000 7fa0c4ff9700 1 -- 172.21.15.3:0/1793914747 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16273 (secure 0 0 0) 0x7fa0b0045a80 con 0x7fa0c80a9da0 2022-07-21T01:36:56.027 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:56.028 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":58,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:36:53.679840+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:36:51.181015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-21T01:36:52.366227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:36:56.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0a0053aa0 msgr2=0x7fa0a0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0a0053aa0 0x7fa0a0055f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fa0b8019000 tx=0x7fa0b8016000).stop 2022-07-21T01:36:56.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0c80a9da0 msgr2=0x7fa0c80ac8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0c80a9da0 0x7fa0c80ac8a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fa0b00011d0 tx=0x7fa0b0007b50).stop 2022-07-21T01:36:56.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 shutdown_connections 2022-07-21T01:36:56.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0c81402d0 0x7fa0c8153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa0a0053aa0 0x7fa0a0055f60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c80b31d0 0x7fa0c814ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 --2- 172.21.15.3:0/1793914747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0c80a9da0 0x7fa0c80ac8a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 >> 172.21.15.3:0/1793914747 conn(0x7fa0c8063870 msgr2=0x7fa0c8060950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 shutdown_connections 2022-07-21T01:36:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:55.989+0000 7fa0cc8d0700 1 -- 172.21.15.3:0/1793914747 wait complete. 2022-07-21T01:36:56.034 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:36:51.181015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:36:52.366227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:36:56.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:36:56.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a359b7700 1 Processor -- start 2022-07-21T01:36:56.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a359b7700 1 -- start start 2022-07-21T01:36:56.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a359b7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3a3013c870 0x7f3a3013ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a359b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a300ad0b0 0x7f3a300ad520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a359b7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a300b2dc0 con 0x7f3a300b29e0 2022-07-21T01:36:56.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a359b7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a300b1340 con 0x7f3a3013c870 2022-07-21T01:36:56.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a359b7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a300b17f0 con 0x7f3a300ad0b0 2022-07-21T01:36:56.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a349b5700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3a300b29e0 0x7f3a300b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:55962/0 (socket says 172.21.15.3:55962) 2022-07-21T01:36:56.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a349b5700 1 -- 172.21.15.3:0/2112325254 learned_addr learned my addr 172.21.15.3:0/2112325254 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:56.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a351b6700 1 --2- 172.21.15.3:0/2112325254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a300ad0b0 0x7f3a300ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a2ffff700 1 --2- 172.21.15.3:0/2112325254 >> v2:172.21.15.143:3300/0 conn(0x7f3a3013c870 0x7f3a3013ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a2f7fe700 1 -- 172.21.15.3:0/2112325254 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868406165 0 0) 0x7f3a300b2dc0 con 0x7f3a300b29e0 2022-07-21T01:36:56.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a351b6700 1 -- 172.21.15.3:0/2112325254 >> v2:172.21.15.143:3300/0 conn(0x7f3a3013c870 msgr2=0x7f3a3013ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:56.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a351b6700 1 --2- 172.21.15.3:0/2112325254 >> v2:172.21.15.143:3300/0 conn(0x7f3a3013c870 0x7f3a3013ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a351b6700 1 -- 172.21.15.3:0/2112325254 >> v1:172.21.15.143:6789/0 conn(0x7f3a300b29e0 legacy=0x7f3a300b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a351b6700 1 -- 172.21.15.3:0/2112325254 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a300b1a90 con 0x7f3a300ad0b0 2022-07-21T01:36:56.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.149+0000 7f3a351b6700 1 --2- 172.21.15.3:0/2112325254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a300ad0b0 0x7f3a300ad520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3a280032f0 tx=0x7f3a280091d0).ready entity=mon.2 client_cookie=ecd4849ab54d2939 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a2f7fe700 1 -- 172.21.15.3:0/2112325254 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a2800b6c0 con 0x7f3a300ad0b0 2022-07-21T01:36:56.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a2f7fe700 1 -- 172.21.15.3:0/2112325254 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2800b820 con 0x7f3a300ad0b0 2022-07-21T01:36:56.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a2f7fe700 1 -- 172.21.15.3:0/2112325254 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a2800b990 con 0x7f3a300ad0b0 2022-07-21T01:36:56.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a359b7700 1 -- 172.21.15.3:0/2112325254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a300ad0b0 msgr2=0x7f3a300ad520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a359b7700 1 --2- 172.21.15.3:0/2112325254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a300ad0b0 0x7f3a300ad520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3a280032f0 tx=0x7f3a280091d0).stop 2022-07-21T01:36:56.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a359b7700 1 -- 172.21.15.3:0/2112325254 shutdown_connections 2022-07-21T01:36:56.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a359b7700 1 --2- 172.21.15.3:0/2112325254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a300ad0b0 0x7f3a300ad520 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a359b7700 1 --2- 172.21.15.3:0/2112325254 >> v2:172.21.15.143:3300/0 conn(0x7f3a3013c870 0x7f3a3013ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a359b7700 1 -- 172.21.15.3:0/2112325254 >> 172.21.15.3:0/2112325254 conn(0x7f3a300613c0 msgr2=0x7f3a30062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:56.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a359b7700 1 -- 172.21.15.3:0/2112325254 shutdown_connections 2022-07-21T01:36:56.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.153+0000 7f3a359b7700 1 -- 172.21.15.3:0/2112325254 wait complete. 2022-07-21T01:36:56.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 Processor -- start 2022-07-21T01:36:56.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 -- start start 2022-07-21T01:36:56.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a300ad0b0 0x7f3a30138040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a300b29e0 0x7f3a30066440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a3013c870 0x7f3a30066930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a30143b10 con 0x7f3a300b29e0 2022-07-21T01:36:56.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a30066e20 con 0x7f3a300ad0b0 2022-07-21T01:36:56.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a300b52e0 con 0x7f3a3013c870 2022-07-21T01:36:56.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a300b29e0 0x7f3a30066440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a300b29e0 0x7f3a30066440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45574/0 (socket says 172.21.15.3:45574) 2022-07-21T01:36:56.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2ffff700 1 -- 172.21.15.3:0/3999815927 learned_addr learned my addr 172.21.15.3:0/3999815927 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:56.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a349b5700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a300ad0b0 0x7f3a30138040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a351b6700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a3013c870 0x7f3a30066930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2ffff700 1 -- 172.21.15.3:0/3999815927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a3013c870 msgr2=0x7f3a30066930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2ffff700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a3013c870 0x7f3a30066930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2ffff700 1 -- 172.21.15.3:0/3999815927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a300ad0b0 msgr2=0x7f3a30138040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2ffff700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a300ad0b0 0x7f3a30138040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2ffff700 1 -- 172.21.15.3:0/3999815927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a28009470 con 0x7f3a300b29e0 2022-07-21T01:36:56.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a349b5700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a300ad0b0 0x7f3a30138040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:56.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2ffff700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a300b29e0 0x7f3a30066440 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f3a1c00c000 tx=0x7f3a1c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2dffb700 1 -- 172.21.15.3:0/3999815927 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a1c011750 con 0x7f3a300b29e0 2022-07-21T01:36:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2dffb700 1 -- 172.21.15.3:0/3999815927 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c0118b0 con 0x7f3a300b29e0 2022-07-21T01:36:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a2dffb700 1 -- 172.21.15.3:0/3999815927 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a1c011c10 con 0x7f3a300b29e0 2022-07-21T01:36:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a300b5580 con 0x7f3a300b29e0 2022-07-21T01:36:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.157+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a30064010 con 0x7f3a300b29e0 2022-07-21T01:36:56.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.161+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a300b5a20 con 0x7f3a300b29e0 2022-07-21T01:36:56.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.161+0000 7f3a2dffb700 1 -- 172.21.15.3:0/3999815927 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3a1c011d70 con 0x7f3a300b29e0 2022-07-21T01:36:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.161+0000 7f3a2dffb700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a100539e0 0x7f3a10055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.161+0000 7f3a2dffb700 1 -- 172.21.15.3:0/3999815927 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3a1c06c430 con 0x7f3a300b29e0 2022-07-21T01:36:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.161+0000 7f3a349b5700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a100539e0 0x7f3a10055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.161+0000 7f3a349b5700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a100539e0 0x7f3a10055ea0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3a2400d000 tx=0x7f3a24007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.169+0000 7f3a2dffb700 1 -- 172.21.15.3:0/3999815927 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3a1c043190 con 0x7f3a300b29e0 2022-07-21T01:36:56.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.293+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3a300574e0 con 0x7f3a300b29e0 2022-07-21T01:36:56.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a2dffb700 1 -- 172.21.15.3:0/3999815927 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v58) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3a1c0467b0 con 0x7f3a300b29e0 2022-07-21T01:36:56.296 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:36:56.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a100539e0 msgr2=0x7f3a10055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a100539e0 0x7f3a10055ea0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3a2400d000 tx=0x7f3a24007a20).stop 2022-07-21T01:36:56.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a300b29e0 msgr2=0x7f3a30066440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a300b29e0 0x7f3a30066440 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f3a1c00c000 tx=0x7f3a1c006df0).stop 2022-07-21T01:36:56.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 shutdown_connections 2022-07-21T01:36:56.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a3013c870 0x7f3a30066930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a300b29e0 0x7f3a30066440 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a100539e0 0x7f3a10055ea0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 --2- 172.21.15.3:0/3999815927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a300ad0b0 0x7f3a30138040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 >> 172.21.15.3:0/3999815927 conn(0x7f3a300613c0 msgr2=0x7f3a30139a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:56.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 shutdown_connections 2022-07-21T01:36:56.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.297+0000 7f3a359b7700 1 -- 172.21.15.3:0/3999815927 wait complete. 2022-07-21T01:36:56.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:36:56.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd69bf8d700 1 Processor -- start 2022-07-21T01:36:56.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd69bf8d700 1 -- start start 2022-07-21T01:36:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd69bf8d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd6940b2050 0x7fd6940b1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd69bf8d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd6940b0140 0x7fd6940b2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd69bf8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 0x7fd6941413e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd69bf8d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6940b05b0 con 0x7fd6940b2050 2022-07-21T01:36:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd69bf8d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6940b2920 con 0x7fd6940b0140 2022-07-21T01:36:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd69bf8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6940b07e0 con 0x7fd6940a7250 2022-07-21T01:36:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd699d29700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd6940b2050 0x7fd6940b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd699d29700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd6940b2050 0x7fd6940b1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45580/0 (socket says 172.21.15.3:45580) 2022-07-21T01:36:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd699d29700 1 -- 172.21.15.3:0/1756774751 learned_addr learned my addr 172.21.15.3:0/1756774751 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd69a52a700 1 --2- 172.21.15.3:0/1756774751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 0x7fd6941413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.421+0000 7fd699528700 1 --2- 172.21.15.3:0/1756774751 >> v2:172.21.15.143:3300/0 conn(0x7fd6940b0140 0x7fd6940b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699528700 1 -- 172.21.15.3:0/1756774751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 msgr2=0x7fd6941413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699528700 1 --2- 172.21.15.3:0/1756774751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 0x7fd6941413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699528700 1 -- 172.21.15.3:0/1756774751 >> v2:172.21.15.3:3300/0 conn(0x7fd6940b2050 msgr2=0x7fd6940b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:56.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699528700 1 --2- 172.21.15.3:0/1756774751 >> v2:172.21.15.3:3300/0 conn(0x7fd6940b2050 0x7fd6940b1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699528700 1 -- 172.21.15.3:0/1756774751 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940b0a80 con 0x7fd6940b0140 2022-07-21T01:36:56.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69a52a700 1 --2- 172.21.15.3:0/1756774751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 0x7fd6941413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:56.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699d29700 1 --2- 172.21.15.3:0/1756774751 >> v2:172.21.15.3:3300/0 conn(0x7fd6940b2050 0x7fd6940b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:36:56.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699528700 1 --2- 172.21.15.3:0/1756774751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6940b0140 0x7fd6940b2430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fd6900032f0 tx=0x7fd6900091d0).ready entity=mon.1 client_cookie=298d44f04d1d6b4b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd698d27700 1 -- 172.21.15.3:0/1756774751 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd69000b600 con 0x7fd6940b0140 2022-07-21T01:36:56.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd698d27700 1 -- 172.21.15.3:0/1756774751 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69000b760 con 0x7fd6940b0140 2022-07-21T01:36:56.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd698d27700 1 -- 172.21.15.3:0/1756774751 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd69000b8d0 con 0x7fd6940b0140 2022-07-21T01:36:56.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 -- 172.21.15.3:0/1756774751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6940b0140 msgr2=0x7fd6940b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/1756774751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6940b0140 0x7fd6940b2430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fd6900032f0 tx=0x7fd6900091d0).stop 2022-07-21T01:36:56.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 -- 172.21.15.3:0/1756774751 shutdown_connections 2022-07-21T01:36:56.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/1756774751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 0x7fd6941413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/1756774751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6940b0140 0x7fd6940b2430 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/1756774751 >> v2:172.21.15.3:3300/0 conn(0x7fd6940b2050 0x7fd6940b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 -- 172.21.15.3:0/1756774751 >> 172.21.15.3:0/1756774751 conn(0x7fd69405f840 msgr2=0x7fd69405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 -- 172.21.15.3:0/1756774751 shutdown_connections 2022-07-21T01:36:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 -- 172.21.15.3:0/1756774751 wait complete. 2022-07-21T01:36:56.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 Processor -- start 2022-07-21T01:36:56.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 -- start start 2022-07-21T01:36:56.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 0x7fd694149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6940b0140 0x7fd6940adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6940b2050 0x7fd6940ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6940ab1d0 con 0x7fd6940b0140 2022-07-21T01:36:56.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6940aafa0 con 0x7fd6940b2050 2022-07-21T01:36:56.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69bf8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd694149680 con 0x7fd6940a7250 2022-07-21T01:36:56.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699528700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6940b0140 0x7fd6940adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699528700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6940b0140 0x7fd6940adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45588/0 (socket says 172.21.15.3:45588) 2022-07-21T01:36:56.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699528700 1 -- 172.21.15.3:0/520214240 learned_addr learned my addr 172.21.15.3:0/520214240 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:56.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd69a52a700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6940b2050 0x7fd6940ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.425+0000 7fd699d29700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 0x7fd694149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd699528700 1 -- 172.21.15.3:0/520214240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 msgr2=0x7fd694149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd699528700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 0x7fd694149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd699528700 1 -- 172.21.15.3:0/520214240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6940b2050 msgr2=0x7fd6940ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd699528700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6940b2050 0x7fd6940ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd699528700 1 -- 172.21.15.3:0/520214240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c00b180 con 0x7fd6940b0140 2022-07-21T01:36:56.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd699528700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6940b0140 0x7fd6940adb20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fd694149810 tx=0x7fd690016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd682ffd700 1 -- 172.21.15.3:0/520214240 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd69000b600 con 0x7fd6940b0140 2022-07-21T01:36:56.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd682ffd700 1 -- 172.21.15.3:0/520214240 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69001f3f0 con 0x7fd6940b0140 2022-07-21T01:36:56.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68400b1e0 con 0x7fd6940b0140 2022-07-21T01:36:56.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd682ffd700 1 -- 172.21.15.3:0/520214240 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd69001f750 con 0x7fd6940b0140 2022-07-21T01:36:56.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd690009470 con 0x7fd6940b0140 2022-07-21T01:36:56.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6940aa9b0 con 0x7fd6940b0140 2022-07-21T01:36:56.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd682ffd700 1 -- 172.21.15.3:0/520214240 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd690026030 con 0x7fd6940b0140 2022-07-21T01:36:56.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd682ffd700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6740539e0 0x7fd674055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd682ffd700 1 -- 172.21.15.3:0/520214240 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd69006cd20 con 0x7fd6940b0140 2022-07-21T01:36:56.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.429+0000 7fd699d29700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6740539e0 0x7fd674055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.433+0000 7fd699d29700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6740539e0 0x7fd674055ea0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fd68400b000 tx=0x7fd684006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.437+0000 7fd682ffd700 1 -- 172.21.15.3:0/520214240 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd6940aa9b0 con 0x7fd6940b0140 2022-07-21T01:36:56.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.557+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd6940aa9b0 con 0x7fd6940b0140 2022-07-21T01:36:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.557+0000 7fd682ffd700 1 -- 172.21.15.3:0/520214240 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v58) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd6940aa9b0 con 0x7fd6940b0140 2022-07-21T01:36:56.560 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:36:56.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6740539e0 msgr2=0x7fd674055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6740539e0 0x7fd674055ea0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fd68400b000 tx=0x7fd684006eb0).stop 2022-07-21T01:36:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6940b0140 msgr2=0x7fd6940adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6940b0140 0x7fd6940adb20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fd694149810 tx=0x7fd690016000).stop 2022-07-21T01:36:56.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 shutdown_connections 2022-07-21T01:36:56.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6740539e0 0x7fd674055ea0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6940b2050 0x7fd6940ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6940b0140 0x7fd6940adb20 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 --2- 172.21.15.3:0/520214240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6940a7250 0x7fd694149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 >> 172.21.15.3:0/520214240 conn(0x7fd69405f840 msgr2=0x7fd69414e270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:56.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 shutdown_connections 2022-07-21T01:36:56.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.561+0000 7fd69bf8d700 1 -- 172.21.15.3:0/520214240 wait complete. 2022-07-21T01:36:56.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:36:56.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.677+0000 7faea5227700 1 Processor -- start 2022-07-21T01:36:56.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.677+0000 7faea5227700 1 -- start start 2022-07-21T01:36:56.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faea00619b0 0x7faea00b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faea00b6ee0 0x7faea0143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea0061d90 con 0x7faea00b5ed0 2022-07-21T01:36:56.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea0142f60 con 0x7faea00619b0 2022-07-21T01:36:56.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea00ba5a0 con 0x7faea00b6ee0 2022-07-21T01:36:56.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faea00619b0 0x7faea00b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea4a26700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7faea00b5ed0 0x7faea00b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53874/0 (socket says 172.21.15.3:53874) 2022-07-21T01:36:56.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea4a26700 1 -- 172.21.15.3:0/1952748176 learned_addr learned my addr 172.21.15.3:0/1952748176 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:56.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9f7fe700 1 --2- 172.21.15.3:0/1952748176 >> v2:172.21.15.143:3300/0 conn(0x7faea00b6ee0 0x7faea0143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9effd700 1 -- 172.21.15.3:0/1952748176 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782282779 0 0) 0x7faea0061d90 con 0x7faea00b5ed0 2022-07-21T01:36:56.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9effd700 1 -- 172.21.15.3:0/1952748176 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae8c003260 con 0x7faea00b5ed0 2022-07-21T01:36:56.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9ffff700 1 -- 172.21.15.3:0/1952748176 >> v2:172.21.15.143:3300/0 conn(0x7faea00b6ee0 msgr2=0x7faea0143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:56.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9ffff700 1 --2- 172.21.15.3:0/1952748176 >> v2:172.21.15.143:3300/0 conn(0x7faea00b6ee0 0x7faea0143890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9ffff700 1 -- 172.21.15.3:0/1952748176 >> v1:172.21.15.3:6789/0 conn(0x7faea00b5ed0 legacy=0x7faea00b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9ffff700 1 -- 172.21.15.3:0/1952748176 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0143150 con 0x7faea00619b0 2022-07-21T01:36:56.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9ffff700 1 --2- 172.21.15.3:0/1952748176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00619b0 0x7faea00b48a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fae94002f70 tx=0x7fae94006df0).ready entity=mon.0 client_cookie=cffe9eac6acf8934 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9effd700 1 -- 172.21.15.3:0/1952748176 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae9400d4f0 con 0x7faea00619b0 2022-07-21T01:36:56.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9effd700 1 -- 172.21.15.3:0/1952748176 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae94000e90 con 0x7faea00619b0 2022-07-21T01:36:56.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7fae9effd700 1 -- 172.21.15.3:0/1952748176 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae9400d840 con 0x7faea00619b0 2022-07-21T01:36:56.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 -- 172.21.15.3:0/1952748176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00619b0 msgr2=0x7faea00b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 --2- 172.21.15.3:0/1952748176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00619b0 0x7faea00b48a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fae94002f70 tx=0x7fae94006df0).stop 2022-07-21T01:36:56.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 -- 172.21.15.3:0/1952748176 shutdown_connections 2022-07-21T01:36:56.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 --2- 172.21.15.3:0/1952748176 >> v2:172.21.15.143:3300/0 conn(0x7faea00b6ee0 0x7faea0143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 --2- 172.21.15.3:0/1952748176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00619b0 0x7faea00b48a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 -- 172.21.15.3:0/1952748176 >> 172.21.15.3:0/1952748176 conn(0x7faea006af80 msgr2=0x7faea005fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:56.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 -- 172.21.15.3:0/1952748176 shutdown_connections 2022-07-21T01:36:56.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.681+0000 7faea5227700 1 -- 172.21.15.3:0/1952748176 wait complete. 2022-07-21T01:36:56.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 Processor -- start 2022-07-21T01:36:56.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 -- start start 2022-07-21T01:36:56.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faea00b5ed0 0x7faea00b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00b6ee0 0x7faea0152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faea00afb20 0x7faea0153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea0153660 con 0x7faea00b6ee0 2022-07-21T01:36:56.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea00af0a0 con 0x7faea00afb20 2022-07-21T01:36:56.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea00af340 con 0x7faea00b5ed0 2022-07-21T01:36:56.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00b6ee0 0x7faea0152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00b6ee0 0x7faea0152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45600/0 (socket says 172.21.15.3:45600) 2022-07-21T01:36:56.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9f7fe700 1 -- 172.21.15.3:0/2920152011 learned_addr learned my addr 172.21.15.3:0/2920152011 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:56.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea4a26700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faea00afb20 0x7faea0153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9ffff700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faea00b5ed0 0x7faea00b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9f7fe700 1 -- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faea00b5ed0 msgr2=0x7faea00b8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9f7fe700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faea00b5ed0 0x7faea00b8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9f7fe700 1 -- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faea00afb20 msgr2=0x7faea0153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9f7fe700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faea00afb20 0x7faea0153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9f7fe700 1 -- 172.21.15.3:0/2920152011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9400b180 con 0x7faea00b6ee0 2022-07-21T01:36:56.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea4a26700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faea00afb20 0x7faea0153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:36:56.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9ffff700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faea00b5ed0 0x7faea00b8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:36:56.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9f7fe700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00b6ee0 0x7faea0152c80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fae9000c000 tx=0x7fae90006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9d7fa700 1 -- 172.21.15.3:0/2920152011 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae90010780 con 0x7faea00b6ee0 2022-07-21T01:36:56.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9d7fa700 1 -- 172.21.15.3:0/2920152011 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae900108e0 con 0x7faea00b6ee0 2022-07-21T01:36:56.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea00af5e0 con 0x7faea00b6ee0 2022-07-21T01:36:56.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7fae9d7fa700 1 -- 172.21.15.3:0/2920152011 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae90010c40 con 0x7faea00b6ee0 2022-07-21T01:36:56.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.685+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea00b00c0 con 0x7faea00b6ee0 2022-07-21T01:36:56.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.689+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea00574e0 con 0x7faea00b6ee0 2022-07-21T01:36:56.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.689+0000 7fae9d7fa700 1 -- 172.21.15.3:0/2920152011 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fae9001c460 con 0x7faea00b6ee0 2022-07-21T01:36:56.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.689+0000 7fae9d7fa700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae80053a10 0x7fae80055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.689+0000 7fae9d7fa700 1 -- 172.21.15.3:0/2920152011 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fae9006c3a0 con 0x7faea00b6ee0 2022-07-21T01:36:56.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.689+0000 7fae9ffff700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae80053a10 0x7fae80055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.689+0000 7fae9ffff700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae80053a10 0x7fae80055ed0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fae94015000 tx=0x7fae94014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.693+0000 7fae9d7fa700 1 -- 172.21.15.3:0/2920152011 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fae90043100 con 0x7faea00b6ee0 2022-07-21T01:36:56.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.821+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faea005db60 con 0x7faea00b6ee0 2022-07-21T01:36:56.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.821+0000 7fae9d7fa700 1 -- 172.21.15.3:0/2920152011 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+10 (secure 0 0 0) 0x7fae90046720 con 0x7faea00b6ee0 2022-07-21T01:36:56.822 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:36:56.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.821+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae80053a10 msgr2=0x7fae80055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.821+0000 7faea5227700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae80053a10 0x7fae80055ed0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fae94015000 tx=0x7fae94014000).stop 2022-07-21T01:36:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.821+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00b6ee0 msgr2=0x7faea0152c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.821+0000 7faea5227700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00b6ee0 0x7faea0152c80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fae9000c000 tx=0x7fae90006df0).stop 2022-07-21T01:36:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.825+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 shutdown_connections 2022-07-21T01:36:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.825+0000 7faea5227700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae80053a10 0x7fae80055ed0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.825+0000 7faea5227700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faea00afb20 0x7faea0153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.825+0000 7faea5227700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faea00b6ee0 0x7faea0152c80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.825+0000 7faea5227700 1 --2- 172.21.15.3:0/2920152011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faea00b5ed0 0x7faea00b8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.825+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 >> 172.21.15.3:0/2920152011 conn(0x7faea006af80 msgr2=0x7faea005be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.825+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 shutdown_connections 2022-07-21T01:36:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.825+0000 7faea5227700 1 -- 172.21.15.3:0/2920152011 wait complete. 2022-07-21T01:36:56.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:36:56.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.949+0000 7f530f59e700 1 Processor -- start 2022-07-21T01:36:56.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.949+0000 7f530f59e700 1 -- start start 2022-07-21T01:36:56.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.949+0000 7f530f59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f53100a9320 0x7f53101466e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.949+0000 7f530f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f53100b0d10 0x7f5310146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.949+0000 7f530f59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53100b12d0 con 0x7f53100b0560 2022-07-21T01:36:56.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.949+0000 7f530f59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53100a9700 con 0x7f53100a9320 2022-07-21T01:36:56.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.949+0000 7f530f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53100b1500 con 0x7f53100b0d10 2022-07-21T01:36:56.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530ed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f53100b0560 0x7f5310140350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53890/0 (socket says 172.21.15.3:53890) 2022-07-21T01:36:56.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530ed9d700 1 -- 172.21.15.3:0/1041223496 learned_addr learned my addr 172.21.15.3:0/1041223496 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:36:56.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530e59c700 1 --2- 172.21.15.3:0/1041223496 >> v2:172.21.15.3:3300/0 conn(0x7f53100a9320 0x7f53101466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530dd9b700 1 --2- 172.21.15.3:0/1041223496 >> v2:172.21.15.143:3300/0 conn(0x7f53100b0d10 0x7f5310146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530d59a700 1 -- 172.21.15.3:0/1041223496 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109421967 0 0) 0x7f53100b12d0 con 0x7f53100b0560 2022-07-21T01:36:56.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530d59a700 1 -- 172.21.15.3:0/1041223496 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f4003260 con 0x7f53100b0560 2022-07-21T01:36:56.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530e59c700 1 -- 172.21.15.3:0/1041223496 >> v2:172.21.15.143:3300/0 conn(0x7f53100b0d10 msgr2=0x7f5310146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:56.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530e59c700 1 --2- 172.21.15.3:0/1041223496 >> v2:172.21.15.143:3300/0 conn(0x7f53100b0d10 0x7f5310146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530e59c700 1 -- 172.21.15.3:0/1041223496 >> v1:172.21.15.3:6789/0 conn(0x7f53100b0560 legacy=0x7f5310140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530d59a700 1 -- 172.21.15.3:0/1041223496 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2624798550 0 0) 0x7f52f4003260 con 0x7f53100b0560 2022-07-21T01:36:56.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530e59c700 1 -- 172.21.15.3:0/1041223496 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53100b17a0 con 0x7f53100a9320 2022-07-21T01:36:56.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530e59c700 1 --2- 172.21.15.3:0/1041223496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100a9320 0x7f53101466e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f5308001b80 tx=0x7f5304006df0).ready entity=mon.0 client_cookie=7413e9bcbe5f06f2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530d59a700 1 -- 172.21.15.3:0/1041223496 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53080019e0 con 0x7f53100a9320 2022-07-21T01:36:56.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530d59a700 1 -- 172.21.15.3:0/1041223496 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5304000e90 con 0x7f53100a9320 2022-07-21T01:36:56.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530d59a700 1 -- 172.21.15.3:0/1041223496 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5304012460 con 0x7f53100a9320 2022-07-21T01:36:56.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530f59e700 1 -- 172.21.15.3:0/1041223496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100a9320 msgr2=0x7f53101466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530f59e700 1 --2- 172.21.15.3:0/1041223496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100a9320 0x7f53101466e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f5308001b80 tx=0x7f5304006df0).stop 2022-07-21T01:36:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530f59e700 1 -- 172.21.15.3:0/1041223496 shutdown_connections 2022-07-21T01:36:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530f59e700 1 --2- 172.21.15.3:0/1041223496 >> v2:172.21.15.143:3300/0 conn(0x7f53100b0d10 0x7f5310146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530f59e700 1 --2- 172.21.15.3:0/1041223496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100a9320 0x7f53101466e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530f59e700 1 -- 172.21.15.3:0/1041223496 >> 172.21.15.3:0/1041223496 conn(0x7f53100612d0 msgr2=0x7f5310062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:56.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530f59e700 1 -- 172.21.15.3:0/1041223496 shutdown_connections 2022-07-21T01:36:56.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.953+0000 7f530f59e700 1 -- 172.21.15.3:0/1041223496 wait complete. 2022-07-21T01:36:56.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 Processor -- start 2022-07-21T01:36:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 -- start start 2022-07-21T01:36:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53100b0560 0x7f5310145950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100b0d10 0x7f53100b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53100b3cb0 0x7f53100b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5310139b90 con 0x7f53100b0d10 2022-07-21T01:36:56.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53100b4f20 con 0x7f53100b3cb0 2022-07-21T01:36:56.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53100b6ac0 con 0x7f53100b0560 2022-07-21T01:36:56.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100b0d10 0x7f53100b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100b0d10 0x7f53100b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45614/0 (socket says 172.21.15.3:45614) 2022-07-21T01:36:56.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530dd9b700 1 -- 172.21.15.3:0/3995308173 learned_addr learned my addr 172.21.15.3:0/3995308173 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:56.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530ed9d700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53100b3cb0 0x7f53100b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530e59c700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53100b0560 0x7f5310145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530dd9b700 1 -- 172.21.15.3:0/3995308173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53100b0560 msgr2=0x7f5310145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530dd9b700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53100b0560 0x7f5310145950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530dd9b700 1 -- 172.21.15.3:0/3995308173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53100b3cb0 msgr2=0x7f53100b5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:56.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530dd9b700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53100b3cb0 0x7f53100b5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:56.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530dd9b700 1 -- 172.21.15.3:0/3995308173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530400b180 con 0x7f53100b0d10 2022-07-21T01:36:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530dd9b700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100b0d10 0x7f53100b37c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f5310145130 tx=0x7f5300006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f52ff7fe700 1 -- 172.21.15.3:0/3995308173 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5300010780 con 0x7f53100b0d10 2022-07-21T01:36:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f52ff7fe700 1 -- 172.21.15.3:0/3995308173 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53000108e0 con 0x7f53100b0d10 2022-07-21T01:36:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53101533c0 con 0x7f53100b0d10 2022-07-21T01:36:56.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f52ff7fe700 1 -- 172.21.15.3:0/3995308173 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5300010c40 con 0x7f53100b0d10 2022-07-21T01:36:56.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.957+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5310144d70 con 0x7f53100b0d10 2022-07-21T01:36:56.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.961+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53100574e0 con 0x7f53100b0d10 2022-07-21T01:36:56.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.961+0000 7f52ff7fe700 1 -- 172.21.15.3:0/3995308173 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5300026430 con 0x7f53100b0d10 2022-07-21T01:36:56.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.961+0000 7f52ff7fe700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52ec053950 0x7f52ec055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:56.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.961+0000 7f52ff7fe700 1 -- 172.21.15.3:0/3995308173 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f530006cf40 con 0x7f53100b0d10 2022-07-21T01:36:56.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.961+0000 7f530e59c700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52ec053950 0x7f52ec055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:56.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.961+0000 7f530e59c700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52ec053950 0x7f52ec055e10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f530401a000 tx=0x7f5304014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:56.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:56.969+0000 7f52ff7fe700 1 -- 172.21.15.3:0/3995308173 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5300043ca0 con 0x7f53100b0d10 2022-07-21T01:36:57.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.093+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f53101475c0 con 0x7f53100b0d10 2022-07-21T01:36:57.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.093+0000 7f52ff7fe700 1 -- 172.21.15.3:0/3995308173 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+10 (secure 0 0 0) 0x7f53000472c0 con 0x7f53100b0d10 2022-07-21T01:36:57.095 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:36:57.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52ec053950 msgr2=0x7f52ec055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:57.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52ec053950 0x7f52ec055e10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f530401a000 tx=0x7f5304014000).stop 2022-07-21T01:36:57.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100b0d10 msgr2=0x7f53100b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:57.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100b0d10 0x7f53100b37c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f5310145130 tx=0x7f5300006df0).stop 2022-07-21T01:36:57.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 shutdown_connections 2022-07-21T01:36:57.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52ec053950 0x7f52ec055e10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53100b3cb0 0x7f53100b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53100b0d10 0x7f53100b37c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 --2- 172.21.15.3:0/3995308173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53100b0560 0x7f5310145950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 >> 172.21.15.3:0/3995308173 conn(0x7f53100612d0 msgr2=0x7f531005be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:57.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 shutdown_connections 2022-07-21T01:36:57.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.097+0000 7f530f59e700 1 -- 172.21.15.3:0/3995308173 wait complete. 2022-07-21T01:36:57.110 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:36:57.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:36:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 Processor -- start 2022-07-21T01:36:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 -- start start 2022-07-21T01:36:57.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f833c0b1f40 0x7f833c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:57.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f833c148530 0x7f833c0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:57.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 0x7f833c1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:57.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0a9250 con 0x7f833c0b1f40 2022-07-21T01:36:57.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0b04e0 con 0x7f833c148530 2022-07-21T01:36:57.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0b0780 con 0x7f833c0a7250 2022-07-21T01:36:57.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f833c0b1f40 0x7f833c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:57.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f833c0b1f40 0x7f833c0b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45620/0 (socket says 172.21.15.3:45620) 2022-07-21T01:36:57.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833b7fe700 1 -- 172.21.15.3:0/2168020142 learned_addr learned my addr 172.21.15.3:0/2168020142 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:57.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833affd700 1 --2- 172.21.15.3:0/2168020142 >> v2:172.21.15.143:3300/0 conn(0x7f833c148530 0x7f833c0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:57.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833bfff700 1 --2- 172.21.15.3:0/2168020142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 0x7f833c1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:57.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833b7fe700 1 -- 172.21.15.3:0/2168020142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 msgr2=0x7f833c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:57.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833b7fe700 1 --2- 172.21.15.3:0/2168020142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 0x7f833c1448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833b7fe700 1 -- 172.21.15.3:0/2168020142 >> v2:172.21.15.143:3300/0 conn(0x7f833c148530 msgr2=0x7f833c0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:36:57.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833b7fe700 1 --2- 172.21.15.3:0/2168020142 >> v2:172.21.15.143:3300/0 conn(0x7f833c148530 0x7f833c0b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833b7fe700 1 -- 172.21.15.3:0/2168020142 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0b0a20 con 0x7f833c0b1f40 2022-07-21T01:36:57.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833affd700 1 --2- 172.21.15.3:0/2168020142 >> v2:172.21.15.143:3300/0 conn(0x7f833c148530 0x7f833c0b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:36:57.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833bfff700 1 --2- 172.21.15.3:0/2168020142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 0x7f833c1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:36:57.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833b7fe700 1 --2- 172.21.15.3:0/2168020142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c0b1f40 0x7f833c0b1160 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f833c1478c0 tx=0x7f832c009290).ready entity=mon.0 client_cookie=88359061485fb64c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:57.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833a7fc700 1 -- 172.21.15.3:0/2168020142 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832c00b860 con 0x7f833c0b1f40 2022-07-21T01:36:57.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833a7fc700 1 -- 172.21.15.3:0/2168020142 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c00b9c0 con 0x7f833c0b1f40 2022-07-21T01:36:57.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f833a7fc700 1 -- 172.21.15.3:0/2168020142 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832c00bb30 con 0x7f833c0b1f40 2022-07-21T01:36:57.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 -- 172.21.15.3:0/2168020142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c0b1f40 msgr2=0x7f833c0b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:57.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 --2- 172.21.15.3:0/2168020142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c0b1f40 0x7f833c0b1160 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f833c1478c0 tx=0x7f832c009290).stop 2022-07-21T01:36:57.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 -- 172.21.15.3:0/2168020142 shutdown_connections 2022-07-21T01:36:57.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 --2- 172.21.15.3:0/2168020142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 0x7f833c1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 --2- 172.21.15.3:0/2168020142 >> v2:172.21.15.143:3300/0 conn(0x7f833c148530 0x7f833c0b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 --2- 172.21.15.3:0/2168020142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c0b1f40 0x7f833c0b1160 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 -- 172.21.15.3:0/2168020142 >> 172.21.15.3:0/2168020142 conn(0x7f833c05f840 msgr2=0x7f833c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:57.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 -- 172.21.15.3:0/2168020142 shutdown_connections 2022-07-21T01:36:57.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.217+0000 7f8341b80700 1 -- 172.21.15.3:0/2168020142 wait complete. 2022-07-21T01:36:57.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 Processor -- start 2022-07-21T01:36:57.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 -- start start 2022-07-21T01:36:57.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 0x7f833c146bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:57.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f833c0b1f40 0x7f833c14ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:57.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c148530 0x7f833c0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:57.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c14b7e0 con 0x7f833c148530 2022-07-21T01:36:57.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c14ba80 con 0x7f833c0b1f40 2022-07-21T01:36:57.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c14a360 con 0x7f833c0a7250 2022-07-21T01:36:57.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c148530 0x7f833c0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:57.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c148530 0x7f833c0adf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45630/0 (socket says 172.21.15.3:45630) 2022-07-21T01:36:57.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833bfff700 1 -- 172.21.15.3:0/3408382666 learned_addr learned my addr 172.21.15.3:0/3408382666 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:36:57.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833affd700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f833c0b1f40 0x7f833c14ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:57.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833b7fe700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 0x7f833c146bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:57.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833bfff700 1 -- 172.21.15.3:0/3408382666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 msgr2=0x7f833c146bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:57.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833bfff700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 0x7f833c146bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833bfff700 1 -- 172.21.15.3:0/3408382666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f833c0b1f40 msgr2=0x7f833c14ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:57.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833bfff700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f833c0b1f40 0x7f833c14ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833bfff700 1 -- 172.21.15.3:0/3408382666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c009530 con 0x7f833c148530 2022-07-21T01:36:57.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f833bfff700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c148530 0x7f833c0adf70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f8334007000 tx=0x7f8334008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:57.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8338ff9700 1 -- 172.21.15.3:0/3408382666 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8334015070 con 0x7f833c148530 2022-07-21T01:36:57.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8338ff9700 1 -- 172.21.15.3:0/3408382666 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833400ce80 con 0x7f833c148530 2022-07-21T01:36:57.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c14a800 con 0x7f833c148530 2022-07-21T01:36:57.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8338ff9700 1 -- 172.21.15.3:0/3408382666 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8334010770 con 0x7f833c148530 2022-07-21T01:36:57.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c0ab840 con 0x7f833c148530 2022-07-21T01:36:57.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.221+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c14a520 con 0x7f833c148530 2022-07-21T01:36:57.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.225+0000 7f8338ff9700 1 -- 172.21.15.3:0/3408382666 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f83340108d0 con 0x7f833c148530 2022-07-21T01:36:57.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.225+0000 7f8338ff9700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f831c0539e0 0x7f831c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:36:57.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.225+0000 7f8338ff9700 1 -- 172.21.15.3:0/3408382666 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f833406c5a0 con 0x7f833c148530 2022-07-21T01:36:57.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.225+0000 7f833b7fe700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f831c0539e0 0x7f831c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:36:57.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.225+0000 7f833b7fe700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f831c0539e0 0x7f831c055ea0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f832c01a000 tx=0x7f832c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:36:57.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.229+0000 7f8338ff9700 1 -- 172.21.15.3:0/3408382666 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8334043300 con 0x7f833c148530 2022-07-21T01:36:57.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.385+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f833c0b0cc0 con 0x7f833c148530 2022-07-21T01:36:57.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8338ff9700 1 -- 172.21.15.3:0/3408382666 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f8334046920 con 0x7f833c148530 2022-07-21T01:36:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:36:57.390 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:36:52.675976+0000","modified":"2022-07-21T01:36:53.679085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/3381448192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3381448192},{"type":"v1","addr":"172.21.15.3:6837","nonce":3381448192}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-21T01:36:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f831c0539e0 msgr2=0x7f831c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f831c0539e0 0x7f831c055ea0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f832c01a000 tx=0x7f832c016000).stop 2022-07-21T01:36:57.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c148530 msgr2=0x7f833c0adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:36:57.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c148530 0x7f833c0adf70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f8334007000 tx=0x7f8334008040).stop 2022-07-21T01:36:57.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 shutdown_connections 2022-07-21T01:36:57.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f833c148530 0x7f833c0adf70 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f831c0539e0 0x7f831c055ea0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f833c0b1f40 0x7f833c14ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 --2- 172.21.15.3:0/3408382666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f833c0a7250 0x7f833c146bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:36:57.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 >> 172.21.15.3:0/3408382666 conn(0x7f833c05f840 msgr2=0x7f833c144df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:36:57.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 shutdown_connections 2022-07-21T01:36:57.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:36:57.389+0000 7f8341b80700 1 -- 172.21.15.3:0/3408382666 wait complete. 2022-07-21T01:36:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2022-07-21T01:36:57.712 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:36:52.675976+0000', 'modified': '2022-07-21T01:36:53.679085+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5586}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5586': {'gid': 5586, 'name': 'a', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/3381448192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 3381448192}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 3381448192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:36:57.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:36:57.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:36:57.836 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":5586,"want_state":"up:active","state":"up:active","rank_uptime":5.1489183560000003,"mdsmap_epoch":23,"osdmap_epoch":58,"osdmap_epoch_barrier":57,"uptime":9.4174909860000007} 2022-07-21T01:36:57.837 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 5586, "mdsmap_epoch": 23, "osdmap_epoch": 58, "osdmap_epoch_barrier": 57, "rank_uptime": 5.148918356, "state": "up:active", "uptime": 9.417490986, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:36:57.838 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:36:57.838 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:36:57.839 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:36:57.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:57.841 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:57.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:36:57.892 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:36:57.893 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:36:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:36:57.952 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:36:57.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:57.953 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:36:57.966 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:36:57.967 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:58.011 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:36:58.030 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:36:58.031 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:36:58.042 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:36:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:36:58.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:58.045 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:36:58.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:36:58.085+0000 7f3c7ecde080 -1 init, newargv = 0x557e1da0b9f0 newargc=9ceph-fuse[ 2022-07-21T01:36:58.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:22500]: starting ceph client 2022-07-21T01:36:58.094 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:36:58.096 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:36:58.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:58.098 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:36:58.098 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:36:58.099 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:36:58.100 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:36:58.101 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:36:58.102 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:36:58.103 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:36:58.103 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:36:58.104 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:36:58.105 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:36:58.106 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:36:58.107 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:36:58.107 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:36:58.108 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:36:58.109 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:36:58.110 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:36:58.111 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:36:58.112 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:36:58.112 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:36:58.113 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:36:58.114 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:36:58.115 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:36:58.116 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:36:58.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22500]: starting fuse 2022-07-21T01:36:58.188 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22500.asok 2022-07-21T01:36:58.193 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22500.asok 2022-07-21T01:36:58.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:58.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22500.asok status 2022-07-21T01:36:58.284 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:36:58.285 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:36:58.285 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:36:58.285 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:36:58.285 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:36:58.286 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:36:58.286 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:36:58.286 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22500", 2022-07-21T01:36:58.286 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:36:58.287 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:36:58.287 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:36:58.287 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:36:58.287 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 5743, 2022-07-21T01:36:58.287 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:36:58.288 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:36:58.288 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:36:58.288 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 5743 2022-07-21T01:36:58.288 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:36:58.289 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:36:58.289 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:36:58.289 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:36:58.289 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3469535881 2022-07-21T01:36:58.290 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:36:58.290 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:36:58.290 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:36:58.290 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:36:58.291 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:36:58.291 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3469535881 2022-07-21T01:36:58.291 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:36:58.291 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.5743 172.21.15.171:0/3469535881", 2022-07-21T01:36:58.292 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/3469535881", 2022-07-21T01:36:58.292 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:36:58.292 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 23, 2022-07-21T01:36:58.292 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 58, 2022-07-21T01:36:58.292 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:36:58.293 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:36:58.293 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:36:58.293 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:36:58.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:58.295 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:36:58.295 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:58.305 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:36:58.305 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:36:58.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:58.306 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:58.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:36:58.335 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:36:58.336 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:36:58.391 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:36:58.392 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:36:58.393 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-21T01:36:58.393 DEBUG:teuthology.orchestra.run.smithi171:> import sys 2022-07-21T01:36:58.394 DEBUG:teuthology.orchestra.run.smithi171:> import time 2022-07-21T01:36:58.394 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:36:58.395 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:36:58.395 DEBUG:teuthology.orchestra.run.smithi171:> n = 1000 2022-07-21T01:36:58.395 DEBUG:teuthology.orchestra.run.smithi171:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-21T01:36:58.396 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:36:58.396 DEBUG:teuthology.orchestra.run.smithi171:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-21T01:36:58.397 DEBUG:teuthology.orchestra.run.smithi171:> os.makedirs(os.path.dirname(abs_path)) 2022-07-21T01:36:58.397 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:36:58.398 DEBUG:teuthology.orchestra.run.smithi171:> for i in range(0, n): 2022-07-21T01:36:58.398 DEBUG:teuthology.orchestra.run.smithi171:> fname = "{0}_{1}".format(abs_path, i) 2022-07-21T01:36:58.399 DEBUG:teuthology.orchestra.run.smithi171:> with open(fname, '"'"'w'"'"') as f: 2022-07-21T01:36:58.399 DEBUG:teuthology.orchestra.run.smithi171:> f.write('"'"'content'"'"') 2022-07-21T01:36:58.399 DEBUG:teuthology.orchestra.run.smithi171:> if True: 2022-07-21T01:36:58.400 DEBUG:teuthology.orchestra.run.smithi171:> f.flush() 2022-07-21T01:36:58.400 DEBUG:teuthology.orchestra.run.smithi171:> os.fsync(f.fileno()) 2022-07-21T01:36:58.401 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:37:01.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:37:01.688 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:37:01.691 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:37:01.694 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:37:09.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:37:09.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 Processor -- start 2022-07-21T01:37:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 -- start start 2022-07-21T01:37:09.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f13700b1f40 0x7f13700b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:09.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13700b04e0 con 0x7f1370148530 2022-07-21T01:37:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13700b0780 con 0x7f13700a7250 2022-07-21T01:37:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13700a9250 con 0x7f13700b1f40 2022-07-21T01:37:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136effd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1370148530 0x7f13700b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53914/0 (socket says 172.21.15.3:53914) 2022-07-21T01:37:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136effd700 1 -- 172.21.15.3:0/3404541425 learned_addr learned my addr 172.21.15.3:0/3404541425 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:37:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136f7fe700 1 --2- 172.21.15.3:0/3404541425 >> v2:172.21.15.3:3300/0 conn(0x7f13700b1f40 0x7f13700b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136e7fc700 1 -- 172.21.15.3:0/3404541425 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937282692 0 0) 0x7f13700b04e0 con 0x7f1370148530 2022-07-21T01:37:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136e7fc700 1 -- 172.21.15.3:0/3404541425 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f135c003260 con 0x7f1370148530 2022-07-21T01:37:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136f7fe700 1 -- 172.21.15.3:0/3404541425 >> v1:172.21.15.143:6789/0 conn(0x7f13700a7250 legacy=0x7f13701448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136e7fc700 1 -- 172.21.15.3:0/3404541425 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1817709909 0 0) 0x7f135c003260 con 0x7f1370148530 2022-07-21T01:37:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136f7fe700 1 -- 172.21.15.3:0/3404541425 >> v1:172.21.15.3:6789/0 conn(0x7f1370148530 legacy=0x7f13700b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136f7fe700 1 -- 172.21.15.3:0/3404541425 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13700b0a20 con 0x7f13700b1f40 2022-07-21T01:37:09.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136f7fe700 1 --2- 172.21.15.3:0/3404541425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700b1f40 0x7f13700b1160 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f13701478c0 tx=0x7f1364006df0).ready entity=mon.0 client_cookie=41d66e10424f4ac1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:09.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136e7fc700 1 -- 172.21.15.3:0/3404541425 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1360001660 con 0x7f13700b1f40 2022-07-21T01:37:09.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136e7fc700 1 -- 172.21.15.3:0/3404541425 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1364000e90 con 0x7f13700b1f40 2022-07-21T01:37:09.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f136e7fc700 1 -- 172.21.15.3:0/3404541425 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f136400d4f0 con 0x7f13700b1f40 2022-07-21T01:37:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 -- 172.21.15.3:0/3404541425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700b1f40 msgr2=0x7f13700b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 --2- 172.21.15.3:0/3404541425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700b1f40 0x7f13700b1160 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f13701478c0 tx=0x7f1364006df0).stop 2022-07-21T01:37:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 -- 172.21.15.3:0/3404541425 shutdown_connections 2022-07-21T01:37:09.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 --2- 172.21.15.3:0/3404541425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700b1f40 0x7f13700b1160 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:09.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 -- 172.21.15.3:0/3404541425 >> 172.21.15.3:0/3404541425 conn(0x7f137005f840 msgr2=0x7f137005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:09.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 -- 172.21.15.3:0/3404541425 shutdown_connections 2022-07-21T01:37:09.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.805+0000 7f1375efe700 1 -- 172.21.15.3:0/3404541425 wait complete. 2022-07-21T01:37:09.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 Processor -- start 2022-07-21T01:37:09.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 -- start start 2022-07-21T01:37:09.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700a7250 0x7f13700a9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:09.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1370148530 0x7f137015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:09.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13700af770 0x7f137014a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:09.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137015a170 con 0x7f13700a7250 2022-07-21T01:37:09.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13700afcf0 con 0x7f13700af770 2022-07-21T01:37:09.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137015a360 con 0x7f1370148530 2022-07-21T01:37:09.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700a7250 0x7f13700a9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:09.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700a7250 0x7f13700a9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45640/0 (socket says 172.21.15.3:45640) 2022-07-21T01:37:09.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136f7fe700 1 -- 172.21.15.3:0/3833232639 learned_addr learned my addr 172.21.15.3:0/3833232639 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:09.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136effd700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1370148530 0x7f137015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:09.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136ffff700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13700af770 0x7f137014a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:09.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136f7fe700 1 -- 172.21.15.3:0/3833232639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1370148530 msgr2=0x7f137015aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:09.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136f7fe700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1370148530 0x7f137015aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136f7fe700 1 -- 172.21.15.3:0/3833232639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13700af770 msgr2=0x7f137014a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136f7fe700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13700af770 0x7f137014a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136f7fe700 1 -- 172.21.15.3:0/3833232639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136400b180 con 0x7f13700a7250 2022-07-21T01:37:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136f7fe700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700a7250 0x7f13700a9910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f13701478c0 tx=0x7f1364014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136cff9700 1 -- 172.21.15.3:0/3833232639 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1364016070 con 0x7f13700a7250 2022-07-21T01:37:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136cff9700 1 -- 172.21.15.3:0/3833232639 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1364013af0 con 0x7f13700a7250 2022-07-21T01:37:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137015a600 con 0x7f13700a7250 2022-07-21T01:37:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f136cff9700 1 -- 172.21.15.3:0/3833232639 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f136400d4f0 con 0x7f13700a7250 2022-07-21T01:37:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.809+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137014a6b0 con 0x7f13700a7250 2022-07-21T01:37:09.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.813+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13700b0cc0 con 0x7f13700a7250 2022-07-21T01:37:09.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.813+0000 7f136cff9700 1 -- 172.21.15.3:0/3833232639 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f136400f030 con 0x7f13700a7250 2022-07-21T01:37:09.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.813+0000 7f136cff9700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1350053a70 0x7f1350055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:09.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.813+0000 7f136effd700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1350053a70 0x7f1350055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:09.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.813+0000 7f136cff9700 1 -- 172.21.15.3:0/3833232639 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f136406d090 con 0x7f13700a7250 2022-07-21T01:37:09.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.813+0000 7f136effd700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1350053a70 0x7f1350055f30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f136000d000 tx=0x7f1360000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:09.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.821+0000 7f136cff9700 1 -- 172.21.15.3:0/3833232639 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1364043f00 con 0x7f13700a7250 2022-07-21T01:37:09.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.981+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13700574e0 con 0x7f13700a7250 2022-07-21T01:37:09.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f136cff9700 1 -- 172.21.15.3:0/3833232639 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f1364047520 con 0x7f13700a7250 2022-07-21T01:37:09.984 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:37:09.984 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:36:52.675976+0000","modified":"2022-07-21T01:36:53.679085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/3381448192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3381448192},{"type":"v1","addr":"172.21.15.3:6837","nonce":3381448192}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-21T01:37:09.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1350053a70 msgr2=0x7f1350055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:09.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1350053a70 0x7f1350055f30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f136000d000 tx=0x7f1360000bc0).stop 2022-07-21T01:37:09.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700a7250 msgr2=0x7f13700a9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:09.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700a7250 0x7f13700a9910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f13701478c0 tx=0x7f1364014000).stop 2022-07-21T01:37:09.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 shutdown_connections 2022-07-21T01:37:09.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1350053a70 0x7f1350055f30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:09.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13700af770 0x7f137014a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:09.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1370148530 0x7f137015aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:09.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 --2- 172.21.15.3:0/3833232639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13700a7250 0x7f13700a9910 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:09.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 >> 172.21.15.3:0/3833232639 conn(0x7f137005f840 msgr2=0x7f1370144a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:09.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 shutdown_connections 2022-07-21T01:37:09.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:09.985+0000 7f1375efe700 1 -- 172.21.15.3:0/3833232639 wait complete. 2022-07-21T01:37:09.989 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2022-07-21T01:37:09.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-21T01:37:10.103 INFO:teuthology.orchestra.run.smithi003.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-21T01:37:10.103 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-21T01:37:10.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:37:10.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.201+0000 7feb9dca6700 1 Processor -- start 2022-07-21T01:37:10.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.201+0000 7feb9dca6700 1 -- start start 2022-07-21T01:37:10.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feb980b2390 0x7feb980abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb98141f40 0x7feb980ac480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb980ac970 con 0x7feb980b0160 2022-07-21T01:37:10.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb981453e0 con 0x7feb980b2390 2022-07-21T01:37:10.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb98145680 con 0x7feb98141f40 2022-07-21T01:37:10.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb977fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feb980b2390 0x7feb980abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb977fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feb980b2390 0x7feb980abf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45646/0 (socket says 172.21.15.3:45646) 2022-07-21T01:37:10.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb8effd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7feb980b0160 0x7feb98141830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53928/0 (socket says 172.21.15.3:53928) 2022-07-21T01:37:10.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb977fe700 1 -- 172.21.15.3:0/2779056733 learned_addr learned my addr 172.21.15.3:0/2779056733 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:10.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb97fff700 1 --2- 172.21.15.3:0/2779056733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb98141f40 0x7feb980ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb977fe700 1 -- 172.21.15.3:0/2779056733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb98141f40 msgr2=0x7feb980ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:10.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb977fe700 1 --2- 172.21.15.3:0/2779056733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb98141f40 0x7feb980ac480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb977fe700 1 -- 172.21.15.3:0/2779056733 >> v1:172.21.15.3:6789/0 conn(0x7feb980b0160 legacy=0x7feb98141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb96ffd700 1 -- 172.21.15.3:0/2779056733 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545413276 0 0) 0x7feb980ac970 con 0x7feb980b0160 2022-07-21T01:37:10.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb977fe700 1 -- 172.21.15.3:0/2779056733 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98141390 con 0x7feb980b2390 2022-07-21T01:37:10.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb977fe700 1 --2- 172.21.15.3:0/2779056733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 0x7feb980abf90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7feb88002f70 tx=0x7feb88006eb0).ready entity=mon.0 client_cookie=d29c9251002526f0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb96ffd700 1 -- 172.21.15.3:0/2779056733 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb8800d5c0 con 0x7feb980b2390 2022-07-21T01:37:10.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb96ffd700 1 -- 172.21.15.3:0/2779056733 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb88000e90 con 0x7feb980b2390 2022-07-21T01:37:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb96ffd700 1 -- 172.21.15.3:0/2779056733 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb8800d910 con 0x7feb980b2390 2022-07-21T01:37:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 -- 172.21.15.3:0/2779056733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 msgr2=0x7feb980abf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 --2- 172.21.15.3:0/2779056733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 0x7feb980abf90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7feb88002f70 tx=0x7feb88006eb0).stop 2022-07-21T01:37:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 -- 172.21.15.3:0/2779056733 shutdown_connections 2022-07-21T01:37:10.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 --2- 172.21.15.3:0/2779056733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb98141f40 0x7feb980ac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 --2- 172.21.15.3:0/2779056733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 0x7feb980abf90 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 -- 172.21.15.3:0/2779056733 >> 172.21.15.3:0/2779056733 conn(0x7feb9805f840 msgr2=0x7feb9805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:10.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 -- 172.21.15.3:0/2779056733 shutdown_connections 2022-07-21T01:37:10.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 -- 172.21.15.3:0/2779056733 wait complete. 2022-07-21T01:37:10.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 Processor -- start 2022-07-21T01:37:10.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.205+0000 7feb9dca6700 1 -- start start 2022-07-21T01:37:10.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb9dca6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb980b0160 0x7feb98140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb9dca6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 0x7feb980b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb9dca6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb98141f40 0x7feb98156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb9dca6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb98066ee0 con 0x7feb980b2390 2022-07-21T01:37:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb9dca6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb98067180 con 0x7feb98141f40 2022-07-21T01:37:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb9dca6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb98155ec0 con 0x7feb980b0160 2022-07-21T01:37:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb8effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 0x7feb980b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb8effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 0x7feb980b0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45654/0 (socket says 172.21.15.3:45654) 2022-07-21T01:37:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb8effd700 1 -- 172.21.15.3:0/101052581 learned_addr learned my addr 172.21.15.3:0/101052581 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb977fe700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb980b0160 0x7feb98140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb97fff700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb98141f40 0x7feb98156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb8effd700 1 -- 172.21.15.3:0/101052581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb980b0160 msgr2=0x7feb98140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb8effd700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb980b0160 0x7feb98140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb8effd700 1 -- 172.21.15.3:0/101052581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb98141f40 msgr2=0x7feb98156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb8effd700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb98141f40 0x7feb98156360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb8effd700 1 -- 172.21.15.3:0/101052581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8800b1e0 con 0x7feb980b2390 2022-07-21T01:37:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb8effd700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 0x7feb980b0c50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7feb80000ed0 tx=0x7feb80007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb957fa700 1 -- 172.21.15.3:0/101052581 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb80007170 con 0x7feb980b2390 2022-07-21T01:37:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb957fa700 1 -- 172.21.15.3:0/101052581 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb800072d0 con 0x7feb980b2390 2022-07-21T01:37:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9813cb80 con 0x7feb980b2390 2022-07-21T01:37:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb957fa700 1 -- 172.21.15.3:0/101052581 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb80004710 con 0x7feb980b2390 2022-07-21T01:37:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.209+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb98156850 con 0x7feb980b2390 2022-07-21T01:37:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.213+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb98067340 con 0x7feb980b2390 2022-07-21T01:37:10.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.213+0000 7feb957fa700 1 -- 172.21.15.3:0/101052581 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feb80004870 con 0x7feb980b2390 2022-07-21T01:37:10.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.213+0000 7feb957fa700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feb7c0539e0 0x7feb7c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.213+0000 7feb957fa700 1 -- 172.21.15.3:0/101052581 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7feb8006b880 con 0x7feb980b2390 2022-07-21T01:37:10.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.213+0000 7feb977fe700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feb7c0539e0 0x7feb7c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.213+0000 7feb977fe700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feb7c0539e0 0x7feb7c055ea0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7feb8801c000 tx=0x7feb88015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.217+0000 7feb957fa700 1 -- 172.21.15.3:0/101052581 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feb800425e0 con 0x7feb980b2390 2022-07-21T01:37:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.381+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb980574e0 con 0x7feb980b2390 2022-07-21T01:37:10.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.381+0000 7feb957fa700 1 -- 172.21.15.3:0/101052581 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2483 (secure 0 0 0) 0x7feb80045c00 con 0x7feb980b2390 2022-07-21T01:37:10.383 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:37:10.383 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:36:52.675976+0000","modified":"2022-07-21T01:36:53.679085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/3381448192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3381448192},{"type":"v1","addr":"172.21.15.3:6837","nonce":3381448192}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-21T01:37:10.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feb7c0539e0 msgr2=0x7feb7c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feb7c0539e0 0x7feb7c055ea0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7feb8801c000 tx=0x7feb88015000).stop 2022-07-21T01:37:10.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 msgr2=0x7feb980b0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 0x7feb980b0c50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7feb80000ed0 tx=0x7feb80007ab0).stop 2022-07-21T01:37:10.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 shutdown_connections 2022-07-21T01:37:10.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feb7c0539e0 0x7feb7c055ea0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb98141f40 0x7feb98156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb980b2390 0x7feb980b0c50 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 --2- 172.21.15.3:0/101052581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb980b0160 0x7feb98140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 >> 172.21.15.3:0/101052581 conn(0x7feb9805f840 msgr2=0x7feb980ab4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:10.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 shutdown_connections 2022-07-21T01:37:10.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.385+0000 7feb9dca6700 1 -- 172.21.15.3:0/101052581 wait complete. 2022-07-21T01:37:10.389 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2022-07-21T01:37:10.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-21T01:37:10.500 INFO:teuthology.orchestra.run.smithi003.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-21T01:37:10.501 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-21T01:37:10.501 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:37:10.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:37:10.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56cbbd700 1 Processor -- start 2022-07-21T01:37:10.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56cbbd700 1 -- start start 2022-07-21T01:37:10.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56cbbd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc5680ad1c0 0x7fc568144170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56cbbd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc568067560 con 0x7fc568067180 2022-07-21T01:37:10.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56cbbd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5680b0160 con 0x7fc5680b0870 2022-07-21T01:37:10.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56cbbd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5680b0400 con 0x7fc5680ad1c0 2022-07-21T01:37:10.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc568067180 0x7fc568143a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53940/0 (socket says 172.21.15.3:53940) 2022-07-21T01:37:10.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56659c700 1 -- 172.21.15.3:0/867319238 learned_addr learned my addr 172.21.15.3:0/867319238 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:37:10.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc565d9b700 1 --2- 172.21.15.3:0/867319238 >> v2:172.21.15.143:3300/0 conn(0x7fc5680ad1c0 0x7fc568144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56559a700 1 -- 172.21.15.3:0/867319238 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586878996 0 0) 0x7fc568067560 con 0x7fc568067180 2022-07-21T01:37:10.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56559a700 1 -- 172.21.15.3:0/867319238 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c003260 con 0x7fc568067180 2022-07-21T01:37:10.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56559a700 1 -- 172.21.15.3:0/867319238 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135110063 0 0) 0x7fc5680b0160 con 0x7fc5680b0870 2022-07-21T01:37:10.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56559a700 1 -- 172.21.15.3:0/867319238 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc568067560 con 0x7fc5680b0870 2022-07-21T01:37:10.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc56559a700 1 -- 172.21.15.3:0/867319238 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 103018191 0 0) 0x7fc54c003260 con 0x7fc568067180 2022-07-21T01:37:10.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc565d9b700 1 -- 172.21.15.3:0/867319238 >> v1:172.21.15.143:6789/0 conn(0x7fc5680b0870 legacy=0x7fc5680b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc565d9b700 1 -- 172.21.15.3:0/867319238 >> v1:172.21.15.3:6789/0 conn(0x7fc568067180 legacy=0x7fc568143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.625+0000 7fc565d9b700 1 -- 172.21.15.3:0/867319238 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5680af530 con 0x7fc5680ad1c0 2022-07-21T01:37:10.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc565d9b700 1 --2- 172.21.15.3:0/867319238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680ad1c0 0x7fc568144170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc55c001820 tx=0x7fc558006df0).ready entity=mon.1 client_cookie=531dbbed3dc43702 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56559a700 1 -- 172.21.15.3:0/867319238 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc55c001680 con 0x7fc5680ad1c0 2022-07-21T01:37:10.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56559a700 1 -- 172.21.15.3:0/867319238 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558000e90 con 0x7fc5680ad1c0 2022-07-21T01:37:10.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56559a700 1 -- 172.21.15.3:0/867319238 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc558012460 con 0x7fc5680ad1c0 2022-07-21T01:37:10.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56cbbd700 1 -- 172.21.15.3:0/867319238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680ad1c0 msgr2=0x7fc568144170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56cbbd700 1 --2- 172.21.15.3:0/867319238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680ad1c0 0x7fc568144170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc55c001820 tx=0x7fc558006df0).stop 2022-07-21T01:37:10.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56cbbd700 1 -- 172.21.15.3:0/867319238 shutdown_connections 2022-07-21T01:37:10.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56cbbd700 1 --2- 172.21.15.3:0/867319238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680ad1c0 0x7fc568144170 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56cbbd700 1 -- 172.21.15.3:0/867319238 >> 172.21.15.3:0/867319238 conn(0x7fc56805f840 msgr2=0x7fc56805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:10.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56cbbd700 1 -- 172.21.15.3:0/867319238 shutdown_connections 2022-07-21T01:37:10.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56cbbd700 1 -- 172.21.15.3:0/867319238 wait complete. 2022-07-21T01:37:10.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56cbbd700 1 Processor -- start 2022-07-21T01:37:10.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.629+0000 7fc56cbbd700 1 -- start start 2022-07-21T01:37:10.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56cbbd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc568067180 0x7fc568146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56cbbd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680b0870 0x7fc5680a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56cbbd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5680a2720 0x7fc5680ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56cbbd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680a12e0 con 0x7fc568067180 2022-07-21T01:37:10.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56cbbd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680a3190 con 0x7fc5680b0870 2022-07-21T01:37:10.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56cbbd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680a2bd0 con 0x7fc5680a2720 2022-07-21T01:37:10.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc568067180 0x7fc568146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc565d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680b0870 0x7fc5680a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc565d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680b0870 0x7fc5680a2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:51114/0 (socket says 172.21.15.3:51114) 2022-07-21T01:37:10.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc565d9b700 1 -- 172.21.15.3:0/3214598551 learned_addr learned my addr 172.21.15.3:0/3214598551 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:10.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc568067180 0x7fc568146e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45668/0 (socket says 172.21.15.3:45668) 2022-07-21T01:37:10.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc566d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5680a2720 0x7fc5680ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56659c700 1 -- 172.21.15.3:0/3214598551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5680a2720 msgr2=0x7fc5680ab4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56659c700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5680a2720 0x7fc5680ab4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56659c700 1 -- 172.21.15.3:0/3214598551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680b0870 msgr2=0x7fc5680a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56659c700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680b0870 0x7fc5680a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56659c700 1 -- 172.21.15.3:0/3214598551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55800b180 con 0x7fc568067180 2022-07-21T01:37:10.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56659c700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc568067180 0x7fc568146e90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7fc55c001860 tx=0x7fc55c001720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc5577fe700 1 -- 172.21.15.3:0/3214598551 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc55c007170 con 0x7fc568067180 2022-07-21T01:37:10.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5680a0cf0 con 0x7fc568067180 2022-07-21T01:37:10.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc5577fe700 1 -- 172.21.15.3:0/3214598551 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c0072d0 con 0x7fc568067180 2022-07-21T01:37:10.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc5577fe700 1 -- 172.21.15.3:0/3214598551 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc55c004710 con 0x7fc568067180 2022-07-21T01:37:10.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc568147380 con 0x7fc568067180 2022-07-21T01:37:10.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.633+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5680574e0 con 0x7fc568067180 2022-07-21T01:37:10.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.637+0000 7fc5577fe700 1 -- 172.21.15.3:0/3214598551 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc55c01b070 con 0x7fc568067180 2022-07-21T01:37:10.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.637+0000 7fc5577fe700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc544053a70 0x7fc544055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.637+0000 7fc5577fe700 1 -- 172.21.15.3:0/3214598551 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc55c06b590 con 0x7fc568067180 2022-07-21T01:37:10.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.637+0000 7fc565d9b700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc544053a70 0x7fc544055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.637+0000 7fc565d9b700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc544053a70 0x7fc544055f30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc55801a000 tx=0x7fc558014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.641+0000 7fc5577fe700 1 -- 172.21.15.3:0/3214598551 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc55c042250 con 0x7fc568067180 2022-07-21T01:37:10.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.805+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5680679a0 con 0x7fc568067180 2022-07-21T01:37:10.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc5577fe700 1 -- 172.21.15.3:0/3214598551 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2483 (secure 0 0 0) 0x7fc55c045870 con 0x7fc568067180 2022-07-21T01:37:10.809 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:37:10.809 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:36:52.675976+0000","modified":"2022-07-21T01:36:53.679085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/3381448192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3381448192},{"type":"v1","addr":"172.21.15.3:6837","nonce":3381448192}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-21T01:37:10.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc544053a70 msgr2=0x7fc544055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc544053a70 0x7fc544055f30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc55801a000 tx=0x7fc558014000).stop 2022-07-21T01:37:10.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc568067180 msgr2=0x7fc568146e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc568067180 0x7fc568146e90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7fc55c001860 tx=0x7fc55c001720).stop 2022-07-21T01:37:10.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 shutdown_connections 2022-07-21T01:37:10.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5680a2720 0x7fc5680ab4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc544053a70 0x7fc544055f30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5680b0870 0x7fc5680a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 --2- 172.21.15.3:0/3214598551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc568067180 0x7fc568146e90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 >> 172.21.15.3:0/3214598551 conn(0x7fc56805f840 msgr2=0x7fc568147dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:10.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 shutdown_connections 2022-07-21T01:37:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.809+0000 7fc56cbbd700 1 -- 172.21.15.3:0/3214598551 wait complete. 2022-07-21T01:37:10.817 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2022-07-21T01:37:10.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 5 2022-07-21T01:37:10.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.937+0000 7f15af59e700 1 Processor -- start 2022-07-21T01:37:10.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15af59e700 1 -- start start 2022-07-21T01:37:10.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15af59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b00b07a0 0x7f15b00af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15af59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b00b5a10 con 0x7f15b0147730 2022-07-21T01:37:10.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15af59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b00ac990 con 0x7f15b00b5630 2022-07-21T01:37:10.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15af59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b0145470 con 0x7f15b00b07a0 2022-07-21T01:37:10.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15aed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f15b0147730 0x7f15b00acdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53958/0 (socket says 172.21.15.3:53958) 2022-07-21T01:37:10.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15aed9d700 1 -- 172.21.15.3:0/995699623 learned_addr learned my addr 172.21.15.3:0/995699623 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:37:10.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15add9b700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b00b07a0 0x7f15b00af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766757829 0 0) 0x7f15b00b5a10 con 0x7f15b0147730 2022-07-21T01:37:10.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1590003260 con 0x7f15b0147730 2022-07-21T01:37:10.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020983633 0 0) 0x7f15b00ac990 con 0x7f15b00b5630 2022-07-21T01:37:10.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b00b5a10 con 0x7f15b00b5630 2022-07-21T01:37:10.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866190352 0 0) 0x7f1590003260 con 0x7f15b0147730 2022-07-21T01:37:10.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b00b07a0 msgr2=0x7f15b00af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:10.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b00b07a0 0x7f15b00af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 >> v1:172.21.15.143:6789/0 conn(0x7f15b00b5630 legacy=0x7f15b0067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b00b1bc0 con 0x7f15b0147730 2022-07-21T01:37:10.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f15a80019e0 con 0x7f15b0147730 2022-07-21T01:37:10.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 >> v1:172.21.15.3:6789/0 conn(0x7f15b0147730 legacy=0x7f15b00acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b00b07a0 0x7f1590003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1590003e20 0x7f1590004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15900049f0 0x7f1590004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b0145470 con 0x7f15900049f0 2022-07-21T01:37:10.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159000b800 con 0x7f1590003e20 2022-07-21T01:37:10.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159000b9c0 con 0x7f15b00b07a0 2022-07-21T01:37:10.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15aed9d700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15900049f0 0x7f1590004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15add9b700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1590003e20 0x7f1590004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ae59c700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b00b07a0 0x7f1590003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15aed9d700 1 -- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b00b07a0 msgr2=0x7f1590003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15aed9d700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b00b07a0 0x7f1590003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15aed9d700 1 -- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1590003e20 msgr2=0x7f1590004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15aed9d700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1590003e20 0x7f1590004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15aed9d700 1 -- 172.21.15.3:0/995699623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1590003260 con 0x7f15900049f0 2022-07-21T01:37:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15aed9d700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15900049f0 0x7f1590004e70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f15a80017b0 tx=0x7f15a8002130).ready entity=mon.0 client_cookie=193d44a625d7672c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15a800cb80 con 0x7f15900049f0 2022-07-21T01:37:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a800cce0 con 0x7f15900049f0 2022-07-21T01:37:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.941+0000 7f15ad59a700 1 -- 172.21.15.3:0/995699623 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15a8005b80 con 0x7f15900049f0 2022-07-21T01:37:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- 172.21.15.3:0/995699623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15900049f0 msgr2=0x7f1590004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 --2- 172.21.15.3:0/995699623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15900049f0 0x7f1590004e70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f15a80017b0 tx=0x7f15a8002130).stop 2022-07-21T01:37:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15aed9d700 1 -- 172.21.15.3:0/995699623 reap_dead start 2022-07-21T01:37:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- 172.21.15.3:0/995699623 shutdown_connections 2022-07-21T01:37:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- 172.21.15.3:0/995699623 >> 172.21.15.3:0/995699623 conn(0x7f15b00626f0 msgr2=0x7f15b0062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- 172.21.15.3:0/995699623 shutdown_connections 2022-07-21T01:37:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- 172.21.15.3:0/995699623 wait complete. 2022-07-21T01:37:10.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 Processor -- start 2022-07-21T01:37:10.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- start start 2022-07-21T01:37:10.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b0067730 0x7f15b0067b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15b0063980 0x7f15b0154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b0154780 0x7f15b0064d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b00ae180 con 0x7f15b0063980 2022-07-21T01:37:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b0154c00 con 0x7f15b0154780 2022-07-21T01:37:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b0066b50 con 0x7f15b0067730 2022-07-21T01:37:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15add9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15b0063980 0x7f15b0154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15add9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15b0063980 0x7f15b0154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45688/0 (socket says 172.21.15.3:45688) 2022-07-21T01:37:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15add9b700 1 -- 172.21.15.3:0/4059463289 learned_addr learned my addr 172.21.15.3:0/4059463289 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b0067730 0x7f15b0067b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15aed9d700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b0154780 0x7f15b0064d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15add9b700 1 -- 172.21.15.3:0/4059463289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b0067730 msgr2=0x7f15b0067b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15add9b700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b0067730 0x7f15b0067b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15add9b700 1 -- 172.21.15.3:0/4059463289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b0154780 msgr2=0x7f15b0064d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15add9b700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b0154780 0x7f15b0064d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15add9b700 1 -- 172.21.15.3:0/4059463289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b0063f90 con 0x7f15b0063980 2022-07-21T01:37:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15aed9d700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b0154780 0x7f15b0064d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:37:10.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15add9b700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15b0063980 0x7f15b0154290 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f15b0064540 tx=0x7f15a0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15a77fe700 1 -- 172.21.15.3:0/4059463289 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15a000f8e0 con 0x7f15b0063980 2022-07-21T01:37:10.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15a77fe700 1 -- 172.21.15.3:0/4059463289 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a0000e90 con 0x7f15b0063980 2022-07-21T01:37:10.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15a77fe700 1 -- 172.21.15.3:0/4059463289 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15a000fc30 con 0x7f15b0063980 2022-07-21T01:37:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- 172.21.15.3:0/4059463289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b0157840 con 0x7f15b0063980 2022-07-21T01:37:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.945+0000 7f15af59e700 1 -- 172.21.15.3:0/4059463289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b0157d50 con 0x7f15b0063980 2022-07-21T01:37:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.949+0000 7f15a77fe700 1 -- 172.21.15.3:0/4059463289 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f15a000fd90 con 0x7f15b0063980 2022-07-21T01:37:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.949+0000 7f15a77fe700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1594053aa0 0x7f1594055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.949+0000 7f15a77fe700 1 -- 172.21.15.3:0/4059463289 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f15a006c8b0 con 0x7f15b0063980 2022-07-21T01:37:10.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.949+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1594053aa0 0x7f1594055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.949+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1594053aa0 0x7f1594055f60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f1598003000 tx=0x7f1598007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15beb5f740 1 Processor -- start 2022-07-21T01:37:10.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15beb5f740 1 -- start start 2022-07-21T01:37:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21aa6f0 0x21aaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e2a90 0x21e2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21ae010 0x21ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15beb5f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x211d210 con 0x21aa6f0 2022-07-21T01:37:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15beb5f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x20fd450 con 0x21ae010 2022-07-21T01:37:10.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15beb5f740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x21aa390 con 0x21e2a90 2022-07-21T01:37:10.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21aa6f0 0x21aaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21aa6f0 0x21aaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45694/0 (socket says 172.21.15.3:45694) 2022-07-21T01:37:10.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15ae59c700 1 -- 172.21.15.3:0/3385214245 learned_addr learned my addr 172.21.15.3:0/3385214245 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:10.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15add9b700 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e2a90 0x21e2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15ae59c700 1 -- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e2a90 msgr2=0x21e2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:10.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15ae59c700 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e2a90 0x21e2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15ae59c700 1 -- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21ae010 msgr2=0x21ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:10.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15aed9d700 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21ae010 0x21ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15ae59c700 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21ae010 0x21ae490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15ae59c700 1 -- 172.21.15.3:0/3385214245 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2154ad0 con 0x21aa6f0 2022-07-21T01:37:10.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15add9b700 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e2a90 0x21e2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:37:10.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15aed9d700 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21ae010 0x21ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:37:10.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15ae59c700 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21aa6f0 0x21aaad0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f1598014000 tx=0x7f1598016000).ready entity=mon.0 client_cookie=42cadd179dc83a67 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15af59e700 1 -- 172.21.15.3:0/3385214245 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15980158e0 con 0x21aa6f0 2022-07-21T01:37:10.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15af59e700 1 -- 172.21.15.3:0/3385214245 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1598015a40 con 0x21aa6f0 2022-07-21T01:37:10.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.953+0000 7f15af59e700 1 -- 172.21.15.3:0/3385214245 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1598015da0 con 0x21aa6f0 2022-07-21T01:37:10.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- 172.21.15.3:0/3385214245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21aa6f0 msgr2=0x21aaad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21aa6f0 0x21aaad0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f1598014000 tx=0x7f1598016000).stop 2022-07-21T01:37:10.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- 172.21.15.3:0/3385214245 shutdown_connections 2022-07-21T01:37:10.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21ae010 0x21ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e2a90 0x21e2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 --2- 172.21.15.3:0/3385214245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21aa6f0 0x21aaad0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- 172.21.15.3:0/3385214245 >> 172.21.15.3:0/3385214245 conn(0x21348f0 msgr2=0x21ab470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:10.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- 172.21.15.3:0/3385214245 shutdown_connections 2022-07-21T01:37:10.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- 172.21.15.3:0/3385214245 wait complete. 2022-07-21T01:37:10.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- start start 2022-07-21T01:37:10.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 Processor -- start 2022-07-21T01:37:10.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 0x21e2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 0x21f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21e4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x21f44a0 con 0x21e2a90 2022-07-21T01:37:10.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x21e70b0 con 0x21aa6f0 2022-07-21T01:37:10.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x21f4700 con 0x21ae010 2022-07-21T01:37:10.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 10 client.0 ll_register_callbacks cb 0x21640b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-21T01:37:10.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15aed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21e4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15aed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21e4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45704/0 (socket says 172.21.15.3:45704) 2022-07-21T01:37:10.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15aed9d700 1 -- 172.21.15.3:0/4205548909 learned_addr learned my addr 172.21.15.3:0/4205548909 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:10.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 0x21e2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15add9b700 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 0x21f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15aed9d700 1 -- 172.21.15.3:0/4205548909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 msgr2=0x21f4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:10.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15aed9d700 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 0x21f4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15aed9d700 1 -- 172.21.15.3:0/4205548909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 msgr2=0x21e2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:10.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15aed9d700 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 0x21e2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15aed9d700 1 -- 172.21.15.3:0/4205548909 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b00b5a10 con 0x21e2a90 2022-07-21T01:37:10.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15aed9d700 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21e4d60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f15b0068920 tx=0x7f15a8015000).ready entity=mon.0 client_cookie=6b9506745b63b52 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:10.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f158d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2022-07-21T01:37:10.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f158d7fa700 1 -- 172.21.15.3:0/4205548909 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15a800cb80 con 0x21e2a90 2022-07-21T01:37:10.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f15beb5f740 1 -- 172.21.15.3:0/4205548909 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x21131c0 con 0x21e2a90 2022-07-21T01:37:10.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.957+0000 7f158d7fa700 1 -- 172.21.15.3:0/4205548909 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f15a80011c0 con 0x21e2a90 2022-07-21T01:37:10.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f158d7fa700 1 -- 172.21.15.3:0/4205548909 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15a8005e70 con 0x21e2a90 2022-07-21T01:37:10.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 10 client.5778 fetch_fsmap learned FSMap version 23 2022-07-21T01:37:10.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 -- 172.21.15.3:0/4205548909 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x210c7b0 con 0x21e2a90 2022-07-21T01:37:10.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f158d7fa700 1 -- 172.21.15.3:0/4205548909 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a8026020 con 0x21e2a90 2022-07-21T01:37:10.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f158d7fa700 1 -- 172.21.15.3:0/4205548909 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 1434+0+0 (secure 0 0 0) 0x7f15a8013450 con 0x21e2a90 2022-07-21T01:37:10.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 10 client.5778 fetch_fsmap finished waiting for FSMap version 23 2022-07-21T01:37:10.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 10 client.5778 resolve_mds: resolved ID 'a' to GID 5586 2022-07-21T01:37:10.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 20 client.5778 populate_metadata read hostname 'smithi003' 2022-07-21T01:37:10.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] conn(0x21e6c10 0x21dd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 4 client.5778 mds_command: new command op to 5586 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-21T01:37:10.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 -- 172.21.15.3:0/4205548909 --> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x210b240 con 0x21e6c10 2022-07-21T01:37:10.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] conn(0x21e6c10 0x21dd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f158d7fa700 10 client.5778 ms_handle_connect on v2:172.21.15.3:6836/3381448192 2022-07-21T01:37:10.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] conn(0x21e6c10 0x21dd980 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=f5fd66ee46ef9ae7 server_cookie=ddfad3de84ef8e00 in_seq=0 out_seq=0 2022-07-21T01:37:10.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f158d7fa700 1 -- 172.21.15.3:0/4205548909 <== mds.0 v2:172.21.15.3:6836/3381448192 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x210b240 con 0x21e6c10 2022-07-21T01:37:10.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f158d7fa700 10 client.5778 handle_command_reply: tid=0 2022-07-21T01:37:10.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 client.5778 shutdown 2022-07-21T01:37:10.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 -- 172.21.15.3:0/4205548909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 msgr2=0x21e4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:10.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21e4d60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f15b0068920 tx=0x7f15a8015000).stop 2022-07-21T01:37:10.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 -- 172.21.15.3:0/4205548909 shutdown_connections 2022-07-21T01:37:10.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] conn(0x21e6c10 0x21dd980 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21e4d60 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 0x21f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4205548909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 0x21e2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:10.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f158d7fa700 0 client.5778 ms_handle_reset on v2:172.21.15.3:6836/3381448192 2022-07-21T01:37:10.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 -- 172.21.15.3:0/4205548909 >> 172.21.15.3:0/4205548909 conn(0x21348f0 msgr2=0x21aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:10.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 -- 172.21.15.3:0/4205548909 shutdown_connections 2022-07-21T01:37:10.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 1 -- 172.21.15.3:0/4205548909 wait complete. 2022-07-21T01:37:10.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.961+0000 7f15beb5f740 20 client.5778 trim_cache size 0 max 16384 2022-07-21T01:37:10.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15beb5f740 1 Processor -- start 2022-07-21T01:37:10.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15beb5f740 1 -- start start 2022-07-21T01:37:10.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 0x21dd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 0x21a70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21a75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:10.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15beb5f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x217ce60 con 0x21e2a90 2022-07-21T01:37:10.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15beb5f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x21f1f30 con 0x21aa6f0 2022-07-21T01:37:10.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15beb5f740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x212ef10 con 0x21ae010 2022-07-21T01:37:10.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15aed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21a75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.977+0000 7f15aed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21a75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45712/0 (socket says 172.21.15.3:45712) 2022-07-21T01:37:10.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15aed9d700 1 -- 172.21.15.3:0/3838743745 learned_addr learned my addr 172.21.15.3:0/3838743745 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:10.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15add9b700 1 --2- 172.21.15.3:0/3838743745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 0x21a70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15ae59c700 1 --2- 172.21.15.3:0/3838743745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 0x21dd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:10.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15aed9d700 1 -- 172.21.15.3:0/3838743745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 msgr2=0x21a70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:11.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15aed9d700 1 --2- 172.21.15.3:0/3838743745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 0x21a70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:11.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15aed9d700 1 -- 172.21.15.3:0/3838743745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 msgr2=0x21dd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:11.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15aed9d700 1 --2- 172.21.15.3:0/3838743745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 0x21dd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:11.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15aed9d700 1 -- 172.21.15.3:0/3838743745 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x21153a0 con 0x21e2a90 2022-07-21T01:37:11.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15aed9d700 1 --2- 172.21.15.3:0/3838743745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21a75b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x21f4d90 tx=0x7f15a801d000).ready entity=mon.0 client_cookie=524d9a058040f2d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:11.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f158f7fe700 1 -- 172.21.15.3:0/3838743745 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15a8005ac0 con 0x21e2a90 2022-07-21T01:37:11.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f158f7fe700 1 -- 172.21.15.3:0/3838743745 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a8026030 con 0x21e2a90 2022-07-21T01:37:11.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f158f7fe700 1 -- 172.21.15.3:0/3838743745 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15a8001cf0 con 0x21e2a90 2022-07-21T01:37:11.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 -- 172.21.15.3:0/3838743745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 msgr2=0x21a75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:11.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 --2- 172.21.15.3:0/3838743745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21a75b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x21f4d90 tx=0x7f15a801d000).stop 2022-07-21T01:37:11.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 -- 172.21.15.3:0/3838743745 shutdown_connections 2022-07-21T01:37:11.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 --2- 172.21.15.3:0/3838743745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21e2a90 0x21a75b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:11.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 --2- 172.21.15.3:0/3838743745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21ae010 0x21a70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:11.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 --2- 172.21.15.3:0/3838743745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21aa6f0 0x21dd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:11.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 -- 172.21.15.3:0/3838743745 >> 172.21.15.3:0/3838743745 conn(0x21348f0 msgr2=0x21a7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:11.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 -- 172.21.15.3:0/3838743745 shutdown_connections 2022-07-21T01:37:11.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 -- 172.21.15.3:0/3838743745 wait complete. 2022-07-21T01:37:11.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 -- start start 2022-07-21T01:37:11.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 Processor -- start 2022-07-21T01:37:11.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21aa6f0 0x2241630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:11.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21ae010 0x2241b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:11.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e2a90 0x2244670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:11.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x21b27e0 con 0x21ae010 2022-07-21T01:37:11.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x21df670 con 0x21e2a90 2022-07-21T01:37:11.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x21ded00 con 0x21aa6f0 2022-07-21T01:37:11.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15beb5f740 10 client.0 ll_register_callbacks cb 0x21e7010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-21T01:37:11.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15add9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21ae010 0x2241b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:11.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15add9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21ae010 0x2241b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45716/0 (socket says 172.21.15.3:45716) 2022-07-21T01:37:11.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15add9b700 1 -- 172.21.15.3:0/4194540079 learned_addr learned my addr 172.21.15.3:0/4194540079 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:11.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.981+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21aa6f0 0x2241630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:11.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15aed9d700 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e2a90 0x2244670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:11.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15ae59c700 1 -- 172.21.15.3:0/4194540079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e2a90 msgr2=0x2244670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:11.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e2a90 0x2244670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:11.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15ae59c700 1 -- 172.21.15.3:0/4194540079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21ae010 msgr2=0x2241b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:11.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21ae010 0x2241b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:11.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15ae59c700 1 -- 172.21.15.3:0/4194540079 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x210c7b0 con 0x21aa6f0 2022-07-21T01:37:11.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15add9b700 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21ae010 0x2241b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:37:11.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15ae59c700 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21aa6f0 0x2241630 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f159802d000 tx=0x7f1598034280).ready entity=mon.2 client_cookie=ffee58e6f267c06e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:11.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f158e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.143:3301/0 2022-07-21T01:37:11.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f158e7fc700 1 -- 172.21.15.3:0/4194540079 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1598033070 con 0x21aa6f0 2022-07-21T01:37:11.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f158e7fc700 1 -- 172.21.15.3:0/4194540079 <== mon.2 v2:172.21.15.143:3301/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f159801a030 con 0x21aa6f0 2022-07-21T01:37:11.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15beb5f740 1 -- 172.21.15.3:0/4194540079 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1598015a40 con 0x21aa6f0 2022-07-21T01:37:11.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f158e7fc700 1 -- 172.21.15.3:0/4194540079 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1598015dc0 con 0x21aa6f0 2022-07-21T01:37:11.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15beb5f740 10 client.5285 fetch_fsmap learned FSMap version 23 2022-07-21T01:37:11.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15beb5f740 1 -- 172.21.15.3:0/4194540079 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x2111d90 con 0x21aa6f0 2022-07-21T01:37:11.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f158e7fc700 1 -- 172.21.15.3:0/4194540079 <== mon.2 v2:172.21.15.143:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1598015a40 con 0x21aa6f0 2022-07-21T01:37:11.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f158e7fc700 1 -- 172.21.15.3:0/4194540079 <== mon.2 v2:172.21.15.143:3301/0 5 ==== fsmap(e 23) v1 ==== 1434+0+0 (secure 0 0 0) 0x7f1598028070 con 0x21aa6f0 2022-07-21T01:37:11.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15beb5f740 10 client.5285 fetch_fsmap finished waiting for FSMap version 23 2022-07-21T01:37:11.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15beb5f740 10 client.5285 resolve_mds: resolved ID 'a' to GID 5586 2022-07-21T01:37:11.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15beb5f740 20 client.5285 populate_metadata read hostname 'smithi003' 2022-07-21T01:37:11.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] conn(0x2244b60 0x2247be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:11.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15beb5f740 4 client.5285 mds_command: new command op to 5586 tid=0[{"prefix": "cache drop", "timeout": 5}] 2022-07-21T01:37:11.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15beb5f740 1 -- 172.21.15.3:0/4194540079 --> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] -- command(tid 0: {"prefix": "cache drop", "timeout": 5}) v1 -- 0x20005e0 con 0x2244b60 2022-07-21T01:37:11.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15add9b700 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] conn(0x2244b60 0x2247be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:11.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f158e7fc700 10 client.5285 ms_handle_connect on v2:172.21.15.3:6836/3381448192 2022-07-21T01:37:11.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:10.985+0000 7f15add9b700 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] conn(0x2244b60 0x2247be0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=36d3572cfeb3f58b server_cookie=1f559d1ede26a9c1 in_seq=0 out_seq=0 2022-07-21T01:37:16.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f158e7fc700 1 -- 172.21.15.3:0/4194540079 <== mds.0 v2:172.21.15.3:6836/3381448192 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+404 (crc 0 0 0) 0x20005e0 con 0x2244b60 2022-07-21T01:37:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f158e7fc700 10 client.5285 handle_command_reply: tid=0 2022-07-21T01:37:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 client.5285 shutdown 2022-07-21T01:37:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 -- 172.21.15.3:0/4194540079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21aa6f0 msgr2=0x2241630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21aa6f0 0x2241630 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f159802d000 tx=0x7f1598034280).stop 2022-07-21T01:37:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 -- 172.21.15.3:0/4194540079 shutdown_connections 2022-07-21T01:37:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.3:6836/3381448192,v1:172.21.15.3:6837/3381448192] conn(0x2244b60 0x2247be0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e2a90 0x2244670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x21ae010 0x2241b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f158e7fc700 0 client.5285 ms_handle_reset on v2:172.21.15.3:6836/3381448192 2022-07-21T01:37:16.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 --2- 172.21.15.3:0/4194540079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21aa6f0 0x2241630 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 -- 172.21.15.3:0/4194540079 >> 172.21.15.3:0/4194540079 conn(0x21348f0 msgr2=0x21a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:16.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 -- 172.21.15.3:0/4194540079 shutdown_connections 2022-07-21T01:37:16.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.173+0000 7f15beb5f740 1 -- 172.21.15.3:0/4194540079 wait complete. 2022-07-21T01:37:16.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15beb5f740 20 client.5285 trim_cache size 0 max 16384 2022-07-21T01:37:16.181 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:37:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "client_recall": { 2022-07-21T01:37:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": -110, 2022-07-21T01:37:16.182 INFO:teuthology.orchestra.run.smithi003.stdout: "message": "(110) Connection timed out", 2022-07-21T01:37:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "recalled": 120 2022-07-21T01:37:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:37:16.183 INFO:teuthology.orchestra.run.smithi003.stdout: "flush_journal": { 2022-07-21T01:37:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2022-07-21T01:37:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "message": "" 2022-07-21T01:37:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:37:16.184 INFO:teuthology.orchestra.run.smithi003.stdout: "trim_cache": { 2022-07-21T01:37:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "trimmed": 0 2022-07-21T01:37:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:37:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2022-07-21T01:37:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": { 2022-07-21T01:37:16.185 INFO:teuthology.orchestra.run.smithi003.stdout: "items": 19972, 2022-07-21T01:37:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes": 3098528 2022-07-21T01:37:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:37:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:37:16.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1879876559999998 2022-07-21T01:37:16.187 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:37:16.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 -- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1594053aa0 msgr2=0x7f1594055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1594053aa0 0x7f1594055f60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f1598003000 tx=0x7f1598007a00).stop 2022-07-21T01:37:16.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 -- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15b0063980 msgr2=0x7f15b0154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15b0063980 0x7f15b0154290 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f15b0064540 tx=0x7f15a0006df0).stop 2022-07-21T01:37:16.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 -- 172.21.15.3:0/4059463289 shutdown_connections 2022-07-21T01:37:16.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1594053aa0 0x7f1594055f60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b0154780 0x7f15b0064d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15b0063980 0x7f15b0154290 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 --2- 172.21.15.3:0/4059463289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b0067730 0x7f15b0067b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 -- 172.21.15.3:0/4059463289 >> 172.21.15.3:0/4059463289 conn(0x7f15b00626f0 msgr2=0x7f15b00b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:16.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 -- 172.21.15.3:0/4059463289 shutdown_connections 2022-07-21T01:37:16.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.177+0000 7f15af59e700 1 -- 172.21.15.3:0/4059463289 wait complete. 2022-07-21T01:37:16.192 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-21T01:37:16.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:16.194 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:37:16.194 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:37:16.202 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-21T01:37:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:37:16.203 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-21T01:37:16.204 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:37:16.204 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:37:16.205 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:37:16.226 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:16.226 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:37:16.226 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:37:16.275 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:37:16.277 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:37:16.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:37:16.278 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:37:16.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:37:16.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.401+0000 7fc20d0c6700 1 Processor -- start 2022-07-21T01:37:16.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.401+0000 7fc20d0c6700 1 -- start start 2022-07-21T01:37:16.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.401+0000 7fc20d0c6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2080b2390 0x7fc2080abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.401+0000 7fc20d0c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b0160 0x7fc2080ac480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.401+0000 7fc20d0c6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2080ac970 con 0x7fc208141830 2022-07-21T01:37:16.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.401+0000 7fc20d0c6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2081453e0 con 0x7fc2080b2390 2022-07-21T01:37:16.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.401+0000 7fc20d0c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc208145680 con 0x7fc2080b0160 2022-07-21T01:37:16.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.401+0000 7fc20659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b0160 0x7fc2080ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc206d9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2080b2390 0x7fc2080abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20759e700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fc208141830 0x7fc208141c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:56128/0 (socket says 172.21.15.3:56128) 2022-07-21T01:37:16.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b0160 0x7fc2080ac480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:52646/0 (socket says 172.21.15.3:52646) 2022-07-21T01:37:16.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20759e700 1 -- 172.21.15.3:0/3976038620 learned_addr learned my addr 172.21.15.3:0/3976038620 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:37:16.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20659c700 1 -- 172.21.15.3:0/3976038620 >> v2:172.21.15.143:3300/0 conn(0x7fc2080b2390 msgr2=0x7fc2080abf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:16.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20659c700 1 --2- 172.21.15.3:0/3976038620 >> v2:172.21.15.143:3300/0 conn(0x7fc2080b2390 0x7fc2080abf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20659c700 1 -- 172.21.15.3:0/3976038620 >> v1:172.21.15.143:6789/0 conn(0x7fc208141830 legacy=0x7fc208141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20659c700 1 -- 172.21.15.3:0/3976038620 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc208141390 con 0x7fc2080b0160 2022-07-21T01:37:16.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20659c700 1 --2- 172.21.15.3:0/3976038620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b0160 0x7fc2080ac480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fc1f8002f70 tx=0x7fc1f8006df0).ready entity=mon.2 client_cookie=cccb8111d5e2e7c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:16.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc205d9b700 1 -- 172.21.15.3:0/3976038620 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1f800d570 con 0x7fc2080b0160 2022-07-21T01:37:16.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc205d9b700 1 -- 172.21.15.3:0/3976038620 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f8000e90 con 0x7fc2080b0160 2022-07-21T01:37:16.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc205d9b700 1 -- 172.21.15.3:0/3976038620 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1f800d8c0 con 0x7fc2080b0160 2022-07-21T01:37:16.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20d0c6700 1 -- 172.21.15.3:0/3976038620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b0160 msgr2=0x7fc2080ac480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20d0c6700 1 --2- 172.21.15.3:0/3976038620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b0160 0x7fc2080ac480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fc1f8002f70 tx=0x7fc1f8006df0).stop 2022-07-21T01:37:16.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20d0c6700 1 -- 172.21.15.3:0/3976038620 shutdown_connections 2022-07-21T01:37:16.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20d0c6700 1 --2- 172.21.15.3:0/3976038620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b0160 0x7fc2080ac480 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20d0c6700 1 --2- 172.21.15.3:0/3976038620 >> v2:172.21.15.143:3300/0 conn(0x7fc2080b2390 0x7fc2080abf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20d0c6700 1 -- 172.21.15.3:0/3976038620 >> 172.21.15.3:0/3976038620 conn(0x7fc20805f840 msgr2=0x7fc20805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:16.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20d0c6700 1 -- 172.21.15.3:0/3976038620 shutdown_connections 2022-07-21T01:37:16.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.405+0000 7fc20d0c6700 1 -- 172.21.15.3:0/3976038620 wait complete. 2022-07-21T01:37:16.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20d0c6700 1 Processor -- start 2022-07-21T01:37:16.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20d0c6700 1 -- start start 2022-07-21T01:37:16.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20d0c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2080b0160 0x7fc208138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20d0c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b2390 0x7fc20813b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20d0c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc208141830 0x7fc20813be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20d0c6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20813d3a0 con 0x7fc2080b0160 2022-07-21T01:37:16.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20d0c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20813c370 con 0x7fc208141830 2022-07-21T01:37:16.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20d0c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc208143240 con 0x7fc2080b2390 2022-07-21T01:37:16.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc206d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2080b0160 0x7fc208138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc206d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2080b0160 0x7fc208138c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45734/0 (socket says 172.21.15.3:45734) 2022-07-21T01:37:16.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc206d9d700 1 -- 172.21.15.3:0/982970742 learned_addr learned my addr 172.21.15.3:0/982970742 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:16.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20659c700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b2390 0x7fc20813b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.409+0000 7fc20759e700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc208141830 0x7fc20813be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc206d9d700 1 -- 172.21.15.3:0/982970742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b2390 msgr2=0x7fc20813b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc206d9d700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b2390 0x7fc20813b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc206d9d700 1 -- 172.21.15.3:0/982970742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc208141830 msgr2=0x7fc20813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc206d9d700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc208141830 0x7fc20813be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc206d9d700 1 -- 172.21.15.3:0/982970742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f800b180 con 0x7fc2080b0160 2022-07-21T01:37:16.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc206d9d700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2080b0160 0x7fc208138c10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fc1fc00c000 tx=0x7fc1fc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:16.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc1f7fff700 1 -- 172.21.15.3:0/982970742 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1fc010780 con 0x7fc2080b0160 2022-07-21T01:37:16.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc1f7fff700 1 -- 172.21.15.3:0/982970742 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1fc0108e0 con 0x7fc2080b0160 2022-07-21T01:37:16.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc208143460 con 0x7fc2080b0160 2022-07-21T01:37:16.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc1f7fff700 1 -- 172.21.15.3:0/982970742 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1fc010c40 con 0x7fc2080b0160 2022-07-21T01:37:16.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc208143930 con 0x7fc2080b0160 2022-07-21T01:37:16.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc208143af0 con 0x7fc2080b0160 2022-07-21T01:37:16.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc1f7fff700 1 -- 172.21.15.3:0/982970742 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc1fc01c460 con 0x7fc2080b0160 2022-07-21T01:37:16.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc1f7fff700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1e4053a10 0x7fc1e4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc1f7fff700 1 -- 172.21.15.3:0/982970742 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc1fc06c490 con 0x7fc2080b0160 2022-07-21T01:37:16.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.413+0000 7fc20659c700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1e4053a10 0x7fc1e4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.417+0000 7fc20659c700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1e4053a10 0x7fc1e4055ed0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fc1f801a000 tx=0x7fc1f8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:16.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.421+0000 7fc1f7fff700 1 -- 172.21.15.3:0/982970742 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc1fc043150 con 0x7fc2080b0160 2022-07-21T01:37:16.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.545+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2080574e0 con 0x7fc2080b0160 2022-07-21T01:37:16.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.545+0000 7fc1f7fff700 1 -- 172.21.15.3:0/982970742 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16273 (secure 0 0 0) 0x7fc1fc046770 con 0x7fc2080b0160 2022-07-21T01:37:16.546 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:37:16.547 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":58,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:36:53.679840+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:36:51.181015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-21T01:36:52.366227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:37:16.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1e4053a10 msgr2=0x7fc1e4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1e4053a10 0x7fc1e4055ed0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fc1f801a000 tx=0x7fc1f8014000).stop 2022-07-21T01:37:16.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2080b0160 msgr2=0x7fc208138c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2080b0160 0x7fc208138c10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fc1fc00c000 tx=0x7fc1fc006df0).stop 2022-07-21T01:37:16.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 shutdown_connections 2022-07-21T01:37:16.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1e4053a10 0x7fc1e4055ed0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc208141830 0x7fc20813be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2080b2390 0x7fc20813b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 --2- 172.21.15.3:0/982970742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2080b0160 0x7fc208138c10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 >> 172.21.15.3:0/982970742 conn(0x7fc20805f840 msgr2=0x7fc20805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:16.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 shutdown_connections 2022-07-21T01:37:16.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.549+0000 7fc20d0c6700 1 -- 172.21.15.3:0/982970742 wait complete. 2022-07-21T01:37:16.565 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:36:51.181015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:36:52.366227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:37:16.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:37:16.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.677+0000 7f391188a700 1 Processor -- start 2022-07-21T01:37:16.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.677+0000 7f391188a700 1 -- start start 2022-07-21T01:37:16.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.677+0000 7f391188a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f390c064cd0 0x7f390c143a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f390c0b3ad0 0x7f390c143f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390c0b2400 con 0x7f390c0b2e80 2022-07-21T01:37:16.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390c144410 con 0x7f390c064cd0 2022-07-21T01:37:16.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390c0623c0 con 0x7f390c0b3ad0 2022-07-21T01:37:16.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f3910888700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f390c064cd0 0x7f390c143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f3910888700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f390c064cd0 0x7f390c143a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45742/0 (socket says 172.21.15.3:45742) 2022-07-21T01:37:16.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f3910888700 1 -- 172.21.15.3:0/2657002105 learned_addr learned my addr 172.21.15.3:0/2657002105 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:16.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f3911089700 1 --2- 172.21.15.3:0/2657002105 >> v2:172.21.15.143:3300/0 conn(0x7f390c0b3ad0 0x7f390c143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f3910888700 1 -- 172.21.15.3:0/2657002105 >> v2:172.21.15.143:3300/0 conn(0x7f390c0b3ad0 msgr2=0x7f390c143f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:16.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f3910888700 1 --2- 172.21.15.3:0/2657002105 >> v2:172.21.15.143:3300/0 conn(0x7f390c0b3ad0 0x7f390c143f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f3910888700 1 -- 172.21.15.3:0/2657002105 >> v1:172.21.15.143:6789/0 conn(0x7f390c0b2e80 legacy=0x7f390c0b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f3910888700 1 -- 172.21.15.3:0/2657002105 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c0650b0 con 0x7f390c064cd0 2022-07-21T01:37:16.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f3910888700 1 --2- 172.21.15.3:0/2657002105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c064cd0 0x7f390c143a30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f38f4002f70 tx=0x7f38f4006eb0).ready entity=mon.0 client_cookie=3037288529ae7a33 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:16.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f390b7fe700 1 -- 172.21.15.3:0/2657002105 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38f400d5c0 con 0x7f390c064cd0 2022-07-21T01:37:16.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f390b7fe700 1 -- 172.21.15.3:0/2657002105 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f4000e90 con 0x7f390c064cd0 2022-07-21T01:37:16.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f390b7fe700 1 -- 172.21.15.3:0/2657002105 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38f400d950 con 0x7f390c064cd0 2022-07-21T01:37:16.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 -- 172.21.15.3:0/2657002105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c064cd0 msgr2=0x7f390c143a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 --2- 172.21.15.3:0/2657002105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c064cd0 0x7f390c143a30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f38f4002f70 tx=0x7f38f4006eb0).stop 2022-07-21T01:37:16.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 -- 172.21.15.3:0/2657002105 shutdown_connections 2022-07-21T01:37:16.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 --2- 172.21.15.3:0/2657002105 >> v2:172.21.15.143:3300/0 conn(0x7f390c0b3ad0 0x7f390c143f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 --2- 172.21.15.3:0/2657002105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c064cd0 0x7f390c143a30 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 -- 172.21.15.3:0/2657002105 >> 172.21.15.3:0/2657002105 conn(0x7f390c062d70 msgr2=0x7f390c05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 -- 172.21.15.3:0/2657002105 shutdown_connections 2022-07-21T01:37:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.681+0000 7f391188a700 1 -- 172.21.15.3:0/2657002105 wait complete. 2022-07-21T01:37:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f391188a700 1 Processor -- start 2022-07-21T01:37:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f391188a700 1 -- start start 2022-07-21T01:37:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f391188a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f390c064cd0 0x7f390c0b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f391188a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c0b2e80 0x7f390c0adac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f391188a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f390c0b3ad0 0x7f390c0adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f391188a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c146350 con 0x7f390c0b2e80 2022-07-21T01:37:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f391188a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c145110 con 0x7f390c0b3ad0 2022-07-21T01:37:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f391188a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c1453b0 con 0x7f390c064cd0 2022-07-21T01:37:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f390bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c0b2e80 0x7f390c0adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f390bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c0b2e80 0x7f390c0adac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45750/0 (socket says 172.21.15.3:45750) 2022-07-21T01:37:16.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f390bfff700 1 -- 172.21.15.3:0/713328033 learned_addr learned my addr 172.21.15.3:0/713328033 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:16.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f3910888700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f390c064cd0 0x7f390c0b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f3911089700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f390c0b3ad0 0x7f390c0adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f390bfff700 1 -- 172.21.15.3:0/713328033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f390c064cd0 msgr2=0x7f390c0b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f390bfff700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f390c064cd0 0x7f390c0b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f390bfff700 1 -- 172.21.15.3:0/713328033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f390c0b3ad0 msgr2=0x7f390c0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f390bfff700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f390c0b3ad0 0x7f390c0adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f390bfff700 1 -- 172.21.15.3:0/713328033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f400b1e0 con 0x7f390c0b2e80 2022-07-21T01:37:16.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f390bfff700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c0b2e80 0x7f390c0adac0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f390c065640 tx=0x7f38fc007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:16.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f3909ffb700 1 -- 172.21.15.3:0/713328033 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38fc007020 con 0x7f390c0b2e80 2022-07-21T01:37:16.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f3909ffb700 1 -- 172.21.15.3:0/713328033 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc007180 con 0x7f390c0b2e80 2022-07-21T01:37:16.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f3909ffb700 1 -- 172.21.15.3:0/713328033 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38fc006560 con 0x7f390c0b2e80 2022-07-21T01:37:16.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.685+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c146c10 con 0x7f390c0b2e80 2022-07-21T01:37:16.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.689+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c146750 con 0x7f390c0b2e80 2022-07-21T01:37:16.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.689+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c0574e0 con 0x7f390c0b2e80 2022-07-21T01:37:16.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.689+0000 7f3909ffb700 1 -- 172.21.15.3:0/713328033 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f38fc010070 con 0x7f390c0b2e80 2022-07-21T01:37:16.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.689+0000 7f3909ffb700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38f0053a10 0x7f38f0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.689+0000 7f3910888700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38f0053a10 0x7f38f0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.689+0000 7f3909ffb700 1 -- 172.21.15.3:0/713328033 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f38fc06b260 con 0x7f390c0b2e80 2022-07-21T01:37:16.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.689+0000 7f3910888700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38f0053a10 0x7f38f0055ed0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f38f401c000 tx=0x7f38f4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:16.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.697+0000 7f3909ffb700 1 -- 172.21.15.3:0/713328033 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f38fc041fa0 con 0x7f390c0b2e80 2022-07-21T01:37:16.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.821+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f390c146520 con 0x7f390c0b2e80 2022-07-21T01:37:16.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.821+0000 7f3909ffb700 1 -- 172.21.15.3:0/713328033 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v58) v1 ==== 110+0+10 (secure 0 0 0) 0x7f38fc0455c0 con 0x7f390c0b2e80 2022-07-21T01:37:16.823 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:37:16.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38f0053a10 msgr2=0x7f38f0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38f0053a10 0x7f38f0055ed0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f38f401c000 tx=0x7f38f4015000).stop 2022-07-21T01:37:16.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c0b2e80 msgr2=0x7f390c0adac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c0b2e80 0x7f390c0adac0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f390c065640 tx=0x7f38fc007c40).stop 2022-07-21T01:37:16.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 shutdown_connections 2022-07-21T01:37:16.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38f0053a10 0x7f38f0055ed0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f390c0b3ad0 0x7f390c0adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f390c0b2e80 0x7f390c0adac0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 --2- 172.21.15.3:0/713328033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f390c064cd0 0x7f390c0b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 >> 172.21.15.3:0/713328033 conn(0x7f390c062d70 msgr2=0x7f390c0b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:16.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 shutdown_connections 2022-07-21T01:37:16.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.825+0000 7f391188a700 1 -- 172.21.15.3:0/713328033 wait complete. 2022-07-21T01:37:16.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:37:16.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f06554d4700 1 Processor -- start 2022-07-21T01:37:16.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f06554d4700 1 -- start start 2022-07-21T01:37:16.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f06554d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0650060560 0x7f06500aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f06554d4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f065013f920 0x7f065013fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f06554d4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0650061b20 con 0x7f06500aa050 2022-07-21T01:37:16.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f06554d4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06501402a0 con 0x7f065013f920 2022-07-21T01:37:16.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f06554d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0650061dc0 con 0x7f0650060560 2022-07-21T01:37:16.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064e7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0650060560 0x7f06500aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064e7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0650060560 0x7f06500aa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:52674/0 (socket says 172.21.15.3:52674) 2022-07-21T01:37:16.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064e7fc700 1 -- 172.21.15.3:0/2665836027 learned_addr learned my addr 172.21.15.3:0/2665836027 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:16.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064f7fe700 1 --2- 172.21.15.3:0/2665836027 >> v2:172.21.15.143:3300/0 conn(0x7f065013f920 0x7f065013fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064e7fc700 1 -- 172.21.15.3:0/2665836027 >> v2:172.21.15.143:3300/0 conn(0x7f065013f920 msgr2=0x7f065013fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:16.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064e7fc700 1 --2- 172.21.15.3:0/2665836027 >> v2:172.21.15.143:3300/0 conn(0x7f065013f920 0x7f065013fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064e7fc700 1 -- 172.21.15.3:0/2665836027 >> v1:172.21.15.143:6789/0 conn(0x7f06500aa050 legacy=0x7f065013ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064e7fc700 1 -- 172.21.15.3:0/2665836027 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0650060d20 con 0x7f0650060560 2022-07-21T01:37:16.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064e7fc700 1 --2- 172.21.15.3:0/2665836027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0650060560 0x7f06500aa430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0644002f70 tx=0x7f0644006eb0).ready entity=mon.2 client_cookie=dc5a1ad2cdc92c82 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:16.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064dffb700 1 -- 172.21.15.3:0/2665836027 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f064400d570 con 0x7f0650060560 2022-07-21T01:37:16.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064dffb700 1 -- 172.21.15.3:0/2665836027 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0644000e90 con 0x7f0650060560 2022-07-21T01:37:16.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f064dffb700 1 -- 172.21.15.3:0/2665836027 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f064400d900 con 0x7f0650060560 2022-07-21T01:37:16.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f06554d4700 1 -- 172.21.15.3:0/2665836027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0650060560 msgr2=0x7f06500aa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.949+0000 7f06554d4700 1 --2- 172.21.15.3:0/2665836027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0650060560 0x7f06500aa430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0644002f70 tx=0x7f0644006eb0).stop 2022-07-21T01:37:16.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 -- 172.21.15.3:0/2665836027 shutdown_connections 2022-07-21T01:37:16.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 --2- 172.21.15.3:0/2665836027 >> v2:172.21.15.143:3300/0 conn(0x7f065013f920 0x7f065013fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 --2- 172.21.15.3:0/2665836027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0650060560 0x7f06500aa430 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 -- 172.21.15.3:0/2665836027 >> 172.21.15.3:0/2665836027 conn(0x7f06500b2b00 msgr2=0x7f06500b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:16.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 -- 172.21.15.3:0/2665836027 shutdown_connections 2022-07-21T01:37:16.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 -- 172.21.15.3:0/2665836027 wait complete. 2022-07-21T01:37:16.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 Processor -- start 2022-07-21T01:37:16.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 -- start start 2022-07-21T01:37:16.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0650060560 0x7f06500b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06500aa050 0x7f0650134350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f065013f920 0x7f06500ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065005f130 con 0x7f0650060560 2022-07-21T01:37:16.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06501437b0 con 0x7f065013f920 2022-07-21T01:37:16.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f06554d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06500b5850 con 0x7f06500aa050 2022-07-21T01:37:16.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0650060560 0x7f06500b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064e7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06500aa050 0x7f0650134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f065013f920 0x7f06500ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064e7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06500aa050 0x7f0650134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:52680/0 (socket says 172.21.15.3:52680) 2022-07-21T01:37:16.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064e7fc700 1 -- 172.21.15.3:0/3793277461 learned_addr learned my addr 172.21.15.3:0/3793277461 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:16.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064effd700 1 -- 172.21.15.3:0/3793277461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06500aa050 msgr2=0x7f0650134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064effd700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06500aa050 0x7f0650134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064effd700 1 -- 172.21.15.3:0/3793277461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f065013f920 msgr2=0x7f06500ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:16.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064effd700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f065013f920 0x7f06500ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:16.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.953+0000 7f064effd700 1 -- 172.21.15.3:0/3793277461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064400b1e0 con 0x7f0650060560 2022-07-21T01:37:16.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f064effd700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0650060560 0x7f06500b4940 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f065005ff20 tx=0x7f0640007b90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:16.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f063ffff700 1 -- 172.21.15.3:0/3793277461 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0640012400 con 0x7f0650060560 2022-07-21T01:37:16.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f063ffff700 1 -- 172.21.15.3:0/3793277461 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0640012560 con 0x7f0650060560 2022-07-21T01:37:16.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06500b4070 con 0x7f0650060560 2022-07-21T01:37:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f063ffff700 1 -- 172.21.15.3:0/3793277461 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06400128c0 con 0x7f0650060560 2022-07-21T01:37:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06500b0120 con 0x7f0650060560 2022-07-21T01:37:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0650060fc0 con 0x7f0650060560 2022-07-21T01:37:16.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f063ffff700 1 -- 172.21.15.3:0/3793277461 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0640019030 con 0x7f0650060560 2022-07-21T01:37:16.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f063ffff700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f062c0539e0 0x7f062c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:16.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.957+0000 7f063ffff700 1 -- 172.21.15.3:0/3793277461 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f064006b2c0 con 0x7f0650060560 2022-07-21T01:37:16.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.961+0000 7f064e7fc700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f062c0539e0 0x7f062c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:16.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.961+0000 7f064e7fc700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f062c0539e0 0x7f062c055ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f064401c000 tx=0x7f0644015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:16.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:16.965+0000 7f063ffff700 1 -- 172.21.15.3:0/3793277461 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0640042080 con 0x7f0650060560 2022-07-21T01:37:17.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.089+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f06500574e0 con 0x7f0650060560 2022-07-21T01:37:17.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f063ffff700 1 -- 172.21.15.3:0/3793277461 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v58) v1 ==== 92+0+10 (secure 0 0 0) 0x7f06400456a0 con 0x7f0650060560 2022-07-21T01:37:17.093 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:37:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f062c0539e0 msgr2=0x7f062c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f062c0539e0 0x7f062c055ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f064401c000 tx=0x7f0644015000).stop 2022-07-21T01:37:17.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0650060560 msgr2=0x7f06500b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0650060560 0x7f06500b4940 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f065005ff20 tx=0x7f0640007b90).stop 2022-07-21T01:37:17.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 shutdown_connections 2022-07-21T01:37:17.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f062c0539e0 0x7f062c055ea0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f065013f920 0x7f06500ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06500aa050 0x7f0650134350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 --2- 172.21.15.3:0/3793277461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0650060560 0x7f06500b4940 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 >> 172.21.15.3:0/3793277461 conn(0x7f06500b2b00 msgr2=0x7f065013e300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:17.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 shutdown_connections 2022-07-21T01:37:17.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.093+0000 7f06554d4700 1 -- 172.21.15.3:0/3793277461 wait complete. 2022-07-21T01:37:17.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:37:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.213+0000 7f88c7c37700 1 Processor -- start 2022-07-21T01:37:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c7c37700 1 -- start start 2022-07-21T01:37:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c7c37700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88c00b4550 0x7f88c013e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c7c37700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c00b4180 con 0x7f88c00b0920 2022-07-21T01:37:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c7c37700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c00afcf0 con 0x7f88c00af910 2022-07-21T01:37:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c7c37700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c013eeb0 con 0x7f88c00b4550 2022-07-21T01:37:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c51d2700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f88c00b0920 0x7f88c013e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54052/0 (socket says 172.21.15.3:54052) 2022-07-21T01:37:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c51d2700 1 -- 172.21.15.3:0/3202831839 learned_addr learned my addr 172.21.15.3:0/3202831839 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:37:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c61d4700 1 --2- 172.21.15.3:0/3202831839 >> v2:172.21.15.143:3300/0 conn(0x7f88c00b4550 0x7f88c013e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11508385 0 0) 0x7f88c00b4180 con 0x7f88c00b0920 2022-07-21T01:37:17.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88ac003260 con 0x7f88c00b0920 2022-07-21T01:37:17.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799229301 0 0) 0x7f88c00afcf0 con 0x7f88c00af910 2022-07-21T01:37:17.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c00b4180 con 0x7f88c00af910 2022-07-21T01:37:17.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 323823702 0 0) 0x7f88ac003260 con 0x7f88c00b0920 2022-07-21T01:37:17.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 >> v2:172.21.15.143:3300/0 conn(0x7f88c00b4550 msgr2=0x7f88c013e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:17.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 --2- 172.21.15.3:0/3202831839 >> v2:172.21.15.143:3300/0 conn(0x7f88c00b4550 0x7f88c013e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 >> v1:172.21.15.143:6789/0 conn(0x7f88c00af910 legacy=0x7f88c00b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c00b3f20 con 0x7f88c00b0920 2022-07-21T01:37:17.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f88b4001660 con 0x7f88c00b0920 2022-07-21T01:37:17.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 >> v1:172.21.15.3:6789/0 conn(0x7f88c00b0920 legacy=0x7f88c013e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88ac0038e0 0x7f88ac003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac004500 0x7f88ac004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004e70 0x7f88ac00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c013eeb0 con 0x7f88ac004500 2022-07-21T01:37:17.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac00bd20 con 0x7f88ac0038e0 2022-07-21T01:37:17.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac00bee0 con 0x7f88ac004e70 2022-07-21T01:37:17.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c61d4700 1 --2- 172.21.15.3:0/3202831839 >> v2:172.21.15.143:3300/0 conn(0x7f88c00b4550 0x7f88c013e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:37:17.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c51d2700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac004500 0x7f88ac004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c59d3700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88ac0038e0 0x7f88ac003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c51d2700 1 -- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004e70 msgr2=0x7f88ac00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:17.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c61d4700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004e70 0x7f88ac00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c51d2700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004e70 0x7f88ac00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c51d2700 1 -- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88ac0038e0 msgr2=0x7f88ac003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:17.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c51d2700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88ac0038e0 0x7f88ac003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c51d2700 1 -- 172.21.15.3:0/3202831839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac003260 con 0x7f88ac004500 2022-07-21T01:37:17.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c59d3700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88ac0038e0 0x7f88ac003d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:37:17.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c61d4700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004e70 0x7f88ac00b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:37:17.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.217+0000 7f88c61d4700 1 -- 172.21.15.3:0/3202831839 reap_dead start 2022-07-21T01:37:17.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c51d2700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac004500 0x7f88ac004980 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f88b4001430 tx=0x7f88c00b1ed0).ready entity=mon.0 client_cookie=66e4a81f402227bf server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:17.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b40071a0 con 0x7f88ac004500 2022-07-21T01:37:17.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b4007300 con 0x7f88ac004500 2022-07-21T01:37:17.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c49d1700 1 -- 172.21.15.3:0/3202831839 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b400c8f0 con 0x7f88ac004500 2022-07-21T01:37:17.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c7c37700 1 -- 172.21.15.3:0/3202831839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac004500 msgr2=0x7f88ac004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c7c37700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac004500 0x7f88ac004980 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f88b4001430 tx=0x7f88c00b1ed0).stop 2022-07-21T01:37:17.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c7c37700 1 -- 172.21.15.3:0/3202831839 shutdown_connections 2022-07-21T01:37:17.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c7c37700 1 --2- 172.21.15.3:0/3202831839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ac004500 0x7f88ac004980 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c7c37700 1 -- 172.21.15.3:0/3202831839 >> 172.21.15.3:0/3202831839 conn(0x7f88c006a2a0 msgr2=0x7f88c005ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:17.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c7c37700 1 -- 172.21.15.3:0/3202831839 shutdown_connections 2022-07-21T01:37:17.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c7c37700 1 -- 172.21.15.3:0/3202831839 wait complete. 2022-07-21T01:37:17.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c7c37700 1 Processor -- start 2022-07-21T01:37:17.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.221+0000 7f88c7c37700 1 -- start start 2022-07-21T01:37:17.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c7c37700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004500 0x7f88c013f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c7c37700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88c0142540 0x7f88c01524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c7c37700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88c00aeea0 0x7f88c0140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c7c37700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c0149fd0 con 0x7f88c00aeea0 2022-07-21T01:37:17.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c7c37700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c014a270 con 0x7f88c0142540 2022-07-21T01:37:17.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c7c37700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c014a510 con 0x7f88ac004500 2022-07-21T01:37:17.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c59d3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004500 0x7f88c013f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c61d4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88c00aeea0 0x7f88c0140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c59d3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004500 0x7f88c013f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:52698/0 (socket says 172.21.15.3:52698) 2022-07-21T01:37:17.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c59d3700 1 -- 172.21.15.3:0/3338092896 learned_addr learned my addr 172.21.15.3:0/3338092896 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:17.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c51d2700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88c0142540 0x7f88c01524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c59d3700 1 -- 172.21.15.3:0/3338092896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88c0142540 msgr2=0x7f88c01524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c59d3700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88c0142540 0x7f88c01524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c59d3700 1 -- 172.21.15.3:0/3338092896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88c00aeea0 msgr2=0x7f88c0140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c59d3700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88c00aeea0 0x7f88c0140850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c59d3700 1 -- 172.21.15.3:0/3338092896 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c013f1f0 con 0x7f88ac004500 2022-07-21T01:37:17.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c59d3700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004500 0x7f88c013f870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f88b000b000 tx=0x7f88b0009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:17.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88beffd700 1 -- 172.21.15.3:0/3338092896 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b0007ea0 con 0x7f88ac004500 2022-07-21T01:37:17.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88beffd700 1 -- 172.21.15.3:0/3338092896 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b00017b0 con 0x7f88ac004500 2022-07-21T01:37:17.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c0140d40 con 0x7f88ac004500 2022-07-21T01:37:17.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88beffd700 1 -- 172.21.15.3:0/3338092896 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b0001b10 con 0x7f88ac004500 2022-07-21T01:37:17.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.225+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c0140ff0 con 0x7f88ac004500 2022-07-21T01:37:17.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.229+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c014a6d0 con 0x7f88ac004500 2022-07-21T01:37:17.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.229+0000 7f88beffd700 1 -- 172.21.15.3:0/3338092896 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f88b0001c70 con 0x7f88ac004500 2022-07-21T01:37:17.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.229+0000 7f88beffd700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88a0053aa0 0x7f88a0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.229+0000 7f88beffd700 1 -- 172.21.15.3:0/3338092896 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f88b006c3c0 con 0x7f88ac004500 2022-07-21T01:37:17.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.229+0000 7f88c51d2700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88a0053aa0 0x7f88a0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.229+0000 7f88c51d2700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88a0053aa0 0x7f88a0055f60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f88b4019000 tx=0x7f88b4013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:17.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.237+0000 7f88beffd700 1 -- 172.21.15.3:0/3338092896 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f88b0043070 con 0x7f88ac004500 2022-07-21T01:37:17.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.365+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88c00574e0 con 0x7f88ac004500 2022-07-21T01:37:17.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88beffd700 1 -- 172.21.15.3:0/3338092896 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+10 (secure 0 0 0) 0x7f88b0046690 con 0x7f88ac004500 2022-07-21T01:37:17.368 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:37:17.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88a0053aa0 msgr2=0x7f88a0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88a0053aa0 0x7f88a0055f60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f88b4019000 tx=0x7f88b4013000).stop 2022-07-21T01:37:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004500 msgr2=0x7f88c013f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004500 0x7f88c013f870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f88b000b000 tx=0x7f88b0009040).stop 2022-07-21T01:37:17.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 shutdown_connections 2022-07-21T01:37:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88c00aeea0 0x7f88c0140850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88a0053aa0 0x7f88a0055f60 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88c0142540 0x7f88c01524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 --2- 172.21.15.3:0/3338092896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88ac004500 0x7f88c013f870 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 >> 172.21.15.3:0/3338092896 conn(0x7f88c006a2a0 msgr2=0x7f88c0144ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 shutdown_connections 2022-07-21T01:37:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.369+0000 7f88c7c37700 1 -- 172.21.15.3:0/3338092896 wait complete. 2022-07-21T01:37:17.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:37:17.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e920700 1 Processor -- start 2022-07-21T01:37:17.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e920700 1 -- start start 2022-07-21T01:37:17.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e920700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6c180a8c50 0x7f6c18152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e920700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c180a7250 con 0x7f6c180b01a0 2022-07-21T01:37:17.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e920700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c180a7480 con 0x7f6c180b2710 2022-07-21T01:37:17.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e920700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c180a7720 con 0x7f6c180a8c50 2022-07-21T01:37:17.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e11f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6c180a8c50 0x7f6c18152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1d11d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6c180b01a0 0x7f6c180a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54070/0 (socket says 172.21.15.3:54070) 2022-07-21T01:37:17.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1d11d700 1 -- 172.21.15.3:0/1469173322 learned_addr learned my addr 172.21.15.3:0/1469173322 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:37:17.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e11f700 1 -- 172.21.15.3:0/1469173322 >> v1:172.21.15.143:6789/0 conn(0x7f6c180b2710 legacy=0x7f6c1814afd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2022-07-21T01:37:17.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e11f700 1 -- 172.21.15.3:0/1469173322 >> v1:172.21.15.3:6789/0 conn(0x7f6c180b01a0 legacy=0x7f6c180a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e11f700 1 -- 172.21.15.3:0/1469173322 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c180a79c0 con 0x7f6c180a8c50 2022-07-21T01:37:17.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1c91c700 1 -- 172.21.15.3:0/1469173322 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012474501 0 0) 0x7f6c180a7250 con 0x7f6c180b01a0 2022-07-21T01:37:17.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.493+0000 7f6c1e11f700 1 --2- 172.21.15.3:0/1469173322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180a8c50 0x7f6c18152d70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f6c18061c10 tx=0x7f6c100091d0).ready entity=mon.0 client_cookie=58b4b43b1c77aba4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:17.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1c91c700 1 -- 172.21.15.3:0/1469173322 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c10009ea0 con 0x7f6c180a8c50 2022-07-21T01:37:17.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1c91c700 1 -- 172.21.15.3:0/1469173322 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1000b860 con 0x7f6c180a8c50 2022-07-21T01:37:17.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1c91c700 1 -- 172.21.15.3:0/1469173322 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c1000b9d0 con 0x7f6c180a8c50 2022-07-21T01:37:17.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 -- 172.21.15.3:0/1469173322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180a8c50 msgr2=0x7f6c18152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 --2- 172.21.15.3:0/1469173322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180a8c50 0x7f6c18152d70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f6c18061c10 tx=0x7f6c100091d0).stop 2022-07-21T01:37:17.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 -- 172.21.15.3:0/1469173322 shutdown_connections 2022-07-21T01:37:17.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 --2- 172.21.15.3:0/1469173322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180a8c50 0x7f6c18152d70 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 -- 172.21.15.3:0/1469173322 >> 172.21.15.3:0/1469173322 conn(0x7f6c18060310 msgr2=0x7f6c180b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:17.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 -- 172.21.15.3:0/1469173322 shutdown_connections 2022-07-21T01:37:17.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 -- 172.21.15.3:0/1469173322 wait complete. 2022-07-21T01:37:17.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 Processor -- start 2022-07-21T01:37:17.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 -- start start 2022-07-21T01:37:17.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c180a8c50 0x7f6c180aa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180b01a0 0x7f6c181495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c180b2710 0x7f6c180ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c18149ac0 con 0x7f6c180b01a0 2022-07-21T01:37:17.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c18159d70 con 0x7f6c180b2710 2022-07-21T01:37:17.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.497+0000 7f6c1e920700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c180aafb0 con 0x7f6c180a8c50 2022-07-21T01:37:17.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d11d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180b01a0 0x7f6c181495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d11d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180b01a0 0x7f6c181495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45798/0 (socket says 172.21.15.3:45798) 2022-07-21T01:37:17.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d11d700 1 -- 172.21.15.3:0/2235358780 learned_addr learned my addr 172.21.15.3:0/2235358780 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:17.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d91e700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c180a8c50 0x7f6c180aa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1e11f700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c180b2710 0x7f6c180ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d11d700 1 -- 172.21.15.3:0/2235358780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c180a8c50 msgr2=0x7f6c180aa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d11d700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c180a8c50 0x7f6c180aa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d11d700 1 -- 172.21.15.3:0/2235358780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c180b2710 msgr2=0x7f6c180ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d11d700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c180b2710 0x7f6c180ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d11d700 1 -- 172.21.15.3:0/2235358780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c10009470 con 0x7f6c180b01a0 2022-07-21T01:37:17.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d11d700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180b01a0 0x7f6c181495d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f6c14000ed0 tx=0x7f6c14007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:17.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c06ffd700 1 -- 172.21.15.3:0/2235358780 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c140070d0 con 0x7f6c180b01a0 2022-07-21T01:37:17.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c06ffd700 1 -- 172.21.15.3:0/2235358780 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c14007230 con 0x7f6c180b01a0 2022-07-21T01:37:17.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c180ab250 con 0x7f6c180b01a0 2022-07-21T01:37:17.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c06ffd700 1 -- 172.21.15.3:0/2235358780 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c14005610 con 0x7f6c180b01a0 2022-07-21T01:37:17.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c180abbe0 con 0x7f6c180b01a0 2022-07-21T01:37:17.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c18159f70 con 0x7f6c180b01a0 2022-07-21T01:37:17.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c06ffd700 1 -- 172.21.15.3:0/2235358780 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6c1401a060 con 0x7f6c180b01a0 2022-07-21T01:37:17.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c06ffd700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bf8053b30 0x7f6bf8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c1d91e700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bf8053b30 0x7f6bf8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.501+0000 7f6c06ffd700 1 -- 172.21.15.3:0/2235358780 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6c1406b790 con 0x7f6c180b01a0 2022-07-21T01:37:17.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.505+0000 7f6c1d91e700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bf8053b30 0x7f6bf8055ff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f6c0800c000 tx=0x7f6c08008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:17.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.509+0000 7f6c06ffd700 1 -- 172.21.15.3:0/2235358780 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6c14042550 con 0x7f6c180b01a0 2022-07-21T01:37:17.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.633+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c180574e0 con 0x7f6c180b01a0 2022-07-21T01:37:17.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.633+0000 7f6c06ffd700 1 -- 172.21.15.3:0/2235358780 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6c14018070 con 0x7f6c180b01a0 2022-07-21T01:37:17.635 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:37:17.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bf8053b30 msgr2=0x7f6bf8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bf8053b30 0x7f6bf8055ff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f6c0800c000 tx=0x7f6c08008040).stop 2022-07-21T01:37:17.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180b01a0 msgr2=0x7f6c181495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180b01a0 0x7f6c181495d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f6c14000ed0 tx=0x7f6c14007a60).stop 2022-07-21T01:37:17.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 shutdown_connections 2022-07-21T01:37:17.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bf8053b30 0x7f6bf8055ff0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c180b2710 0x7f6c180ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c180b01a0 0x7f6c181495d0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 --2- 172.21.15.3:0/2235358780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c180a8c50 0x7f6c180aa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 >> 172.21.15.3:0/2235358780 conn(0x7f6c18060310 msgr2=0x7f6c1805ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:17.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 shutdown_connections 2022-07-21T01:37:17.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.637+0000 7f6c1e920700 1 -- 172.21.15.3:0/2235358780 wait complete. 2022-07-21T01:37:17.655 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:37:17.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:37:17.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 Processor -- start 2022-07-21T01:37:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 -- start start 2022-07-21T01:37:17.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6cac0b38d0 0x7f6cac060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6cac060e40 0x7f6cac0612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cac060670 con 0x7f6cac060290 2022-07-21T01:37:17.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cac065c30 con 0x7f6cac060e40 2022-07-21T01:37:17.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cac065ed0 con 0x7f6cac0b38d0 2022-07-21T01:37:17.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6caad9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6cac060290 0x7f6cac066610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54084/0 (socket says 172.21.15.3:54084) 2022-07-21T01:37:17.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6caad9d700 1 -- 172.21.15.3:0/3783440074 learned_addr learned my addr 172.21.15.3:0/3783440074 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:37:17.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cab59e700 1 --2- 172.21.15.3:0/3783440074 >> v2:172.21.15.3:3300/0 conn(0x7f6cac060e40 0x7f6cac0612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6caa59c700 1 --2- 172.21.15.3:0/3783440074 >> v2:172.21.15.143:3300/0 conn(0x7f6cac0b38d0 0x7f6cac060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6ca9d9b700 1 -- 172.21.15.3:0/3783440074 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595344565 0 0) 0x7f6cac060670 con 0x7f6cac060290 2022-07-21T01:37:17.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6ca9d9b700 1 -- 172.21.15.3:0/3783440074 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c90003260 con 0x7f6cac060290 2022-07-21T01:37:17.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cab59e700 1 -- 172.21.15.3:0/3783440074 >> v2:172.21.15.143:3300/0 conn(0x7f6cac0b38d0 msgr2=0x7f6cac060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:37:17.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cab59e700 1 --2- 172.21.15.3:0/3783440074 >> v2:172.21.15.143:3300/0 conn(0x7f6cac0b38d0 0x7f6cac060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cab59e700 1 -- 172.21.15.3:0/3783440074 >> v1:172.21.15.3:6789/0 conn(0x7f6cac060290 legacy=0x7f6cac066610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cab59e700 1 -- 172.21.15.3:0/3783440074 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cac066170 con 0x7f6cac060e40 2022-07-21T01:37:17.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cab59e700 1 --2- 172.21.15.3:0/3783440074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060e40 0x7f6cac0612b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f6ca40032f0 tx=0x7f6ca40091d0).ready entity=mon.0 client_cookie=9091202a8d6b6b9e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6ca9d9b700 1 -- 172.21.15.3:0/3783440074 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ca400b600 con 0x7f6cac060e40 2022-07-21T01:37:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6ca9d9b700 1 -- 172.21.15.3:0/3783440074 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca400b760 con 0x7f6cac060e40 2022-07-21T01:37:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6ca9d9b700 1 -- 172.21.15.3:0/3783440074 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ca400b8d0 con 0x7f6cac060e40 2022-07-21T01:37:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 -- 172.21.15.3:0/3783440074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060e40 msgr2=0x7f6cac0612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 --2- 172.21.15.3:0/3783440074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060e40 0x7f6cac0612b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f6ca40032f0 tx=0x7f6ca40091d0).stop 2022-07-21T01:37:17.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 -- 172.21.15.3:0/3783440074 shutdown_connections 2022-07-21T01:37:17.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 --2- 172.21.15.3:0/3783440074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060e40 0x7f6cac0612b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 --2- 172.21.15.3:0/3783440074 >> v2:172.21.15.143:3300/0 conn(0x7f6cac0b38d0 0x7f6cac060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 -- 172.21.15.3:0/3783440074 >> 172.21.15.3:0/3783440074 conn(0x7f6cac05d0c0 msgr2=0x7f6cac05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:17.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 -- 172.21.15.3:0/3783440074 shutdown_connections 2022-07-21T01:37:17.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.765+0000 7f6cb1047700 1 -- 172.21.15.3:0/3783440074 wait complete. 2022-07-21T01:37:17.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.769+0000 7f6cb1047700 1 Processor -- start 2022-07-21T01:37:17.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.769+0000 7f6cb1047700 1 -- start start 2022-07-21T01:37:17.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cb1047700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060290 0x7f6cac0ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cb1047700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cac060e40 0x7f6cac0b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cb1047700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cac0b38d0 0x7f6cac15a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cb1047700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac0adfa0 con 0x7f6cac060290 2022-07-21T01:37:17.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cb1047700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac0ae240 con 0x7f6cac0b38d0 2022-07-21T01:37:17.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cb1047700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac15a080 con 0x7f6cac060e40 2022-07-21T01:37:17.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060290 0x7f6cac0ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060290 0x7f6cac0ad6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45810/0 (socket says 172.21.15.3:45810) 2022-07-21T01:37:17.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caad9d700 1 -- 172.21.15.3:0/1884062966 learned_addr learned my addr 172.21.15.3:0/1884062966 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:37:17.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cab59e700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cac0b38d0 0x7f6cac15a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caa59c700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cac060e40 0x7f6cac0b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caad9d700 1 -- 172.21.15.3:0/1884062966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cac060e40 msgr2=0x7f6cac0b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caad9d700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cac060e40 0x7f6cac0b1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caad9d700 1 -- 172.21.15.3:0/1884062966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cac0b38d0 msgr2=0x7f6cac15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caad9d700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cac0b38d0 0x7f6cac15a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caad9d700 1 -- 172.21.15.3:0/1884062966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca4009470 con 0x7f6cac060290 2022-07-21T01:37:17.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cab59e700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cac0b38d0 0x7f6cac15a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:37:17.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6caad9d700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060290 0x7f6cac0ad6e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f6c9c001110 tx=0x7f6c9c007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:17.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6c9bfff700 1 -- 172.21.15.3:0/1884062966 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c9c0070d0 con 0x7f6cac060290 2022-07-21T01:37:17.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cac0a1970 con 0x7f6cac060290 2022-07-21T01:37:17.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6c9bfff700 1 -- 172.21.15.3:0/1884062966 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c007230 con 0x7f6cac060290 2022-07-21T01:37:17.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6c9bfff700 1 -- 172.21.15.3:0/1884062966 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c9c005610 con 0x7f6cac060290 2022-07-21T01:37:17.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac156930 con 0x7f6cac060290 2022-07-21T01:37:17.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.773+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cac15a240 con 0x7f6cac060290 2022-07-21T01:37:17.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.777+0000 7f6c9bfff700 1 -- 172.21.15.3:0/1884062966 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6c9c010070 con 0x7f6cac060290 2022-07-21T01:37:17.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.777+0000 7f6c9bfff700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6c88053aa0 0x7f6c88055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:37:17.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.777+0000 7f6c9bfff700 1 -- 172.21.15.3:0/1884062966 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6c9c06b720 con 0x7f6cac060290 2022-07-21T01:37:17.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.777+0000 7f6caa59c700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6c88053aa0 0x7f6c88055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:37:17.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.777+0000 7f6caa59c700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6c88053aa0 0x7f6c88055f60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f6ca000b000 tx=0x7f6ca0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:37:17.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.785+0000 7f6c9bfff700 1 -- 172.21.15.3:0/1884062966 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6c9c042440 con 0x7f6cac060290 2022-07-21T01:37:17.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.949+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cac0574e0 con 0x7f6cac060290 2022-07-21T01:37:17.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.949+0000 7f6c9bfff700 1 -- 172.21.15.3:0/1884062966 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f6c9c045a60 con 0x7f6cac060290 2022-07-21T01:37:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:37:17.950 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:36:52.675976+0000","modified":"2022-07-21T01:36:53.679085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/3381448192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3381448192},{"type":"v1","addr":"172.21.15.3:6837","nonce":3381448192}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-21T01:37:17.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6c88053aa0 msgr2=0x7f6c88055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6c88053aa0 0x7f6c88055f60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f6ca000b000 tx=0x7f6ca0006df0).stop 2022-07-21T01:37:17.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060290 msgr2=0x7f6cac0ad6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:37:17.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060290 0x7f6cac0ad6e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f6c9c001110 tx=0x7f6c9c007aa0).stop 2022-07-21T01:37:17.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 shutdown_connections 2022-07-21T01:37:17.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6c88053aa0 0x7f6c88055f60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cac0b38d0 0x7f6cac15a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cac060e40 0x7f6cac0b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 --2- 172.21.15.3:0/1884062966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6cac060290 0x7f6cac0ad6e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:37:17.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 >> 172.21.15.3:0/1884062966 conn(0x7f6cac05d0c0 msgr2=0x7f6cac14e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:37:17.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 shutdown_connections 2022-07-21T01:37:17.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:37:17.953+0000 7f6cb1047700 1 -- 172.21.15.3:0/1884062966 wait complete. 2022-07-21T01:37:17.955 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2022-07-21T01:37:18.265 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:36:52.675976+0000', 'modified': '2022-07-21T01:36:53.679085+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5586}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5586': {'gid': 5586, 'name': 'a', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/3381448192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 3381448192}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 3381448192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:37:18.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:37:18.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:37:18.386 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":5586,"want_state":"up:active","state":"up:active","rank_uptime":25.698764945000001,"mdsmap_epoch":23,"osdmap_epoch":58,"osdmap_epoch_barrier":57,"uptime":29.967341345000001} 2022-07-21T01:37:18.386 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 5586, "mdsmap_epoch": 23, "osdmap_epoch": 58, "osdmap_epoch_barrier": 57, "rank_uptime": 25.698764945, "state": "up:active", "uptime": 29.967341345, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:37:18.387 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:37:18.387 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:37:18.388 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:37:18.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:18.389 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:37:18.399 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:37:18.457 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:37:18.458 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:37:18.470 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:37:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:37:18.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:18.475 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:37:18.485 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:37:18.486 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:37:18.531 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:37:18.548 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:37:18.549 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:37:18.560 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:37:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:37:18.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:18.564 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:37:18.607 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:37:18.609+0000 7f15c1070080 -1 init, newargv = 0x55db43c88800 newargc=9ceph-fuse[22577]: starting ceph client 2022-07-21T01:37:18.607 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr: 2022-07-21T01:37:18.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22577]: starting fuse 2022-07-21T01:37:19.714 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:37:19.790 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:37:19.791 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:37:19.803 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:37:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:37:19.804 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:19.805 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:37:19.855 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:37:19.856 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:37:19.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:19.857 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:37:19.858 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:37:19.858 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:37:19.859 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:37:19.860 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:37:19.861 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:37:19.862 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:37:19.862 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:37:19.863 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:37:19.864 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:37:19.865 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:37:19.866 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:37:19.866 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:37:19.867 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:37:19.868 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:37:19.869 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:37:19.870 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:37:19.870 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:37:19.871 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:37:19.872 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:37:19.873 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:37:19.873 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:37:19.874 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:37:19.920 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22577.asok 2022-07-21T01:37:19.925 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22577.asok 2022-07-21T01:37:19.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:19.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22577.asok status 2022-07-21T01:37:20.017 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:37:20.017 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:37:20.017 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:37:20.018 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:37:20.018 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:37:20.019 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:37:20.019 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:37:20.019 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22577", 2022-07-21T01:37:20.020 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:37:20.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:37:20.021 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:37:20.021 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:37:20.022 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 5827, 2022-07-21T01:37:20.025 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:37:20.025 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:37:20.025 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:37:20.026 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 5827 2022-07-21T01:37:20.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:37:20.026 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:37:20.026 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:37:20.027 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:37:20.027 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3475819407 2022-07-21T01:37:20.028 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:37:20.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:37:20.028 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:37:20.028 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:37:20.029 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:37:20.029 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3475819407 2022-07-21T01:37:20.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:37:20.029 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.5827 172.21.15.171:0/3475819407", 2022-07-21T01:37:20.030 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/3475819407", 2022-07-21T01:37:20.030 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:37:20.030 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 23, 2022-07-21T01:37:20.031 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 58, 2022-07-21T01:37:20.031 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:37:20.031 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:37:20.031 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:37:20.032 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:37:20.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:20.034 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:37:20.035 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:37:20.044 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:37:20.045 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:37:20.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:37:20.047 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:37:31.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:37:31.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:37:31.748 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:37:31.750 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:38:01.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:38:01.808 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:38:01.820 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:38:01.825 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:38:13.416 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:38:13.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:13.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:13.461 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:13.483 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:13.507 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:38:13.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:13.509 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:13.510 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:13.519 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:38:13.521 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:38:13.521 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:38:13.521 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:38:13.522 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:13.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22577]: fuse finished with error 0 and tester_r 0 2022-07-21T01:38:13.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:13.580 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:13.581 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:13.588 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:38:13.648 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:38:13.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:38:13.680 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:13.730 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:13.731 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:13.731 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:38:13.739 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:38:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:38:13.741 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:38:13.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:38:13.742 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:38:13.748 INFO:teuthology.orchestra.run.smithi174.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:38:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:38:13.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:38:13.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f6204446700 1 Processor -- start 2022-07-21T01:38:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f6204446700 1 -- start start 2022-07-21T01:38:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f6204446700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f61fc067180 0x7f61fc143a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f6204446700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61fc0b0870 0x7f61fc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f6204446700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc067560 con 0x7f61fc0ad1c0 2022-07-21T01:38:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f6204446700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61fc0b0160 con 0x7f61fc067180 2022-07-21T01:38:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f6204446700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61fc0b0400 con 0x7f61fc0b0870 2022-07-21T01:38:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62021e2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f61fc067180 0x7f61fc143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62021e2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f61fc067180 0x7f61fc143a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45818/0 (socket says 172.21.15.3:45818) 2022-07-21T01:38:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62021e2700 1 -- 172.21.15.3:0/2613823331 learned_addr learned my addr 172.21.15.3:0/2613823331 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62029e3700 1 --2- 172.21.15.3:0/2613823331 >> v2:172.21.15.143:3300/0 conn(0x7f61fc0b0870 0x7f61fc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62021e2700 1 -- 172.21.15.3:0/2613823331 >> v2:172.21.15.143:3300/0 conn(0x7f61fc0b0870 msgr2=0x7f61fc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62021e2700 1 --2- 172.21.15.3:0/2613823331 >> v2:172.21.15.143:3300/0 conn(0x7f61fc0b0870 0x7f61fc0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62021e2700 1 -- 172.21.15.3:0/2613823331 >> v1:172.21.15.143:6789/0 conn(0x7f61fc0ad1c0 legacy=0x7f61fc143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62021e2700 1 -- 172.21.15.3:0/2613823331 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc0af530 con 0x7f61fc067180 2022-07-21T01:38:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62021e2700 1 --2- 172.21.15.3:0/2613823331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc067180 0x7f61fc143a60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f61f0002f70 tx=0x7f61f0006eb0).ready entity=mon.0 client_cookie=41f7dda4e8128950 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:13.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62011e0700 1 -- 172.21.15.3:0/2613823331 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61f000d5c0 con 0x7f61fc067180 2022-07-21T01:38:13.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62011e0700 1 -- 172.21.15.3:0/2613823331 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f0000e90 con 0x7f61fc067180 2022-07-21T01:38:13.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.861+0000 7f62011e0700 1 -- 172.21.15.3:0/2613823331 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61f000d950 con 0x7f61fc067180 2022-07-21T01:38:13.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 -- 172.21.15.3:0/2613823331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc067180 msgr2=0x7f61fc143a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:13.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 --2- 172.21.15.3:0/2613823331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc067180 0x7f61fc143a60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f61f0002f70 tx=0x7f61f0006eb0).stop 2022-07-21T01:38:13.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 -- 172.21.15.3:0/2613823331 shutdown_connections 2022-07-21T01:38:13.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 --2- 172.21.15.3:0/2613823331 >> v2:172.21.15.143:3300/0 conn(0x7f61fc0b0870 0x7f61fc0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:13.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 --2- 172.21.15.3:0/2613823331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc067180 0x7f61fc143a60 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:13.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 -- 172.21.15.3:0/2613823331 >> 172.21.15.3:0/2613823331 conn(0x7f61fc05f840 msgr2=0x7f61fc05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:13.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 -- 172.21.15.3:0/2613823331 shutdown_connections 2022-07-21T01:38:13.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 -- 172.21.15.3:0/2613823331 wait complete. 2022-07-21T01:38:13.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 Processor -- start 2022-07-21T01:38:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.865+0000 7f6204446700 1 -- start start 2022-07-21T01:38:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f6204446700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61fc067180 0x7f61fc1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f6204446700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61fc0b0870 0x7f61fc0a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f6204446700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc0a40b0 0x7f61fc0a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f6204446700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc156da0 con 0x7f61fc0a40b0 2022-07-21T01:38:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f6204446700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc09fcf0 con 0x7f61fc067180 2022-07-21T01:38:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f6204446700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc09ff90 con 0x7f61fc0b0870 2022-07-21T01:38:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62029e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc0a40b0 0x7f61fc0a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62029e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc0a40b0 0x7f61fc0a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45826/0 (socket says 172.21.15.3:45826) 2022-07-21T01:38:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62029e3700 1 -- 172.21.15.3:0/1819069298 learned_addr learned my addr 172.21.15.3:0/1819069298 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62021e2700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61fc067180 0x7f61fc1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62019e1700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61fc0b0870 0x7f61fc0a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62029e3700 1 -- 172.21.15.3:0/1819069298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61fc0b0870 msgr2=0x7f61fc0a3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62029e3700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61fc0b0870 0x7f61fc0a3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62029e3700 1 -- 172.21.15.3:0/1819069298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61fc067180 msgr2=0x7f61fc1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62029e3700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61fc067180 0x7f61fc1568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62029e3700 1 -- 172.21.15.3:0/1819069298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f000b1e0 con 0x7f61fc0a40b0 2022-07-21T01:38:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62019e1700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61fc0b0870 0x7f61fc0a3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f62029e3700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc0a40b0 0x7f61fc0a1150 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f61ec00c000 tx=0x7f61ec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f61fb7fe700 1 -- 172.21.15.3:0/1819069298 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61ec011750 con 0x7f61fc0a40b0 2022-07-21T01:38:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f61fb7fe700 1 -- 172.21.15.3:0/1819069298 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ec0118b0 con 0x7f61fc0a40b0 2022-07-21T01:38:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc0a0230 con 0x7f61fc0a40b0 2022-07-21T01:38:13.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f61fb7fe700 1 -- 172.21.15.3:0/1819069298 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61ec011c10 con 0x7f61fc0a40b0 2022-07-21T01:38:13.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.869+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc0a17e0 con 0x7f61fc0a40b0 2022-07-21T01:38:13.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.873+0000 7f61fb7fe700 1 -- 172.21.15.3:0/1819069298 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f61ec011d70 con 0x7f61fc0a40b0 2022-07-21T01:38:13.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.873+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61fc0a1dd0 con 0x7f61fc0a40b0 2022-07-21T01:38:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.873+0000 7f61fb7fe700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61dc053a70 0x7f61dc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.873+0000 7f61fb7fe700 1 -- 172.21.15.3:0/1819069298 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f61ec06c3a0 con 0x7f61fc0a40b0 2022-07-21T01:38:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.873+0000 7f62021e2700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61dc053a70 0x7f61dc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.873+0000 7f62021e2700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61dc053a70 0x7f61dc055f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f61f001c000 tx=0x7f61f0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:13.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:13.877+0000 7f61fb7fe700 1 -- 172.21.15.3:0/1819069298 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f61ec043160 con 0x7f61fc0a40b0 2022-07-21T01:38:14.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.001+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f61fc0574e0 con 0x7f61fc0a40b0 2022-07-21T01:38:14.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.005+0000 7f61fb7fe700 1 -- 172.21.15.3:0/1819069298 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v58) v1 ==== 81+0+34039 (secure 0 0 0) 0x7f61ec046780 con 0x7f61fc0a40b0 2022-07-21T01:38:14.004 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:14.004 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:38:14.004 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 58, 2022-07-21T01:38:14.004 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:38:14.005 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:38:14.005 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:36:53.679840+0000", 2022-07-21T01:38:14.005 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:38:14.005 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:38:14.005 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:38:14.006 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:38:14.007 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:38:14.007 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:38:14.007 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:38:14.007 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:38:14.008 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:38:14.008 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:14.008 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:38:14.008 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:38:14.008 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:38:14.009 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:38:14.009 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:38:14.009 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 10, 2022-07-21T01:38:14.009 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:38:14.010 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:38:14.010 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:38:14.010 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:38:14.011 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:38:14.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.011 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:38:14.011 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:38:14.011 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:38:14.012 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:14.012 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:14.012 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:14.012 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:14.013 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:14.013 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:14.013 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:14.013 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:14.013 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:38:14.014 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:38:14.014 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:38:14.014 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:38:14.014 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:38:14.015 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:14.015 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:14.015 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:14.015 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:14.015 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:14.016 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:14.016 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:14.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.016 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:38:14.017 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:14.017 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:14.017 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:14.017 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:14.017 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:14.018 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:14.018 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:14.018 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:14.018 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:14.019 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:14.019 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:14.019 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:14.019 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:14.019 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:14.020 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:14.020 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:14.020 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:14.020 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:14.021 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:14.021 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:14.021 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:14.021 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:14.022 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:14.022 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:14.022 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:14.022 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:14.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.023 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:14.023 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:14.023 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:14.024 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:14.024 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:14.024 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:14.024 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:14.024 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:14.025 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:14.025 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:14.025 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:14.025 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:38:14.026 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:38:14.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.026 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:14.026 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:38:14.027 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.027 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:38:14.028 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:38:14.028 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:38:14.028 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:14.029 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:14.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:14.029 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:14.029 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:14.030 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:14.030 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:14.030 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:14.030 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:38:14.031 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:38:14.031 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:38:14.031 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:38:14.031 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:38:14.031 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:14.032 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:14.032 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:14.033 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:14.033 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:14.033 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:14.033 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:14.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.034 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:38:14.034 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:14.034 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:14.035 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:14.035 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:14.035 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:14.035 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:14.036 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:14.036 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:14.037 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:14.037 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:14.037 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:14.037 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:14.038 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:14.038 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:14.038 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:14.038 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:14.039 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:14.039 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:14.039 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:14.039 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:14.040 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:14.040 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:14.040 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:14.040 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:14.041 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:14.041 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:14.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.041 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:14.042 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:14.042 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:14.042 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:14.042 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:14.042 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:14.043 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:14.043 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:14.043 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:14.043 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:14.044 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:14.044 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:38:14.044 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:14.044 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:38:14.045 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.045 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 9, 2022-07-21T01:38:14.046 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:38:14.046 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:36:51.181015+0000", 2022-07-21T01:38:14.046 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:14.047 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:14.047 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:14.105 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:14.106 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:14.106 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:14.106 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:14.107 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:14.107 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:38:14.107 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:38:14.107 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:38:14.108 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:38:14.108 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:38:14.108 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:14.108 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:14.109 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:14.109 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:14.109 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:14.109 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:14.109 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:14.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.110 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "57", 2022-07-21T01:38:14.110 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:14.110 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:14.111 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:14.111 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:14.111 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:14.112 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:14.113 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:14.113 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:14.114 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:14.114 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:14.115 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:14.115 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:14.116 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:14.116 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:14.117 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:14.117 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:14.118 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:14.118 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:14.119 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:14.119 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:14.120 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:14.120 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:14.120 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:14.121 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:14.121 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:14.122 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:14.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.123 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:14.123 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:14.124 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:14.124 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:14.125 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:14.125 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:14.126 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:14.126 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:14.127 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:14.127 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:14.128 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:14.128 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:38:14.129 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:38:14.129 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:38:14.130 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:38:14.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.131 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:14.131 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:38:14.132 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:38:14.132 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.133 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.134 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 10, 2022-07-21T01:38:14.135 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:38:14.135 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:36:52.366227+0000", 2022-07-21T01:38:14.136 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:14.136 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:14.137 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:14.137 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:14.138 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:14.138 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:14.138 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:14.139 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:14.139 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:38:14.139 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:38:14.139 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:38:14.140 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:38:14.140 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:38:14.140 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:14.140 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:14.141 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:14.141 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:14.141 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:14.141 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:14.142 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:14.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.142 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "58", 2022-07-21T01:38:14.142 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:14.143 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:14.143 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:14.143 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:14.143 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:14.144 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:14.144 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:14.144 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:14.144 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:14.145 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:14.145 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:14.145 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:14.145 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:14.146 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:14.146 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:14.146 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:14.146 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:14.147 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:14.147 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:14.147 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:14.147 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:14.148 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:14.148 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:14.148 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:14.148 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:14.149 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:14.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.149 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:14.149 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:14.150 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:14.150 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:14.150 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:14.150 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:14.151 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:14.151 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:14.151 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:14.151 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:14.152 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:14.152 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:38:14.152 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:14.152 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:38:14.152 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:38:14.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.153 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:14.154 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:38:14.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.154 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:38:14.154 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:38:14.155 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:14.155 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:14.155 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:14.155 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:14.156 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:14.156 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:14.156 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:38:14.156 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:14.157 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:14.157 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:14.157 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:14.157 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.158 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.158 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:38:14.158 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:14.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.159 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.159 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:38:14.159 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:14.160 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.160 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.160 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:14.161 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.161 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:38:14.162 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:14.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.162 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.163 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:38:14.163 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:14.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.163 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.164 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:14.164 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.164 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.165 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:38:14.165 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:14.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.166 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.166 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:38:14.166 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:14.166 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.167 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.167 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:14.167 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.168 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:38:14.168 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:14.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.169 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:38:14.170 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:14.170 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.170 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.171 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:38:14.171 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:38:14.171 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:38:14.171 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:38:14.172 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:14.172 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:14.172 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:14.172 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.173 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:38:14.173 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:38:14.173 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:14.174 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:14.174 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:14.174 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:14.174 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:14.175 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:14.175 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:14.175 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:14.175 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:14.176 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:14.176 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:14.176 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.177 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.177 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:38:14.177 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:14.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.178 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:38:14.178 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:14.179 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.179 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.179 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:14.180 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.180 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.180 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:38:14.181 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:14.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.181 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.182 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:38:14.182 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:14.183 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.183 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.184 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:14.185 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.186 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.186 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:38:14.187 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:14.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.188 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.188 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:38:14.188 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:14.189 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.189 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.189 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:14.190 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.190 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.191 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:38:14.191 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:14.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.192 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:38:14.192 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:14.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.193 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.193 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:38:14.193 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:38:14.193 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:38:14.194 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:38:14.194 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:14.194 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:14.195 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:14.195 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.196 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:38:14.196 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:38:14.196 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:14.196 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:14.197 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:14.197 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:14.197 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:14.198 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:14.198 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:38:14.198 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:14.198 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:14.199 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:14.199 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:14.199 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.200 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.200 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:38:14.200 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:14.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.201 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.201 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:38:14.202 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:14.202 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.203 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.203 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:14.203 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.204 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.204 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:38:14.204 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:14.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.205 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.205 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:38:14.206 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:14.206 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.206 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.207 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:14.207 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.208 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.208 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:38:14.208 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:14.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.209 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.209 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:38:14.210 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:14.210 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.210 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.211 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:14.212 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.212 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.212 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:38:14.213 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:14.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.213 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.214 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:38:14.214 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:14.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.215 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.216 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:38:14.217 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:38:14.217 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:38:14.218 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:38:14.218 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:14.219 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:14.219 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:14.220 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.221 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:38:14.221 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:38:14.222 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:14.222 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:14.223 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:14.223 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:14.224 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:14.224 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:14.225 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:38:14.225 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 55, 2022-07-21T01:38:14.226 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:14.227 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:14.227 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:14.228 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.229 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.229 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:38:14.230 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:14.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.231 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.232 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:38:14.232 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:14.233 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.233 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.234 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:14.234 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.236 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.236 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:38:14.237 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:14.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.238 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.239 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:38:14.239 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:14.240 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.240 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.241 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:14.242 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.242 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.243 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:38:14.244 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:14.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.245 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.245 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:38:14.245 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:14.246 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.246 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.247 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:14.247 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.247 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.247 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:38:14.248 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:14.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.248 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.249 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:38:14.249 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:14.249 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.249 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.250 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:38:14.250 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:38:14.250 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:38:14.251 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:38:14.251 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:14.251 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:14.251 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:14.251 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.252 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:38:14.252 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:38:14.253 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:14.253 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:14.253 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:14.253 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:14.254 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:14.254 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:14.254 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:14.254 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:14.255 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:14.255 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:14.255 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:14.255 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.256 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.256 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:38:14.257 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:14.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.257 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.257 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:38:14.258 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:14.258 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.258 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.259 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:14.259 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.259 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.260 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:38:14.260 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:14.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.261 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.261 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:38:14.261 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:14.261 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.262 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.262 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:14.262 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.263 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.263 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:38:14.263 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:14.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.264 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.264 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:38:14.264 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:14.265 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.265 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.265 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:14.266 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.280 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.280 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:38:14.280 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:14.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.281 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.281 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:38:14.281 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:14.282 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.282 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.282 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:38:14.283 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:38:14.283 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:38:14.283 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:38:14.283 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:14.284 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:14.284 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:14.284 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.285 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:38:14.285 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:38:14.285 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:14.285 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:14.286 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:14.286 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:14.286 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:14.286 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:14.287 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:14.287 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:14.287 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:14.288 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:14.288 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:14.288 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.288 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.289 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:38:14.289 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:14.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.290 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.290 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:38:14.290 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:14.290 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.291 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.291 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:14.291 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.292 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.292 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:38:14.293 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:14.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.294 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.294 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:38:14.294 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:14.294 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.295 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.295 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:14.296 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.296 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.296 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:38:14.297 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:14.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.298 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.298 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:38:14.298 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:14.298 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.299 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.299 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:14.299 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.300 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.300 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:38:14.301 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:14.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.301 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.302 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:38:14.302 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:14.303 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.303 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.304 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:38:14.304 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:38:14.304 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:38:14.305 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:38:14.305 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:14.305 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:14.305 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:14.306 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.307 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:38:14.307 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:38:14.307 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:14.307 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:14.308 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:14.308 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:14.308 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:14.308 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:14.309 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:14.309 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:14.309 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:14.309 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:14.310 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:14.310 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.311 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.311 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:38:14.311 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:14.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.312 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.312 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:38:14.312 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:14.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.313 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.313 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:14.314 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.314 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.314 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:38:14.315 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:14.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.315 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.316 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:38:14.316 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:14.317 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.317 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.317 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:14.318 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.318 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.319 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:38:14.319 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:14.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.320 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.320 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:38:14.320 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:14.320 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.320 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.321 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:14.321 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.322 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.322 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:38:14.322 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:14.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.415 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:38:14.415 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:14.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.415 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.416 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:38:14.416 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:38:14.416 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:38:14.417 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:38:14.417 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:14.417 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:14.418 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:14.418 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.419 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:38:14.419 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:38:14.419 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:14.419 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:14.420 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:14.420 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:14.420 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:14.420 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:14.420 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:14.421 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 55, 2022-07-21T01:38:14.421 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:14.421 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:14.421 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:14.422 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.422 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:38:14.423 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:14.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.423 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.423 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:38:14.424 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:14.424 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.424 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.425 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:14.425 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.425 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.425 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:38:14.426 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:14.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.426 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.427 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:38:14.427 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:14.427 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.427 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.428 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:14.428 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.429 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:38:14.429 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:14.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.430 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.430 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:38:14.430 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:14.431 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.431 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.431 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:14.432 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:14.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.432 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:14.433 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:38:14.433 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:14.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:14.434 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:38:14.434 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:14.434 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.434 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.435 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:38:14.435 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:38:14.435 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:38:14.436 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:38:14.436 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:14.436 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:14.436 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:14.437 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:14.437 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.437 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:14.437 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:38:14.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.438 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:38:14.438 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:14.438 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:14.438 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:14.439 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:14.439 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:14.439 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:38:14.439 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:14.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.440 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:38:14.440 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:14.440 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:14.441 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:14.441 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:14.441 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:14.441 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:38:14.441 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:14.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.442 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:38:14.442 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:14.443 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:14.443 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:14.443 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:14.443 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:14.444 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:38:14.444 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:14.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:38:14.445 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:14.445 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:14.445 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:14.445 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:14.446 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:14.446 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:38:14.446 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:14.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:38:14.447 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:14.447 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:14.447 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:14.448 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:14.448 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:14.448 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:38:14.448 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:14.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:38:14.449 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:14.450 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:14.450 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:14.450 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:14.450 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:14.450 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:38:14.451 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:14.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:38:14.452 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:14.452 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:14.452 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:14.452 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:14.453 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:14.453 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:38:14.453 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:14.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:14.454 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:38:14.454 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:14.454 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:14.455 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:14.455 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:14.455 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:14.455 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:38:14.455 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:14.456 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.456 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:14.456 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:38:14.456 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:38:14.457 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:38:14.457 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:38:14.457 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:38:14.457 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:38:14.457 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:38:14.458 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:38:14.458 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:38:14.458 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:38:14.458 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:38:14.459 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:38:14.459 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:38:14.459 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:38:14.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:38:14.460 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:38:14.460 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:38:14.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.460 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:38:14.461 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:38:14.461 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:38:14.461 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:38:14.461 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:38:14.462 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:38:14.462 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:38:14.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:14.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:14.464 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:38:14.464 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:38:14.464 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:38:14.465 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:38:14.465 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:38:14.465 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:38:14.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.005+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61dc053a70 msgr2=0x7f61dc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.005+0000 7f6204446700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61dc053a70 0x7f61dc055f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f61f001c000 tx=0x7f61f0015000).stop 2022-07-21T01:38:14.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.005+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc0a40b0 msgr2=0x7f61fc0a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.005+0000 7f6204446700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc0a40b0 0x7f61fc0a1150 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f61ec00c000 tx=0x7f61ec006df0).stop 2022-07-21T01:38:14.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.009+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 shutdown_connections 2022-07-21T01:38:14.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.009+0000 7f6204446700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61fc0a40b0 0x7f61fc0a1150 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.009+0000 7f6204446700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61fc0b0870 0x7f61fc0a3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.009+0000 7f6204446700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61dc053a70 0x7f61dc055f30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.009+0000 7f6204446700 1 --2- 172.21.15.3:0/1819069298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61fc067180 0x7f61fc1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.009+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 >> 172.21.15.3:0/1819069298 conn(0x7f61fc05f840 msgr2=0x7f61fc0b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:14.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.009+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 shutdown_connections 2022-07-21T01:38:14.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.009+0000 7f6204446700 1 -- 172.21.15.3:0/1819069298 wait complete. 2022-07-21T01:38:14.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:14.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f262e412700 1 Processor -- start 2022-07-21T01:38:14.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f262e412700 1 -- start start 2022-07-21T01:38:14.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f262e412700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f26280b1f40 0x7f26280b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f262e412700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280a7250 0x7f26281448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f262e412700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280b04e0 con 0x7f2628148530 2022-07-21T01:38:14.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f262e412700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26280a9250 con 0x7f26280b1f40 2022-07-21T01:38:14.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f262e412700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26280b0780 con 0x7f26280a7250 2022-07-21T01:38:14.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2627fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f26280b1f40 0x7f26280b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f262c9af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280a7250 0x7f26281448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2627fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f26280b1f40 0x7f26280b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45834/0 (socket says 172.21.15.3:45834) 2022-07-21T01:38:14.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f262c9af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280a7250 0x7f26281448b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:52752/0 (socket says 172.21.15.3:52752) 2022-07-21T01:38:14.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2627fff700 1 -- 172.21.15.3:0/3869422779 learned_addr learned my addr 172.21.15.3:0/3869422779 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:14.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f26277fe700 1 --1- 172.21.15.3:0/3869422779 >> v1:172.21.15.143:6789/0 conn(0x7f2628148530 0x7f26280b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:56228/0 (socket says 172.21.15.3:56228) 2022-07-21T01:38:14.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2627fff700 1 -- 172.21.15.3:0/3869422779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280a7250 msgr2=0x7f26281448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:14.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2627fff700 1 --2- 172.21.15.3:0/3869422779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280a7250 0x7f26281448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2627fff700 1 -- 172.21.15.3:0/3869422779 >> v1:172.21.15.143:6789/0 conn(0x7f2628148530 legacy=0x7f26280b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2627fff700 1 -- 172.21.15.3:0/3869422779 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26280b0a20 con 0x7f26280b1f40 2022-07-21T01:38:14.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2627fff700 1 --2- 172.21.15.3:0/3869422779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26280b1f40 0x7f26280b1160 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f26281478c0 tx=0x7f261c006eb0).ready entity=mon.0 client_cookie=d568ff41eafbfa79 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:14.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2626ffd700 1 -- 172.21.15.3:0/3869422779 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f261c00d5c0 con 0x7f26280b1f40 2022-07-21T01:38:14.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2626ffd700 1 -- 172.21.15.3:0/3869422779 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c000e90 con 0x7f26280b1f40 2022-07-21T01:38:14.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.581+0000 7f2626ffd700 1 -- 172.21.15.3:0/3869422779 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f261c00d950 con 0x7f26280b1f40 2022-07-21T01:38:14.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 -- 172.21.15.3:0/3869422779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26280b1f40 msgr2=0x7f26280b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 --2- 172.21.15.3:0/3869422779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26280b1f40 0x7f26280b1160 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f26281478c0 tx=0x7f261c006eb0).stop 2022-07-21T01:38:14.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 -- 172.21.15.3:0/3869422779 shutdown_connections 2022-07-21T01:38:14.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 --2- 172.21.15.3:0/3869422779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280a7250 0x7f26281448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 --2- 172.21.15.3:0/3869422779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26280b1f40 0x7f26280b1160 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 -- 172.21.15.3:0/3869422779 >> 172.21.15.3:0/3869422779 conn(0x7f262805f840 msgr2=0x7f262805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:14.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 -- 172.21.15.3:0/3869422779 shutdown_connections 2022-07-21T01:38:14.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 -- 172.21.15.3:0/3869422779 wait complete. 2022-07-21T01:38:14.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 Processor -- start 2022-07-21T01:38:14.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 -- start start 2022-07-21T01:38:14.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26280a7250 0x7f2628149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280b1f40 0x7f262814b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2628148530 0x7f26280ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26280abed0 con 0x7f2628148530 2022-07-21T01:38:14.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26280ab600 con 0x7f26280a7250 2022-07-21T01:38:14.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262e412700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26280ab8a0 con 0x7f26280b1f40 2022-07-21T01:38:14.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262c9af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2628148530 0x7f26280ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262c9af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2628148530 0x7f26280ac300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45842/0 (socket says 172.21.15.3:45842) 2022-07-21T01:38:14.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f262c9af700 1 -- 172.21.15.3:0/742725681 learned_addr learned my addr 172.21.15.3:0/742725681 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:14.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f26277fe700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280b1f40 0x7f262814b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.585+0000 7f2627fff700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26280a7250 0x7f2628149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f262c9af700 1 -- 172.21.15.3:0/742725681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280b1f40 msgr2=0x7f262814b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f262c9af700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280b1f40 0x7f262814b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f262c9af700 1 -- 172.21.15.3:0/742725681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26280a7250 msgr2=0x7f2628149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f262c9af700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26280a7250 0x7f2628149e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f262c9af700 1 -- 172.21.15.3:0/742725681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c00b1e0 con 0x7f2628148530 2022-07-21T01:38:14.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f2627fff700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26280a7250 0x7f2628149e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:14.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f262c9af700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2628148530 0x7f26280ac300 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f2620006e00 tx=0x7f26200091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:14.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f26257fa700 1 -- 172.21.15.3:0/742725681 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f262000c670 con 0x7f2628148530 2022-07-21T01:38:14.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f26257fa700 1 -- 172.21.15.3:0/742725681 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262000c7d0 con 0x7f2628148530 2022-07-21T01:38:14.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26280abb40 con 0x7f2628148530 2022-07-21T01:38:14.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f26257fa700 1 -- 172.21.15.3:0/742725681 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f262000cb30 con 0x7f2628148530 2022-07-21T01:38:14.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26280ab1d0 con 0x7f2628148530 2022-07-21T01:38:14.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f26257fa700 1 -- 172.21.15.3:0/742725681 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2620022070 con 0x7f2628148530 2022-07-21T01:38:14.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26280ac040 con 0x7f2628148530 2022-07-21T01:38:14.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f26257fa700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2608053b30 0x7f2608055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f26257fa700 1 -- 172.21.15.3:0/742725681 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f262006c5a0 con 0x7f2628148530 2022-07-21T01:38:14.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f2627fff700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2608053b30 0x7f2608055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.589+0000 7f2627fff700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2608053b30 0x7f2608055ff0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f261c01a000 tx=0x7f261c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:14.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.597+0000 7f26257fa700 1 -- 172.21.15.3:0/742725681 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f26280ac040 con 0x7f2628148530 2022-07-21T01:38:14.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.761+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26280af0a0 con 0x7f2628148530 2022-07-21T01:38:14.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.761+0000 7f26257fa700 1 -- 172.21.15.3:0/742725681 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f2620043360 con 0x7f2628148530 2022-07-21T01:38:14.761 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:14.761 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:36:52.675976+0000","modified":"2022-07-21T01:36:53.679085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/3381448192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3381448192},{"type":"v1","addr":"172.21.15.3:6837","nonce":3381448192}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-21T01:38:14.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2608053b30 msgr2=0x7f2608055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2608053b30 0x7f2608055ff0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f261c01a000 tx=0x7f261c015000).stop 2022-07-21T01:38:14.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2628148530 msgr2=0x7f26280ac300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2628148530 0x7f26280ac300 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f2620006e00 tx=0x7f26200091d0).stop 2022-07-21T01:38:14.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 shutdown_connections 2022-07-21T01:38:14.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2628148530 0x7f26280ac300 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26280b1f40 0x7f262814b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2608053b30 0x7f2608055ff0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 --2- 172.21.15.3:0/742725681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26280a7250 0x7f2628149e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 >> 172.21.15.3:0/742725681 conn(0x7f262805f840 msgr2=0x7f26280b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:14.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 shutdown_connections 2022-07-21T01:38:14.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.765+0000 7f262e412700 1 -- 172.21.15.3:0/742725681 wait complete. 2022-07-21T01:38:14.766 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2022-07-21T01:38:14.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:38:14.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac258cc700 1 Processor -- start 2022-07-21T01:38:14.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac258cc700 1 -- start start 2022-07-21T01:38:14.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac258cc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fac200b27e0 0x7fac20144240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac258cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200b1f70 0x7fac20140270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac258cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac200b2410 con 0x7fac20140760 2022-07-21T01:38:14.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac258cc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2013dfd0 con 0x7fac200b27e0 2022-07-21T01:38:14.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac258cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac20143860 con 0x7fac200b1f70 2022-07-21T01:38:14.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fac200b27e0 0x7fac20144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fac200b27e0 0x7fac20144240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45846/0 (socket says 172.21.15.3:45846) 2022-07-21T01:38:14.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1effd700 1 -- 172.21.15.3:0/615239064 learned_addr learned my addr 172.21.15.3:0/615239064 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:14.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1e7fc700 1 --2- 172.21.15.3:0/615239064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200b1f70 0x7fac20140270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1effd700 1 -- 172.21.15.3:0/615239064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200b1f70 msgr2=0x7fac20140270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:14.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1effd700 1 --2- 172.21.15.3:0/615239064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200b1f70 0x7fac20140270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1effd700 1 -- 172.21.15.3:0/615239064 >> v1:172.21.15.143:6789/0 conn(0x7fac20140760 legacy=0x7fac200ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1effd700 1 -- 172.21.15.3:0/615239064 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac20143b00 con 0x7fac200b27e0 2022-07-21T01:38:14.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1e7fc700 1 --2- 172.21.15.3:0/615239064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200b1f70 0x7fac20140270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:14.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.877+0000 7fac1effd700 1 --2- 172.21.15.3:0/615239064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b27e0 0x7fac20144240 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fac10002f70 tx=0x7fac10006eb0).ready entity=mon.0 client_cookie=21c53e2b31d49f5e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:14.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1dffb700 1 -- 172.21.15.3:0/615239064 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac1000f9c0 con 0x7fac200b27e0 2022-07-21T01:38:14.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1dffb700 1 -- 172.21.15.3:0/615239064 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac10000e90 con 0x7fac200b27e0 2022-07-21T01:38:14.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1dffb700 1 -- 172.21.15.3:0/615239064 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac1000fd50 con 0x7fac200b27e0 2022-07-21T01:38:14.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- 172.21.15.3:0/615239064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b27e0 msgr2=0x7fac20144240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 --2- 172.21.15.3:0/615239064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b27e0 0x7fac20144240 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fac10002f70 tx=0x7fac10006eb0).stop 2022-07-21T01:38:14.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- 172.21.15.3:0/615239064 shutdown_connections 2022-07-21T01:38:14.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 --2- 172.21.15.3:0/615239064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200b1f70 0x7fac20140270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 --2- 172.21.15.3:0/615239064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b27e0 0x7fac20144240 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- 172.21.15.3:0/615239064 >> 172.21.15.3:0/615239064 conn(0x7fac2005ceb0 msgr2=0x7fac20063130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:14.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- 172.21.15.3:0/615239064 shutdown_connections 2022-07-21T01:38:14.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- 172.21.15.3:0/615239064 wait complete. 2022-07-21T01:38:14.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 Processor -- start 2022-07-21T01:38:14.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- start start 2022-07-21T01:38:14.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b1f70 0x7fac200af620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200b27e0 0x7fac200b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac20140760 0x7fac200b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac200b4a30 con 0x7fac200b1f70 2022-07-21T01:38:14.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac20138340 con 0x7fac200b27e0 2022-07-21T01:38:14.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac200b4d20 con 0x7fac20140760 2022-07-21T01:38:14.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b1f70 0x7fac200af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b1f70 0x7fac200af620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45852/0 (socket says 172.21.15.3:45852) 2022-07-21T01:38:14.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1effd700 1 -- 172.21.15.3:0/1437697413 learned_addr learned my addr 172.21.15.3:0/1437697413 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:14.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1f7fe700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac20140760 0x7fac200b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1e7fc700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200b27e0 0x7fac200b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1effd700 1 -- 172.21.15.3:0/1437697413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac20140760 msgr2=0x7fac200b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1effd700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac20140760 0x7fac200b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1effd700 1 -- 172.21.15.3:0/1437697413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200b27e0 msgr2=0x7fac200b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:14.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1effd700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200b27e0 0x7fac200b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:14.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1effd700 1 -- 172.21.15.3:0/1437697413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1000b1e0 con 0x7fac200b1f70 2022-07-21T01:38:14.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac1effd700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b1f70 0x7fac200af620 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7fac10002f70 tx=0x7fac10017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:14.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac17fff700 1 -- 172.21.15.3:0/1437697413 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac10020070 con 0x7fac200b1f70 2022-07-21T01:38:14.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac17fff700 1 -- 172.21.15.3:0/1437697413 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1001a670 con 0x7fac200b1f70 2022-07-21T01:38:14.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.881+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2013e290 con 0x7fac200b1f70 2022-07-21T01:38:14.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.885+0000 7fac17fff700 1 -- 172.21.15.3:0/1437697413 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac1001a9d0 con 0x7fac200b1f70 2022-07-21T01:38:14.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.885+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2013a430 con 0x7fac200b1f70 2022-07-21T01:38:14.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.885+0000 7fac17fff700 1 -- 172.21.15.3:0/1437697413 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fac1001ab30 con 0x7fac200b1f70 2022-07-21T01:38:14.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.885+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac20143da0 con 0x7fac200b1f70 2022-07-21T01:38:14.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.885+0000 7fac17fff700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac04053950 0x7fac04055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:14.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.885+0000 7fac17fff700 1 -- 172.21.15.3:0/1437697413 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fac1006f850 con 0x7fac200b1f70 2022-07-21T01:38:14.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.885+0000 7fac1e7fc700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac04053950 0x7fac04055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:14.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.885+0000 7fac1e7fc700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac04053950 0x7fac04055e10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fac0800c000 tx=0x7fac08008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:14.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:14.889+0000 7fac17fff700 1 -- 172.21.15.3:0/1437697413 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fac10046610 con 0x7fac200b1f70 2022-07-21T01:38:15.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.049+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fac200574e0 con 0x7fac200b1f70 2022-07-21T01:38:15.691 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:38:15.693+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:38:15.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.705+0000 7fac17fff700 1 -- 172.21.15.3:0/1437697413 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) v1 ==== 161+0+0 (secure 0 0 0) 0x7fac10049c30 con 0x7fac200b1f70 2022-07-21T01:38:15.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac04053950 msgr2=0x7fac04055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:15.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac04053950 0x7fac04055e10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fac0800c000 tx=0x7fac08008040).stop 2022-07-21T01:38:15.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b1f70 msgr2=0x7fac200af620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:15.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b1f70 0x7fac200af620 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7fac10002f70 tx=0x7fac10017000).stop 2022-07-21T01:38:15.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 shutdown_connections 2022-07-21T01:38:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac20140760 0x7fac200b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac04053950 0x7fac04055e10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200b27e0 0x7fac200b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 --2- 172.21.15.3:0/1437697413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac200b1f70 0x7fac200af620 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 >> 172.21.15.3:0/1437697413 conn(0x7fac2005ceb0 msgr2=0x7fac201449f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 shutdown_connections 2022-07-21T01:38:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.709+0000 7fac258cc700 1 -- 172.21.15.3:0/1437697413 wait complete. 2022-07-21T01:38:15.712 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:38:15.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:15.744 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:38:15.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba3b26700 1 Processor -- start 2022-07-21T01:38:15.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba3b26700 1 -- start start 2022-07-21T01:38:15.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba3b26700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4b9c148530 0x7f4b9c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:15.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba3b26700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c0b04e0 con 0x7f4b9c0a7250 2022-07-21T01:38:15.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba3b26700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c0b0780 con 0x7f4b9c0b1f40 2022-07-21T01:38:15.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba3b26700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c0a9250 con 0x7f4b9c148530 2022-07-21T01:38:15.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba20c3700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4b9c0a7250 0x7f4b9c1448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54154/0 (socket says 172.21.15.3:54154) 2022-07-21T01:38:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba20c3700 1 -- 172.21.15.3:0/3061385396 learned_addr learned my addr 172.21.15.3:0/3061385396 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba10c1700 1 --2- 172.21.15.3:0/3061385396 >> v2:172.21.15.143:3300/0 conn(0x7f4b9c148530 0x7f4b9c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba08c0700 1 -- 172.21.15.3:0/3061385396 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57095031 0 0) 0x7f4b9c0b04e0 con 0x7f4b9c0a7250 2022-07-21T01:38:15.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba08c0700 1 -- 172.21.15.3:0/3061385396 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b80003260 con 0x7f4b9c0a7250 2022-07-21T01:38:15.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba08c0700 1 -- 172.21.15.3:0/3061385396 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513389274 0 0) 0x7f4b9c0b0780 con 0x7f4b9c0b1f40 2022-07-21T01:38:15.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba08c0700 1 -- 172.21.15.3:0/3061385396 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b9c0b04e0 con 0x7f4b9c0b1f40 2022-07-21T01:38:15.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba10c1700 1 -- 172.21.15.3:0/3061385396 >> v1:172.21.15.143:6789/0 conn(0x7f4b9c0b1f40 legacy=0x7f4b9c0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:15.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba10c1700 1 -- 172.21.15.3:0/3061385396 >> v1:172.21.15.3:6789/0 conn(0x7f4b9c0a7250 legacy=0x7f4b9c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:15.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba10c1700 1 -- 172.21.15.3:0/3061385396 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c0b0a20 con 0x7f4b9c148530 2022-07-21T01:38:15.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba10c1700 1 --2- 172.21.15.3:0/3061385396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 0x7f4b9c0b1160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f4b9c1478c0 tx=0x7f4b8c006df0).ready entity=mon.1 client_cookie=442cb7ae16ec6ed3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:15.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba08c0700 1 -- 172.21.15.3:0/3061385396 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b8c00d610 con 0x7f4b9c148530 2022-07-21T01:38:15.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba08c0700 1 -- 172.21.15.3:0/3061385396 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8c000e90 con 0x7f4b9c148530 2022-07-21T01:38:15.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.837+0000 7f4ba08c0700 1 -- 172.21.15.3:0/3061385396 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b8c00d960 con 0x7f4b9c148530 2022-07-21T01:38:15.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.841+0000 7f4ba3b26700 1 -- 172.21.15.3:0/3061385396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 msgr2=0x7f4b9c0b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:15.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.841+0000 7f4ba3b26700 1 --2- 172.21.15.3:0/3061385396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 0x7f4b9c0b1160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f4b9c1478c0 tx=0x7f4b8c006df0).stop 2022-07-21T01:38:15.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.841+0000 7f4ba3b26700 1 -- 172.21.15.3:0/3061385396 shutdown_connections 2022-07-21T01:38:15.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.841+0000 7f4ba3b26700 1 --2- 172.21.15.3:0/3061385396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 0x7f4b9c0b1160 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:15.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.841+0000 7f4ba3b26700 1 -- 172.21.15.3:0/3061385396 >> 172.21.15.3:0/3061385396 conn(0x7f4b9c05f840 msgr2=0x7f4b9c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:15.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.841+0000 7f4ba3b26700 1 -- 172.21.15.3:0/3061385396 shutdown_connections 2022-07-21T01:38:15.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.841+0000 7f4ba3b26700 1 -- 172.21.15.3:0/3061385396 wait complete. 2022-07-21T01:38:15.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.841+0000 7f4ba3b26700 1 Processor -- start 2022-07-21T01:38:15.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.841+0000 7f4ba3b26700 1 -- start start 2022-07-21T01:38:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba3b26700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b9c0a7250 0x7f4b9c0ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba3b26700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b9c0b1f40 0x7f4b9c14a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba3b26700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 0x7f4b9c14a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba3b26700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c149f80 con 0x7f4b9c0a7250 2022-07-21T01:38:15.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba3b26700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0ab270 con 0x7f4b9c148530 2022-07-21T01:38:15.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba3b26700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0ab510 con 0x7f4b9c0b1f40 2022-07-21T01:38:15.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba18c2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b9c0a7250 0x7f4b9c0ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:15.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba18c2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b9c0a7250 0x7f4b9c0ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45878/0 (socket says 172.21.15.3:45878) 2022-07-21T01:38:15.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba18c2700 1 -- 172.21.15.3:0/2568585529 learned_addr learned my addr 172.21.15.3:0/2568585529 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:15.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba10c1700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b9c0b1f40 0x7f4b9c14a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:15.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba20c3700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 0x7f4b9c14a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:15.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba18c2700 1 -- 172.21.15.3:0/2568585529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b9c0b1f40 msgr2=0x7f4b9c14a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:15.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba18c2700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b9c0b1f40 0x7f4b9c14a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:15.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba18c2700 1 -- 172.21.15.3:0/2568585529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 msgr2=0x7f4b9c14a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:15.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba18c2700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 0x7f4b9c14a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:15.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba18c2700 1 -- 172.21.15.3:0/2568585529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c00b180 con 0x7f4b9c0a7250 2022-07-21T01:38:15.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba20c3700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 0x7f4b9c14a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:15.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba18c2700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b9c0a7250 0x7f4b9c0ae290 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f4b98001490 tx=0x7f4b98007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:15.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4b92ffd700 1 -- 172.21.15.3:0/2568585529 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b98002240 con 0x7f4b9c0a7250 2022-07-21T01:38:15.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4b92ffd700 1 -- 172.21.15.3:0/2568585529 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b980023a0 con 0x7f4b9c0a7250 2022-07-21T01:38:15.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c0ab7b0 con 0x7f4b9c0a7250 2022-07-21T01:38:15.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4b92ffd700 1 -- 172.21.15.3:0/2568585529 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b98001650 con 0x7f4b9c0a7250 2022-07-21T01:38:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9c0abcf0 con 0x7f4b9c0a7250 2022-07-21T01:38:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4b92ffd700 1 -- 172.21.15.3:0/2568585529 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4b98010030 con 0x7f4b9c0a7250 2022-07-21T01:38:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c0ac110 con 0x7f4b9c0a7250 2022-07-21T01:38:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4b92ffd700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4b84053b30 0x7f4b84055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4b92ffd700 1 -- 172.21.15.3:0/2568585529 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f4b9806c080 con 0x7f4b9c0a7250 2022-07-21T01:38:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.845+0000 7f4ba10c1700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4b84053b30 0x7f4b84055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.849+0000 7f4ba10c1700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4b84053b30 0x7f4b84055ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4b8c018000 tx=0x7f4b8c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:15.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:15.853+0000 7f4b92ffd700 1 -- 172.21.15.3:0/2568585529 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4b98042d50 con 0x7f4b9c0a7250 2022-07-21T01:38:16.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.013+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b9c0574e0 con 0x7f4b9c0a7250 2022-07-21T01:38:16.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4b92ffd700 1 -- 172.21.15.3:0/2568585529 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2125 (secure 0 0 0) 0x7f4b98046370 con 0x7f4b9c0a7250 2022-07-21T01:38:16.016 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:16.016 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:36:52.675976+0000","modified":"2022-07-21T01:38:15.694395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-21T01:38:16.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4b84053b30 msgr2=0x7f4b84055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4b84053b30 0x7f4b84055ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4b8c018000 tx=0x7f4b8c014000).stop 2022-07-21T01:38:16.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b9c0a7250 msgr2=0x7f4b9c0ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b9c0a7250 0x7f4b9c0ae290 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f4b98001490 tx=0x7f4b98007e10).stop 2022-07-21T01:38:16.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 shutdown_connections 2022-07-21T01:38:16.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4b84053b30 0x7f4b84055ff0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9c148530 0x7f4b9c14a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b9c0b1f40 0x7f4b9c14a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 --2- 172.21.15.3:0/2568585529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b9c0a7250 0x7f4b9c0ae290 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 >> 172.21.15.3:0/2568585529 conn(0x7f4b9c05f840 msgr2=0x7f4b9c1449c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:16.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 shutdown_connections 2022-07-21T01:38:16.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.017+0000 7f4ba3b26700 1 -- 172.21.15.3:0/2568585529 wait complete. 2022-07-21T01:38:16.024 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 24 2022-07-21T01:38:16.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:38:16.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.133+0000 7f7fdf5ba700 1 Processor -- start 2022-07-21T01:38:16.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.133+0000 7f7fdf5ba700 1 -- start start 2022-07-21T01:38:16.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fdf5ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8060290 0x7f7fd8064ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fdf5ba700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd8060670 con 0x7f7fd80b3a60 2022-07-21T01:38:16.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fdf5ba700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd80b2540 con 0x7f7fd81445d0 2022-07-21T01:38:16.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fdf5ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8065e10 con 0x7f7fd8060290 2022-07-21T01:38:16.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fddb57700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7fd80b3a60 0x7f7fd814b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54170/0 (socket says 172.21.15.3:54170) 2022-07-21T01:38:16.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fddb57700 1 -- 172.21.15.3:0/1550245891 learned_addr learned my addr 172.21.15.3:0/1550245891 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:16.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fdd356700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8060290 0x7f7fd8064ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112993396 0 0) 0x7f7fd8060670 con 0x7f7fd80b3a60 2022-07-21T01:38:16.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fbc003260 con 0x7f7fd80b3a60 2022-07-21T01:38:16.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579114653 0 0) 0x7f7fd80b2540 con 0x7f7fd81445d0 2022-07-21T01:38:16.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd8060670 con 0x7f7fd81445d0 2022-07-21T01:38:16.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1434226850 0 0) 0x7f7fbc003260 con 0x7f7fd80b3a60 2022-07-21T01:38:16.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8060290 msgr2=0x7f7fd8064ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:16.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8060290 0x7f7fd8064ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 >> v1:172.21.15.143:6789/0 conn(0x7f7fd81445d0 legacy=0x7f7fd8144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd80b27e0 con 0x7f7fd80b3a60 2022-07-21T01:38:16.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f7fd0001660 con 0x7f7fd80b3a60 2022-07-21T01:38:16.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 >> v1:172.21.15.3:6789/0 conn(0x7f7fd80b3a60 legacy=0x7f7fd814b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fbc003910 0x7f7fbc003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fbc004500 0x7f7fbc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fbc004e70 0x7f7fbc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8065e10 con 0x7f7fbc004e70 2022-07-21T01:38:16.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc00bc40 con 0x7f7fbc003910 2022-07-21T01:38:16.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc00be00 con 0x7f7fbc004500 2022-07-21T01:38:16.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fdd356700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8060290 0x7f7fd8064ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:16.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fddb57700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fbc004e70 0x7f7fbc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fdcb55700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fbc004500 0x7f7fbc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fdd356700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fbc003910 0x7f7fbc003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fddb57700 1 -- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fbc004500 msgr2=0x7f7fbc004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:16.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fddb57700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fbc004500 0x7f7fbc004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fddb57700 1 -- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fbc003910 msgr2=0x7f7fbc003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:16.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fddb57700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fbc003910 0x7f7fbc003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fddb57700 1 -- 172.21.15.3:0/1550245891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc003260 con 0x7f7fbc004e70 2022-07-21T01:38:16.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fddb57700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fbc004e70 0x7f7fbc00b750 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f7fd0001430 tx=0x7f7fd0001dd0).ready entity=mon.0 client_cookie=c11b14b7ad713a1c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:16.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd00070d0 con 0x7f7fbc004e70 2022-07-21T01:38:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0007230 con 0x7f7fbc004e70 2022-07-21T01:38:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.137+0000 7f7fcffff700 1 -- 172.21.15.3:0/1550245891 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd000c8c0 con 0x7f7fbc004e70 2022-07-21T01:38:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.141+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/1550245891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fbc004e70 msgr2=0x7f7fbc00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.141+0000 7f7fdf5ba700 1 --2- 172.21.15.3:0/1550245891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fbc004e70 0x7f7fbc00b750 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f7fd0001430 tx=0x7f7fd0001dd0).stop 2022-07-21T01:38:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.141+0000 7f7fddb57700 1 -- 172.21.15.3:0/1550245891 reap_dead start 2022-07-21T01:38:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.141+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/1550245891 shutdown_connections 2022-07-21T01:38:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.141+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/1550245891 >> 172.21.15.3:0/1550245891 conn(0x7f7fd805d0f0 msgr2=0x7f7fd805f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.141+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/1550245891 shutdown_connections 2022-07-21T01:38:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.141+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/1550245891 wait complete. 2022-07-21T01:38:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.141+0000 7f7fdf5ba700 1 Processor -- start 2022-07-21T01:38:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.141+0000 7f7fdf5ba700 1 -- start start 2022-07-21T01:38:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdf5ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8141420 0x7f7fd813cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdf5ba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd80ae8b0 0x7f7fd813d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdf5ba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd80b05d0 0x7f7fd8153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdf5ba700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80b14c0 con 0x7f7fd80b05d0 2022-07-21T01:38:16.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdf5ba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd813e8c0 con 0x7f7fd80ae8b0 2022-07-21T01:38:16.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdf5ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8141800 con 0x7f7fd8141420 2022-07-21T01:38:16.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fddb57700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd80b05d0 0x7f7fd8153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdcb55700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd80ae8b0 0x7f7fd813d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdd356700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8141420 0x7f7fd813cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdcb55700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd80ae8b0 0x7f7fd813d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:51348/0 (socket says 172.21.15.3:51348) 2022-07-21T01:38:16.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fddb57700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd80b05d0 0x7f7fd8153d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45902/0 (socket says 172.21.15.3:45902) 2022-07-21T01:38:16.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdcb55700 1 -- 172.21.15.3:0/3471410599 learned_addr learned my addr 172.21.15.3:0/3471410599 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:16.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fddb57700 1 -- 172.21.15.3:0/3471410599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8141420 msgr2=0x7f7fd813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fddb57700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8141420 0x7f7fd813cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fddb57700 1 -- 172.21.15.3:0/3471410599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd80ae8b0 msgr2=0x7f7fd813d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fddb57700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd80ae8b0 0x7f7fd813d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fddb57700 1 -- 172.21.15.3:0/3471410599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd80aef40 con 0x7f7fd80b05d0 2022-07-21T01:38:16.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fddb57700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd80b05d0 0x7f7fd8153d60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f7fd8150840 tx=0x7f7fd0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:16.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fce7fc700 1 -- 172.21.15.3:0/3471410599 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd00070d0 con 0x7f7fd80b05d0 2022-07-21T01:38:16.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fce7fc700 1 -- 172.21.15.3:0/3471410599 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0000e40 con 0x7f7fd80b05d0 2022-07-21T01:38:16.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd813e620 con 0x7f7fd80b05d0 2022-07-21T01:38:16.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fce7fc700 1 -- 172.21.15.3:0/3471410599 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd00047d0 con 0x7f7fd80b05d0 2022-07-21T01:38:16.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd80b02e0 con 0x7f7fd80b05d0 2022-07-21T01:38:16.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.145+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd80574e0 con 0x7f7fd80b05d0 2022-07-21T01:38:16.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.149+0000 7f7fce7fc700 1 -- 172.21.15.3:0/3471410599 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7fd0010070 con 0x7f7fd80b05d0 2022-07-21T01:38:16.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.149+0000 7f7fce7fc700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7fc0053a10 0x7f7fc0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.149+0000 7f7fce7fc700 1 -- 172.21.15.3:0/3471410599 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f7fd006b760 con 0x7f7fd80b05d0 2022-07-21T01:38:16.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.149+0000 7f7fdd356700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7fc0053a10 0x7f7fc0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.149+0000 7f7fdd356700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7fc0053a10 0x7f7fc0055ed0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f7fc800b000 tx=0x7f7fc8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:16.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.153+0000 7f7fce7fc700 1 -- 172.21.15.3:0/3471410599 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7fd00423c0 con 0x7f7fd80b05d0 2022-07-21T01:38:16.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.317+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7fd8069a70 con 0x7f7fd80b05d0 2022-07-21T01:38:16.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.709+0000 7f7fce7fc700 1 -- 172.21.15.3:0/3471410599 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7fd00459e0 con 0x7f7fd80b05d0 2022-07-21T01:38:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7fc0053a10 msgr2=0x7f7fc0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7fc0053a10 0x7f7fc0055ed0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f7fc800b000 tx=0x7f7fc8008040).stop 2022-07-21T01:38:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd80b05d0 msgr2=0x7f7fd8153d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd80b05d0 0x7f7fd8153d60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f7fd8150840 tx=0x7f7fd0015000).stop 2022-07-21T01:38:16.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 shutdown_connections 2022-07-21T01:38:16.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd80b05d0 0x7f7fd8153d60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7fc0053a10 0x7f7fc0055ed0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fd80ae8b0 0x7f7fd813d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 --2- 172.21.15.3:0/3471410599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fd8141420 0x7f7fd813cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 >> 172.21.15.3:0/3471410599 conn(0x7f7fd805d0f0 msgr2=0x7f7fd80b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:16.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 shutdown_connections 2022-07-21T01:38:16.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.713+0000 7f7fdf5ba700 1 -- 172.21.15.3:0/3471410599 wait complete. 2022-07-21T01:38:16.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:38:16.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f0689766700 1 Processor -- start 2022-07-21T01:38:16.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f0689766700 1 -- start start 2022-07-21T01:38:16.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f0689766700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06840b6710 0x7f06840b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f0689766700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f06840b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f0689766700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0684144240 con 0x7f0684061ec0 2022-07-21T01:38:16.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f0689766700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06840b6fe0 con 0x7f06840b6710 2022-07-21T01:38:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f0689766700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0684143d00 con 0x7f06840b4550 2022-07-21T01:38:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f06827fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f06840b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f06827fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f06840b6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:52826/0 (socket says 172.21.15.3:52826) 2022-07-21T01:38:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f06827fc700 1 -- 172.21.15.3:0/3919668268 learned_addr learned my addr 172.21.15.3:0/3919668268 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.861+0000 7f0682ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06840b6710 0x7f06840b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f06827fc700 1 -- 172.21.15.3:0/3919668268 >> v2:172.21.15.143:3300/0 conn(0x7f06840b6710 msgr2=0x7f06840b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:16.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f06827fc700 1 --2- 172.21.15.3:0/3919668268 >> v2:172.21.15.143:3300/0 conn(0x7f06840b6710 0x7f06840b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f06827fc700 1 -- 172.21.15.3:0/3919668268 >> v1:172.21.15.143:6789/0 conn(0x7f0684061ec0 legacy=0x7f0684146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0681ffb700 1 -- 172.21.15.3:0/3919668268 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162011726 0 0) 0x7f0684144240 con 0x7f0684061ec0 2022-07-21T01:38:16.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f06827fc700 1 -- 172.21.15.3:0/3919668268 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684143fa0 con 0x7f06840b4550 2022-07-21T01:38:16.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f06827fc700 1 --2- 172.21.15.3:0/3919668268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f06840b6af0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f0674002f70 tx=0x7f0674006eb0).ready entity=mon.2 client_cookie=a156a7ddfaf8e6b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:16.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0681ffb700 1 -- 172.21.15.3:0/3919668268 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f067400d570 con 0x7f06840b4550 2022-07-21T01:38:16.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0681ffb700 1 -- 172.21.15.3:0/3919668268 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0674000e90 con 0x7f06840b4550 2022-07-21T01:38:16.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0681ffb700 1 -- 172.21.15.3:0/3919668268 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f067400d8c0 con 0x7f06840b4550 2022-07-21T01:38:16.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 -- 172.21.15.3:0/3919668268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 msgr2=0x7f06840b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 --2- 172.21.15.3:0/3919668268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f06840b6af0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f0674002f70 tx=0x7f0674006eb0).stop 2022-07-21T01:38:16.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 -- 172.21.15.3:0/3919668268 shutdown_connections 2022-07-21T01:38:16.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 --2- 172.21.15.3:0/3919668268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f06840b6af0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 --2- 172.21.15.3:0/3919668268 >> v2:172.21.15.143:3300/0 conn(0x7f06840b6710 0x7f06840b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 -- 172.21.15.3:0/3919668268 >> 172.21.15.3:0/3919668268 conn(0x7f068405c620 msgr2=0x7f068405ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:16.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 -- 172.21.15.3:0/3919668268 shutdown_connections 2022-07-21T01:38:16.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 -- 172.21.15.3:0/3919668268 wait complete. 2022-07-21T01:38:16.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 Processor -- start 2022-07-21T01:38:16.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.865+0000 7f0689766700 1 -- start start 2022-07-21T01:38:16.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0689766700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0684061ec0 0x7f0684152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0689766700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f0684153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0689766700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06840b6710 0x7f0684144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0689766700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06841526b0 con 0x7f0684061ec0 2022-07-21T01:38:16.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0689766700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0684153610 con 0x7f06840b6710 2022-07-21T01:38:16.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0689766700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06840b0530 con 0x7f06840b4550 2022-07-21T01:38:16.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0682ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0684061ec0 0x7f0684152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0682ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0684061ec0 0x7f0684152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45914/0 (socket says 172.21.15.3:45914) 2022-07-21T01:38:16.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0682ffd700 1 -- 172.21.15.3:0/2837460025 learned_addr learned my addr 172.21.15.3:0/2837460025 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:16.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f06827fc700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f0684153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f06837fe700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06840b6710 0x7f0684144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0682ffd700 1 -- 172.21.15.3:0/2837460025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 msgr2=0x7f0684153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0682ffd700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f0684153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0682ffd700 1 -- 172.21.15.3:0/2837460025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06840b6710 msgr2=0x7f0684144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:16.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0682ffd700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06840b6710 0x7f0684144680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:16.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0682ffd700 1 -- 172.21.15.3:0/2837460025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067400b1e0 con 0x7f0684061ec0 2022-07-21T01:38:16.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0682ffd700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0684061ec0 0x7f0684152c30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f066c00c000 tx=0x7f066c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:16.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f067bfff700 1 -- 172.21.15.3:0/2837460025 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f066c011750 con 0x7f0684061ec0 2022-07-21T01:38:16.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f067bfff700 1 -- 172.21.15.3:0/2837460025 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c0118b0 con 0x7f0684061ec0 2022-07-21T01:38:16.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06841528a0 con 0x7f0684061ec0 2022-07-21T01:38:16.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f067bfff700 1 -- 172.21.15.3:0/2837460025 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f066c011c10 con 0x7f0684061ec0 2022-07-21T01:38:16.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.869+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06840b7940 con 0x7f0684061ec0 2022-07-21T01:38:16.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.873+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06840b0670 con 0x7f0684061ec0 2022-07-21T01:38:16.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.873+0000 7f067bfff700 1 -- 172.21.15.3:0/2837460025 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f066c011d70 con 0x7f0684061ec0 2022-07-21T01:38:16.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.873+0000 7f067bfff700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0668053a70 0x7f0668055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:16.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.873+0000 7f067bfff700 1 -- 172.21.15.3:0/2837460025 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f066c06c4c0 con 0x7f0684061ec0 2022-07-21T01:38:16.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.873+0000 7f06827fc700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0668053a70 0x7f0668055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:16.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.873+0000 7f06827fc700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0668053a70 0x7f0668055f30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f067401c000 tx=0x7f0674015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:16.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:16.881+0000 7f067bfff700 1 -- 172.21.15.3:0/2837460025 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f066c043230 con 0x7f0684061ec0 2022-07-21T01:38:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.001+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0684061780 con 0x7f0684061ec0 2022-07-21T01:38:17.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.713+0000 7f067bfff700 1 -- 172.21.15.3:0/2837460025 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v60) v1 ==== 185+0+0 (secure 0 0 0) 0x7f066c046850 con 0x7f0684061ec0 2022-07-21T01:38:17.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0668053a70 msgr2=0x7f0668055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:17.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0668053a70 0x7f0668055f30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f067401c000 tx=0x7f0674015000).stop 2022-07-21T01:38:17.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0684061ec0 msgr2=0x7f0684152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:17.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0684061ec0 0x7f0684152c30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f066c00c000 tx=0x7f066c006df0).stop 2022-07-21T01:38:17.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 shutdown_connections 2022-07-21T01:38:17.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0668053a70 0x7f0668055f30 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0x7f067401c000 tx=0x7f0674015000).stop 2022-07-21T01:38:17.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06840b6710 0x7f0684144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:17.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06840b4550 0x7f0684153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:17.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 --2- 172.21.15.3:0/2837460025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0684061ec0 0x7f0684152c30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:17.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.717+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 >> 172.21.15.3:0/2837460025 conn(0x7f068405c620 msgr2=0x7f06841299e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:17.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.721+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 shutdown_connections 2022-07-21T01:38:17.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.721+0000 7f0689766700 1 -- 172.21.15.3:0/2837460025 wait complete. 2022-07-21T01:38:17.725 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:38:17.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:38:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.853+0000 7f8e2d518700 1 Processor -- start 2022-07-21T01:38:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.853+0000 7f8e2d518700 1 -- start start 2022-07-21T01:38:17.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e2d518700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e280b0560 0x7f8e28146df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:17.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e2d518700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e280b12d0 con 0x7f8e280a9320 2022-07-21T01:38:17.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e2d518700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e280a9700 con 0x7f8e280b0d10 2022-07-21T01:38:17.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e2d518700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e280b1500 con 0x7f8e280b0560 2022-07-21T01:38:17.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e27fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8e280a9320 0x7f8e281466e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54202/0 (socket says 172.21.15.3:54202) 2022-07-21T01:38:17.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e27fff700 1 -- 172.21.15.3:0/3480539122 learned_addr learned my addr 172.21.15.3:0/3480539122 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:17.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e2cd17700 1 --2- 172.21.15.3:0/3480539122 >> v2:172.21.15.143:3300/0 conn(0x7f8e280b0560 0x7f8e28146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:17.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e277fe700 1 -- 172.21.15.3:0/3480539122 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590780998 0 0) 0x7f8e280b12d0 con 0x7f8e280a9320 2022-07-21T01:38:17.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e277fe700 1 -- 172.21.15.3:0/3480539122 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e08003260 con 0x7f8e280a9320 2022-07-21T01:38:17.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e277fe700 1 -- 172.21.15.3:0/3480539122 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593570105 0 0) 0x7f8e280a9700 con 0x7f8e280b0d10 2022-07-21T01:38:17.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e277fe700 1 -- 172.21.15.3:0/3480539122 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e280b12d0 con 0x7f8e280b0d10 2022-07-21T01:38:17.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e2cd17700 1 -- 172.21.15.3:0/3480539122 >> v1:172.21.15.143:6789/0 conn(0x7f8e280b0d10 legacy=0x7f8e28140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:17.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e2cd17700 1 -- 172.21.15.3:0/3480539122 >> v1:172.21.15.3:6789/0 conn(0x7f8e280a9320 legacy=0x7f8e281466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:17.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e2cd17700 1 -- 172.21.15.3:0/3480539122 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e280b17a0 con 0x7f8e280b0560 2022-07-21T01:38:17.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e2cd17700 1 --2- 172.21.15.3:0/3480539122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e280b0560 0x7f8e28146df0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f8e200032f0 tx=0x7f8e200091d0).ready entity=mon.1 client_cookie=9556266fc094d11a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:17.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.857+0000 7f8e277fe700 1 -- 172.21.15.3:0/3480539122 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e2000b600 con 0x7f8e280b0560 2022-07-21T01:38:17.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e277fe700 1 -- 172.21.15.3:0/3480539122 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2000b760 con 0x7f8e280b0560 2022-07-21T01:38:17.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e277fe700 1 -- 172.21.15.3:0/3480539122 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e2000b8d0 con 0x7f8e280b0560 2022-07-21T01:38:17.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e2d518700 1 -- 172.21.15.3:0/3480539122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e280b0560 msgr2=0x7f8e28146df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:17.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e2d518700 1 --2- 172.21.15.3:0/3480539122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e280b0560 0x7f8e28146df0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f8e200032f0 tx=0x7f8e200091d0).stop 2022-07-21T01:38:17.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e2d518700 1 -- 172.21.15.3:0/3480539122 shutdown_connections 2022-07-21T01:38:17.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e2d518700 1 --2- 172.21.15.3:0/3480539122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e280b0560 0x7f8e28146df0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:17.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e2d518700 1 -- 172.21.15.3:0/3480539122 >> 172.21.15.3:0/3480539122 conn(0x7f8e280612d0 msgr2=0x7f8e28062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:17.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e2d518700 1 -- 172.21.15.3:0/3480539122 shutdown_connections 2022-07-21T01:38:17.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e2d518700 1 -- 172.21.15.3:0/3480539122 wait complete. 2022-07-21T01:38:17.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.861+0000 7f8e2d518700 1 Processor -- start 2022-07-21T01:38:17.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2d518700 1 -- start start 2022-07-21T01:38:17.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2d518700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8e280a9320 0x7f8e280b5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:17.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2d518700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e280b0560 0x7f8e280aa750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:17.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2d518700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e280b0d10 0x7f8e2813afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:17.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2d518700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e280a9a40 con 0x7f8e280a9320 2022-07-21T01:38:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2d518700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e280b3940 con 0x7f8e280b0560 2022-07-21T01:38:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2d518700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e280ab070 con 0x7f8e280b0d10 2022-07-21T01:38:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e27fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8e280a9320 0x7f8e280b5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e27fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8e280a9320 0x7f8e280b5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45928/0 (socket says 172.21.15.3:45928) 2022-07-21T01:38:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e27fff700 1 -- 172.21.15.3:0/242136412 learned_addr learned my addr 172.21.15.3:0/242136412 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2cd17700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e280b0d10 0x7f8e2813afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e1ffff700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e280b0560 0x7f8e280aa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:17.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e27fff700 1 -- 172.21.15.3:0/242136412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e280b0d10 msgr2=0x7f8e2813afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:17.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e27fff700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e280b0d10 0x7f8e2813afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:17.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e27fff700 1 -- 172.21.15.3:0/242136412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e280b0560 msgr2=0x7f8e280aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:17.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e27fff700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e280b0560 0x7f8e280aa750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:17.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e27fff700 1 -- 172.21.15.3:0/242136412 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e20009470 con 0x7f8e280a9320 2022-07-21T01:38:17.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e27fff700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8e280a9320 0x7f8e280b5fe0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f8e180011d0 tx=0x7f8e18007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:17.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e25ffb700 1 -- 172.21.15.3:0/242136412 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e1800ce50 con 0x7f8e280a9320 2022-07-21T01:38:17.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e25ffb700 1 -- 172.21.15.3:0/242136412 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e180070d0 con 0x7f8e280a9320 2022-07-21T01:38:17.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e280ab310 con 0x7f8e280a9320 2022-07-21T01:38:17.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e25ffb700 1 -- 172.21.15.3:0/242136412 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e18007430 con 0x7f8e280a9320 2022-07-21T01:38:17.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.865+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e280b5960 con 0x7f8e280a9320 2022-07-21T01:38:17.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.869+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e280574e0 con 0x7f8e280a9320 2022-07-21T01:38:17.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.869+0000 7f8e25ffb700 1 -- 172.21.15.3:0/242136412 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8e18010070 con 0x7f8e280a9320 2022-07-21T01:38:17.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.869+0000 7f8e25ffb700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8e0c053950 0x7f8e0c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:17.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.869+0000 7f8e25ffb700 1 -- 172.21.15.3:0/242136412 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f8e1806b560 con 0x7f8e280a9320 2022-07-21T01:38:17.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.869+0000 7f8e1ffff700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8e0c053950 0x7f8e0c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:17.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.869+0000 7f8e1ffff700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8e0c053950 0x7f8e0c055e10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8e1000d000 tx=0x7f8e10007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:17.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:17.881+0000 7f8e25ffb700 1 -- 172.21.15.3:0/242136412 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8e180423e0 con 0x7f8e280a9320 2022-07-21T01:38:18.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.009+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8e281475c0 con 0x7f8e280a9320 2022-07-21T01:38:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.721+0000 7f8e25ffb700 1 -- 172.21.15.3:0/242136412 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v61) v1 ==== 173+0+0 (secure 0 0 0) 0x7f8e18045a00 con 0x7f8e280a9320 2022-07-21T01:38:18.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8e0c053950 msgr2=0x7f8e0c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:18.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8e0c053950 0x7f8e0c055e10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8e1000d000 tx=0x7f8e10007a00).stop 2022-07-21T01:38:18.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8e280a9320 msgr2=0x7f8e280b5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:18.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8e280a9320 0x7f8e280b5fe0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f8e180011d0 tx=0x7f8e18007b50).stop 2022-07-21T01:38:18.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 shutdown_connections 2022-07-21T01:38:18.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e280b0d10 0x7f8e2813afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:18.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8e0c053950 0x7f8e0c055e10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:18.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e280b0560 0x7f8e280aa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 --2- 172.21.15.3:0/242136412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8e280a9320 0x7f8e280b5fe0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 >> 172.21.15.3:0/242136412 conn(0x7f8e280612d0 msgr2=0x7f8e28141010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 shutdown_connections 2022-07-21T01:38:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.729+0000 7f8e2d518700 1 -- 172.21.15.3:0/242136412 wait complete. 2022-07-21T01:38:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:38:18.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:38:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f98948cd700 1 Processor -- start 2022-07-21T01:38:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f98948cd700 1 -- start start 2022-07-21T01:38:18.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f98948cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98900b07a0 0x7f98900af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f98948cd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9890147730 0x7f98900acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f98948cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98900b5a10 con 0x7f98900b5630 2022-07-21T01:38:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f98948cd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9890145470 con 0x7f9890147730 2022-07-21T01:38:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f98948cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98900ae180 con 0x7f98900b07a0 2022-07-21T01:38:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f988ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9890147730 0x7f98900acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f988ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9890147730 0x7f98900acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45940/0 (socket says 172.21.15.3:45940) 2022-07-21T01:38:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f988ffff700 1 -- 172.21.15.3:0/1066081804 learned_addr learned my addr 172.21.15.3:0/1066081804 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.861+0000 7f988effd700 1 --2- 172.21.15.3:0/1066081804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98900b07a0 0x7f98900af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f988ffff700 1 -- 172.21.15.3:0/1066081804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98900b07a0 msgr2=0x7f98900af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f988ffff700 1 --2- 172.21.15.3:0/1066081804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98900b07a0 0x7f98900af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:18.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f988ffff700 1 -- 172.21.15.3:0/1066081804 >> v1:172.21.15.143:6789/0 conn(0x7f98900b5630 legacy=0x7f9890067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:18.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f988ffff700 1 -- 172.21.15.3:0/1066081804 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900ac990 con 0x7f9890147730 2022-07-21T01:38:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f988ffff700 1 --2- 172.21.15.3:0/1066081804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9890147730 0x7f98900acdf0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f98880032f0 tx=0x7f9888009290).ready entity=mon.0 client_cookie=63b335498615bf8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f988e7fc700 1 -- 172.21.15.3:0/1066081804 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f988800b8a0 con 0x7f9890147730 2022-07-21T01:38:18.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f988e7fc700 1 -- 172.21.15.3:0/1066081804 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988800ba00 con 0x7f9890147730 2022-07-21T01:38:18.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f988e7fc700 1 -- 172.21.15.3:0/1066081804 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f988800bb70 con 0x7f9890147730 2022-07-21T01:38:18.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f98948cd700 1 -- 172.21.15.3:0/1066081804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9890147730 msgr2=0x7f98900acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:18.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f98948cd700 1 --2- 172.21.15.3:0/1066081804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9890147730 0x7f98900acdf0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f98880032f0 tx=0x7f9888009290).stop 2022-07-21T01:38:18.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f98948cd700 1 -- 172.21.15.3:0/1066081804 shutdown_connections 2022-07-21T01:38:18.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f98948cd700 1 --2- 172.21.15.3:0/1066081804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9890147730 0x7f98900acdf0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:18.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f98948cd700 1 --2- 172.21.15.3:0/1066081804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98900b07a0 0x7f98900af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:18.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f98948cd700 1 -- 172.21.15.3:0/1066081804 >> 172.21.15.3:0/1066081804 conn(0x7f98900626f0 msgr2=0x7f9890062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:18.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f98948cd700 1 -- 172.21.15.3:0/1066081804 shutdown_connections 2022-07-21T01:38:18.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f98948cd700 1 -- 172.21.15.3:0/1066081804 wait complete. 2022-07-21T01:38:18.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.865+0000 7f98948cd700 1 Processor -- start 2022-07-21T01:38:18.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f98948cd700 1 -- start start 2022-07-21T01:38:18.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f98948cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98900b07a0 0x7f98900b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:18.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f98948cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98900b5630 0x7f9890156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:18.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f98948cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9890147730 0x7f98901572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:18.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f98948cd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9890063380 con 0x7f98900b5630 2022-07-21T01:38:18.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f98948cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9890063620 con 0x7f98900b07a0 2022-07-21T01:38:18.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f98948cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98900638c0 con 0x7f9890147730 2022-07-21T01:38:18.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98900b5630 0x7f9890156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:18.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98900b07a0 0x7f98900b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:18.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98900b5630 0x7f9890156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45944/0 (socket says 172.21.15.3:45944) 2022-07-21T01:38:18.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988effd700 1 -- 172.21.15.3:0/3192173500 learned_addr learned my addr 172.21.15.3:0/3192173500 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:18.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988ffff700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9890147730 0x7f98901572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:18.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988effd700 1 -- 172.21.15.3:0/3192173500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9890147730 msgr2=0x7f98901572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:18.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988effd700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9890147730 0x7f98901572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:18.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988effd700 1 -- 172.21.15.3:0/3192173500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98900b07a0 msgr2=0x7f98900b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:18.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988effd700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98900b07a0 0x7f98900b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:18.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988effd700 1 -- 172.21.15.3:0/3192173500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9888009530 con 0x7f98900b5630 2022-07-21T01:38:18.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988effd700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98900b5630 0x7f9890156e00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f988000c000 tx=0x7f9880006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:18.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988cff9700 1 -- 172.21.15.3:0/3192173500 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9880010780 con 0x7f98900b5630 2022-07-21T01:38:18.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988cff9700 1 -- 172.21.15.3:0/3192173500 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98800108e0 con 0x7f98900b5630 2022-07-21T01:38:18.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9890063b60 con 0x7f98900b5630 2022-07-21T01:38:18.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f988cff9700 1 -- 172.21.15.3:0/3192173500 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9880010c40 con 0x7f98900b5630 2022-07-21T01:38:18.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.869+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9890064170 con 0x7f98900b5630 2022-07-21T01:38:18.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.873+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98900574e0 con 0x7f98900b5630 2022-07-21T01:38:18.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.873+0000 7f988cff9700 1 -- 172.21.15.3:0/3192173500 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f988001c460 con 0x7f98900b5630 2022-07-21T01:38:18.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.873+0000 7f988cff9700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9870053aa0 0x7f9870055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:18.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.873+0000 7f988cff9700 1 -- 172.21.15.3:0/3192173500 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f988006c280 con 0x7f98900b5630 2022-07-21T01:38:18.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.873+0000 7f988f7fe700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9870053aa0 0x7f9870055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:18.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.873+0000 7f988f7fe700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9870053aa0 0x7f9870055f60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f988400d000 tx=0x7f9884007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:18.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:18.877+0000 7f988cff9700 1 -- 172.21.15.3:0/3192173500 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9880043130 con 0x7f98900b5630 2022-07-21T01:38:19.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.009+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9890061020 con 0x7f98900b5630 2022-07-21T01:38:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.013+0000 7f988cff9700 1 -- 172.21.15.3:0/3192173500 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9880046750 con 0x7f98900b5630 2022-07-21T01:38:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9870053aa0 msgr2=0x7f9870055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9870053aa0 0x7f9870055f60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f988400d000 tx=0x7f9884007b90).stop 2022-07-21T01:38:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98900b5630 msgr2=0x7f9890156e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98900b5630 0x7f9890156e00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f988000c000 tx=0x7f9880006df0).stop 2022-07-21T01:38:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 shutdown_connections 2022-07-21T01:38:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9890147730 0x7f98901572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98900b5630 0x7f9890156e00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9870053aa0 0x7f9870055f60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 --2- 172.21.15.3:0/3192173500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98900b07a0 0x7f98900b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 >> 172.21.15.3:0/3192173500 conn(0x7f98900626f0 msgr2=0x7f9890067500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:19.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 shutdown_connections 2022-07-21T01:38:19.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.017+0000 7f98948cd700 1 -- 172.21.15.3:0/3192173500 wait complete. 2022-07-21T01:38:19.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-07-21T01:38:19.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 Processor -- start 2022-07-21T01:38:19.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 -- start start 2022-07-21T01:38:19.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab240b6510 0x7fab2413d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fab240b0e60 0x7fab2413d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fab24064500 0x7fab240609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2413dc80 con 0x7fab24064500 2022-07-21T01:38:19.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab24066af0 con 0x7fab240b0e60 2022-07-21T01:38:19.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab24066d90 con 0x7fab240b6510 2022-07-21T01:38:19.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab23fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fab24064500 0x7fab240609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab237fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab240b6510 0x7fab2413d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab23fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fab24064500 0x7fab240609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45954/0 (socket says 172.21.15.3:45954) 2022-07-21T01:38:19.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab23fff700 1 -- 172.21.15.3:0/102773561 learned_addr learned my addr 172.21.15.3:0/102773561 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:19.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab22ffd700 1 --2- 172.21.15.3:0/102773561 >> v2:172.21.15.143:3300/0 conn(0x7fab240b0e60 0x7fab2413d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab23fff700 1 -- 172.21.15.3:0/102773561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab240b6510 msgr2=0x7fab2413d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab23fff700 1 --2- 172.21.15.3:0/102773561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab240b6510 0x7fab2413d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab23fff700 1 -- 172.21.15.3:0/102773561 >> v2:172.21.15.143:3300/0 conn(0x7fab240b0e60 msgr2=0x7fab2413d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab23fff700 1 --2- 172.21.15.3:0/102773561 >> v2:172.21.15.143:3300/0 conn(0x7fab240b0e60 0x7fab2413d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab23fff700 1 -- 172.21.15.3:0/102773561 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab24067030 con 0x7fab24064500 2022-07-21T01:38:19.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab23fff700 1 --2- 172.21.15.3:0/102773561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab24064500 0x7fab240609c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fab1c002f70 tx=0x7fab1c006eb0).ready entity=mon.0 client_cookie=eb0f943a6b5e8ba9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:19.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab227fc700 1 -- 172.21.15.3:0/102773561 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab1c00d570 con 0x7fab24064500 2022-07-21T01:38:19.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab227fc700 1 -- 172.21.15.3:0/102773561 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1c000e90 con 0x7fab24064500 2022-07-21T01:38:19.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab227fc700 1 -- 172.21.15.3:0/102773561 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab1c00d900 con 0x7fab24064500 2022-07-21T01:38:19.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 -- 172.21.15.3:0/102773561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab24064500 msgr2=0x7fab240609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 --2- 172.21.15.3:0/102773561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab24064500 0x7fab240609c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fab1c002f70 tx=0x7fab1c006eb0).stop 2022-07-21T01:38:19.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 -- 172.21.15.3:0/102773561 shutdown_connections 2022-07-21T01:38:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 --2- 172.21.15.3:0/102773561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab24064500 0x7fab240609c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 --2- 172.21.15.3:0/102773561 >> v2:172.21.15.143:3300/0 conn(0x7fab240b0e60 0x7fab2413d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 --2- 172.21.15.3:0/102773561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab240b6510 0x7fab2413d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 -- 172.21.15.3:0/102773561 >> 172.21.15.3:0/102773561 conn(0x7fab24063800 msgr2=0x7fab24063be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 -- 172.21.15.3:0/102773561 shutdown_connections 2022-07-21T01:38:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.137+0000 7fab28913700 1 -- 172.21.15.3:0/102773561 wait complete. 2022-07-21T01:38:19.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 Processor -- start 2022-07-21T01:38:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 -- start start 2022-07-21T01:38:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab24064500 0x7fab240ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab240b0e60 0x7fab241444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab240b6510 0x7fab241449e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab24142e30 con 0x7fab240b6510 2022-07-21T01:38:19.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab24144ed0 con 0x7fab240b0e60 2022-07-21T01:38:19.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab24143d00 con 0x7fab24064500 2022-07-21T01:38:19.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab23fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab240b6510 0x7fab241449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab23fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab240b6510 0x7fab241449e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45960/0 (socket says 172.21.15.3:45960) 2022-07-21T01:38:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab23fff700 1 -- 172.21.15.3:0/4116577074 learned_addr learned my addr 172.21.15.3:0/4116577074 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab22ffd700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab240b0e60 0x7fab241444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab237fe700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab24064500 0x7fab240ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab23fff700 1 -- 172.21.15.3:0/4116577074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab24064500 msgr2=0x7fab240ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab23fff700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab24064500 0x7fab240ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab23fff700 1 -- 172.21.15.3:0/4116577074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab240b0e60 msgr2=0x7fab241444f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab23fff700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab240b0e60 0x7fab241444f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab23fff700 1 -- 172.21.15.3:0/4116577074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1c00b1e0 con 0x7fab240b6510 2022-07-21T01:38:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab237fe700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab24064500 0x7fab240ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab23fff700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab240b6510 0x7fab241449e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fab24142f70 tx=0x7fab1c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab20ff9700 1 -- 172.21.15.3:0/4116577074 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab1c011760 con 0x7fab240b6510 2022-07-21T01:38:19.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab20ff9700 1 -- 172.21.15.3:0/4116577074 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1c0118c0 con 0x7fab240b6510 2022-07-21T01:38:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab241455b0 con 0x7fab240b6510 2022-07-21T01:38:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab20ff9700 1 -- 172.21.15.3:0/4116577074 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab1c011c20 con 0x7fab240b6510 2022-07-21T01:38:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.141+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab24143860 con 0x7fab240b6510 2022-07-21T01:38:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.145+0000 7fab20ff9700 1 -- 172.21.15.3:0/4116577074 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fab1c025030 con 0x7fab240b6510 2022-07-21T01:38:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.145+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab24145770 con 0x7fab240b6510 2022-07-21T01:38:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.145+0000 7fab20ff9700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab08053950 0x7fab08055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.145+0000 7fab20ff9700 1 -- 172.21.15.3:0/4116577074 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fab1c06cfb0 con 0x7fab240b6510 2022-07-21T01:38:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.145+0000 7fab237fe700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab08053950 0x7fab08055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.145+0000 7fab237fe700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab08053950 0x7fab08055e10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fab0c00b000 tx=0x7fab0c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.149+0000 7fab20ff9700 1 -- 172.21.15.3:0/4116577074 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fab1c043f40 con 0x7fab240b6510 2022-07-21T01:38:19.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.313+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7fab24143ec0 con 0x7fab240b6510 2022-07-21T01:38:19.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.725+0000 7fab20ff9700 1 -- 172.21.15.3:0/4116577074 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v214) v1 ==== 152+0+0 (secure 0 0 0) 0x7fab1c0230d0 con 0x7fab240b6510 2022-07-21T01:38:19.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab08053950 msgr2=0x7fab08055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab08053950 0x7fab08055e10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fab0c00b000 tx=0x7fab0c008040).stop 2022-07-21T01:38:19.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab240b6510 msgr2=0x7fab241449e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab240b6510 0x7fab241449e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fab24142f70 tx=0x7fab1c015000).stop 2022-07-21T01:38:19.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 shutdown_connections 2022-07-21T01:38:19.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab240b6510 0x7fab241449e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fab08053950 0x7fab08055e10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab240b0e60 0x7fab241444f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 --2- 172.21.15.3:0/4116577074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab24064500 0x7fab240ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 >> 172.21.15.3:0/4116577074 conn(0x7fab24063800 msgr2=0x7fab240b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:19.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 shutdown_connections 2022-07-21T01:38:19.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.729+0000 7fab28913700 1 -- 172.21.15.3:0/4116577074 wait complete. 2022-07-21T01:38:19.741 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-21T01:38:19.742 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-21T01:38:19.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-07-21T01:38:19.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.849+0000 7efd5c0cd700 1 Processor -- start 2022-07-21T01:38:19.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.849+0000 7efd5c0cd700 1 -- start start 2022-07-21T01:38:19.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0630 0x7efd540adec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efd540aa2b0 0x7efd540ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd540b2c80 con 0x7efd540b0a10 2022-07-21T01:38:19.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd540b2900 con 0x7efd540aa2b0 2022-07-21T01:38:19.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd540b2eb0 con 0x7efd540b0630 2022-07-21T01:38:19.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd59668700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efd540aa2b0 0x7efd540ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd59668700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efd540aa2b0 0x7efd540ae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45966/0 (socket says 172.21.15.3:45966) 2022-07-21T01:38:19.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd59668700 1 -- 172.21.15.3:0/650433762 learned_addr learned my addr 172.21.15.3:0/650433762 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:19.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd59e69700 1 --2- 172.21.15.3:0/650433762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0630 0x7efd540adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd59668700 1 -- 172.21.15.3:0/650433762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0630 msgr2=0x7efd540adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:19.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd59668700 1 --2- 172.21.15.3:0/650433762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0630 0x7efd540adec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd59668700 1 -- 172.21.15.3:0/650433762 >> v1:172.21.15.143:6789/0 conn(0x7efd540b0a10 legacy=0x7efd541438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd59668700 1 -- 172.21.15.3:0/650433762 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd540a9d00 con 0x7efd540aa2b0 2022-07-21T01:38:19.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd59668700 1 --2- 172.21.15.3:0/650433762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 0x7efd540ae3b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7efd44002f70 tx=0x7efd44006eb0).ready entity=mon.0 client_cookie=7d829f699f571ba1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:19.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd58e67700 1 -- 172.21.15.3:0/650433762 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd4400d570 con 0x7efd540aa2b0 2022-07-21T01:38:19.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd58e67700 1 -- 172.21.15.3:0/650433762 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd44000e90 con 0x7efd540aa2b0 2022-07-21T01:38:19.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd58e67700 1 -- 172.21.15.3:0/650433762 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd4400d900 con 0x7efd540aa2b0 2022-07-21T01:38:19.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 -- 172.21.15.3:0/650433762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 msgr2=0x7efd540ae3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 --2- 172.21.15.3:0/650433762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 0x7efd540ae3b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7efd44002f70 tx=0x7efd44006eb0).stop 2022-07-21T01:38:19.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 -- 172.21.15.3:0/650433762 shutdown_connections 2022-07-21T01:38:19.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 --2- 172.21.15.3:0/650433762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 0x7efd540ae3b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 --2- 172.21.15.3:0/650433762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0630 0x7efd540adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 -- 172.21.15.3:0/650433762 >> 172.21.15.3:0/650433762 conn(0x7efd54060950 msgr2=0x7efd54060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:19.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 -- 172.21.15.3:0/650433762 shutdown_connections 2022-07-21T01:38:19.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.853+0000 7efd5c0cd700 1 -- 172.21.15.3:0/650433762 wait complete. 2022-07-21T01:38:19.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 Processor -- start 2022-07-21T01:38:19.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 -- start start 2022-07-21T01:38:19.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 0x7efd5413cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd540b0630 0x7efd540643e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0a10 0x7efd540648d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5413de40 con 0x7efd540aa2b0 2022-07-21T01:38:19.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd54064dc0 con 0x7efd540b0630 2022-07-21T01:38:19.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5413d770 con 0x7efd540b0a10 2022-07-21T01:38:19.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59e69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 0x7efd5413cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59e69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 0x7efd5413cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45970/0 (socket says 172.21.15.3:45970) 2022-07-21T01:38:19.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59e69700 1 -- 172.21.15.3:0/1651212262 learned_addr learned my addr 172.21.15.3:0/1651212262 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:19.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59668700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd540b0630 0x7efd540643e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5a66a700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0a10 0x7efd540648d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59e69700 1 -- 172.21.15.3:0/1651212262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0a10 msgr2=0x7efd540648d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59e69700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0a10 0x7efd540648d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59e69700 1 -- 172.21.15.3:0/1651212262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd540b0630 msgr2=0x7efd540643e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:19.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59e69700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd540b0630 0x7efd540643e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:19.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59e69700 1 -- 172.21.15.3:0/1651212262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4400b1e0 con 0x7efd540aa2b0 2022-07-21T01:38:19.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd59e69700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 0x7efd5413cff0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7efd54064f00 tx=0x7efd50006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:19.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd42ffd700 1 -- 172.21.15.3:0/1651212262 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd50011920 con 0x7efd540aa2b0 2022-07-21T01:38:19.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd541500d0 con 0x7efd540aa2b0 2022-07-21T01:38:19.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd42ffd700 1 -- 172.21.15.3:0/1651212262 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd50011a80 con 0x7efd540aa2b0 2022-07-21T01:38:19.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd42ffd700 1 -- 172.21.15.3:0/1651212262 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd50011de0 con 0x7efd540aa2b0 2022-07-21T01:38:19.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.857+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5413d9a0 con 0x7efd540aa2b0 2022-07-21T01:38:19.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.861+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd540574e0 con 0x7efd540aa2b0 2022-07-21T01:38:19.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.861+0000 7efd42ffd700 1 -- 172.21.15.3:0/1651212262 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efd50021070 con 0x7efd540aa2b0 2022-07-21T01:38:19.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.861+0000 7efd42ffd700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd34053b00 0x7efd34055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:19.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.861+0000 7efd42ffd700 1 -- 172.21.15.3:0/1651212262 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5686+0+0 (secure 0 0 0) 0x7efd5006db40 con 0x7efd540aa2b0 2022-07-21T01:38:19.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.861+0000 7efd59668700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd34053b00 0x7efd34055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:19.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.861+0000 7efd59668700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd34053b00 0x7efd34055fc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7efd4401c000 tx=0x7efd44015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:19.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:19.865+0000 7efd42ffd700 1 -- 172.21.15.3:0/1651212262 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd50044b40 con 0x7efd540aa2b0 2022-07-21T01:38:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.033+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7efd540447e0 con 0x7efd540aa2b0 2022-07-21T01:38:20.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd42ffd700 1 -- 172.21.15.3:0/1651212262 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v215) v1 ==== 150+0+0 (secure 0 0 0) 0x7efd50048160 con 0x7efd540aa2b0 2022-07-21T01:38:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd34053b00 msgr2=0x7efd34055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd34053b00 0x7efd34055fc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7efd4401c000 tx=0x7efd44015000).stop 2022-07-21T01:38:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 msgr2=0x7efd5413cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:20.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 0x7efd5413cff0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7efd54064f00 tx=0x7efd50006df0).stop 2022-07-21T01:38:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 shutdown_connections 2022-07-21T01:38:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd540b0a10 0x7efd540648d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd34053b00 0x7efd34055fc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd540b0630 0x7efd540643e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:20.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 --2- 172.21.15.3:0/1651212262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd540aa2b0 0x7efd5413cff0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:20.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 >> 172.21.15.3:0/1651212262 conn(0x7efd54060950 msgr2=0x7efd540b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:20.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 shutdown_connections 2022-07-21T01:38:20.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.733+0000 7efd5c0cd700 1 -- 172.21.15.3:0/1651212262 wait complete. 2022-07-21T01:38:20.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:38:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbb59e700 1 Processor -- start 2022-07-21T01:38:20.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbb59e700 1 -- start start 2022-07-21T01:38:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 0x7f3dbc061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbb59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3dbc1405d0 0x7f3dbc140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbb59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc0aee80 con 0x7f3dbc0ad750 2022-07-21T01:38:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbb59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dbc0b70b0 con 0x7f3dbc1405d0 2022-07-21T01:38:20.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc0af0b0 con 0x7f3dbc0614b0 2022-07-21T01:38:20.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3dbc1405d0 0x7f3dbc140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:20.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3dbc1405d0 0x7f3dbc140a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45982/0 (socket says 172.21.15.3:45982) 2022-07-21T01:38:20.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbad9d700 1 -- 172.21.15.3:0/627003684 learned_addr learned my addr 172.21.15.3:0/627003684 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:20.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dba59c700 1 --2- 172.21.15.3:0/627003684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 0x7f3dbc061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:20.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbad9d700 1 -- 172.21.15.3:0/627003684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 msgr2=0x7f3dbc061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:20.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbad9d700 1 --2- 172.21.15.3:0/627003684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 0x7f3dbc061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:20.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbad9d700 1 -- 172.21.15.3:0/627003684 >> v1:172.21.15.143:6789/0 conn(0x7f3dbc0ad750 legacy=0x7f3dbc0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:20.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbad9d700 1 -- 172.21.15.3:0/627003684 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc0af350 con 0x7f3dbc1405d0 2022-07-21T01:38:20.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbad9d700 1 --2- 172.21.15.3:0/627003684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 0x7f3dbc140a60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f3db40032f0 tx=0x7f3db4009290).ready entity=mon.0 client_cookie=ac1b114f99b16d35 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:20.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3db959a700 1 -- 172.21.15.3:0/627003684 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3db400b8a0 con 0x7f3dbc1405d0 2022-07-21T01:38:20.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3db959a700 1 -- 172.21.15.3:0/627003684 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db400ba00 con 0x7f3dbc1405d0 2022-07-21T01:38:20.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3db959a700 1 -- 172.21.15.3:0/627003684 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3db400bb70 con 0x7f3dbc1405d0 2022-07-21T01:38:20.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbb59e700 1 -- 172.21.15.3:0/627003684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 msgr2=0x7f3dbc140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:20.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.869+0000 7f3dbb59e700 1 --2- 172.21.15.3:0/627003684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 0x7f3dbc140a60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f3db40032f0 tx=0x7f3db4009290).stop 2022-07-21T01:38:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.873+0000 7f3dbb59e700 1 -- 172.21.15.3:0/627003684 shutdown_connections 2022-07-21T01:38:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.873+0000 7f3dbb59e700 1 --2- 172.21.15.3:0/627003684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 0x7f3dbc140a60 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.873+0000 7f3dbb59e700 1 --2- 172.21.15.3:0/627003684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 0x7f3dbc061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.873+0000 7f3dbb59e700 1 -- 172.21.15.3:0/627003684 >> 172.21.15.3:0/627003684 conn(0x7f3dbc066f70 msgr2=0x7f3dbc067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:20.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.873+0000 7f3dbb59e700 1 -- 172.21.15.3:0/627003684 shutdown_connections 2022-07-21T01:38:20.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.873+0000 7f3dbb59e700 1 -- 172.21.15.3:0/627003684 wait complete. 2022-07-21T01:38:20.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.873+0000 7f3dbb59e700 1 Processor -- start 2022-07-21T01:38:20.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.873+0000 7f3dbb59e700 1 -- start start 2022-07-21T01:38:20.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 0x7f3dbc0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:20.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbb59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dbc0ad750 0x7f3dbc0b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:20.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbb59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 0x7f3dbc0b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:20.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbb59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc153360 con 0x7f3dbc1405d0 2022-07-21T01:38:20.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbb59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc0aaa40 con 0x7f3dbc0ad750 2022-07-21T01:38:20.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc0aace0 con 0x7f3dbc0614b0 2022-07-21T01:38:20.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 0x7f3dbc0b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:20.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 0x7f3dbc0b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45988/0 (socket says 172.21.15.3:45988) 2022-07-21T01:38:20.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbad9d700 1 -- 172.21.15.3:0/84107263 learned_addr learned my addr 172.21.15.3:0/84107263 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:20.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3db9d9b700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dbc0ad750 0x7f3dbc0b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:20.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dba59c700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 0x7f3dbc0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:20.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbad9d700 1 -- 172.21.15.3:0/84107263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 msgr2=0x7f3dbc0b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:20.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbad9d700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 0x7f3dbc0b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:20.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbad9d700 1 -- 172.21.15.3:0/84107263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dbc0ad750 msgr2=0x7f3dbc0b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbad9d700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dbc0ad750 0x7f3dbc0b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbad9d700 1 -- 172.21.15.3:0/84107263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db4009530 con 0x7f3dbc1405d0 2022-07-21T01:38:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbad9d700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 0x7f3dbc0b0bb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f3db40032f0 tx=0x7f3db4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:20.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3db37fe700 1 -- 172.21.15.3:0/84107263 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3db400b8a0 con 0x7f3dbc1405d0 2022-07-21T01:38:20.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3db37fe700 1 -- 172.21.15.3:0/84107263 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db4021440 con 0x7f3dbc1405d0 2022-07-21T01:38:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dbc152f60 con 0x7f3dbc1405d0 2022-07-21T01:38:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3db37fe700 1 -- 172.21.15.3:0/84107263 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3db40217a0 con 0x7f3dbc1405d0 2022-07-21T01:38:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dbc0ab9d0 con 0x7f3dbc1405d0 2022-07-21T01:38:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.877+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dbc0aaea0 con 0x7f3dbc1405d0 2022-07-21T01:38:20.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.881+0000 7f3db37fe700 1 -- 172.21.15.3:0/84107263 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3db400f070 con 0x7f3dbc1405d0 2022-07-21T01:38:20.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.881+0000 7f3db37fe700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3da0053950 0x7f3da0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:20.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.881+0000 7f3db37fe700 1 -- 172.21.15.3:0/84107263 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f3db406c1c0 con 0x7f3dbc1405d0 2022-07-21T01:38:20.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.881+0000 7f3dba59c700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3da0053950 0x7f3da0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:20.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.881+0000 7f3dba59c700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3da0053950 0x7f3da0055e10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f3da400b000 tx=0x7f3da4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:20.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:20.885+0000 7f3db37fe700 1 -- 172.21.15.3:0/84107263 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3db4043150 con 0x7f3dbc1405d0 2022-07-21T01:38:21.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.017+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3dbc0af5f0 con 0x7f3dbc1405d0 2022-07-21T01:38:21.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3db37fe700 1 -- 172.21.15.3:0/84107263 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3db4015020 con 0x7f3dbc1405d0 2022-07-21T01:38:21.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3da0053950 msgr2=0x7f3da0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3da0053950 0x7f3da0055e10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f3da400b000 tx=0x7f3da4006df0).stop 2022-07-21T01:38:21.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 msgr2=0x7f3dbc0b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 0x7f3dbc0b0bb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f3db40032f0 tx=0x7f3db4016000).stop 2022-07-21T01:38:21.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 shutdown_connections 2022-07-21T01:38:21.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3dbc1405d0 0x7f3dbc0b0bb0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3da0053950 0x7f3da0055e10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dbc0ad750 0x7f3dbc0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 --2- 172.21.15.3:0/84107263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dbc0614b0 0x7f3dbc0b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 >> 172.21.15.3:0/84107263 conn(0x7f3dbc066f70 msgr2=0x7f3dbc0b8990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 shutdown_connections 2022-07-21T01:38:21.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.025+0000 7f3dbb59e700 1 -- 172.21.15.3:0/84107263 wait complete. 2022-07-21T01:38:21.037 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:38:21.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:21.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:21.075 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:21.097 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:21.119 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:38:21.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:21.120 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:21.120 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:21.145 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:38:21.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:38:21.147 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:38:21.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:21.148 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:21.149 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:38:21.156 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:38:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:38:21.158 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:38:21.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:38:21.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b32147700 1 Processor -- start 2022-07-21T01:38:21.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b32147700 1 -- start start 2022-07-21T01:38:21.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b32147700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2b2c0ad750 0x7f2b2c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b32147700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2b2c1405d0 0x7f2b2c140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b32147700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2c0aee80 con 0x7f2b2c0614b0 2022-07-21T01:38:21.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b32147700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b2c0b70b0 con 0x7f2b2c1405d0 2022-07-21T01:38:21.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b32147700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b2c0af0b0 con 0x7f2b2c0ad750 2022-07-21T01:38:21.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b31946700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2b2c1405d0 0x7f2b2c140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b31145700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2b2c0614b0 0x7f2b2c0b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54272/0 (socket says 172.21.15.3:54272) 2022-07-21T01:38:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b31145700 1 -- 172.21.15.3:0/313754980 learned_addr learned my addr 172.21.15.3:0/313754980 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.273+0000 7f2b30944700 1 --2- 172.21.15.3:0/313754980 >> v2:172.21.15.143:3300/0 conn(0x7f2b2c0ad750 0x7f2b2c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b31946700 1 -- 172.21.15.3:0/313754980 >> v2:172.21.15.143:3300/0 conn(0x7f2b2c0ad750 msgr2=0x7f2b2c061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b31946700 1 --2- 172.21.15.3:0/313754980 >> v2:172.21.15.143:3300/0 conn(0x7f2b2c0ad750 0x7f2b2c061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b31946700 1 -- 172.21.15.3:0/313754980 >> v1:172.21.15.3:6789/0 conn(0x7f2b2c0614b0 legacy=0x7f2b2c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b31946700 1 -- 172.21.15.3:0/313754980 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c0af350 con 0x7f2b2c1405d0 2022-07-21T01:38:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b23fff700 1 -- 172.21.15.3:0/313754980 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951213367 0 0) 0x7f2b2c0aee80 con 0x7f2b2c0614b0 2022-07-21T01:38:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b30944700 1 --2- 172.21.15.3:0/313754980 >> v2:172.21.15.143:3300/0 conn(0x7f2b2c0ad750 0x7f2b2c061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b31946700 1 --2- 172.21.15.3:0/313754980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c1405d0 0x7f2b2c140a80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f2b240032f0 tx=0x7f2b240091d0).ready entity=mon.0 client_cookie=b389f817ffd8c44a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b23fff700 1 -- 172.21.15.3:0/313754980 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b24009ea0 con 0x7f2b2c1405d0 2022-07-21T01:38:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b23fff700 1 -- 172.21.15.3:0/313754980 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2400b870 con 0x7f2b2c1405d0 2022-07-21T01:38:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b23fff700 1 -- 172.21.15.3:0/313754980 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b2400b9e0 con 0x7f2b2c1405d0 2022-07-21T01:38:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 -- 172.21.15.3:0/313754980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c1405d0 msgr2=0x7f2b2c140a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 --2- 172.21.15.3:0/313754980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c1405d0 0x7f2b2c140a80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f2b240032f0 tx=0x7f2b240091d0).stop 2022-07-21T01:38:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 -- 172.21.15.3:0/313754980 shutdown_connections 2022-07-21T01:38:21.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 --2- 172.21.15.3:0/313754980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c1405d0 0x7f2b2c140a80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 --2- 172.21.15.3:0/313754980 >> v2:172.21.15.143:3300/0 conn(0x7f2b2c0ad750 0x7f2b2c061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 -- 172.21.15.3:0/313754980 >> 172.21.15.3:0/313754980 conn(0x7f2b2c066f70 msgr2=0x7f2b2c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:21.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 -- 172.21.15.3:0/313754980 shutdown_connections 2022-07-21T01:38:21.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 -- 172.21.15.3:0/313754980 wait complete. 2022-07-21T01:38:21.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 Processor -- start 2022-07-21T01:38:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.277+0000 7f2b32147700 1 -- start start 2022-07-21T01:38:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b32147700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b2c0614b0 0x7f2b2c152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b32147700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c0ad750 0x7f2b2c141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b32147700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b2c1405d0 0x7f2b2c0aa340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b32147700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c14a110 con 0x7f2b2c0ad750 2022-07-21T01:38:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b32147700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c1531a0 con 0x7f2b2c1405d0 2022-07-21T01:38:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b32147700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0a9e80 con 0x7f2b2c0614b0 2022-07-21T01:38:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b30944700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c0ad750 0x7f2b2c141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b30944700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c0ad750 0x7f2b2c141b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46000/0 (socket says 172.21.15.3:46000) 2022-07-21T01:38:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b30944700 1 -- 172.21.15.3:0/2094160770 learned_addr learned my addr 172.21.15.3:0/2094160770 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b31946700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b2c1405d0 0x7f2b2c0aa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b31145700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b2c0614b0 0x7f2b2c152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b30944700 1 -- 172.21.15.3:0/2094160770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b2c0614b0 msgr2=0x7f2b2c152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b30944700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b2c0614b0 0x7f2b2c152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b30944700 1 -- 172.21.15.3:0/2094160770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b2c1405d0 msgr2=0x7f2b2c0aa340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b30944700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b2c1405d0 0x7f2b2c0aa340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b30944700 1 -- 172.21.15.3:0/2094160770 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24009470 con 0x7f2b2c0ad750 2022-07-21T01:38:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b31145700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b2c0614b0 0x7f2b2c152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b30944700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c0ad750 0x7f2b2c141b10 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f2b2800a000 tx=0x7f2b28008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b227fc700 1 -- 172.21.15.3:0/2094160770 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b28011780 con 0x7f2b2c0ad750 2022-07-21T01:38:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b227fc700 1 -- 172.21.15.3:0/2094160770 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b280118e0 con 0x7f2b2c0ad750 2022-07-21T01:38:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2c14a370 con 0x7f2b2c0ad750 2022-07-21T01:38:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b227fc700 1 -- 172.21.15.3:0/2094160770 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b28011c40 con 0x7f2b2c0ad750 2022-07-21T01:38:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c142000 con 0x7f2b2c0ad750 2022-07-21T01:38:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.281+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c0af5f0 con 0x7f2b2c0ad750 2022-07-21T01:38:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.285+0000 7f2b227fc700 1 -- 172.21.15.3:0/2094160770 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2b28012420 con 0x7f2b2c0ad750 2022-07-21T01:38:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.285+0000 7f2b227fc700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b14053980 0x7f2b14055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.285+0000 7f2b227fc700 1 -- 172.21.15.3:0/2094160770 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f2b2806baf0 con 0x7f2b2c0ad750 2022-07-21T01:38:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.285+0000 7f2b31145700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b14053980 0x7f2b14055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.285+0000 7f2b31145700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b14053980 0x7f2b14055e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f2b1c00b000 tx=0x7f2b1c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:21.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.289+0000 7f2b227fc700 1 -- 172.21.15.3:0/2094160770 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b28042b30 con 0x7f2b2c0ad750 2022-07-21T01:38:21.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.417+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2b2c0574e0 con 0x7f2b2c0ad750 2022-07-21T01:38:21.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.421+0000 7f2b227fc700 1 -- 172.21.15.3:0/2094160770 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v61) v1 ==== 81+0+29218 (secure 0 0 0) 0x7f2b28046150 con 0x7f2b2c0ad750 2022-07-21T01:38:21.421 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:21.421 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:38:21.421 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 61, 2022-07-21T01:38:21.421 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:38:21.422 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:38:21.422 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:38:18.718942+0000", 2022-07-21T01:38:21.422 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:38:21.423 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:38:21.423 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:38:21.423 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:38:21.423 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:38:21.424 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:38:21.424 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:38:21.424 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:38:21.424 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:38:21.425 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:21.425 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:38:21.425 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:38:21.425 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:38:21.426 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:38:21.426 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:38:21.426 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 10, 2022-07-21T01:38:21.426 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:38:21.427 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:38:21.427 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:38:21.427 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:38:21.427 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:38:21.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.428 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:38:21.429 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:38:21.429 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:38:21.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:21.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:21.430 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:21.430 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:21.430 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:21.431 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:21.431 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:21.431 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:21.431 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:38:21.432 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:38:21.432 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:38:21.432 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:38:21.432 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:38:21.433 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:21.433 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:21.433 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:21.433 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:21.434 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:21.434 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:21.434 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:21.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.435 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:38:21.435 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:21.435 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:21.435 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:21.436 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:21.436 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:21.436 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:21.436 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:21.437 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:21.437 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:21.437 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:21.438 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:21.438 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:21.438 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:21.438 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:21.439 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:21.439 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:21.440 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:21.440 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:21.440 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:21.440 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:21.441 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:21.441 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:21.441 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:21.441 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:21.442 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:21.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:21.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.443 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:21.443 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:21.443 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:21.443 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:21.444 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:21.444 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:21.444 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:21.445 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:21.445 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:21.445 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:21.445 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:21.446 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:38:21.446 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:38:21.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.447 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:21.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:38:21.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.450 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:38:21.451 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:38:21.451 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:38:21.452 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:21.452 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:21.453 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:21.453 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:21.454 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:21.454 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:21.455 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:21.455 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:21.456 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:38:21.456 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:38:21.456 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:38:21.457 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:38:21.457 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:38:21.458 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:21.458 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:21.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:21.458 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:21.459 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:21.459 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:21.459 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:21.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.460 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:38:21.460 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:21.460 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:21.461 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:21.461 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:21.461 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:21.462 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:21.462 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:21.462 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:21.463 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:21.463 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:21.463 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:21.464 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:21.464 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:21.464 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:21.464 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:21.465 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:21.465 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:21.465 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:21.465 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:21.466 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:21.466 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:21.466 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:21.467 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:21.467 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:21.467 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:21.467 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:21.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.468 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:21.468 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:21.468 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:21.469 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:21.469 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:21.469 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:21.470 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:21.470 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:21.470 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:21.471 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:21.471 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:21.472 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:38:21.472 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:21.473 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:38:21.473 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.474 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.474 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:21.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:38:21.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:38:21.476 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:38:21.477 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:21.477 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:21.478 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:21.479 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:21.479 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:21.480 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:21.480 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:38:21.481 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:21.481 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:21.482 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:21.482 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:21.483 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.484 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.484 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:38:21.485 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:21.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.487 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:38:21.487 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:21.488 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.488 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.489 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:21.490 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.491 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:38:21.492 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:21.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:38:21.494 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:21.495 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.495 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.496 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:21.497 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.498 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.498 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:38:21.499 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:21.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.500 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.501 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:38:21.501 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:21.502 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.502 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.503 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:21.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.505 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.505 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:38:21.506 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:21.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:38:21.508 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:21.509 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.509 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.510 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:38:21.511 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:38:21.511 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:38:21.512 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:38:21.512 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:21.513 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:21.513 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:21.514 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.515 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:38:21.516 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:38:21.517 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:21.517 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:21.517 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:21.518 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:21.518 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:21.518 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:21.519 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:21.519 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:21.519 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:21.520 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:21.520 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:21.520 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.521 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.521 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:38:21.522 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:21.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.522 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.523 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:38:21.523 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:21.523 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.524 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.524 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:21.525 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.525 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.526 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:38:21.526 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:21.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.527 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:38:21.527 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:21.528 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.528 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.529 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:21.529 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.530 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.530 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:38:21.530 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:21.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.531 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:38:21.532 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:21.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.532 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.533 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:21.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:38:21.534 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:21.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.535 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.535 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:38:21.536 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:21.536 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.536 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.537 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:38:21.537 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:38:21.537 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:38:21.538 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:38:21.538 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:21.538 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:21.539 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:21.539 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.541 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:38:21.541 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:38:21.542 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:21.542 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:21.543 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:21.543 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:21.543 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:21.544 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:21.544 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:38:21.545 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:21.545 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:21.546 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:21.546 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:21.547 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.548 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:38:21.549 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:21.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.550 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:38:21.551 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:21.551 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.552 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.553 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:21.553 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.554 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:38:21.555 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:21.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.556 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.556 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:38:21.557 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:21.557 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.557 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.558 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:21.558 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.559 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.559 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:38:21.559 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:21.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.560 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.560 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:38:21.561 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:21.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.561 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.562 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:21.562 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.563 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.563 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:38:21.563 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:21.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.564 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.564 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:38:21.565 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:21.565 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.565 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.566 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:38:21.566 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:38:21.566 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:38:21.566 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:38:21.567 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:21.567 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:21.567 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:21.567 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.568 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:38:21.568 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:38:21.568 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:21.569 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:21.569 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:21.569 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:21.569 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:21.570 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:21.570 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:38:21.570 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 55, 2022-07-21T01:38:21.570 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:21.571 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:21.571 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:21.571 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.572 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:38:21.572 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:21.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.573 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.573 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:38:21.573 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:21.573 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.574 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.574 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:21.574 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.575 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:38:21.575 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:21.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.576 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.576 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:38:21.577 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:21.577 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.577 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.578 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:21.578 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.579 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:38:21.579 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:21.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.579 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.580 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:38:21.580 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:21.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.580 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.581 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:21.581 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.582 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.582 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:38:21.582 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:21.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.583 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.583 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:38:21.583 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:21.584 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.584 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.584 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:38:21.585 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:38:21.585 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:38:21.585 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:38:21.585 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:21.585 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:21.586 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:21.586 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.587 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:38:21.587 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:38:21.588 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:21.588 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:21.588 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:21.589 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:21.589 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:21.590 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:21.590 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:21.591 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:21.591 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:21.592 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:21.592 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:21.593 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.594 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.594 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:38:21.595 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:21.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.596 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:38:21.597 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:21.597 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.598 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.599 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:21.599 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.600 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.600 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:38:21.601 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:21.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.601 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.602 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:38:21.602 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:21.602 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.602 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.603 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:21.603 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.604 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.604 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:38:21.604 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:21.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.605 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:38:21.605 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:21.606 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.606 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.607 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:21.607 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.607 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.608 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:38:21.608 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:21.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.609 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:38:21.609 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:21.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.609 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.610 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:38:21.610 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:38:21.610 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:38:21.611 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:38:21.611 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:21.611 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:21.611 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:21.612 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.612 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:38:21.613 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:38:21.613 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:21.613 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:21.613 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:21.614 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:21.614 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:21.614 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:21.614 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:21.615 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:21.615 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:21.615 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:21.615 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:21.616 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.616 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.616 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:38:21.616 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:21.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.617 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.617 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:38:21.618 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:21.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.618 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.619 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:21.619 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.619 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.620 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:38:21.620 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:21.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.621 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.621 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:38:21.621 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:21.621 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.622 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.622 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:21.623 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.623 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:38:21.624 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:21.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.624 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.624 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:38:21.625 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:21.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.625 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.626 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:21.626 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.627 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:38:21.627 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:21.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.628 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.628 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:38:21.628 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:21.628 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.629 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.629 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:38:21.629 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:38:21.630 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:38:21.630 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:38:21.630 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:21.630 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:21.631 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:21.631 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.632 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:38:21.632 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:38:21.632 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:21.632 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:21.633 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:21.633 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:21.633 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:21.633 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:21.634 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:21.634 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 56, 2022-07-21T01:38:21.634 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:21.634 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:21.635 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:21.635 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.636 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:38:21.636 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:21.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.637 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.637 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:38:21.637 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:21.637 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.637 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.638 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:21.638 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.639 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.639 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:38:21.639 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:21.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.640 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:38:21.640 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:21.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.641 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.641 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:21.641 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.642 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.643 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:38:21.643 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:21.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.644 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.645 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:38:21.645 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:21.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.647 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:21.648 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.649 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.649 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:38:21.650 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:21.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.651 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:38:21.652 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:21.652 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.654 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:38:21.654 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:38:21.654 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:38:21.655 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:38:21.655 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:21.656 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:21.656 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:21.657 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.658 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:38:21.658 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:38:21.658 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:21.659 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:21.659 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:21.659 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:21.659 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:21.660 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:21.660 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:21.660 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 55, 2022-07-21T01:38:21.660 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:21.661 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:21.661 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:21.661 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.662 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.662 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:38:21.662 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:21.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.663 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:38:21.664 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:21.664 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.664 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.665 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:21.665 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.666 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.666 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:38:21.667 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:21.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.667 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:38:21.668 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:21.668 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.668 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.669 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:21.669 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.669 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.670 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:38:21.670 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:21.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.671 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.671 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:38:21.671 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:21.671 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.672 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.672 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:21.672 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:21.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:21.673 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:38:21.673 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:21.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:21.674 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:38:21.674 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:21.675 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.675 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.675 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:38:21.675 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:38:21.676 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:38:21.676 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:38:21.676 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:21.676 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:21.677 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:21.677 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:21.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.677 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:21.678 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:38:21.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.678 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:38:21.678 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:21.679 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:21.679 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:21.679 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:21.679 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:21.680 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:38:21.680 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:21.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.681 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:38:21.681 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:21.681 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:21.681 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:21.682 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:21.682 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:21.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:38:21.682 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:21.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.683 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:38:21.683 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:21.684 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:21.684 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:21.684 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:21.684 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:21.685 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:38:21.685 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:21.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.686 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:38:21.686 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:21.686 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:21.686 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:21.687 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:21.687 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:21.687 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:38:21.687 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:21.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.688 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:38:21.688 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:21.689 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:21.689 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:21.689 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:21.690 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:21.690 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:38:21.690 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:21.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.691 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:38:21.691 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:21.692 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:21.692 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:21.692 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:21.692 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:21.692 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:38:21.693 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:21.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.693 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:38:21.694 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:21.694 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:21.694 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:21.694 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:21.695 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:21.695 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:38:21.695 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:21.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:21.696 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:38:21.696 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:21.696 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:21.696 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:21.697 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:21.697 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:21.697 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:38:21.697 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:21.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.698 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:21.698 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:38:21.698 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:38:21.699 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:38:21.699 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:38:21.699 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:38:21.699 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6837/3381448192": "2022-07-22T01:38:15.051974+0000", 2022-07-21T01:38:21.699 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6836/3381448192": "2022-07-22T01:38:15.051974+0000" 2022-07-21T01:38:21.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.700 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:38:21.700 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:38:21.700 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:38:21.701 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:38:21.701 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:38:21.701 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:38:21.701 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:38:21.702 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:38:21.702 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:38:21.702 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:38:21.702 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:38:21.702 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:38:21.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.703 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:38:21.703 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:38:21.703 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:38:21.704 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:38:21.704 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:38:21.704 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:38:21.704 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:38:21.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:21.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:21.705 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:38:21.705 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:38:21.705 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:38:21.706 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:38:21.706 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:38:21.706 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:38:21.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b14053980 msgr2=0x7f2b14055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b14053980 0x7f2b14055e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f2b1c00b000 tx=0x7f2b1c009040).stop 2022-07-21T01:38:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c0ad750 msgr2=0x7f2b2c141b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c0ad750 0x7f2b2c141b10 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f2b2800a000 tx=0x7f2b28008040).stop 2022-07-21T01:38:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 shutdown_connections 2022-07-21T01:38:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b14053980 0x7f2b14055e40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b2c1405d0 0x7f2b2c0aa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b2c0ad750 0x7f2b2c141b10 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 --2- 172.21.15.3:0/2094160770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b2c0614b0 0x7f2b2c152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 >> 172.21.15.3:0/2094160770 conn(0x7f2b2c066f70 msgr2=0x7f2b2c0b8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 shutdown_connections 2022-07-21T01:38:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.425+0000 7f2b32147700 1 -- 172.21.15.3:0/2094160770 wait complete. 2022-07-21T01:38:21.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:21.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 Processor -- start 2022-07-21T01:38:21.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 -- start start 2022-07-21T01:38:21.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efd580b4550 0x7efd580b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efd580b6af0 0x7efd5812bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd58144240 con 0x7efd580b6710 2022-07-21T01:38:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd58143d00 con 0x7efd580b6af0 2022-07-21T01:38:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd58143fa0 con 0x7efd580b4550 2022-07-21T01:38:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5e741700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efd580b6af0 0x7efd5812bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5d73f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efd580b4550 0x7efd580b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5e741700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efd580b6af0 0x7efd5812bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46010/0 (socket says 172.21.15.3:46010) 2022-07-21T01:38:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5d73f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efd580b4550 0x7efd580b5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:51456/0 (socket says 172.21.15.3:51456) 2022-07-21T01:38:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5e741700 1 -- 172.21.15.3:0/2699223238 learned_addr learned my addr 172.21.15.3:0/2699223238 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5df40700 1 --1- 172.21.15.3:0/2699223238 >> v1:172.21.15.143:6789/0 conn(0x7efd580b6710 0x7efd58061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:56402/0 (socket says 172.21.15.3:56402) 2022-07-21T01:38:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5d73f700 1 -- 172.21.15.3:0/2699223238 >> v2:172.21.15.3:3300/0 conn(0x7efd580b6af0 msgr2=0x7efd5812bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:21.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5d73f700 1 --2- 172.21.15.3:0/2699223238 >> v2:172.21.15.3:3300/0 conn(0x7efd580b6af0 0x7efd5812bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5d73f700 1 -- 172.21.15.3:0/2699223238 >> v1:172.21.15.143:6789/0 conn(0x7efd580b6710 legacy=0x7efd58061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5d73f700 1 -- 172.21.15.3:0/2699223238 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58061780 con 0x7efd580b4550 2022-07-21T01:38:21.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5d73f700 1 --2- 172.21.15.3:0/2699223238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd580b4550 0x7efd580b5d80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7efd581432b0 tx=0x7efd48006df0).ready entity=mon.1 client_cookie=cf4ad3c2b60122d7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:21.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5cf3e700 1 -- 172.21.15.3:0/2699223238 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd4800d5c0 con 0x7efd580b4550 2022-07-21T01:38:21.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5cf3e700 1 -- 172.21.15.3:0/2699223238 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd48000e90 con 0x7efd580b4550 2022-07-21T01:38:21.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd5cf3e700 1 -- 172.21.15.3:0/2699223238 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd4800d910 con 0x7efd580b4550 2022-07-21T01:38:21.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 -- 172.21.15.3:0/2699223238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd580b4550 msgr2=0x7efd580b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 --2- 172.21.15.3:0/2699223238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd580b4550 0x7efd580b5d80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7efd581432b0 tx=0x7efd48006df0).stop 2022-07-21T01:38:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 -- 172.21.15.3:0/2699223238 shutdown_connections 2022-07-21T01:38:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 --2- 172.21.15.3:0/2699223238 >> v2:172.21.15.3:3300/0 conn(0x7efd580b6af0 0x7efd5812bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 --2- 172.21.15.3:0/2699223238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd580b4550 0x7efd580b5d80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 -- 172.21.15.3:0/2699223238 >> 172.21.15.3:0/2699223238 conn(0x7efd5805c620 msgr2=0x7efd5805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:21.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 -- 172.21.15.3:0/2699223238 shutdown_connections 2022-07-21T01:38:21.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.825+0000 7efd601a4700 1 -- 172.21.15.3:0/2699223238 wait complete. 2022-07-21T01:38:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 Processor -- start 2022-07-21T01:38:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 -- start start 2022-07-21T01:38:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd580b4550 0x7efd58127df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd580b6710 0x7efd58129910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd580b6af0 0x7efd581261b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd580b0800 con 0x7efd580b4550 2022-07-21T01:38:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd58156700 con 0x7efd580b6af0 2022-07-21T01:38:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd580b0360 con 0x7efd580b6710 2022-07-21T01:38:21.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5df40700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd580b4550 0x7efd58127df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5df40700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd580b4550 0x7efd58127df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46012/0 (socket says 172.21.15.3:46012) 2022-07-21T01:38:21.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5df40700 1 -- 172.21.15.3:0/149654769 learned_addr learned my addr 172.21.15.3:0/149654769 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:21.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5e741700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd580b6af0 0x7efd581261b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5d73f700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd580b6710 0x7efd58129910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5df40700 1 -- 172.21.15.3:0/149654769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd580b6710 msgr2=0x7efd58129910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5df40700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd580b6710 0x7efd58129910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5df40700 1 -- 172.21.15.3:0/149654769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd580b6af0 msgr2=0x7efd581261b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:21.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5df40700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd580b6af0 0x7efd581261b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:21.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5df40700 1 -- 172.21.15.3:0/149654769 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4800b180 con 0x7efd580b4550 2022-07-21T01:38:21.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd5df40700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd580b4550 0x7efd58127df0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7efd580b1e10 tx=0x7efd54007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:21.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd46ffd700 1 -- 172.21.15.3:0/149654769 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd540070d0 con 0x7efd580b4550 2022-07-21T01:38:21.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd46ffd700 1 -- 172.21.15.3:0/149654769 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd54007230 con 0x7efd580b4550 2022-07-21T01:38:21.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd46ffd700 1 -- 172.21.15.3:0/149654769 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd54005550 con 0x7efd580b4550 2022-07-21T01:38:21.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd58156420 con 0x7efd580b4550 2022-07-21T01:38:21.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.829+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd58129e00 con 0x7efd580b4550 2022-07-21T01:38:21.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.833+0000 7efd46ffd700 1 -- 172.21.15.3:0/149654769 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efd5401a070 con 0x7efd580b4550 2022-07-21T01:38:21.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.833+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd580b0520 con 0x7efd580b4550 2022-07-21T01:38:21.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.833+0000 7efd46ffd700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd380539e0 0x7efd38055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:21.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.833+0000 7efd46ffd700 1 -- 172.21.15.3:0/149654769 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5686+0+0 (secure 0 0 0) 0x7efd5406b040 con 0x7efd580b4550 2022-07-21T01:38:21.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.833+0000 7efd5d73f700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd380539e0 0x7efd38055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:21.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.833+0000 7efd5d73f700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd380539e0 0x7efd38055ea0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7efd4801a000 tx=0x7efd48014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:21.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:21.841+0000 7efd46ffd700 1 -- 172.21.15.3:0/149654769 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd54041fe0 con 0x7efd580b4550 2022-07-21T01:38:22.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.009+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd58061a20 con 0x7efd580b4550 2022-07-21T01:38:22.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.009+0000 7efd46ffd700 1 -- 172.21.15.3:0/149654769 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+1553 (secure 0 0 0) 0x7efd54018020 con 0x7efd580b4550 2022-07-21T01:38:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:22.009 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21},{"gid":5850,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2711503784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2711503784},{"type":"v1","addr":"172.21.15.3:6837","nonce":2711503784}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":25}],"filesystems":[]} 2022-07-21T01:38:22.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.009+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd380539e0 msgr2=0x7efd38055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.009+0000 7efd601a4700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd380539e0 0x7efd38055ea0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7efd4801a000 tx=0x7efd48014000).stop 2022-07-21T01:38:22.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.009+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd580b4550 msgr2=0x7efd58127df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.009+0000 7efd601a4700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd580b4550 0x7efd58127df0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7efd580b1e10 tx=0x7efd54007c40).stop 2022-07-21T01:38:22.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.013+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 shutdown_connections 2022-07-21T01:38:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.013+0000 7efd601a4700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efd380539e0 0x7efd38055ea0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.013+0000 7efd601a4700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd580b6af0 0x7efd581261b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.013+0000 7efd601a4700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd580b6710 0x7efd58129910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.013+0000 7efd601a4700 1 --2- 172.21.15.3:0/149654769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd580b4550 0x7efd58127df0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.013+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 >> 172.21.15.3:0/149654769 conn(0x7efd5805c620 msgr2=0x7efd58128f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:22.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.013+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 shutdown_connections 2022-07-21T01:38:22.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.013+0000 7efd601a4700 1 -- 172.21.15.3:0/149654769 wait complete. 2022-07-21T01:38:22.013 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 25 2022-07-21T01:38:22.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.129+0000 7f31db78c700 1 Processor -- start 2022-07-21T01:38:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.129+0000 7f31db78c700 1 -- start start 2022-07-21T01:38:22.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31d40b1f40 0x7f31d40b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 0x7f31d40b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f31d40a7250 0x7f31d41448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d40a9250 con 0x7f31d40a7250 2022-07-21T01:38:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d40b04e0 con 0x7f31d40b1f40 2022-07-21T01:38:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40b0780 con 0x7f31d4148530 2022-07-21T01:38:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9d29700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f31d40a7250 0x7f31d41448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9d29700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f31d40a7250 0x7f31d41448b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46022/0 (socket says 172.21.15.3:46022) 2022-07-21T01:38:22.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9d29700 1 -- 172.21.15.3:0/2435137586 learned_addr learned my addr 172.21.15.3:0/2435137586 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:22.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d8d27700 1 --2- 172.21.15.3:0/2435137586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 0x7f31d40b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9528700 1 --2- 172.21.15.3:0/2435137586 >> v2:172.21.15.143:3300/0 conn(0x7f31d40b1f40 0x7f31d40b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9d29700 1 -- 172.21.15.3:0/2435137586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 msgr2=0x7f31d40b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9d29700 1 --2- 172.21.15.3:0/2435137586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 0x7f31d40b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9d29700 1 -- 172.21.15.3:0/2435137586 >> v2:172.21.15.143:3300/0 conn(0x7f31d40b1f40 msgr2=0x7f31d40b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9d29700 1 --2- 172.21.15.3:0/2435137586 >> v2:172.21.15.143:3300/0 conn(0x7f31d40b1f40 0x7f31d40b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9d29700 1 -- 172.21.15.3:0/2435137586 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d40b0a20 con 0x7f31d40a7250 2022-07-21T01:38:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d8d27700 1 --2- 172.21.15.3:0/2435137586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 0x7f31d40b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31d9d29700 1 --2- 172.21.15.3:0/2435137586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40a7250 0x7f31d41448b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f31d0002f70 tx=0x7f31d0006eb0).ready entity=mon.0 client_cookie=62a52af388bab251 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31cbfff700 1 -- 172.21.15.3:0/2435137586 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31d000d570 con 0x7f31d40a7250 2022-07-21T01:38:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31cbfff700 1 -- 172.21.15.3:0/2435137586 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d0000e90 con 0x7f31d40a7250 2022-07-21T01:38:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31cbfff700 1 -- 172.21.15.3:0/2435137586 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31d000d900 con 0x7f31d40a7250 2022-07-21T01:38:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 -- 172.21.15.3:0/2435137586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40a7250 msgr2=0x7f31d41448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 --2- 172.21.15.3:0/2435137586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40a7250 0x7f31d41448b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f31d0002f70 tx=0x7f31d0006eb0).stop 2022-07-21T01:38:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 -- 172.21.15.3:0/2435137586 shutdown_connections 2022-07-21T01:38:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 --2- 172.21.15.3:0/2435137586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40a7250 0x7f31d41448b0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 --2- 172.21.15.3:0/2435137586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 0x7f31d40b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 --2- 172.21.15.3:0/2435137586 >> v2:172.21.15.143:3300/0 conn(0x7f31d40b1f40 0x7f31d40b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 -- 172.21.15.3:0/2435137586 >> 172.21.15.3:0/2435137586 conn(0x7f31d405f840 msgr2=0x7f31d405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 -- 172.21.15.3:0/2435137586 shutdown_connections 2022-07-21T01:38:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.133+0000 7f31db78c700 1 -- 172.21.15.3:0/2435137586 wait complete. 2022-07-21T01:38:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31db78c700 1 Processor -- start 2022-07-21T01:38:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31db78c700 1 -- start start 2022-07-21T01:38:22.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31db78c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31d40a7250 0x7f31d40ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31db78c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40b1f40 0x7f31d40ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31db78c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 0x7f31d414b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31db78c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40a9750 con 0x7f31d40b1f40 2022-07-21T01:38:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31db78c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40abc60 con 0x7f31d40a7250 2022-07-21T01:38:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31db78c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40abf00 con 0x7f31d4148530 2022-07-21T01:38:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31d8d27700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40b1f40 0x7f31d40ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31d8d27700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40b1f40 0x7f31d40ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46028/0 (socket says 172.21.15.3:46028) 2022-07-21T01:38:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31d8d27700 1 -- 172.21.15.3:0/1509062999 learned_addr learned my addr 172.21.15.3:0/1509062999 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31d9528700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31d40a7250 0x7f31d40ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.137+0000 7f31d9d29700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 0x7f31d414b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31d8d27700 1 -- 172.21.15.3:0/1509062999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 msgr2=0x7f31d414b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31d8d27700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 0x7f31d414b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31d8d27700 1 -- 172.21.15.3:0/1509062999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31d40a7250 msgr2=0x7f31d40ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31d8d27700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31d40a7250 0x7f31d40ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31d8d27700 1 -- 172.21.15.3:0/1509062999 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc009470 con 0x7f31d40b1f40 2022-07-21T01:38:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31d8d27700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40b1f40 0x7f31d40ae300 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f31cc00c000 tx=0x7f31cc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31ca7fc700 1 -- 172.21.15.3:0/1509062999 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31cc00c850 con 0x7f31d40b1f40 2022-07-21T01:38:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31ca7fc700 1 -- 172.21.15.3:0/1509062999 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc00c9b0 con 0x7f31d40b1f40 2022-07-21T01:38:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d000b1e0 con 0x7f31d40b1f40 2022-07-21T01:38:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31ca7fc700 1 -- 172.21.15.3:0/1509062999 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31cc00cd10 con 0x7f31d40b1f40 2022-07-21T01:38:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d414ad10 con 0x7f31d40b1f40 2022-07-21T01:38:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d40abac0 con 0x7f31d40b1f40 2022-07-21T01:38:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.141+0000 7f31ca7fc700 1 -- 172.21.15.3:0/1509062999 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f31cc022070 con 0x7f31d40b1f40 2022-07-21T01:38:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.145+0000 7f31ca7fc700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31bc053b00 0x7f31bc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.145+0000 7f31d9528700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31bc053b00 0x7f31bc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.145+0000 7f31ca7fc700 1 -- 172.21.15.3:0/1509062999 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f31cc06c570 con 0x7f31d40b1f40 2022-07-21T01:38:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.145+0000 7f31d9528700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31bc053b00 0x7f31bc055fc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f31c400b000 tx=0x7f31c4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.153+0000 7f31ca7fc700 1 -- 172.21.15.3:0/1509062999 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f31d40abac0 con 0x7f31d40b1f40 2022-07-21T01:38:22.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.313+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31d40abac0 con 0x7f31d40b1f40 2022-07-21T01:38:22.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31ca7fc700 1 -- 172.21.15.3:0/1509062999 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+1553 (secure 0 0 0) 0x7f31d40abac0 con 0x7f31d40b1f40 2022-07-21T01:38:22.316 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:22.316 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5153,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/4189899323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4189899323},{"type":"v1","addr":"172.21.15.143:6833","nonce":4189899323}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22},{"gid":5580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/14590865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":14590865},{"type":"v1","addr":"172.21.15.3:6835","nonce":14590865}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21},{"gid":5850,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2711503784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2711503784},{"type":"v1","addr":"172.21.15.3:6837","nonce":2711503784}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":25}],"filesystems":[]} 2022-07-21T01:38:22.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31bc053b00 msgr2=0x7f31bc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31bc053b00 0x7f31bc055fc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f31c400b000 tx=0x7f31c4006df0).stop 2022-07-21T01:38:22.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40b1f40 msgr2=0x7f31d40ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40b1f40 0x7f31d40ae300 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f31cc00c000 tx=0x7f31cc0091d0).stop 2022-07-21T01:38:22.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 shutdown_connections 2022-07-21T01:38:22.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31d4148530 0x7f31d414b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31d40b1f40 0x7f31d40ae300 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31bc053b00 0x7f31bc055fc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 --2- 172.21.15.3:0/1509062999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31d40a7250 0x7f31d40ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 >> 172.21.15.3:0/1509062999 conn(0x7f31d405f840 msgr2=0x7f31d40b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:22.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 shutdown_connections 2022-07-21T01:38:22.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.317+0000 7f31db78c700 1 -- 172.21.15.3:0/1509062999 wait complete. 2022-07-21T01:38:22.324 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 25 2022-07-21T01:38:22.329 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:38:22.329 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:38:22.330 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:38:22.330 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:38:22.331 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:38:22.332 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:38:22.333 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:38:22.333 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:38:22.333 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:38:22.333 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:38:22.334 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:38:22.334 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:38:22.347 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:38:22.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:38:22.350 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:38:22.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:38:22.354 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:38:22.354 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:38:22.358 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:38:22.359 DEBUG:teuthology.parallel:result is None 2022-07-21T01:38:22.361 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:38:22.361 DEBUG:teuthology.parallel:result is None 2022-07-21T01:38:22.398 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:38:22.399 DEBUG:teuthology.parallel:result is None 2022-07-21T01:38:22.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:38:22.423 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:38:22.430 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:38:22.473 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:38:22.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.493+0000 7f614beb5700 1 Processor -- start 2022-07-21T01:38:22.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.493+0000 7f614beb5700 1 -- start start 2022-07-21T01:38:22.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61440ae3c0 0x7f6144143450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f614405c620 0x7f6144143940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f61440b0af0 0x7f61440b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614405f9c0 con 0x7f61440b0af0 2022-07-21T01:38:22.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614405fc60 con 0x7f61440ae3c0 2022-07-21T01:38:22.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614405ff00 con 0x7f614405c620 2022-07-21T01:38:22.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614a452700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f61440b0af0 0x7f61440b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614a452700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f61440b0af0 0x7f61440b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46058/0 (socket says 172.21.15.3:46058) 2022-07-21T01:38:22.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614a452700 1 -- 172.21.15.3:0/3337066268 learned_addr learned my addr 172.21.15.3:0/3337066268 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:22.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f6149450700 1 --2- 172.21.15.3:0/3337066268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f614405c620 0x7f6144143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f6149c51700 1 --2- 172.21.15.3:0/3337066268 >> v2:172.21.15.143:3300/0 conn(0x7f61440ae3c0 0x7f6144143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614a452700 1 -- 172.21.15.3:0/3337066268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f614405c620 msgr2=0x7f6144143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:22.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614a452700 1 --2- 172.21.15.3:0/3337066268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f614405c620 0x7f6144143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614a452700 1 -- 172.21.15.3:0/3337066268 >> v2:172.21.15.143:3300/0 conn(0x7f61440ae3c0 msgr2=0x7f6144143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:22.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614a452700 1 --2- 172.21.15.3:0/3337066268 >> v2:172.21.15.143:3300/0 conn(0x7f61440ae3c0 0x7f6144143450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614a452700 1 -- 172.21.15.3:0/3337066268 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61440601a0 con 0x7f61440b0af0 2022-07-21T01:38:22.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f6149c51700 1 --2- 172.21.15.3:0/3337066268 >> v2:172.21.15.143:3300/0 conn(0x7f61440ae3c0 0x7f6144143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:22.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f6149450700 1 --2- 172.21.15.3:0/3337066268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f614405c620 0x7f6144143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:22.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614a452700 1 --2- 172.21.15.3:0/3337066268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61440b0af0 0x7f61440b2720 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f613c002f70 tx=0x7f613c006eb0).ready entity=mon.0 client_cookie=6cdfbe23f6fb1a54 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:22.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f6148c4f700 1 -- 172.21.15.3:0/3337066268 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f613c00d570 con 0x7f61440b0af0 2022-07-21T01:38:22.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f6148c4f700 1 -- 172.21.15.3:0/3337066268 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c000e90 con 0x7f61440b0af0 2022-07-21T01:38:22.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f6148c4f700 1 -- 172.21.15.3:0/3337066268 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f613c00d900 con 0x7f61440b0af0 2022-07-21T01:38:22.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 -- 172.21.15.3:0/3337066268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61440b0af0 msgr2=0x7f61440b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/3337066268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61440b0af0 0x7f61440b2720 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f613c002f70 tx=0x7f613c006eb0).stop 2022-07-21T01:38:22.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 -- 172.21.15.3:0/3337066268 shutdown_connections 2022-07-21T01:38:22.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/3337066268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61440b0af0 0x7f61440b2720 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/3337066268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f614405c620 0x7f6144143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/3337066268 >> v2:172.21.15.143:3300/0 conn(0x7f61440ae3c0 0x7f6144143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 -- 172.21.15.3:0/3337066268 >> 172.21.15.3:0/3337066268 conn(0x7f614405ceb0 msgr2=0x7f6144063130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:22.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 -- 172.21.15.3:0/3337066268 shutdown_connections 2022-07-21T01:38:22.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 -- 172.21.15.3:0/3337066268 wait complete. 2022-07-21T01:38:22.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 Processor -- start 2022-07-21T01:38:22.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.497+0000 7f614beb5700 1 -- start start 2022-07-21T01:38:22.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614beb5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614405c620 0x7f6144144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614beb5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61440ae3c0 0x7f61441529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614beb5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61440b0af0 0x7f6144152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614beb5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6144152330 con 0x7f614405c620 2022-07-21T01:38:22.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614beb5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61441565b0 con 0x7f61440b0af0 2022-07-21T01:38:22.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614beb5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6144152520 con 0x7f61440ae3c0 2022-07-21T01:38:22.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149c51700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614405c620 0x7f6144144b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149c51700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614405c620 0x7f6144144b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46062/0 (socket says 172.21.15.3:46062) 2022-07-21T01:38:22.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149c51700 1 -- 172.21.15.3:0/85561321 learned_addr learned my addr 172.21.15.3:0/85561321 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:22.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614a452700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61440b0af0 0x7f6144152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149450700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61440ae3c0 0x7f61441529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149c51700 1 -- 172.21.15.3:0/85561321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61440ae3c0 msgr2=0x7f61441529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149c51700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61440ae3c0 0x7f61441529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149c51700 1 -- 172.21.15.3:0/85561321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61440b0af0 msgr2=0x7f6144152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:22.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149c51700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61440b0af0 0x7f6144152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:22.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149c51700 1 -- 172.21.15.3:0/85561321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c00b1e0 con 0x7f614405c620 2022-07-21T01:38:22.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149450700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61440ae3c0 0x7f61441529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:22.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6149c51700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614405c620 0x7f6144144b10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f6140006cc0 tx=0x7f6140008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:22.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6132ffd700 1 -- 172.21.15.3:0/85561321 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61400127e0 con 0x7f614405c620 2022-07-21T01:38:22.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6132ffd700 1 -- 172.21.15.3:0/85561321 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6140012940 con 0x7f614405c620 2022-07-21T01:38:22.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61441562e0 con 0x7f614405c620 2022-07-21T01:38:22.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6132ffd700 1 -- 172.21.15.3:0/85561321 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6140012ca0 con 0x7f614405c620 2022-07-21T01:38:22.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61440a4990 con 0x7f614405c620 2022-07-21T01:38:22.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f6132ffd700 1 -- 172.21.15.3:0/85561321 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6140013480 con 0x7f614405c620 2022-07-21T01:38:22.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.501+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6144152720 con 0x7f614405c620 2022-07-21T01:38:22.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.505+0000 7f6132ffd700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6124053950 0x7f6124055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:22.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.505+0000 7f6132ffd700 1 -- 172.21.15.3:0/85561321 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f614006c1c0 con 0x7f614405c620 2022-07-21T01:38:22.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.505+0000 7f6149450700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6124053950 0x7f6124055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:22.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.505+0000 7f6149450700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6124053950 0x7f6124055e10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f613400c000 tx=0x7f6134008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:22.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.509+0000 7f6132ffd700 1 -- 172.21.15.3:0/85561321 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6144152720 con 0x7f614405c620 2022-07-21T01:38:22.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:22.653+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f6144152720 con 0x7f614405c620 2022-07-21T01:38:23.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.493+0000 7f6132ffd700 1 -- 172.21.15.3:0/85561321 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v63) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6144152720 con 0x7f614405c620 2022-07-21T01:38:23.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6124053950 msgr2=0x7f6124055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:23.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6124053950 0x7f6124055e10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f613400c000 tx=0x7f6134008040).stop 2022-07-21T01:38:23.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614405c620 msgr2=0x7f6144144b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:23.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614405c620 0x7f6144144b10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f6140006cc0 tx=0x7f6140008040).stop 2022-07-21T01:38:23.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 shutdown_connections 2022-07-21T01:38:23.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6124053950 0x7f6124055e10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61440b0af0 0x7f6144152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61440ae3c0 0x7f61441529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 --2- 172.21.15.3:0/85561321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f614405c620 0x7f6144144b10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 >> 172.21.15.3:0/85561321 conn(0x7f614405ceb0 msgr2=0x7f61440ae7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:23.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 shutdown_connections 2022-07-21T01:38:23.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.497+0000 7f614beb5700 1 -- 172.21.15.3:0/85561321 wait complete. 2022-07-21T01:38:23.501 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:38:23.510 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:38:23.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:38:23.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.665+0000 7fb10f59e700 1 Processor -- start 2022-07-21T01:38:23.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.665+0000 7fb10f59e700 1 -- start start 2022-07-21T01:38:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb1100b65c0 0x7fb110061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb110147d80 0x7fb110144d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10f59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1100b2100 con 0x7fb1100b41f0 2022-07-21T01:38:23.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1100b1630 con 0x7fb1100b65c0 2022-07-21T01:38:23.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100b69a0 con 0x7fb110147d80 2022-07-21T01:38:23.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10dd9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb1100b41f0 0x7fb110144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54352/0 (socket says 172.21.15.3:54352) 2022-07-21T01:38:23.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10dd9b700 1 -- 172.21.15.3:0/141720136 learned_addr learned my addr 172.21.15.3:0/141720136 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:23.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10ed9d700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb110147d80 0x7fb110144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:23.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10e59c700 1 --2- 172.21.15.3:0/141720136 >> v2:172.21.15.143:3300/0 conn(0x7fb1100b65c0 0x7fb110061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:23.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735499750 0 0) 0x7fb1100b2100 con 0x7fb1100b41f0 2022-07-21T01:38:23.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4003260 con 0x7fb1100b41f0 2022-07-21T01:38:23.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3575895563 0 0) 0x7fb0f4003260 con 0x7fb1100b41f0 2022-07-21T01:38:23.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb110147d80 msgr2=0x7fb110144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb110147d80 0x7fb110144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 >> v2:172.21.15.143:3300/0 conn(0x7fb1100b65c0 msgr2=0x7fb110061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 --2- 172.21.15.3:0/141720136 >> v2:172.21.15.143:3300/0 conn(0x7fb1100b65c0 0x7fb110061600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1101482a0 con 0x7fb1100b41f0 2022-07-21T01:38:23.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fb0f8001660 con 0x7fb1100b41f0 2022-07-21T01:38:23.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10e59c700 1 --2- 172.21.15.3:0/141720136 >> v2:172.21.15.143:3300/0 conn(0x7fb1100b65c0 0x7fb110061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:23.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 >> v1:172.21.15.3:6789/0 conn(0x7fb1100b41f0 legacy=0x7fb110144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:23.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb110147d80 0x7fb0f4003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:23.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0f4003c80 0x7fb0f4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:23.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb0f4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100b1630 con 0x7fb0f40049f0 2022-07-21T01:38:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100b69a0 con 0x7fb0f4003c80 2022-07-21T01:38:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f400b780 con 0x7fb110147d80 2022-07-21T01:38:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10ed9d700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb0f4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:23.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10dd9b700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0f4003c80 0x7fb0f4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:23.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10e59c700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb110147d80 0x7fb0f4003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:23.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10ed9d700 1 -- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb110147d80 msgr2=0x7fb0f4003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:23.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10ed9d700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb110147d80 0x7fb0f4003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10ed9d700 1 -- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0f4003c80 msgr2=0x7fb0f4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10ed9d700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0f4003c80 0x7fb0f4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10ed9d700 1 -- 172.21.15.3:0/141720136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4003260 con 0x7fb0f40049f0 2022-07-21T01:38:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10ed9d700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb0f4004e70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fb10800c000 tx=0x7fb1080091d0).ready entity=mon.0 client_cookie=380fc083c411eef6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb10800e780 con 0x7fb0f40049f0 2022-07-21T01:38:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10800e8e0 con 0x7fb0f40049f0 2022-07-21T01:38:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.669+0000 7fb10d59a700 1 -- 172.21.15.3:0/141720136 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb10800ec40 con 0x7fb0f40049f0 2022-07-21T01:38:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 -- 172.21.15.3:0/141720136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 msgr2=0x7fb0f4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 --2- 172.21.15.3:0/141720136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb0f4004e70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fb10800c000 tx=0x7fb1080091d0).stop 2022-07-21T01:38:23.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10ed9d700 1 -- 172.21.15.3:0/141720136 reap_dead start 2022-07-21T01:38:23.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 -- 172.21.15.3:0/141720136 shutdown_connections 2022-07-21T01:38:23.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 -- 172.21.15.3:0/141720136 >> 172.21.15.3:0/141720136 conn(0x7fb11005c620 msgr2=0x7fb11005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:23.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 -- 172.21.15.3:0/141720136 shutdown_connections 2022-07-21T01:38:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 -- 172.21.15.3:0/141720136 wait complete. 2022-07-21T01:38:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 Processor -- start 2022-07-21T01:38:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 -- start start 2022-07-21T01:38:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb1100b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1100aec10 0x7fb1100b87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb110145290 0x7fb1101436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100b8380 con 0x7fb0f40049f0 2022-07-21T01:38:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1101468b0 con 0x7fb110145290 2022-07-21T01:38:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb110146b50 con 0x7fb1100aec10 2022-07-21T01:38:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb1100b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1100aec10 0x7fb1100b87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb1100b0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46084/0 (socket says 172.21.15.3:46084) 2022-07-21T01:38:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10ed9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb110145290 0x7fb1101436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:23.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.673+0000 7fb10e59c700 1 -- 172.21.15.3:0/2114376395 learned_addr learned my addr 172.21.15.3:0/2114376395 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:23.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10e59c700 1 -- 172.21.15.3:0/2114376395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1100aec10 msgr2=0x7fb1100b87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:23.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10e59c700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1100aec10 0x7fb1100b87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10e59c700 1 -- 172.21.15.3:0/2114376395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb110145290 msgr2=0x7fb1101436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10e59c700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb110145290 0x7fb1101436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10e59c700 1 -- 172.21.15.3:0/2114376395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1101573b0 con 0x7fb0f40049f0 2022-07-21T01:38:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10dd9b700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1100aec10 0x7fb1100b87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10e59c700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb1100b0650 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7fb100006a80 tx=0x7fb10000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb1077fe700 1 -- 172.21.15.3:0/2114376395 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb100012800 con 0x7fb0f40049f0 2022-07-21T01:38:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb1077fe700 1 -- 172.21.15.3:0/2114376395 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb100012960 con 0x7fb0f40049f0 2022-07-21T01:38:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb1077fe700 1 -- 172.21.15.3:0/2114376395 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb100012cc0 con 0x7fb0f40049f0 2022-07-21T01:38:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb110157650 con 0x7fb0f40049f0 2022-07-21T01:38:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1101464e0 con 0x7fb0f40049f0 2022-07-21T01:38:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1100b84f0 con 0x7fb0f40049f0 2022-07-21T01:38:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.677+0000 7fb1077fe700 1 -- 172.21.15.3:0/2114376395 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb100014470 con 0x7fb0f40049f0 2022-07-21T01:38:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.681+0000 7fb1077fe700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb0e8053b30 0x7fb0e8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.681+0000 7fb1077fe700 1 -- 172.21.15.3:0/2114376395 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fb10006c050 con 0x7fb0f40049f0 2022-07-21T01:38:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.681+0000 7fb10dd9b700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb0e8053b30 0x7fb0e8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.681+0000 7fb10dd9b700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb0e8053b30 0x7fb0e8055ff0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fb0f8005000 tx=0x7fb0f8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.685+0000 7fb1077fe700 1 -- 172.21.15.3:0/2114376395 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb1100b84f0 con 0x7fb0f40049f0 2022-07-21T01:38:23.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.853+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb1100b84c0 con 0x7fb0f40049f0 2022-07-21T01:38:23.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb1077fe700 1 -- 172.21.15.3:0/2114376395 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fb1000430f0 con 0x7fb0f40049f0 2022-07-21T01:38:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:23.856 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:38:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:38:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:38:23.858 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:38:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:38:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:38:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:38:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:38:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:38:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:38:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:38:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:38:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:38:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.875 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.875 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.876 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.877 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:38:23.878 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:38:23.878 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.878 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.879 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.879 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.880 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.881 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:38:23.882 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:38:23.882 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.883 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.883 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.884 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.884 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.885 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:38:23.886 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:38:23.886 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.887 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.887 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.888 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.888 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.889 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:38:23.890 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:38:23.890 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.891 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.891 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.892 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.892 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.894 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:38:23.894 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:38:23.895 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.895 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.895 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.896 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.896 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.897 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:38:23.898 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:38:23.898 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.910 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.910 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.910 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.911 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:38:23.911 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:38:23.912 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.912 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.912 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.912 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.913 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:38:23.914 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:38:23.914 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.914 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:38:23.914 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:38:23.915 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:38:23.915 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.915 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:38:23.916 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:38:23.916 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:38:23.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:38:23.917 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:38:23.917 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.918 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:38:23.918 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:38:23.918 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:38:23.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:38:23.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:38:23.919 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.920 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.920 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:38:23.920 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:38:23.921 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:38:23.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:38:23.921 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.922 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:23.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:23.922 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:38:23.923 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:38:23.923 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:38:23.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:38:23.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:38:23.923 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:38:23.924 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.924 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:23.924 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:23.924 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:38:23.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb0e8053b30 msgr2=0x7fb0e8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:23.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb0e8053b30 0x7fb0e8055ff0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fb0f8005000 tx=0x7fb0f8009040).stop 2022-07-21T01:38:23.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 msgr2=0x7fb1100b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:23.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb1100b0650 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7fb100006a80 tx=0x7fb10000b040).stop 2022-07-21T01:38:23.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 shutdown_connections 2022-07-21T01:38:23.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb0e8053b30 0x7fb0e8055ff0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb110145290 0x7fb1101436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1100aec10 0x7fb1100b87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 --2- 172.21.15.3:0/2114376395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f40049f0 0x7fb1100b0650 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:23.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 >> 172.21.15.3:0/2114376395 conn(0x7fb11005c620 msgr2=0x7fb1100b97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:23.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 shutdown_connections 2022-07-21T01:38:23.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:23.857+0000 7fb10f59e700 1 -- 172.21.15.3:0/2114376395 wait complete. 2022-07-21T01:38:23.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:38:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.041+0000 7f979538e700 1 Processor -- start 2022-07-21T01:38:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.041+0000 7f979538e700 1 -- start start 2022-07-21T01:38:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.041+0000 7f979538e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f97900b08a0 0x7f9790142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.041+0000 7f979538e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9790143ab0 con 0x7f97900b3260 2022-07-21T01:38:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.041+0000 7f979538e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97900b0c80 con 0x7f97900ad000 2022-07-21T01:38:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.041+0000 7f979538e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97900ae810 con 0x7f97900b08a0 2022-07-21T01:38:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.041+0000 7f978f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f97900b3260 0x7f97901433a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54372/0 (socket says 172.21.15.3:54372) 2022-07-21T01:38:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.041+0000 7f978f7fe700 1 -- 172.21.15.3:0/2650755616 learned_addr learned my addr 172.21.15.3:0/2650755616 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978ffff700 1 --2- 172.21.15.3:0/2650755616 >> v2:172.21.15.143:3300/0 conn(0x7f97900b08a0 0x7f9790142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557046181 0 0) 0x7f9790143ab0 con 0x7f97900b3260 2022-07-21T01:38:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9770004370 con 0x7f97900b3260 2022-07-21T01:38:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 159961171 0 0) 0x7f9770004370 con 0x7f97900b3260 2022-07-21T01:38:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 >> v2:172.21.15.143:3300/0 conn(0x7f97900b08a0 msgr2=0x7f9790142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 --2- 172.21.15.3:0/2650755616 >> v2:172.21.15.143:3300/0 conn(0x7f97900b08a0 0x7f9790142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 >> v1:172.21.15.143:6789/0 conn(0x7f97900ad000 legacy=0x7f979013fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900aeab0 con 0x7f97900b3260 2022-07-21T01:38:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f9778001660 con 0x7f97900b3260 2022-07-21T01:38:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 >> v1:172.21.15.3:6789/0 conn(0x7f97900b3260 legacy=0x7f97901433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97700048b0 0x7f9770004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9770005610 0x7f9770005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9770005fd0 0x7f977000c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97900ae810 con 0x7f97700048b0 2022-07-21T01:38:24.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977000ce90 con 0x7f9770005610 2022-07-21T01:38:24.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977000d050 con 0x7f9770005fd0 2022-07-21T01:38:24.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978ffff700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97700048b0 0x7f9770004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978f7fe700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9770005610 0x7f9770005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f9794b8d700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9770005fd0 0x7f977000c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978ffff700 1 -- 172.21.15.3:0/2650755616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9770005fd0 msgr2=0x7f977000c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:24.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978ffff700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9770005fd0 0x7f977000c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978ffff700 1 -- 172.21.15.3:0/2650755616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9770005610 msgr2=0x7f9770005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978ffff700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9770005610 0x7f9770005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978ffff700 1 -- 172.21.15.3:0/2650755616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9770004370 con 0x7f97700048b0 2022-07-21T01:38:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f9794b8d700 1 -- 172.21.15.3:0/2650755616 reap_dead start 2022-07-21T01:38:24.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978ffff700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97700048b0 0x7f9770004cf0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f978000c000 tx=0x7f9780006df0).ready entity=mon.0 client_cookie=cc63721378225647 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f978000e830 con 0x7f97700048b0 2022-07-21T01:38:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978000e990 con 0x7f97700048b0 2022-07-21T01:38:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f978effd700 1 -- 172.21.15.3:0/2650755616 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f978000ecf0 con 0x7f97700048b0 2022-07-21T01:38:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f979538e700 1 -- 172.21.15.3:0/2650755616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97700048b0 msgr2=0x7f9770004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f979538e700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97700048b0 0x7f9770004cf0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f978000c000 tx=0x7f9780006df0).stop 2022-07-21T01:38:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f979538e700 1 -- 172.21.15.3:0/2650755616 shutdown_connections 2022-07-21T01:38:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f979538e700 1 --2- 172.21.15.3:0/2650755616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97700048b0 0x7f9770004cf0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f979538e700 1 -- 172.21.15.3:0/2650755616 >> 172.21.15.3:0/2650755616 conn(0x7f97900612d0 msgr2=0x7f9790062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f979538e700 1 -- 172.21.15.3:0/2650755616 shutdown_connections 2022-07-21T01:38:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.045+0000 7f979538e700 1 -- 172.21.15.3:0/2650755616 wait complete. 2022-07-21T01:38:24.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 Processor -- start 2022-07-21T01:38:24.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 -- start start 2022-07-21T01:38:24.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97700048b0 0x7f9790063e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9790066120 0x7f979013ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f979013d1c0 0x7f979014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9790067e70 con 0x7f979013d1c0 2022-07-21T01:38:24.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979009ee60 con 0x7f9790066120 2022-07-21T01:38:24.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979009ebc0 con 0x7f97700048b0 2022-07-21T01:38:24.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f9794b8d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f979013d1c0 0x7f979014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f978f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9790066120 0x7f979013ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f9794b8d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f979013d1c0 0x7f979014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46108/0 (socket says 172.21.15.3:46108) 2022-07-21T01:38:24.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f9794b8d700 1 -- 172.21.15.3:0/1523638080 learned_addr learned my addr 172.21.15.3:0/1523638080 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:24.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f978ffff700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97700048b0 0x7f9790063e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f9794b8d700 1 -- 172.21.15.3:0/1523638080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97700048b0 msgr2=0x7f9790063e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f9794b8d700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97700048b0 0x7f9790063e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f9794b8d700 1 -- 172.21.15.3:0/1523638080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9790066120 msgr2=0x7f979013ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f9794b8d700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9790066120 0x7f979013ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f9794b8d700 1 -- 172.21.15.3:0/1523638080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900679d0 con 0x7f979013d1c0 2022-07-21T01:38:24.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f9794b8d700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f979013d1c0 0x7f979014e1f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f9788001210 tx=0x7f9788009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f978d7fa700 1 -- 172.21.15.3:0/1523638080 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9788003220 con 0x7f979013d1c0 2022-07-21T01:38:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f978d7fa700 1 -- 172.21.15.3:0/1523638080 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9788003380 con 0x7f979013d1c0 2022-07-21T01:38:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9790064490 con 0x7f979013d1c0 2022-07-21T01:38:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f978d7fa700 1 -- 172.21.15.3:0/1523638080 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9788001770 con 0x7f979013d1c0 2022-07-21T01:38:24.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.049+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9790064ef0 con 0x7f979013d1c0 2022-07-21T01:38:24.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.053+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9790067b60 con 0x7f979013d1c0 2022-07-21T01:38:24.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.053+0000 7f978d7fa700 1 -- 172.21.15.3:0/1523638080 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f978801c060 con 0x7f979013d1c0 2022-07-21T01:38:24.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.053+0000 7f978d7fa700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9774053980 0x7f9774055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.053+0000 7f978d7fa700 1 -- 172.21.15.3:0/1523638080 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f978806b5d0 con 0x7f979013d1c0 2022-07-21T01:38:24.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.053+0000 7f978ffff700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9774053980 0x7f9774055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.053+0000 7f978ffff700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9774053980 0x7f9774055e40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f978001b000 tx=0x7f9780019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.061+0000 7f978d7fa700 1 -- 172.21.15.3:0/1523638080 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f97880426d0 con 0x7f979013d1c0 2022-07-21T01:38:24.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.185+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97900574e0 con 0x7f979013d1c0 2022-07-21T01:38:24.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.185+0000 7f978d7fa700 1 -- 172.21.15.3:0/1523638080 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+13415 (secure 0 0 0) 0x7f978801a020 con 0x7f979013d1c0 2022-07-21T01:38:24.186 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:24.186 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":63,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:38:23.485077+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:38:24.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9774053980 msgr2=0x7f9774055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9774053980 0x7f9774055e40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f978001b000 tx=0x7f9780019000).stop 2022-07-21T01:38:24.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f979013d1c0 msgr2=0x7f979014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f979013d1c0 0x7f979014e1f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f9788001210 tx=0x7f9788009040).stop 2022-07-21T01:38:24.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 shutdown_connections 2022-07-21T01:38:24.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f979013d1c0 0x7f979014e1f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9774053980 0x7f9774055e40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9790066120 0x7f979013ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 --2- 172.21.15.3:0/1523638080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97700048b0 0x7f9790063e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 >> 172.21.15.3:0/1523638080 conn(0x7f97900612d0 msgr2=0x7f97900ac250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:24.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 shutdown_connections 2022-07-21T01:38:24.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.189+0000 7f979538e700 1 -- 172.21.15.3:0/1523638080 wait complete. 2022-07-21T01:38:24.200 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:38:24.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:38:24.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.309+0000 7faa2388f700 1 Processor -- start 2022-07-21T01:38:24.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.309+0000 7faa2388f700 1 -- start start 2022-07-21T01:38:24.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa2388f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faa1c0b2b30 0x7faa1c0b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa2388f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c140100 0x7faa1c0b1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa2388f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c0aecc0 con 0x7faa1c0b5b80 2022-07-21T01:38:24.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa2388f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1c0aea20 con 0x7faa1c0b2b30 2022-07-21T01:38:24.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa2388f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c13f9c0 con 0x7faa1c140100 2022-07-21T01:38:24.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa21e2c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c140100 0x7faa1c0b1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa20e2a700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7faa1c0b5b80 0x7faa1c0ae1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:56510/0 (socket says 172.21.15.3:56510) 2022-07-21T01:38:24.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa20e2a700 1 -- 172.21.15.3:0/1528480647 learned_addr learned my addr 172.21.15.3:0/1528480647 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:24.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa2162b700 1 --2- 172.21.15.3:0/1528480647 >> v2:172.21.15.143:3300/0 conn(0x7faa1c0b2b30 0x7faa1c0b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa0bfff700 1 -- 172.21.15.3:0/1528480647 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055502951 0 0) 0x7faa1c0aecc0 con 0x7faa1c0b5b80 2022-07-21T01:38:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa0bfff700 1 -- 172.21.15.3:0/1528480647 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa04003260 con 0x7faa1c0b5b80 2022-07-21T01:38:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa21e2c700 1 -- 172.21.15.3:0/1528480647 >> v2:172.21.15.143:3300/0 conn(0x7faa1c0b2b30 msgr2=0x7faa1c0b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa21e2c700 1 --2- 172.21.15.3:0/1528480647 >> v2:172.21.15.143:3300/0 conn(0x7faa1c0b2b30 0x7faa1c0b3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa21e2c700 1 -- 172.21.15.3:0/1528480647 >> v1:172.21.15.143:6789/0 conn(0x7faa1c0b5b80 legacy=0x7faa1c0ae1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa21e2c700 1 -- 172.21.15.3:0/1528480647 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c13fc60 con 0x7faa1c140100 2022-07-21T01:38:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa2162b700 1 --2- 172.21.15.3:0/1528480647 >> v2:172.21.15.143:3300/0 conn(0x7faa1c0b2b30 0x7faa1c0b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa21e2c700 1 --2- 172.21.15.3:0/1528480647 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c140100 0x7faa1c0b1930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7faa140032f0 tx=0x7faa1400a370).ready entity=mon.2 client_cookie=69ffed8f1a8c8d4f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.313+0000 7faa0bfff700 1 -- 172.21.15.3:0/1528480647 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa1400cba0 con 0x7faa1c140100 2022-07-21T01:38:24.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa0bfff700 1 -- 172.21.15.3:0/1528480647 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1400cd00 con 0x7faa1c140100 2022-07-21T01:38:24.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa0bfff700 1 -- 172.21.15.3:0/1528480647 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa14010600 con 0x7faa1c140100 2022-07-21T01:38:24.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 -- 172.21.15.3:0/1528480647 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c140100 msgr2=0x7faa1c0b1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 --2- 172.21.15.3:0/1528480647 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c140100 0x7faa1c0b1930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7faa140032f0 tx=0x7faa1400a370).stop 2022-07-21T01:38:24.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 -- 172.21.15.3:0/1528480647 shutdown_connections 2022-07-21T01:38:24.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 --2- 172.21.15.3:0/1528480647 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c140100 0x7faa1c0b1930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 --2- 172.21.15.3:0/1528480647 >> v2:172.21.15.143:3300/0 conn(0x7faa1c0b2b30 0x7faa1c0b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 -- 172.21.15.3:0/1528480647 >> 172.21.15.3:0/1528480647 conn(0x7faa1c06a2a0 msgr2=0x7faa1c05ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:24.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 -- 172.21.15.3:0/1528480647 shutdown_connections 2022-07-21T01:38:24.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 -- 172.21.15.3:0/1528480647 wait complete. 2022-07-21T01:38:24.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 Processor -- start 2022-07-21T01:38:24.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 -- start start 2022-07-21T01:38:24.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa1c0b2b30 0x7faa1c1419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c0b5b80 0x7faa1c134170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa1c140100 0x7faa1c134660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c13f150 con 0x7faa1c0b2b30 2022-07-21T01:38:24.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c137de0 con 0x7faa1c140100 2022-07-21T01:38:24.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2388f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c13c4e0 con 0x7faa1c0b5b80 2022-07-21T01:38:24.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.317+0000 7faa2162b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa1c0b2b30 0x7faa1c1419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2162b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa1c0b2b30 0x7faa1c1419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46118/0 (socket says 172.21.15.3:46118) 2022-07-21T01:38:24.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2162b700 1 -- 172.21.15.3:0/454570577 learned_addr learned my addr 172.21.15.3:0/454570577 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:24.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa20e2a700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c0b5b80 0x7faa1c134170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa21e2c700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa1c140100 0x7faa1c134660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2162b700 1 -- 172.21.15.3:0/454570577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c0b5b80 msgr2=0x7faa1c134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2162b700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c0b5b80 0x7faa1c134170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2162b700 1 -- 172.21.15.3:0/454570577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa1c140100 msgr2=0x7faa1c134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2162b700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa1c140100 0x7faa1c134660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2162b700 1 -- 172.21.15.3:0/454570577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1800b180 con 0x7faa1c0b2b30 2022-07-21T01:38:24.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2162b700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa1c0b2b30 0x7faa1c1419e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7faa1800c000 tx=0x7faa18006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa0a7fc700 1 -- 172.21.15.3:0/454570577 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa1800c5f0 con 0x7faa1c0b2b30 2022-07-21T01:38:24.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa0a7fc700 1 -- 172.21.15.3:0/454570577 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1800c750 con 0x7faa1c0b2b30 2022-07-21T01:38:24.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1400a610 con 0x7faa1c0b2b30 2022-07-21T01:38:24.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa0a7fc700 1 -- 172.21.15.3:0/454570577 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa1800cab0 con 0x7faa1c0b2b30 2022-07-21T01:38:24.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c137b40 con 0x7faa1c0b2b30 2022-07-21T01:38:24.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c0b2f10 con 0x7faa1c0b2b30 2022-07-21T01:38:24.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.321+0000 7faa0a7fc700 1 -- 172.21.15.3:0/454570577 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faa1800cc10 con 0x7faa1c0b2b30 2022-07-21T01:38:24.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.325+0000 7faa0a7fc700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9fc0539e0 0x7fa9fc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.325+0000 7faa0a7fc700 1 -- 172.21.15.3:0/454570577 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5600+0+0 (secure 0 0 0) 0x7faa1806c300 con 0x7faa1c0b2b30 2022-07-21T01:38:24.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.325+0000 7faa20e2a700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9fc0539e0 0x7fa9fc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.325+0000 7faa20e2a700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9fc0539e0 0x7fa9fc055ea0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7faa0c00d000 tx=0x7faa0c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.329+0000 7faa0a7fc700 1 -- 172.21.15.3:0/454570577 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faa180432f0 con 0x7faa1c0b2b30 2022-07-21T01:38:24.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.457+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7faa1c0574e0 con 0x7faa1c0b2b30 2022-07-21T01:38:24.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.457+0000 7faa0a7fc700 1 -- 172.21.15.3:0/454570577 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v63) v1 ==== 110+0+10 (secure 0 0 0) 0x7faa18046910 con 0x7faa1c0b2b30 2022-07-21T01:38:24.457 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:38:24.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9fc0539e0 msgr2=0x7fa9fc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9fc0539e0 0x7fa9fc055ea0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7faa0c00d000 tx=0x7faa0c009040).stop 2022-07-21T01:38:24.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa1c0b2b30 msgr2=0x7faa1c1419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa1c0b2b30 0x7faa1c1419e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7faa1800c000 tx=0x7faa18006df0).stop 2022-07-21T01:38:24.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 shutdown_connections 2022-07-21T01:38:24.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9fc0539e0 0x7fa9fc055ea0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa1c140100 0x7faa1c134660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1c0b5b80 0x7faa1c134170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 --2- 172.21.15.3:0/454570577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa1c0b2b30 0x7faa1c1419e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 >> 172.21.15.3:0/454570577 conn(0x7faa1c06a2a0 msgr2=0x7faa1c0b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:24.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 shutdown_connections 2022-07-21T01:38:24.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.461+0000 7faa2388f700 1 -- 172.21.15.3:0/454570577 wait complete. 2022-07-21T01:38:24.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:38:24.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.581+0000 7f3f81da0700 1 Processor -- start 2022-07-21T01:38:24.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.581+0000 7f3f81da0700 1 -- start start 2022-07-21T01:38:24.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 0x7f3f7c0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3f7c05c6f0 0x7f3f7c0a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c0aa630 con 0x7f3f7c060330 2022-07-21T01:38:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c0b15e0 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0af930 con 0x7f3f7c0af550 2022-07-21T01:38:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f8159f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3f7c05c6f0 0x7f3f7c0a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f8159f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3f7c05c6f0 0x7f3f7c0a8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46130/0 (socket says 172.21.15.3:46130) 2022-07-21T01:38:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f8159f700 1 -- 172.21.15.3:0/2328829037 learned_addr learned my addr 172.21.15.3:0/2328829037 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f80d9e700 1 --2- 172.21.15.3:0/2328829037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 0x7f3f7c0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f8159f700 1 -- 172.21.15.3:0/2328829037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 msgr2=0x7f3f7c0b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f8159f700 1 --2- 172.21.15.3:0/2328829037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 0x7f3f7c0b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f8159f700 1 -- 172.21.15.3:0/2328829037 >> v1:172.21.15.143:6789/0 conn(0x7f3f7c060330 legacy=0x7f3f7c1445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f8159f700 1 -- 172.21.15.3:0/2328829037 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c05fd10 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f8159f700 1 --2- 172.21.15.3:0/2328829037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 0x7f3f7c0a8d80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f3f740032f0 tx=0x7f3f74009290).ready entity=mon.0 client_cookie=ceb51e1eeec7f67b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f7b7fe700 1 -- 172.21.15.3:0/2328829037 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f7400b8a0 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f7b7fe700 1 -- 172.21.15.3:0/2328829037 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7400ba00 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f7b7fe700 1 -- 172.21.15.3:0/2328829037 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f7400bb70 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 -- 172.21.15.3:0/2328829037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 msgr2=0x7f3f7c0a8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 --2- 172.21.15.3:0/2328829037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 0x7f3f7c0a8d80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f3f740032f0 tx=0x7f3f74009290).stop 2022-07-21T01:38:24.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 -- 172.21.15.3:0/2328829037 shutdown_connections 2022-07-21T01:38:24.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 --2- 172.21.15.3:0/2328829037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 0x7f3f7c0a8d80 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 --2- 172.21.15.3:0/2328829037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 0x7f3f7c0b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 -- 172.21.15.3:0/2328829037 >> 172.21.15.3:0/2328829037 conn(0x7f3f7c0612d0 msgr2=0x7f3f7c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:24.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 -- 172.21.15.3:0/2328829037 shutdown_connections 2022-07-21T01:38:24.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.585+0000 7f3f81da0700 1 -- 172.21.15.3:0/2328829037 wait complete. 2022-07-21T01:38:24.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f81da0700 1 Processor -- start 2022-07-21T01:38:24.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f81da0700 1 -- start start 2022-07-21T01:38:24.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f81da0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 0x7f3f7c150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f81da0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7c060330 0x7f3f7c065310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f81da0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 0x7f3f7c140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f81da0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c13f050 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f81da0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c13f280 con 0x7f3f7c060330 2022-07-21T01:38:24.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f81da0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c13f520 con 0x7f3f7c0af550 2022-07-21T01:38:24.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f80d9e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 0x7f3f7c150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f80d9e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 0x7f3f7c150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46132/0 (socket says 172.21.15.3:46132) 2022-07-21T01:38:24.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f80d9e700 1 -- 172.21.15.3:0/2852552601 learned_addr learned my addr 172.21.15.3:0/2852552601 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:24.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f8159f700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 0x7f3f7c140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.589+0000 7f3f7bfff700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7c060330 0x7f3f7c065310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f80d9e700 1 -- 172.21.15.3:0/2852552601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 msgr2=0x7f3f7c140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f80d9e700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 0x7f3f7c140550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f80d9e700 1 -- 172.21.15.3:0/2852552601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7c060330 msgr2=0x7f3f7c065310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f80d9e700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7c060330 0x7f3f7c065310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f80d9e700 1 -- 172.21.15.3:0/2852552601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f74009530 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f80d9e700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 0x7f3f7c150f50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f3f6400c000 tx=0x7f3f64006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f79ffb700 1 -- 172.21.15.3:0/2852552601 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f64011750 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f79ffb700 1 -- 172.21.15.3:0/2852552601 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f640118b0 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c05f840 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f79ffb700 1 -- 172.21.15.3:0/2852552601 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f64011c10 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c063ad0 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c05fa70 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f79ffb700 1 -- 172.21.15.3:0/2852552601 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3f64011d70 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f79ffb700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f60053950 0x7f3f60055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.593+0000 7f3f79ffb700 1 -- 172.21.15.3:0/2852552601 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f3f6406c0f0 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.597+0000 7f3f7bfff700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f60053950 0x7f3f60055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.597+0000 7f3f7bfff700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f60053950 0x7f3f60055e10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f3f6c00d000 tx=0x7f3f6c000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.601+0000 7f3f79ffb700 1 -- 172.21.15.3:0/2852552601 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f64043190 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3f7c0574e0 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f79ffb700 1 -- 172.21.15.3:0/2852552601 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v63) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3f640467b0 con 0x7f3f7c05c6f0 2022-07-21T01:38:24.724 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:38:24.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f60053950 msgr2=0x7f3f60055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f60053950 0x7f3f60055e10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f3f6c00d000 tx=0x7f3f6c000bc0).stop 2022-07-21T01:38:24.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 msgr2=0x7f3f7c150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 0x7f3f7c150f50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f3f6400c000 tx=0x7f3f64006df0).stop 2022-07-21T01:38:24.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 shutdown_connections 2022-07-21T01:38:24.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f7c0af550 0x7f3f7c140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f60053950 0x7f3f60055e10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7c060330 0x7f3f7c065310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 --2- 172.21.15.3:0/2852552601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f7c05c6f0 0x7f3f7c150f50 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 >> 172.21.15.3:0/2852552601 conn(0x7f3f7c0612d0 msgr2=0x7f3f7c0ac1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:24.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 shutdown_connections 2022-07-21T01:38:24.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.725+0000 7f3f81da0700 1 -- 172.21.15.3:0/2852552601 wait complete. 2022-07-21T01:38:24.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:38:24.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.837+0000 7ff40359e700 1 Processor -- start 2022-07-21T01:38:24.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.837+0000 7ff40359e700 1 -- start start 2022-07-21T01:38:24.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff4040ad670 0x7ff4040b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40359e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4040b0c40 con 0x7ff40405c620 2022-07-21T01:38:24.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4040b0ee0 con 0x7ff4040af120 2022-07-21T01:38:24.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4040ae9e0 con 0x7ff4040ad670 2022-07-21T01:38:24.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff402d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff40405c620 0x7ff40413fa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54424/0 (socket says 172.21.15.3:54424) 2022-07-21T01:38:24.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff402d9d700 1 -- 172.21.15.3:0/1743704394 learned_addr learned my addr 172.21.15.3:0/1743704394 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:24.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40259c700 1 --2- 172.21.15.3:0/1743704394 >> v2:172.21.15.143:3300/0 conn(0x7ff4040ad670 0x7ff4040b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999313724 0 0) 0x7ff4040b0c40 con 0x7ff40405c620 2022-07-21T01:38:24.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e8003260 con 0x7ff40405c620 2022-07-21T01:38:24.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015159811 0 0) 0x7ff4040b0ee0 con 0x7ff4040af120 2022-07-21T01:38:24.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4040b0c40 con 0x7ff4040af120 2022-07-21T01:38:24.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 533122354 0 0) 0x7ff3e8003260 con 0x7ff40405c620 2022-07-21T01:38:24.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 >> v2:172.21.15.143:3300/0 conn(0x7ff4040ad670 msgr2=0x7ff4040b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:24.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 --2- 172.21.15.3:0/1743704394 >> v2:172.21.15.143:3300/0 conn(0x7ff4040ad670 0x7ff4040b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 >> v1:172.21.15.143:6789/0 conn(0x7ff4040af120 legacy=0x7ff4040b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4040aec80 con 0x7ff40405c620 2022-07-21T01:38:24.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7ff3fc0019e0 con 0x7ff40405c620 2022-07-21T01:38:24.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 >> v1:172.21.15.3:6789/0 conn(0x7ff40405c620 legacy=0x7ff40413fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3e80038e0 0x7ff3e8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3e8004500 0x7ff3e8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e8004e70 0x7ff3e800d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4040ae9e0 con 0x7ff3e8004500 2022-07-21T01:38:24.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e800df00 con 0x7ff3e8004e70 2022-07-21T01:38:24.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e800e0c0 con 0x7ff3e80038e0 2022-07-21T01:38:24.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff401d9b700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3e8004500 0x7ff3e8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff402d9d700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e8004e70 0x7ff3e800d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40259c700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3e80038e0 0x7ff3e8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff401d9b700 1 -- 172.21.15.3:0/1743704394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3e80038e0 msgr2=0x7ff3e8003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:24.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff401d9b700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3e80038e0 0x7ff3e8003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff401d9b700 1 -- 172.21.15.3:0/1743704394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e8004e70 msgr2=0x7ff3e800d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:24.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff401d9b700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e8004e70 0x7ff3e800d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff401d9b700 1 -- 172.21.15.3:0/1743704394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e8003260 con 0x7ff3e8004500 2022-07-21T01:38:24.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff402d9d700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e8004e70 0x7ff3e800d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:24.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff402d9d700 1 -- 172.21.15.3:0/1743704394 reap_dead start 2022-07-21T01:38:24.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff401d9b700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3e8004500 0x7ff3e8004980 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7ff3f4001110 tx=0x7ff3f4007a00).ready entity=mon.0 client_cookie=7bb83af8096fbad6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3f4007d10 con 0x7ff3e8004500 2022-07-21T01:38:24.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f4007070 con 0x7ff3e8004500 2022-07-21T01:38:24.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40159a700 1 -- 172.21.15.3:0/1743704394 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3f40073d0 con 0x7ff3e8004500 2022-07-21T01:38:24.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40359e700 1 -- 172.21.15.3:0/1743704394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3e8004500 msgr2=0x7ff3e8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.841+0000 7ff40359e700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3e8004500 0x7ff3e8004980 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7ff3f4001110 tx=0x7ff3f4007a00).stop 2022-07-21T01:38:24.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- 172.21.15.3:0/1743704394 shutdown_connections 2022-07-21T01:38:24.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 --2- 172.21.15.3:0/1743704394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3e8004500 0x7ff3e8004980 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- 172.21.15.3:0/1743704394 >> 172.21.15.3:0/1743704394 conn(0x7ff404061360 msgr2=0x7ff404062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:24.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- 172.21.15.3:0/1743704394 shutdown_connections 2022-07-21T01:38:24.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- 172.21.15.3:0/1743704394 wait complete. 2022-07-21T01:38:24.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 Processor -- start 2022-07-21T01:38:24.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- start start 2022-07-21T01:38:24.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4040b5990 0x7ff404065a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4040abcb0 0x7ff40414e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff40414e7a0 0x7ff404067290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff404141c70 con 0x7ff4040abcb0 2022-07-21T01:38:24.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4040642e0 con 0x7ff4040b5990 2022-07-21T01:38:24.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4040b5d70 con 0x7ff40414e7a0 2022-07-21T01:38:24.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff401d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4040abcb0 0x7ff40414e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff401d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4040abcb0 0x7ff40414e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46154/0 (socket says 172.21.15.3:46154) 2022-07-21T01:38:24.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff401d9b700 1 -- 172.21.15.3:0/4096611177 learned_addr learned my addr 172.21.15.3:0/4096611177 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:24.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff402d9d700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff40414e7a0 0x7ff404067290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40259c700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4040b5990 0x7ff404065a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff401d9b700 1 -- 172.21.15.3:0/4096611177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff40414e7a0 msgr2=0x7ff404067290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff401d9b700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff40414e7a0 0x7ff404067290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff401d9b700 1 -- 172.21.15.3:0/4096611177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4040b5990 msgr2=0x7ff404065a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff401d9b700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4040b5990 0x7ff404065a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff401d9b700 1 -- 172.21.15.3:0/4096611177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4040ac120 con 0x7ff4040abcb0 2022-07-21T01:38:24.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff401d9b700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4040abcb0 0x7ff40414e2b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7ff3f4001110 tx=0x7ff3f4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff3f37fe700 1 -- 172.21.15.3:0/4096611177 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3f4007d10 con 0x7ff4040abcb0 2022-07-21T01:38:24.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff3f37fe700 1 -- 172.21.15.3:0/4096611177 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f4013850 con 0x7ff4040abcb0 2022-07-21T01:38:24.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4040ac380 con 0x7ff4040abcb0 2022-07-21T01:38:24.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff3f37fe700 1 -- 172.21.15.3:0/4096611177 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3f4006ea0 con 0x7ff4040abcb0 2022-07-21T01:38:24.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.845+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4040655f0 con 0x7ff4040abcb0 2022-07-21T01:38:24.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.849+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4040657f0 con 0x7ff4040abcb0 2022-07-21T01:38:24.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.849+0000 7ff3f37fe700 1 -- 172.21.15.3:0/4096611177 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff3f4010070 con 0x7ff4040abcb0 2022-07-21T01:38:24.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.849+0000 7ff3f37fe700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3e0053a70 0x7ff3e0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:24.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.849+0000 7ff3f37fe700 1 -- 172.21.15.3:0/4096611177 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5600+0+0 (secure 0 0 0) 0x7ff3f406cb40 con 0x7ff4040abcb0 2022-07-21T01:38:24.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.849+0000 7ff40259c700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3e0053a70 0x7ff3e0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:24.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.849+0000 7ff40259c700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3e0053a70 0x7ff3e0055f30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff3f800b000 tx=0x7ff3f8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:24.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.853+0000 7ff3f37fe700 1 -- 172.21.15.3:0/4096611177 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff4040657f0 con 0x7ff4040abcb0 2022-07-21T01:38:24.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.981+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff4040657f0 con 0x7ff4040abcb0 2022-07-21T01:38:24.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff3f37fe700 1 -- 172.21.15.3:0/4096611177 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) v1 ==== 84+0+143 (secure 0 0 0) 0x7ff4040657f0 con 0x7ff4040abcb0 2022-07-21T01:38:24.983 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:24.983 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:38:24.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:24.984 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:38:24.984 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:38:24.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:24.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:24.984 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:38:24.985 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:38:24.985 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:24.985 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:38:24.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3e0053a70 msgr2=0x7ff3e0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3e0053a70 0x7ff3e0055f30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff3f800b000 tx=0x7ff3f8008040).stop 2022-07-21T01:38:24.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4040abcb0 msgr2=0x7ff40414e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:24.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4040abcb0 0x7ff40414e2b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7ff3f4001110 tx=0x7ff3f4015000).stop 2022-07-21T01:38:24.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 shutdown_connections 2022-07-21T01:38:24.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff40414e7a0 0x7ff404067290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4040abcb0 0x7ff40414e2b0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3e0053a70 0x7ff3e0055f30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 --2- 172.21.15.3:0/4096611177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4040b5990 0x7ff404065a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:24.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 >> 172.21.15.3:0/4096611177 conn(0x7ff404061360 msgr2=0x7ff40405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:24.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 shutdown_connections 2022-07-21T01:38:24.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:24.985+0000 7ff40359e700 1 -- 172.21.15.3:0/4096611177 wait complete. 2022-07-21T01:38:24.996 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:38:24.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:38:25.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e7fff700 1 Processor -- start 2022-07-21T01:38:25.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e7fff700 1 -- start start 2022-07-21T01:38:25.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80619b0 0x7fa9e80b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa9e80b5ed0 0x7fa9e80b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e7fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e8061d90 con 0x7fa9e80b6ee0 2022-07-21T01:38:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e7fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e8143fa0 con 0x7fa9e80b5ed0 2022-07-21T01:38:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e8142f60 con 0x7fa9e80619b0 2022-07-21T01:38:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa9e80b5ed0 0x7fa9e80b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa9e80b5ed0 0x7fa9e80b5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46164/0 (socket says 172.21.15.3:46164) 2022-07-21T01:38:25.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e77fe700 1 -- 172.21.15.3:0/3483603964 learned_addr learned my addr 172.21.15.3:0/3483603964 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:25.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e6ffd700 1 --2- 172.21.15.3:0/3483603964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80619b0 0x7fa9e80b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.105+0000 7fa9e67fc700 1 --1- 172.21.15.3:0/3483603964 >> v1:172.21.15.143:6789/0 conn(0x7fa9e80b6ee0 0x7fa9e8143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:56558/0 (socket says 172.21.15.3:56558) 2022-07-21T01:38:25.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e77fe700 1 -- 172.21.15.3:0/3483603964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80619b0 msgr2=0x7fa9e80b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:25.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e77fe700 1 --2- 172.21.15.3:0/3483603964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80619b0 0x7fa9e80b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e77fe700 1 -- 172.21.15.3:0/3483603964 >> v1:172.21.15.143:6789/0 conn(0x7fa9e80b6ee0 legacy=0x7fa9e8143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e77fe700 1 -- 172.21.15.3:0/3483603964 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e80ba5a0 con 0x7fa9e80b5ed0 2022-07-21T01:38:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e77fe700 1 --2- 172.21.15.3:0/3483603964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80b5ed0 0x7fa9e80b5190 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fa9d8002f70 tx=0x7fa9d8006eb0).ready entity=mon.0 client_cookie=8c0452cf2fa23800 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e5ffb700 1 -- 172.21.15.3:0/3483603964 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9d800d570 con 0x7fa9e80b5ed0 2022-07-21T01:38:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e5ffb700 1 -- 172.21.15.3:0/3483603964 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d8000e90 con 0x7fa9e80b5ed0 2022-07-21T01:38:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e5ffb700 1 -- 172.21.15.3:0/3483603964 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9d800d900 con 0x7fa9e80b5ed0 2022-07-21T01:38:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e7fff700 1 -- 172.21.15.3:0/3483603964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80b5ed0 msgr2=0x7fa9e80b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e7fff700 1 --2- 172.21.15.3:0/3483603964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80b5ed0 0x7fa9e80b5190 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fa9d8002f70 tx=0x7fa9d8006eb0).stop 2022-07-21T01:38:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e7fff700 1 -- 172.21.15.3:0/3483603964 shutdown_connections 2022-07-21T01:38:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e7fff700 1 --2- 172.21.15.3:0/3483603964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80b5ed0 0x7fa9e80b5190 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e7fff700 1 --2- 172.21.15.3:0/3483603964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80619b0 0x7fa9e80b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e7fff700 1 -- 172.21.15.3:0/3483603964 >> 172.21.15.3:0/3483603964 conn(0x7fa9e806af80 msgr2=0x7fa9e805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e7fff700 1 -- 172.21.15.3:0/3483603964 shutdown_connections 2022-07-21T01:38:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.109+0000 7fa9e7fff700 1 -- 172.21.15.3:0/3483603964 wait complete. 2022-07-21T01:38:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e7fff700 1 Processor -- start 2022-07-21T01:38:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e7fff700 1 -- start start 2022-07-21T01:38:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80619b0 0x7fa9e80b8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80b5ed0 0x7fa9e8145810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9e80b6ee0 0x7fa9e8145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e7fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e8145420 con 0x7fa9e80619b0 2022-07-21T01:38:25.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e7fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80afdd0 con 0x7fa9e80b6ee0 2022-07-21T01:38:25.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80b0070 con 0x7fa9e80b5ed0 2022-07-21T01:38:25.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80619b0 0x7fa9e80b8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80619b0 0x7fa9e80b8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46166/0 (socket says 172.21.15.3:46166) 2022-07-21T01:38:25.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e6ffd700 1 -- 172.21.15.3:0/2757444383 learned_addr learned my addr 172.21.15.3:0/2757444383 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:25.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e67fc700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80b5ed0 0x7fa9e8145810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e77fe700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9e80b6ee0 0x7fa9e8145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e6ffd700 1 -- 172.21.15.3:0/2757444383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80b5ed0 msgr2=0x7fa9e8145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e6ffd700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80b5ed0 0x7fa9e8145810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e6ffd700 1 -- 172.21.15.3:0/2757444383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9e80b6ee0 msgr2=0x7fa9e8145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e6ffd700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9e80b6ee0 0x7fa9e8145d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e6ffd700 1 -- 172.21.15.3:0/2757444383 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d800b1e0 con 0x7fa9e80619b0 2022-07-21T01:38:25.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e77fe700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9e80b6ee0 0x7fa9e8145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:38:25.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e6ffd700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80619b0 0x7fa9e80b8690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7fa9e000c000 tx=0x7fa9e00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:25.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9d7fff700 1 -- 172.21.15.3:0/2757444383 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9e000c850 con 0x7fa9e80619b0 2022-07-21T01:38:25.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e80b0310 con 0x7fa9e80619b0 2022-07-21T01:38:25.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9d7fff700 1 -- 172.21.15.3:0/2757444383 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e000c9b0 con 0x7fa9e80619b0 2022-07-21T01:38:25.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.113+0000 7fa9d7fff700 1 -- 172.21.15.3:0/2757444383 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9e000cd10 con 0x7fa9e80619b0 2022-07-21T01:38:25.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.117+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e81134b0 con 0x7fa9e80619b0 2022-07-21T01:38:25.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.117+0000 7fa9d7fff700 1 -- 172.21.15.3:0/2757444383 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa9e0022070 con 0x7fa9e80619b0 2022-07-21T01:38:25.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.117+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e80b04a0 con 0x7fa9e80619b0 2022-07-21T01:38:25.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.117+0000 7fa9d7fff700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9c40539e0 0x7fa9c4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.117+0000 7fa9d7fff700 1 -- 172.21.15.3:0/2757444383 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa9e006c570 con 0x7fa9e80619b0 2022-07-21T01:38:25.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.117+0000 7fa9e67fc700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9c40539e0 0x7fa9c4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.117+0000 7fa9e67fc700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9c40539e0 0x7fa9c4055ea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fa9dc00b000 tx=0x7fa9dc007160).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:25.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.125+0000 7fa9d7fff700 1 -- 172.21.15.3:0/2757444383 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9e00435d0 con 0x7fa9e80619b0 2022-07-21T01:38:25.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.253+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fa9e80574e0 con 0x7fa9e80619b0 2022-07-21T01:38:25.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.765+0000 7fa9d7fff700 1 -- 172.21.15.3:0/2757444383 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v64) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa9e0046bf0 con 0x7fa9e80619b0 2022-07-21T01:38:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9c40539e0 msgr2=0x7fa9c4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9c40539e0 0x7fa9c4055ea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fa9dc00b000 tx=0x7fa9dc007160).stop 2022-07-21T01:38:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80619b0 msgr2=0x7fa9e80b8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80619b0 0x7fa9e80b8690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7fa9e000c000 tx=0x7fa9e00091d0).stop 2022-07-21T01:38:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 shutdown_connections 2022-07-21T01:38:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa9c40539e0 0x7fa9c4055ea0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9e80b6ee0 0x7fa9e8145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9e80b5ed0 0x7fa9e8145810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 --2- 172.21.15.3:0/2757444383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9e80619b0 0x7fa9e80b8690 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 >> 172.21.15.3:0/2757444383 conn(0x7fa9e806af80 msgr2=0x7fa9e8142600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:25.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 shutdown_connections 2022-07-21T01:38:25.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.769+0000 7fa9e7fff700 1 -- 172.21.15.3:0/2757444383 wait complete. 2022-07-21T01:38:25.776 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:38:25.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:38:25.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0506d4d700 1 Processor -- start 2022-07-21T01:38:25.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0506d4d700 1 -- start start 2022-07-21T01:38:25.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0506d4d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 0x7f05001403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0506d4d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0500143cf0 con 0x7f05000b4820 2022-07-21T01:38:25.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0506d4d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0500143f90 con 0x7f05000b4310 2022-07-21T01:38:25.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0506d4d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05001476a0 con 0x7f0500146ae0 2022-07-21T01:38:25.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f05052ea700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f05000b4820 0x7f05000b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54458/0 (socket says 172.21.15.3:54458) 2022-07-21T01:38:25.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f05052ea700 1 -- 172.21.15.3:0/2711429672 learned_addr learned my addr 172.21.15.3:0/2711429672 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:25.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0504ae9700 1 --2- 172.21.15.3:0/2711429672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 0x7f05001403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f04ff7fe700 1 -- 172.21.15.3:0/2711429672 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881213008 0 0) 0x7f0500143cf0 con 0x7f05000b4820 2022-07-21T01:38:25.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f04ff7fe700 1 -- 172.21.15.3:0/2711429672 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec003260 con 0x7f05000b4820 2022-07-21T01:38:25.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f04ff7fe700 1 -- 172.21.15.3:0/2711429672 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072984981 0 0) 0x7f0500143f90 con 0x7f05000b4310 2022-07-21T01:38:25.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f04ff7fe700 1 -- 172.21.15.3:0/2711429672 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0500143cf0 con 0x7f05000b4310 2022-07-21T01:38:25.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f04ff7fe700 1 -- 172.21.15.3:0/2711429672 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278545689 0 0) 0x7f04ec003260 con 0x7f05000b4820 2022-07-21T01:38:25.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0504ae9700 1 -- 172.21.15.3:0/2711429672 >> v1:172.21.15.143:6789/0 conn(0x7f05000b4310 legacy=0x7f05001408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0504ae9700 1 -- 172.21.15.3:0/2711429672 >> v1:172.21.15.3:6789/0 conn(0x7f05000b4820 legacy=0x7f05000b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0504ae9700 1 -- 172.21.15.3:0/2711429672 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05000b0e20 con 0x7f0500146ae0 2022-07-21T01:38:25.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.897+0000 7f0504ae9700 1 --2- 172.21.15.3:0/2711429672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 0x7f05001403c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f04f8001490 tx=0x7f04f0006df0).ready entity=mon.2 client_cookie=f865856a40a1e137 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:25.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f04ff7fe700 1 -- 172.21.15.3:0/2711429672 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04f80019e0 con 0x7f0500146ae0 2022-07-21T01:38:25.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f04ff7fe700 1 -- 172.21.15.3:0/2711429672 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f0000e90 con 0x7f0500146ae0 2022-07-21T01:38:25.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f04ff7fe700 1 -- 172.21.15.3:0/2711429672 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04f0012460 con 0x7f0500146ae0 2022-07-21T01:38:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 -- 172.21.15.3:0/2711429672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 msgr2=0x7f05001403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 --2- 172.21.15.3:0/2711429672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 0x7f05001403c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f04f8001490 tx=0x7f04f0006df0).stop 2022-07-21T01:38:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 -- 172.21.15.3:0/2711429672 shutdown_connections 2022-07-21T01:38:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 --2- 172.21.15.3:0/2711429672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 0x7f05001403c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 -- 172.21.15.3:0/2711429672 >> 172.21.15.3:0/2711429672 conn(0x7f050005c620 msgr2=0x7f050005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 -- 172.21.15.3:0/2711429672 shutdown_connections 2022-07-21T01:38:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 -- 172.21.15.3:0/2711429672 wait complete. 2022-07-21T01:38:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 Processor -- start 2022-07-21T01:38:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 -- start start 2022-07-21T01:38:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.901+0000 7f0506d4d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05000b4310 0x7f05000b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0506d4d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05000b4820 0x7f0500142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0506d4d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 0x7f05000ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0506d4d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05000b07c0 con 0x7f05000b4310 2022-07-21T01:38:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0506d4d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0500142860 con 0x7f05000b4820 2022-07-21T01:38:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0506d4d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0500142320 con 0x7f0500146ae0 2022-07-21T01:38:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0504ae9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05000b4310 0x7f05000b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0504ae9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05000b4310 0x7f05000b8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46180/0 (socket says 172.21.15.3:46180) 2022-07-21T01:38:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0504ae9700 1 -- 172.21.15.3:0/1356127363 learned_addr learned my addr 172.21.15.3:0/1356127363 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f04fffff700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05000b4820 0x7f0500142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f05052ea700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 0x7f05000ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0504ae9700 1 -- 172.21.15.3:0/1356127363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 msgr2=0x7f05000ada50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0504ae9700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 0x7f05000ada50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0504ae9700 1 -- 172.21.15.3:0/1356127363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05000b4820 msgr2=0x7f0500142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0504ae9700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05000b4820 0x7f0500142cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:25.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0504ae9700 1 -- 172.21.15.3:0/1356127363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f000b180 con 0x7f05000b4310 2022-07-21T01:38:25.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0504ae9700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05000b4310 0x7f05000b8a10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f05000ad7b0 tx=0x7f04f0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:25.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f04fdffb700 1 -- 172.21.15.3:0/1356127363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04f000d9a0 con 0x7f05000b4310 2022-07-21T01:38:25.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f04fdffb700 1 -- 172.21.15.3:0/1356127363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f000db00 con 0x7f05000b4310 2022-07-21T01:38:25.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f04fdffb700 1 -- 172.21.15.3:0/1356127363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04f000de60 con 0x7f05000b4310 2022-07-21T01:38:25.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05001425c0 con 0x7f05000b4310 2022-07-21T01:38:25.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05000ae5d0 con 0x7f05000b4310 2022-07-21T01:38:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.905+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05000b1300 con 0x7f05000b4310 2022-07-21T01:38:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.909+0000 7f04fdffb700 1 -- 172.21.15.3:0/1356127363 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f04f001c070 con 0x7f05000b4310 2022-07-21T01:38:25.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.909+0000 7f04fdffb700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f04e0053a10 0x7f04e0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:25.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.909+0000 7f04fdffb700 1 -- 172.21.15.3:0/1356127363 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f04f006c7d0 con 0x7f05000b4310 2022-07-21T01:38:25.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.909+0000 7f04fffff700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f04e0053a10 0x7f04e0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:25.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.909+0000 7f04fffff700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f04e0053a10 0x7f04e0055ed0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f04f400d000 tx=0x7f04f4007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:25.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:25.917+0000 7f04fdffb700 1 -- 172.21.15.3:0/1356127363 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f04f0043760 con 0x7f05000b4310 2022-07-21T01:38:26.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.041+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f05000ad7f0 con 0x7f05000b4310 2022-07-21T01:38:26.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.485+0000 7f04fdffb700 1 -- 172.21.15.3:0/1356127363 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v65) v1 ==== 125+0+0 (secure 0 0 0) 0x7f04f00230a0 con 0x7f05000b4310 2022-07-21T01:38:26.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f04e0053a10 msgr2=0x7f04e0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f04e0053a10 0x7f04e0055ed0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f04f400d000 tx=0x7f04f4007a20).stop 2022-07-21T01:38:26.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05000b4310 msgr2=0x7f05000b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05000b4310 0x7f05000b8a10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f05000ad7b0 tx=0x7f04f0014000).stop 2022-07-21T01:38:26.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 shutdown_connections 2022-07-21T01:38:26.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0500146ae0 0x7f05000ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f04e0053a10 0x7f04e0055ed0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05000b4820 0x7f0500142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 --2- 172.21.15.3:0/1356127363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05000b4310 0x7f05000b8a10 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 >> 172.21.15.3:0/1356127363 conn(0x7f050005c620 msgr2=0x7f05000b6140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:26.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 shutdown_connections 2022-07-21T01:38:26.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.489+0000 7f0506d4d700 1 -- 172.21.15.3:0/1356127363 wait complete. 2022-07-21T01:38:26.495 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:38:26.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:38:26.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.617+0000 7fd2dbfff700 1 Processor -- start 2022-07-21T01:38:26.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.617+0000 7fd2dbfff700 1 -- start start 2022-07-21T01:38:26.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.617+0000 7fd2dbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2dc066780 0x7fd2dc066b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.617+0000 7fd2dbfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2dc05f8e0 con 0x7fd2dc140e40 2022-07-21T01:38:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2dbfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2dc0b1630 con 0x7fd2dc0b1bc0 2022-07-21T01:38:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2dbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc05fb10 con 0x7fd2dc066780 2022-07-21T01:38:26.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2db7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd2dc140e40 0x7fd2dc13e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54472/0 (socket says 172.21.15.3:54472) 2022-07-21T01:38:26.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2db7fe700 1 -- 172.21.15.3:0/2659044039 learned_addr learned my addr 172.21.15.3:0/2659044039 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:26.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2daffd700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2dc066780 0x7fd2dc066b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675481455 0 0) 0x7fd2dc05f8e0 con 0x7fd2dc140e40 2022-07-21T01:38:26.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2bc003260 con 0x7fd2dc140e40 2022-07-21T01:38:26.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3101975830 0 0) 0x7fd2bc003260 con 0x7fd2dc140e40 2022-07-21T01:38:26.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2dc066780 msgr2=0x7fd2dc066b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:26.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2dc066780 0x7fd2dc066b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 >> v1:172.21.15.143:6789/0 conn(0x7fd2dc0b1bc0 legacy=0x7fd2dc05ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc0b18d0 con 0x7fd2dc140e40 2022-07-21T01:38:26.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fd2d4001660 con 0x7fd2dc140e40 2022-07-21T01:38:26.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 >> v1:172.21.15.3:6789/0 conn(0x7fd2dc140e40 legacy=0x7fd2dc13e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2bc003780 0x7fd2bc003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2bc004500 0x7fd2bc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2dc066780 0x7fd2bc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc05fb10 con 0x7fd2bc004500 2022-07-21T01:38:26.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00b750 con 0x7fd2bc003780 2022-07-21T01:38:26.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc00b8c0 con 0x7fd2dc066780 2022-07-21T01:38:26.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2da7fc700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2bc004500 0x7fd2bc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2daffd700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2bc003780 0x7fd2bc003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2db7fe700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2dc066780 0x7fd2bc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2da7fc700 1 -- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2dc066780 msgr2=0x7fd2bc004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:26.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2da7fc700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2dc066780 0x7fd2bc004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2da7fc700 1 -- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2bc003780 msgr2=0x7fd2bc003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:26.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2da7fc700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2bc003780 0x7fd2bc003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2da7fc700 1 -- 172.21.15.3:0/2659044039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc003260 con 0x7fd2bc004500 2022-07-21T01:38:26.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2da7fc700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2bc004500 0x7fd2bc004980 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7fd2c40013a0 tx=0x7fd2c4007d60).ready entity=mon.0 client_cookie=f5c9036a68de4a89 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:26.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2c40016c0 con 0x7fd2bc004500 2022-07-21T01:38:26.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c4006890 con 0x7fd2bc004500 2022-07-21T01:38:26.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2d9ffb700 1 -- 172.21.15.3:0/2659044039 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2c4006bf0 con 0x7fd2bc004500 2022-07-21T01:38:26.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2dbfff700 1 -- 172.21.15.3:0/2659044039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2bc004500 msgr2=0x7fd2bc004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2dbfff700 1 --2- 172.21.15.3:0/2659044039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2bc004500 0x7fd2bc004980 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7fd2c40013a0 tx=0x7fd2c4007d60).stop 2022-07-21T01:38:26.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2db7fe700 1 -- 172.21.15.3:0/2659044039 reap_dead start 2022-07-21T01:38:26.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2dbfff700 1 -- 172.21.15.3:0/2659044039 shutdown_connections 2022-07-21T01:38:26.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2dbfff700 1 -- 172.21.15.3:0/2659044039 >> 172.21.15.3:0/2659044039 conn(0x7fd2dc063870 msgr2=0x7fd2dc063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:26.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2dbfff700 1 -- 172.21.15.3:0/2659044039 shutdown_connections 2022-07-21T01:38:26.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.621+0000 7fd2dbfff700 1 -- 172.21.15.3:0/2659044039 wait complete. 2022-07-21T01:38:26.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 Processor -- start 2022-07-21T01:38:26.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 -- start start 2022-07-21T01:38:26.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc004500 0x7fd2dc0660d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2dc142200 0x7fd2dc13c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2dc0b00e0 0x7fd2dc155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc142640 con 0x7fd2dc142200 2022-07-21T01:38:26.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc0ae960 con 0x7fd2dc0b00e0 2022-07-21T01:38:26.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2dc064ff0 con 0x7fd2bc004500 2022-07-21T01:38:26.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2da7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2dc142200 0x7fd2dc13c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2da7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2dc142200 0x7fd2dc13c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46202/0 (socket says 172.21.15.3:46202) 2022-07-21T01:38:26.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2da7fc700 1 -- 172.21.15.3:0/661783281 learned_addr learned my addr 172.21.15.3:0/661783281 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:26.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2daffd700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc004500 0x7fd2dc0660d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2db7fe700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2dc0b00e0 0x7fd2dc155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2da7fc700 1 -- 172.21.15.3:0/661783281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc004500 msgr2=0x7fd2dc0660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2da7fc700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc004500 0x7fd2dc0660d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2da7fc700 1 -- 172.21.15.3:0/661783281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2dc0b00e0 msgr2=0x7fd2dc155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2da7fc700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2dc0b00e0 0x7fd2dc155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2da7fc700 1 -- 172.21.15.3:0/661783281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc0af190 con 0x7fd2dc142200 2022-07-21T01:38:26.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2da7fc700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2dc142200 0x7fd2dc13c810 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7fd2dc0b60c0 tx=0x7fd2c4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:26.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2d3fff700 1 -- 172.21.15.3:0/661783281 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2c40016c0 con 0x7fd2dc142200 2022-07-21T01:38:26.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2d3fff700 1 -- 172.21.15.3:0/661783281 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c400c930 con 0x7fd2dc142200 2022-07-21T01:38:26.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2d3fff700 1 -- 172.21.15.3:0/661783281 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2c400cc90 con 0x7fd2dc142200 2022-07-21T01:38:26.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2dc0aeb90 con 0x7fd2dc142200 2022-07-21T01:38:26.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2dc13cd00 con 0x7fd2dc142200 2022-07-21T01:38:26.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.625+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2dc0b0700 con 0x7fd2dc142200 2022-07-21T01:38:26.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.629+0000 7fd2d3fff700 1 -- 172.21.15.3:0/661783281 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd2c401d070 con 0x7fd2dc142200 2022-07-21T01:38:26.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.629+0000 7fd2d3fff700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c0053b30 0x7fd2c0055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.629+0000 7fd2d3fff700 1 -- 172.21.15.3:0/661783281 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6304+0+0 (secure 0 0 0) 0x7fd2c406cff0 con 0x7fd2dc142200 2022-07-21T01:38:26.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.629+0000 7fd2daffd700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c0053b30 0x7fd2c0055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.629+0000 7fd2daffd700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c0053b30 0x7fd2c0055ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fd2cc00b000 tx=0x7fd2cc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:26.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.637+0000 7fd2d3fff700 1 -- 172.21.15.3:0/661783281 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd2c4043d20 con 0x7fd2dc142200 2022-07-21T01:38:26.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.805+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fd2dc1412f0 con 0x7fd2dc142200 2022-07-21T01:38:26.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.817+0000 7fd2d3fff700 1 -- 172.21.15.3:0/661783281 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 11 and data pool 12 v28) v1 ==== 189+0+0 (secure 0 0 0) 0x7fd2c401a070 con 0x7fd2dc142200 2022-07-21T01:38:26.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.817+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c0053b30 msgr2=0x7fd2c0055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.817+0000 7fd2dbfff700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c0053b30 0x7fd2c0055ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fd2cc00b000 tx=0x7fd2cc006df0).stop 2022-07-21T01:38:26.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.817+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2dc142200 msgr2=0x7fd2dc13c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.817+0000 7fd2dbfff700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2dc142200 0x7fd2dc13c810 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7fd2dc0b60c0 tx=0x7fd2c4015000).stop 2022-07-21T01:38:26.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.821+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 shutdown_connections 2022-07-21T01:38:26.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.821+0000 7fd2dbfff700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c0053b30 0x7fd2c0055ff0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.821+0000 7fd2dbfff700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2dc0b00e0 0x7fd2dc155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.821+0000 7fd2dbfff700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2dc142200 0x7fd2dc13c810 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.821+0000 7fd2dbfff700 1 --2- 172.21.15.3:0/661783281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc004500 0x7fd2dc0660d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.821+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 >> 172.21.15.3:0/661783281 conn(0x7fd2dc063870 msgr2=0x7fd2dc0b2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:26.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.821+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 shutdown_connections 2022-07-21T01:38:26.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.821+0000 7fd2dbfff700 1 -- 172.21.15.3:0/661783281 wait complete. 2022-07-21T01:38:26.821 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 11 and data pool 12 2022-07-21T01:38:26.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:38:26.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88efd28700 1 Processor -- start 2022-07-21T01:38:26.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88efd28700 1 -- start start 2022-07-21T01:38:26.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88efd28700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e80617e0 0x7f88e8066760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88efd28700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88e8066c50 0x7f88e814db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88efd28700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e8060670 con 0x7f88e8060290 2022-07-21T01:38:26.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88efd28700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e8060e00 con 0x7f88e8066c50 2022-07-21T01:38:26.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88efd28700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80610a0 con 0x7f88e80617e0 2022-07-21T01:38:26.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88ed2c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e80617e0 0x7f88e8066760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88edac4700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f88e8060290 0x7f88e8065b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:56606/0 (socket says 172.21.15.3:56606) 2022-07-21T01:38:26.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88edac4700 1 -- 172.21.15.3:0/2995146793 learned_addr learned my addr 172.21.15.3:0/2995146793 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:26.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88ee2c5700 1 --2- 172.21.15.3:0/2995146793 >> v2:172.21.15.143:3300/0 conn(0x7f88e8066c50 0x7f88e814db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88ed2c3700 1 -- 172.21.15.3:0/2995146793 >> v2:172.21.15.143:3300/0 conn(0x7f88e8066c50 msgr2=0x7f88e814db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:26.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88ed2c3700 1 --2- 172.21.15.3:0/2995146793 >> v2:172.21.15.143:3300/0 conn(0x7f88e8066c50 0x7f88e814db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88ed2c3700 1 -- 172.21.15.3:0/2995146793 >> v1:172.21.15.143:6789/0 conn(0x7f88e8060290 legacy=0x7f88e8065b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.925+0000 7f88ed2c3700 1 -- 172.21.15.3:0/2995146793 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e8061340 con 0x7f88e80617e0 2022-07-21T01:38:26.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88ed2c3700 1 --2- 172.21.15.3:0/2995146793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e80617e0 0x7f88e8066760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f88e4002f70 tx=0x7f88e4006df0).ready entity=mon.2 client_cookie=a9c5ed3edba08c8c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:26.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88ecac2700 1 -- 172.21.15.3:0/2995146793 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88e400d570 con 0x7f88e80617e0 2022-07-21T01:38:26.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88ecac2700 1 -- 172.21.15.3:0/2995146793 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e4000e90 con 0x7f88e80617e0 2022-07-21T01:38:26.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88ecac2700 1 -- 172.21.15.3:0/2995146793 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88e400d8c0 con 0x7f88e80617e0 2022-07-21T01:38:26.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 -- 172.21.15.3:0/2995146793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e80617e0 msgr2=0x7f88e8066760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 --2- 172.21.15.3:0/2995146793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e80617e0 0x7f88e8066760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f88e4002f70 tx=0x7f88e4006df0).stop 2022-07-21T01:38:26.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 -- 172.21.15.3:0/2995146793 shutdown_connections 2022-07-21T01:38:26.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 --2- 172.21.15.3:0/2995146793 >> v2:172.21.15.143:3300/0 conn(0x7f88e8066c50 0x7f88e814db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 --2- 172.21.15.3:0/2995146793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e80617e0 0x7f88e8066760 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 -- 172.21.15.3:0/2995146793 >> 172.21.15.3:0/2995146793 conn(0x7f88e805d0c0 msgr2=0x7f88e805f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:26.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 -- 172.21.15.3:0/2995146793 shutdown_connections 2022-07-21T01:38:26.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 -- 172.21.15.3:0/2995146793 wait complete. 2022-07-21T01:38:26.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 Processor -- start 2022-07-21T01:38:26.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.929+0000 7f88efd28700 1 -- start start 2022-07-21T01:38:26.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88efd28700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e8060290 0x7f88e80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88efd28700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e80617e0 0x7f88e8146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88efd28700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e8066c50 0x7f88e8156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88efd28700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80a6000 con 0x7f88e8066c50 2022-07-21T01:38:26.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88efd28700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80a4250 con 0x7f88e80617e0 2022-07-21T01:38:26.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88efd28700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80a5d60 con 0x7f88e8060290 2022-07-21T01:38:26.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ee2c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e8066c50 0x7f88e8156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ee2c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e8066c50 0x7f88e8156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46220/0 (socket says 172.21.15.3:46220) 2022-07-21T01:38:26.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ee2c5700 1 -- 172.21.15.3:0/455493437 learned_addr learned my addr 172.21.15.3:0/455493437 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:26.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88edac4700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e8060290 0x7f88e80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ed2c3700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e80617e0 0x7f88e8146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ee2c5700 1 -- 172.21.15.3:0/455493437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e8060290 msgr2=0x7f88e80a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ee2c5700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e8060290 0x7f88e80a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ee2c5700 1 -- 172.21.15.3:0/455493437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e80617e0 msgr2=0x7f88e8146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:26.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ee2c5700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e80617e0 0x7f88e8146c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:26.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ee2c5700 1 -- 172.21.15.3:0/455493437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e400b180 con 0x7f88e8066c50 2022-07-21T01:38:26.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88ee2c5700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e8066c50 0x7f88e8156110 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f88e0006e00 tx=0x7f88e00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:26.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88d6ffd700 1 -- 172.21.15.3:0/455493437 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88e0012600 con 0x7f88e8066c50 2022-07-21T01:38:26.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88d6ffd700 1 -- 172.21.15.3:0/455493437 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e0012760 con 0x7f88e8066c50 2022-07-21T01:38:26.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e80a44b0 con 0x7f88e8066c50 2022-07-21T01:38:26.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88d6ffd700 1 -- 172.21.15.3:0/455493437 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88e0012ac0 con 0x7f88e8066c50 2022-07-21T01:38:26.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.933+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e80a5930 con 0x7f88e8066c50 2022-07-21T01:38:26.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.937+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e80a4640 con 0x7f88e8066c50 2022-07-21T01:38:26.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.937+0000 7f88d6ffd700 1 -- 172.21.15.3:0/455493437 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f88e0022070 con 0x7f88e8066c50 2022-07-21T01:38:26.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.937+0000 7f88d6ffd700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053b30 0x7f88c8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:26.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.937+0000 7f88d6ffd700 1 -- 172.21.15.3:0/455493437 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f88e006c570 con 0x7f88e8066c50 2022-07-21T01:38:26.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.937+0000 7f88edac4700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053b30 0x7f88c8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:26.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.937+0000 7f88edac4700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053b30 0x7f88c8055ff0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f88d800d000 tx=0x7f88d8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:26.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:26.945+0000 7f88d6ffd700 1 -- 172.21.15.3:0/455493437 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f88e0043290 con 0x7f88e8066c50 2022-07-21T01:38:27.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.073+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88e80574e0 con 0x7f88e8066c50 2022-07-21T01:38:27.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88d6ffd700 1 -- 172.21.15.3:0/455493437 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16287 (secure 0 0 0) 0x7f88e0016090 con 0x7f88e8066c50 2022-07-21T01:38:27.074 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:27.075 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":67,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:38:26.807803+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:38:25.308080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-21T01:38:26.095494+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:38:27.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053b30 msgr2=0x7f88c8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053b30 0x7f88c8055ff0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f88d800d000 tx=0x7f88d8009040).stop 2022-07-21T01:38:27.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e8066c50 msgr2=0x7f88e8156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e8066c50 0x7f88e8156110 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f88e0006e00 tx=0x7f88e00091d0).stop 2022-07-21T01:38:27.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 shutdown_connections 2022-07-21T01:38:27.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e8066c50 0x7f88e8156110 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053b30 0x7f88c8055ff0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e80617e0 0x7f88e8146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 --2- 172.21.15.3:0/455493437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e8060290 0x7f88e80a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 >> 172.21.15.3:0/455493437 conn(0x7f88e805d0c0 msgr2=0x7f88e80b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:27.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 shutdown_connections 2022-07-21T01:38:27.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.077+0000 7f88efd28700 1 -- 172.21.15.3:0/455493437 wait complete. 2022-07-21T01:38:27.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:38:27.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f8984fdd700 1 Processor -- start 2022-07-21T01:38:27.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f8984fdd700 1 -- start start 2022-07-21T01:38:27.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f8984fdd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89800b2710 0x7f898014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f8984fdd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89800a7250 con 0x7f89800b01a0 2022-07-21T01:38:27.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f8984fdd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89800a7480 con 0x7f89800a8c50 2022-07-21T01:38:27.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f8984fdd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89800a7720 con 0x7f89800b2710 2022-07-21T01:38:27.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f897f7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89800b2710 0x7f898014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f897effd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f89800b01a0 0x7f89800a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54504/0 (socket says 172.21.15.3:54504) 2022-07-21T01:38:27.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f897effd700 1 -- 172.21.15.3:0/134244541 learned_addr learned my addr 172.21.15.3:0/134244541 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:27.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f897e7fc700 1 -- 172.21.15.3:0/134244541 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435330884 0 0) 0x7f89800a7250 con 0x7f89800b01a0 2022-07-21T01:38:27.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.193+0000 7f897e7fc700 1 -- 172.21.15.3:0/134244541 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8960003260 con 0x7f89800b01a0 2022-07-21T01:38:27.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897e7fc700 1 -- 172.21.15.3:0/134244541 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356526080 0 0) 0x7f89800a7480 con 0x7f89800a8c50 2022-07-21T01:38:27.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897e7fc700 1 -- 172.21.15.3:0/134244541 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89800a7250 con 0x7f89800a8c50 2022-07-21T01:38:27.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897f7fe700 1 -- 172.21.15.3:0/134244541 >> v1:172.21.15.143:6789/0 conn(0x7f89800a8c50 legacy=0x7f8980151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897e7fc700 1 -- 172.21.15.3:0/134244541 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2444128281 0 0) 0x7f8960003260 con 0x7f89800b01a0 2022-07-21T01:38:27.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897f7fe700 1 -- 172.21.15.3:0/134244541 >> v1:172.21.15.3:6789/0 conn(0x7f89800b01a0 legacy=0x7f89800a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897f7fe700 1 -- 172.21.15.3:0/134244541 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89800a79c0 con 0x7f89800b2710 2022-07-21T01:38:27.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897f7fe700 1 --2- 172.21.15.3:0/134244541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b2710 0x7f898014afd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f8968001800 tx=0x7f8970006df0).ready entity=mon.0 client_cookie=9bcc9e87be47d6ab server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:27.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897e7fc700 1 -- 172.21.15.3:0/134244541 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8968001660 con 0x7f89800b2710 2022-07-21T01:38:27.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897e7fc700 1 -- 172.21.15.3:0/134244541 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8970000e90 con 0x7f89800b2710 2022-07-21T01:38:27.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f897e7fc700 1 -- 172.21.15.3:0/134244541 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8970012460 con 0x7f89800b2710 2022-07-21T01:38:27.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f8984fdd700 1 -- 172.21.15.3:0/134244541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b2710 msgr2=0x7f898014afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f8984fdd700 1 --2- 172.21.15.3:0/134244541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b2710 0x7f898014afd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f8968001800 tx=0x7f8970006df0).stop 2022-07-21T01:38:27.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f8984fdd700 1 -- 172.21.15.3:0/134244541 shutdown_connections 2022-07-21T01:38:27.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f8984fdd700 1 --2- 172.21.15.3:0/134244541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b2710 0x7f898014afd0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f8984fdd700 1 -- 172.21.15.3:0/134244541 >> 172.21.15.3:0/134244541 conn(0x7f8980060310 msgr2=0x7f89800b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:27.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f8984fdd700 1 -- 172.21.15.3:0/134244541 shutdown_connections 2022-07-21T01:38:27.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f8984fdd700 1 -- 172.21.15.3:0/134244541 wait complete. 2022-07-21T01:38:27.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f8984fdd700 1 Processor -- start 2022-07-21T01:38:27.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.197+0000 7f8984fdd700 1 -- start start 2022-07-21T01:38:27.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f8984fdd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89800a8c50 0x7f8980148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f8984fdd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b01a0 0x7f8980156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f8984fdd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8980065bb0 0x7f898011d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f8984fdd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898011d0d0 con 0x7f89800b01a0 2022-07-21T01:38:27.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f8984fdd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8980156e30 con 0x7f89800a8c50 2022-07-21T01:38:27.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f8984fdd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8980063ae0 con 0x7f8980065bb0 2022-07-21T01:38:27.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b01a0 0x7f8980156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b01a0 0x7f8980156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46232/0 (socket says 172.21.15.3:46232) 2022-07-21T01:38:27.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897effd700 1 -- 172.21.15.3:0/1787479846 learned_addr learned my addr 172.21.15.3:0/1787479846 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:27.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897f7fe700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89800a8c50 0x7f8980148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897ffff700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8980065bb0 0x7f898011d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897effd700 1 -- 172.21.15.3:0/1787479846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8980065bb0 msgr2=0x7f898011d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897effd700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8980065bb0 0x7f898011d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897effd700 1 -- 172.21.15.3:0/1787479846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89800a8c50 msgr2=0x7f8980148b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897effd700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89800a8c50 0x7f8980148b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897effd700 1 -- 172.21.15.3:0/1787479846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897000b180 con 0x7f89800b01a0 2022-07-21T01:38:27.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897effd700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b01a0 0x7f8980156940 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f8968000fe0 tx=0x7f8968001720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:27.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897cff9700 1 -- 172.21.15.3:0/1787479846 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89680070d0 con 0x7f89800b01a0 2022-07-21T01:38:27.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897cff9700 1 -- 172.21.15.3:0/1787479846 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8968007230 con 0x7f89800b01a0 2022-07-21T01:38:27.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8980063d80 con 0x7f89800b01a0 2022-07-21T01:38:27.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f897cff9700 1 -- 172.21.15.3:0/1787479846 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8968005610 con 0x7f89800b01a0 2022-07-21T01:38:27.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89800636f0 con 0x7f89800b01a0 2022-07-21T01:38:27.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.201+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89800574e0 con 0x7f89800b01a0 2022-07-21T01:38:27.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.205+0000 7f897cff9700 1 -- 172.21.15.3:0/1787479846 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f896801a060 con 0x7f89800b01a0 2022-07-21T01:38:27.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.205+0000 7f897cff9700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8964053a70 0x7f8964055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.205+0000 7f897cff9700 1 -- 172.21.15.3:0/1787479846 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f896806b930 con 0x7f89800b01a0 2022-07-21T01:38:27.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.205+0000 7f897f7fe700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8964053a70 0x7f8964055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.205+0000 7f897f7fe700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8964053a70 0x7f8964055f30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f897001a000 tx=0x7f8970014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:27.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.209+0000 7f897cff9700 1 -- 172.21.15.3:0/1787479846 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f89680425e0 con 0x7f89800b01a0 2022-07-21T01:38:27.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.337+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f898014a800 con 0x7f89800b01a0 2022-07-21T01:38:27.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.337+0000 7f897cff9700 1 -- 172.21.15.3:0/1787479846 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16287 (secure 0 0 0) 0x7f8968018070 con 0x7f89800b01a0 2022-07-21T01:38:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:27.339 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":67,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:38:26.807803+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:38:25.308080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-21T01:38:26.095494+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:38:27.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8964053a70 msgr2=0x7f8964055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8964053a70 0x7f8964055f30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f897001a000 tx=0x7f8970014000).stop 2022-07-21T01:38:27.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b01a0 msgr2=0x7f8980156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b01a0 0x7f8980156940 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f8968000fe0 tx=0x7f8968001720).stop 2022-07-21T01:38:27.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 shutdown_connections 2022-07-21T01:38:27.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8980065bb0 0x7f898011d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89800b01a0 0x7f8980156940 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8964053a70 0x7f8964055f30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 --2- 172.21.15.3:0/1787479846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89800a8c50 0x7f8980148b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 >> 172.21.15.3:0/1787479846 conn(0x7f8980060310 msgr2=0x7f89800af3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:27.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 shutdown_connections 2022-07-21T01:38:27.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.341+0000 7f8984fdd700 1 -- 172.21.15.3:0/1787479846 wait complete. 2022-07-21T01:38:27.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:38:27.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.461+0000 7fe6fec13700 1 Processor -- start 2022-07-21T01:38:27.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.461+0000 7fe6fec13700 1 -- start start 2022-07-21T01:38:27.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe6f80b6ee0 0x7fe6f80b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80b5ed0 0x7fe6f80b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f8061d90 con 0x7fe6f80619b0 2022-07-21T01:38:27.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f8143fa0 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8142f60 con 0x7fe6f80b5ed0 2022-07-21T01:38:27.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fd410700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe6f80b6ee0 0x7fe6f80b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fd410700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe6f80b6ee0 0x7fe6f80b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46240/0 (socket says 172.21.15.3:46240) 2022-07-21T01:38:27.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fd410700 1 -- 172.21.15.3:0/1265661214 learned_addr learned my addr 172.21.15.3:0/1265661214 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:27.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fe412700 1 --2- 172.21.15.3:0/1265661214 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80b5ed0 0x7fe6f80b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fd410700 1 -- 172.21.15.3:0/1265661214 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80b5ed0 msgr2=0x7fe6f80b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:27.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fd410700 1 --2- 172.21.15.3:0/1265661214 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80b5ed0 0x7fe6f80b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fd410700 1 -- 172.21.15.3:0/1265661214 >> v1:172.21.15.143:6789/0 conn(0x7fe6f80619b0 legacy=0x7fe6f8143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fd410700 1 -- 172.21.15.3:0/1265661214 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f80ba5a0 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fd410700 1 --2- 172.21.15.3:0/1265661214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 0x7fe6f80b48a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7fe6f80b6880 tx=0x7fe6f4009290).ready entity=mon.0 client_cookie=f8ac043de7e8c522 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:27.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fcc0f700 1 -- 172.21.15.3:0/1265661214 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f400b860 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fcc0f700 1 -- 172.21.15.3:0/1265661214 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f400b9c0 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fcc0f700 1 -- 172.21.15.3:0/1265661214 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f400bb30 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 -- 172.21.15.3:0/1265661214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 msgr2=0x7fe6f80b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 --2- 172.21.15.3:0/1265661214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 0x7fe6f80b48a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7fe6f80b6880 tx=0x7fe6f4009290).stop 2022-07-21T01:38:27.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 -- 172.21.15.3:0/1265661214 shutdown_connections 2022-07-21T01:38:27.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 --2- 172.21.15.3:0/1265661214 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80b5ed0 0x7fe6f80b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 --2- 172.21.15.3:0/1265661214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 0x7fe6f80b48a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 -- 172.21.15.3:0/1265661214 >> 172.21.15.3:0/1265661214 conn(0x7fe6f806af80 msgr2=0x7fe6f805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:27.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 -- 172.21.15.3:0/1265661214 shutdown_connections 2022-07-21T01:38:27.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.465+0000 7fe6fec13700 1 -- 172.21.15.3:0/1265661214 wait complete. 2022-07-21T01:38:27.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fec13700 1 Processor -- start 2022-07-21T01:38:27.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fec13700 1 -- start start 2022-07-21T01:38:27.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fec13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80619b0 0x7fe6f81465b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fec13700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6f80b5ed0 0x7fe6f8114be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fec13700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 0x7fe6f81150d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fec13700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8113590 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fec13700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80aefa0 con 0x7fe6f80b5ed0 2022-07-21T01:38:27.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fec13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80af240 con 0x7fe6f80619b0 2022-07-21T01:38:27.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fe412700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 0x7fe6f81150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fe412700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 0x7fe6f81150d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46248/0 (socket says 172.21.15.3:46248) 2022-07-21T01:38:27.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fe412700 1 -- 172.21.15.3:0/1617359392 learned_addr learned my addr 172.21.15.3:0/1617359392 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:27.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fdc11700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80619b0 0x7fe6f81465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.469+0000 7fe6fd410700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6f80b5ed0 0x7fe6f8114be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fe412700 1 -- 172.21.15.3:0/1617359392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80619b0 msgr2=0x7fe6f81465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fe412700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80619b0 0x7fe6f81465b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fe412700 1 -- 172.21.15.3:0/1617359392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6f80b5ed0 msgr2=0x7fe6f8114be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fe412700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6f80b5ed0 0x7fe6f8114be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fe412700 1 -- 172.21.15.3:0/1617359392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4009530 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fe412700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 0x7fe6f81150d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fe6f000c000 tx=0x7fe6f0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:27.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6eeffd700 1 -- 172.21.15.3:0/1617359392 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f0011750 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6eeffd700 1 -- 172.21.15.3:0/1617359392 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f00118b0 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f80af4e0 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6eeffd700 1 -- 172.21.15.3:0/1617359392 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f0011c10 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f8147010 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f80574e0 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6eeffd700 1 -- 172.21.15.3:0/1617359392 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe6f0011d70 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6eeffd700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6d8053b00 0x7fe6d8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6eeffd700 1 -- 172.21.15.3:0/1617359392 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe6f006c460 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.473+0000 7fe6fdc11700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6d8053b00 0x7fe6d8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.477+0000 7fe6fdc11700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6d8053b00 0x7fe6d8055fc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fe6e800d000 tx=0x7fe6e8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:27.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.481+0000 7fe6eeffd700 1 -- 172.21.15.3:0/1617359392 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe6f0043180 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.645+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe6f805db60 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.817+0000 7fe6eeffd700 1 -- 172.21.15.3:0/1617359392 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe6f00467a0 con 0x7fe6f80b6ee0 2022-07-21T01:38:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6d8053b00 msgr2=0x7fe6d8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6d8053b00 0x7fe6d8055fc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fe6e800d000 tx=0x7fe6e8000bc0).stop 2022-07-21T01:38:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 msgr2=0x7fe6f81150d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 0x7fe6f81150d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fe6f000c000 tx=0x7fe6f0006df0).stop 2022-07-21T01:38:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 shutdown_connections 2022-07-21T01:38:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6f80b6ee0 0x7fe6f81150d0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6d8053b00 0x7fe6d8055fc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6f80b5ed0 0x7fe6f8114be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 --2- 172.21.15.3:0/1617359392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6f80619b0 0x7fe6f81465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 >> 172.21.15.3:0/1617359392 conn(0x7fe6f806af80 msgr2=0x7fe6f813f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:27.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 shutdown_connections 2022-07-21T01:38:27.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.821+0000 7fe6fec13700 1 -- 172.21.15.3:0/1617359392 wait complete. 2022-07-21T01:38:27.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:27.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.957+0000 7faa8359e700 1 Processor -- start 2022-07-21T01:38:27.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.957+0000 7faa8359e700 1 -- start start 2022-07-21T01:38:27.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.957+0000 7faa8359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840b3190 0x7faa840b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.957+0000 7faa8359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faa840a9da0 0x7faa84147e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.957+0000 7faa8359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa840b3570 con 0x7faa840b1a80 2022-07-21T01:38:27.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.957+0000 7faa8359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa840b2980 con 0x7faa840a9da0 2022-07-21T01:38:27.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.957+0000 7faa8359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa840b0e00 con 0x7faa840b3190 2022-07-21T01:38:27.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa82d9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faa840a9da0 0x7faa84147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa8259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840b3190 0x7faa840b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa7bfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7faa840b1a80 0x7faa840656a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:56648/0 (socket says 172.21.15.3:56648) 2022-07-21T01:38:27.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa7bfff700 1 -- 172.21.15.3:0/2895609477 learned_addr learned my addr 172.21.15.3:0/2895609477 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:27.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa81d9b700 1 -- 172.21.15.3:0/2895609477 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274301030 0 0) 0x7faa840b3570 con 0x7faa840b1a80 2022-07-21T01:38:27.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa81d9b700 1 -- 172.21.15.3:0/2895609477 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa64003260 con 0x7faa840b1a80 2022-07-21T01:38:27.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa82d9d700 1 -- 172.21.15.3:0/2895609477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840b3190 msgr2=0x7faa840b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:27.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa82d9d700 1 --2- 172.21.15.3:0/2895609477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840b3190 0x7faa840b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa82d9d700 1 -- 172.21.15.3:0/2895609477 >> v1:172.21.15.143:6789/0 conn(0x7faa840b1a80 legacy=0x7faa840656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa82d9d700 1 -- 172.21.15.3:0/2895609477 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa840b10a0 con 0x7faa840a9da0 2022-07-21T01:38:27.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa82d9d700 1 --2- 172.21.15.3:0/2895609477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa840a9da0 0x7faa84147e30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7faa7c0032f0 tx=0x7faa7c0091d0).ready entity=mon.1 client_cookie=282f7ebceb960627 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa81d9b700 1 -- 172.21.15.3:0/2895609477 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa7c00b600 con 0x7faa840a9da0 2022-07-21T01:38:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa81d9b700 1 -- 172.21.15.3:0/2895609477 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c00b760 con 0x7faa840a9da0 2022-07-21T01:38:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa81d9b700 1 -- 172.21.15.3:0/2895609477 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa7c00b8d0 con 0x7faa840a9da0 2022-07-21T01:38:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa8359e700 1 -- 172.21.15.3:0/2895609477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa840a9da0 msgr2=0x7faa84147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa8359e700 1 --2- 172.21.15.3:0/2895609477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa840a9da0 0x7faa84147e30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7faa7c0032f0 tx=0x7faa7c0091d0).stop 2022-07-21T01:38:27.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa8359e700 1 -- 172.21.15.3:0/2895609477 shutdown_connections 2022-07-21T01:38:27.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa8359e700 1 --2- 172.21.15.3:0/2895609477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa840a9da0 0x7faa84147e30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa8359e700 1 --2- 172.21.15.3:0/2895609477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840b3190 0x7faa840b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa8359e700 1 -- 172.21.15.3:0/2895609477 >> 172.21.15.3:0/2895609477 conn(0x7faa84063870 msgr2=0x7faa84063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:27.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa8359e700 1 -- 172.21.15.3:0/2895609477 shutdown_connections 2022-07-21T01:38:27.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.961+0000 7faa8359e700 1 -- 172.21.15.3:0/2895609477 wait complete. 2022-07-21T01:38:27.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.965+0000 7faa8359e700 1 Processor -- start 2022-07-21T01:38:27.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.965+0000 7faa8359e700 1 -- start start 2022-07-21T01:38:27.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.965+0000 7faa8359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840a9da0 0x7faa84144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.965+0000 7faa8359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa840b1a80 0x7faa84141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.965+0000 7faa8359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa840b3190 0x7faa84142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.965+0000 7faa8359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8415a940 con 0x7faa840b1a80 2022-07-21T01:38:27.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.965+0000 7faa8359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa84142980 con 0x7faa840b3190 2022-07-21T01:38:27.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.965+0000 7faa8359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8415a6a0 con 0x7faa840a9da0 2022-07-21T01:38:27.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa840b1a80 0x7faa84141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa840b1a80 0x7faa84141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46260/0 (socket says 172.21.15.3:46260) 2022-07-21T01:38:27.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa8259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840a9da0 0x7faa84144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa82d9d700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa840b3190 0x7faa84142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7bfff700 1 -- 172.21.15.3:0/1711878128 learned_addr learned my addr 172.21.15.3:0/1711878128 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:27.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7bfff700 1 -- 172.21.15.3:0/1711878128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840a9da0 msgr2=0x7faa84144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7bfff700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840a9da0 0x7faa84144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7bfff700 1 -- 172.21.15.3:0/1711878128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa840b3190 msgr2=0x7faa84142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:27.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7bfff700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa840b3190 0x7faa84142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:27.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7bfff700 1 -- 172.21.15.3:0/1711878128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c009470 con 0x7faa840b1a80 2022-07-21T01:38:27.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7bfff700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa840b1a80 0x7faa84141fa0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7faa6c001110 tx=0x7faa6c007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:27.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7b7fe700 1 -- 172.21.15.3:0/1711878128 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa6c0070d0 con 0x7faa840b1a80 2022-07-21T01:38:27.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7b7fe700 1 -- 172.21.15.3:0/1711878128 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c007230 con 0x7faa840b1a80 2022-07-21T01:38:27.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa840b0530 con 0x7faa840b1a80 2022-07-21T01:38:27.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa7b7fe700 1 -- 172.21.15.3:0/1711878128 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa6c005610 con 0x7faa840b1a80 2022-07-21T01:38:27.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa840b09d0 con 0x7faa840b1a80 2022-07-21T01:38:27.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.969+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa840574e0 con 0x7faa840b1a80 2022-07-21T01:38:27.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.973+0000 7faa7b7fe700 1 -- 172.21.15.3:0/1711878128 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faa6c01a070 con 0x7faa840b1a80 2022-07-21T01:38:27.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.973+0000 7faa7b7fe700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa68053a70 0x7faa68055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:27.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.973+0000 7faa7b7fe700 1 -- 172.21.15.3:0/1711878128 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7faa6c06b020 con 0x7faa840b1a80 2022-07-21T01:38:27.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.973+0000 7faa8259c700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa68053a70 0x7faa68055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:27.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.973+0000 7faa8259c700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa68053a70 0x7faa68055f30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7faa7400b000 tx=0x7faa74006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:27.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:27.977+0000 7faa7b7fe700 1 -- 172.21.15.3:0/1711878128 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faa6c041de0 con 0x7faa840b1a80 2022-07-21T01:38:28.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.145+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa84069a70 con 0x7faa840b1a80 2022-07-21T01:38:28.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.145+0000 7faa7b7fe700 1 -- 172.21.15.3:0/1711878128 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2490 (secure 0 0 0) 0x7faa6c045400 con 0x7faa840b1a80 2022-07-21T01:38:28.146 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:28.146 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":29,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:38:26.808059+0000","modified":"2022-07-21T01:38:27.811394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/2524169670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2524169670},{"type":"v1","addr":"172.21.15.143:6833","nonce":2524169670}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-21T01:38:28.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa68053a70 msgr2=0x7faa68055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa68053a70 0x7faa68055f30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7faa7400b000 tx=0x7faa74006df0).stop 2022-07-21T01:38:28.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa840b1a80 msgr2=0x7faa84141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa840b1a80 0x7faa84141fa0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7faa6c001110 tx=0x7faa6c007a60).stop 2022-07-21T01:38:28.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 shutdown_connections 2022-07-21T01:38:28.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa68053a70 0x7faa68055f30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa840b3190 0x7faa84142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa840b1a80 0x7faa84141fa0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 --2- 172.21.15.3:0/1711878128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa840a9da0 0x7faa84144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 >> 172.21.15.3:0/1711878128 conn(0x7faa84063870 msgr2=0x7faa84145340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:28.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 shutdown_connections 2022-07-21T01:38:28.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.149+0000 7faa8359e700 1 -- 172.21.15.3:0/1711878128 wait complete. 2022-07-21T01:38:28.154 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 29 2022-07-21T01:38:28.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:38:28.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd918d70700 1 Processor -- start 2022-07-21T01:38:28.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd918d70700 1 -- start start 2022-07-21T01:38:28.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd918d70700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9140b2710 0x7fd9141401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd918d70700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd9140b0270 0x7fd9141406d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd918d70700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd914067a10 con 0x7fd9140ac880 2022-07-21T01:38:28.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd918d70700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9140ad740 con 0x7fd9140b0270 2022-07-21T01:38:28.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd918d70700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd914067cb0 con 0x7fd9140b2710 2022-07-21T01:38:28.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd912ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd9140b0270 0x7fd9141406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd912ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd9140b0270 0x7fd9141406d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46266/0 (socket says 172.21.15.3:46266) 2022-07-21T01:38:28.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd912ffd700 1 -- 172.21.15.3:0/1629136490 learned_addr learned my addr 172.21.15.3:0/1629136490 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:28.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd9137fe700 1 --2- 172.21.15.3:0/1629136490 >> v2:172.21.15.143:3300/0 conn(0x7fd9140b2710 0x7fd9141401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd912ffd700 1 -- 172.21.15.3:0/1629136490 >> v2:172.21.15.143:3300/0 conn(0x7fd9140b2710 msgr2=0x7fd9141401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:28.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd912ffd700 1 --2- 172.21.15.3:0/1629136490 >> v2:172.21.15.143:3300/0 conn(0x7fd9140b2710 0x7fd9141401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd912ffd700 1 -- 172.21.15.3:0/1629136490 >> v1:172.21.15.143:6789/0 conn(0x7fd9140ac880 legacy=0x7fd914143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd912ffd700 1 -- 172.21.15.3:0/1629136490 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd914147160 con 0x7fd9140b0270 2022-07-21T01:38:28.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd912ffd700 1 --2- 172.21.15.3:0/1629136490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 0x7fd9141406d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fd904002f70 tx=0x7fd904006eb0).ready entity=mon.0 client_cookie=2eb680a2e4954b55 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:28.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd9127fc700 1 -- 172.21.15.3:0/1629136490 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd90400d610 con 0x7fd9140b0270 2022-07-21T01:38:28.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd9127fc700 1 -- 172.21.15.3:0/1629136490 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd904000e90 con 0x7fd9140b0270 2022-07-21T01:38:28.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.265+0000 7fd9127fc700 1 -- 172.21.15.3:0/1629136490 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd90400d9a0 con 0x7fd9140b0270 2022-07-21T01:38:28.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 -- 172.21.15.3:0/1629136490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 msgr2=0x7fd9141406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 --2- 172.21.15.3:0/1629136490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 0x7fd9141406d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fd904002f70 tx=0x7fd904006eb0).stop 2022-07-21T01:38:28.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 -- 172.21.15.3:0/1629136490 shutdown_connections 2022-07-21T01:38:28.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 --2- 172.21.15.3:0/1629136490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 0x7fd9141406d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 --2- 172.21.15.3:0/1629136490 >> v2:172.21.15.143:3300/0 conn(0x7fd9140b2710 0x7fd9141401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 -- 172.21.15.3:0/1629136490 >> 172.21.15.3:0/1629136490 conn(0x7fd914060310 msgr2=0x7fd9140b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:28.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 -- 172.21.15.3:0/1629136490 shutdown_connections 2022-07-21T01:38:28.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 -- 172.21.15.3:0/1629136490 wait complete. 2022-07-21T01:38:28.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 Processor -- start 2022-07-21T01:38:28.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.269+0000 7fd918d70700 1 -- start start 2022-07-21T01:38:28.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd918d70700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9140ac880 0x7fd9140ab720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd918d70700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 0x7fd914152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd918d70700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9140b2710 0x7fd914141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd918d70700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91413f300 con 0x7fd9140b0270 2022-07-21T01:38:28.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd918d70700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd914143470 con 0x7fd9140b2710 2022-07-21T01:38:28.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd918d70700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd914143710 con 0x7fd9140ac880 2022-07-21T01:38:28.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd912ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 0x7fd914152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd912ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 0x7fd914152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46274/0 (socket says 172.21.15.3:46274) 2022-07-21T01:38:28.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd912ffd700 1 -- 172.21.15.3:0/1820338182 learned_addr learned my addr 172.21.15.3:0/1820338182 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:28.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd913fff700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9140b2710 0x7fd914141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd9137fe700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9140ac880 0x7fd9140ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd912ffd700 1 -- 172.21.15.3:0/1820338182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9140ac880 msgr2=0x7fd9140ab720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd912ffd700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9140ac880 0x7fd9140ab720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd912ffd700 1 -- 172.21.15.3:0/1820338182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9140b2710 msgr2=0x7fd914141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd912ffd700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9140b2710 0x7fd914141570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd912ffd700 1 -- 172.21.15.3:0/1820338182 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90400b1e0 con 0x7fd9140b0270 2022-07-21T01:38:28.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd912ffd700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 0x7fd914152d40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fd904002f70 tx=0x7fd904015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:28.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd910ff9700 1 -- 172.21.15.3:0/1820338182 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd904011860 con 0x7fd9140b0270 2022-07-21T01:38:28.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd910ff9700 1 -- 172.21.15.3:0/1820338182 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9040119c0 con 0x7fd9140b0270 2022-07-21T01:38:28.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd910ff9700 1 -- 172.21.15.3:0/1820338182 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd904011d20 con 0x7fd9140b0270 2022-07-21T01:38:28.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd914152600 con 0x7fd9140b0270 2022-07-21T01:38:28.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9140a1960 con 0x7fd9140b0270 2022-07-21T01:38:28.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.273+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9140574e0 con 0x7fd9140b0270 2022-07-21T01:38:28.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.277+0000 7fd910ff9700 1 -- 172.21.15.3:0/1820338182 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd90401a040 con 0x7fd9140b0270 2022-07-21T01:38:28.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.277+0000 7fd910ff9700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8f4053950 0x7fd8f4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.277+0000 7fd910ff9700 1 -- 172.21.15.3:0/1820338182 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd90406d330 con 0x7fd9140b0270 2022-07-21T01:38:28.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.277+0000 7fd9137fe700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8f4053950 0x7fd8f4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.277+0000 7fd9137fe700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8f4053950 0x7fd8f4055e10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fd90800c000 tx=0x7fd908008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:28.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.281+0000 7fd910ff9700 1 -- 172.21.15.3:0/1820338182 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd9040440f0 con 0x7fd9140b0270 2022-07-21T01:38:28.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.405+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd914152aa0 con 0x7fd9140b0270 2022-07-21T01:38:28.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.405+0000 7fd910ff9700 1 -- 172.21.15.3:0/1820338182 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16274 (secure 0 0 0) 0x7fd904023070 con 0x7fd9140b0270 2022-07-21T01:38:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:28.406 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":68,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:38:27.812077+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:38:25.308080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-21T01:38:26.095494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:38:28.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8f4053950 msgr2=0x7fd8f4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8f4053950 0x7fd8f4055e10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fd90800c000 tx=0x7fd908008040).stop 2022-07-21T01:38:28.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 msgr2=0x7fd914152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 0x7fd914152d40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fd904002f70 tx=0x7fd904015000).stop 2022-07-21T01:38:28.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 shutdown_connections 2022-07-21T01:38:28.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8f4053950 0x7fd8f4055e10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9140b2710 0x7fd914141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9140b0270 0x7fd914152d40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 --2- 172.21.15.3:0/1820338182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9140ac880 0x7fd9140ab720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 >> 172.21.15.3:0/1820338182 conn(0x7fd914060310 msgr2=0x7fd91405bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:28.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 shutdown_connections 2022-07-21T01:38:28.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.409+0000 7fd918d70700 1 -- 172.21.15.3:0/1820338182 wait complete. 2022-07-21T01:38:28.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:38:28.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.529+0000 7fcb2633a700 1 Processor -- start 2022-07-21T01:38:28.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.529+0000 7fcb2633a700 1 -- start start 2022-07-21T01:38:28.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcb200b1180 0x7fcb20141920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200aac20 con 0x7fcb2013e8d0 2022-07-21T01:38:28.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200b2c90 con 0x7fcb200b28b0 2022-07-21T01:38:28.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb20140e20 con 0x7fcb200b1180 2022-07-21T01:38:28.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb24b37700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcb2013e8d0 0x7fcb20141210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54562/0 (socket says 172.21.15.3:54562) 2022-07-21T01:38:28.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb24b37700 1 -- 172.21.15.3:0/2147454578 learned_addr learned my addr 172.21.15.3:0/2147454578 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:28.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb25b39700 1 --2- 172.21.15.3:0/2147454578 >> v2:172.21.15.143:3300/0 conn(0x7fcb200b1180 0x7fcb20141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb17fff700 1 -- 172.21.15.3:0/2147454578 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531791630 0 0) 0x7fcb1c001110 con 0x7fcb2013e8d0 2022-07-21T01:38:28.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb17fff700 1 -- 172.21.15.3:0/2147454578 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb04003260 con 0x7fcb2013e8d0 2022-07-21T01:38:28.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb25b39700 1 -- 172.21.15.3:0/2147454578 >> v1:172.21.15.143:6789/0 conn(0x7fcb200b28b0 legacy=0x7fcb2013e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb25b39700 1 -- 172.21.15.3:0/2147454578 >> v1:172.21.15.3:6789/0 conn(0x7fcb2013e8d0 legacy=0x7fcb20141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb17fff700 1 -- 172.21.15.3:0/2147454578 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852955336 0 0) 0x7fcb04003260 con 0x7fcb2013e8d0 2022-07-21T01:38:28.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb25b39700 1 -- 172.21.15.3:0/2147454578 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200aa440 con 0x7fcb200b1180 2022-07-21T01:38:28.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb25b39700 1 --2- 172.21.15.3:0/2147454578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb200b1180 0x7fcb20141920 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fcb1c001990 tx=0x7fcb180091d0).ready entity=mon.1 client_cookie=6c4d6757ea439ec6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:28.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb17fff700 1 -- 172.21.15.3:0/2147454578 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb1c0017f0 con 0x7fcb200b1180 2022-07-21T01:38:28.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb17fff700 1 -- 172.21.15.3:0/2147454578 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1800b580 con 0x7fcb200b1180 2022-07-21T01:38:28.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb17fff700 1 -- 172.21.15.3:0/2147454578 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb1800b6f0 con 0x7fcb200b1180 2022-07-21T01:38:28.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 -- 172.21.15.3:0/2147454578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb200b1180 msgr2=0x7fcb20141920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 --2- 172.21.15.3:0/2147454578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb200b1180 0x7fcb20141920 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fcb1c001990 tx=0x7fcb180091d0).stop 2022-07-21T01:38:28.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 -- 172.21.15.3:0/2147454578 shutdown_connections 2022-07-21T01:38:28.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 --2- 172.21.15.3:0/2147454578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb200b1180 0x7fcb20141920 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 -- 172.21.15.3:0/2147454578 >> 172.21.15.3:0/2147454578 conn(0x7fcb20060310 msgr2=0x7fcb200b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:28.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 -- 172.21.15.3:0/2147454578 shutdown_connections 2022-07-21T01:38:28.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.533+0000 7fcb2633a700 1 -- 172.21.15.3:0/2147454578 wait complete. 2022-07-21T01:38:28.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 Processor -- start 2022-07-21T01:38:28.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 -- start start 2022-07-21T01:38:28.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb200b1180 0x7fcb201436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb200b28b0 0x7fcb20150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb20144970 0x7fcb20150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20143cf0 con 0x7fcb200b28b0 2022-07-21T01:38:28.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20143f90 con 0x7fcb20144970 2022-07-21T01:38:28.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20144230 con 0x7fcb200b1180 2022-07-21T01:38:28.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb24b37700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb200b28b0 0x7fcb20150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb24b37700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb200b28b0 0x7fcb20150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46288/0 (socket says 172.21.15.3:46288) 2022-07-21T01:38:28.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb24b37700 1 -- 172.21.15.3:0/1713001480 learned_addr learned my addr 172.21.15.3:0/1713001480 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:28.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb25b39700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb20144970 0x7fcb20150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb25338700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb200b1180 0x7fcb201436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb25b39700 1 -- 172.21.15.3:0/1713001480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb200b1180 msgr2=0x7fcb201436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb25b39700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb200b1180 0x7fcb201436e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb25b39700 1 -- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb200b28b0 msgr2=0x7fcb20150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb25b39700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb200b28b0 0x7fcb20150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb25b39700 1 -- 172.21.15.3:0/1713001480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb18009470 con 0x7fcb20144970 2022-07-21T01:38:28.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb24b37700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb200b28b0 0x7fcb20150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:28.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb25b39700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb20144970 0x7fcb20150750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fcb1c001990 tx=0x7fcb18016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:28.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb167fc700 1 -- 172.21.15.3:0/1713001480 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb1800b6f0 con 0x7fcb20144970 2022-07-21T01:38:28.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb201444d0 con 0x7fcb20144970 2022-07-21T01:38:28.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb167fc700 1 -- 172.21.15.3:0/1713001480 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb180032f0 con 0x7fcb20144970 2022-07-21T01:38:28.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb167fc700 1 -- 172.21.15.3:0/1713001480 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb18021480 con 0x7fcb20144970 2022-07-21T01:38:28.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.537+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb20142300 con 0x7fcb20144970 2022-07-21T01:38:28.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.541+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb20144660 con 0x7fcb20144970 2022-07-21T01:38:28.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.541+0000 7fcb167fc700 1 -- 172.21.15.3:0/1713001480 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb1800f070 con 0x7fcb20144970 2022-07-21T01:38:28.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.541+0000 7fcb167fc700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 0x7fcb08055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.541+0000 7fcb167fc700 1 -- 172.21.15.3:0/1713001480 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcb1806d380 con 0x7fcb20144970 2022-07-21T01:38:28.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.541+0000 7fcb25338700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 0x7fcb08055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.541+0000 7fcb25338700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 0x7fcb08055ea0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fcb1000d000 tx=0x7fcb10000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:28.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.549+0000 7fcb167fc700 1 -- 172.21.15.3:0/1713001480 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb18044060 con 0x7fcb20144970 2022-07-21T01:38:28.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.717+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fcb200574e0 con 0x7fcb20144970 2022-07-21T01:38:28.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb167fc700 1 -- 172.21.15.3:0/1713001480 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v21) v1 ==== 184+0+0 (secure 0 0 0) 0x7fcb180260e0 con 0x7fcb20144970 2022-07-21T01:38:28.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 msgr2=0x7fcb08055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 0x7fcb08055ea0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fcb1000d000 tx=0x7fcb10000bc0).stop 2022-07-21T01:38:28.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb20144970 msgr2=0x7fcb20150750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb20144970 0x7fcb20150750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fcb1c001990 tx=0x7fcb18016000).stop 2022-07-21T01:38:28.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 shutdown_connections 2022-07-21T01:38:28.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb080539e0 0x7fcb08055ea0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb20144970 0x7fcb20150750 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb200b28b0 0x7fcb20150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 --2- 172.21.15.3:0/1713001480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb200b1180 0x7fcb201436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.729+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 >> 172.21.15.3:0/1713001480 conn(0x7fcb20060310 msgr2=0x7fcb20141130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:28.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.733+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 shutdown_connections 2022-07-21T01:38:28.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.733+0000 7fcb2633a700 1 -- 172.21.15.3:0/1713001480 wait complete. 2022-07-21T01:38:28.734 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:38:28.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:38:28.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3cd39f700 1 Processor -- start 2022-07-21T01:38:28.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3cd39f700 1 -- start start 2022-07-21T01:38:28.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3cd39f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd3c814b950 0x7fd3c814dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3cd39f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c80b60e0 con 0x7fd3c80b41f0 2022-07-21T01:38:28.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3cd39f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c80b2100 con 0x7fd3c80b6420 2022-07-21T01:38:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3cd39f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c80b23a0 con 0x7fd3c814b950 2022-07-21T01:38:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3bf7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd3c80b41f0 0x7fd3c8061d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54574/0 (socket says 172.21.15.3:54574) 2022-07-21T01:38:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3bf7fe700 1 -- 172.21.15.3:0/4264788005 learned_addr learned my addr 172.21.15.3:0/4264788005 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3ccb9e700 1 --2- 172.21.15.3:0/4264788005 >> v2:172.21.15.143:3300/0 conn(0x7fd3c814b950 0x7fd3c814dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659573698 0 0) 0x7fd3b0001110 con 0x7fd3c80b41f0 2022-07-21T01:38:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac003260 con 0x7fd3c80b41f0 2022-07-21T01:38:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2783163662 0 0) 0x7fd3ac003260 con 0x7fd3c80b41f0 2022-07-21T01:38:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 >> v2:172.21.15.143:3300/0 conn(0x7fd3c814b950 msgr2=0x7fd3c814dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.857+0000 7fd3c77fe700 1 --2- 172.21.15.3:0/4264788005 >> v2:172.21.15.143:3300/0 conn(0x7fd3c814b950 0x7fd3c814dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 >> v1:172.21.15.143:6789/0 conn(0x7fd3c80b6420 legacy=0x7fd3c8061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c80b2640 con 0x7fd3c80b41f0 2022-07-21T01:38:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fd3b00017f0 con 0x7fd3c80b41f0 2022-07-21T01:38:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 >> v1:172.21.15.3:6789/0 conn(0x7fd3c80b41f0 legacy=0x7fd3c8061d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3ac003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3ac004500 0x7fd3ac004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3ac004e70 0x7fd3ac00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c80b23a0 con 0x7fd3ac003750 2022-07-21T01:38:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac00bd20 con 0x7fd3ac004e70 2022-07-21T01:38:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac00bee0 con 0x7fd3ac004500 2022-07-21T01:38:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c7fff700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3ac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3ccb9e700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3ac004e70 0x7fd3ac00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3bf7fe700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3ac004500 0x7fd3ac004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c7fff700 1 -- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3ac004500 msgr2=0x7fd3ac004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c7fff700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3ac004500 0x7fd3ac004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c7fff700 1 -- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3ac004e70 msgr2=0x7fd3ac00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c7fff700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3ac004e70 0x7fd3ac00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c7fff700 1 -- 172.21.15.3:0/4264788005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac003260 con 0x7fd3ac003750 2022-07-21T01:38:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3ccb9e700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3ac004e70 0x7fd3ac00b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:28.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3bf7fe700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3ac004500 0x7fd3ac004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:28.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3ccb9e700 1 -- 172.21.15.3:0/4264788005 reap_dead start 2022-07-21T01:38:28.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c7fff700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3ac003b70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fd3c8060320 tx=0x7fd3b8007a00).ready entity=mon.0 client_cookie=66107b9870047485 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:28.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3b8007d10 con 0x7fd3ac003750 2022-07-21T01:38:28.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b8007030 con 0x7fd3ac003750 2022-07-21T01:38:28.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3c77fe700 1 -- 172.21.15.3:0/4264788005 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3b8007390 con 0x7fd3ac003750 2022-07-21T01:38:28.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3cd39f700 1 -- 172.21.15.3:0/4264788005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 msgr2=0x7fd3ac003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3cd39f700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3ac003b70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fd3c8060320 tx=0x7fd3b8007a00).stop 2022-07-21T01:38:28.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3cd39f700 1 -- 172.21.15.3:0/4264788005 shutdown_connections 2022-07-21T01:38:28.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3cd39f700 1 --2- 172.21.15.3:0/4264788005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3ac003b70 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3cd39f700 1 -- 172.21.15.3:0/4264788005 >> 172.21.15.3:0/4264788005 conn(0x7fd3c805c620 msgr2=0x7fd3c805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:28.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3cd39f700 1 -- 172.21.15.3:0/4264788005 shutdown_connections 2022-07-21T01:38:28.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.861+0000 7fd3cd39f700 1 -- 172.21.15.3:0/4264788005 wait complete. 2022-07-21T01:38:28.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 Processor -- start 2022-07-21T01:38:28.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 -- start start 2022-07-21T01:38:28.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3c812b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3c814a510 0x7fd3c80adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8149080 0x7fd3c8152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c814a330 con 0x7fd3ac003750 2022-07-21T01:38:28.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c80ae480 con 0x7fd3c8149080 2022-07-21T01:38:28.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c814a090 con 0x7fd3c814a510 2022-07-21T01:38:28.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3c812b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3c812b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46306/0 (socket says 172.21.15.3:46306) 2022-07-21T01:38:28.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c7fff700 1 -- 172.21.15.3:0/3364452759 learned_addr learned my addr 172.21.15.3:0/3364452759 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:28.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3bf7fe700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3c814a510 0x7fd3c80adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3ccb9e700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8149080 0x7fd3c8152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c7fff700 1 -- 172.21.15.3:0/3364452759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3c814a510 msgr2=0x7fd3c80adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c7fff700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3c814a510 0x7fd3c80adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c7fff700 1 -- 172.21.15.3:0/3364452759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8149080 msgr2=0x7fd3c8152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:28.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c7fff700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8149080 0x7fd3c8152e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:28.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c7fff700 1 -- 172.21.15.3:0/3364452759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c80ad850 con 0x7fd3ac003750 2022-07-21T01:38:28.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c7fff700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3c812b610 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fd3c8060320 tx=0x7fd3b8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:28.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c5ffb700 1 -- 172.21.15.3:0/3364452759 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3b8007d10 con 0x7fd3ac003750 2022-07-21T01:38:28.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c5ffb700 1 -- 172.21.15.3:0/3364452759 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b80127b0 con 0x7fd3ac003750 2022-07-21T01:38:28.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c80adaf0 con 0x7fd3ac003750 2022-07-21T01:38:28.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3c5ffb700 1 -- 172.21.15.3:0/3364452759 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3b8012b10 con 0x7fd3ac003750 2022-07-21T01:38:28.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c8153360 con 0x7fd3ac003750 2022-07-21T01:38:28.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.865+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c80adc80 con 0x7fd3ac003750 2022-07-21T01:38:28.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.869+0000 7fd3c5ffb700 1 -- 172.21.15.3:0/3364452759 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd3b801d070 con 0x7fd3ac003750 2022-07-21T01:38:28.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.869+0000 7fd3c5ffb700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd3a0053b00 0x7fd3a0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:28.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.869+0000 7fd3c5ffb700 1 -- 172.21.15.3:0/3364452759 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd3b8001e40 con 0x7fd3ac003750 2022-07-21T01:38:28.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.869+0000 7fd3bf7fe700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd3a0053b00 0x7fd3a0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:28.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.869+0000 7fd3bf7fe700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd3a0053b00 0x7fd3a0055fc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fd3b000d000 tx=0x7fd3b0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:28.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:28.873+0000 7fd3c5ffb700 1 -- 172.21.15.3:0/3364452759 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd3b8043c40 con 0x7fd3ac003750 2022-07-21T01:38:29.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.045+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd3c80b28e0 con 0x7fd3ac003750 2022-07-21T01:38:29.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.049+0000 7fd3c5ffb700 1 -- 172.21.15.3:0/3364452759 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v22) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd3b8047260 con 0x7fd3ac003750 2022-07-21T01:38:29.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd3a0053b00 msgr2=0x7fd3a0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd3a0053b00 0x7fd3a0055fc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fd3b000d000 tx=0x7fd3b0009040).stop 2022-07-21T01:38:29.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 msgr2=0x7fd3c812b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3c812b610 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fd3c8060320 tx=0x7fd3b8015000).stop 2022-07-21T01:38:29.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 shutdown_connections 2022-07-21T01:38:29.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd3a0053b00 0x7fd3a0055fc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8149080 0x7fd3c8152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3c814a510 0x7fd3c80adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 --2- 172.21.15.3:0/3364452759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3ac003750 0x7fd3c812b610 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 >> 172.21.15.3:0/3364452759 conn(0x7fd3c805c620 msgr2=0x7fd3c8143110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:29.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 shutdown_connections 2022-07-21T01:38:29.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.053+0000 7fd3cd39f700 1 -- 172.21.15.3:0/3364452759 wait complete. 2022-07-21T01:38:29.054 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:38:29.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:29.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.165+0000 7fef78f5f700 1 Processor -- start 2022-07-21T01:38:29.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.165+0000 7fef78f5f700 1 -- start start 2022-07-21T01:38:29.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fef740aa2b0 0x7fef740ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef740b0a10 0x7fef74140730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef740b2c80 con 0x7fef740b0630 2022-07-21T01:38:29.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef740b2900 con 0x7fef740aa2b0 2022-07-21T01:38:29.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef740b2eb0 con 0x7fef740b0a10 2022-07-21T01:38:29.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef71d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fef740aa2b0 0x7fef740ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef7259c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fef740b0630 0x7fef740adec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54596/0 (socket says 172.21.15.3:54596) 2022-07-21T01:38:29.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef71d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fef740aa2b0 0x7fef740ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46314/0 (socket says 172.21.15.3:46314) 2022-07-21T01:38:29.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef7259c700 1 -- 172.21.15.3:0/1655465510 learned_addr learned my addr 172.21.15.3:0/1655465510 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef72d9d700 1 --2- 172.21.15.3:0/1655465510 >> v2:172.21.15.143:3300/0 conn(0x7fef740b0a10 0x7fef74140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef71d9b700 1 -- 172.21.15.3:0/1655465510 >> v2:172.21.15.143:3300/0 conn(0x7fef740b0a10 msgr2=0x7fef74140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef71d9b700 1 --2- 172.21.15.3:0/1655465510 >> v2:172.21.15.143:3300/0 conn(0x7fef740b0a10 0x7fef74140730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef71d9b700 1 -- 172.21.15.3:0/1655465510 >> v1:172.21.15.3:6789/0 conn(0x7fef740b0630 legacy=0x7fef740adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef7159a700 1 -- 172.21.15.3:0/1655465510 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382438999 0 0) 0x7fef740b2c80 con 0x7fef740b0630 2022-07-21T01:38:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef71d9b700 1 -- 172.21.15.3:0/1655465510 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef740a9d00 con 0x7fef740aa2b0 2022-07-21T01:38:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef71d9b700 1 --2- 172.21.15.3:0/1655465510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740aa2b0 0x7fef740ae5d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fef5c002f70 tx=0x7fef5c006df0).ready entity=mon.0 client_cookie=e5054a3d688869af server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef7159a700 1 -- 172.21.15.3:0/1655465510 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef5c00d5c0 con 0x7fef740aa2b0 2022-07-21T01:38:29.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef7159a700 1 -- 172.21.15.3:0/1655465510 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5c000e90 con 0x7fef740aa2b0 2022-07-21T01:38:29.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef7159a700 1 -- 172.21.15.3:0/1655465510 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef5c00d910 con 0x7fef740aa2b0 2022-07-21T01:38:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 -- 172.21.15.3:0/1655465510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740aa2b0 msgr2=0x7fef740ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 --2- 172.21.15.3:0/1655465510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740aa2b0 0x7fef740ae5d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fef5c002f70 tx=0x7fef5c006df0).stop 2022-07-21T01:38:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 -- 172.21.15.3:0/1655465510 shutdown_connections 2022-07-21T01:38:29.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 --2- 172.21.15.3:0/1655465510 >> v2:172.21.15.143:3300/0 conn(0x7fef740b0a10 0x7fef74140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 --2- 172.21.15.3:0/1655465510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740aa2b0 0x7fef740ae5d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 -- 172.21.15.3:0/1655465510 >> 172.21.15.3:0/1655465510 conn(0x7fef74060950 msgr2=0x7fef74060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:29.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 -- 172.21.15.3:0/1655465510 shutdown_connections 2022-07-21T01:38:29.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.169+0000 7fef78f5f700 1 -- 172.21.15.3:0/1655465510 wait complete. 2022-07-21T01:38:29.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef78f5f700 1 Processor -- start 2022-07-21T01:38:29.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef78f5f700 1 -- start start 2022-07-21T01:38:29.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef78f5f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef740aa2b0 0x7fef74136060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef78f5f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740b0630 0x7fef74063a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef78f5f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef740b0a10 0x7fef7412d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef78f5f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef74067b80 con 0x7fef740b0630 2022-07-21T01:38:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef78f5f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef741395e0 con 0x7fef740b0a10 2022-07-21T01:38:29.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef78f5f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef74063f50 con 0x7fef740aa2b0 2022-07-21T01:38:29.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef71d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740b0630 0x7fef74063a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef71d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740b0630 0x7fef74063a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46322/0 (socket says 172.21.15.3:46322) 2022-07-21T01:38:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef71d9b700 1 -- 172.21.15.3:0/1572463064 learned_addr learned my addr 172.21.15.3:0/1572463064 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef72d9d700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef740b0a10 0x7fef7412d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef7259c700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef740aa2b0 0x7fef74136060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef71d9b700 1 -- 172.21.15.3:0/1572463064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef740aa2b0 msgr2=0x7fef74136060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef71d9b700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef740aa2b0 0x7fef74136060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef71d9b700 1 -- 172.21.15.3:0/1572463064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef740b0a10 msgr2=0x7fef7412d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef71d9b700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef740b0a10 0x7fef7412d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef71d9b700 1 -- 172.21.15.3:0/1572463064 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c00b180 con 0x7fef740b0630 2022-07-21T01:38:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef7259c700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef740aa2b0 0x7fef74136060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.173+0000 7fef71d9b700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740b0630 0x7fef74063a60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fef5c002f70 tx=0x7fef5c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:29.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef6b7fe700 1 -- 172.21.15.3:0/1572463064 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef5c013e90 con 0x7fef740b0630 2022-07-21T01:38:29.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef6b7fe700 1 -- 172.21.15.3:0/1572463064 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5c01e400 con 0x7fef740b0630 2022-07-21T01:38:29.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef6b7fe700 1 -- 172.21.15.3:0/1572463064 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef5c01e760 con 0x7fef740b0630 2022-07-21T01:38:29.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef74139340 con 0x7fef740b0630 2022-07-21T01:38:29.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef740b4ee0 con 0x7fef740b0630 2022-07-21T01:38:29.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef740574e0 con 0x7fef740b0630 2022-07-21T01:38:29.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef6b7fe700 1 -- 172.21.15.3:0/1572463064 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef5c01c070 con 0x7fef740b0630 2022-07-21T01:38:29.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef6b7fe700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef580539e0 0x7fef58055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef6b7fe700 1 -- 172.21.15.3:0/1572463064 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fef5c06cb10 con 0x7fef740b0630 2022-07-21T01:38:29.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef7259c700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef580539e0 0x7fef58055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.177+0000 7fef7259c700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef580539e0 0x7fef58055ea0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fef6400b000 tx=0x7fef64009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:29.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.185+0000 7fef6b7fe700 1 -- 172.21.15.3:0/1572463064 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef5c0438d0 con 0x7fef740b0630 2022-07-21T01:38:29.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef740447e0 con 0x7fef740b0630 2022-07-21T01:38:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef6b7fe700 1 -- 172.21.15.3:0/1572463064 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2490 (secure 0 0 0) 0x7fef5c023020 con 0x7fef740b0630 2022-07-21T01:38:29.348 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:29.348 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":29,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:38:26.808059+0000","modified":"2022-07-21T01:38:27.811394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/2524169670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2524169670},{"type":"v1","addr":"172.21.15.143:6833","nonce":2524169670}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-21T01:38:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef580539e0 msgr2=0x7fef58055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef580539e0 0x7fef58055ea0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fef6400b000 tx=0x7fef64009040).stop 2022-07-21T01:38:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740b0630 msgr2=0x7fef74063a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740b0630 0x7fef74063a60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fef5c002f70 tx=0x7fef5c014000).stop 2022-07-21T01:38:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 shutdown_connections 2022-07-21T01:38:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef580539e0 0x7fef58055ea0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef740b0a10 0x7fef7412d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef740b0630 0x7fef74063a60 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 --2- 172.21.15.3:0/1572463064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef740aa2b0 0x7fef74136060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 >> 172.21.15.3:0/1572463064 conn(0x7fef74060950 msgr2=0x7fef7413a220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 shutdown_connections 2022-07-21T01:38:29.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.349+0000 7fef78f5f700 1 -- 172.21.15.3:0/1572463064 wait complete. 2022-07-21T01:38:29.353 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 29 2022-07-21T01:38:29.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:38:26.808059+0000', 'modified': '2022-07-21T01:38:27.811394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5934}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5934': {'gid': 5934, 'name': 'b', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6833/2524169670', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 2524169670}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 2524169670}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:38:29.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:38:29.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-21T01:38:29.783 INFO:teuthology.orchestra.run.smithi143.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":5934,"want_state":"up:active","state":"up:active","rank_uptime":2.966083421,"mdsmap_epoch":29,"osdmap_epoch":68,"osdmap_epoch_barrier":67,"uptime":7.2995154009999998} 2022-07-21T01:38:29.784 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 5934, "mdsmap_epoch": 29, "osdmap_epoch": 68, "osdmap_epoch_barrier": 67, "rank_uptime": 2.966083421, "state": "up:active", "uptime": 7.299515401, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:38:29.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:38:29.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc17505a700 1 Processor -- start 2022-07-21T01:38:29.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc17505a700 1 -- start start 2022-07-21T01:38:29.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc17505a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc1700b3190 0x7fc1700b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc17505a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b1a80 0x7fc1700656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc17505a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc1700a9da0 0x7fc170065b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc17505a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1700b2980 con 0x7fc1700b3190 2022-07-21T01:38:29.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc17505a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1700b3570 con 0x7fc1700a9da0 2022-07-21T01:38:29.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc17505a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b0e00 con 0x7fc1700b1a80 2022-07-21T01:38:29.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc1700b3190 0x7fc1700b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc1700b3190 0x7fc1700b2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46328/0 (socket says 172.21.15.3:46328) 2022-07-21T01:38:29.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16ffff700 1 -- 172.21.15.3:0/1791097343 learned_addr learned my addr 172.21.15.3:0/1791097343 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:29.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16f7fe700 1 --2- 172.21.15.3:0/1791097343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b1a80 0x7fc1700656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc174859700 1 --2- 172.21.15.3:0/1791097343 >> v2:172.21.15.143:3300/0 conn(0x7fc1700a9da0 0x7fc170065b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16ffff700 1 -- 172.21.15.3:0/1791097343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b1a80 msgr2=0x7fc1700656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:29.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16ffff700 1 --2- 172.21.15.3:0/1791097343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b1a80 0x7fc1700656a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16ffff700 1 -- 172.21.15.3:0/1791097343 >> v2:172.21.15.143:3300/0 conn(0x7fc1700a9da0 msgr2=0x7fc170065b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:29.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16ffff700 1 --2- 172.21.15.3:0/1791097343 >> v2:172.21.15.143:3300/0 conn(0x7fc1700a9da0 0x7fc170065b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16ffff700 1 -- 172.21.15.3:0/1791097343 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700b10a0 con 0x7fc1700b3190 2022-07-21T01:38:29.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc174859700 1 --2- 172.21.15.3:0/1791097343 >> v2:172.21.15.143:3300/0 conn(0x7fc1700a9da0 0x7fc170065b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:29.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16ffff700 1 --2- 172.21.15.3:0/1791097343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b3190 0x7fc1700b2490 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fc160002f70 tx=0x7fc160006eb0).ready entity=mon.0 client_cookie=866db94101862539 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:29.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16effd700 1 -- 172.21.15.3:0/1791097343 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc16000d570 con 0x7fc1700b3190 2022-07-21T01:38:29.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16effd700 1 -- 172.21.15.3:0/1791097343 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc160000e90 con 0x7fc1700b3190 2022-07-21T01:38:29.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.901+0000 7fc16effd700 1 -- 172.21.15.3:0/1791097343 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc16000d900 con 0x7fc1700b3190 2022-07-21T01:38:29.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 -- 172.21.15.3:0/1791097343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b3190 msgr2=0x7fc1700b2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 --2- 172.21.15.3:0/1791097343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b3190 0x7fc1700b2490 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fc160002f70 tx=0x7fc160006eb0).stop 2022-07-21T01:38:29.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 -- 172.21.15.3:0/1791097343 shutdown_connections 2022-07-21T01:38:29.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 --2- 172.21.15.3:0/1791097343 >> v2:172.21.15.143:3300/0 conn(0x7fc1700a9da0 0x7fc170065b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 --2- 172.21.15.3:0/1791097343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b1a80 0x7fc1700656a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 --2- 172.21.15.3:0/1791097343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b3190 0x7fc1700b2490 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 -- 172.21.15.3:0/1791097343 >> 172.21.15.3:0/1791097343 conn(0x7fc170063870 msgr2=0x7fc170063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:29.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 -- 172.21.15.3:0/1791097343 shutdown_connections 2022-07-21T01:38:29.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 -- 172.21.15.3:0/1791097343 wait complete. 2022-07-21T01:38:29.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 Processor -- start 2022-07-21T01:38:29.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 -- start start 2022-07-21T01:38:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1700a9da0 0x7fc1700ab6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b1a80 0x7fc1700b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b3190 0x7fc1701564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1701452d0 con 0x7fc1700b1a80 2022-07-21T01:38:29.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b5b20 con 0x7fc1700a9da0 2022-07-21T01:38:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.905+0000 7fc17505a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700b58c0 con 0x7fc1700b3190 2022-07-21T01:38:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b1a80 0x7fc1700b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1700a9da0 0x7fc1700ab6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b1a80 0x7fc1700b4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46336/0 (socket says 172.21.15.3:46336) 2022-07-21T01:38:29.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16f7fe700 1 -- 172.21.15.3:0/2972395941 learned_addr learned my addr 172.21.15.3:0/2972395941 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:29.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc174859700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b3190 0x7fc1701564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16f7fe700 1 -- 172.21.15.3:0/2972395941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b3190 msgr2=0x7fc1701564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16f7fe700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b3190 0x7fc1701564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16f7fe700 1 -- 172.21.15.3:0/2972395941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1700a9da0 msgr2=0x7fc1700ab6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:29.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16f7fe700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1700a9da0 0x7fc1700ab6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:29.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16f7fe700 1 -- 172.21.15.3:0/2972395941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16000b1e0 con 0x7fc1700b1a80 2022-07-21T01:38:29.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16ffff700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1700a9da0 0x7fc1700ab6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:29.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16f7fe700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b1a80 0x7fc1700b4eb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fc15800c000 tx=0x7fc1580091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16d7fa700 1 -- 172.21.15.3:0/2972395941 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc15800c670 con 0x7fc1700b1a80 2022-07-21T01:38:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16d7fa700 1 -- 172.21.15.3:0/2972395941 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15800c7d0 con 0x7fc1700b1a80 2022-07-21T01:38:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1700aaf90 con 0x7fc1700b1a80 2022-07-21T01:38:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc16d7fa700 1 -- 172.21.15.3:0/2972395941 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc15800cb30 con 0x7fc1700b1a80 2022-07-21T01:38:29.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc170141580 con 0x7fc1700b1a80 2022-07-21T01:38:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.909+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1700574e0 con 0x7fc1700b1a80 2022-07-21T01:38:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.913+0000 7fc16d7fa700 1 -- 172.21.15.3:0/2972395941 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc158018020 con 0x7fc1700b1a80 2022-07-21T01:38:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.913+0000 7fc16d7fa700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1540539e0 0x7fc154055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:29.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.913+0000 7fc16d7fa700 1 -- 172.21.15.3:0/2972395941 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc15806c5e0 con 0x7fc1700b1a80 2022-07-21T01:38:29.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.913+0000 7fc16ffff700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1540539e0 0x7fc154055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:29.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.913+0000 7fc16ffff700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1540539e0 0x7fc154055ea0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fc16001c000 tx=0x7fc160015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:29.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:29.917+0000 7fc16d7fa700 1 -- 172.21.15.3:0/2972395941 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc1580433a0 con 0x7fc1700b1a80 2022-07-21T01:38:30.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.045+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc170069a70 con 0x7fc1700b1a80 2022-07-21T01:38:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc16d7fa700 1 -- 172.21.15.3:0/2972395941 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16274 (secure 0 0 0) 0x7fc1580469c0 con 0x7fc1700b1a80 2022-07-21T01:38:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:30.047 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":68,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:38:27.812077+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:38:25.308080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-21T01:38:26.095494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:38:30.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1540539e0 msgr2=0x7fc154055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1540539e0 0x7fc154055ea0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fc16001c000 tx=0x7fc160015000).stop 2022-07-21T01:38:30.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b1a80 msgr2=0x7fc1700b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b1a80 0x7fc1700b4eb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fc15800c000 tx=0x7fc1580091d0).stop 2022-07-21T01:38:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 shutdown_connections 2022-07-21T01:38:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1700b3190 0x7fc1701564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1700b1a80 0x7fc1700b4eb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc1540539e0 0x7fc154055ea0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 --2- 172.21.15.3:0/2972395941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1700a9da0 0x7fc1700ab6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 >> 172.21.15.3:0/2972395941 conn(0x7fc170063870 msgr2=0x7fc170145570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:30.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 shutdown_connections 2022-07-21T01:38:30.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.049+0000 7fc17505a700 1 -- 172.21.15.3:0/2972395941 wait complete. 2022-07-21T01:38:30.064 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:38:25.308080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:38:26.095494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:38:30.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:38:30.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d77fff700 1 Processor -- start 2022-07-21T01:38:30.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d77fff700 1 -- start start 2022-07-21T01:38:30.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d77fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6d780619b0 0x7f6d780b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d77fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6d780b6ee0 0x7f6d78143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d77fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d78061d90 con 0x7f6d780b5ed0 2022-07-21T01:38:30.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d77fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d78142f60 con 0x7f6d780b6ee0 2022-07-21T01:38:30.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d77fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d780ba5a0 con 0x7f6d780619b0 2022-07-21T01:38:30.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d777fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6d780b5ed0 0x7f6d780b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54626/0 (socket says 172.21.15.3:54626) 2022-07-21T01:38:30.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d777fe700 1 -- 172.21.15.3:0/808329862 learned_addr learned my addr 172.21.15.3:0/808329862 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:30.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d767fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6d780b6ee0 0x7f6d78143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d76ffd700 1 --2- 172.21.15.3:0/808329862 >> v2:172.21.15.143:3300/0 conn(0x7f6d780619b0 0x7f6d780b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d75ffb700 1 -- 172.21.15.3:0/808329862 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65407618 0 0) 0x7f6d78061d90 con 0x7f6d780b5ed0 2022-07-21T01:38:30.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d75ffb700 1 -- 172.21.15.3:0/808329862 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d5c003260 con 0x7f6d780b5ed0 2022-07-21T01:38:30.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d767fc700 1 -- 172.21.15.3:0/808329862 >> v2:172.21.15.143:3300/0 conn(0x7f6d780619b0 msgr2=0x7f6d780b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:30.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d767fc700 1 --2- 172.21.15.3:0/808329862 >> v2:172.21.15.143:3300/0 conn(0x7f6d780619b0 0x7f6d780b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d767fc700 1 -- 172.21.15.3:0/808329862 >> v1:172.21.15.3:6789/0 conn(0x7f6d780b5ed0 legacy=0x7f6d780b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d767fc700 1 -- 172.21.15.3:0/808329862 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78143150 con 0x7f6d780b6ee0 2022-07-21T01:38:30.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d76ffd700 1 --2- 172.21.15.3:0/808329862 >> v2:172.21.15.143:3300/0 conn(0x7f6d780619b0 0x7f6d780b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:30.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d767fc700 1 --2- 172.21.15.3:0/808329862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 0x7f6d78143890 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f6d6c002f70 tx=0x7f6d6c006df0).ready entity=mon.0 client_cookie=a8065883d123dbc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d75ffb700 1 -- 172.21.15.3:0/808329862 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6c00d4f0 con 0x7f6d780b6ee0 2022-07-21T01:38:30.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d75ffb700 1 -- 172.21.15.3:0/808329862 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6c000e90 con 0x7f6d780b6ee0 2022-07-21T01:38:30.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.177+0000 7f6d75ffb700 1 -- 172.21.15.3:0/808329862 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6c00d840 con 0x7f6d780b6ee0 2022-07-21T01:38:30.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 -- 172.21.15.3:0/808329862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 msgr2=0x7f6d78143890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 --2- 172.21.15.3:0/808329862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 0x7f6d78143890 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f6d6c002f70 tx=0x7f6d6c006df0).stop 2022-07-21T01:38:30.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 -- 172.21.15.3:0/808329862 shutdown_connections 2022-07-21T01:38:30.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 --2- 172.21.15.3:0/808329862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 0x7f6d78143890 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 --2- 172.21.15.3:0/808329862 >> v2:172.21.15.143:3300/0 conn(0x7f6d780619b0 0x7f6d780b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 -- 172.21.15.3:0/808329862 >> 172.21.15.3:0/808329862 conn(0x7f6d7806af80 msgr2=0x7f6d7805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:30.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 -- 172.21.15.3:0/808329862 shutdown_connections 2022-07-21T01:38:30.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 -- 172.21.15.3:0/808329862 wait complete. 2022-07-21T01:38:30.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 Processor -- start 2022-07-21T01:38:30.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.181+0000 7f6d77fff700 1 -- start start 2022-07-21T01:38:30.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d77fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d780619b0 0x7f6d780b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d77fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d780b5ed0 0x7f6d7813bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d77fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 0x7f6d781443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d77fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78152270 con 0x7f6d780b6ee0 2022-07-21T01:38:30.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d77fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78152460 con 0x7f6d780b5ed0 2022-07-21T01:38:30.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d77fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78152700 con 0x7f6d780619b0 2022-07-21T01:38:30.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 0x7f6d781443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d76ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d780619b0 0x7f6d780b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 0x7f6d781443f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46352/0 (socket says 172.21.15.3:46352) 2022-07-21T01:38:30.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d76ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d780619b0 0x7f6d780b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:53264/0 (socket says 172.21.15.3:53264) 2022-07-21T01:38:30.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d767fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d780b5ed0 0x7f6d7813bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d76ffd700 1 -- 172.21.15.3:0/130505652 learned_addr learned my addr 172.21.15.3:0/130505652 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:30.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d76ffd700 1 -- 172.21.15.3:0/130505652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d780b5ed0 msgr2=0x7f6d7813bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d76ffd700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d780b5ed0 0x7f6d7813bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d76ffd700 1 -- 172.21.15.3:0/130505652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 msgr2=0x7f6d781443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d76ffd700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 0x7f6d781443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d76ffd700 1 -- 172.21.15.3:0/130505652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6800b180 con 0x7f6d780619b0 2022-07-21T01:38:30.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d76ffd700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d780619b0 0x7f6d780b81f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f6d6800c000 tx=0x7f6d68006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d67fff700 1 -- 172.21.15.3:0/130505652 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6800c7e0 con 0x7f6d780619b0 2022-07-21T01:38:30.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d67fff700 1 -- 172.21.15.3:0/130505652 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6800c940 con 0x7f6d780619b0 2022-07-21T01:38:30.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6c00b180 con 0x7f6d780619b0 2022-07-21T01:38:30.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d67fff700 1 -- 172.21.15.3:0/130505652 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6800cca0 con 0x7f6d780619b0 2022-07-21T01:38:30.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d78152c40 con 0x7f6d780619b0 2022-07-21T01:38:30.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d780574e0 con 0x7f6d780619b0 2022-07-21T01:38:30.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.185+0000 7f6d67fff700 1 -- 172.21.15.3:0/130505652 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6d6801c460 con 0x7f6d780619b0 2022-07-21T01:38:30.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.189+0000 7f6d67fff700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d54053b00 0x7f6d54055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.189+0000 7f6d67fff700 1 -- 172.21.15.3:0/130505652 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6d6806c6c0 con 0x7f6d780619b0 2022-07-21T01:38:30.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.189+0000 7f6d767fc700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d54053b00 0x7f6d54055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.189+0000 7f6d767fc700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d54053b00 0x7f6d54055fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f6d6c01a000 tx=0x7f6d6c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.197+0000 7f6d67fff700 1 -- 172.21.15.3:0/130505652 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d68043310 con 0x7f6d780619b0 2022-07-21T01:38:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.321+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f6d78152840 con 0x7f6d780619b0 2022-07-21T01:38:30.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.321+0000 7f6d67fff700 1 -- 172.21.15.3:0/130505652 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v68) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6d6800ce00 con 0x7f6d780619b0 2022-07-21T01:38:30.322 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:38:30.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d54053b00 msgr2=0x7f6d54055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d54053b00 0x7f6d54055fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f6d6c01a000 tx=0x7f6d6c014000).stop 2022-07-21T01:38:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d780619b0 msgr2=0x7f6d780b81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d780619b0 0x7f6d780b81f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f6d6800c000 tx=0x7f6d68006df0).stop 2022-07-21T01:38:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 shutdown_connections 2022-07-21T01:38:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d780b6ee0 0x7f6d781443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6d54053b00 0x7f6d54055fc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d780b5ed0 0x7f6d7813bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 --2- 172.21.15.3:0/130505652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d780619b0 0x7f6d780b81f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 >> 172.21.15.3:0/130505652 conn(0x7f6d7806af80 msgr2=0x7f6d7805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 shutdown_connections 2022-07-21T01:38:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.325+0000 7f6d77fff700 1 -- 172.21.15.3:0/130505652 wait complete. 2022-07-21T01:38:30.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:38:30.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.441+0000 7f2af3fff700 1 Processor -- start 2022-07-21T01:38:30.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.441+0000 7f2af3fff700 1 -- start start 2022-07-21T01:38:30.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2af40b31f0 0x7f2af40b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2af40aadb0 0x7f2af4064d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af40b2510 con 0x7f2af40b7500 2022-07-21T01:38:30.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af41443e0 con 0x7f2af40aadb0 2022-07-21T01:38:30.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af40b3700 con 0x7f2af40b31f0 2022-07-21T01:38:30.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af2ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2af40b7500 0x7f2af413e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54636/0 (socket says 172.21.15.3:54636) 2022-07-21T01:38:30.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af2ffd700 1 -- 172.21.15.3:0/4231100913 learned_addr learned my addr 172.21.15.3:0/4231100913 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:30.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af37fe700 1 --2- 172.21.15.3:0/4231100913 >> v2:172.21.15.3:3300/0 conn(0x7f2af40aadb0 0x7f2af4064d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af27fc700 1 --2- 172.21.15.3:0/4231100913 >> v2:172.21.15.143:3300/0 conn(0x7f2af40b31f0 0x7f2af40b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af37fe700 1 -- 172.21.15.3:0/4231100913 >> v2:172.21.15.143:3300/0 conn(0x7f2af40b31f0 msgr2=0x7f2af40b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:30.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af37fe700 1 --2- 172.21.15.3:0/4231100913 >> v2:172.21.15.143:3300/0 conn(0x7f2af40b31f0 0x7f2af40b78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af37fe700 1 -- 172.21.15.3:0/4231100913 >> v1:172.21.15.3:6789/0 conn(0x7f2af40b7500 legacy=0x7f2af413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af1ffb700 1 -- 172.21.15.3:0/4231100913 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632931039 0 0) 0x7f2af40b2510 con 0x7f2af40b7500 2022-07-21T01:38:30.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af37fe700 1 -- 172.21.15.3:0/4231100913 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af40b2740 con 0x7f2af40aadb0 2022-07-21T01:38:30.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af27fc700 1 --2- 172.21.15.3:0/4231100913 >> v2:172.21.15.143:3300/0 conn(0x7f2af40b31f0 0x7f2af40b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:30.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af37fe700 1 --2- 172.21.15.3:0/4231100913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40aadb0 0x7f2af4064d30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f2aec0032f0 tx=0x7f2aec0091d0).ready entity=mon.0 client_cookie=c594e86fcfff99e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af1ffb700 1 -- 172.21.15.3:0/4231100913 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2aec00b680 con 0x7f2af40aadb0 2022-07-21T01:38:30.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af1ffb700 1 -- 172.21.15.3:0/4231100913 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec00b7e0 con 0x7f2af40aadb0 2022-07-21T01:38:30.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af1ffb700 1 -- 172.21.15.3:0/4231100913 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2aec00b950 con 0x7f2af40aadb0 2022-07-21T01:38:30.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 -- 172.21.15.3:0/4231100913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40aadb0 msgr2=0x7f2af4064d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 --2- 172.21.15.3:0/4231100913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40aadb0 0x7f2af4064d30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f2aec0032f0 tx=0x7f2aec0091d0).stop 2022-07-21T01:38:30.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 -- 172.21.15.3:0/4231100913 shutdown_connections 2022-07-21T01:38:30.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 --2- 172.21.15.3:0/4231100913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40aadb0 0x7f2af4064d30 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 --2- 172.21.15.3:0/4231100913 >> v2:172.21.15.143:3300/0 conn(0x7f2af40b31f0 0x7f2af40b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 -- 172.21.15.3:0/4231100913 >> 172.21.15.3:0/4231100913 conn(0x7f2af405f840 msgr2=0x7f2af405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:30.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 -- 172.21.15.3:0/4231100913 shutdown_connections 2022-07-21T01:38:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.445+0000 7f2af3fff700 1 -- 172.21.15.3:0/4231100913 wait complete. 2022-07-21T01:38:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 Processor -- start 2022-07-21T01:38:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 -- start start 2022-07-21T01:38:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2af40aadb0 0x7f2af413fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2af40b31f0 0x7f2af40ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40b7500 0x7f2af40b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af40acb60 con 0x7f2af40b7500 2022-07-21T01:38:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af40ad1f0 con 0x7f2af40b31f0 2022-07-21T01:38:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af40abe20 con 0x7f2af40aadb0 2022-07-21T01:38:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40b7500 0x7f2af40b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40b7500 0x7f2af40b1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46366/0 (socket says 172.21.15.3:46366) 2022-07-21T01:38:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af37fe700 1 -- 172.21.15.3:0/3306696154 learned_addr learned my addr 172.21.15.3:0/3306696154 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af27fc700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2af40b31f0 0x7f2af40ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af2ffd700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2af40aadb0 0x7f2af413fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af37fe700 1 -- 172.21.15.3:0/3306696154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2af40aadb0 msgr2=0x7f2af413fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af37fe700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2af40aadb0 0x7f2af413fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af37fe700 1 -- 172.21.15.3:0/3306696154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2af40b31f0 msgr2=0x7f2af40ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af37fe700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2af40b31f0 0x7f2af40ad3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af37fe700 1 -- 172.21.15.3:0/3306696154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc00b180 con 0x7f2af40b7500 2022-07-21T01:38:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af37fe700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40b7500 0x7f2af40b1670 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f2af41501e0 tx=0x7f2aec016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2aebfff700 1 -- 172.21.15.3:0/3306696154 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2aec00b680 con 0x7f2af40b7500 2022-07-21T01:38:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec009470 con 0x7f2af40b7500 2022-07-21T01:38:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2aebfff700 1 -- 172.21.15.3:0/3306696154 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec021630 con 0x7f2af40b7500 2022-07-21T01:38:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2aebfff700 1 -- 172.21.15.3:0/3306696154 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2aec021990 con 0x7f2af40b7500 2022-07-21T01:38:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.449+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af40ac2f0 con 0x7f2af40b7500 2022-07-21T01:38:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.453+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af40acef0 con 0x7f2af40b7500 2022-07-21T01:38:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.453+0000 7f2aebfff700 1 -- 172.21.15.3:0/3306696154 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2aec027020 con 0x7f2af40b7500 2022-07-21T01:38:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.453+0000 7f2aebfff700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ad8053a70 0x7f2ad8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.453+0000 7f2aebfff700 1 -- 172.21.15.3:0/3306696154 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2aec06d470 con 0x7f2af40b7500 2022-07-21T01:38:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.453+0000 7f2af2ffd700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ad8053a70 0x7f2ad8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.453+0000 7f2af2ffd700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ad8053a70 0x7f2ad8055f30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f2ae400d000 tx=0x7f2ae4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.461+0000 7f2aebfff700 1 -- 172.21.15.3:0/3306696154 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2aec044190 con 0x7f2af40b7500 2022-07-21T01:38:30.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.585+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2af40574e0 con 0x7f2af40b7500 2022-07-21T01:38:30.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2aebfff700 1 -- 172.21.15.3:0/3306696154 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v68) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2aec0150a0 con 0x7f2af40b7500 2022-07-21T01:38:30.588 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:38:30.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ad8053a70 msgr2=0x7f2ad8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ad8053a70 0x7f2ad8055f30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f2ae400d000 tx=0x7f2ae4009040).stop 2022-07-21T01:38:30.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40b7500 msgr2=0x7f2af40b1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40b7500 0x7f2af40b1670 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f2af41501e0 tx=0x7f2aec016000).stop 2022-07-21T01:38:30.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 shutdown_connections 2022-07-21T01:38:30.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2af40b7500 0x7f2af40b1670 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ad8053a70 0x7f2ad8055f30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2af40b31f0 0x7f2af40ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 --2- 172.21.15.3:0/3306696154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2af40aadb0 0x7f2af413fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 >> 172.21.15.3:0/3306696154 conn(0x7f2af405f840 msgr2=0x7f2af40b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:30.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 shutdown_connections 2022-07-21T01:38:30.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.589+0000 7f2af3fff700 1 -- 172.21.15.3:0/3306696154 wait complete. 2022-07-21T01:38:30.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:38:30.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 Processor -- start 2022-07-21T01:38:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 -- start start 2022-07-21T01:38:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f36ec0b6ee0 0x7f36ec0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec061d90 con 0x7f36ec0b5ed0 2022-07-21T01:38:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec0ba5a0 con 0x7f36ec0619b0 2022-07-21T01:38:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36ec143fa0 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2296700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f36ec0b5ed0 0x7f36ec0b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54652/0 (socket says 172.21.15.3:54652) 2022-07-21T01:38:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2296700 1 -- 172.21.15.3:0/355091928 learned_addr learned my addr 172.21.15.3:0/355091928 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f1294700 1 --2- 172.21.15.3:0/355091928 >> v2:172.21.15.3:3300/0 conn(0x7f36ec0b6ee0 0x7f36ec0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f0a93700 1 -- 172.21.15.3:0/355091928 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143341802 0 0) 0x7f36ec061d90 con 0x7f36ec0b5ed0 2022-07-21T01:38:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f0a93700 1 -- 172.21.15.3:0/355091928 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d4003260 con 0x7f36ec0b5ed0 2022-07-21T01:38:30.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f0a93700 1 -- 172.21.15.3:0/355091928 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369942443 0 0) 0x7f36ec0ba5a0 con 0x7f36ec0619b0 2022-07-21T01:38:30.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f0a93700 1 -- 172.21.15.3:0/355091928 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ec061d90 con 0x7f36ec0619b0 2022-07-21T01:38:30.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f0a93700 1 -- 172.21.15.3:0/355091928 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2642506073 0 0) 0x7f36d4003260 con 0x7f36ec0b5ed0 2022-07-21T01:38:30.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f1294700 1 -- 172.21.15.3:0/355091928 >> v1:172.21.15.143:6789/0 conn(0x7f36ec0619b0 legacy=0x7f36ec143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f1294700 1 -- 172.21.15.3:0/355091928 >> v1:172.21.15.3:6789/0 conn(0x7f36ec0b5ed0 legacy=0x7f36ec0b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f1294700 1 -- 172.21.15.3:0/355091928 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec143150 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f1294700 1 --2- 172.21.15.3:0/355091928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 0x7f36ec0b48a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f36e4001490 tx=0x7f36dc006df0).ready entity=mon.0 client_cookie=6c42503cba3bde47 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f0a93700 1 -- 172.21.15.3:0/355091928 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36e4001a00 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f0a93700 1 -- 172.21.15.3:0/355091928 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc000e90 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f0a93700 1 -- 172.21.15.3:0/355091928 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36dc012460 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 -- 172.21.15.3:0/355091928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 msgr2=0x7f36ec0b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 --2- 172.21.15.3:0/355091928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 0x7f36ec0b48a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f36e4001490 tx=0x7f36dc006df0).stop 2022-07-21T01:38:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 -- 172.21.15.3:0/355091928 shutdown_connections 2022-07-21T01:38:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 --2- 172.21.15.3:0/355091928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 0x7f36ec0b48a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.713+0000 7f36f2a97700 1 -- 172.21.15.3:0/355091928 >> 172.21.15.3:0/355091928 conn(0x7f36ec06af80 msgr2=0x7f36ec05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 -- 172.21.15.3:0/355091928 shutdown_connections 2022-07-21T01:38:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 -- 172.21.15.3:0/355091928 wait complete. 2022-07-21T01:38:30.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 Processor -- start 2022-07-21T01:38:30.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 -- start start 2022-07-21T01:38:30.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36ec0619b0 0x7f36ec146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36ec0b5ed0 0x7f36ec114940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 0x7f36ec114e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec1525a0 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec145f50 con 0x7f36ec0619b0 2022-07-21T01:38:30.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec0af300 con 0x7f36ec0b5ed0 2022-07-21T01:38:30.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2296700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 0x7f36ec114e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2296700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 0x7f36ec114e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46380/0 (socket says 172.21.15.3:46380) 2022-07-21T01:38:30.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2296700 1 -- 172.21.15.3:0/330894804 learned_addr learned my addr 172.21.15.3:0/330894804 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:30.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2296700 1 -- 172.21.15.3:0/330894804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36ec0b5ed0 msgr2=0x7f36ec114940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2296700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36ec0b5ed0 0x7f36ec114940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:38:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2296700 1 -- 172.21.15.3:0/330894804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36ec0619b0 msgr2=0x7f36ec146fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2296700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36ec0619b0 0x7f36ec146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:38:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2296700 1 -- 172.21.15.3:0/330894804 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc00b180 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2296700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 0x7f36ec114e30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f36e4001be0 tx=0x7f36e4002140).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36daffd700 1 -- 172.21.15.3:0/330894804 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36e4004ab0 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36daffd700 1 -- 172.21.15.3:0/330894804 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e4004c10 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec145cb0 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.717+0000 7f36daffd700 1 -- 172.21.15.3:0/330894804 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36e4007170 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.721+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec115320 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.721+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ec0af760 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.721+0000 7f36daffd700 1 -- 172.21.15.3:0/330894804 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f36e4010070 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.721+0000 7f36daffd700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36cc0539e0 0x7f36cc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.721+0000 7f36daffd700 1 -- 172.21.15.3:0/330894804 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f36e406bf40 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.721+0000 7f36f1a95700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36cc0539e0 0x7f36cc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.721+0000 7f36f1a95700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36cc0539e0 0x7f36cc055ea0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f36e8001110 tx=0x7f36e80076a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.729+0000 7f36daffd700 1 -- 172.21.15.3:0/330894804 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f36ec0af760 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.853+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f36ec0af760 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.853+0000 7f36daffd700 1 -- 172.21.15.3:0/330894804 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v68) v1 ==== 104+0+10 (secure 0 0 0) 0x7f36ec0af760 con 0x7f36ec0b6ee0 2022-07-21T01:38:30.854 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:38:30.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36cc0539e0 msgr2=0x7f36cc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36cc0539e0 0x7f36cc055ea0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f36e8001110 tx=0x7f36e80076a0).stop 2022-07-21T01:38:30.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 msgr2=0x7f36ec114e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 0x7f36ec114e30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f36e4001be0 tx=0x7f36e4002140).stop 2022-07-21T01:38:30.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 shutdown_connections 2022-07-21T01:38:30.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36ec0b6ee0 0x7f36ec114e30 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36ec0b5ed0 0x7f36ec114940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:38:30.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36cc0539e0 0x7f36cc055ea0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 --2- 172.21.15.3:0/330894804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36ec0619b0 0x7f36ec146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:38:30.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 >> 172.21.15.3:0/330894804 conn(0x7f36ec06af80 msgr2=0x7f36ec13f580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:30.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 shutdown_connections 2022-07-21T01:38:30.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.857+0000 7f36f2a97700 1 -- 172.21.15.3:0/330894804 wait complete. 2022-07-21T01:38:30.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:38:30.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e3897700 1 Processor -- start 2022-07-21T01:38:30.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e3897700 1 -- start start 2022-07-21T01:38:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e3897700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc0b1b10 0x7fb5dc0b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e3897700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5dc061860 0x7fb5dc060d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e3897700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5dc139e30 0x7fb5dc13a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e3897700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5dc140220 con 0x7fb5dc061860 2022-07-21T01:38:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e3897700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5dc1404c0 con 0x7fb5dc139e30 2022-07-21T01:38:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e3897700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc140760 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e0e32700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5dc061860 0x7fb5dc060d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e0e32700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5dc061860 0x7fb5dc060d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46386/0 (socket says 172.21.15.3:46386) 2022-07-21T01:38:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e0e32700 1 -- 172.21.15.3:0/1238618845 learned_addr learned my addr 172.21.15.3:0/1238618845 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e1e34700 1 --2- 172.21.15.3:0/1238618845 >> v2:172.21.15.143:3300/0 conn(0x7fb5dc139e30 0x7fb5dc13a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e1633700 1 --2- 172.21.15.3:0/1238618845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc0b1b10 0x7fb5dc0b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e0e32700 1 -- 172.21.15.3:0/1238618845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc0b1b10 msgr2=0x7fb5dc0b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e0e32700 1 --2- 172.21.15.3:0/1238618845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc0b1b10 0x7fb5dc0b1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e0e32700 1 -- 172.21.15.3:0/1238618845 >> v2:172.21.15.143:3300/0 conn(0x7fb5dc139e30 msgr2=0x7fb5dc13a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e0e32700 1 --2- 172.21.15.3:0/1238618845 >> v2:172.21.15.143:3300/0 conn(0x7fb5dc139e30 0x7fb5dc13a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e0e32700 1 -- 172.21.15.3:0/1238618845 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc140a00 con 0x7fb5dc061860 2022-07-21T01:38:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.977+0000 7fb5e0e32700 1 --2- 172.21.15.3:0/1238618845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc061860 0x7fb5dc060d00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fb5cc002f70 tx=0x7fb5cc006eb0).ready entity=mon.0 client_cookie=2f858abb9298c0be server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5d3fff700 1 -- 172.21.15.3:0/1238618845 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5cc00d570 con 0x7fb5dc061860 2022-07-21T01:38:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5d3fff700 1 -- 172.21.15.3:0/1238618845 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5cc000e90 con 0x7fb5dc061860 2022-07-21T01:38:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5d3fff700 1 -- 172.21.15.3:0/1238618845 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5cc00d900 con 0x7fb5dc061860 2022-07-21T01:38:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 -- 172.21.15.3:0/1238618845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc061860 msgr2=0x7fb5dc060d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 --2- 172.21.15.3:0/1238618845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc061860 0x7fb5dc060d00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fb5cc002f70 tx=0x7fb5cc006eb0).stop 2022-07-21T01:38:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 -- 172.21.15.3:0/1238618845 shutdown_connections 2022-07-21T01:38:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 --2- 172.21.15.3:0/1238618845 >> v2:172.21.15.143:3300/0 conn(0x7fb5dc139e30 0x7fb5dc13a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 --2- 172.21.15.3:0/1238618845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc061860 0x7fb5dc060d00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 --2- 172.21.15.3:0/1238618845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc0b1b10 0x7fb5dc0b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 -- 172.21.15.3:0/1238618845 >> 172.21.15.3:0/1238618845 conn(0x7fb5dc0b2b00 msgr2=0x7fb5dc0b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 -- 172.21.15.3:0/1238618845 shutdown_connections 2022-07-21T01:38:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 -- 172.21.15.3:0/1238618845 wait complete. 2022-07-21T01:38:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 Processor -- start 2022-07-21T01:38:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 -- start start 2022-07-21T01:38:30.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc061860 0x7fb5dc1417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc0b1b10 0x7fb5dc155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5dc139e30 0x7fb5dc156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc13d5e0 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc0b3c20 con 0x7fb5dc139e30 2022-07-21T01:38:30.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e3897700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc13d810 con 0x7fb5dc061860 2022-07-21T01:38:30.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e0e32700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc0b1b10 0x7fb5dc155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e0e32700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc0b1b10 0x7fb5dc155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46390/0 (socket says 172.21.15.3:46390) 2022-07-21T01:38:30.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e0e32700 1 -- 172.21.15.3:0/2403002187 learned_addr learned my addr 172.21.15.3:0/2403002187 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:30.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e1633700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc061860 0x7fb5dc1417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e1e34700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5dc139e30 0x7fb5dc156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e0e32700 1 -- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc061860 msgr2=0x7fb5dc1417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e0e32700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc061860 0x7fb5dc1417e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e0e32700 1 -- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5dc139e30 msgr2=0x7fb5dc156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:30.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e0e32700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5dc139e30 0x7fb5dc156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:30.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e0e32700 1 -- 172.21.15.3:0/2403002187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc00b1e0 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e1e34700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5dc139e30 0x7fb5dc156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:30.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e1633700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc061860 0x7fb5dc1417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:30.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5e0e32700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc0b1b10 0x7fb5dc155e40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fb5cc002f70 tx=0x7fb5cc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.981+0000 7fb5d27fc700 1 -- 172.21.15.3:0/2403002187 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5cc011950 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5d27fc700 1 -- 172.21.15.3:0/2403002187 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5cc011ab0 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc0b3e80 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5d27fc700 1 -- 172.21.15.3:0/2403002187 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5cc011e10 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc13afa0 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5dc0ad110 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5d27fc700 1 -- 172.21.15.3:0/2403002187 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb5cc01a040 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5d27fc700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5c4053950 0x7fb5c4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:30.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5d27fc700 1 -- 172.21.15.3:0/2403002187 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb5cc06cff0 con 0x7fb5dc0b1b10 2022-07-21T01:38:30.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5e1633700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5c4053950 0x7fb5c4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:30.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.985+0000 7fb5e1633700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5c4053950 0x7fb5c4055e10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fb5d800b000 tx=0x7fb5d8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:30.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:30.993+0000 7fb5d27fc700 1 -- 172.21.15.3:0/2403002187 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5cc043c60 con 0x7fb5dc0b1b10 2022-07-21T01:38:31.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb5dc0574e0 con 0x7fb5dc0b1b10 2022-07-21T01:38:31.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5d27fc700 1 -- 172.21.15.3:0/2403002187 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v68) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb5cc023020 con 0x7fb5dc0b1b10 2022-07-21T01:38:31.120 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:38:31.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5c4053950 msgr2=0x7fb5c4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:31.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5c4053950 0x7fb5c4055e10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fb5d800b000 tx=0x7fb5d8008040).stop 2022-07-21T01:38:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc0b1b10 msgr2=0x7fb5dc155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc0b1b10 0x7fb5dc155e40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fb5cc002f70 tx=0x7fb5cc015000).stop 2022-07-21T01:38:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 shutdown_connections 2022-07-21T01:38:31.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb5c4053950 0x7fb5c4055e10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5dc139e30 0x7fb5dc156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5dc0b1b10 0x7fb5dc155e40 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 --2- 172.21.15.3:0/2403002187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5dc061860 0x7fb5dc1417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 >> 172.21.15.3:0/2403002187 conn(0x7fb5dc0b2b00 msgr2=0x7fb5dc13f270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:31.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 shutdown_connections 2022-07-21T01:38:31.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.121+0000 7fb5e3897700 1 -- 172.21.15.3:0/2403002187 wait complete. 2022-07-21T01:38:31.132 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:38:31.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 Processor -- start 2022-07-21T01:38:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 -- start start 2022-07-21T01:38:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbc780b3260 0x7fbc781435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc78143ab0 con 0x7fbc780b08a0 2022-07-21T01:38:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc780b0c80 con 0x7fbc780ad000 2022-07-21T01:38:31.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc780ae810 con 0x7fbc780b3260 2022-07-21T01:38:31.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbc780b08a0 0x7fbc78142eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54678/0 (socket says 172.21.15.3:54678) 2022-07-21T01:38:31.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7659c700 1 -- 172.21.15.3:0/1944377205 learned_addr learned my addr 172.21.15.3:0/1944377205 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:31.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc75d9b700 1 --2- 172.21.15.3:0/1944377205 >> v2:172.21.15.3:3300/0 conn(0x7fbc780b3260 0x7fbc781435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:31.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7559a700 1 -- 172.21.15.3:0/1944377205 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262709819 0 0) 0x7fbc78143ab0 con 0x7fbc780b08a0 2022-07-21T01:38:31.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7559a700 1 -- 172.21.15.3:0/1944377205 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc5c003260 con 0x7fbc780b08a0 2022-07-21T01:38:31.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7559a700 1 -- 172.21.15.3:0/1944377205 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1250871084 0 0) 0x7fbc5c003260 con 0x7fbc780b08a0 2022-07-21T01:38:31.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc75d9b700 1 -- 172.21.15.3:0/1944377205 >> v1:172.21.15.143:6789/0 conn(0x7fbc780ad000 legacy=0x7fbc7813fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:31.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc75d9b700 1 -- 172.21.15.3:0/1944377205 >> v1:172.21.15.3:6789/0 conn(0x7fbc780b08a0 legacy=0x7fbc78142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:31.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc75d9b700 1 -- 172.21.15.3:0/1944377205 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc780aeab0 con 0x7fbc780b3260 2022-07-21T01:38:31.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc75d9b700 1 --2- 172.21.15.3:0/1944377205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 0x7fbc781435c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fbc68001800 tx=0x7fbc6c006df0).ready entity=mon.0 client_cookie=5da247851b83a57 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:31.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7559a700 1 -- 172.21.15.3:0/1944377205 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc68001660 con 0x7fbc780b3260 2022-07-21T01:38:31.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7559a700 1 -- 172.21.15.3:0/1944377205 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c000e90 con 0x7fbc780b3260 2022-07-21T01:38:31.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7559a700 1 -- 172.21.15.3:0/1944377205 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc6c011400 con 0x7fbc780b3260 2022-07-21T01:38:31.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 -- 172.21.15.3:0/1944377205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 msgr2=0x7fbc781435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 --2- 172.21.15.3:0/1944377205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 0x7fbc781435c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fbc68001800 tx=0x7fbc6c006df0).stop 2022-07-21T01:38:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 -- 172.21.15.3:0/1944377205 shutdown_connections 2022-07-21T01:38:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 --2- 172.21.15.3:0/1944377205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 0x7fbc781435c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 -- 172.21.15.3:0/1944377205 >> 172.21.15.3:0/1944377205 conn(0x7fbc780612d0 msgr2=0x7fbc78062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 -- 172.21.15.3:0/1944377205 shutdown_connections 2022-07-21T01:38:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.241+0000 7fbc7759e700 1 -- 172.21.15.3:0/1944377205 wait complete. 2022-07-21T01:38:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7759e700 1 Processor -- start 2022-07-21T01:38:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7759e700 1 -- start start 2022-07-21T01:38:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc780ad000 0x7fbc78156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc780b08a0 0x7fbc780b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 0x7fbc78066950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:31.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78064800 con 0x7fbc780b3260 2022-07-21T01:38:31.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78156710 con 0x7fbc780b08a0 2022-07-21T01:38:31.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78064cb0 con 0x7fbc780ad000 2022-07-21T01:38:31.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc76d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 0x7fbc78066950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:31.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc76d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 0x7fbc78066950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46408/0 (socket says 172.21.15.3:46408) 2022-07-21T01:38:31.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc76d9d700 1 -- 172.21.15.3:0/254741681 learned_addr learned my addr 172.21.15.3:0/254741681 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:31.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7659c700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc780ad000 0x7fbc78156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:31.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc75d9b700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc780b08a0 0x7fbc780b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:31.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc76d9d700 1 -- 172.21.15.3:0/254741681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc780ad000 msgr2=0x7fbc78156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:31.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc76d9d700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc780ad000 0x7fbc78156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc76d9d700 1 -- 172.21.15.3:0/254741681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc780b08a0 msgr2=0x7fbc780b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc76d9d700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc780b08a0 0x7fbc780b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc76d9d700 1 -- 172.21.15.3:0/254741681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c00b180 con 0x7fbc780b3260 2022-07-21T01:38:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc7659c700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc780ad000 0x7fbc78156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.245+0000 7fbc76d9d700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 0x7fbc78066950 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7fbc780a2f30 tx=0x7fbc70007eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc677fe700 1 -- 172.21.15.3:0/254741681 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc70010510 con 0x7fbc780b3260 2022-07-21T01:38:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc677fe700 1 -- 172.21.15.3:0/254741681 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc70010670 con 0x7fbc780b3260 2022-07-21T01:38:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc78156c10 con 0x7fbc780b3260 2022-07-21T01:38:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc677fe700 1 -- 172.21.15.3:0/254741681 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc700107e0 con 0x7fbc780b3260 2022-07-21T01:38:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc78156970 con 0x7fbc780b3260 2022-07-21T01:38:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc780574e0 con 0x7fbc780b3260 2022-07-21T01:38:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc677fe700 1 -- 172.21.15.3:0/254741681 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbc70010070 con 0x7fbc780b3260 2022-07-21T01:38:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc677fe700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc54053a10 0x7fbc54055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc7659c700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc54053a10 0x7fbc54055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.249+0000 7fbc677fe700 1 -- 172.21.15.3:0/254741681 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fbc7006bef0 con 0x7fbc780b3260 2022-07-21T01:38:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.253+0000 7fbc7659c700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc54053a10 0x7fbc54055ed0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fbc6800b000 tx=0x7fbc68000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.257+0000 7fbc677fe700 1 -- 172.21.15.3:0/254741681 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc70042ba0 con 0x7fbc780b3260 2022-07-21T01:38:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.421+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc780b2e00 con 0x7fbc780b3260 2022-07-21T01:38:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.421+0000 7fbc677fe700 1 -- 172.21.15.3:0/254741681 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2490 (secure 0 0 0) 0x7fbc700461c0 con 0x7fbc780b3260 2022-07-21T01:38:31.421 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:31.421 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":29,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:38:26.808059+0000","modified":"2022-07-21T01:38:27.811394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/2524169670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2524169670},{"type":"v1","addr":"172.21.15.143:6833","nonce":2524169670}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-21T01:38:31.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc54053a10 msgr2=0x7fbc54055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:31.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc54053a10 0x7fbc54055ed0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fbc6800b000 tx=0x7fbc68000bc0).stop 2022-07-21T01:38:31.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 msgr2=0x7fbc78066950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:31.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 0x7fbc78066950 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7fbc780a2f30 tx=0x7fbc70007eb0).stop 2022-07-21T01:38:31.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 shutdown_connections 2022-07-21T01:38:31.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc780b3260 0x7fbc78066950 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbc54053a10 0x7fbc54055ed0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc780b08a0 0x7fbc780b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 --2- 172.21.15.3:0/254741681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc780ad000 0x7fbc78156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:31.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 >> 172.21.15.3:0/254741681 conn(0x7fbc780612d0 msgr2=0x7fbc78142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:31.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 shutdown_connections 2022-07-21T01:38:31.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:31.425+0000 7fbc7759e700 1 -- 172.21.15.3:0/254741681 wait complete. 2022-07-21T01:38:31.426 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 29 2022-07-21T01:38:31.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:38:26.808059+0000', 'modified': '2022-07-21T01:38:27.811394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5934}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5934': {'gid': 5934, 'name': 'b', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6833/2524169670', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 2524169670}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 2524169670}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:38:31.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:38:31.738 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-21T01:38:31.854 INFO:teuthology.orchestra.run.smithi143.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":5934,"want_state":"up:active","state":"up:active","rank_uptime":5.036171854,"mdsmap_epoch":29,"osdmap_epoch":68,"osdmap_epoch_barrier":67,"uptime":9.3695952780000002} 2022-07-21T01:38:31.855 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 5934, "mdsmap_epoch": 29, "osdmap_epoch": 68, "osdmap_epoch_barrier": 67, "rank_uptime": 5.036171854, "state": "up:active", "uptime": 9.369595278, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:38:31.855 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:38:31.856 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:38:31.857 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:38:31.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:31.858 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:31.867 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:38:31.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:38:31.885 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:38:31.886 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:38:31.888 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:38:31.892 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:38:31.931 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:38:31.941 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:38:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:38:31.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:31.945 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:38:31.953 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:38:31.954 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:32.020 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:38:32.037 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:38:32.037 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:38:32.046 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:38:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:38:32.048 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:32.048 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:38:32.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:38:32.097+0000 7f7f1f97c080 -1 init, newargv = 0x561da08888b0 newargc=9 2022-07-21T01:38:32.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22678]: starting ceph client 2022-07-21T01:38:32.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22678]: starting fuse 2022-07-21T01:38:33.154 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:38:33.175 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:38:33.176 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:38:33.234 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:38:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:38:33.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:33.238 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:38:33.246 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:38:33.247 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:38:33.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:33.249 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:38:33.249 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:38:33.250 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:38:33.250 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:38:33.251 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:38:33.251 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:38:33.252 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:38:33.252 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:38:33.253 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:38:33.253 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:38:33.254 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:38:33.254 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:38:33.255 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:38:33.255 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:38:33.256 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:38:33.256 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:38:33.257 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:38:33.257 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:38:33.258 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:38:33.258 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:38:33.259 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:38:33.259 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:38:33.260 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:38:33.306 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22678.asok 2022-07-21T01:38:33.311 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22678.asok 2022-07-21T01:38:33.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:33.312 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22678.asok status 2022-07-21T01:38:33.451 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:38:33.451 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:38:33.451 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:38:33.452 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:38:33.452 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:38:33.452 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:38:33.452 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:38:33.453 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22678", 2022-07-21T01:38:33.453 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:38:33.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:38:33.453 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:38:33.454 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:38:33.454 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 6106, 2022-07-21T01:38:33.454 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:38:33.454 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:38:33.455 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:38:33.455 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 6106 2022-07-21T01:38:33.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:38:33.455 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:38:33.455 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:38:33.456 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:38:33.456 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 184545098 2022-07-21T01:38:33.456 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:38:33.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:38:33.457 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:38:33.457 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:38:33.457 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:38:33.458 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 184545098 2022-07-21T01:38:33.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:38:33.458 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.6106 172.21.15.171:0/184545098", 2022-07-21T01:38:33.458 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/184545098", 2022-07-21T01:38:33.459 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:38:33.459 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 29, 2022-07-21T01:38:33.459 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 68, 2022-07-21T01:38:33.460 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:38:33.460 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:38:33.460 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:38:33.460 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:38:33.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:33.462 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:33.463 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:33.471 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:38:33.472 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:38:33.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:33.473 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:33.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:33.492 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:33.493 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:33.500 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:38:33.502 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:38:33.502 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-21T01:38:33.503 DEBUG:teuthology.orchestra.run.smithi171:> import sys 2022-07-21T01:38:33.503 DEBUG:teuthology.orchestra.run.smithi171:> import time 2022-07-21T01:38:33.504 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:38:33.504 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:38:33.505 DEBUG:teuthology.orchestra.run.smithi171:> n = 1000 2022-07-21T01:38:33.506 DEBUG:teuthology.orchestra.run.smithi171:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-21T01:38:33.506 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:38:33.507 DEBUG:teuthology.orchestra.run.smithi171:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-21T01:38:33.507 DEBUG:teuthology.orchestra.run.smithi171:> os.makedirs(os.path.dirname(abs_path)) 2022-07-21T01:38:33.508 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:38:33.508 DEBUG:teuthology.orchestra.run.smithi171:> for i in range(0, n): 2022-07-21T01:38:33.508 DEBUG:teuthology.orchestra.run.smithi171:> fname = "{0}_{1}".format(abs_path, i) 2022-07-21T01:38:33.509 DEBUG:teuthology.orchestra.run.smithi171:> with open(fname, '"'"'w'"'"') as f: 2022-07-21T01:38:33.509 DEBUG:teuthology.orchestra.run.smithi171:> f.write('"'"'content'"'"') 2022-07-21T01:38:33.510 DEBUG:teuthology.orchestra.run.smithi171:> if True: 2022-07-21T01:38:33.510 DEBUG:teuthology.orchestra.run.smithi171:> f.flush() 2022-07-21T01:38:33.510 DEBUG:teuthology.orchestra.run.smithi171:> os.fsync(f.fileno()) 2022-07-21T01:38:33.511 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:38:43.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:43.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.937+0000 7fdb60f52700 1 Processor -- start 2022-07-21T01:38:43.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.937+0000 7fdb60f52700 1 -- start start 2022-07-21T01:38:43.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.937+0000 7fdb60f52700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdb5c0b01a0 0x7fdb5c0a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:43.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.937+0000 7fdb60f52700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdb5c0afd60 0x7fdb5c0a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:43.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.937+0000 7fdb60f52700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0a7250 con 0x7fdb5c0b2710 2022-07-21T01:38:43.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.937+0000 7fdb60f52700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c0a7480 con 0x7fdb5c0afd60 2022-07-21T01:38:43.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.937+0000 7fdb60f52700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c0a7720 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.937+0000 7fdb5bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdb5c0afd60 0x7fdb5c0a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:43.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdb5c0afd60 0x7fdb5c0a8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46416/0 (socket says 172.21.15.3:46416) 2022-07-21T01:38:43.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5bfff700 1 -- 172.21.15.3:0/2541955677 learned_addr learned my addr 172.21.15.3:0/2541955677 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:43.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5affd700 1 --2- 172.21.15.3:0/2541955677 >> v2:172.21.15.143:3300/0 conn(0x7fdb5c0b01a0 0x7fdb5c0a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:43.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5bfff700 1 -- 172.21.15.3:0/2541955677 >> v2:172.21.15.143:3300/0 conn(0x7fdb5c0b01a0 msgr2=0x7fdb5c0a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:43.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5bfff700 1 --2- 172.21.15.3:0/2541955677 >> v2:172.21.15.143:3300/0 conn(0x7fdb5c0b01a0 0x7fdb5c0a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:43.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5bfff700 1 -- 172.21.15.3:0/2541955677 >> v1:172.21.15.143:6789/0 conn(0x7fdb5c0b2710 legacy=0x7fdb5c14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:43.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5bfff700 1 -- 172.21.15.3:0/2541955677 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0a79c0 con 0x7fdb5c0afd60 2022-07-21T01:38:43.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5bfff700 1 --2- 172.21.15.3:0/2541955677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0afd60 0x7fdb5c0a8a30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fdb5c06b950 tx=0x7fdb54009290).ready entity=mon.0 client_cookie=3329e84a3bfbfcd1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:43.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5a7fc700 1 -- 172.21.15.3:0/2541955677 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5400b830 con 0x7fdb5c0afd60 2022-07-21T01:38:43.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5a7fc700 1 -- 172.21.15.3:0/2541955677 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5400b990 con 0x7fdb5c0afd60 2022-07-21T01:38:43.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb5a7fc700 1 -- 172.21.15.3:0/2541955677 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5400bb40 con 0x7fdb5c0afd60 2022-07-21T01:38:43.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 -- 172.21.15.3:0/2541955677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0afd60 msgr2=0x7fdb5c0a8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:43.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 --2- 172.21.15.3:0/2541955677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0afd60 0x7fdb5c0a8a30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fdb5c06b950 tx=0x7fdb54009290).stop 2022-07-21T01:38:43.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 -- 172.21.15.3:0/2541955677 shutdown_connections 2022-07-21T01:38:43.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 --2- 172.21.15.3:0/2541955677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0afd60 0x7fdb5c0a8a30 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:43.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 --2- 172.21.15.3:0/2541955677 >> v2:172.21.15.143:3300/0 conn(0x7fdb5c0b01a0 0x7fdb5c0a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:43.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 -- 172.21.15.3:0/2541955677 >> 172.21.15.3:0/2541955677 conn(0x7fdb5c060310 msgr2=0x7fdb5c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:43.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 -- 172.21.15.3:0/2541955677 shutdown_connections 2022-07-21T01:38:43.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 -- 172.21.15.3:0/2541955677 wait complete. 2022-07-21T01:38:43.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 Processor -- start 2022-07-21T01:38:43.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 -- start start 2022-07-21T01:38:43.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb5c0afd60 0x7fdb5c0a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:43.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0b01a0 0x7fdb5c0aa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:43.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb5c0b2710 0x7fdb5c0aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:43.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c14b810 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c1572d0 con 0x7fdb5c0afd60 2022-07-21T01:38:43.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.941+0000 7fdb60f52700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c157570 con 0x7fdb5c0b2710 2022-07-21T01:38:43.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0b01a0 0x7fdb5c0aa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:43.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0b01a0 0x7fdb5c0aa8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46420/0 (socket says 172.21.15.3:46420) 2022-07-21T01:38:43.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5affd700 1 -- 172.21.15.3:0/2272602164 learned_addr learned my addr 172.21.15.3:0/2272602164 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:43.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5b7fe700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb5c0afd60 0x7fdb5c0a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:43.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5bfff700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb5c0b2710 0x7fdb5c0aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:43.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5affd700 1 -- 172.21.15.3:0/2272602164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb5c0b2710 msgr2=0x7fdb5c0aad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:43.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5affd700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb5c0b2710 0x7fdb5c0aad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:43.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5affd700 1 -- 172.21.15.3:0/2272602164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb5c0afd60 msgr2=0x7fdb5c0a9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:43.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5affd700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb5c0afd60 0x7fdb5c0a9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:43.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5affd700 1 -- 172.21.15.3:0/2272602164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54009530 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb5affd700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0b01a0 0x7fdb5c0aa8a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7fdb4c00a000 tx=0x7fdb4c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:43.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb58ff9700 1 -- 172.21.15.3:0/2272602164 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb4c011780 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb58ff9700 1 -- 172.21.15.3:0/2272602164 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c0118e0 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c14b0d0 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb58ff9700 1 -- 172.21.15.3:0/2272602164 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb4c011c40 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c14a580 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c0574e0 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.945+0000 7fdb58ff9700 1 -- 172.21.15.3:0/2272602164 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdb4c01c460 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.949+0000 7fdb58ff9700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb3c053b00 0x7fdb3c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:43.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.949+0000 7fdb58ff9700 1 -- 172.21.15.3:0/2272602164 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdb4c06c380 con 0x7fdb5c0b01a0 2022-07-21T01:38:43.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.949+0000 7fdb5b7fe700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb3c053b00 0x7fdb3c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:43.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.949+0000 7fdb5b7fe700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb3c053b00 0x7fdb3c055fc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fdb5000c000 tx=0x7fdb50008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:43.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:43.953+0000 7fdb58ff9700 1 -- 172.21.15.3:0/2272602164 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb4c043040 con 0x7fdb5c0b01a0 2022-07-21T01:38:44.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.117+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb5c14b570 con 0x7fdb5c0b01a0 2022-07-21T01:38:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb58ff9700 1 -- 172.21.15.3:0/2272602164 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2490 (secure 0 0 0) 0x7fdb4c046660 con 0x7fdb5c0b01a0 2022-07-21T01:38:44.122 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:44.123 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":29,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:38:26.808059+0000","modified":"2022-07-21T01:38:27.811394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/2524169670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2524169670},{"type":"v1","addr":"172.21.15.143:6833","nonce":2524169670}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-21T01:38:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb3c053b00 msgr2=0x7fdb3c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb3c053b00 0x7fdb3c055fc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fdb5000c000 tx=0x7fdb50008040).stop 2022-07-21T01:38:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0b01a0 msgr2=0x7fdb5c0aa8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0b01a0 0x7fdb5c0aa8a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7fdb4c00a000 tx=0x7fdb4c008040).stop 2022-07-21T01:38:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 shutdown_connections 2022-07-21T01:38:44.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb5c0b2710 0x7fdb5c0aad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb5c0b01a0 0x7fdb5c0aa8a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb3c053b00 0x7fdb3c055fc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 --2- 172.21.15.3:0/2272602164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb5c0afd60 0x7fdb5c0a9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 >> 172.21.15.3:0/2272602164 conn(0x7fdb5c060310 msgr2=0x7fdb5c0add00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:44.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 shutdown_connections 2022-07-21T01:38:44.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.121+0000 7fdb60f52700 1 -- 172.21.15.3:0/2272602164 wait complete. 2022-07-21T01:38:44.127 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 29 2022-07-21T01:38:44.132 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_caps 20 2022-07-21T01:38:44.240 INFO:teuthology.orchestra.run.smithi143.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-21T01:38:44.240 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-21T01:38:44.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.353+0000 7f491759e700 1 Processor -- start 2022-07-21T01:38:44.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.353+0000 7f491759e700 1 -- start start 2022-07-21T01:38:44.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.353+0000 7f491759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49180b01a0 0x7f49180a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.353+0000 7f491759e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180a7250 con 0x7f49180afd60 2022-07-21T01:38:44.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.353+0000 7f491759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180a7480 con 0x7f49180b2710 2022-07-21T01:38:44.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.353+0000 7f491759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49180a7720 con 0x7f49180b01a0 2022-07-21T01:38:44.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f49180afd60 0x7f4918151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54710/0 (socket says 172.21.15.3:54710) 2022-07-21T01:38:44.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 -- 172.21.15.3:0/3046364051 learned_addr learned my addr 172.21.15.3:0/3046364051 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:44.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f490ffff700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49180b01a0 0x7f49180a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542191045 0 0) 0x7f49180a7250 con 0x7f49180afd60 2022-07-21T01:38:44.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f8003260 con 0x7f49180afd60 2022-07-21T01:38:44.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2262000222 0 0) 0x7f48f8003260 con 0x7f49180afd60 2022-07-21T01:38:44.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49180b01a0 msgr2=0x7f49180a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:44.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49180b01a0 0x7f49180a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 >> v1:172.21.15.143:6789/0 conn(0x7f49180b2710 legacy=0x7f491814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180a79c0 con 0x7f49180afd60 2022-07-21T01:38:44.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f49100019e0 con 0x7f49180afd60 2022-07-21T01:38:44.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 >> v1:172.21.15.3:6789/0 conn(0x7f49180afd60 legacy=0x7f4918151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49180b01a0 0x7f48f8003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f8003cb0 0x7f48f8004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48f80049f0 0x7f48f8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49180a7720 con 0x7f48f80049f0 2022-07-21T01:38:44.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f800b800 con 0x7f48f8003cb0 2022-07-21T01:38:44.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f800b9c0 con 0x7f49180b01a0 2022-07-21T01:38:44.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48f80049f0 0x7f48f8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f491659c700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49180b01a0 0x7f48f8003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f490ffff700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f8003cb0 0x7f48f8004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 -- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49180b01a0 msgr2=0x7f48f8003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:44.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49180b01a0 0x7f48f8003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 -- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f8003cb0 msgr2=0x7f48f8004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:44.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f8003cb0 0x7f48f8004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 -- 172.21.15.3:0/3046364051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f8003260 con 0x7f48f80049f0 2022-07-21T01:38:44.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48f80049f0 0x7f48f8004e70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f49100016f0 tx=0x7f491806a5f0).ready entity=mon.0 client_cookie=42eb69184ed4f42d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:44.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f491000cc70 con 0x7f48f80049f0 2022-07-21T01:38:44.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491000cdd0 con 0x7f48f80049f0 2022-07-21T01:38:44.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4915d9b700 1 -- 172.21.15.3:0/3046364051 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4910005b80 con 0x7f48f80049f0 2022-07-21T01:38:44.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f491759e700 1 -- 172.21.15.3:0/3046364051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48f80049f0 msgr2=0x7f48f8004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f491759e700 1 --2- 172.21.15.3:0/3046364051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48f80049f0 0x7f48f8004e70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f49100016f0 tx=0x7f491806a5f0).stop 2022-07-21T01:38:44.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f4916d9d700 1 -- 172.21.15.3:0/3046364051 reap_dead start 2022-07-21T01:38:44.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f491759e700 1 -- 172.21.15.3:0/3046364051 shutdown_connections 2022-07-21T01:38:44.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f491759e700 1 -- 172.21.15.3:0/3046364051 >> 172.21.15.3:0/3046364051 conn(0x7f4918060310 msgr2=0x7f49180b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:44.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f491759e700 1 -- 172.21.15.3:0/3046364051 shutdown_connections 2022-07-21T01:38:44.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.357+0000 7f491759e700 1 -- 172.21.15.3:0/3046364051 wait complete. 2022-07-21T01:38:44.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 Processor -- start 2022-07-21T01:38:44.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 -- start start 2022-07-21T01:38:44.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80049f0 0x7f49180a96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491815a870 0x7f49181566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49180a9f70 0x7f4918156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4918159c30 con 0x7f49180a9f70 2022-07-21T01:38:44.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4918159e90 con 0x7f48f80049f0 2022-07-21T01:38:44.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491815a130 con 0x7f491815a870 2022-07-21T01:38:44.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f4916d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49180a9f70 0x7f4918156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f4916d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49180a9f70 0x7f4918156bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46440/0 (socket says 172.21.15.3:46440) 2022-07-21T01:38:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f4916d9d700 1 -- 172.21.15.3:0/2585596574 learned_addr learned my addr 172.21.15.3:0/2585596574 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80049f0 0x7f49180a96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f490ffff700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491815a870 0x7f49181566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f4916d9d700 1 -- 172.21.15.3:0/2585596574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491815a870 msgr2=0x7f49181566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f4916d9d700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491815a870 0x7f49181566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f4916d9d700 1 -- 172.21.15.3:0/2585596574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80049f0 msgr2=0x7f49180a96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f4916d9d700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80049f0 0x7f49180a96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f4916d9d700 1 -- 172.21.15.3:0/2585596574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491815a3d0 con 0x7f49180a9f70 2022-07-21T01:38:44.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491659c700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80049f0 0x7f49180a96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:44.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f4916d9d700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49180a9f70 0x7f4918156bd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f4910001c40 tx=0x7f4910013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:44.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f490f7fe700 1 -- 172.21.15.3:0/2585596574 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f491001a030 con 0x7f49180a9f70 2022-07-21T01:38:44.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f490f7fe700 1 -- 172.21.15.3:0/2585596574 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4910018030 con 0x7f49180a9f70 2022-07-21T01:38:44.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4918064590 con 0x7f49180a9f70 2022-07-21T01:38:44.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f490f7fe700 1 -- 172.21.15.3:0/2585596574 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f491000cc70 con 0x7f49180a9f70 2022-07-21T01:38:44.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.361+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4918066a50 con 0x7f49180a9f70 2022-07-21T01:38:44.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.365+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491815a560 con 0x7f49180a9f70 2022-07-21T01:38:44.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.365+0000 7f490f7fe700 1 -- 172.21.15.3:0/2585596574 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4910010070 con 0x7f49180a9f70 2022-07-21T01:38:44.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.365+0000 7f490f7fe700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053b30 0x7f48fc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.365+0000 7f490f7fe700 1 -- 172.21.15.3:0/2585596574 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f491006c5a0 con 0x7f49180a9f70 2022-07-21T01:38:44.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.365+0000 7f491659c700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053b30 0x7f48fc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.365+0000 7f491659c700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053b30 0x7f48fc055ff0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f4908003000 tx=0x7f4908007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:44.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.373+0000 7f490f7fe700 1 -- 172.21.15.3:0/2585596574 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4910043360 con 0x7f49180a9f70 2022-07-21T01:38:44.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.533+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49180574e0 con 0x7f49180a9f70 2022-07-21T01:38:44.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f490f7fe700 1 -- 172.21.15.3:0/2585596574 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2490 (secure 0 0 0) 0x7f4910046980 con 0x7f49180a9f70 2022-07-21T01:38:44.534 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:44.535 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":29,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:38:26.808059+0000","modified":"2022-07-21T01:38:27.811394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/2524169670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2524169670},{"type":"v1","addr":"172.21.15.143:6833","nonce":2524169670}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-21T01:38:44.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053b30 msgr2=0x7f48fc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053b30 0x7f48fc055ff0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f4908003000 tx=0x7f4908007a00).stop 2022-07-21T01:38:44.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49180a9f70 msgr2=0x7f4918156bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49180a9f70 0x7f4918156bd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f4910001c40 tx=0x7f4910013000).stop 2022-07-21T01:38:44.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 shutdown_connections 2022-07-21T01:38:44.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49180a9f70 0x7f4918156bd0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491815a870 0x7f49181566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053b30 0x7f48fc055ff0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 --2- 172.21.15.3:0/2585596574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80049f0 0x7f49180a96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 >> 172.21.15.3:0/2585596574 conn(0x7f4918060310 msgr2=0x7f4918149bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:44.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 shutdown_connections 2022-07-21T01:38:44.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.537+0000 7f491759e700 1 -- 172.21.15.3:0/2585596574 wait complete. 2022-07-21T01:38:44.540 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 29 2022-07-21T01:38:44.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-21T01:38:44.657 INFO:teuthology.orchestra.run.smithi143.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-21T01:38:44.658 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-21T01:38:44.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:44.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.765+0000 7fd44af4c700 1 Processor -- start 2022-07-21T01:38:44.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.765+0000 7fd44af4c700 1 -- start start 2022-07-21T01:38:44.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b2710 0x7fd4441401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd4440b0270 0x7fd4441406d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd444067a10 con 0x7fd4440ac880 2022-07-21T01:38:44.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4440ad740 con 0x7fd4440b0270 2022-07-21T01:38:44.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd444067cb0 con 0x7fd4440b2710 2022-07-21T01:38:44.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd449749700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd4440b0270 0x7fd4441406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd449749700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd4440b0270 0x7fd4441406d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:51894/0 (socket says 172.21.15.3:51894) 2022-07-21T01:38:44.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd449749700 1 -- 172.21.15.3:0/2145333330 learned_addr learned my addr 172.21.15.3:0/2145333330 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:44.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd449f4a700 1 --2- 172.21.15.3:0/2145333330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b2710 0x7fd4441401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd449749700 1 -- 172.21.15.3:0/2145333330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b2710 msgr2=0x7fd4441401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:44.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd449749700 1 --2- 172.21.15.3:0/2145333330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b2710 0x7fd4441401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd449749700 1 -- 172.21.15.3:0/2145333330 >> v1:172.21.15.143:6789/0 conn(0x7fd4440ac880 legacy=0x7fd444143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd449749700 1 -- 172.21.15.3:0/2145333330 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd444147160 con 0x7fd4440b0270 2022-07-21T01:38:44.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd449749700 1 --2- 172.21.15.3:0/2145333330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4440b0270 0x7fd4441406d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fd440002f70 tx=0x7fd440006eb0).ready entity=mon.1 client_cookie=f27ef0418da6c24f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:44.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd448f48700 1 -- 172.21.15.3:0/2145333330 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd44000d570 con 0x7fd4440b0270 2022-07-21T01:38:44.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd448f48700 1 -- 172.21.15.3:0/2145333330 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd440000e90 con 0x7fd4440b0270 2022-07-21T01:38:44.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd448f48700 1 -- 172.21.15.3:0/2145333330 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd44000d900 con 0x7fd4440b0270 2022-07-21T01:38:44.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 -- 172.21.15.3:0/2145333330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4440b0270 msgr2=0x7fd4441406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 --2- 172.21.15.3:0/2145333330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4440b0270 0x7fd4441406d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fd440002f70 tx=0x7fd440006eb0).stop 2022-07-21T01:38:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 -- 172.21.15.3:0/2145333330 shutdown_connections 2022-07-21T01:38:44.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 --2- 172.21.15.3:0/2145333330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4440b0270 0x7fd4441406d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 --2- 172.21.15.3:0/2145333330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b2710 0x7fd4441401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 -- 172.21.15.3:0/2145333330 >> 172.21.15.3:0/2145333330 conn(0x7fd444060310 msgr2=0x7fd4440b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 -- 172.21.15.3:0/2145333330 shutdown_connections 2022-07-21T01:38:44.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.769+0000 7fd44af4c700 1 -- 172.21.15.3:0/2145333330 wait complete. 2022-07-21T01:38:44.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44af4c700 1 Processor -- start 2022-07-21T01:38:44.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44af4c700 1 -- start start 2022-07-21T01:38:44.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44af4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4440ac880 0x7fd4441461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44af4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b0270 0x7fd4440a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44af4c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4440b2710 0x7fd4440a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44af4c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd444152570 con 0x7fd4440b2710 2022-07-21T01:38:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44af4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44409ea40 con 0x7fd4440ac880 2022-07-21T01:38:44.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44af4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd444142e30 con 0x7fd4440b0270 2022-07-21T01:38:44.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449749700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b0270 0x7fd4440a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44a74b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4440b2710 0x7fd4440a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449749700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b0270 0x7fd4440a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:53370/0 (socket says 172.21.15.3:53370) 2022-07-21T01:38:44.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44a74b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4440b2710 0x7fd4440a1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46456/0 (socket says 172.21.15.3:46456) 2022-07-21T01:38:44.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449749700 1 -- 172.21.15.3:0/3739104670 learned_addr learned my addr 172.21.15.3:0/3739104670 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:44.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449f4a700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4440ac880 0x7fd4441461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449749700 1 -- 172.21.15.3:0/3739104670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4440ac880 msgr2=0x7fd4441461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449749700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4440ac880 0x7fd4441461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449749700 1 -- 172.21.15.3:0/3739104670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4440b2710 msgr2=0x7fd4440a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449749700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4440b2710 0x7fd4440a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449749700 1 -- 172.21.15.3:0/3739104670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44000b1e0 con 0x7fd4440b0270 2022-07-21T01:38:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd449749700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b0270 0x7fd4440a0b50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fd444143100 tx=0x7fd440015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd432ffd700 1 -- 172.21.15.3:0/3739104670 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd440011610 con 0x7fd4440b0270 2022-07-21T01:38:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd432ffd700 1 -- 172.21.15.3:0/3739104670 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd440011770 con 0x7fd4440b0270 2022-07-21T01:38:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44409ec70 con 0x7fd4440b0270 2022-07-21T01:38:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.773+0000 7fd432ffd700 1 -- 172.21.15.3:0/3739104670 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd440011ad0 con 0x7fd4440b0270 2022-07-21T01:38:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.777+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4441466e0 con 0x7fd4440b0270 2022-07-21T01:38:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.777+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd44409ee00 con 0x7fd4440b0270 2022-07-21T01:38:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.777+0000 7fd432ffd700 1 -- 172.21.15.3:0/3739104670 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd44001a040 con 0x7fd4440b0270 2022-07-21T01:38:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.777+0000 7fd432ffd700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd424053950 0x7fd424055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:44.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.777+0000 7fd432ffd700 1 -- 172.21.15.3:0/3739104670 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd44006ce60 con 0x7fd4440b0270 2022-07-21T01:38:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.777+0000 7fd449f4a700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd424053950 0x7fd424055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.777+0000 7fd449f4a700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd424053950 0x7fd424055e10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fd43400b000 tx=0x7fd434006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:44.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.785+0000 7fd432ffd700 1 -- 172.21.15.3:0/3739104670 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd440043b80 con 0x7fd4440b0270 2022-07-21T01:38:44.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.949+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4440574e0 con 0x7fd4440b0270 2022-07-21T01:38:44.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd432ffd700 1 -- 172.21.15.3:0/3739104670 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2490 (secure 0 0 0) 0x7fd440023020 con 0x7fd4440b0270 2022-07-21T01:38:44.950 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:44.950 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":29,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:38:26.808059+0000","modified":"2022-07-21T01:38:27.811394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/2524169670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2524169670},{"type":"v1","addr":"172.21.15.143:6833","nonce":2524169670}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-21T01:38:44.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd424053950 msgr2=0x7fd424055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd424053950 0x7fd424055e10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fd43400b000 tx=0x7fd434006df0).stop 2022-07-21T01:38:44.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b0270 msgr2=0x7fd4440a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:44.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b0270 0x7fd4440a0b50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fd444143100 tx=0x7fd440015000).stop 2022-07-21T01:38:44.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 shutdown_connections 2022-07-21T01:38:44.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4440b2710 0x7fd4440a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4440b0270 0x7fd4440a0b50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd424053950 0x7fd424055e10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 --2- 172.21.15.3:0/3739104670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4440ac880 0x7fd4441461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:44.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 >> 172.21.15.3:0/3739104670 conn(0x7fd444060310 msgr2=0x7fd4441446d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:44.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 shutdown_connections 2022-07-21T01:38:44.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:44.953+0000 7fd44af4c700 1 -- 172.21.15.3:0/3739104670 wait complete. 2022-07-21T01:38:44.955 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 29 2022-07-21T01:38:44.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b cache drop 10 2022-07-21T01:38:45.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc8a7f700 1 Processor -- start 2022-07-21T01:38:45.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc8a7f700 1 -- start start 2022-07-21T01:38:45.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc8a7f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7dc405fd90 0x7f7dc40b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc8a7f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7dc40b5b20 0x7f7dc40b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc8a7f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc41440f0 con 0x7f7dc40b66f0 2022-07-21T01:38:45.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc8a7f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc40ba840 con 0x7f7dc405fd90 2022-07-21T01:38:45.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc8a7f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc4060170 con 0x7f7dc40b5b20 2022-07-21T01:38:45.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7db9d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7dc40b66f0 0x7f7dc40ba130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54742/0 (socket says 172.21.15.3:54742) 2022-07-21T01:38:45.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7dc405fd90 0x7f7dc40b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7db9d9b700 1 -- 172.21.15.3:0/839079152 learned_addr learned my addr 172.21.15.3:0/839079152 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:45.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/839079152 >> v2:172.21.15.143:3300/0 conn(0x7f7dc40b5b20 0x7f7dc40b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc1d9b700 1 -- 172.21.15.3:0/839079152 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014559763 0 0) 0x7f7dc41440f0 con 0x7f7dc40b66f0 2022-07-21T01:38:45.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc1d9b700 1 -- 172.21.15.3:0/839079152 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da4003260 con 0x7f7dc40b66f0 2022-07-21T01:38:45.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc259c700 1 -- 172.21.15.3:0/839079152 >> v2:172.21.15.143:3300/0 conn(0x7f7dc40b5b20 msgr2=0x7f7dc40b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:45.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc259c700 1 --2- 172.21.15.3:0/839079152 >> v2:172.21.15.143:3300/0 conn(0x7f7dc40b5b20 0x7f7dc40b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc259c700 1 -- 172.21.15.3:0/839079152 >> v1:172.21.15.3:6789/0 conn(0x7f7dc40b66f0 legacy=0x7f7dc40ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:45.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc259c700 1 -- 172.21.15.3:0/839079152 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc4143be0 con 0x7f7dc405fd90 2022-07-21T01:38:45.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/839079152 >> v2:172.21.15.143:3300/0 conn(0x7f7dc40b5b20 0x7f7dc40b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:45.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc259c700 1 --2- 172.21.15.3:0/839079152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc405fd90 0x7f7dc40b4340 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f7db4002f70 tx=0x7f7db4006df0).ready entity=mon.0 client_cookie=90bd2a71236877ac server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:45.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc1d9b700 1 -- 172.21.15.3:0/839079152 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7db400d570 con 0x7f7dc405fd90 2022-07-21T01:38:45.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc1d9b700 1 -- 172.21.15.3:0/839079152 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db4000e90 con 0x7f7dc405fd90 2022-07-21T01:38:45.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.073+0000 7f7dc1d9b700 1 -- 172.21.15.3:0/839079152 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7db400d8c0 con 0x7f7dc405fd90 2022-07-21T01:38:45.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/839079152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc405fd90 msgr2=0x7f7dc40b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:45.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 --2- 172.21.15.3:0/839079152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc405fd90 0x7f7dc40b4340 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f7db4002f70 tx=0x7f7db4006df0).stop 2022-07-21T01:38:45.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/839079152 shutdown_connections 2022-07-21T01:38:45.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 --2- 172.21.15.3:0/839079152 >> v2:172.21.15.143:3300/0 conn(0x7f7dc40b5b20 0x7f7dc40b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 --2- 172.21.15.3:0/839079152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc405fd90 0x7f7dc40b4340 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/839079152 >> 172.21.15.3:0/839079152 conn(0x7f7dc4062530 msgr2=0x7f7dc4066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:45.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/839079152 shutdown_connections 2022-07-21T01:38:45.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/839079152 wait complete. 2022-07-21T01:38:45.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 Processor -- start 2022-07-21T01:38:45.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.077+0000 7f7dc8a7f700 1 -- start start 2022-07-21T01:38:45.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc8a7f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dc405fd90 0x7f7dc41565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc8a7f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dc40b5b20 0x7f7dc4146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc8a7f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc40b66f0 0x7f7dc4141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc8a7f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4155bf0 con 0x7f7dc40b66f0 2022-07-21T01:38:45.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc8a7f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4155e90 con 0x7f7dc405fd90 2022-07-21T01:38:45.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc8a7f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4156130 con 0x7f7dc40b5b20 2022-07-21T01:38:45.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc40b66f0 0x7f7dc4141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc40b66f0 0x7f7dc4141040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46470/0 (socket says 172.21.15.3:46470) 2022-07-21T01:38:45.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc2d9d700 1 -- 172.21.15.3:0/2515188840 learned_addr learned my addr 172.21.15.3:0/2515188840 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:45.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc259c700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dc405fd90 0x7f7dc41565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dc40b5b20 0x7f7dc4146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc2d9d700 1 -- 172.21.15.3:0/2515188840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dc40b5b20 msgr2=0x7f7dc4146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:45.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dc40b5b20 0x7f7dc4146ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc2d9d700 1 -- 172.21.15.3:0/2515188840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dc405fd90 msgr2=0x7f7dc41565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:45.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dc405fd90 0x7f7dc41565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc2d9d700 1 -- 172.21.15.3:0/2515188840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db400b180 con 0x7f7dc40b66f0 2022-07-21T01:38:45.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc40b66f0 0x7f7dc4141040 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f7dbc006cc0 tx=0x7f7dbc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:45.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dbbfff700 1 -- 172.21.15.3:0/2515188840 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dbc013620 con 0x7f7dc40b66f0 2022-07-21T01:38:45.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dbbfff700 1 -- 172.21.15.3:0/2515188840 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc013780 con 0x7f7dc40b66f0 2022-07-21T01:38:45.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/2515188840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc4148690 con 0x7f7dc40b66f0 2022-07-21T01:38:45.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dbbfff700 1 -- 172.21.15.3:0/2515188840 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dbc013ae0 con 0x7f7dc40b66f0 2022-07-21T01:38:45.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.081+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/2515188840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc4147390 con 0x7f7dc40b66f0 2022-07-21T01:38:45.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.085+0000 7f7dbbfff700 1 -- 172.21.15.3:0/2515188840 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7dbc013c40 con 0x7f7dc40b66f0 2022-07-21T01:38:45.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.085+0000 7f7dbbfff700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7da8053a70 0x7f7da8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.085+0000 7f7dbbfff700 1 -- 172.21.15.3:0/2515188840 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7dbc06bf10 con 0x7f7dc40b66f0 2022-07-21T01:38:45.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.085+0000 7f7dc259c700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7da8053a70 0x7f7da8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.085+0000 7f7dc259c700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7da8053a70 0x7f7da8055f30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f7db401a000 tx=0x7f7db4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:45.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.085+0000 7f7dd2cd6740 1 Processor -- start 2022-07-21T01:38:45.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.085+0000 7f7dd2cd6740 1 -- start start 2022-07-21T01:38:45.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1a82580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a7d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 0x1ab0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x19f2210 con 0x1a81130 2022-07-21T01:38:45.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x19d2450 con 0x1ab06d0 2022-07-21T01:38:45.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a815b0 con 0x1a7c830 2022-07-21T01:38:45.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7db9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a7d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7db9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a7d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46474/0 (socket says 172.21.15.3:46474) 2022-07-21T01:38:45.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3524016083 learned_addr learned my addr 172.21.15.3:0/3524016083 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:45.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1a82580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 0x1ab0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3524016083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 msgr2=0x1a82580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:45.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1a82580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3524016083 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 msgr2=0x1ab0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:45.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 0x1ab0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3524016083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1a29ad0 con 0x1a81130 2022-07-21T01:38:45.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1a82580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:45.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a7d600 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x19d7b30 tx=0x7f7dac009040).ready entity=mon.0 client_cookie=417f15cef6e6016 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:45.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/3524016083 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dac007e70 con 0x1a81130 2022-07-21T01:38:45.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/3524016083 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac0070d0 con 0x1a81130 2022-07-21T01:38:45.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/3524016083 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dac007430 con 0x1a81130 2022-07-21T01:38:45.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3524016083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 msgr2=0x1a7d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:45.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a7d600 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x19d7b30 tx=0x7f7dac009040).stop 2022-07-21T01:38:45.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3524016083 shutdown_connections 2022-07-21T01:38:45.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 0x1ab0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a7d600 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3524016083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1a82580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3524016083 >> 172.21.15.3:0/3524016083 conn(0x1a098f0 msgr2=0x1a81e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:45.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3524016083 shutdown_connections 2022-07-21T01:38:45.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3524016083 wait complete. 2022-07-21T01:38:45.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 -- start start 2022-07-21T01:38:45.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.089+0000 7f7dd2cd6740 1 Processor -- start 2022-07-21T01:38:45.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 0x1ac6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1aae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1ab06d0 0x1aaea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ab53d0 con 0x1a81130 2022-07-21T01:38:45.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ab5600 con 0x1a7c830 2022-07-21T01:38:45.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ac5ed0 con 0x1ab06d0 2022-07-21T01:38:45.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 10 client.0 ll_register_callbacks cb 0x1a390b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-21T01:38:45.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7db9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1aae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7db9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1aae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46482/0 (socket says 172.21.15.3:46482) 2022-07-21T01:38:45.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3229891453 learned_addr learned my addr 172.21.15.3:0/3229891453 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:45.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 0x1ac6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1ab06d0 0x1aaea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1ab06d0 msgr2=0x1aaea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:45.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1ab06d0 0x1aaea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 msgr2=0x1ac6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:45.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 0x1ac6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3229891453 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da4003260 con 0x1a81130 2022-07-21T01:38:45.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 0x1ac6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:45.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1aae570 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x19d7b30 tx=0x7f7dac018000).ready entity=mon.0 client_cookie=ab51378d94ea3452 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:45.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7da17fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2022-07-21T01:38:45.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7da17fa700 1 -- 172.21.15.3:0/3229891453 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dac005800 con 0x1a81130 2022-07-21T01:38:45.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7da17fa700 1 -- 172.21.15.3:0/3229891453 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7dac021db0 con 0x1a81130 2022-07-21T01:38:45.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3229891453 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x19e81c0 con 0x1a81130 2022-07-21T01:38:45.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7da17fa700 1 -- 172.21.15.3:0/3229891453 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dac005960 con 0x1a81130 2022-07-21T01:38:45.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 10 client.6117 fetch_fsmap learned FSMap version 29 2022-07-21T01:38:45.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3229891453 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x19e17b0 con 0x1a81130 2022-07-21T01:38:45.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7da17fa700 1 -- 172.21.15.3:0/3229891453 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x19e81c0 con 0x1a81130 2022-07-21T01:38:45.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7da17fa700 1 -- 172.21.15.3:0/3229891453 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 1434+0+0 (secure 0 0 0) 0x7f7dac01e070 con 0x1a81130 2022-07-21T01:38:45.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 10 client.6117 fetch_fsmap finished waiting for FSMap version 29 2022-07-21T01:38:45.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 10 client.6117 resolve_mds: resolved ID 'b' to GID 5934 2022-07-21T01:38:45.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 20 client.6117 populate_metadata read hostname 'smithi003' 2022-07-21T01:38:45.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] conn(0x1ab7c70 0x1ac94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 4 client.6117 mds_command: new command op to 5934 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-21T01:38:45.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3229891453 --> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x19e0240 con 0x1ab7c70 2022-07-21T01:38:45.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] conn(0x1ab7c70 0x1ac94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] conn(0x1ab7c70 0x1ac94a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=81a6ce5455cef0d0 server_cookie=f6ba51452975ddcf in_seq=0 out_seq=0 2022-07-21T01:38:45.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.093+0000 7f7da17fa700 10 client.6117 ms_handle_connect on v2:172.21.15.143:6832/2524169670 2022-07-21T01:38:45.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7da17fa700 1 -- 172.21.15.3:0/3229891453 <== mds.0 v2:172.21.15.143:6832/2524169670 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x19e0240 con 0x1ab7c70 2022-07-21T01:38:45.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7da17fa700 10 client.6117 handle_command_reply: tid=0 2022-07-21T01:38:45.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 client.6117 shutdown 2022-07-21T01:38:45.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3229891453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 msgr2=0x1aae570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:45.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1aae570 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x19d7b30 tx=0x7f7dac018000).stop 2022-07-21T01:38:45.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3229891453 shutdown_connections 2022-07-21T01:38:45.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] conn(0x1ab7c70 0x1ac94a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1ab06d0 0x1aaea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1aae570 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3229891453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 0x1ac6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7da17fa700 0 client.6117 ms_handle_reset on v2:172.21.15.143:6832/2524169670 2022-07-21T01:38:45.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3229891453 >> 172.21.15.3:0/3229891453 conn(0x1a098f0 msgr2=0x1aaf860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:45.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3229891453 shutdown_connections 2022-07-21T01:38:45.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3229891453 wait complete. 2022-07-21T01:38:45.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.097+0000 7f7dd2cd6740 20 client.6117 trim_cache size 0 max 16384 2022-07-21T01:38:45.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 Processor -- start 2022-07-21T01:38:45.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 -- start start 2022-07-21T01:38:45.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1aaf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a804e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 0x1aae260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a7ed70 con 0x1a81130 2022-07-21T01:38:45.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x19dc840 con 0x1ab06d0 2022-07-21T01:38:45.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x19e81c0 con 0x1a7c830 2022-07-21T01:38:45.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7db9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a804e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7db9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a804e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46488/0 (socket says 172.21.15.3:46488) 2022-07-21T01:38:45.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3719942160 learned_addr learned my addr 172.21.15.3:0/3719942160 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:45.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1aaf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 0x1aae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 msgr2=0x1aaf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:45.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1aaf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 msgr2=0x1aae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:45.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 0x1aae260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7db9d9b700 1 -- 172.21.15.3:0/3719942160 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x19ea3a0 con 0x1a81130 2022-07-21T01:38:45.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 0x1aae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:45.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1aaf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:45.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a804e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x19d7b30 tx=0x7f7dac01e000).ready entity=mon.0 client_cookie=ddb1416dc9bba7b0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:45.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7da37fe700 1 -- 172.21.15.3:0/3719942160 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dac0114c0 con 0x1a81130 2022-07-21T01:38:45.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7da37fe700 1 -- 172.21.15.3:0/3719942160 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac011620 con 0x1a81130 2022-07-21T01:38:45.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7da37fe700 1 -- 172.21.15.3:0/3719942160 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dac011790 con 0x1a81130 2022-07-21T01:38:45.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3719942160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 msgr2=0x1a804e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:45.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a804e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x19d7b30 tx=0x7f7dac01e000).stop 2022-07-21T01:38:45.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3719942160 shutdown_connections 2022-07-21T01:38:45.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1ab06d0 0x1aae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1a81130 0x1a804e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3719942160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a7c830 0x1aaf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3719942160 >> 172.21.15.3:0/3719942160 conn(0x1a098f0 msgr2=0x1ab3510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:45.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3719942160 shutdown_connections 2022-07-21T01:38:45.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.113+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3719942160 wait complete. 2022-07-21T01:38:45.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 -- start start 2022-07-21T01:38:45.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 Processor -- start 2022-07-21T01:38:45.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 0x1b18450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a81130 0x1b18940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1ab06d0 0x1aad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a83e90 con 0x1ab06d0 2022-07-21T01:38:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a7f080 con 0x1a7c830 2022-07-21T01:38:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a81f70 con 0x1a81130 2022-07-21T01:38:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 10 client.0 ll_register_callbacks cb 0x1ab6930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-21T01:38:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1ab06d0 0x1aad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1ab06d0 0x1aad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46498/0 (socket says 172.21.15.3:46498) 2022-07-21T01:38:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc2d9d700 1 -- 172.21.15.3:0/3908540313 learned_addr learned my addr 172.21.15.3:0/3908540313 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7db9d9b700 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a81130 0x1b18940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 0x1b18450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc2d9d700 1 -- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a81130 msgr2=0x1b18940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:45.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a81130 0x1b18940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc2d9d700 1 -- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 msgr2=0x1b18450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:45.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 0x1b18450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:45.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc2d9d700 1 -- 172.21.15.3:0/3908540313 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x19e17b0 con 0x1ab06d0 2022-07-21T01:38:45.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7da27fc700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2022-07-21T01:38:45.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dc2d9d700 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1ab06d0 0x1aad1f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f7dbc024000 tx=0x7f7dbc018000).ready entity=mon.0 client_cookie=c4db9b1d47e11ba server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:45.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7da27fc700 1 -- 172.21.15.3:0/3908540313 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dbc046820 con 0x1ab06d0 2022-07-21T01:38:45.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3908540313 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7dac0070d0 con 0x1ab06d0 2022-07-21T01:38:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7da27fc700 1 -- 172.21.15.3:0/3908540313 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7dbc028d90 con 0x1ab06d0 2022-07-21T01:38:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7da27fc700 1 -- 172.21.15.3:0/3908540313 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dbc005a70 con 0x1ab06d0 2022-07-21T01:38:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 10 client.6123 fetch_fsmap learned FSMap version 29 2022-07-21T01:38:45.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3908540313 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x19e6d90 con 0x1ab06d0 2022-07-21T01:38:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7da27fc700 1 -- 172.21.15.3:0/3908540313 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac0070d0 con 0x1ab06d0 2022-07-21T01:38:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7da27fc700 1 -- 172.21.15.3:0/3908540313 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 1434+0+0 (secure 0 0 0) 0x7f7dbc02f020 con 0x1ab06d0 2022-07-21T01:38:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 10 client.6123 fetch_fsmap finished waiting for FSMap version 29 2022-07-21T01:38:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 10 client.6123 resolve_mds: resolved ID 'b' to GID 5934 2022-07-21T01:38:45.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 20 client.6123 populate_metadata read hostname 'smithi003' 2022-07-21T01:38:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] conn(0x1a81910 0x1aad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 4 client.6123 mds_command: new command op to 5934 tid=0[{"prefix": "cache drop", "timeout": 10}] 2022-07-21T01:38:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.117+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3908540313 --> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] -- command(tid 0: {"prefix": "cache drop", "timeout": 10}) v1 -- 0x18d55e0 con 0x1a81910 2022-07-21T01:38:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.121+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] conn(0x1a81910 0x1aad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.121+0000 7f7da27fc700 10 client.6123 ms_handle_connect on v2:172.21.15.143:6832/2524169670 2022-07-21T01:38:45.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:45.121+0000 7f7dc259c700 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] conn(0x1a81910 0x1aad6e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=5a7825e8a22293c0 server_cookie=998151918c22fb8 in_seq=0 out_seq=0 2022-07-21T01:38:55.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7da27fc700 1 -- 172.21.15.3:0/3908540313 <== mds.0 v2:172.21.15.143:6832/2524169670 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+406 (crc 0 0 0) 0x18d55e0 con 0x1a81910 2022-07-21T01:38:55.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7da27fc700 10 client.6123 handle_command_reply: tid=0 2022-07-21T01:38:55.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 client.6123 shutdown 2022-07-21T01:38:55.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3908540313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1ab06d0 msgr2=0x1aad1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:55.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1ab06d0 0x1aad1f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f7dbc024000 tx=0x7f7dbc018000).stop 2022-07-21T01:38:55.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3908540313 shutdown_connections 2022-07-21T01:38:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:6832/2524169670,v1:172.21.15.143:6833/2524169670] conn(0x1a81910 0x1aad6e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x1ab06d0 0x1aad1f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a81130 0x1b18940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7da27fc700 0 client.6123 ms_handle_reset on v2:172.21.15.143:6832/2524169670 2022-07-21T01:38:55.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 --2- 172.21.15.3:0/3908540313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1a7c830 0x1b18450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3908540313 >> 172.21.15.3:0/3908540313 conn(0x1a098f0 msgr2=0x1a7d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:55.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.405+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3908540313 shutdown_connections 2022-07-21T01:38:55.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dd2cd6740 1 -- 172.21.15.3:0/3908540313 wait complete. 2022-07-21T01:38:55.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dd2cd6740 20 client.6123 trim_cache size 0 max 16384 2022-07-21T01:38:55.410 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:38:55.410 INFO:teuthology.orchestra.run.smithi003.stdout: "client_recall": { 2022-07-21T01:38:55.411 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": -110, 2022-07-21T01:38:55.411 INFO:teuthology.orchestra.run.smithi003.stdout: "message": "(110) Connection timed out", 2022-07-21T01:38:55.411 INFO:teuthology.orchestra.run.smithi003.stdout: "recalled": 240 2022-07-21T01:38:55.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:55.412 INFO:teuthology.orchestra.run.smithi003.stdout: "flush_journal": { 2022-07-21T01:38:55.412 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2022-07-21T01:38:55.413 INFO:teuthology.orchestra.run.smithi003.stdout: "message": "" 2022-07-21T01:38:55.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:55.413 INFO:teuthology.orchestra.run.smithi003.stdout: "trim_cache": { 2022-07-21T01:38:55.413 INFO:teuthology.orchestra.run.smithi003.stdout: "trimmed": 220 2022-07-21T01:38:55.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:55.414 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2022-07-21T01:38:55.414 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": { 2022-07-21T01:38:55.414 INFO:teuthology.orchestra.run.smithi003.stdout: "items": 15422, 2022-07-21T01:38:55.415 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes": 2428256 2022-07-21T01:38:55.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:55.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:55.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 10.285050504999999 2022-07-21T01:38:55.416 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:38:55.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7da8053a70 msgr2=0x7f7da8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:55.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7da8053a70 0x7f7da8055f30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f7db401a000 tx=0x7f7db4014000).stop 2022-07-21T01:38:55.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc40b66f0 msgr2=0x7f7dc4141040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:55.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc40b66f0 0x7f7dc4141040 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f7dbc006cc0 tx=0x7f7dbc008040).stop 2022-07-21T01:38:55.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/2515188840 shutdown_connections 2022-07-21T01:38:55.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7dc40b66f0 0x7f7dc4141040 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dc40b5b20 0x7f7dc4146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7da8053a70 0x7f7da8055f30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 --2- 172.21.15.3:0/2515188840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dc405fd90 0x7f7dc41565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/2515188840 >> 172.21.15.3:0/2515188840 conn(0x7f7dc4062530 msgr2=0x7f7dc405ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:55.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/2515188840 shutdown_connections 2022-07-21T01:38:55.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.409+0000 7f7dc8a7f700 1 -- 172.21.15.3:0/2515188840 wait complete. 2022-07-21T01:38:55.421 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:38:55.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:55.442 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:55.460 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:55.483 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:38:55.506 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:38:55.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:55.507 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:55.507 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:55.534 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:38:55.535 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:38:55.536 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:38:55.536 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:38:55.537 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:55.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22678]: fuse finished with error 0 and tester_r 0 2022-07-21T01:38:55.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:55.554 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:55.555 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:55.600 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:38:55.601 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:38:55.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:38:55.720 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:38:55.728 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:38:55.729 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:38:55.729 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:38:55.736 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:38:55.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:38:55.738 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:38:55.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:38:55.739 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:38:55.790 INFO:teuthology.orchestra.run.smithi174.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:38:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:38:55.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:38:55.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.905+0000 7f3f1317b700 1 Processor -- start 2022-07-21T01:38:55.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.905+0000 7f3f1317b700 1 -- start start 2022-07-21T01:38:55.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f0c13e8d0 0x7f3f0c141210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:55.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3f0c0b1180 0x7f3f0c141700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:55.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c0aac20 con 0x7f3f0c0b28b0 2022-07-21T01:38:55.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c140e20 con 0x7f3f0c0b1180 2022-07-21T01:38:55.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c0b2c90 con 0x7f3f0c13e8d0 2022-07-21T01:38:55.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1297a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3f0c0b1180 0x7f3f0c141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:55.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1297a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3f0c0b1180 0x7f3f0c141700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46506/0 (socket says 172.21.15.3:46506) 2022-07-21T01:38:55.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1297a700 1 -- 172.21.15.3:0/551518616 learned_addr learned my addr 172.21.15.3:0/551518616 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:55.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f11978700 1 --2- 172.21.15.3:0/551518616 >> v2:172.21.15.143:3300/0 conn(0x7f3f0c13e8d0 0x7f3f0c141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:55.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1297a700 1 -- 172.21.15.3:0/551518616 >> v2:172.21.15.143:3300/0 conn(0x7f3f0c13e8d0 msgr2=0x7f3f0c141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:55.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1297a700 1 --2- 172.21.15.3:0/551518616 >> v2:172.21.15.143:3300/0 conn(0x7f3f0c13e8d0 0x7f3f0c141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1297a700 1 -- 172.21.15.3:0/551518616 >> v1:172.21.15.143:6789/0 conn(0x7f3f0c0b28b0 legacy=0x7f3f0c13e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:55.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1297a700 1 -- 172.21.15.3:0/551518616 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c0aa440 con 0x7f3f0c0b1180 2022-07-21T01:38:55.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f11978700 1 --2- 172.21.15.3:0/551518616 >> v2:172.21.15.143:3300/0 conn(0x7f3f0c13e8d0 0x7f3f0c141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:55.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1297a700 1 --2- 172.21.15.3:0/551518616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 0x7f3f0c141700 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f3f00002f70 tx=0x7f3f00006eb0).ready entity=mon.0 client_cookie=8a89237a38ce08e9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:55.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f11177700 1 -- 172.21.15.3:0/551518616 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f0000d610 con 0x7f3f0c0b1180 2022-07-21T01:38:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f11177700 1 -- 172.21.15.3:0/551518616 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f00000e90 con 0x7f3f0c0b1180 2022-07-21T01:38:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f11177700 1 -- 172.21.15.3:0/551518616 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f0000d9a0 con 0x7f3f0c0b1180 2022-07-21T01:38:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 -- 172.21.15.3:0/551518616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 msgr2=0x7f3f0c141700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 --2- 172.21.15.3:0/551518616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 0x7f3f0c141700 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f3f00002f70 tx=0x7f3f00006eb0).stop 2022-07-21T01:38:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 -- 172.21.15.3:0/551518616 shutdown_connections 2022-07-21T01:38:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 --2- 172.21.15.3:0/551518616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 0x7f3f0c141700 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 --2- 172.21.15.3:0/551518616 >> v2:172.21.15.143:3300/0 conn(0x7f3f0c13e8d0 0x7f3f0c141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 -- 172.21.15.3:0/551518616 >> 172.21.15.3:0/551518616 conn(0x7f3f0c060310 msgr2=0x7f3f0c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:55.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 -- 172.21.15.3:0/551518616 shutdown_connections 2022-07-21T01:38:55.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.909+0000 7f3f1317b700 1 -- 172.21.15.3:0/551518616 wait complete. 2022-07-21T01:38:55.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f1317b700 1 Processor -- start 2022-07-21T01:38:55.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f1317b700 1 -- start start 2022-07-21T01:38:55.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f1317b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 0x7f3f0c0643d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:55.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f1317b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0c0b28b0 0x7f3f0c0648c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:55.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f1317b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c143d50 0x7f3f0c1441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:55.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f1317b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c064db0 con 0x7f3f0c0b1180 2022-07-21T01:38:55.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f1317b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c1430d0 con 0x7f3f0c143d50 2022-07-21T01:38:55.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f1317b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c143300 con 0x7f3f0c0b28b0 2022-07-21T01:38:55.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f12179700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 0x7f3f0c0643d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:55.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f12179700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 0x7f3f0c0643d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46508/0 (socket says 172.21.15.3:46508) 2022-07-21T01:38:55.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f12179700 1 -- 172.21.15.3:0/3734025054 learned_addr learned my addr 172.21.15.3:0/3734025054 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:55.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f11978700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0c0b28b0 0x7f3f0c0648c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:55.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.913+0000 7f3f1297a700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c143d50 0x7f3f0c1441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:55.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3f12179700 1 -- 172.21.15.3:0/3734025054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0c0b28b0 msgr2=0x7f3f0c0648c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:55.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3f12179700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0c0b28b0 0x7f3f0c0648c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3f12179700 1 -- 172.21.15.3:0/3734025054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c143d50 msgr2=0x7f3f0c1441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:55.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3f12179700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c143d50 0x7f3f0c1441d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:55.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3f12179700 1 -- 172.21.15.3:0/3734025054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0000b1e0 con 0x7f3f0c0b1180 2022-07-21T01:38:55.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3f12179700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 0x7f3f0c0643d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f3f0c064ef0 tx=0x7f3f04007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:55.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3eff7fe700 1 -- 172.21.15.3:0/3734025054 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f04012400 con 0x7f3f0c0b1180 2022-07-21T01:38:55.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3eff7fe700 1 -- 172.21.15.3:0/3734025054 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f04012560 con 0x7f3f0c0b1180 2022-07-21T01:38:55.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c1435a0 con 0x7f3f0c0b1180 2022-07-21T01:38:55.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3eff7fe700 1 -- 172.21.15.3:0/3734025054 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f040128c0 con 0x7f3f0c0b1180 2022-07-21T01:38:55.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c142de0 con 0x7f3f0c0b1180 2022-07-21T01:38:55.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c143730 con 0x7f3f0c0b1180 2022-07-21T01:38:55.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3eff7fe700 1 -- 172.21.15.3:0/3734025054 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3f04010070 con 0x7f3f0c0b1180 2022-07-21T01:38:55.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.917+0000 7f3eff7fe700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3eec053950 0x7f3eec055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:55.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.921+0000 7f3eff7fe700 1 -- 172.21.15.3:0/3734025054 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3f0406adf0 con 0x7f3f0c0b1180 2022-07-21T01:38:55.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.921+0000 7f3f11978700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3eec053950 0x7f3eec055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:55.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.921+0000 7f3f11978700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3eec053950 0x7f3eec055e10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3f0800d000 tx=0x7f3f08008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:55.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:55.925+0000 7f3eff7fe700 1 -- 172.21.15.3:0/3734025054 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f04041bb0 con 0x7f3f0c0b1180 2022-07-21T01:38:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.049+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f3f0c0aac20 con 0x7f3f0c0b1180 2022-07-21T01:38:56.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3eff7fe700 1 -- 172.21.15.3:0/3734025054 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v68) v1 ==== 81+0+34040 (secure 0 0 0) 0x7f3f0c0aac20 con 0x7f3f0c0b1180 2022-07-21T01:38:56.051 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:56.052 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:38:56.052 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 68, 2022-07-21T01:38:56.053 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:38:56.053 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:38:56.054 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:38:27.812077+0000", 2022-07-21T01:38:56.054 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:38:56.055 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:38:56.055 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:38:56.056 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:38:56.056 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:38:56.057 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:38:56.057 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:38:56.058 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:38:56.058 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:38:56.058 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:56.059 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:38:56.059 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:38:56.059 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:38:56.060 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:38:56.060 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:38:56.060 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 12, 2022-07-21T01:38:56.061 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:38:56.061 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:38:56.061 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:38:56.061 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:38:56.062 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:38:56.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.062 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:38:56.063 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:38:56.063 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:38:56.063 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:56.063 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:56.064 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:56.064 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:56.064 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:56.064 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:38:56.066 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:38:56.066 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:38:56.066 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:38:56.067 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:38:56.067 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:56.067 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:56.068 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:56.068 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:56.068 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:56.068 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:56.069 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:56.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.069 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:38:56.070 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:56.070 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:56.071 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:56.071 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:56.071 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:56.072 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:56.072 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:56.072 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:56.073 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:56.073 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:56.073 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:56.074 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:56.074 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:56.074 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:56.074 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:56.075 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:56.075 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:56.075 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:56.076 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:56.076 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:56.076 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:56.077 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:56.077 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:56.077 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:56.077 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:56.078 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:56.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.079 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:56.079 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:56.080 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:56.080 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:56.080 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:56.081 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:56.081 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:56.081 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:56.082 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:56.082 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:56.082 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:56.082 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:38:56.083 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:38:56.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.083 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:56.084 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:38:56.084 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.085 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:38:56.085 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:38:56.085 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:38:56.085 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:56.086 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:56.086 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:38:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:38:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:38:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:38:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:38:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:56.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:38:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:38:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:38:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 11, 2022-07-21T01:38:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:38:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:38:25.308080+0000", 2022-07-21T01:38:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:38:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:38:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:38:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:38:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:38:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "67", 2022-07-21T01:38:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:38:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:38:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:38:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:38:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:38:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:38:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 12, 2022-07-21T01:38:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:38:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:38:26.095494+0000", 2022-07-21T01:38:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:38:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:38:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:38:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:38:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:38:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:38:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:38:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:38:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:38:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:38:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:38:56.128 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:38:56.128 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:38:56.128 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:38:56.128 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:38:56.129 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:38:56.129 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:38:56.129 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:38:56.129 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:38:56.130 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:38:56.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.130 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2022-07-21T01:38:56.130 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:38:56.131 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:38:56.131 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:38:56.131 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:38:56.132 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:38:56.132 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:38:56.132 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:38:56.133 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:38:56.133 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:38:56.133 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:38:56.133 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:38:56.134 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:38:56.134 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:38:56.134 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:38:56.134 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:38:56.135 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:38:56.135 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:38:56.135 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:38:56.135 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:38:56.136 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:38:56.136 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:38:56.136 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:38:56.136 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:38:56.137 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:38:56.137 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:38:56.137 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:38:56.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.138 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:38:56.138 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:38:56.138 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:38:56.138 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:38:56.139 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:38:56.139 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:38:56.139 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:38:56.139 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:38:56.140 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:38:56.140 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:38:56.140 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:38:56.141 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:38:56.141 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:38:56.142 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:38:56.142 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:38:56.142 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.143 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.143 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.143 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:56.143 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:38:56.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.144 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:38:56.144 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:38:56.144 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:56.145 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:56.145 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:56.145 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:56.146 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:56.146 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:56.146 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:38:56.146 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 65, 2022-07-21T01:38:56.147 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:56.147 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:56.147 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:56.147 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.148 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.148 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:38:56.149 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:56.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.149 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.150 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:38:56.150 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:56.150 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.151 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.151 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:56.151 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.152 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.152 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:38:56.152 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:56.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.153 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.154 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:38:56.154 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:56.154 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.154 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.155 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:56.155 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.156 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.156 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:38:56.157 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:56.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.157 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.158 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:38:56.158 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:56.158 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.159 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.159 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:56.159 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.160 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.160 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:38:56.160 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:56.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.161 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:38:56.162 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:38:56.162 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.162 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.163 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:38:56.163 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:38:56.163 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:38:56.164 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:38:56.164 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:56.164 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:38:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:38:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:56.167 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:56.167 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:56.167 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:56.168 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:56.168 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:56.168 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 65, 2022-07-21T01:38:56.168 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:56.169 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:56.169 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:56.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.170 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.170 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:38:56.170 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:56.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.171 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.172 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:38:56.172 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:56.172 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.173 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.173 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:56.173 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.174 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.174 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:38:56.175 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:56.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.176 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.176 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:38:56.176 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:56.176 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.177 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:56.178 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.179 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:38:56.179 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:56.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.180 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.180 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:38:56.180 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:56.181 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.181 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.181 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:56.182 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.182 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.183 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:38:56.183 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:56.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.184 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:38:56.184 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:38:56.184 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.185 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.185 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:38:56.185 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:38:56.186 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:38:56.186 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:38:56.186 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:56.186 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:56.187 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:56.187 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.188 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:38:56.188 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:38:56.188 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:56.189 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:56.189 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:56.189 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:56.189 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:56.190 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:56.190 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:38:56.190 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 64, 2022-07-21T01:38:56.190 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:56.191 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:56.191 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:56.191 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.192 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.192 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:38:56.192 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:56.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.193 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.194 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:38:56.194 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:56.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.194 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.195 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:56.195 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.196 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.196 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:38:56.196 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:56.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.197 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:38:56.197 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:56.198 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.198 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.198 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:56.199 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.199 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.199 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:38:56.200 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:56.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.201 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.201 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:38:56.201 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:56.201 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.201 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.202 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:56.202 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.203 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.203 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:38:56.203 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:56.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.204 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.204 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:38:56.205 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:38:56.205 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.205 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.206 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:38:56.206 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:38:56.206 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:38:56.207 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:38:56.207 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:56.207 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:56.207 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:56.208 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.208 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:38:56.209 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:38:56.209 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:56.209 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:56.209 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:56.210 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:56.210 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:56.210 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:56.210 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:38:56.211 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 55, 2022-07-21T01:38:56.211 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:56.211 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:56.211 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:56.212 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.212 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.212 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:38:56.213 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:56.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.213 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.214 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:38:56.214 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:56.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.215 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.215 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:56.215 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.216 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.216 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:38:56.216 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:56.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.217 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.217 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:38:56.218 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:56.218 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.218 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.219 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:56.219 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.219 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.220 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:38:56.220 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:56.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.221 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.221 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:38:56.221 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:56.221 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.222 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.222 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:56.223 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.223 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.223 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:38:56.224 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:56.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.224 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.225 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:38:56.225 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:38:56.225 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.226 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.226 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:38:56.226 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:38:56.227 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:38:56.227 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:38:56.227 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:56.228 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:56.228 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:56.228 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.229 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:38:56.229 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:38:56.229 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:56.230 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:56.230 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:56.230 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:56.231 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:56.231 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:56.231 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:56.231 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 66, 2022-07-21T01:38:56.232 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:56.232 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:56.232 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:56.233 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.233 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.233 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:38:56.234 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:56.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.234 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.235 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:38:56.235 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:56.235 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.236 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.236 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:56.236 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.237 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.237 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:38:56.237 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:56.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.238 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.238 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:38:56.238 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:56.239 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.239 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.240 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:56.240 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.240 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.241 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:38:56.241 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:56.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.242 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.242 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:38:56.242 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:56.242 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.243 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.243 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:56.243 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.244 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.244 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:38:56.245 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:56.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.245 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.246 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:38:56.246 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:38:56.246 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.246 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.247 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:38:56.247 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:38:56.247 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:38:56.248 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:38:56.248 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:56.248 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:56.248 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:56.249 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.249 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:38:56.250 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:38:56.250 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:56.250 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:56.250 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:56.250 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:56.251 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:56.251 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:56.251 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:56.251 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 64, 2022-07-21T01:38:56.252 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:56.252 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:56.252 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:56.252 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.253 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.253 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:38:56.253 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:56.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.254 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.254 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:38:56.254 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:56.255 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.255 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.255 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:56.256 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.256 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.257 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:38:56.257 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:56.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.257 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.258 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:38:56.258 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:56.258 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.259 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.259 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:56.259 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.260 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.260 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:38:56.260 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:56.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.261 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.261 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:38:56.262 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:56.262 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.262 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.263 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:56.263 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.263 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.264 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:38:56.264 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:56.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.265 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.265 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:38:56.265 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:38:56.266 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.266 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.266 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:38:56.267 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:38:56.267 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:38:56.267 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:38:56.267 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:56.268 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:56.268 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:56.268 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.269 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:38:56.269 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:38:56.270 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:56.270 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:56.270 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:56.270 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:56.271 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:56.271 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:56.271 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:56.271 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 65, 2022-07-21T01:38:56.272 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:56.272 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:56.272 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:56.272 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.273 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.273 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:38:56.274 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:56.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.274 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.275 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:38:56.275 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:56.275 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.276 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.276 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:56.276 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.277 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.277 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:38:56.278 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:56.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.278 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.279 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:38:56.279 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:56.279 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.279 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.280 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:56.280 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.281 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.281 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:38:56.281 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:56.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.283 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.283 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:38:56.283 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:56.283 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.284 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.284 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:56.285 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.285 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.285 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:38:56.286 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:56.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.287 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.287 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:38:56.287 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:38:56.287 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.288 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.288 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:38:56.288 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:38:56.289 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:38:56.289 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:38:56.289 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:56.289 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:56.290 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:56.290 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.291 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:38:56.291 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:38:56.291 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:38:56.292 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:38:56.292 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:38:56.292 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:38:56.292 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:38:56.293 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:38:56.293 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:38:56.293 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 64, 2022-07-21T01:38:56.293 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:38:56.294 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:38:56.294 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:38:56.294 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.295 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.295 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:38:56.295 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:56.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.296 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.296 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:38:56.296 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:56.297 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.297 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.297 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:38:56.298 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.298 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.298 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:38:56.299 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:56.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.299 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.300 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:38:56.300 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:56.300 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.300 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.301 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:38:56.301 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.302 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.302 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:38:56.302 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:56.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.303 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.303 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:38:56.303 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:56.304 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.304 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.304 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:38:56.305 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:38:56.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.305 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:38:56.305 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:38:56.306 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:56.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.306 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:38:56.307 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:38:56.307 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:38:56.307 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.307 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.308 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:38:56.308 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:38:56.309 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:38:56.309 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:38:56.309 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:38:56.309 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:38:56.310 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:38:56.310 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:38:56.310 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.310 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:56.311 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:38:56.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.311 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:38:56.311 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:56.312 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:56.312 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:56.312 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:56.312 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:56.313 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:38:56.313 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:56.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.314 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:38:56.314 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:56.314 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:56.314 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:56.315 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:56.315 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:56.315 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:38:56.315 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:56.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.316 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:38:56.316 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:56.317 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:56.317 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:56.317 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:56.317 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:56.318 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:38:56.318 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:56.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.319 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:38:56.319 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:56.319 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:56.320 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:56.320 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:56.320 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:56.320 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:38:56.321 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:56.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.321 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:38:56.322 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:56.322 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:56.322 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:56.322 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:56.323 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:56.323 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:38:56.323 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:56.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.324 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:38:56.324 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:56.324 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:56.325 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:56.325 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:56.325 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:56.325 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:38:56.326 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:56.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.327 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:38:56.327 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:56.327 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:56.327 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:56.328 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:56.328 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:56.328 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:38:56.328 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:56.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:38:56.329 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:38:56.329 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:38:56.330 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:38:56.330 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:38:56.330 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:38:56.330 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:38:56.331 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:38:56.331 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:38:56.331 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.331 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:38:56.332 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:38:56.332 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:38:56.332 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:38:56.332 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:38:56.333 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:38:56.333 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:38:56.333 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:38:56.333 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:38:56.334 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:38:56.334 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:38:56.334 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:38:56.335 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:38:56.335 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:38:56.335 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:38:56.335 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:38:56.336 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:38:56.336 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:38:56.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.336 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:38:56.337 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:38:56.337 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:38:56.337 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:38:56.337 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:38:56.338 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:38:56.338 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:38:56.338 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:38:56.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:38:56.339 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:38:56.339 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:38:56.339 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:38:56.339 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:38:56.340 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:38:56.340 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:38:56.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3eec053950 msgr2=0x7f3eec055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3eec053950 0x7f3eec055e10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3f0800d000 tx=0x7f3f08008040).stop 2022-07-21T01:38:56.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 msgr2=0x7f3f0c0643d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 0x7f3f0c0643d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f3f0c064ef0 tx=0x7f3f04007c50).stop 2022-07-21T01:38:56.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 shutdown_connections 2022-07-21T01:38:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3eec053950 0x7f3eec055e10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c143d50 0x7f3f0c1441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0c0b28b0 0x7f3f0c0648c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 --2- 172.21.15.3:0/3734025054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f0c0b1180 0x7f3f0c0643d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 >> 172.21.15.3:0/3734025054 conn(0x7f3f0c060310 msgr2=0x7f3f0c141130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:56.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 shutdown_connections 2022-07-21T01:38:56.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.053+0000 7f3f1317b700 1 -- 172.21.15.3:0/3734025054 wait complete. 2022-07-21T01:38:56.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:56.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f901a825700 1 Processor -- start 2022-07-21T01:38:56.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f901a825700 1 -- start start 2022-07-21T01:38:56.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f901a825700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90140b1160 0x7f90141448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f901a825700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90140b04e0 con 0x7f9014148530 2022-07-21T01:38:56.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f901a825700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90140b0780 con 0x7f90140b1f40 2022-07-21T01:38:56.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f901a825700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90140a9250 con 0x7f90140b1160 2022-07-21T01:38:56.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f90137fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9014148530 0x7f90140a7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54798/0 (socket says 172.21.15.3:54798) 2022-07-21T01:38:56.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f90137fe700 1 -- 172.21.15.3:0/1189857793 learned_addr learned my addr 172.21.15.3:0/1189857793 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:56.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9018dc2700 1 --2- 172.21.15.3:0/1189857793 >> v2:172.21.15.143:3300/0 conn(0x7f90140b1160 0x7f90141448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080892315 0 0) 0x7f90140b04e0 con 0x7f9014148530 2022-07-21T01:38:56.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff8003260 con 0x7f9014148530 2022-07-21T01:38:56.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395243115 0 0) 0x7f90140b0780 con 0x7f90140b1f40 2022-07-21T01:38:56.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90140b04e0 con 0x7f90140b1f40 2022-07-21T01:38:56.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1851755389 0 0) 0x7f8ff8003260 con 0x7f9014148530 2022-07-21T01:38:56.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 >> v2:172.21.15.143:3300/0 conn(0x7f90140b1160 msgr2=0x7f90141448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:56.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 --2- 172.21.15.3:0/1189857793 >> v2:172.21.15.143:3300/0 conn(0x7f90140b1160 0x7f90141448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 >> v1:172.21.15.143:6789/0 conn(0x7f90140b1f40 legacy=0x7f90140b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90140b0a20 con 0x7f9014148530 2022-07-21T01:38:56.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f90040019e0 con 0x7f9014148530 2022-07-21T01:38:56.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 >> v1:172.21.15.3:6789/0 conn(0x7f9014148530 legacy=0x7f90140a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ff80038e0 0x7f8ff8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ff8004500 0x7f8ff8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004e70 0x7f8ff800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff800bcf0 con 0x7f8ff8004500 2022-07-21T01:38:56.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff800be30 con 0x7f8ff8004e70 2022-07-21T01:38:56.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff800bff0 con 0x7f8ff80038e0 2022-07-21T01:38:56.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9018dc2700 1 --2- 172.21.15.3:0/1189857793 >> v2:172.21.15.143:3300/0 conn(0x7f90140b1160 0x7f90141448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:56.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f90137fe700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ff8004500 0x7f8ff8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9013fff700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ff80038e0 0x7f8ff8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9018dc2700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004e70 0x7f8ff800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f90137fe700 1 -- 172.21.15.3:0/1189857793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ff80038e0 msgr2=0x7f8ff8003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:56.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f90137fe700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ff80038e0 0x7f8ff8003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f90137fe700 1 -- 172.21.15.3:0/1189857793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004e70 msgr2=0x7f8ff800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:56.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f90137fe700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004e70 0x7f8ff800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f90137fe700 1 -- 172.21.15.3:0/1189857793 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff8003260 con 0x7f8ff8004500 2022-07-21T01:38:56.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9018dc2700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004e70 0x7f8ff800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:56.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.461+0000 7f9018dc2700 1 -- 172.21.15.3:0/1189857793 reap_dead start 2022-07-21T01:38:56.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f90137fe700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ff8004500 0x7f8ff8004980 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f90040017b0 tx=0x7f90140b0910).ready entity=mon.0 client_cookie=4546dd0a4999508c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:56.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f900400ccb0 con 0x7f8ff8004500 2022-07-21T01:38:56.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900400ce10 con 0x7f8ff8004500 2022-07-21T01:38:56.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f9012ffd700 1 -- 172.21.15.3:0/1189857793 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9004005b80 con 0x7f8ff8004500 2022-07-21T01:38:56.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f901a825700 1 -- 172.21.15.3:0/1189857793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ff8004500 msgr2=0x7f8ff8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f901a825700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ff8004500 0x7f8ff8004980 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f90040017b0 tx=0x7f90140b0910).stop 2022-07-21T01:38:56.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f901a825700 1 -- 172.21.15.3:0/1189857793 shutdown_connections 2022-07-21T01:38:56.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f901a825700 1 --2- 172.21.15.3:0/1189857793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ff8004500 0x7f8ff8004980 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f901a825700 1 -- 172.21.15.3:0/1189857793 >> 172.21.15.3:0/1189857793 conn(0x7f901405f840 msgr2=0x7f901405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:56.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f901a825700 1 -- 172.21.15.3:0/1189857793 shutdown_connections 2022-07-21T01:38:56.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f901a825700 1 -- 172.21.15.3:0/1189857793 wait complete. 2022-07-21T01:38:56.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.465+0000 7f901a825700 1 Processor -- start 2022-07-21T01:38:56.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f901a825700 1 -- start start 2022-07-21T01:38:56.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f901a825700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004500 0x7f901414d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f901a825700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90140af4b0 0x7f9014145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f901a825700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90140af990 0x7f9014156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f901a825700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9014145c20 con 0x7f90140af990 2022-07-21T01:38:56.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f901a825700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9014145e40 con 0x7f8ff8004500 2022-07-21T01:38:56.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f901a825700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90140aead0 con 0x7f90140af4b0 2022-07-21T01:38:56.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9018dc2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90140af990 0x7f9014156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9018dc2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90140af990 0x7f9014156990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46532/0 (socket says 172.21.15.3:46532) 2022-07-21T01:38:56.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9018dc2700 1 -- 172.21.15.3:0/623695465 learned_addr learned my addr 172.21.15.3:0/623695465 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:56.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9013fff700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004500 0x7f901414d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f90137fe700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90140af4b0 0x7f9014145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9018dc2700 1 -- 172.21.15.3:0/623695465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90140af4b0 msgr2=0x7f9014145730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9018dc2700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90140af4b0 0x7f9014145730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9018dc2700 1 -- 172.21.15.3:0/623695465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004500 msgr2=0x7f901414d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9018dc2700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004500 0x7f901414d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9018dc2700 1 -- 172.21.15.3:0/623695465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90140aed70 con 0x7f90140af990 2022-07-21T01:38:56.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9013fff700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004500 0x7f901414d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:56.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f9018dc2700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90140af990 0x7f9014156990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f9014145d60 tx=0x7f900c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:56.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f90117fa700 1 -- 172.21.15.3:0/623695465 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f900c0126e0 con 0x7f90140af990 2022-07-21T01:38:56.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f90117fa700 1 -- 172.21.15.3:0/623695465 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c012840 con 0x7f90140af990 2022-07-21T01:38:56.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90140af010 con 0x7f90140af990 2022-07-21T01:38:56.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f90117fa700 1 -- 172.21.15.3:0/623695465 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f900c012ba0 con 0x7f90140af990 2022-07-21T01:38:56.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90141570a0 con 0x7f90140af990 2022-07-21T01:38:56.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.469+0000 7f90117fa700 1 -- 172.21.15.3:0/623695465 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f900c01c030 con 0x7f90140af990 2022-07-21T01:38:56.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.473+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90140af1a0 con 0x7f90140af990 2022-07-21T01:38:56.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.473+0000 7f90117fa700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fec053b00 0x7f8fec055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.473+0000 7f90117fa700 1 -- 172.21.15.3:0/623695465 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f900c06bda0 con 0x7f90140af990 2022-07-21T01:38:56.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.473+0000 7f9013fff700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fec053b00 0x7f8fec055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.473+0000 7f9013fff700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fec053b00 0x7f8fec055fc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f8ffc003000 tx=0x7f8ffc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:56.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.477+0000 7f90117fa700 1 -- 172.21.15.3:0/623695465 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f900c042a50 con 0x7f90140af990 2022-07-21T01:38:56.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.637+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90140b0cc0 con 0x7f90140af990 2022-07-21T01:38:56.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f90117fa700 1 -- 172.21.15.3:0/623695465 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2490 (secure 0 0 0) 0x7f900c046070 con 0x7f90140af990 2022-07-21T01:38:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:56.639 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":29,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:38:26.808059+0000","modified":"2022-07-21T01:38:27.811394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5934":{"gid":5934,"name":"b","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/2524169670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2524169670},{"type":"v1","addr":"172.21.15.143:6833","nonce":2524169670}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-21T01:38:56.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fec053b00 msgr2=0x7f8fec055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fec053b00 0x7f8fec055fc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f8ffc003000 tx=0x7f8ffc009040).stop 2022-07-21T01:38:56.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90140af990 msgr2=0x7f9014156990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90140af990 0x7f9014156990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f9014145d60 tx=0x7f900c008040).stop 2022-07-21T01:38:56.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 shutdown_connections 2022-07-21T01:38:56.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90140af990 0x7f9014156990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90140af4b0 0x7f9014145730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fec053b00 0x7f8fec055fc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 --2- 172.21.15.3:0/623695465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ff8004500 0x7f901414d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 >> 172.21.15.3:0/623695465 conn(0x7f901405f840 msgr2=0x7f90140b4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:56.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 shutdown_connections 2022-07-21T01:38:56.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.641+0000 7f901a825700 1 -- 172.21.15.3:0/623695465 wait complete. 2022-07-21T01:38:56.644 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 29 2022-07-21T01:38:56.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:38:56.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c67fe1700 1 Processor -- start 2022-07-21T01:38:56.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c67fe1700 1 -- start start 2022-07-21T01:38:56.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c67fe1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2c600b79a0 0x7f2c60144730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c67fe1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c60144c20 0x7f2c60065110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c67fe1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c600b3360 con 0x7f2c600b1300 2022-07-21T01:38:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c67fe1700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c60145080 con 0x7f2c600b79a0 2022-07-21T01:38:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c67fe1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c600b7d80 con 0x7f2c60144c20 2022-07-21T01:38:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c65d7d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2c600b79a0 0x7f2c60144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c65d7d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2c600b79a0 0x7f2c60144730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46536/0 (socket says 172.21.15.3:46536) 2022-07-21T01:38:56.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c65d7d700 1 -- 172.21.15.3:0/1661597497 learned_addr learned my addr 172.21.15.3:0/1661597497 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:56.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c6657e700 1 --2- 172.21.15.3:0/1661597497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c60144c20 0x7f2c60065110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c65d7d700 1 -- 172.21.15.3:0/1661597497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c60144c20 msgr2=0x7f2c60065110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:56.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c65d7d700 1 --2- 172.21.15.3:0/1661597497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c60144c20 0x7f2c60065110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c65d7d700 1 -- 172.21.15.3:0/1661597497 >> v1:172.21.15.143:6789/0 conn(0x7f2c600b1300 legacy=0x7f2c60064a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c65d7d700 1 -- 172.21.15.3:0/1661597497 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60067750 con 0x7f2c600b79a0 2022-07-21T01:38:56.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c65d7d700 1 --2- 172.21.15.3:0/1661597497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 0x7f2c60144730 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f2c5c002f70 tx=0x7f2c5c006eb0).ready entity=mon.0 client_cookie=a354a84c089a3eb2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:56.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c64d7b700 1 -- 172.21.15.3:0/1661597497 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c5c00d5c0 con 0x7f2c600b79a0 2022-07-21T01:38:56.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c64d7b700 1 -- 172.21.15.3:0/1661597497 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c000e90 con 0x7f2c600b79a0 2022-07-21T01:38:56.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.765+0000 7f2c64d7b700 1 -- 172.21.15.3:0/1661597497 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c5c00d950 con 0x7f2c600b79a0 2022-07-21T01:38:56.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1661597497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 msgr2=0x7f2c60144730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 --2- 172.21.15.3:0/1661597497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 0x7f2c60144730 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f2c5c002f70 tx=0x7f2c5c006eb0).stop 2022-07-21T01:38:56.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1661597497 shutdown_connections 2022-07-21T01:38:56.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 --2- 172.21.15.3:0/1661597497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c60144c20 0x7f2c60065110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 --2- 172.21.15.3:0/1661597497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 0x7f2c60144730 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1661597497 >> 172.21.15.3:0/1661597497 conn(0x7f2c6005f840 msgr2=0x7f2c6005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:56.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1661597497 shutdown_connections 2022-07-21T01:38:56.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1661597497 wait complete. 2022-07-21T01:38:56.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 Processor -- start 2022-07-21T01:38:56.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 -- start start 2022-07-21T01:38:56.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c600b1300 0x7f2c60142e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 0x7f2c600b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60144c20 0x7f2c601538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c600ad500 con 0x7f2c600b79a0 2022-07-21T01:38:56.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c600afdd0 con 0x7f2c60144c20 2022-07-21T01:38:56.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c67fe1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c600ad730 con 0x7f2c600b1300 2022-07-21T01:38:56.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6557c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 0x7f2c600b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6557c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 0x7f2c600b04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46544/0 (socket says 172.21.15.3:46544) 2022-07-21T01:38:56.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6557c700 1 -- 172.21.15.3:0/1600016896 learned_addr learned my addr 172.21.15.3:0/1600016896 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:56.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c65d7d700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c600b1300 0x7f2c60142e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6657e700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60144c20 0x7f2c601538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6557c700 1 -- 172.21.15.3:0/1600016896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c600b1300 msgr2=0x7f2c60142e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6557c700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c600b1300 0x7f2c60142e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6557c700 1 -- 172.21.15.3:0/1600016896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60144c20 msgr2=0x7f2c601538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:56.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6557c700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60144c20 0x7f2c601538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:56.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6557c700 1 -- 172.21.15.3:0/1600016896 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c00b1e0 con 0x7f2c600b79a0 2022-07-21T01:38:56.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c65d7d700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c600b1300 0x7f2c60142e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:38:56.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.769+0000 7f2c6557c700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 0x7f2c600b04a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f2c60067e30 tx=0x7f2c50007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:56.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c56ffd700 1 -- 172.21.15.3:0/1600016896 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c5000ce30 con 0x7f2c600b79a0 2022-07-21T01:38:56.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c56ffd700 1 -- 172.21.15.3:0/1600016896 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5000f470 con 0x7f2c600b79a0 2022-07-21T01:38:56.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c600b0000 con 0x7f2c600b79a0 2022-07-21T01:38:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c56ffd700 1 -- 172.21.15.3:0/1600016896 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c5000f7d0 con 0x7f2c600b79a0 2022-07-21T01:38:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c60153dc0 con 0x7f2c600b79a0 2022-07-21T01:38:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c600b0190 con 0x7f2c600b79a0 2022-07-21T01:38:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c56ffd700 1 -- 172.21.15.3:0/1600016896 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2c50019030 con 0x7f2c600b79a0 2022-07-21T01:38:56.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c56ffd700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c48053b00 0x7f2c48055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:56.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c65d7d700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c48053b00 0x7f2c48055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:56.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.773+0000 7f2c56ffd700 1 -- 172.21.15.3:0/1600016896 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2c5006b4a0 con 0x7f2c600b79a0 2022-07-21T01:38:56.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.777+0000 7f2c65d7d700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c48053b00 0x7f2c48055fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2c5c01a000 tx=0x7f2c5c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:56.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.777+0000 7f2c56ffd700 1 -- 172.21.15.3:0/1600016896 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2c50042200 con 0x7f2c600b79a0 2022-07-21T01:38:56.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:56.941+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2c600ab000 con 0x7f2c600b79a0 2022-07-21T01:38:57.485 INFO:tasks.ceph.mds.b.smithi143.stderr:2022-07-21T01:38:57.489+0000 7fc30c8c7700 -1 MDSIOContextBase: failed with -108, restarting... 2022-07-21T01:38:57.506 INFO:tasks.ceph.mds.b.smithi143.stderr: -1> 2022-07-21T01:38:57.489+0000 7fc30c8c7700 -1 MDSIOContextBase: failed with -108, restarting... 2022-07-21T01:38:57.528 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:38:57.709 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:38:57.709+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:38:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c56ffd700 1 -- 172.21.15.3:0/1600016896 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2c50045820 con 0x7f2c600b79a0 2022-07-21T01:38:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c48053b00 msgr2=0x7f2c48055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:57.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c48053b00 0x7f2c48055fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2c5c01a000 tx=0x7f2c5c015000).stop 2022-07-21T01:38:57.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 msgr2=0x7f2c600b04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:57.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 0x7f2c600b04a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f2c60067e30 tx=0x7f2c50007c50).stop 2022-07-21T01:38:57.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 shutdown_connections 2022-07-21T01:38:57.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2c48053b00 0x7f2c48055fc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:57.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60144c20 0x7f2c601538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:57.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c600b79a0 0x7f2c600b04a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:57.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 --2- 172.21.15.3:0/1600016896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c600b1300 0x7f2c60142e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:57.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 >> 172.21.15.3:0/1600016896 conn(0x7f2c6005f840 msgr2=0x7f2c60146020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:57.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.725+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 shutdown_connections 2022-07-21T01:38:57.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.729+0000 7f2c67fe1700 1 -- 172.21.15.3:0/1600016896 wait complete. 2022-07-21T01:38:57.730 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:38:57.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:38:57.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.849+0000 7f2d9e52f700 1 Processor -- start 2022-07-21T01:38:57.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.849+0000 7f2d9e52f700 1 -- start start 2022-07-21T01:38:57.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 0x7f2d980ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:57.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980b2c80 con 0x7f2d980b0630 2022-07-21T01:38:57.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980b2900 con 0x7f2d980b0a10 2022-07-21T01:38:57.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d980b2eb0 con 0x7f2d980aa2b0 2022-07-21T01:38:57.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d977fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 0x7f2d980ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:57.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d97fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2d980b0630 0x7f2d980adec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54832/0 (socket says 172.21.15.3:54832) 2022-07-21T01:38:57.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d97fff700 1 -- 172.21.15.3:0/3516816465 learned_addr learned my addr 172.21.15.3:0/3516816465 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:57.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9cacc700 1 --1- 172.21.15.3:0/3516816465 >> v1:172.21.15.143:6789/0 conn(0x7f2d980b0a10 0x7f2d98140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:56950/0 (socket says 172.21.15.3:56950) 2022-07-21T01:38:57.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d96ffd700 1 -- 172.21.15.3:0/3516816465 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685158807 0 0) 0x7f2d980b2c80 con 0x7f2d980b0630 2022-07-21T01:38:57.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d96ffd700 1 -- 172.21.15.3:0/3516816465 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d84003260 con 0x7f2d980b0630 2022-07-21T01:38:57.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d96ffd700 1 -- 172.21.15.3:0/3516816465 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069888760 0 0) 0x7f2d980b2900 con 0x7f2d980b0a10 2022-07-21T01:38:57.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d96ffd700 1 -- 172.21.15.3:0/3516816465 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d980b2c80 con 0x7f2d980b0a10 2022-07-21T01:38:57.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d977fe700 1 -- 172.21.15.3:0/3516816465 >> v1:172.21.15.143:6789/0 conn(0x7f2d980b0a10 legacy=0x7f2d98140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:57.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d977fe700 1 -- 172.21.15.3:0/3516816465 >> v1:172.21.15.3:6789/0 conn(0x7f2d980b0630 legacy=0x7f2d980adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:57.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d977fe700 1 -- 172.21.15.3:0/3516816465 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980a9d00 con 0x7f2d980aa2b0 2022-07-21T01:38:57.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d977fe700 1 --2- 172.21.15.3:0/3516816465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 0x7f2d980ae5d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f2d880032f0 tx=0x7f2d880091d0).ready entity=mon.2 client_cookie=35bc43bfdf178feb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:57.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d96ffd700 1 -- 172.21.15.3:0/3516816465 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8800b640 con 0x7f2d980aa2b0 2022-07-21T01:38:57.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d96ffd700 1 -- 172.21.15.3:0/3516816465 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8800b7a0 con 0x7f2d980aa2b0 2022-07-21T01:38:57.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d96ffd700 1 -- 172.21.15.3:0/3516816465 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8800b910 con 0x7f2d980aa2b0 2022-07-21T01:38:57.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3516816465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 msgr2=0x7f2d980ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:57.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 --2- 172.21.15.3:0/3516816465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 0x7f2d980ae5d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f2d880032f0 tx=0x7f2d880091d0).stop 2022-07-21T01:38:57.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3516816465 shutdown_connections 2022-07-21T01:38:57.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 --2- 172.21.15.3:0/3516816465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 0x7f2d980ae5d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:57.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3516816465 >> 172.21.15.3:0/3516816465 conn(0x7f2d98060950 msgr2=0x7f2d98060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:57.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3516816465 shutdown_connections 2022-07-21T01:38:57.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3516816465 wait complete. 2022-07-21T01:38:57.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.853+0000 7f2d9e52f700 1 Processor -- start 2022-07-21T01:38:57.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9e52f700 1 -- start start 2022-07-21T01:38:57.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9e52f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 0x7f2d980b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:57.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9e52f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d980b0630 0x7f2d980a8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:57.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9e52f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d980b0a10 0x7f2d980a8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:57.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9e52f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98066100 con 0x7f2d980b0630 2022-07-21T01:38:57.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9e52f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d980a4200 con 0x7f2d980b0a10 2022-07-21T01:38:57.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9e52f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98065bc0 con 0x7f2d980aa2b0 2022-07-21T01:38:57.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d977fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d980b0630 0x7f2d980a8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:57.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d977fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d980b0630 0x7f2d980a8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46558/0 (socket says 172.21.15.3:46558) 2022-07-21T01:38:57.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d977fe700 1 -- 172.21.15.3:0/3174294874 learned_addr learned my addr 172.21.15.3:0/3174294874 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:57.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d97fff700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 0x7f2d980b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:57.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9cacc700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d980b0a10 0x7f2d980a8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:57.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d977fe700 1 -- 172.21.15.3:0/3174294874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 msgr2=0x7f2d980b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:57.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d977fe700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 0x7f2d980b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:57.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d977fe700 1 -- 172.21.15.3:0/3174294874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d980b0a10 msgr2=0x7f2d980a8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:57.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d977fe700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d980b0a10 0x7f2d980a8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:57.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d977fe700 1 -- 172.21.15.3:0/3174294874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d88009470 con 0x7f2d980b0630 2022-07-21T01:38:57.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d977fe700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d980b0630 0x7f2d980a8680 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f2d880032f0 tx=0x7f2d88016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:57.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d957fa700 1 -- 172.21.15.3:0/3174294874 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8800b910 con 0x7f2d980b0630 2022-07-21T01:38:57.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d957fa700 1 -- 172.21.15.3:0/3174294874 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d88011e00 con 0x7f2d980b0630 2022-07-21T01:38:57.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d98065e60 con 0x7f2d980b0630 2022-07-21T01:38:57.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d957fa700 1 -- 172.21.15.3:0/3174294874 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8800b640 con 0x7f2d980b0630 2022-07-21T01:38:57.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.857+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d980b1690 con 0x7f2d980b0630 2022-07-21T01:38:57.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.861+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d980574e0 con 0x7f2d980b0630 2022-07-21T01:38:57.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.861+0000 7f2d957fa700 1 -- 172.21.15.3:0/3174294874 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2d88026030 con 0x7f2d980b0630 2022-07-21T01:38:57.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.861+0000 7f2d957fa700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d78053b00 0x7f2d78055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:57.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.861+0000 7f2d957fa700 1 -- 172.21.15.3:0/3174294874 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f2d8806ca20 con 0x7f2d980b0630 2022-07-21T01:38:57.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.861+0000 7f2d97fff700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d78053b00 0x7f2d78055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:57.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.861+0000 7f2d97fff700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d78053b00 0x7f2d78055fc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f2d8c00d000 tx=0x7f2d8c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:57.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:57.865+0000 7f2d957fa700 1 -- 172.21.15.3:0/3174294874 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d88043790 con 0x7f2d980b0630 2022-07-21T01:38:58.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.033+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d980447e0 con 0x7f2d980b0630 2022-07-21T01:38:58.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.033+0000 7f2d957fa700 1 -- 172.21.15.3:0/3174294874 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2126 (secure 0 0 0) 0x7f2d88024860 con 0x7f2d980b0630 2022-07-21T01:38:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:38:58.033 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":30,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:38:26.808059+0000","modified":"2022-07-21T01:38:57.712889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-21T01:38:58.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.033+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d78053b00 msgr2=0x7f2d78055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:58.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.033+0000 7f2d9e52f700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d78053b00 0x7f2d78055fc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f2d8c00d000 tx=0x7f2d8c009040).stop 2022-07-21T01:38:58.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.033+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d980b0630 msgr2=0x7f2d980a8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:58.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.033+0000 7f2d9e52f700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d980b0630 0x7f2d980a8680 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f2d880032f0 tx=0x7f2d88016000).stop 2022-07-21T01:38:58.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.037+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 shutdown_connections 2022-07-21T01:38:58.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.037+0000 7f2d9e52f700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2d78053b00 0x7f2d78055fc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.037+0000 7f2d9e52f700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d980b0a10 0x7f2d980a8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.037+0000 7f2d9e52f700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d980b0630 0x7f2d980a8680 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.037+0000 7f2d9e52f700 1 --2- 172.21.15.3:0/3174294874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d980aa2b0 0x7f2d980b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.037+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 >> 172.21.15.3:0/3174294874 conn(0x7f2d98060950 msgr2=0x7f2d980b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:58.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.037+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 shutdown_connections 2022-07-21T01:38:58.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.037+0000 7f2d9e52f700 1 -- 172.21.15.3:0/3174294874 wait complete. 2022-07-21T01:38:58.038 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 30 2022-07-21T01:38:58.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:38:58.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.149+0000 7fd6cabc6700 1 Processor -- start 2022-07-21T01:38:58.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.149+0000 7fd6cabc6700 1 -- start start 2022-07-21T01:38:58.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd6c40b0740 0x7fd6c40b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:58.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 0x7fd6c40b1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:58.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c4143ab0 con 0x7fd6c40b38a0 2022-07-21T01:38:58.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c4143d50 con 0x7fd6c40b0740 2022-07-21T01:38:58.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c4143ff0 con 0x7fd6c40a9da0 2022-07-21T01:38:58.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6c8962700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd6c40b0740 0x7fd6c40b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:58.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6bbfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd6c40b38a0 0x7fd6c413d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54844/0 (socket says 172.21.15.3:54844) 2022-07-21T01:38:58.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6bbfff700 1 -- 172.21.15.3:0/864804339 learned_addr learned my addr 172.21.15.3:0/864804339 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:58.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6c9163700 1 --2- 172.21.15.3:0/864804339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 0x7fd6c40b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:58.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6bb7fe700 1 -- 172.21.15.3:0/864804339 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462077902 0 0) 0x7fd6c4143ab0 con 0x7fd6c40b38a0 2022-07-21T01:38:58.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6bb7fe700 1 -- 172.21.15.3:0/864804339 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b0003260 con 0x7fd6c40b38a0 2022-07-21T01:38:58.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6c8962700 1 -- 172.21.15.3:0/864804339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 msgr2=0x7fd6c40b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:38:58.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6c8962700 1 --2- 172.21.15.3:0/864804339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 0x7fd6c40b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6c8962700 1 -- 172.21.15.3:0/864804339 >> v1:172.21.15.3:6789/0 conn(0x7fd6c40b38a0 legacy=0x7fd6c413d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:58.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6c8962700 1 -- 172.21.15.3:0/864804339 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c4144290 con 0x7fd6c40b0740 2022-07-21T01:38:58.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6c8962700 1 --2- 172.21.15.3:0/864804339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 0x7fd6c40b0b20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fd6c0002f70 tx=0x7fd6c0006df0).ready entity=mon.0 client_cookie=a9d5049498ee0bd2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:58.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6bb7fe700 1 -- 172.21.15.3:0/864804339 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6c000d4a0 con 0x7fd6c40b0740 2022-07-21T01:38:58.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6bb7fe700 1 -- 172.21.15.3:0/864804339 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c0000e90 con 0x7fd6c40b0740 2022-07-21T01:38:58.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6bb7fe700 1 -- 172.21.15.3:0/864804339 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6c000d7f0 con 0x7fd6c40b0740 2022-07-21T01:38:58.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 -- 172.21.15.3:0/864804339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 msgr2=0x7fd6c40b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:58.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 --2- 172.21.15.3:0/864804339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 0x7fd6c40b0b20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fd6c0002f70 tx=0x7fd6c0006df0).stop 2022-07-21T01:38:58.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 -- 172.21.15.3:0/864804339 shutdown_connections 2022-07-21T01:38:58.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 --2- 172.21.15.3:0/864804339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 0x7fd6c40b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 --2- 172.21.15.3:0/864804339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 0x7fd6c40b0b20 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 -- 172.21.15.3:0/864804339 >> 172.21.15.3:0/864804339 conn(0x7fd6c4060950 msgr2=0x7fd6c4060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:58.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 -- 172.21.15.3:0/864804339 shutdown_connections 2022-07-21T01:38:58.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.153+0000 7fd6cabc6700 1 -- 172.21.15.3:0/864804339 wait complete. 2022-07-21T01:38:58.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 Processor -- start 2022-07-21T01:38:58.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 -- start start 2022-07-21T01:38:58.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 0x7fd6c405fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:58.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 0x7fd6c40aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:58.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6c40b38a0 0x7fd6c40aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:58.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c40aed80 con 0x7fd6c40b0740 2022-07-21T01:38:58.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c40abb40 con 0x7fd6c40b38a0 2022-07-21T01:38:58.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c40677c0 con 0x7fd6c40a9da0 2022-07-21T01:38:58.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6bbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 0x7fd6c40aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:58.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6c8962700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 0x7fd6c405fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:58.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6bbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 0x7fd6c40aaa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46574/0 (socket says 172.21.15.3:46574) 2022-07-21T01:38:58.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6bbfff700 1 -- 172.21.15.3:0/1347080486 learned_addr learned my addr 172.21.15.3:0/1347080486 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:58.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6c9163700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6c40b38a0 0x7fd6c40aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:58.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6bbfff700 1 -- 172.21.15.3:0/1347080486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 msgr2=0x7fd6c405fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:58.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6bbfff700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 0x7fd6c405fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6bbfff700 1 -- 172.21.15.3:0/1347080486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6c40b38a0 msgr2=0x7fd6c40aaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:58.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6bbfff700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6c40b38a0 0x7fd6c40aaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6bbfff700 1 -- 172.21.15.3:0/1347080486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c000b180 con 0x7fd6c40b0740 2022-07-21T01:38:58.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6bbfff700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 0x7fd6c40aaa10 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fd6c4140ea0 tx=0x7fd6b4007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:58.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6b9ffb700 1 -- 172.21.15.3:0/1347080486 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6b400cda0 con 0x7fd6c40b0740 2022-07-21T01:38:58.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6b9ffb700 1 -- 172.21.15.3:0/1347080486 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b40070d0 con 0x7fd6c40b0740 2022-07-21T01:38:58.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c40666a0 con 0x7fd6c40b0740 2022-07-21T01:38:58.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6b9ffb700 1 -- 172.21.15.3:0/1347080486 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6b4007430 con 0x7fd6c40b0740 2022-07-21T01:38:58.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.157+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c4067c50 con 0x7fd6c40b0740 2022-07-21T01:38:58.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.161+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c40aa720 con 0x7fd6c40b0740 2022-07-21T01:38:58.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.161+0000 7fd6b9ffb700 1 -- 172.21.15.3:0/1347080486 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd6b401a070 con 0x7fd6c40b0740 2022-07-21T01:38:58.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.161+0000 7fd6b9ffb700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6a4053950 0x7fd6a4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:58.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.161+0000 7fd6b9ffb700 1 -- 172.21.15.3:0/1347080486 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fd6b406b240 con 0x7fd6c40b0740 2022-07-21T01:38:58.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.161+0000 7fd6c8962700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6a4053950 0x7fd6a4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:58.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.161+0000 7fd6c8962700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6a4053950 0x7fd6a4055e10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fd6c001a000 tx=0x7fd6c0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:58.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.169+0000 7fd6b9ffb700 1 -- 172.21.15.3:0/1347080486 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd6b4041f10 con 0x7fd6c40b0740 2022-07-21T01:38:58.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.337+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd6c413ddf0 con 0x7fd6c40b0740 2022-07-21T01:38:58.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.957+0000 7fd6b9ffb700 1 -- 172.21.15.3:0/1347080486 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd6b4045530 con 0x7fd6c40b0740 2022-07-21T01:38:58.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6a4053950 msgr2=0x7fd6a4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:58.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6a4053950 0x7fd6a4055e10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fd6c001a000 tx=0x7fd6c0014000).stop 2022-07-21T01:38:58.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 msgr2=0x7fd6c40aaa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:58.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 0x7fd6c40aaa10 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fd6c4140ea0 tx=0x7fd6b4007a60).stop 2022-07-21T01:38:58.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 shutdown_connections 2022-07-21T01:38:58.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd6a4053950 0x7fd6a4055e10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6c40b38a0 0x7fd6c40aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c40b0740 0x7fd6c40aaa10 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 --2- 172.21.15.3:0/1347080486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6c40a9da0 0x7fd6c405fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:58.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 >> 172.21.15.3:0/1347080486 conn(0x7fd6c4060950 msgr2=0x7fd6c4141ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:58.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 shutdown_connections 2022-07-21T01:38:58.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:58.961+0000 7fd6cabc6700 1 -- 172.21.15.3:0/1347080486 wait complete. 2022-07-21T01:38:58.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:38:59.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.097+0000 7fe60e359700 1 Processor -- start 2022-07-21T01:38:59.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.097+0000 7fe60e359700 1 -- start start 2022-07-21T01:38:59.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe6080612f0 0x7fe608143b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:59.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6080671e0 con 0x7fe6080b2930 2022-07-21T01:38:59.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe608065540 con 0x7fe6080b7840 2022-07-21T01:38:59.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6080b7c20 con 0x7fe6080612f0 2022-07-21T01:38:59.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60cb56700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe6080b2930 0x7fe608143400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54860/0 (socket says 172.21.15.3:54860) 2022-07-21T01:38:59.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60cb56700 1 -- 172.21.15.3:0/3493887803 learned_addr learned my addr 172.21.15.3:0/3493887803 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:38:59.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60db58700 1 --2- 172.21.15.3:0/3493887803 >> v2:172.21.15.143:3300/0 conn(0x7fe6080612f0 0x7fe608143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:59.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe5f7fff700 1 -- 172.21.15.3:0/3493887803 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835320635 0 0) 0x7fe6080671e0 con 0x7fe6080b2930 2022-07-21T01:38:59.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe5f7fff700 1 -- 172.21.15.3:0/3493887803 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f0003260 con 0x7fe6080b2930 2022-07-21T01:38:59.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60db58700 1 -- 172.21.15.3:0/3493887803 >> v1:172.21.15.143:6789/0 conn(0x7fe6080b7840 legacy=0x7fe608060be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:59.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60db58700 1 -- 172.21.15.3:0/3493887803 >> v1:172.21.15.3:6789/0 conn(0x7fe6080b2930 legacy=0x7fe608143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:59.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60db58700 1 -- 172.21.15.3:0/3493887803 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608066ce0 con 0x7fe6080612f0 2022-07-21T01:38:59.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60db58700 1 --2- 172.21.15.3:0/3493887803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080612f0 0x7fe608143b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fe6000032f0 tx=0x7fe6000091d0).ready entity=mon.1 client_cookie=82d7662f0b8fe857 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:59.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe5f7fff700 1 -- 172.21.15.3:0/3493887803 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe60000b6c0 con 0x7fe6080612f0 2022-07-21T01:38:59.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe5f7fff700 1 -- 172.21.15.3:0/3493887803 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60000b820 con 0x7fe6080612f0 2022-07-21T01:38:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe5f7fff700 1 -- 172.21.15.3:0/3493887803 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe60000b990 con 0x7fe6080612f0 2022-07-21T01:38:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 -- 172.21.15.3:0/3493887803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080612f0 msgr2=0x7fe608143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:59.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 --2- 172.21.15.3:0/3493887803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080612f0 0x7fe608143b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fe6000032f0 tx=0x7fe6000091d0).stop 2022-07-21T01:38:59.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 -- 172.21.15.3:0/3493887803 shutdown_connections 2022-07-21T01:38:59.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 --2- 172.21.15.3:0/3493887803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080612f0 0x7fe608143b10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:59.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 -- 172.21.15.3:0/3493887803 >> 172.21.15.3:0/3493887803 conn(0x7fe60805f840 msgr2=0x7fe60805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:59.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 -- 172.21.15.3:0/3493887803 shutdown_connections 2022-07-21T01:38:59.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.101+0000 7fe60e359700 1 -- 172.21.15.3:0/3493887803 wait complete. 2022-07-21T01:38:59.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60e359700 1 Processor -- start 2022-07-21T01:38:59.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60e359700 1 -- start start 2022-07-21T01:38:59.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60e359700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6080612f0 0x7fe608143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:59.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60e359700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080b2930 0x7fe60813d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:59.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60e359700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6080b7840 0x7fe60813dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:59.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60e359700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080b8180 con 0x7fe6080b7840 2022-07-21T01:38:59.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60e359700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60813e240 con 0x7fe6080b2930 2022-07-21T01:38:59.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60e359700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080ab7d0 con 0x7fe6080612f0 2022-07-21T01:38:59.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60db58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6080b7840 0x7fe60813dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:59.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60cb56700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080b2930 0x7fe60813d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:59.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60db58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6080b7840 0x7fe60813dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46588/0 (socket says 172.21.15.3:46588) 2022-07-21T01:38:59.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60db58700 1 -- 172.21.15.3:0/2264025877 learned_addr learned my addr 172.21.15.3:0/2264025877 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:38:59.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.105+0000 7fe60d357700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6080612f0 0x7fe608143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:59.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60db58700 1 -- 172.21.15.3:0/2264025877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6080612f0 msgr2=0x7fe608143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:59.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60db58700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6080612f0 0x7fe608143260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:59.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60db58700 1 -- 172.21.15.3:0/2264025877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080b2930 msgr2=0x7fe60813d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:59.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60db58700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080b2930 0x7fe60813d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:59.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60db58700 1 -- 172.21.15.3:0/2264025877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe600009470 con 0x7fe6080b7840 2022-07-21T01:38:59.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60cb56700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080b2930 0x7fe60813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:38:59.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60db58700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6080b7840 0x7fe60813dd50 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7fe6000032f0 tx=0x7fe600016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:59.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe5f67fc700 1 -- 172.21.15.3:0/2264025877 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe60000b6c0 con 0x7fe6080b7840 2022-07-21T01:38:59.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe5f67fc700 1 -- 172.21.15.3:0/2264025877 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe600021630 con 0x7fe6080b7840 2022-07-21T01:38:59.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60813fa70 con 0x7fe6080b7840 2022-07-21T01:38:59.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe5f67fc700 1 -- 172.21.15.3:0/2264025877 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe600021990 con 0x7fe6080b7840 2022-07-21T01:38:59.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60813ffb0 con 0x7fe6080b7840 2022-07-21T01:38:59.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6080574e0 con 0x7fe6080b7840 2022-07-21T01:38:59.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.109+0000 7fe5f67fc700 1 -- 172.21.15.3:0/2264025877 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe600027020 con 0x7fe6080b7840 2022-07-21T01:38:59.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.113+0000 7fe5f67fc700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5e8053a70 0x7fe5e8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:38:59.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.113+0000 7fe5f67fc700 1 -- 172.21.15.3:0/2264025877 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fe60006d4d0 con 0x7fe6080b7840 2022-07-21T01:38:59.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.113+0000 7fe60d357700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5e8053a70 0x7fe5e8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:38:59.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.113+0000 7fe60d357700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5e8053a70 0x7fe5e8055f30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fe5f800d000 tx=0x7fe5f8007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:38:59.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.117+0000 7fe5f67fc700 1 -- 172.21.15.3:0/2264025877 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe600044240 con 0x7fe6080b7840 2022-07-21T01:38:59.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.245+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe608060330 con 0x7fe6080b7840 2022-07-21T01:38:59.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.965+0000 7fe5f67fc700 1 -- 172.21.15.3:0/2264025877 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v70) v1 ==== 185+0+0 (secure 0 0 0) 0x7fe6000150e0 con 0x7fe6080b7840 2022-07-21T01:38:59.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5e8053a70 msgr2=0x7fe5e8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5e8053a70 0x7fe5e8055f30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fe5f800d000 tx=0x7fe5f8007be0).stop 2022-07-21T01:38:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6080b7840 msgr2=0x7fe60813dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:38:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6080b7840 0x7fe60813dd50 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7fe6000032f0 tx=0x7fe600016000).stop 2022-07-21T01:38:59.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 shutdown_connections 2022-07-21T01:38:59.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6080b7840 0x7fe60813dd50 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:59.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5e8053a70 0x7fe5e8055f30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:59.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6080b2930 0x7fe60813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:59.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 --2- 172.21.15.3:0/2264025877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6080612f0 0x7fe608143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:38:59.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 >> 172.21.15.3:0/2264025877 conn(0x7fe60805f840 msgr2=0x7fe6080b6410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:38:59.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 shutdown_connections 2022-07-21T01:38:59.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:38:59.969+0000 7fe60e359700 1 -- 172.21.15.3:0/2264025877 wait complete. 2022-07-21T01:38:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:38:59.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:39:00.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.097+0000 7f9bad9ac700 1 Processor -- start 2022-07-21T01:39:00.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.097+0000 7f9bad9ac700 1 -- start start 2022-07-21T01:39:00.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.097+0000 7f9bad9ac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9ba80b3260 0x7f9ba81435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:00.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.097+0000 7f9bad9ac700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba8143ab0 con 0x7f9ba80ad000 2022-07-21T01:39:00.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.097+0000 7f9bad9ac700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba80b0c80 con 0x7f9ba80b08a0 2022-07-21T01:39:00.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.097+0000 7f9bad9ac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba80ae810 con 0x7f9ba80b3260 2022-07-21T01:39:00.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.097+0000 7f9bad1ab700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9ba80ad000 0x7f9ba813fa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54876/0 (socket says 172.21.15.3:54876) 2022-07-21T01:39:00.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.097+0000 7f9bad1ab700 1 -- 172.21.15.3:0/2006610293 learned_addr learned my addr 172.21.15.3:0/2006610293 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:00.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba7fff700 1 --2- 172.21.15.3:0/2006610293 >> v2:172.21.15.143:3300/0 conn(0x7f9ba80b3260 0x7f9ba81435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:00.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba77fe700 1 -- 172.21.15.3:0/2006610293 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261788522 0 0) 0x7f9ba8143ab0 con 0x7f9ba80ad000 2022-07-21T01:39:00.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba77fe700 1 -- 172.21.15.3:0/2006610293 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b94004370 con 0x7f9ba80ad000 2022-07-21T01:39:00.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba7fff700 1 -- 172.21.15.3:0/2006610293 >> v1:172.21.15.143:6789/0 conn(0x7f9ba80b08a0 legacy=0x7f9ba8142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:00.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba7fff700 1 -- 172.21.15.3:0/2006610293 >> v1:172.21.15.3:6789/0 conn(0x7f9ba80ad000 legacy=0x7f9ba813fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:00.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba7fff700 1 -- 172.21.15.3:0/2006610293 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba80aeab0 con 0x7f9ba80b3260 2022-07-21T01:39:00.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba7fff700 1 --2- 172.21.15.3:0/2006610293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ba80b3260 0x7f9ba81435c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f9b98002f70 tx=0x7f9b98006df0).ready entity=mon.1 client_cookie=6d599b5ff66ac229 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:00.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba77fe700 1 -- 172.21.15.3:0/2006610293 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b9800d5c0 con 0x7f9ba80b3260 2022-07-21T01:39:00.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba77fe700 1 -- 172.21.15.3:0/2006610293 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b98000e90 con 0x7f9ba80b3260 2022-07-21T01:39:00.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9ba77fe700 1 -- 172.21.15.3:0/2006610293 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b9800d910 con 0x7f9ba80b3260 2022-07-21T01:39:00.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9bad9ac700 1 -- 172.21.15.3:0/2006610293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ba80b3260 msgr2=0x7f9ba81435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:00.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9bad9ac700 1 --2- 172.21.15.3:0/2006610293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ba80b3260 0x7f9ba81435c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f9b98002f70 tx=0x7f9b98006df0).stop 2022-07-21T01:39:00.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9bad9ac700 1 -- 172.21.15.3:0/2006610293 shutdown_connections 2022-07-21T01:39:00.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9bad9ac700 1 --2- 172.21.15.3:0/2006610293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ba80b3260 0x7f9ba81435c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:00.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9bad9ac700 1 -- 172.21.15.3:0/2006610293 >> 172.21.15.3:0/2006610293 conn(0x7f9ba80612d0 msgr2=0x7f9ba8062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:00.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9bad9ac700 1 -- 172.21.15.3:0/2006610293 shutdown_connections 2022-07-21T01:39:00.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9bad9ac700 1 -- 172.21.15.3:0/2006610293 wait complete. 2022-07-21T01:39:00.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9bad9ac700 1 Processor -- start 2022-07-21T01:39:00.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.101+0000 7f9bad9ac700 1 -- start start 2022-07-21T01:39:00.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bad9ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ba80ad000 0x7f9ba80abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:00.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bad9ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ba80b08a0 0x7f9ba81528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:00.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bad9ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ba80b3260 0x7f9ba8152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:00.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bad9ac700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba80a3250 con 0x7f9ba80b08a0 2022-07-21T01:39:00.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bad9ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba8152530 con 0x7f9ba80b3260 2022-07-21T01:39:00.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bad9ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba8065b30 con 0x7f9ba80ad000 2022-07-21T01:39:00.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ba80b08a0 0x7f9ba81528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:00.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ba80b08a0 0x7f9ba81528e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46598/0 (socket says 172.21.15.3:46598) 2022-07-21T01:39:00.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba7fff700 1 -- 172.21.15.3:0/3555752736 learned_addr learned my addr 172.21.15.3:0/3555752736 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:00.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bac9aa700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ba80ad000 0x7f9ba80abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:00.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bad1ab700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ba80b3260 0x7f9ba8152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:00.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba7fff700 1 -- 172.21.15.3:0/3555752736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ba80ad000 msgr2=0x7f9ba80abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:00.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba7fff700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ba80ad000 0x7f9ba80abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:00.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba7fff700 1 -- 172.21.15.3:0/3555752736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ba80b3260 msgr2=0x7f9ba8152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:00.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba7fff700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ba80b3260 0x7f9ba8152dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:00.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba7fff700 1 -- 172.21.15.3:0/3555752736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9800b180 con 0x7f9ba80b08a0 2022-07-21T01:39:00.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba7fff700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ba80b08a0 0x7f9ba81528e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f9b98002f70 tx=0x7f9b98014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:00.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba5ffb700 1 -- 172.21.15.3:0/3555752736 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b98016070 con 0x7f9ba80b08a0 2022-07-21T01:39:00.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba5ffb700 1 -- 172.21.15.3:0/3555752736 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9800de40 con 0x7f9ba80b08a0 2022-07-21T01:39:00.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9ba5ffb700 1 -- 172.21.15.3:0/3555752736 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b98013da0 con 0x7f9ba80b08a0 2022-07-21T01:39:00.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba8065510 con 0x7f9ba80b08a0 2022-07-21T01:39:00.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.105+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba8064e70 con 0x7f9ba80b08a0 2022-07-21T01:39:00.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.109+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba8152680 con 0x7f9ba80b08a0 2022-07-21T01:39:00.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.109+0000 7f9ba5ffb700 1 -- 172.21.15.3:0/3555752736 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9b9800f030 con 0x7f9ba80b08a0 2022-07-21T01:39:00.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.109+0000 7f9ba5ffb700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b88053a70 0x7f9b88055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:00.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.109+0000 7f9ba5ffb700 1 -- 172.21.15.3:0/3555752736 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f9b9806cb80 con 0x7f9ba80b08a0 2022-07-21T01:39:00.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.109+0000 7f9bac9aa700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b88053a70 0x7f9b88055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:00.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.109+0000 7f9bac9aa700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b88053a70 0x7f9b88055f30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f9b9c00d000 tx=0x7f9b9c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:00.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.117+0000 7f9ba5ffb700 1 -- 172.21.15.3:0/3555752736 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9b98043a40 con 0x7f9ba80b08a0 2022-07-21T01:39:00.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.245+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9ba80574e0 con 0x7f9ba80b08a0 2022-07-21T01:39:00.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.969+0000 7f9ba5ffb700 1 -- 172.21.15.3:0/3555752736 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v71) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9b98023070 con 0x7f9ba80b08a0 2022-07-21T01:39:00.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b88053a70 msgr2=0x7f9b88055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:00.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b88053a70 0x7f9b88055f30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f9b9c00d000 tx=0x7f9b9c007b90).stop 2022-07-21T01:39:00.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ba80b08a0 msgr2=0x7f9ba81528e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:00.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ba80b08a0 0x7f9ba81528e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f9b98002f70 tx=0x7f9b98014000).stop 2022-07-21T01:39:00.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 shutdown_connections 2022-07-21T01:39:00.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b88053a70 0x7f9b88055f30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:00.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ba80b3260 0x7f9ba8152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:00.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ba80b08a0 0x7f9ba81528e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:00.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 --2- 172.21.15.3:0/3555752736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ba80ad000 0x7f9ba80abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:00.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 >> 172.21.15.3:0/3555752736 conn(0x7f9ba80612d0 msgr2=0x7f9ba8142340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:00.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 shutdown_connections 2022-07-21T01:39:00.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:00.973+0000 7f9bad9ac700 1 -- 172.21.15.3:0/3555752736 wait complete. 2022-07-21T01:39:00.974 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:39:00.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:39:01.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe2941b2700 1 Processor -- start 2022-07-21T01:39:01.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe2941b2700 1 -- start start 2022-07-21T01:39:01.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe2941b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c0b2a40 0x7fe28c0b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe2941b2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c062290 con 0x7fe28c0affd0 2022-07-21T01:39:01.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe2941b2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c0b2500 con 0x7fe28c061eb0 2022-07-21T01:39:01.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe2941b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c0b22e0 con 0x7fe28c0b2a40 2022-07-21T01:39:01.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe29274f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe28c0affd0 0x7fe28c13ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54890/0 (socket says 172.21.15.3:54890) 2022-07-21T01:39:01.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe29274f700 1 -- 172.21.15.3:0/3754305563 learned_addr learned my addr 172.21.15.3:0/3754305563 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:01.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe29174d700 1 --2- 172.21.15.3:0/3754305563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c0b2a40 0x7fe28c0b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe290f4c700 1 -- 172.21.15.3:0/3754305563 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246592103 0 0) 0x7fe28c062290 con 0x7fe28c0affd0 2022-07-21T01:39:01.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.097+0000 7fe290f4c700 1 -- 172.21.15.3:0/3754305563 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe274003260 con 0x7fe28c0affd0 2022-07-21T01:39:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe29174d700 1 -- 172.21.15.3:0/3754305563 >> v1:172.21.15.143:6789/0 conn(0x7fe28c061eb0 legacy=0x7fe28c0af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe29174d700 1 -- 172.21.15.3:0/3754305563 >> v1:172.21.15.3:6789/0 conn(0x7fe28c0affd0 legacy=0x7fe28c13ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe29174d700 1 -- 172.21.15.3:0/3754305563 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c0b27a0 con 0x7fe28c0b2a40 2022-07-21T01:39:01.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe29174d700 1 --2- 172.21.15.3:0/3754305563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c0b2a40 0x7fe28c0b1df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe288002f70 tx=0x7fe288006df0).ready entity=mon.2 client_cookie=e80116361eb1cdb9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:01.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe290f4c700 1 -- 172.21.15.3:0/3754305563 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe288014070 con 0x7fe28c0b2a40 2022-07-21T01:39:01.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe290f4c700 1 -- 172.21.15.3:0/3754305563 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe288000e90 con 0x7fe28c0b2a40 2022-07-21T01:39:01.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe290f4c700 1 -- 172.21.15.3:0/3754305563 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe28800fb60 con 0x7fe28c0b2a40 2022-07-21T01:39:01.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe2941b2700 1 -- 172.21.15.3:0/3754305563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c0b2a40 msgr2=0x7fe28c0b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe2941b2700 1 --2- 172.21.15.3:0/3754305563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c0b2a40 0x7fe28c0b1df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe288002f70 tx=0x7fe288006df0).stop 2022-07-21T01:39:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe2941b2700 1 -- 172.21.15.3:0/3754305563 shutdown_connections 2022-07-21T01:39:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe2941b2700 1 --2- 172.21.15.3:0/3754305563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c0b2a40 0x7fe28c0b1df0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe2941b2700 1 -- 172.21.15.3:0/3754305563 >> 172.21.15.3:0/3754305563 conn(0x7fe28c063b40 msgr2=0x7fe28c063f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe2941b2700 1 -- 172.21.15.3:0/3754305563 shutdown_connections 2022-07-21T01:39:01.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe2941b2700 1 -- 172.21.15.3:0/3754305563 wait complete. 2022-07-21T01:39:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe2941b2700 1 Processor -- start 2022-07-21T01:39:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.101+0000 7fe2941b2700 1 -- start start 2022-07-21T01:39:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe2941b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c061eb0 0x7fe28c13a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe2941b2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe28c0affd0 0x7fe28c065860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe2941b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe28c0b2a40 0x7fe28c065d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe2941b2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c14eda0 con 0x7fe28c0affd0 2022-07-21T01:39:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe2941b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c0654b0 con 0x7fe28c0b2a40 2022-07-21T01:39:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe2941b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c065210 con 0x7fe28c061eb0 2022-07-21T01:39:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29174d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe28c0affd0 0x7fe28c065860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29274f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe28c0b2a40 0x7fe28c065d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29174d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe28c0affd0 0x7fe28c065860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46614/0 (socket says 172.21.15.3:46614) 2022-07-21T01:39:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29174d700 1 -- 172.21.15.3:0/3159545385 learned_addr learned my addr 172.21.15.3:0/3159545385 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe291f4e700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c061eb0 0x7fe28c13a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29174d700 1 -- 172.21.15.3:0/3159545385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c061eb0 msgr2=0x7fe28c13a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29174d700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c061eb0 0x7fe28c13a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29174d700 1 -- 172.21.15.3:0/3159545385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe28c0b2a40 msgr2=0x7fe28c065d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29174d700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe28c0b2a40 0x7fe28c065d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29174d700 1 -- 172.21.15.3:0/3159545385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28800b180 con 0x7fe28c0affd0 2022-07-21T01:39:01.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29274f700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe28c0b2a40 0x7fe28c065d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:01.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe291f4e700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c061eb0 0x7fe28c13a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:01.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe29174d700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe28c0affd0 0x7fe28c065860 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fe288002f70 tx=0x7fe288016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:01.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe27effd700 1 -- 172.21.15.3:0/3159545385 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe288010d60 con 0x7fe28c0affd0 2022-07-21T01:39:01.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe27effd700 1 -- 172.21.15.3:0/3159545385 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28801d450 con 0x7fe28c0affd0 2022-07-21T01:39:01.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c14f000 con 0x7fe28c0affd0 2022-07-21T01:39:01.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe27effd700 1 -- 172.21.15.3:0/3159545385 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe28801d7b0 con 0x7fe28c0affd0 2022-07-21T01:39:01.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.105+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c066920 con 0x7fe28c0affd0 2022-07-21T01:39:01.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.109+0000 7fe27effd700 1 -- 172.21.15.3:0/3159545385 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe28801d910 con 0x7fe28c0affd0 2022-07-21T01:39:01.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.109+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c14f190 con 0x7fe28c0affd0 2022-07-21T01:39:01.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.109+0000 7fe27effd700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe26c053a70 0x7fe26c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.109+0000 7fe27effd700 1 -- 172.21.15.3:0/3159545385 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fe28806fe10 con 0x7fe28c0affd0 2022-07-21T01:39:01.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.109+0000 7fe291f4e700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe26c053a70 0x7fe26c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.109+0000 7fe291f4e700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe26c053a70 0x7fe26c055f30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fe28000d000 tx=0x7fe280009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:01.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.113+0000 7fe27effd700 1 -- 172.21.15.3:0/3159545385 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe288046eb0 con 0x7fe28c0affd0 2022-07-21T01:39:01.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.241+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe28c0af180 con 0x7fe28c0affd0 2022-07-21T01:39:01.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe27effd700 1 -- 172.21.15.3:0/3159545385 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe288006980 con 0x7fe28c0affd0 2022-07-21T01:39:01.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe26c053a70 msgr2=0x7fe26c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe26c053a70 0x7fe26c055f30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fe28000d000 tx=0x7fe280009040).stop 2022-07-21T01:39:01.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe28c0affd0 msgr2=0x7fe28c065860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe28c0affd0 0x7fe28c065860 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fe288002f70 tx=0x7fe288016000).stop 2022-07-21T01:39:01.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 shutdown_connections 2022-07-21T01:39:01.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe26c053a70 0x7fe26c055f30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe28c0b2a40 0x7fe28c065d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe28c0affd0 0x7fe28c065860 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 --2- 172.21.15.3:0/3159545385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe28c061eb0 0x7fe28c13a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 >> 172.21.15.3:0/3159545385 conn(0x7fe28c063b40 msgr2=0x7fe28c144f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:01.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 shutdown_connections 2022-07-21T01:39:01.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.249+0000 7fe2941b2700 1 -- 172.21.15.3:0/3159545385 wait complete. 2022-07-21T01:39:01.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-07-21T01:39:01.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.369+0000 7f3ccbfff700 1 Processor -- start 2022-07-21T01:39:01.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.369+0000 7f3ccbfff700 1 -- start start 2022-07-21T01:39:01.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ccc0af6e0 0x7f3ccc147730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ccc0b5a10 con 0x7f3ccc0b5630 2022-07-21T01:39:01.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ccc0ac990 con 0x7f3ccc0b07a0 2022-07-21T01:39:01.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ccc145470 con 0x7f3ccc0af6e0 2022-07-21T01:39:01.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccb7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ccc0af6e0 0x7f3ccc147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccaffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3ccc0b5630 0x7f3ccc067730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54902/0 (socket says 172.21.15.3:54902) 2022-07-21T01:39:01.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccb7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ccc0af6e0 0x7f3ccc147730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46624/0 (socket says 172.21.15.3:46624) 2022-07-21T01:39:01.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccaffd700 1 -- 172.21.15.3:0/2954318557 learned_addr learned my addr 172.21.15.3:0/2954318557 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:01.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3cc9ffb700 1 -- 172.21.15.3:0/2954318557 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824669582 0 0) 0x7f3ccc0ac990 con 0x7f3ccc0b07a0 2022-07-21T01:39:01.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccb7fe700 1 -- 172.21.15.3:0/2954318557 >> v1:172.21.15.143:6789/0 conn(0x7f3ccc0b07a0 legacy=0x7f3ccc0acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccb7fe700 1 -- 172.21.15.3:0/2954318557 >> v1:172.21.15.3:6789/0 conn(0x7f3ccc0b5630 legacy=0x7f3ccc067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccb7fe700 1 -- 172.21.15.3:0/2954318557 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc0b1bc0 con 0x7f3ccc0af6e0 2022-07-21T01:39:01.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccb7fe700 1 --2- 172.21.15.3:0/2954318557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0af6e0 0x7f3ccc147730 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f3cc0001110 tx=0x7f3cc4006df0).ready entity=mon.0 client_cookie=aa2af5c9ac33e47c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:01.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3cc9ffb700 1 -- 172.21.15.3:0/2954318557 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc400d5c0 con 0x7f3ccc0af6e0 2022-07-21T01:39:01.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3cc9ffb700 1 -- 172.21.15.3:0/2954318557 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc4000e90 con 0x7f3ccc0af6e0 2022-07-21T01:39:01.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3cc9ffb700 1 -- 172.21.15.3:0/2954318557 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc400d910 con 0x7f3ccc0af6e0 2022-07-21T01:39:01.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 -- 172.21.15.3:0/2954318557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0af6e0 msgr2=0x7f3ccc147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 --2- 172.21.15.3:0/2954318557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0af6e0 0x7f3ccc147730 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f3cc0001110 tx=0x7f3cc4006df0).stop 2022-07-21T01:39:01.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 -- 172.21.15.3:0/2954318557 shutdown_connections 2022-07-21T01:39:01.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 --2- 172.21.15.3:0/2954318557 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0af6e0 0x7f3ccc147730 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 -- 172.21.15.3:0/2954318557 >> 172.21.15.3:0/2954318557 conn(0x7f3ccc0626f0 msgr2=0x7f3ccc062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:01.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 -- 172.21.15.3:0/2954318557 shutdown_connections 2022-07-21T01:39:01.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.373+0000 7f3ccbfff700 1 -- 172.21.15.3:0/2954318557 wait complete. 2022-07-21T01:39:01.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccbfff700 1 Processor -- start 2022-07-21T01:39:01.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccbfff700 1 -- start start 2022-07-21T01:39:01.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ccc0af6e0 0x7f3ccc146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0b07a0 0x7f3ccc1572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ccc0b5630 0x7f3ccc1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccbfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0ae180 con 0x7f3ccc0b07a0 2022-07-21T01:39:01.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccbfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc066a00 con 0x7f3ccc0b5630 2022-07-21T01:39:01.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc156e70 con 0x7f3ccc0af6e0 2022-07-21T01:39:01.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3cca7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0b07a0 0x7f3ccc1572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3cca7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0b07a0 0x7f3ccc1572d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46628/0 (socket says 172.21.15.3:46628) 2022-07-21T01:39:01.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3cca7fc700 1 -- 172.21.15.3:0/1829211033 learned_addr learned my addr 172.21.15.3:0/1829211033 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:01.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccb7fe700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ccc0b5630 0x7f3ccc1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccaffd700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ccc0af6e0 0x7f3ccc146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccb7fe700 1 -- 172.21.15.3:0/1829211033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ccc0af6e0 msgr2=0x7f3ccc146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccb7fe700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ccc0af6e0 0x7f3ccc146880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccb7fe700 1 -- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0b07a0 msgr2=0x7f3ccc1572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccb7fe700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0b07a0 0x7f3ccc1572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccb7fe700 1 -- 172.21.15.3:0/1829211033 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc400b180 con 0x7f3ccc0b5630 2022-07-21T01:39:01.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3cca7fc700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0b07a0 0x7f3ccc1572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:01.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3ccb7fe700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ccc0b5630 0x7f3ccc1577c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f3ccc0a7980 tx=0x7f3cc4014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:01.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3cbbfff700 1 -- 172.21.15.3:0/1829211033 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc4016070 con 0x7f3ccc0b5630 2022-07-21T01:39:01.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.377+0000 7f3cbbfff700 1 -- 172.21.15.3:0/1829211033 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc400de40 con 0x7f3ccc0b5630 2022-07-21T01:39:01.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.381+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ccc157cb0 con 0x7f3ccc0b5630 2022-07-21T01:39:01.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.381+0000 7f3cbbfff700 1 -- 172.21.15.3:0/1829211033 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc4002f50 con 0x7f3ccc0b5630 2022-07-21T01:39:01.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.381+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc15a3d0 con 0x7f3ccc0b5630 2022-07-21T01:39:01.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.381+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc0b1680 con 0x7f3ccc0b5630 2022-07-21T01:39:01.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.381+0000 7f3cbbfff700 1 -- 172.21.15.3:0/1829211033 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3cc400f030 con 0x7f3ccc0b5630 2022-07-21T01:39:01.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.381+0000 7f3cbbfff700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca80539e0 0x7f3ca8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:01.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.381+0000 7f3cbbfff700 1 -- 172.21.15.3:0/1829211033 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f3cc406ca50 con 0x7f3ccc0b5630 2022-07-21T01:39:01.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.381+0000 7f3ccaffd700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca80539e0 0x7f3ca8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:01.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.381+0000 7f3ccaffd700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca80539e0 0x7f3ca8055ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f3cc000d000 tx=0x7f3cc0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:01.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.389+0000 7f3cbbfff700 1 -- 172.21.15.3:0/1829211033 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3cc4043960 con 0x7f3ccc0b5630 2022-07-21T01:39:01.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.561+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7f3ccc0574e0 con 0x7f3ccc0b5630 2022-07-21T01:39:01.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.973+0000 7f3cbbfff700 1 -- 172.21.15.3:0/1829211033 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v244) v1 ==== 147+0+0 (secure 0 0 0) 0x7f3cc4023070 con 0x7f3ccc0b5630 2022-07-21T01:39:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca80539e0 msgr2=0x7f3ca8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca80539e0 0x7f3ca8055ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f3cc000d000 tx=0x7f3cc0009040).stop 2022-07-21T01:39:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ccc0b5630 msgr2=0x7f3ccc1577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ccc0b5630 0x7f3ccc1577c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f3ccc0a7980 tx=0x7f3cc4014000).stop 2022-07-21T01:39:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 shutdown_connections 2022-07-21T01:39:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca80539e0 0x7f3ca8055ea0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ccc0b5630 0x7f3ccc1577c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ccc0b07a0 0x7f3ccc1572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 --2- 172.21.15.3:0/1829211033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ccc0af6e0 0x7f3ccc146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:01.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 >> 172.21.15.3:0/1829211033 conn(0x7f3ccc0626f0 msgr2=0x7f3ccc143b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:01.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 shutdown_connections 2022-07-21T01:39:01.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:01.977+0000 7f3ccbfff700 1 -- 172.21.15.3:0/1829211033 wait complete. 2022-07-21T01:39:01.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:39:01.986 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:39:01.989 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:39:01.992 INFO:tasks.cephfs_test_runner:test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-21T01:39:01.993 INFO:tasks.cephfs_test_runner:Starting test: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) 2022-07-21T01:39:01.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-07-21T01:39:01.996 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:39:02.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.181+0000 7f9d1e043700 1 Processor -- start 2022-07-21T01:39:02.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.181+0000 7f9d1e043700 1 -- start start 2022-07-21T01:39:02.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.181+0000 7f9d1e043700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18143710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:02.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.181+0000 7f9d1e043700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d18067a10 con 0x7f9d180b0270 2022-07-21T01:39:02.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.181+0000 7f9d1e043700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d18067cb0 con 0x7f9d180b2710 2022-07-21T01:39:02.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.181+0000 7f9d1e043700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180ad740 con 0x7f9d180ac880 2022-07-21T01:39:02.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1c840700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9d180b0270 0x7f9d18143000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54918/0 (socket says 172.21.15.3:54918) 2022-07-21T01:39:02.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1c840700 1 -- 172.21.15.3:0/1226031644 learned_addr learned my addr 172.21.15.3:0/1226031644 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:02.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1d842700 1 --2- 172.21.15.3:0/1226031644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:02.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d0ffff700 1 -- 172.21.15.3:0/1226031644 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252641396 0 0) 0x7f9d18067a10 con 0x7f9d180b0270 2022-07-21T01:39:02.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d0ffff700 1 -- 172.21.15.3:0/1226031644 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cfc003260 con 0x7f9d180b0270 2022-07-21T01:39:02.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d0ffff700 1 -- 172.21.15.3:0/1226031644 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589513655 0 0) 0x7f9d18067cb0 con 0x7f9d180b2710 2022-07-21T01:39:02.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1d842700 1 -- 172.21.15.3:0/1226031644 >> v1:172.21.15.143:6789/0 conn(0x7f9d180b2710 legacy=0x7f9d181401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:02.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1d842700 1 -- 172.21.15.3:0/1226031644 >> v1:172.21.15.3:6789/0 conn(0x7f9d180b0270 legacy=0x7f9d18143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:02.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1d842700 1 -- 172.21.15.3:0/1226031644 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d18147160 con 0x7f9d180ac880 2022-07-21T01:39:02.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1d842700 1 --2- 172.21.15.3:0/1226031644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18143710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9d14001800 tx=0x7f9d100091d0).ready entity=mon.2 client_cookie=6f725f3d3dcd7931 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:02.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d0ffff700 1 -- 172.21.15.3:0/1226031644 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d14001660 con 0x7f9d180ac880 2022-07-21T01:39:02.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d0ffff700 1 -- 172.21.15.3:0/1226031644 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1000b580 con 0x7f9d180ac880 2022-07-21T01:39:02.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d0ffff700 1 -- 172.21.15.3:0/1226031644 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d1000b6f0 con 0x7f9d180ac880 2022-07-21T01:39:02.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1e043700 1 -- 172.21.15.3:0/1226031644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 msgr2=0x7f9d18143710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:02.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1e043700 1 --2- 172.21.15.3:0/1226031644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18143710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9d14001800 tx=0x7f9d100091d0).stop 2022-07-21T01:39:02.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1e043700 1 -- 172.21.15.3:0/1226031644 shutdown_connections 2022-07-21T01:39:02.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1e043700 1 --2- 172.21.15.3:0/1226031644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18143710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:02.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1e043700 1 -- 172.21.15.3:0/1226031644 >> 172.21.15.3:0/1226031644 conn(0x7f9d18060310 msgr2=0x7f9d180b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:02.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1e043700 1 -- 172.21.15.3:0/1226031644 shutdown_connections 2022-07-21T01:39:02.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.185+0000 7f9d1e043700 1 -- 172.21.15.3:0/1226031644 wait complete. 2022-07-21T01:39:02.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 Processor -- start 2022-07-21T01:39:02.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 -- start start 2022-07-21T01:39:02.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:02.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d180b0270 0x7f9d18152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:02.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180b2710 0x7f9d181559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:02.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d181523d0 con 0x7f9d180b2710 2022-07-21T01:39:02.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180abca0 con 0x7f9d180b0270 2022-07-21T01:39:02.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d181525c0 con 0x7f9d180ac880 2022-07-21T01:39:02.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d842700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180b2710 0x7f9d181559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:02.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d842700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180b2710 0x7f9d181559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46644/0 (socket says 172.21.15.3:46644) 2022-07-21T01:39:02.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d842700 1 -- 172.21.15.3:0/3501087591 learned_addr learned my addr 172.21.15.3:0/3501087591 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:02.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d041700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:02.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1c840700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d180b0270 0x7f9d18152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d842700 1 -- 172.21.15.3:0/3501087591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 msgr2=0x7f9d18142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d842700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d842700 1 -- 172.21.15.3:0/3501087591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d180b0270 msgr2=0x7f9d18152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d842700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d180b0270 0x7f9d18152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d842700 1 -- 172.21.15.3:0/3501087591 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d10009470 con 0x7f9d180b2710 2022-07-21T01:39:02.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d041700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:02.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1d842700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180b2710 0x7f9d181559a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f9d14001800 tx=0x7f9d18067ba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:02.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d0e7fc700 1 -- 172.21.15.3:0/3501087591 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d10021070 con 0x7f9d180b2710 2022-07-21T01:39:02.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d0e7fc700 1 -- 172.21.15.3:0/3501087591 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d10011b40 con 0x7f9d180b2710 2022-07-21T01:39:02.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d18152860 con 0x7f9d180b2710 2022-07-21T01:39:02.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d0e7fc700 1 -- 172.21.15.3:0/3501087591 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d10011ea0 con 0x7f9d180b2710 2022-07-21T01:39:02.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.189+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d180a8cc0 con 0x7f9d180b2710 2022-07-21T01:39:02.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.193+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d181529f0 con 0x7f9d180b2710 2022-07-21T01:39:02.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.193+0000 7f9d0e7fc700 1 -- 172.21.15.3:0/3501087591 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9d1000f070 con 0x7f9d180b2710 2022-07-21T01:39:02.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.193+0000 7f9d0e7fc700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9d00053950 0x7f9d00055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:02.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.193+0000 7f9d0e7fc700 1 -- 172.21.15.3:0/3501087591 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f9d1006d1f0 con 0x7f9d180b2710 2022-07-21T01:39:02.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.193+0000 7f9d1d041700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9d00053950 0x7f9d00055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:02.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.193+0000 7f9d1d041700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9d00053950 0x7f9d00055e10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f9d0800d000 tx=0x7f9d08009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:02.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.197+0000 7f9d0e7fc700 1 -- 172.21.15.3:0/3501087591 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9d10044290 con 0x7f9d180b2710 2022-07-21T01:39:02.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.365+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7f9d18143c00 con 0x7f9d180b2710 2022-07-21T01:39:02.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.973+0000 7f9d0e7fc700 1 -- 172.21.15.3:0/3501087591 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v245) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9d100478b0 con 0x7f9d180b2710 2022-07-21T01:39:02.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9d00053950 msgr2=0x7f9d00055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:02.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9d00053950 0x7f9d00055e10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f9d0800d000 tx=0x7f9d08009040).stop 2022-07-21T01:39:02.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180b2710 msgr2=0x7f9d181559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:02.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180b2710 0x7f9d181559a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f9d14001800 tx=0x7f9d18067ba0).stop 2022-07-21T01:39:02.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 shutdown_connections 2022-07-21T01:39:02.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180b2710 0x7f9d181559a0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:02.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9d00053950 0x7f9d00055e10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:02.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d180b0270 0x7f9d18152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:02.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 --2- 172.21.15.3:0/3501087591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d180ac880 0x7f9d18142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:02.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 >> 172.21.15.3:0/3501087591 conn(0x7f9d18060310 msgr2=0x7f9d180b4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:02.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 shutdown_connections 2022-07-21T01:39:02.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:02.977+0000 7f9d1e043700 1 -- 172.21.15.3:0/3501087591 wait complete. 2022-07-21T01:39:02.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:39:03.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27f4c3700 1 Processor -- start 2022-07-21T01:39:03.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27f4c3700 1 -- start start 2022-07-21T01:39:03.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27f4c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 0x7fa278061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27f4c3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa2781405d0 0x7fa278140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27f4c3700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2780aee80 con 0x7fa2780614b0 2022-07-21T01:39:03.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27f4c3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2780b70b0 con 0x7fa2781405d0 2022-07-21T01:39:03.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27f4c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2780af0b0 con 0x7fa2780ad750 2022-07-21T01:39:03.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27e4c1700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa2780614b0 0x7fa2780b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54928/0 (socket says 172.21.15.3:54928) 2022-07-21T01:39:03.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27ecc2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa2781405d0 0x7fa278140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27e4c1700 1 -- 172.21.15.3:0/2344952997 learned_addr learned my addr 172.21.15.3:0/2344952997 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:03.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.109+0000 7fa27dcc0700 1 --2- 172.21.15.3:0/2344952997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 0x7fa278061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27d4bf700 1 -- 172.21.15.3:0/2344952997 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920543474 0 0) 0x7fa2780aee80 con 0x7fa2780614b0 2022-07-21T01:39:03.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27d4bf700 1 -- 172.21.15.3:0/2344952997 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa260003260 con 0x7fa2780614b0 2022-07-21T01:39:03.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27ecc2700 1 -- 172.21.15.3:0/2344952997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 msgr2=0x7fa278061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:03.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27ecc2700 1 --2- 172.21.15.3:0/2344952997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 0x7fa278061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27ecc2700 1 -- 172.21.15.3:0/2344952997 >> v1:172.21.15.3:6789/0 conn(0x7fa2780614b0 legacy=0x7fa2780b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27ecc2700 1 -- 172.21.15.3:0/2344952997 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2780af350 con 0x7fa2781405d0 2022-07-21T01:39:03.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27ecc2700 1 --2- 172.21.15.3:0/2344952997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 0x7fa278140a80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fa2700032f0 tx=0x7fa2700091d0).ready entity=mon.0 client_cookie=7d30c619257a1888 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:03.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27d4bf700 1 -- 172.21.15.3:0/2344952997 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa27000b600 con 0x7fa2781405d0 2022-07-21T01:39:03.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27d4bf700 1 -- 172.21.15.3:0/2344952997 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27000b760 con 0x7fa2781405d0 2022-07-21T01:39:03.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27d4bf700 1 -- 172.21.15.3:0/2344952997 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa27000b8d0 con 0x7fa2781405d0 2022-07-21T01:39:03.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27f4c3700 1 -- 172.21.15.3:0/2344952997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 msgr2=0x7fa278140a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27f4c3700 1 --2- 172.21.15.3:0/2344952997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 0x7fa278140a80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fa2700032f0 tx=0x7fa2700091d0).stop 2022-07-21T01:39:03.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27f4c3700 1 -- 172.21.15.3:0/2344952997 shutdown_connections 2022-07-21T01:39:03.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27f4c3700 1 --2- 172.21.15.3:0/2344952997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 0x7fa278140a80 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27f4c3700 1 --2- 172.21.15.3:0/2344952997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 0x7fa278061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27f4c3700 1 -- 172.21.15.3:0/2344952997 >> 172.21.15.3:0/2344952997 conn(0x7fa278066f70 msgr2=0x7fa278067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:03.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27f4c3700 1 -- 172.21.15.3:0/2344952997 shutdown_connections 2022-07-21T01:39:03.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.113+0000 7fa27f4c3700 1 -- 172.21.15.3:0/2344952997 wait complete. 2022-07-21T01:39:03.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27f4c3700 1 Processor -- start 2022-07-21T01:39:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27f4c3700 1 -- start start 2022-07-21T01:39:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27f4c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2780614b0 0x7fa2780abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27f4c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 0x7fa2780b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27f4c3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 0x7fa2780b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27f4c3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2780ab0f0 con 0x7fa2781405d0 2022-07-21T01:39:03.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27f4c3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2780aa320 con 0x7fa2780614b0 2022-07-21T01:39:03.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27f4c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2780aa5c0 con 0x7fa2780ad750 2022-07-21T01:39:03.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27ecc2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 0x7fa2780b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27e4c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2780614b0 0x7fa2780abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27ecc2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 0x7fa2780b39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46658/0 (socket says 172.21.15.3:46658) 2022-07-21T01:39:03.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27ecc2700 1 -- 172.21.15.3:0/540992524 learned_addr learned my addr 172.21.15.3:0/540992524 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:03.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27dcc0700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 0x7fa2780b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27ecc2700 1 -- 172.21.15.3:0/540992524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 msgr2=0x7fa2780b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27ecc2700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 0x7fa2780b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27ecc2700 1 -- 172.21.15.3:0/540992524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2780614b0 msgr2=0x7fa2780abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27ecc2700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2780614b0 0x7fa2780abfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27ecc2700 1 -- 172.21.15.3:0/540992524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa270009470 con 0x7fa2781405d0 2022-07-21T01:39:03.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.117+0000 7fa27dcc0700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 0x7fa2780b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:39:03.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa27ecc2700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 0x7fa2780b39f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fa2700032f0 tx=0x7fa270016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:03.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa26b7fe700 1 -- 172.21.15.3:0/540992524 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa27000b600 con 0x7fa2781405d0 2022-07-21T01:39:03.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa26b7fe700 1 -- 172.21.15.3:0/540992524 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa270011c10 con 0x7fa2781405d0 2022-07-21T01:39:03.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2780b3ee0 con 0x7fa2781405d0 2022-07-21T01:39:03.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa26b7fe700 1 -- 172.21.15.3:0/540992524 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa27001f3f0 con 0x7fa2781405d0 2022-07-21T01:39:03.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2780aacc0 con 0x7fa2781405d0 2022-07-21T01:39:03.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2780aa780 con 0x7fa2781405d0 2022-07-21T01:39:03.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa26b7fe700 1 -- 172.21.15.3:0/540992524 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa270026030 con 0x7fa2781405d0 2022-07-21T01:39:03.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa26b7fe700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa258053980 0x7fa258055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa26b7fe700 1 -- 172.21.15.3:0/540992524 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fa27006c7b0 con 0x7fa2781405d0 2022-07-21T01:39:03.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.121+0000 7fa27e4c1700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa258053980 0x7fa258055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.125+0000 7fa27e4c1700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa258053980 0x7fa258055e40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fa26c00d000 tx=0x7fa26c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:03.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.129+0000 7fa26b7fe700 1 -- 172.21.15.3:0/540992524 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa270043850 con 0x7fa2781405d0 2022-07-21T01:39:03.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.257+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa2780af5f0 con 0x7fa2781405d0 2022-07-21T01:39:03.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.265+0000 7fa26b7fe700 1 -- 172.21.15.3:0/540992524 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa2700150c0 con 0x7fa2781405d0 2022-07-21T01:39:03.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa258053980 msgr2=0x7fa258055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa258053980 0x7fa258055e40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fa26c00d000 tx=0x7fa26c009040).stop 2022-07-21T01:39:03.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 msgr2=0x7fa2780b39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 0x7fa2780b39f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fa2700032f0 tx=0x7fa270016000).stop 2022-07-21T01:39:03.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 shutdown_connections 2022-07-21T01:39:03.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2781405d0 0x7fa2780b39f0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2780ad750 0x7fa2780b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa258053980 0x7fa258055e40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 --2- 172.21.15.3:0/540992524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2780614b0 0x7fa2780abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 >> 172.21.15.3:0/540992524 conn(0x7fa278066f70 msgr2=0x7fa2780b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:03.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 shutdown_connections 2022-07-21T01:39:03.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.269+0000 7fa27f4c3700 1 -- 172.21.15.3:0/540992524 wait complete. 2022-07-21T01:39:03.281 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:39:03.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:39:03.303 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:39:03.325 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:39:03.347 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:39:03.369 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:39:03.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:03.370 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:39:03.370 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:39:03.378 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:39:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:39:03.379 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:39:03.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:03.380 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:39:03.381 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:39:03.388 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:39:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:39:03.389 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:39:03.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:39:03.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.501+0000 7faef7b76700 1 Processor -- start 2022-07-21T01:39:03.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.501+0000 7faef7b76700 1 -- start start 2022-07-21T01:39:03.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faef00b68f0 0x7faef013ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef013d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef013d860 con 0x7faef00aa210 2022-07-21T01:39:03.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef00ac6e0 con 0x7faef00b68f0 2022-07-21T01:39:03.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef00ac980 con 0x7faef005ff40 2022-07-21T01:39:03.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef6113700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef013d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef5912700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faef00b68f0 0x7faef013ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef6113700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef013d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:53582/0 (socket says 172.21.15.3:53582) 2022-07-21T01:39:03.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef6113700 1 -- 172.21.15.3:0/2844336786 learned_addr learned my addr 172.21.15.3:0/2844336786 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef6113700 1 -- 172.21.15.3:0/2844336786 >> v2:172.21.15.143:3300/0 conn(0x7faef00b68f0 msgr2=0x7faef013ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef6113700 1 --2- 172.21.15.3:0/2844336786 >> v2:172.21.15.143:3300/0 conn(0x7faef00b68f0 0x7faef013ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef6113700 1 -- 172.21.15.3:0/2844336786 >> v1:172.21.15.143:6789/0 conn(0x7faef00aa210 legacy=0x7faef0146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef6113700 1 -- 172.21.15.3:0/2844336786 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef00acc20 con 0x7faef005ff40 2022-07-21T01:39:03.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef6113700 1 --2- 172.21.15.3:0/2844336786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef013d370 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7faee80032f0 tx=0x7faee8009290).ready entity=mon.2 client_cookie=139bae6b5541c357 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:03.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef4910700 1 -- 172.21.15.3:0/2844336786 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee800b8a0 con 0x7faef005ff40 2022-07-21T01:39:03.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef4910700 1 -- 172.21.15.3:0/2844336786 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee800ba00 con 0x7faef005ff40 2022-07-21T01:39:03.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef4910700 1 -- 172.21.15.3:0/2844336786 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee800bb70 con 0x7faef005ff40 2022-07-21T01:39:03.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 -- 172.21.15.3:0/2844336786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 msgr2=0x7faef013d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 --2- 172.21.15.3:0/2844336786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef013d370 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7faee80032f0 tx=0x7faee8009290).stop 2022-07-21T01:39:03.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 -- 172.21.15.3:0/2844336786 shutdown_connections 2022-07-21T01:39:03.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 --2- 172.21.15.3:0/2844336786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef013d370 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 --2- 172.21.15.3:0/2844336786 >> v2:172.21.15.143:3300/0 conn(0x7faef00b68f0 0x7faef013ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 -- 172.21.15.3:0/2844336786 >> 172.21.15.3:0/2844336786 conn(0x7faef00b3610 msgr2=0x7faef00b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:03.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 -- 172.21.15.3:0/2844336786 shutdown_connections 2022-07-21T01:39:03.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.505+0000 7faef7b76700 1 -- 172.21.15.3:0/2844336786 wait complete. 2022-07-21T01:39:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 Processor -- start 2022-07-21T01:39:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 -- start start 2022-07-21T01:39:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef00654c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef00aa210 0x7faef013f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faef00b68f0 0x7faef013c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef0067210 con 0x7faef00aa210 2022-07-21T01:39:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef0065ca0 con 0x7faef00b68f0 2022-07-21T01:39:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef00674b0 con 0x7faef005ff40 2022-07-21T01:39:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5111700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef00aa210 0x7faef013f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5111700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef00aa210 0x7faef013f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46670/0 (socket says 172.21.15.3:46670) 2022-07-21T01:39:03.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5111700 1 -- 172.21.15.3:0/751403051 learned_addr learned my addr 172.21.15.3:0/751403051 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:03.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5912700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef00654c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef6113700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faef00b68f0 0x7faef013c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5111700 1 -- 172.21.15.3:0/751403051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 msgr2=0x7faef00654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5111700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef00654c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5111700 1 -- 172.21.15.3:0/751403051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faef00b68f0 msgr2=0x7faef013c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5111700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faef00b68f0 0x7faef013c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5111700 1 -- 172.21.15.3:0/751403051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8009530 con 0x7faef00aa210 2022-07-21T01:39:03.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef6113700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faef00b68f0 0x7faef013c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:03.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef5111700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef00aa210 0x7faef013f690 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7faef00b5f50 tx=0x7faee4007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:03.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faee2ffd700 1 -- 172.21.15.3:0/751403051 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee4012400 con 0x7faef00aa210 2022-07-21T01:39:03.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faee2ffd700 1 -- 172.21.15.3:0/751403051 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee4012560 con 0x7faef00aa210 2022-07-21T01:39:03.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef0065240 con 0x7faef00aa210 2022-07-21T01:39:03.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faee2ffd700 1 -- 172.21.15.3:0/751403051 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee40128c0 con 0x7faef00aa210 2022-07-21T01:39:03.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.509+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef00ad5a0 con 0x7faef00aa210 2022-07-21T01:39:03.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.513+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef00aae50 con 0x7faef00aa210 2022-07-21T01:39:03.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.513+0000 7faee2ffd700 1 -- 172.21.15.3:0/751403051 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faee4019030 con 0x7faef00aa210 2022-07-21T01:39:03.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.513+0000 7faee2ffd700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faed0053b30 0x7faed0055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.513+0000 7faee2ffd700 1 -- 172.21.15.3:0/751403051 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5686+0+0 (secure 0 0 0) 0x7faee406b210 con 0x7faef00aa210 2022-07-21T01:39:03.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.513+0000 7faef5912700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faed0053b30 0x7faed0055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.513+0000 7faef5912700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faed0053b30 0x7faed0055ff0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7faeec00b000 tx=0x7faeec006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:03.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.521+0000 7faee2ffd700 1 -- 172.21.15.3:0/751403051 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faee4042140 con 0x7faef00aa210 2022-07-21T01:39:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.653+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7faef00acec0 con 0x7faef00aa210 2022-07-21T01:39:03.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.653+0000 7faee2ffd700 1 -- 172.21.15.3:0/751403051 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v71) v1 ==== 81+0+29222 (secure 0 0 0) 0x7faee4045760 con 0x7faef00aa210 2022-07-21T01:39:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:03.654 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:39:03.654 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 71, 2022-07-21T01:39:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:39:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:39:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:39:00.962480+0000", 2022-07-21T01:39:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:39:03.655 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:39:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:39:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:39:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:39:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:39:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:39:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:39:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:39:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:39:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:39:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:39:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:39:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:39:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:39:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 12, 2022-07-21T01:39:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:39:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:39:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:39:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:39:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:39:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:39:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:39:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:39:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:39:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:39:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:39:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:39:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:39:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:39:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:39:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:39:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:39:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:39:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:39:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:39:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:39:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:39:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:39:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:39:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:39:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:39:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:39:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:39:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:39:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:39:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:39:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:39:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:39:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:39:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:39:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:39:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:39:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:39:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:39:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:39:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:39:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:39:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:39:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:39:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:39:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:39:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:39:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:39:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:39:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:39:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:39:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:39:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:39:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:39:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:39:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:39:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:39:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:39:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:39:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:39:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:39:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:39:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:39:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:39:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:39:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:39:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:39:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:39:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:39:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:39:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:39:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:39:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:39:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:39:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:39:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:39:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:39:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:39:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:39:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:39:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:39:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:39:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:39:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:39:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:39:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:39:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:39:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:39:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:39:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:39:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:39:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:39:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:39:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:39:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:39:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:39:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:39:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:39:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:39:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:39:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:39:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:39:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:39:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:39:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:39:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:39:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:39:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:39:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:39:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:39:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:39:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:39:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:39:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:39:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:39:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:39:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:39:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:39:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:39:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:39:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:39:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:39:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:39:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:39:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:39:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:39:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:39:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:39:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:39:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:39:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:39:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:39:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:39:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:39:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.693 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:39:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:39:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.694 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:39:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:39:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:39:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:39:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:39:03.695 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:39:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:39:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:39:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:39:03.696 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 65, 2022-07-21T01:39:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:39:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:39:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:39:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:39:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:39:03.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:39:03.699 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:39:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:39:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:39:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:39:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.702 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:39:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:39:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:39:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.704 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:39:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:39:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:39:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:39:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.706 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:39:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:39:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:39:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:39:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:39:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.709 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:39:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:39:03.710 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:39:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:39:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:39:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:39:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:39:03.711 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:39:03.712 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:39:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:39:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:39:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:39:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:39:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:39:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:39:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:39:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 65, 2022-07-21T01:39:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:39:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:39:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:39:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:39:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:39:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.716 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:39:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:39:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.717 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:39:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.718 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:39:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:39:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:39:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:39:03.720 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:39:03.721 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:39:03.722 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:39:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.723 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:39:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:39:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.724 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:39:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.725 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.726 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:39:03.726 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:39:03.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:39:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:39:03.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:39:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:39:03.728 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:39:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:39:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:39:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:39:03.729 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:39:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.730 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:39:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:39:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:39:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:39:03.731 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:39:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:39:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:39:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:39:03.732 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:39:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 64, 2022-07-21T01:39:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:39:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:39:03.733 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:39:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.734 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:39:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:39:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:39:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:39:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.736 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:39:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.737 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:39:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:39:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:39:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:39:03.739 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:39:03.740 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:39:03.741 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:39:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.742 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:39:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:39:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:39:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.744 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:39:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:39:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.745 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:39:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:39:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.746 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:39:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:39:03.747 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:39:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:39:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:39:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:39:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:39:03.748 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:39:03.749 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:39:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:39:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:39:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:39:03.750 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:39:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:39:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:39:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:39:03.751 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 55, 2022-07-21T01:39:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:39:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:39:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:39:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:39:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:39:03.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:39:03.754 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:39:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.755 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:39:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.756 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:39:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:39:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.757 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:39:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:39:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.758 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:39:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:39:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:39:03.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:39:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:39:03.761 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:39:03.762 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:39:03.763 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:39:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.764 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:39:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:39:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.765 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:39:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:39:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:39:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:39:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:39:03.766 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:39:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:39:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:39:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:39:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:39:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:39:03.768 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:39:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:39:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:39:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:39:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:39:03.769 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 66, 2022-07-21T01:39:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:39:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:39:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:39:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:39:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:39:03.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:39:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:39:03.772 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:39:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:39:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:39:03.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:39:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:39:03.775 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:39:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:39:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:39:03.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:39:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:39:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:39:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:39:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:39:03.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:39:03.781 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:39:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:39:03.782 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:39:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:39:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:39:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:39:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:39:03.783 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:39:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.784 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:39:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:39:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:39:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:39:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:39:03.785 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:39:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:39:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:39:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:39:03.786 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 64, 2022-07-21T01:39:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:39:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:39:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:39:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:39:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:39:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:39:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:39:03.789 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.790 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.790 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:39:03.790 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.791 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.791 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:39:03.791 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:39:03.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.792 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.792 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:39:03.792 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:39:03.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.793 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.793 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:39:03.793 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.794 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.794 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:39:03.794 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:39:03.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.795 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.795 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:39:03.795 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:39:03.795 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.796 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.796 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:39:03.796 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.797 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:39:03.797 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:39:03.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.798 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.798 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:39:03.798 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:39:03.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.799 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.799 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:39:03.799 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:39:03.799 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:39:03.800 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:39:03.800 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:39:03.800 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:39:03.800 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:39:03.801 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.801 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:39:03.801 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:39:03.802 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:39:03.802 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:39:03.802 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:39:03.802 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:39:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:39:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:39:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:39:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 65, 2022-07-21T01:39:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:39:03.804 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:39:03.804 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:39:03.804 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.805 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.805 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:39:03.805 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:39:03.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.806 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.806 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:39:03.806 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:39:03.806 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.806 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.807 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:39:03.807 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.808 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.808 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:39:03.808 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:39:03.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.809 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.809 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:39:03.809 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:39:03.809 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.809 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.810 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:39:03.810 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.810 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.811 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:39:03.811 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:39:03.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.812 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.812 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:39:03.812 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:39:03.812 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.812 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.813 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:39:03.813 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.814 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.814 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:39:03.814 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:39:03.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.815 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.815 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:39:03.815 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:39:03.815 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.816 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.816 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:39:03.816 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:39:03.816 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:39:03.817 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:39:03.817 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:39:03.817 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:39:03.817 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:39:03.817 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.818 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:39:03.818 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:39:03.819 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:39:03.819 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:39:03.819 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:39:03.819 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:39:03.820 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:39:03.820 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:39:03.820 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:39:03.820 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 64, 2022-07-21T01:39:03.821 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:39:03.821 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:39:03.821 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:39:03.821 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.822 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:39:03.822 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:39:03.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.823 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.823 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:39:03.824 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:39:03.824 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.824 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.825 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:39:03.825 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.825 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.826 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:39:03.826 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:39:03.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.827 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:39:03.827 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:39:03.827 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.828 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.828 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:39:03.828 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.829 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.829 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:39:03.829 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:39:03.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.830 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.830 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:39:03.831 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:39:03.831 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.831 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.832 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:39:03.832 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:39:03.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.832 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:39:03.832 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:39:03.833 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:39:03.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.833 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:39:03.834 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:39:03.834 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:39:03.834 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.834 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.835 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:39:03.835 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:39:03.835 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:39:03.836 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:39:03.836 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:39:03.836 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:39:03.836 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:39:03.837 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:03.837 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.837 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:39:03.837 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:39:03.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.838 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:39:03.838 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:39:03.838 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:39:03.838 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:39:03.839 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:39:03.839 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:39:03.839 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:39:03.839 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:39:03.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.840 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:39:03.840 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:39:03.840 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:39:03.840 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:39:03.841 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:39:03.841 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:39:03.841 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:39:03.841 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:39:03.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.842 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:39:03.842 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:39:03.843 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:39:03.843 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:39:03.843 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:39:03.843 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:39:03.844 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:39:03.844 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:39:03.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.844 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:39:03.845 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:39:03.845 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:39:03.845 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:39:03.845 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:39:03.846 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:39:03.846 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:39:03.846 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:39:03.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.847 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:39:03.847 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:39:03.847 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:39:03.847 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:39:03.848 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:39:03.848 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:39:03.848 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:39:03.848 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:39:03.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.849 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:39:03.849 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:39:03.849 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:39:03.850 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:39:03.850 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:39:03.850 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:39:03.850 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:39:03.851 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:39:03.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.851 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:39:03.851 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:39:03.852 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:39:03.852 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:39:03.852 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:39:03.853 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:39:03.853 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:39:03.853 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:39:03.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:03.854 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:39:03.854 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:39:03.854 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:39:03.854 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:39:03.855 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:39:03.855 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:39:03.855 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:39:03.855 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:39:03.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.856 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:39:03.856 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:39:03.856 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:39:03.856 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:39:03.857 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:39:03.857 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:39:03.857 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.143:6833/2524169670": "2022-07-22T01:38:56.945811+0000", 2022-07-21T01:39:03.857 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.143:6832/2524169670": "2022-07-22T01:38:56.945811+0000" 2022-07-21T01:39:03.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.858 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:39:03.858 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:39:03.858 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:39:03.858 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:39:03.859 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:39:03.859 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:39:03.859 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:39:03.859 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:39:03.859 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:39:03.860 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:39:03.860 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:39:03.860 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:39:03.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.860 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:39:03.861 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:39:03.861 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:39:03.861 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:39:03.861 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:39:03.862 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:39:03.862 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:39:03.862 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:03.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:03.862 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:39:03.863 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:39:03.863 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:39:03.863 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:39:03.863 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:39:03.863 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:39:03.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faed0053b30 msgr2=0x7faed0055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faed0053b30 0x7faed0055ff0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7faeec00b000 tx=0x7faeec006df0).stop 2022-07-21T01:39:03.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef00aa210 msgr2=0x7faef013f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef00aa210 0x7faef013f690 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7faef00b5f50 tx=0x7faee4007c50).stop 2022-07-21T01:39:03.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 shutdown_connections 2022-07-21T01:39:03.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faed0053b30 0x7faed0055ff0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faef00b68f0 0x7faef013c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef00aa210 0x7faef013f690 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 --2- 172.21.15.3:0/751403051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faef005ff40 0x7faef00654c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 >> 172.21.15.3:0/751403051 conn(0x7faef00b3610 msgr2=0x7faef0141eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:03.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 shutdown_connections 2022-07-21T01:39:03.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.657+0000 7faef7b76700 1 -- 172.21.15.3:0/751403051 wait complete. 2022-07-21T01:39:03.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:39:03.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b4621700 1 Processor -- start 2022-07-21T01:39:03.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b4621700 1 -- start start 2022-07-21T01:39:03.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b4621700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb3ac067180 0x7fb3ac143a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b4621700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac0b0870 0x7fb3ac0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:03.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b4621700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ac067560 con 0x7fb3ac0ad1c0 2022-07-21T01:39:03.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b4621700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ac0b0160 con 0x7fb3ac067180 2022-07-21T01:39:03.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b4621700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0b0400 con 0x7fb3ac0b0870 2022-07-21T01:39:03.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b23bd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb3ac067180 0x7fb3ac143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b1bbc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb3ac0ad1c0 0x7fb3ac143f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54958/0 (socket says 172.21.15.3:54958) 2022-07-21T01:39:03.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b1bbc700 1 -- 172.21.15.3:0/3281713935 learned_addr learned my addr 172.21.15.3:0/3281713935 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:03.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b2bbe700 1 --2- 172.21.15.3:0/3281713935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac0b0870 0x7fb3ac0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:03.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b23bd700 1 -- 172.21.15.3:0/3281713935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac0b0870 msgr2=0x7fb3ac0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:03.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b23bd700 1 --2- 172.21.15.3:0/3281713935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac0b0870 0x7fb3ac0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b23bd700 1 -- 172.21.15.3:0/3281713935 >> v1:172.21.15.3:6789/0 conn(0x7fb3ac0ad1c0 legacy=0x7fb3ac143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b13bb700 1 -- 172.21.15.3:0/3281713935 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299347289 0 0) 0x7fb3ac067560 con 0x7fb3ac0ad1c0 2022-07-21T01:39:03.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b23bd700 1 -- 172.21.15.3:0/3281713935 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac0af530 con 0x7fb3ac067180 2022-07-21T01:39:03.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.981+0000 7fb3b23bd700 1 --2- 172.21.15.3:0/3281713935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac067180 0x7fb3ac143a60 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fb3a40032f0 tx=0x7fb3a40091d0).ready entity=mon.0 client_cookie=8ed4f93bc51c1d66 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:03.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b13bb700 1 -- 172.21.15.3:0/3281713935 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3a4009ea0 con 0x7fb3ac067180 2022-07-21T01:39:03.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b13bb700 1 -- 172.21.15.3:0/3281713935 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a400b8a0 con 0x7fb3ac067180 2022-07-21T01:39:03.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b13bb700 1 -- 172.21.15.3:0/3281713935 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3a400ba10 con 0x7fb3ac067180 2022-07-21T01:39:03.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 -- 172.21.15.3:0/3281713935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac067180 msgr2=0x7fb3ac143a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:03.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 --2- 172.21.15.3:0/3281713935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac067180 0x7fb3ac143a60 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fb3a40032f0 tx=0x7fb3a40091d0).stop 2022-07-21T01:39:03.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 -- 172.21.15.3:0/3281713935 shutdown_connections 2022-07-21T01:39:03.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 --2- 172.21.15.3:0/3281713935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac0b0870 0x7fb3ac0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 --2- 172.21.15.3:0/3281713935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac067180 0x7fb3ac143a60 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:03.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 -- 172.21.15.3:0/3281713935 >> 172.21.15.3:0/3281713935 conn(0x7fb3ac05f840 msgr2=0x7fb3ac05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:03.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 -- 172.21.15.3:0/3281713935 shutdown_connections 2022-07-21T01:39:03.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 -- 172.21.15.3:0/3281713935 wait complete. 2022-07-21T01:39:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 Processor -- start 2022-07-21T01:39:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.985+0000 7fb3b4621700 1 -- start start 2022-07-21T01:39:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b4621700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac067180 0x7fb3ac0b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b4621700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0b0870 0x7fb3ac152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b4621700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ac0b5d90 0x7fb3ac152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b4621700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0abe20 con 0x7fb3ac0b0870 2022-07-21T01:39:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b4621700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0ac630 con 0x7fb3ac0b5d90 2022-07-21T01:39:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b4621700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac140410 con 0x7fb3ac067180 2022-07-21T01:39:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b1bbc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0b0870 0x7fb3ac152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b23bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac067180 0x7fb3ac0b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b1bbc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0b0870 0x7fb3ac152970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46684/0 (socket says 172.21.15.3:46684) 2022-07-21T01:39:04.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b1bbc700 1 -- 172.21.15.3:0/253231660 learned_addr learned my addr 172.21.15.3:0/253231660 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:04.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b2bbe700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ac0b5d90 0x7fb3ac152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b1bbc700 1 -- 172.21.15.3:0/253231660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac067180 msgr2=0x7fb3ac0b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b1bbc700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac067180 0x7fb3ac0b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b1bbc700 1 -- 172.21.15.3:0/253231660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ac0b5d90 msgr2=0x7fb3ac152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b1bbc700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ac0b5d90 0x7fb3ac152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b1bbc700 1 -- 172.21.15.3:0/253231660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a4009470 con 0x7fb3ac0b0870 2022-07-21T01:39:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b1bbc700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0b0870 0x7fb3ac152970 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fb39c000ed0 tx=0x7fb39c007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb39b7fe700 1 -- 172.21.15.3:0/253231660 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb39c0070d0 con 0x7fb3ac0b0870 2022-07-21T01:39:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb39b7fe700 1 -- 172.21.15.3:0/253231660 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c007230 con 0x7fb3ac0b0870 2022-07-21T01:39:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb39b7fe700 1 -- 172.21.15.3:0/253231660 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb39c005610 con 0x7fb3ac0b0870 2022-07-21T01:39:04.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ac1406b0 con 0x7fb3ac0b0870 2022-07-21T01:39:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.989+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ac0ac2b0 con 0x7fb3ac0b0870 2022-07-21T01:39:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.993+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ac0ac830 con 0x7fb3ac0b0870 2022-07-21T01:39:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.993+0000 7fb39b7fe700 1 -- 172.21.15.3:0/253231660 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb39c010070 con 0x7fb3ac0b0870 2022-07-21T01:39:04.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.993+0000 7fb39b7fe700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb38c0539e0 0x7fb38c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.993+0000 7fb39b7fe700 1 -- 172.21.15.3:0/253231660 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fb39c06b320 con 0x7fb3ac0b0870 2022-07-21T01:39:04.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.993+0000 7fb3b23bd700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb38c0539e0 0x7fb38c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.993+0000 7fb3b23bd700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb38c0539e0 0x7fb38c055ea0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fb3a401b000 tx=0x7fb3a4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:04.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:03.997+0000 7fb39b7fe700 1 -- 172.21.15.3:0/253231660 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb39c0423c0 con 0x7fb3ac0b0870 2022-07-21T01:39:04.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.157+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3ac140840 con 0x7fb3ac0b0870 2022-07-21T01:39:04.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.157+0000 7fb39b7fe700 1 -- 172.21.15.3:0/253231660 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1559 (secure 0 0 0) 0x7fb39c0459e0 con 0x7fb3ac0b0870 2022-07-21T01:39:04.157 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:04.158 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5564,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1568809646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1568809646},{"type":"v1","addr":"172.21.15.143:6833","nonce":1568809646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":32},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[]} 2022-07-21T01:39:04.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb38c0539e0 msgr2=0x7fb38c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb38c0539e0 0x7fb38c055ea0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fb3a401b000 tx=0x7fb3a4016000).stop 2022-07-21T01:39:04.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0b0870 msgr2=0x7fb3ac152970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0b0870 0x7fb3ac152970 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fb39c000ed0 tx=0x7fb39c007a80).stop 2022-07-21T01:39:04.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 shutdown_connections 2022-07-21T01:39:04.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb38c0539e0 0x7fb38c055ea0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ac0b5d90 0x7fb3ac152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0b0870 0x7fb3ac152970 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 --2- 172.21.15.3:0/253231660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3ac067180 0x7fb3ac0b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 >> 172.21.15.3:0/253231660 conn(0x7fb3ac05f840 msgr2=0x7fb3ac143a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:04.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 shutdown_connections 2022-07-21T01:39:04.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.161+0000 7fb3b4621700 1 -- 172.21.15.3:0/253231660 wait complete. 2022-07-21T01:39:04.163 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 32 2022-07-21T01:39:04.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:39:04.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.277+0000 7f6bd679d700 1 Processor -- start 2022-07-21T01:39:04.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.277+0000 7f6bd679d700 1 -- start start 2022-07-21T01:39:04.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6bd0148530 0x7f6bd00b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6bd00a7250 0x7f6bd01448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd00b04e0 con 0x7f6bd00b1f40 2022-07-21T01:39:04.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd00a9250 con 0x7f6bd00a7250 2022-07-21T01:39:04.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd00b0780 con 0x7f6bd0148530 2022-07-21T01:39:04.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bcffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6bd00b1f40 0x7f6bd00b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54970/0 (socket says 172.21.15.3:54970) 2022-07-21T01:39:04.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bcffff700 1 -- 172.21.15.3:0/1784924373 learned_addr learned my addr 172.21.15.3:0/1784924373 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:04.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd4d3a700 1 --2- 172.21.15.3:0/1784924373 >> v2:172.21.15.3:3300/0 conn(0x7f6bd00a7250 0x7f6bd01448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bcf7fe700 1 --2- 172.21.15.3:0/1784924373 >> v2:172.21.15.143:3300/0 conn(0x7f6bd0148530 0x7f6bd00b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bceffd700 1 -- 172.21.15.3:0/1784924373 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042950016 0 0) 0x7f6bd00b04e0 con 0x7f6bd00b1f40 2022-07-21T01:39:04.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bceffd700 1 -- 172.21.15.3:0/1784924373 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bbc003260 con 0x7f6bd00b1f40 2022-07-21T01:39:04.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bcf7fe700 1 -- 172.21.15.3:0/1784924373 >> v2:172.21.15.3:3300/0 conn(0x7f6bd00a7250 msgr2=0x7f6bd01448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:04.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bcf7fe700 1 --2- 172.21.15.3:0/1784924373 >> v2:172.21.15.3:3300/0 conn(0x7f6bd00a7250 0x7f6bd01448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bcf7fe700 1 -- 172.21.15.3:0/1784924373 >> v1:172.21.15.3:6789/0 conn(0x7f6bd00b1f40 legacy=0x7f6bd00b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bcf7fe700 1 -- 172.21.15.3:0/1784924373 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd00b0a20 con 0x7f6bd0148530 2022-07-21T01:39:04.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bcf7fe700 1 --2- 172.21.15.3:0/1784924373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd0148530 0x7f6bd00b1160 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f6bd01478c0 tx=0x7f6bc0006df0).ready entity=mon.1 client_cookie=7a2f48a803dabe23 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:04.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bceffd700 1 -- 172.21.15.3:0/1784924373 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc000d4f0 con 0x7f6bd0148530 2022-07-21T01:39:04.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bceffd700 1 -- 172.21.15.3:0/1784924373 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc0000e90 con 0x7f6bd0148530 2022-07-21T01:39:04.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bceffd700 1 -- 172.21.15.3:0/1784924373 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc000d840 con 0x7f6bd0148530 2022-07-21T01:39:04.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 -- 172.21.15.3:0/1784924373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd0148530 msgr2=0x7f6bd00b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 --2- 172.21.15.3:0/1784924373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd0148530 0x7f6bd00b1160 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f6bd01478c0 tx=0x7f6bc0006df0).stop 2022-07-21T01:39:04.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 -- 172.21.15.3:0/1784924373 shutdown_connections 2022-07-21T01:39:04.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 --2- 172.21.15.3:0/1784924373 >> v2:172.21.15.3:3300/0 conn(0x7f6bd00a7250 0x7f6bd01448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 --2- 172.21.15.3:0/1784924373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd0148530 0x7f6bd00b1160 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 -- 172.21.15.3:0/1784924373 >> 172.21.15.3:0/1784924373 conn(0x7f6bd005f840 msgr2=0x7f6bd005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:04.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 -- 172.21.15.3:0/1784924373 shutdown_connections 2022-07-21T01:39:04.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.281+0000 7f6bd679d700 1 -- 172.21.15.3:0/1784924373 wait complete. 2022-07-21T01:39:04.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bd679d700 1 Processor -- start 2022-07-21T01:39:04.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bd679d700 1 -- start start 2022-07-21T01:39:04.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bd679d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bd00a7250 0x7f6bd014a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bd679d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd00b1f40 0x7f6bd00ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bd679d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd0148530 0x7f6bd00a9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bd679d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd01574a0 con 0x7f6bd00a7250 2022-07-21T01:39:04.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bd679d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd014bab0 con 0x7f6bd00b1f40 2022-07-21T01:39:04.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bd679d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd014b590 con 0x7f6bd0148530 2022-07-21T01:39:04.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bd00a7250 0x7f6bd014a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bd00a7250 0x7f6bd014a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46696/0 (socket says 172.21.15.3:46696) 2022-07-21T01:39:04.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcffff700 1 -- 172.21.15.3:0/713863679 learned_addr learned my addr 172.21.15.3:0/713863679 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:04.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcf7fe700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd00b1f40 0x7f6bd00ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bd4d3a700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd0148530 0x7f6bd00a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcffff700 1 -- 172.21.15.3:0/713863679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd0148530 msgr2=0x7f6bd00a9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcffff700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd0148530 0x7f6bd00a9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcffff700 1 -- 172.21.15.3:0/713863679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd00b1f40 msgr2=0x7f6bd00ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcffff700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd00b1f40 0x7f6bd00ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcffff700 1 -- 172.21.15.3:0/713863679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc000b180 con 0x7f6bd00a7250 2022-07-21T01:39:04.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.285+0000 7f6bcffff700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bd00a7250 0x7f6bd014a750 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f6bc8001490 tx=0x7f6bc8007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:04.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bcd7fa700 1 -- 172.21.15.3:0/713863679 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc8005ba0 con 0x7f6bd00a7250 2022-07-21T01:39:04.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bcd7fa700 1 -- 172.21.15.3:0/713863679 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8005d00 con 0x7f6bd00a7250 2022-07-21T01:39:04.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd014b810 con 0x7f6bd00a7250 2022-07-21T01:39:04.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bcd7fa700 1 -- 172.21.15.3:0/713863679 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc80072c0 con 0x7f6bd00a7250 2022-07-21T01:39:04.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd014acc0 con 0x7f6bd00a7250 2022-07-21T01:39:04.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd00b0cc0 con 0x7f6bd00a7250 2022-07-21T01:39:04.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bcd7fa700 1 -- 172.21.15.3:0/713863679 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6bc801a070 con 0x7f6bd00a7250 2022-07-21T01:39:04.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bcd7fa700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053aa0 0x7f6bb0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bcd7fa700 1 -- 172.21.15.3:0/713863679 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f6bc806b380 con 0x7f6bd00a7250 2022-07-21T01:39:04.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bcf7fe700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053aa0 0x7f6bb0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.289+0000 7f6bcf7fe700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053aa0 0x7f6bb0055f60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f6bc001a000 tx=0x7f6bc0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:04.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.297+0000 7f6bcd7fa700 1 -- 172.21.15.3:0/713863679 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6bc80423c0 con 0x7f6bd00a7250 2022-07-21T01:39:04.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.461+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bd00574e0 con 0x7f6bd00a7250 2022-07-21T01:39:04.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.461+0000 7f6bcd7fa700 1 -- 172.21.15.3:0/713863679 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1559 (secure 0 0 0) 0x7f6bc8018070 con 0x7f6bd00a7250 2022-07-21T01:39:04.460 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:04.460 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5564,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1568809646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1568809646},{"type":"v1","addr":"172.21.15.143:6833","nonce":1568809646}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":32},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/4267673848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":4267673848},{"type":"v1","addr":"172.21.15.3:6835","nonce":4267673848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27},{"gid":5928,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/2003836554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2003836554},{"type":"v1","addr":"172.21.15.3:6837","nonce":2003836554}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":26}],"filesystems":[]} 2022-07-21T01:39:04.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053aa0 msgr2=0x7f6bb0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053aa0 0x7f6bb0055f60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f6bc001a000 tx=0x7f6bc0014000).stop 2022-07-21T01:39:04.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bd00a7250 msgr2=0x7f6bd014a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bd00a7250 0x7f6bd014a750 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f6bc8001490 tx=0x7f6bc8007e50).stop 2022-07-21T01:39:04.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 shutdown_connections 2022-07-21T01:39:04.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd0148530 0x7f6bd00a9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053aa0 0x7f6bb0055f60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd00b1f40 0x7f6bd00ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 --2- 172.21.15.3:0/713863679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bd00a7250 0x7f6bd014a750 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 >> 172.21.15.3:0/713863679 conn(0x7f6bd005f840 msgr2=0x7f6bd0147900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:04.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 shutdown_connections 2022-07-21T01:39:04.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.465+0000 7f6bd679d700 1 -- 172.21.15.3:0/713863679 wait complete. 2022-07-21T01:39:04.465 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 32 2022-07-21T01:39:04.472 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:39:04.473 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:39:04.473 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:39:04.473 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:39:04.474 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:39:04.474 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:39:04.474 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:39:04.475 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:39:04.475 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:39:04.475 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:39:04.475 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:39:04.476 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:39:04.490 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:39:04.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:39:04.492 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:39:04.493 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:39:04.493 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:39:04.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:39:04.495 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:39:04.495 DEBUG:teuthology.parallel:result is None 2022-07-21T01:39:04.496 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:39:04.497 DEBUG:teuthology.parallel:result is None 2022-07-21T01:39:04.497 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:39:04.498 DEBUG:teuthology.parallel:result is None 2022-07-21T01:39:04.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:39:04.556 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:39:04.566 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:39:04.569 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:39:04.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.601+0000 7fe10652a700 1 Processor -- start 2022-07-21T01:39:04.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.601+0000 7fe10652a700 1 -- start start 2022-07-21T01:39:04.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b7080 0x7fe10012ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe1000b4300 0x7fe100146890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe100061e00 con 0x7fe100061a20 2022-07-21T01:39:04.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1000b2e40 con 0x7fe1000b4300 2022-07-21T01:39:04.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10012c090 con 0x7fe1000b7080 2022-07-21T01:39:04.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0ff7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b7080 0x7fe10012ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe104ac7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe1000b4300 0x7fe100146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0ff7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b7080 0x7fe10012ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:53644/0 (socket says 172.21.15.3:53644) 2022-07-21T01:39:04.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0ff7fe700 1 -- 172.21.15.3:0/3568211688 learned_addr learned my addr 172.21.15.3:0/3568211688 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:04.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0ff7fe700 1 -- 172.21.15.3:0/3568211688 >> v2:172.21.15.143:3300/0 conn(0x7fe1000b4300 msgr2=0x7fe100146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:04.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0ff7fe700 1 --2- 172.21.15.3:0/3568211688 >> v2:172.21.15.143:3300/0 conn(0x7fe1000b4300 0x7fe100146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0ff7fe700 1 -- 172.21.15.3:0/3568211688 >> v1:172.21.15.143:6789/0 conn(0x7fe100061a20 legacy=0x7fe10012b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0ff7fe700 1 -- 172.21.15.3:0/3568211688 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10012c330 con 0x7fe1000b7080 2022-07-21T01:39:04.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe104ac7700 1 --2- 172.21.15.3:0/3568211688 >> v2:172.21.15.143:3300/0 conn(0x7fe1000b4300 0x7fe100146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:04.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0ff7fe700 1 --2- 172.21.15.3:0/3568211688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b7080 0x7fe10012ba00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe0e80032f0 tx=0x7fe0e8009290).ready entity=mon.2 client_cookie=7062f12f9e06c1c1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:04.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0feffd700 1 -- 172.21.15.3:0/3568211688 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0e800b8a0 con 0x7fe1000b7080 2022-07-21T01:39:04.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0feffd700 1 -- 172.21.15.3:0/3568211688 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e800ba00 con 0x7fe1000b7080 2022-07-21T01:39:04.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe0feffd700 1 -- 172.21.15.3:0/3568211688 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0e800bb70 con 0x7fe1000b7080 2022-07-21T01:39:04.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 -- 172.21.15.3:0/3568211688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b7080 msgr2=0x7fe10012ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 --2- 172.21.15.3:0/3568211688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b7080 0x7fe10012ba00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe0e80032f0 tx=0x7fe0e8009290).stop 2022-07-21T01:39:04.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 -- 172.21.15.3:0/3568211688 shutdown_connections 2022-07-21T01:39:04.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 --2- 172.21.15.3:0/3568211688 >> v2:172.21.15.143:3300/0 conn(0x7fe1000b4300 0x7fe100146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 --2- 172.21.15.3:0/3568211688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b7080 0x7fe10012ba00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 -- 172.21.15.3:0/3568211688 >> 172.21.15.3:0/3568211688 conn(0x7fe100066f70 msgr2=0x7fe100067380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 -- 172.21.15.3:0/3568211688 shutdown_connections 2022-07-21T01:39:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.605+0000 7fe10652a700 1 -- 172.21.15.3:0/3568211688 wait complete. 2022-07-21T01:39:04.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe10652a700 1 Processor -- start 2022-07-21T01:39:04.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe10652a700 1 -- start start 2022-07-21T01:39:04.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe10652a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe100061a20 0x7fe100144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe10652a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b4300 0x7fe100142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe10652a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1000b7080 0x7fe100143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe10652a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe100142460 con 0x7fe100061a20 2022-07-21T01:39:04.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe10652a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe100142a00 con 0x7fe1000b7080 2022-07-21T01:39:04.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe10652a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe100143810 con 0x7fe1000b4300 2022-07-21T01:39:04.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0fffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe100061a20 0x7fe100144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0fffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe100061a20 0x7fe100144c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46734/0 (socket says 172.21.15.3:46734) 2022-07-21T01:39:04.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0fffff700 1 -- 172.21.15.3:0/1942414875 learned_addr learned my addr 172.21.15.3:0/1942414875 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:04.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe104ac7700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1000b7080 0x7fe100143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0ff7fe700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b4300 0x7fe100142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0fffff700 1 -- 172.21.15.3:0/1942414875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b4300 msgr2=0x7fe100142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0fffff700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b4300 0x7fe100142e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0fffff700 1 -- 172.21.15.3:0/1942414875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1000b7080 msgr2=0x7fe100143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:04.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0fffff700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1000b7080 0x7fe100143320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:04.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0fffff700 1 -- 172.21.15.3:0/1942414875 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f800b180 con 0x7fe100061a20 2022-07-21T01:39:04.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.609+0000 7fe0fffff700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe100061a20 0x7fe100144c90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fe1000ad590 tx=0x7fe0f0007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:04.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.613+0000 7fe0fd7fa700 1 -- 172.21.15.3:0/1942414875 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0f0007030 con 0x7fe100061a20 2022-07-21T01:39:04.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.613+0000 7fe0fd7fa700 1 -- 172.21.15.3:0/1942414875 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f0007190 con 0x7fe100061a20 2022-07-21T01:39:04.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.613+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e8009530 con 0x7fe100061a20 2022-07-21T01:39:04.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.613+0000 7fe0fd7fa700 1 -- 172.21.15.3:0/1942414875 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0f0006510 con 0x7fe100061a20 2022-07-21T01:39:04.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.613+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1001410f0 con 0x7fe100061a20 2022-07-21T01:39:04.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.617+0000 7fe0fd7fa700 1 -- 172.21.15.3:0/1942414875 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe0f0006670 con 0x7fe100061a20 2022-07-21T01:39:04.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.617+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe100142b90 con 0x7fe100061a20 2022-07-21T01:39:04.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.617+0000 7fe0fd7fa700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe0e4053a70 0x7fe0e4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:04.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.617+0000 7fe0fd7fa700 1 -- 172.21.15.3:0/1942414875 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fe0f006b680 con 0x7fe100061a20 2022-07-21T01:39:04.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.617+0000 7fe0ff7fe700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe0e4053a70 0x7fe0e4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:04.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.617+0000 7fe0ff7fe700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe0e4053a70 0x7fe0e4055f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fe0e801b000 tx=0x7fe0e8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:04.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.621+0000 7fe0fd7fa700 1 -- 172.21.15.3:0/1942414875 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe100142b90 con 0x7fe100061a20 2022-07-21T01:39:04.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:04.757+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fe100142b90 con 0x7fe100061a20 2022-07-21T01:39:05.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.617+0000 7fe0fd7fa700 1 -- 172.21.15.3:0/1942414875 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v75) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe100142b90 con 0x7fe100061a20 2022-07-21T01:39:05.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe0e4053a70 msgr2=0x7fe0e4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:05.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe0e4053a70 0x7fe0e4055f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fe0e801b000 tx=0x7fe0e8016000).stop 2022-07-21T01:39:05.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe100061a20 msgr2=0x7fe100144c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:05.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe100061a20 0x7fe100144c90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fe1000ad590 tx=0x7fe0f0007be0).stop 2022-07-21T01:39:05.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 shutdown_connections 2022-07-21T01:39:05.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe0e4053a70 0x7fe0e4055f30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1000b7080 0x7fe100143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1000b4300 0x7fe100142e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 --2- 172.21.15.3:0/1942414875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe100061a20 0x7fe100144c90 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 >> 172.21.15.3:0/1942414875 conn(0x7fe100066f70 msgr2=0x7fe1001452f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:05.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 shutdown_connections 2022-07-21T01:39:05.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.621+0000 7fe10652a700 1 -- 172.21.15.3:0/1942414875 wait complete. 2022-07-21T01:39:05.623 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:39:05.634 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:39:05.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:39:05.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1f59e700 1 Processor -- start 2022-07-21T01:39:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1f59e700 1 -- start start 2022-07-21T01:39:05.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8b200b2730 0x7f8b200666b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:05.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1f59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8b20066ba0 0x7f8b2014dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:05.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1f59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b200654c0 con 0x7f8b200b3190 2022-07-21T01:39:05.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1f59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b20062190 con 0x7f8b20066ba0 2022-07-21T01:39:05.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1e59c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8b200b3190 0x7f8b20060ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55026/0 (socket says 172.21.15.3:55026) 2022-07-21T01:39:05.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1e59c700 1 -- 172.21.15.3:0/4126157376 learned_addr learned my addr 172.21.15.3:0/4126157376 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:05.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1ed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8b20066ba0 0x7f8b2014dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:05.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1f59e700 1 -- 172.21.15.3:0/4126157376 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b200b3570 con 0x7f8b200b2730 2022-07-21T01:39:05.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b15d9b700 1 --2- 172.21.15.3:0/4126157376 >> v2:172.21.15.143:3300/0 conn(0x7f8b200b2730 0x7f8b200666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:05.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.753+0000 7f8b1dd9b700 1 -- 172.21.15.3:0/4126157376 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581262471 0 0) 0x7f8b200654c0 con 0x7f8b200b3190 2022-07-21T01:39:05.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1dd9b700 1 -- 172.21.15.3:0/4126157376 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b00003260 con 0x7f8b200b3190 2022-07-21T01:39:05.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1ed9d700 1 -- 172.21.15.3:0/4126157376 >> v2:172.21.15.143:3300/0 conn(0x7f8b200b2730 msgr2=0x7f8b200666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:05.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1ed9d700 1 --2- 172.21.15.3:0/4126157376 >> v2:172.21.15.143:3300/0 conn(0x7f8b200b2730 0x7f8b200666b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1ed9d700 1 -- 172.21.15.3:0/4126157376 >> v1:172.21.15.3:6789/0 conn(0x7f8b200b3190 legacy=0x7f8b20060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:05.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1ed9d700 1 -- 172.21.15.3:0/4126157376 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2005fbf0 con 0x7f8b20066ba0 2022-07-21T01:39:05.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1ed9d700 1 --2- 172.21.15.3:0/4126157376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b20066ba0 0x7f8b2014dab0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f8b180032f0 tx=0x7f8b180091d0).ready entity=mon.0 client_cookie=8a401bc244bea56e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:05.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1dd9b700 1 -- 172.21.15.3:0/4126157376 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b1800b680 con 0x7f8b20066ba0 2022-07-21T01:39:05.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1dd9b700 1 -- 172.21.15.3:0/4126157376 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1800b7e0 con 0x7f8b20066ba0 2022-07-21T01:39:05.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1dd9b700 1 -- 172.21.15.3:0/4126157376 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b1800b950 con 0x7f8b20066ba0 2022-07-21T01:39:05.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1f59e700 1 -- 172.21.15.3:0/4126157376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b20066ba0 msgr2=0x7f8b2014dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:05.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1f59e700 1 --2- 172.21.15.3:0/4126157376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b20066ba0 0x7f8b2014dab0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f8b180032f0 tx=0x7f8b180091d0).stop 2022-07-21T01:39:05.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1f59e700 1 -- 172.21.15.3:0/4126157376 shutdown_connections 2022-07-21T01:39:05.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1f59e700 1 --2- 172.21.15.3:0/4126157376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b20066ba0 0x7f8b2014dab0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1f59e700 1 --2- 172.21.15.3:0/4126157376 >> v2:172.21.15.143:3300/0 conn(0x7f8b200b2730 0x7f8b200666b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1f59e700 1 -- 172.21.15.3:0/4126157376 >> 172.21.15.3:0/4126157376 conn(0x7f8b20063870 msgr2=0x7f8b20063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1f59e700 1 -- 172.21.15.3:0/4126157376 shutdown_connections 2022-07-21T01:39:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.757+0000 7f8b1f59e700 1 -- 172.21.15.3:0/4126157376 wait complete. 2022-07-21T01:39:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 Processor -- start 2022-07-21T01:39:05.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 -- start start 2022-07-21T01:39:05.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b20066ba0 0x7f8b20159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:05.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b200b2730 0x7f8b200aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:05.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b200b3190 0x7f8b200ac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:05.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b200b6630 con 0x7f8b200b3190 2022-07-21T01:39:05.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b20141200 con 0x7f8b20066ba0 2022-07-21T01:39:05.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b200b6860 con 0x7f8b200b2730 2022-07-21T01:39:05.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b200b3190 0x7f8b200ac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b200b3190 0x7f8b200ac4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46756/0 (socket says 172.21.15.3:46756) 2022-07-21T01:39:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1ed9d700 1 -- 172.21.15.3:0/2642812799 learned_addr learned my addr 172.21.15.3:0/2642812799 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1e59c700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b20066ba0 0x7f8b20159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b15d9b700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b200b2730 0x7f8b200aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1ed9d700 1 -- 172.21.15.3:0/2642812799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b200b2730 msgr2=0x7f8b200aa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1ed9d700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b200b2730 0x7f8b200aa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1ed9d700 1 -- 172.21.15.3:0/2642812799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b20066ba0 msgr2=0x7f8b20159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1ed9d700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b20066ba0 0x7f8b20159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1ed9d700 1 -- 172.21.15.3:0/2642812799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18009470 con 0x7f8b200b3190 2022-07-21T01:39:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1e59c700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b20066ba0 0x7f8b20159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1ed9d700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b200b3190 0x7f8b200ac4d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f8b180032f0 tx=0x7f8b18016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b17fff700 1 -- 172.21.15.3:0/2642812799 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b18011d70 con 0x7f8b200b3190 2022-07-21T01:39:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b17fff700 1 -- 172.21.15.3:0/2642812799 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b18021440 con 0x7f8b200b3190 2022-07-21T01:39:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b200b6b00 con 0x7f8b200b3190 2022-07-21T01:39:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b17fff700 1 -- 172.21.15.3:0/2642812799 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b180217a0 con 0x7f8b200b3190 2022-07-21T01:39:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.761+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b200b1230 con 0x7f8b200b3190 2022-07-21T01:39:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.765+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b200b6c90 con 0x7f8b200b3190 2022-07-21T01:39:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.765+0000 7f8b17fff700 1 -- 172.21.15.3:0/2642812799 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8b18027020 con 0x7f8b200b3190 2022-07-21T01:39:05.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.765+0000 7f8b17fff700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b04053b00 0x7f8b04055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:05.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.765+0000 7f8b17fff700 1 -- 172.21.15.3:0/2642812799 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f8b1806d0a0 con 0x7f8b200b3190 2022-07-21T01:39:05.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.765+0000 7f8b1e59c700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b04053b00 0x7f8b04055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:05.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.765+0000 7f8b1e59c700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b04053b00 0x7f8b04055fc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f8b1000d000 tx=0x7f8b10009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:05.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.769+0000 7f8b17fff700 1 -- 172.21.15.3:0/2642812799 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8b180441a0 con 0x7f8b200b3190 2022-07-21T01:39:05.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.937+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8b2005fe90 con 0x7f8b200b3190 2022-07-21T01:39:05.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b17fff700 1 -- 172.21.15.3:0/2642812799 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f8b18015090 con 0x7f8b200b3190 2022-07-21T01:39:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:05.939 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:39:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:39:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:39:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:39:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:39:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:39:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:39:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:39:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:39:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:39:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:39:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:39:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:39:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:39:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:39:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:39:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:39:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:39:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:39:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:39:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:39:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:39:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:39:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.960 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:39:05.960 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:39:05.960 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.960 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:39:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:39:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:39:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:39:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:39:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:39:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:39:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:39:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:39:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:39:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:39:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:39:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:39:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:39:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:39:05.969 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:39:05.969 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:39:05.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.969 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.970 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:39:05.971 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:39:05.971 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.971 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:39:05.971 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:39:05.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.972 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:05.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:05.972 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:39:05.973 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:39:05.973 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:39:05.973 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:39:05.973 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:39:05.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:39:05.974 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.974 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:05.974 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:39:05.975 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:39:05.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b04053b00 msgr2=0x7f8b04055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:05.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b04053b00 0x7f8b04055fc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f8b1000d000 tx=0x7f8b10009040).stop 2022-07-21T01:39:05.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b200b3190 msgr2=0x7f8b200ac4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b200b3190 0x7f8b200ac4d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f8b180032f0 tx=0x7f8b18016000).stop 2022-07-21T01:39:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 shutdown_connections 2022-07-21T01:39:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b200b3190 0x7f8b200ac4d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b200b2730 0x7f8b200aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b04053b00 0x7f8b04055fc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 --2- 172.21.15.3:0/2642812799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b20066ba0 0x7f8b20159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:05.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 >> 172.21.15.3:0/2642812799 conn(0x7f8b20063870 msgr2=0x7f8b201488f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:05.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 shutdown_connections 2022-07-21T01:39:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:05.941+0000 7f8b1f59e700 1 -- 172.21.15.3:0/2642812799 wait complete. 2022-07-21T01:39:05.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:39:06.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6aa4856700 1 Processor -- start 2022-07-21T01:39:06.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6aa4856700 1 -- start start 2022-07-21T01:39:06.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6aa4856700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6aa00af450 0x7f6aa00af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6aa4856700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00ad4b0 0x7f6aa01432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6aa4856700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa00aff30 con 0x7f6aa01437d0 2022-07-21T01:39:06.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6aa4856700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa005c7a0 con 0x7f6aa00af450 2022-07-21T01:39:06.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6aa4856700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa0140990 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00ad4b0 0x7f6aa01432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9ed9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6aa00af450 0x7f6aa00af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9f59e700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f6aa01437d0 0x7f6aa00a8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:57158/0 (socket says 172.21.15.3:57158) 2022-07-21T01:39:06.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00ad4b0 0x7f6aa01432e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:53676/0 (socket says 172.21.15.3:53676) 2022-07-21T01:39:06.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9ed9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6aa00af450 0x7f6aa00af830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:52208/0 (socket says 172.21.15.3:52208) 2022-07-21T01:39:06.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9f59e700 1 -- 172.21.15.3:0/1862923468 learned_addr learned my addr 172.21.15.3:0/1862923468 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:06.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9e59c700 1 -- 172.21.15.3:0/1862923468 >> v2:172.21.15.143:3300/0 conn(0x7f6aa00af450 msgr2=0x7f6aa00af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:06.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9e59c700 1 --2- 172.21.15.3:0/1862923468 >> v2:172.21.15.143:3300/0 conn(0x7f6aa00af450 0x7f6aa00af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9e59c700 1 -- 172.21.15.3:0/1862923468 >> v1:172.21.15.143:6789/0 conn(0x7f6aa01437d0 legacy=0x7f6aa00a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9e59c700 1 -- 172.21.15.3:0/1862923468 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa00a9550 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.085+0000 7f6a9ed9d700 1 --2- 172.21.15.3:0/1862923468 >> v2:172.21.15.143:3300/0 conn(0x7f6aa00af450 0x7f6aa00af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:06.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6a9e59c700 1 --2- 172.21.15.3:0/1862923468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00ad4b0 0x7f6aa01432e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f6aa013fdc0 tx=0x7f6a94006e80).ready entity=mon.2 client_cookie=fa95cf131848261 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6a9dd9b700 1 -- 172.21.15.3:0/1862923468 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a9400f8a0 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6a9dd9b700 1 -- 172.21.15.3:0/1862923468 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a94000e90 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6a9dd9b700 1 -- 172.21.15.3:0/1862923468 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a9400fbf0 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6aa4856700 1 -- 172.21.15.3:0/1862923468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00ad4b0 msgr2=0x7f6aa01432e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6aa4856700 1 --2- 172.21.15.3:0/1862923468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00ad4b0 0x7f6aa01432e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f6aa013fdc0 tx=0x7f6a94006e80).stop 2022-07-21T01:39:06.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6aa4856700 1 -- 172.21.15.3:0/1862923468 shutdown_connections 2022-07-21T01:39:06.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6aa4856700 1 --2- 172.21.15.3:0/1862923468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00ad4b0 0x7f6aa01432e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6aa4856700 1 --2- 172.21.15.3:0/1862923468 >> v2:172.21.15.143:3300/0 conn(0x7f6aa00af450 0x7f6aa00af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6aa4856700 1 -- 172.21.15.3:0/1862923468 >> 172.21.15.3:0/1862923468 conn(0x7f6aa00612d0 msgr2=0x7f6aa0062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6aa4856700 1 -- 172.21.15.3:0/1862923468 shutdown_connections 2022-07-21T01:39:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.089+0000 7f6aa4856700 1 -- 172.21.15.3:0/1862923468 wait complete. 2022-07-21T01:39:06.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6aa4856700 1 Processor -- start 2022-07-21T01:39:06.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6aa4856700 1 -- start start 2022-07-21T01:39:06.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6aa4856700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6aa00ad4b0 0x7f6aa00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6aa4856700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00af450 0x7f6aa00a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6aa4856700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa01437d0 0x7f6aa00abeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6aa4856700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa0141f10 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6aa4856700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa01421b0 con 0x7f6aa01437d0 2022-07-21T01:39:06.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6aa4856700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa0142450 con 0x7f6aa00af450 2022-07-21T01:39:06.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6aa00ad4b0 0x7f6aa00b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6aa00ad4b0 0x7f6aa00b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46764/0 (socket says 172.21.15.3:46764) 2022-07-21T01:39:06.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9ed9d700 1 -- 172.21.15.3:0/3783334677 learned_addr learned my addr 172.21.15.3:0/3783334677 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:06.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9f59e700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa01437d0 0x7f6aa00abeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9e59c700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00af450 0x7f6aa00a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9ed9d700 1 -- 172.21.15.3:0/3783334677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00af450 msgr2=0x7f6aa00a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9ed9d700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00af450 0x7f6aa00a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9ed9d700 1 -- 172.21.15.3:0/3783334677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa01437d0 msgr2=0x7f6aa00abeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9ed9d700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa01437d0 0x7f6aa00abeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9ed9d700 1 -- 172.21.15.3:0/3783334677 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9000b180 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a9ed9d700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6aa00ad4b0 0x7f6aa00b1d50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f6a9000c000 tx=0x7f6a90006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.093+0000 7f6a8ffff700 1 -- 172.21.15.3:0/3783334677 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a9000c7e0 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.097+0000 7f6a8ffff700 1 -- 172.21.15.3:0/3783334677 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9000c940 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.097+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9400d410 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.097+0000 7f6a8ffff700 1 -- 172.21.15.3:0/3783334677 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a9000cca0 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.097+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa0155a70 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.097+0000 7f6a8ffff700 1 -- 172.21.15.3:0/3783334677 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6a9001c460 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.097+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa00574e0 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.097+0000 7f6a8ffff700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a7c0539e0 0x7f6a7c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.097+0000 7f6a8ffff700 1 -- 172.21.15.3:0/3783334677 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f6a9006c2c0 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.097+0000 7f6a9e59c700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a7c0539e0 0x7f6a7c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.101+0000 7f6a9e59c700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a7c0539e0 0x7f6a7c055ea0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f6a9401c000 tx=0x7f6a94006e80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.105+0000 7f6a8ffff700 1 -- 172.21.15.3:0/3783334677 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6a90043360 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.229+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6aa0061b20 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.229+0000 7f6a8ffff700 1 -- 172.21.15.3:0/3783334677 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+13415 (secure 0 0 0) 0x7f6a9000ce00 con 0x7f6aa00ad4b0 2022-07-21T01:39:06.228 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:06.228 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":75,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:39:05.611342+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:39:06.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.229+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a7c0539e0 msgr2=0x7f6a7c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a7c0539e0 0x7f6a7c055ea0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f6a9401c000 tx=0x7f6a94006e80).stop 2022-07-21T01:39:06.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6aa00ad4b0 msgr2=0x7f6aa00b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6aa00ad4b0 0x7f6aa00b1d50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f6a9000c000 tx=0x7f6a90006df0).stop 2022-07-21T01:39:06.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 shutdown_connections 2022-07-21T01:39:06.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a7c0539e0 0x7f6a7c055ea0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa01437d0 0x7f6aa00abeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa00af450 0x7f6aa00a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 --2- 172.21.15.3:0/3783334677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6aa00ad4b0 0x7f6aa00b1d50 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 >> 172.21.15.3:0/3783334677 conn(0x7f6aa00612d0 msgr2=0x7f6aa00b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:06.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 shutdown_connections 2022-07-21T01:39:06.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.233+0000 7f6aa4856700 1 -- 172.21.15.3:0/3783334677 wait complete. 2022-07-21T01:39:06.241 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:39:06.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:39:06.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61ec8cc700 1 Processor -- start 2022-07-21T01:39:06.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61ec8cc700 1 -- start start 2022-07-21T01:39:06.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61ec8cc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f61e0047a50 0x7f61e002a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61ec8cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61e0045110 0x7f61e0006530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61ec8cc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e0045890 con 0x7f61e0005900 2022-07-21T01:39:06.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61ec8cc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e002a190 con 0x7f61e0047a50 2022-07-21T01:39:06.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61ec8cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e002a430 con 0x7f61e0045110 2022-07-21T01:39:06.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e6d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f61e0047a50 0x7f61e002a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f61e0005900 0x7f61e00dba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55054/0 (socket says 172.21.15.3:55054) 2022-07-21T01:39:06.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e659c700 1 -- 172.21.15.3:0/4223295163 learned_addr learned my addr 172.21.15.3:0/4223295163 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:06.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e759e700 1 --2- 172.21.15.3:0/4223295163 >> v2:172.21.15.143:3300/0 conn(0x7f61e0045110 0x7f61e0006530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e6d9d700 1 -- 172.21.15.3:0/4223295163 >> v2:172.21.15.143:3300/0 conn(0x7f61e0045110 msgr2=0x7f61e0006530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:06.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e6d9d700 1 --2- 172.21.15.3:0/4223295163 >> v2:172.21.15.143:3300/0 conn(0x7f61e0045110 0x7f61e0006530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e5d9b700 1 -- 172.21.15.3:0/4223295163 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386512357 0 0) 0x7f61e0045890 con 0x7f61e0005900 2022-07-21T01:39:06.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e6d9d700 1 -- 172.21.15.3:0/4223295163 >> v1:172.21.15.3:6789/0 conn(0x7f61e0005900 legacy=0x7f61e00dba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e6d9d700 1 -- 172.21.15.3:0/4223295163 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e0045b30 con 0x7f61e0047a50 2022-07-21T01:39:06.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.353+0000 7f61e6d9d700 1 --2- 172.21.15.3:0/4223295163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 0x7f61e002a8d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f61d8002f70 tx=0x7f61d8006df0).ready entity=mon.0 client_cookie=1ea5b9aca9eb9419 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61e5d9b700 1 -- 172.21.15.3:0/4223295163 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61d800d4a0 con 0x7f61e0047a50 2022-07-21T01:39:06.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61e5d9b700 1 -- 172.21.15.3:0/4223295163 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d8000e90 con 0x7f61e0047a50 2022-07-21T01:39:06.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61e5d9b700 1 -- 172.21.15.3:0/4223295163 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61d800d7f0 con 0x7f61e0047a50 2022-07-21T01:39:06.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 -- 172.21.15.3:0/4223295163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 msgr2=0x7f61e002a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 --2- 172.21.15.3:0/4223295163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 0x7f61e002a8d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f61d8002f70 tx=0x7f61d8006df0).stop 2022-07-21T01:39:06.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 -- 172.21.15.3:0/4223295163 shutdown_connections 2022-07-21T01:39:06.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 --2- 172.21.15.3:0/4223295163 >> v2:172.21.15.143:3300/0 conn(0x7f61e0045110 0x7f61e0006530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 --2- 172.21.15.3:0/4223295163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 0x7f61e002a8d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 -- 172.21.15.3:0/4223295163 >> 172.21.15.3:0/4223295163 conn(0x7f61e004b1c0 msgr2=0x7f61e00471a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 -- 172.21.15.3:0/4223295163 shutdown_connections 2022-07-21T01:39:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 -- 172.21.15.3:0/4223295163 wait complete. 2022-07-21T01:39:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 Processor -- start 2022-07-21T01:39:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.357+0000 7f61ec8cc700 1 -- start start 2022-07-21T01:39:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61ec8cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e0005900 0x7f61e00304a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61ec8cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e0045110 0x7f61e002d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61ec8cc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 0x7f61e00d77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61ec8cc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00e7760 con 0x7f61e0047a50 2022-07-21T01:39:06.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61ec8cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0030130 con 0x7f61e0005900 2022-07-21T01:39:06.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61ec8cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e002fc30 con 0x7f61e0045110 2022-07-21T01:39:06.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 0x7f61e00d77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 0x7f61e00d77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46782/0 (socket says 172.21.15.3:46782) 2022-07-21T01:39:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e759e700 1 -- 172.21.15.3:0/732118155 learned_addr learned my addr 172.21.15.3:0/732118155 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e659c700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e0045110 0x7f61e002d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e6d9d700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e0005900 0x7f61e00304a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e759e700 1 -- 172.21.15.3:0/732118155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e0045110 msgr2=0x7f61e002d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e759e700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e0045110 0x7f61e002d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e759e700 1 -- 172.21.15.3:0/732118155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e0005900 msgr2=0x7f61e00304a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e759e700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e0005900 0x7f61e00304a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e759e700 1 -- 172.21.15.3:0/732118155 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d800b120 con 0x7f61e0047a50 2022-07-21T01:39:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61e759e700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 0x7f61e00d77e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f61e8062000 tx=0x7f61e805f750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61d7fff700 1 -- 172.21.15.3:0/732118155 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e8086070 con 0x7f61e0047a50 2022-07-21T01:39:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61d7fff700 1 -- 172.21.15.3:0/732118155 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e805dae0 con 0x7f61e0047a50 2022-07-21T01:39:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e00e7950 con 0x7f61e0047a50 2022-07-21T01:39:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61d7fff700 1 -- 172.21.15.3:0/732118155 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e8069a70 con 0x7f61e0047a50 2022-07-21T01:39:06.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e00e7bb0 con 0x7f61e0047a50 2022-07-21T01:39:06.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e0005170 con 0x7f61e0047a50 2022-07-21T01:39:06.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.361+0000 7f61d7fff700 1 -- 172.21.15.3:0/732118155 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f61e8069bd0 con 0x7f61e0047a50 2022-07-21T01:39:06.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.365+0000 7f61d7fff700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c4053980 0x7f61c4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.365+0000 7f61d7fff700 1 -- 172.21.15.3:0/732118155 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f61e80d9fd0 con 0x7f61e0047a50 2022-07-21T01:39:06.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.365+0000 7f61e6d9d700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c4053980 0x7f61c4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.365+0000 7f61e6d9d700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c4053980 0x7f61c4055e40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f61d801a000 tx=0x7f61d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.369+0000 7f61d7fff700 1 -- 172.21.15.3:0/732118155 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f61e80b10d0 con 0x7f61e0047a50 2022-07-21T01:39:06.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.493+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f61e002a7f0 con 0x7f61e0047a50 2022-07-21T01:39:06.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.497+0000 7f61d7fff700 1 -- 172.21.15.3:0/732118155 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v75) v1 ==== 110+0+10 (secure 0 0 0) 0x7f61e80b46f0 con 0x7f61e0047a50 2022-07-21T01:39:06.494 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:39:06.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.497+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c4053980 msgr2=0x7f61c4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.497+0000 7f61ec8cc700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c4053980 0x7f61c4055e40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f61d801a000 tx=0x7f61d8014000).stop 2022-07-21T01:39:06.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.497+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 msgr2=0x7f61e00d77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.497+0000 7f61ec8cc700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 0x7f61e00d77e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f61e8062000 tx=0x7f61e805f750).stop 2022-07-21T01:39:06.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.501+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 shutdown_connections 2022-07-21T01:39:06.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.501+0000 7f61ec8cc700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61e0047a50 0x7f61e00d77e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.501+0000 7f61ec8cc700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61e0045110 0x7f61e002d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.501+0000 7f61ec8cc700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f61c4053980 0x7f61c4055e40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.501+0000 7f61ec8cc700 1 --2- 172.21.15.3:0/732118155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e0005900 0x7f61e00304a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.501+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 >> 172.21.15.3:0/732118155 conn(0x7f61e004b1c0 msgr2=0x7f61e00d5800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:06.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.501+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 shutdown_connections 2022-07-21T01:39:06.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.501+0000 7f61ec8cc700 1 -- 172.21.15.3:0/732118155 wait complete. 2022-07-21T01:39:06.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:39:06.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0699089700 1 Processor -- start 2022-07-21T01:39:06.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0699089700 1 -- start start 2022-07-21T01:39:06.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0699089700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 0x7f0694064bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0699089700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06940b38a0 0x7f06941470d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0699089700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06941436f0 0x7f0694143b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0699089700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06940b4f60 con 0x7f06940b38a0 2022-07-21T01:39:06.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0699089700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06941475c0 con 0x7f06941436f0 2022-07-21T01:39:06.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0699089700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06940b4a60 con 0x7f06940647d0 2022-07-21T01:39:06.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f06937fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06940b38a0 0x7f06941470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f06937fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06940b38a0 0x7f06941470d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46788/0 (socket says 172.21.15.3:46788) 2022-07-21T01:39:06.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f06937fe700 1 -- 172.21.15.3:0/1667676028 learned_addr learned my addr 172.21.15.3:0/1667676028 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:06.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0693fff700 1 --2- 172.21.15.3:0/1667676028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 0x7f0694064bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0698888700 1 --2- 172.21.15.3:0/1667676028 >> v2:172.21.15.143:3300/0 conn(0x7f06941436f0 0x7f0694143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f06937fe700 1 -- 172.21.15.3:0/1667676028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 msgr2=0x7f0694064bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:06.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f06937fe700 1 --2- 172.21.15.3:0/1667676028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 0x7f0694064bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f06937fe700 1 -- 172.21.15.3:0/1667676028 >> v2:172.21.15.143:3300/0 conn(0x7f06941436f0 msgr2=0x7f0694143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:06.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f06937fe700 1 --2- 172.21.15.3:0/1667676028 >> v2:172.21.15.143:3300/0 conn(0x7f06941436f0 0x7f0694143b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f06937fe700 1 -- 172.21.15.3:0/1667676028 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06940b4cc0 con 0x7f06940b38a0 2022-07-21T01:39:06.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f0698888700 1 --2- 172.21.15.3:0/1667676028 >> v2:172.21.15.143:3300/0 conn(0x7f06941436f0 0x7f0694143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:06.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.625+0000 7f06937fe700 1 --2- 172.21.15.3:0/1667676028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06940b38a0 0x7f06941470d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f067c002f70 tx=0x7f067c006eb0).ready entity=mon.0 client_cookie=1060cae1c3cc9dc8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0692ffd700 1 -- 172.21.15.3:0/1667676028 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f067c00d570 con 0x7f06940b38a0 2022-07-21T01:39:06.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0692ffd700 1 -- 172.21.15.3:0/1667676028 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c000e90 con 0x7f06940b38a0 2022-07-21T01:39:06.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0692ffd700 1 -- 172.21.15.3:0/1667676028 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f067c00d900 con 0x7f06940b38a0 2022-07-21T01:39:06.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 -- 172.21.15.3:0/1667676028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06940b38a0 msgr2=0x7f06941470d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 --2- 172.21.15.3:0/1667676028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06940b38a0 0x7f06941470d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f067c002f70 tx=0x7f067c006eb0).stop 2022-07-21T01:39:06.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 -- 172.21.15.3:0/1667676028 shutdown_connections 2022-07-21T01:39:06.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 --2- 172.21.15.3:0/1667676028 >> v2:172.21.15.143:3300/0 conn(0x7f06941436f0 0x7f0694143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 --2- 172.21.15.3:0/1667676028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06940b38a0 0x7f06941470d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 --2- 172.21.15.3:0/1667676028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 0x7f0694064bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 -- 172.21.15.3:0/1667676028 >> 172.21.15.3:0/1667676028 conn(0x7f069405f840 msgr2=0x7f069405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:06.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 -- 172.21.15.3:0/1667676028 shutdown_connections 2022-07-21T01:39:06.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 -- 172.21.15.3:0/1667676028 wait complete. 2022-07-21T01:39:06.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 Processor -- start 2022-07-21T01:39:06.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 -- start start 2022-07-21T01:39:06.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 0x7f0694145180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06940b38a0 0x7f06941469d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06941436f0 0x7f0694155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0694147370 con 0x7f06941436f0 2022-07-21T01:39:06.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06940af750 con 0x7f06940b38a0 2022-07-21T01:39:06.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0699089700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06940ae9e0 con 0x7f06940647d0 2022-07-21T01:39:06.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0698888700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06941436f0 0x7f0694155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0698888700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06941436f0 0x7f0694155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46796/0 (socket says 172.21.15.3:46796) 2022-07-21T01:39:06.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.629+0000 7f0698888700 1 -- 172.21.15.3:0/4171836197 learned_addr learned my addr 172.21.15.3:0/4171836197 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:06.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f06937fe700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06940b38a0 0x7f06941469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0693fff700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 0x7f0694145180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0698888700 1 -- 172.21.15.3:0/4171836197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 msgr2=0x7f0694145180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0698888700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 0x7f0694145180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0698888700 1 -- 172.21.15.3:0/4171836197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06940b38a0 msgr2=0x7f06941469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0698888700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06940b38a0 0x7f06941469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0698888700 1 -- 172.21.15.3:0/4171836197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c00b1e0 con 0x7f06941436f0 2022-07-21T01:39:06.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0698888700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06941436f0 0x7f0694155aa0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f068c006cc0 tx=0x7f068c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f06917fa700 1 -- 172.21.15.3:0/4171836197 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f068c013620 con 0x7f06941436f0 2022-07-21T01:39:06.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f06917fa700 1 -- 172.21.15.3:0/4171836197 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068c013780 con 0x7f06941436f0 2022-07-21T01:39:06.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06940aec80 con 0x7f06941436f0 2022-07-21T01:39:06.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f06917fa700 1 -- 172.21.15.3:0/4171836197 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f068c013ae0 con 0x7f06941436f0 2022-07-21T01:39:06.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0694144490 con 0x7f06941436f0 2022-07-21T01:39:06.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0694143250 con 0x7f06941436f0 2022-07-21T01:39:06.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f06917fa700 1 -- 172.21.15.3:0/4171836197 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f068c013c40 con 0x7f06941436f0 2022-07-21T01:39:06.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f06917fa700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06780539e0 0x7f0678055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f06917fa700 1 -- 172.21.15.3:0/4171836197 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f068c06c190 con 0x7f06941436f0 2022-07-21T01:39:06.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.633+0000 7f0693fff700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06780539e0 0x7f0678055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.637+0000 7f0693fff700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06780539e0 0x7f0678055ea0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f068400b000 tx=0x7f0684006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.645+0000 7f06917fa700 1 -- 172.21.15.3:0/4171836197 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f068c043230 con 0x7f06941436f0 2022-07-21T01:39:06.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.769+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f06940574e0 con 0x7f06941436f0 2022-07-21T01:39:06.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f06917fa700 1 -- 172.21.15.3:0/4171836197 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v75) v1 ==== 92+0+10 (secure 0 0 0) 0x7f068c046850 con 0x7f06941436f0 2022-07-21T01:39:06.770 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:39:06.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06780539e0 msgr2=0x7f0678055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06780539e0 0x7f0678055ea0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f068400b000 tx=0x7f0684006df0).stop 2022-07-21T01:39:06.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06941436f0 msgr2=0x7f0694155aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06941436f0 0x7f0694155aa0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f068c006cc0 tx=0x7f068c008040).stop 2022-07-21T01:39:06.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 shutdown_connections 2022-07-21T01:39:06.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06941436f0 0x7f0694155aa0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06780539e0 0x7f0678055ea0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06940b38a0 0x7f06941469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 --2- 172.21.15.3:0/4171836197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06940647d0 0x7f0694145180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 >> 172.21.15.3:0/4171836197 conn(0x7f069405f840 msgr2=0x7f0694148d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:06.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 shutdown_connections 2022-07-21T01:39:06.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.773+0000 7f0699089700 1 -- 172.21.15.3:0/4171836197 wait complete. 2022-07-21T01:39:06.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:39:06.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728ef0b700 1 Processor -- start 2022-07-21T01:39:06.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728ef0b700 1 -- start start 2022-07-21T01:39:06.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728ef0b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72880b0d10 0x7f7288146df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728ef0b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880b0560 0x7f7288140350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728ef0b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72880b12d0 con 0x7f72880a9320 2022-07-21T01:39:06.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728ef0b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72880a9700 con 0x7f72880b0d10 2022-07-21T01:39:06.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728ef0b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72880b1500 con 0x7f72880b0560 2022-07-21T01:39:06.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728d708700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72880b0d10 0x7f7288146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728d708700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72880b0d10 0x7f7288146df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46804/0 (socket says 172.21.15.3:46804) 2022-07-21T01:39:06.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728d708700 1 -- 172.21.15.3:0/448879308 learned_addr learned my addr 172.21.15.3:0/448879308 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:06.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728e70a700 1 --2- 172.21.15.3:0/448879308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880b0560 0x7f7288140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728d708700 1 -- 172.21.15.3:0/448879308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880b0560 msgr2=0x7f7288140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:06.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728d708700 1 --2- 172.21.15.3:0/448879308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880b0560 0x7f7288140350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728d708700 1 -- 172.21.15.3:0/448879308 >> v1:172.21.15.143:6789/0 conn(0x7f72880a9320 legacy=0x7f72881466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728cf07700 1 -- 172.21.15.3:0/448879308 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518387285 0 0) 0x7f72880b12d0 con 0x7f72880a9320 2022-07-21T01:39:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728d708700 1 -- 172.21.15.3:0/448879308 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72880b17a0 con 0x7f72880b0d10 2022-07-21T01:39:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728e70a700 1 --2- 172.21.15.3:0/448879308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880b0560 0x7f7288140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.885+0000 7f728d708700 1 --2- 172.21.15.3:0/448879308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0d10 0x7f7288146df0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f727c002f70 tx=0x7f727c006eb0).ready entity=mon.0 client_cookie=103d95283520e7df server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728cf07700 1 -- 172.21.15.3:0/448879308 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f727c00d5c0 con 0x7f72880b0d10 2022-07-21T01:39:06.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728cf07700 1 -- 172.21.15.3:0/448879308 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c000e90 con 0x7f72880b0d10 2022-07-21T01:39:06.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728cf07700 1 -- 172.21.15.3:0/448879308 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f727c00d910 con 0x7f72880b0d10 2022-07-21T01:39:06.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728ef0b700 1 -- 172.21.15.3:0/448879308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0d10 msgr2=0x7f7288146df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728ef0b700 1 --2- 172.21.15.3:0/448879308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0d10 0x7f7288146df0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f727c002f70 tx=0x7f727c006eb0).stop 2022-07-21T01:39:06.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728ef0b700 1 -- 172.21.15.3:0/448879308 shutdown_connections 2022-07-21T01:39:06.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728ef0b700 1 --2- 172.21.15.3:0/448879308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880b0560 0x7f7288140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728ef0b700 1 --2- 172.21.15.3:0/448879308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0d10 0x7f7288146df0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728ef0b700 1 -- 172.21.15.3:0/448879308 >> 172.21.15.3:0/448879308 conn(0x7f72880612d0 msgr2=0x7f7288062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:06.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728ef0b700 1 -- 172.21.15.3:0/448879308 shutdown_connections 2022-07-21T01:39:06.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.889+0000 7f728ef0b700 1 -- 172.21.15.3:0/448879308 wait complete. 2022-07-21T01:39:06.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728ef0b700 1 Processor -- start 2022-07-21T01:39:06.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728ef0b700 1 -- start start 2022-07-21T01:39:06.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728ef0b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880a9320 0x7f72880b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728ef0b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0560 0x7f72880b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728ef0b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72880b0d10 0x7f7288142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728ef0b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7288152ea0 con 0x7f72880b0560 2022-07-21T01:39:06.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728ef0b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7288153110 con 0x7f72880b0d10 2022-07-21T01:39:06.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728ef0b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72881533b0 con 0x7f72880a9320 2022-07-21T01:39:06.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728d708700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0560 0x7f72880b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728d708700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0560 0x7f72880b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46808/0 (socket says 172.21.15.3:46808) 2022-07-21T01:39:06.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728d708700 1 -- 172.21.15.3:0/2990195402 learned_addr learned my addr 172.21.15.3:0/2990195402 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:06.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728df09700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880a9320 0x7f72880b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728e70a700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72880b0d10 0x7f7288142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728d708700 1 -- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880a9320 msgr2=0x7f72880b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728d708700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880a9320 0x7f72880b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728d708700 1 -- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72880b0d10 msgr2=0x7f7288142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:06.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728d708700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72880b0d10 0x7f7288142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:06.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728d708700 1 -- 172.21.15.3:0/2990195402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c00b1e0 con 0x7f72880b0560 2022-07-21T01:39:06.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728e70a700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72880b0d10 0x7f7288142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:06.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728df09700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880a9320 0x7f72880b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:06.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.893+0000 7f728d708700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0560 0x7f72880b3f20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f727c002f70 tx=0x7f727c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f727affd700 1 -- 172.21.15.3:0/2990195402 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f727c006be0 con 0x7f72880b0560 2022-07-21T01:39:06.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f727affd700 1 -- 172.21.15.3:0/2990195402 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c00de40 con 0x7f72880b0560 2022-07-21T01:39:06.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72880aa090 con 0x7f72880b0560 2022-07-21T01:39:06.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f727affd700 1 -- 172.21.15.3:0/2990195402 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f727c011d70 con 0x7f72880b0560 2022-07-21T01:39:06.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72880a9e00 con 0x7f72880b0560 2022-07-21T01:39:06.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72880aa220 con 0x7f72880b0560 2022-07-21T01:39:06.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f727affd700 1 -- 172.21.15.3:0/2990195402 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f727c025030 con 0x7f72880b0560 2022-07-21T01:39:06.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f727affd700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7268053b30 0x7f7268055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:06.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f727affd700 1 -- 172.21.15.3:0/2990195402 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f727c06ccb0 con 0x7f72880b0560 2022-07-21T01:39:06.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.897+0000 7f728df09700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7268053b30 0x7f7268055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:06.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.901+0000 7f728df09700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7268053b30 0x7f7268055ff0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f728400d000 tx=0x7f7284007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:06.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:06.905+0000 7f727affd700 1 -- 172.21.15.3:0/2990195402 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f727c043db0 con 0x7f72880b0560 2022-07-21T01:39:07.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.033+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f72880a9e00 con 0x7f72880b0560 2022-07-21T01:39:07.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.037+0000 7f727affd700 1 -- 172.21.15.3:0/2990195402 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) v1 ==== 84+0+143 (secure 0 0 0) 0x7f72880a9e00 con 0x7f72880b0560 2022-07-21T01:39:07.035 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:07.036 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:39:07.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:07.036 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:39:07.036 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:39:07.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:39:07.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:39:07.037 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:39:07.037 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:39:07.038 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:39:07.039 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:39:07.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.037+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7268053b30 msgr2=0x7f7268055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:07.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.037+0000 7f728ef0b700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7268053b30 0x7f7268055ff0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f728400d000 tx=0x7f7284007f10).stop 2022-07-21T01:39:07.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.037+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0560 msgr2=0x7f72880b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:07.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.037+0000 7f728ef0b700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0560 0x7f72880b3f20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f727c002f70 tx=0x7f727c015000).stop 2022-07-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.041+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 shutdown_connections 2022-07-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.041+0000 7f728ef0b700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7268053b30 0x7f7268055ff0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.041+0000 7f728ef0b700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72880b0d10 0x7f7288142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.041+0000 7f728ef0b700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72880b0560 0x7f72880b3f20 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.041+0000 7f728ef0b700 1 --2- 172.21.15.3:0/2990195402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72880a9320 0x7f72880b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.041+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 >> 172.21.15.3:0/2990195402 conn(0x7f72880612d0 msgr2=0x7f728813b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.041+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 shutdown_connections 2022-07-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.041+0000 7f728ef0b700 1 -- 172.21.15.3:0/2990195402 wait complete. 2022-07-21T01:39:07.052 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:39:07.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:39:07.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01e540700 1 Processor -- start 2022-07-21T01:39:07.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01e540700 1 -- start start 2022-07-21T01:39:07.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01e540700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 0x7ff0180af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:07.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01e540700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff0180b07a0 0x7ff018147730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:07.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01e540700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0180b5a10 con 0x7ff018067730 2022-07-21T01:39:07.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01e540700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff018145470 con 0x7ff0180b07a0 2022-07-21T01:39:07.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01e540700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0180ae180 con 0x7ff0180b5630 2022-07-21T01:39:07.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01cd3d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff0180b07a0 0x7ff018147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:07.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01dd3f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff018067730 0x7ff0180acdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55096/0 (socket says 172.21.15.3:55096) 2022-07-21T01:39:07.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01dd3f700 1 -- 172.21.15.3:0/102145467 learned_addr learned my addr 172.21.15.3:0/102145467 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:07.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01d53e700 1 --2- 172.21.15.3:0/102145467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 0x7ff0180af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:07.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff017fff700 1 -- 172.21.15.3:0/102145467 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108324985 0 0) 0x7ff0180b5a10 con 0x7ff018067730 2022-07-21T01:39:07.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff017fff700 1 -- 172.21.15.3:0/102145467 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff004003260 con 0x7ff018067730 2022-07-21T01:39:07.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01cd3d700 1 -- 172.21.15.3:0/102145467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 msgr2=0x7ff0180af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:07.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01cd3d700 1 --2- 172.21.15.3:0/102145467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 0x7ff0180af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:07.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01cd3d700 1 -- 172.21.15.3:0/102145467 >> v1:172.21.15.3:6789/0 conn(0x7ff018067730 legacy=0x7ff0180acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:07.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01cd3d700 1 -- 172.21.15.3:0/102145467 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0180ac990 con 0x7ff0180b07a0 2022-07-21T01:39:07.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff01cd3d700 1 --2- 172.21.15.3:0/102145467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 0x7ff018147730 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7ff00c002f70 tx=0x7ff00c006df0).ready entity=mon.0 client_cookie=31b8a127d9d6c3dd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:07.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.165+0000 7ff017fff700 1 -- 172.21.15.3:0/102145467 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff00c00d4a0 con 0x7ff0180b07a0 2022-07-21T01:39:07.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff017fff700 1 -- 172.21.15.3:0/102145467 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c000e90 con 0x7ff0180b07a0 2022-07-21T01:39:07.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff017fff700 1 -- 172.21.15.3:0/102145467 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff00c00d7f0 con 0x7ff0180b07a0 2022-07-21T01:39:07.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- 172.21.15.3:0/102145467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 msgr2=0x7ff018147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 --2- 172.21.15.3:0/102145467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 0x7ff018147730 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7ff00c002f70 tx=0x7ff00c006df0).stop 2022-07-21T01:39:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- 172.21.15.3:0/102145467 shutdown_connections 2022-07-21T01:39:07.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 --2- 172.21.15.3:0/102145467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 0x7ff018147730 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:07.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 --2- 172.21.15.3:0/102145467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 0x7ff0180af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:07.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- 172.21.15.3:0/102145467 >> 172.21.15.3:0/102145467 conn(0x7ff0180626f0 msgr2=0x7ff018062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:07.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- 172.21.15.3:0/102145467 shutdown_connections 2022-07-21T01:39:07.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- 172.21.15.3:0/102145467 wait complete. 2022-07-21T01:39:07.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 Processor -- start 2022-07-21T01:39:07.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- start start 2022-07-21T01:39:07.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018067730 0x7ff018066400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:07.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 0x7ff018144b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:07.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 0x7ff018156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:07.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff018065d00 con 0x7ff0180b07a0 2022-07-21T01:39:07.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff018146a70 con 0x7ff018067730 2022-07-21T01:39:07.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0181467d0 con 0x7ff0180b5630 2022-07-21T01:39:07.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01cd3d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 0x7ff018144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:07.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01cd3d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 0x7ff018144b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46822/0 (socket says 172.21.15.3:46822) 2022-07-21T01:39:07.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01cd3d700 1 -- 172.21.15.3:0/4277973172 learned_addr learned my addr 172.21.15.3:0/4277973172 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:07.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01d53e700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018067730 0x7ff018066400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:07.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01dd3f700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 0x7ff018156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:07.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01cd3d700 1 -- 172.21.15.3:0/4277973172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 msgr2=0x7ff018156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:07.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01cd3d700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 0x7ff018156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:07.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01cd3d700 1 -- 172.21.15.3:0/4277973172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018067730 msgr2=0x7ff018066400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:07.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01cd3d700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018067730 0x7ff018066400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:07.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01cd3d700 1 -- 172.21.15.3:0/4277973172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c00b180 con 0x7ff0180b07a0 2022-07-21T01:39:07.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01d53e700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018067730 0x7ff018066400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:07.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01cd3d700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 0x7ff018144b50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7ff00c002f70 tx=0x7ff00c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:07.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff0167fc700 1 -- 172.21.15.3:0/4277973172 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff00c016070 con 0x7ff0180b07a0 2022-07-21T01:39:07.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff0167fc700 1 -- 172.21.15.3:0/4277973172 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c013a40 con 0x7ff0180b07a0 2022-07-21T01:39:07.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff018065f60 con 0x7ff0180b07a0 2022-07-21T01:39:07.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff0167fc700 1 -- 172.21.15.3:0/4277973172 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff00c013da0 con 0x7ff0180b07a0 2022-07-21T01:39:07.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.169+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff018147760 con 0x7ff0180b07a0 2022-07-21T01:39:07.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.173+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0180647d0 con 0x7ff0180b07a0 2022-07-21T01:39:07.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.173+0000 7ff0167fc700 1 -- 172.21.15.3:0/4277973172 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff00c00f030 con 0x7ff0180b07a0 2022-07-21T01:39:07.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.173+0000 7ff0167fc700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feff8053a70 0x7feff8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:07.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.173+0000 7ff0167fc700 1 -- 172.21.15.3:0/4277973172 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5600+0+0 (secure 0 0 0) 0x7ff00c06c930 con 0x7ff0180b07a0 2022-07-21T01:39:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.173+0000 7ff01d53e700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feff8053a70 0x7feff8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.173+0000 7ff01d53e700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feff8053a70 0x7feff8055f30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7ff01000b000 tx=0x7ff010008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:07.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.185+0000 7ff0167fc700 1 -- 172.21.15.3:0/4277973172 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff0180647d0 con 0x7ff0180b07a0 2022-07-21T01:39:07.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:07.313+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7ff0180647d0 con 0x7ff0180b07a0 2022-07-21T01:39:08.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff0167fc700 1 -- 172.21.15.3:0/4277973172 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v76) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff0180647d0 con 0x7ff0180b07a0 2022-07-21T01:39:08.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feff8053a70 msgr2=0x7feff8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:08.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feff8053a70 0x7feff8055f30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7ff01000b000 tx=0x7ff010008040).stop 2022-07-21T01:39:08.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 msgr2=0x7ff018144b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:08.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 0x7ff018144b50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7ff00c002f70 tx=0x7ff00c014000).stop 2022-07-21T01:39:08.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 shutdown_connections 2022-07-21T01:39:08.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0180b5630 0x7ff018156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0180b07a0 0x7ff018144b50 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feff8053a70 0x7feff8055f30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 --2- 172.21.15.3:0/4277973172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018067730 0x7ff018066400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 >> 172.21.15.3:0/4277973172 conn(0x7ff0180626f0 msgr2=0x7ff018147d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:08.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 shutdown_connections 2022-07-21T01:39:08.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.293+0000 7ff01e540700 1 -- 172.21.15.3:0/4277973172 wait complete. 2022-07-21T01:39:08.297 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:39:08.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:39:08.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f24edd700 1 Processor -- start 2022-07-21T01:39:08.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f24edd700 1 -- start start 2022-07-21T01:39:08.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f24edd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6f200614b0 0x7f6f20061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:08.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f24edd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f200ad750 0x7f6f200b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:08.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f24edd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6f201405d0 0x7f6f20140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:08.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f24edd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f200aee80 con 0x7f6f200614b0 2022-07-21T01:39:08.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f24edd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f200b70b0 con 0x7f6f201405d0 2022-07-21T01:39:08.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f24edd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f200af0b0 con 0x7f6f200ad750 2022-07-21T01:39:08.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1f7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6f200614b0 0x7f6f20061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:08.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6f201405d0 0x7f6f20140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:08.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f200ad750 0x7f6f200b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:08.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1f7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6f200614b0 0x7f6f20061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46828/0 (socket says 172.21.15.3:46828) 2022-07-21T01:39:08.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1f7fe700 1 -- 172.21.15.3:0/665897724 learned_addr learned my addr 172.21.15.3:0/665897724 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:08.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1f7fe700 1 -- 172.21.15.3:0/665897724 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f200ad750 msgr2=0x7f6f200b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:08.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1f7fe700 1 --2- 172.21.15.3:0/665897724 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f200ad750 0x7f6f200b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1f7fe700 1 -- 172.21.15.3:0/665897724 >> v2:172.21.15.143:3300/0 conn(0x7f6f201405d0 msgr2=0x7f6f20140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:08.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1f7fe700 1 --2- 172.21.15.3:0/665897724 >> v2:172.21.15.143:3300/0 conn(0x7f6f201405d0 0x7f6f20140a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1f7fe700 1 -- 172.21.15.3:0/665897724 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f200af350 con 0x7f6f200614b0 2022-07-21T01:39:08.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1f7fe700 1 --2- 172.21.15.3:0/665897724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200614b0 0x7f6f20061890 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f6f10002f70 tx=0x7f6f10006eb0).ready entity=mon.0 client_cookie=61d217c2c8e69c4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:08.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1e7fc700 1 -- 172.21.15.3:0/665897724 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f1000d570 con 0x7f6f200614b0 2022-07-21T01:39:08.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1e7fc700 1 -- 172.21.15.3:0/665897724 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f10000e90 con 0x7f6f200614b0 2022-07-21T01:39:08.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.421+0000 7f6f1e7fc700 1 -- 172.21.15.3:0/665897724 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f1000d900 con 0x7f6f200614b0 2022-07-21T01:39:08.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 -- 172.21.15.3:0/665897724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200614b0 msgr2=0x7f6f20061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:08.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 --2- 172.21.15.3:0/665897724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200614b0 0x7f6f20061890 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f6f10002f70 tx=0x7f6f10006eb0).stop 2022-07-21T01:39:08.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 -- 172.21.15.3:0/665897724 shutdown_connections 2022-07-21T01:39:08.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 --2- 172.21.15.3:0/665897724 >> v2:172.21.15.143:3300/0 conn(0x7f6f201405d0 0x7f6f20140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 --2- 172.21.15.3:0/665897724 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f200ad750 0x7f6f200b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 --2- 172.21.15.3:0/665897724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200614b0 0x7f6f20061890 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 -- 172.21.15.3:0/665897724 >> 172.21.15.3:0/665897724 conn(0x7f6f20066f70 msgr2=0x7f6f20067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:08.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 -- 172.21.15.3:0/665897724 shutdown_connections 2022-07-21T01:39:08.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 -- 172.21.15.3:0/665897724 wait complete. 2022-07-21T01:39:08.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 Processor -- start 2022-07-21T01:39:08.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.425+0000 7f6f24edd700 1 -- start start 2022-07-21T01:39:08.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f24edd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f200614b0 0x7f6f20145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:08.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f24edd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200ad750 0x7f6f200afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:08.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f24edd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f201405d0 0x7f6f200affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:08.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f24edd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f20153550 con 0x7f6f200ad750 2022-07-21T01:39:08.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f24edd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2009ed50 con 0x7f6f200614b0 2022-07-21T01:39:08.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f24edd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2009eff0 con 0x7f6f201405d0 2022-07-21T01:39:08.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200ad750 0x7f6f200afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:08.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200ad750 0x7f6f200afac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46836/0 (socket says 172.21.15.3:46836) 2022-07-21T01:39:08.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1effd700 1 -- 172.21.15.3:0/1055512723 learned_addr learned my addr 172.21.15.3:0/1055512723 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:08.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1f7fe700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f200614b0 0x7f6f20145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:08.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1ffff700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f201405d0 0x7f6f200affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:08.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1effd700 1 -- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f201405d0 msgr2=0x7f6f200affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:08.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1effd700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f201405d0 0x7f6f200affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1effd700 1 -- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f200614b0 msgr2=0x7f6f20145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1effd700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f200614b0 0x7f6f20145610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1effd700 1 -- 172.21.15.3:0/1055512723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1000b1e0 con 0x7f6f200ad750 2022-07-21T01:39:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1f7fe700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f200614b0 0x7f6f20145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1ffff700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f201405d0 0x7f6f200affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1effd700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200ad750 0x7f6f200afac0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f6f0800c000 tx=0x7f6f08006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1cff9700 1 -- 172.21.15.3:0/1055512723 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f08010780 con 0x7f6f200ad750 2022-07-21T01:39:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1cff9700 1 -- 172.21.15.3:0/1055512723 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f080108e0 con 0x7f6f200ad750 2022-07-21T01:39:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2009f290 con 0x7f6f200ad750 2022-07-21T01:39:08.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f1cff9700 1 -- 172.21.15.3:0/1055512723 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f08010c40 con 0x7f6f200ad750 2022-07-21T01:39:08.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.429+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f20145b00 con 0x7f6f200ad750 2022-07-21T01:39:08.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.433+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2009f420 con 0x7f6f200ad750 2022-07-21T01:39:08.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.433+0000 7f6f1cff9700 1 -- 172.21.15.3:0/1055512723 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6f0801c460 con 0x7f6f200ad750 2022-07-21T01:39:08.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.433+0000 7f6f1cff9700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6f04053950 0x7f6f04055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:08.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.433+0000 7f6f1cff9700 1 -- 172.21.15.3:0/1055512723 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f6f0806c2c0 con 0x7f6f200ad750 2022-07-21T01:39:08.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.433+0000 7f6f1f7fe700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6f04053950 0x7f6f04055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:08.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.433+0000 7f6f1f7fe700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6f04053950 0x7f6f04055e10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f6f1001c000 tx=0x7f6f10015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:08.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.441+0000 7f6f1cff9700 1 -- 172.21.15.3:0/1055512723 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6f08043060 con 0x7f6f200ad750 2022-07-21T01:39:08.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:08.561+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f6f200af5f0 con 0x7f6f200ad750 2022-07-21T01:39:09.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.305+0000 7f6f1cff9700 1 -- 172.21.15.3:0/1055512723 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v77) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6f08046680 con 0x7f6f200ad750 2022-07-21T01:39:09.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6f04053950 msgr2=0x7f6f04055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6f04053950 0x7f6f04055e10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f6f1001c000 tx=0x7f6f10015000).stop 2022-07-21T01:39:09.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200ad750 msgr2=0x7f6f200afac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200ad750 0x7f6f200afac0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f6f0800c000 tx=0x7f6f08006df0).stop 2022-07-21T01:39:09.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 shutdown_connections 2022-07-21T01:39:09.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f201405d0 0x7f6f200affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f200ad750 0x7f6f200afac0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6f04053950 0x7f6f04055e10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 --2- 172.21.15.3:0/1055512723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f200614b0 0x7f6f20145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 >> 172.21.15.3:0/1055512723 conn(0x7f6f20066f70 msgr2=0x7f6f201469a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:09.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 shutdown_connections 2022-07-21T01:39:09.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.309+0000 7f6f24edd700 1 -- 172.21.15.3:0/1055512723 wait complete. 2022-07-21T01:39:09.313 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:39:09.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:39:09.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.437+0000 7f3c508fb700 1 Processor -- start 2022-07-21T01:39:09.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.437+0000 7f3c508fb700 1 -- start start 2022-07-21T01:39:09.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3c4c066dc0 0x7f3c4c14eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4c0654c0 con 0x7f3c4c0b2730 2022-07-21T01:39:09.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4c0b3570 con 0x7f3c4c0b3190 2022-07-21T01:39:09.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4c062190 con 0x7f3c4c066dc0 2022-07-21T01:39:09.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3c4c066dc0 0x7f3c4c14eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4affd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3c4c0b2730 0x7f3c4c0666b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55124/0 (socket says 172.21.15.3:55124) 2022-07-21T01:39:09.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4affd700 1 -- 172.21.15.3:0/2575848463 learned_addr learned my addr 172.21.15.3:0/2575848463 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:09.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4a7fc700 1 -- 172.21.15.3:0/2575848463 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51412886 0 0) 0x7f3c4c0654c0 con 0x7f3c4c0b2730 2022-07-21T01:39:09.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4bfff700 1 -- 172.21.15.3:0/2575848463 >> v1:172.21.15.143:6789/0 conn(0x7f3c4c0b3190 legacy=0x7f3c4c060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4bfff700 1 -- 172.21.15.3:0/2575848463 >> v1:172.21.15.3:6789/0 conn(0x7f3c4c0b2730 legacy=0x7f3c4c0666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4bfff700 1 -- 172.21.15.3:0/2575848463 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c05fbf0 con 0x7f3c4c066dc0 2022-07-21T01:39:09.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4bfff700 1 --2- 172.21.15.3:0/2575848463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c066dc0 0x7f3c4c14eac0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f3c440032f0 tx=0x7f3c440091d0).ready entity=mon.0 client_cookie=e46b55caf3f16a1e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:09.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4a7fc700 1 -- 172.21.15.3:0/2575848463 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c44009ea0 con 0x7f3c4c066dc0 2022-07-21T01:39:09.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4a7fc700 1 -- 172.21.15.3:0/2575848463 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4400b8a0 con 0x7f3c4c066dc0 2022-07-21T01:39:09.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c4a7fc700 1 -- 172.21.15.3:0/2575848463 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c4400ba10 con 0x7f3c4c066dc0 2022-07-21T01:39:09.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 -- 172.21.15.3:0/2575848463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c066dc0 msgr2=0x7f3c4c14eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 --2- 172.21.15.3:0/2575848463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c066dc0 0x7f3c4c14eac0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f3c440032f0 tx=0x7f3c440091d0).stop 2022-07-21T01:39:09.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 -- 172.21.15.3:0/2575848463 shutdown_connections 2022-07-21T01:39:09.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 --2- 172.21.15.3:0/2575848463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c066dc0 0x7f3c4c14eac0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 -- 172.21.15.3:0/2575848463 >> 172.21.15.3:0/2575848463 conn(0x7f3c4c063870 msgr2=0x7f3c4c063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:09.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 -- 172.21.15.3:0/2575848463 shutdown_connections 2022-07-21T01:39:09.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.441+0000 7f3c508fb700 1 -- 172.21.15.3:0/2575848463 wait complete. 2022-07-21T01:39:09.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c508fb700 1 Processor -- start 2022-07-21T01:39:09.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c508fb700 1 -- start start 2022-07-21T01:39:09.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c508fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c4c066dc0 0x7f3c4c1402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c508fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c0b2730 0x7f3c4c13e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c508fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c0b3190 0x7f3c4c13ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c508fb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c0a4c40 con 0x7f3c4c0b2730 2022-07-21T01:39:09.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c508fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c152320 con 0x7f3c4c0b3190 2022-07-21T01:39:09.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c508fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c13f220 con 0x7f3c4c066dc0 2022-07-21T01:39:09.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c0b2730 0x7f3c4c13e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c0b2730 0x7f3c4c13e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46850/0 (socket says 172.21.15.3:46850) 2022-07-21T01:39:09.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4affd700 1 -- 172.21.15.3:0/1360180988 learned_addr learned my addr 172.21.15.3:0/1360180988 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:09.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4bfff700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c0b3190 0x7f3c4c13ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4b7fe700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c4c066dc0 0x7f3c4c1402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4affd700 1 -- 172.21.15.3:0/1360180988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c4c066dc0 msgr2=0x7f3c4c1402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4affd700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c4c066dc0 0x7f3c4c1402c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4affd700 1 -- 172.21.15.3:0/1360180988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c0b3190 msgr2=0x7f3c4c13ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4affd700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c0b3190 0x7f3c4c13ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4affd700 1 -- 172.21.15.3:0/1360180988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c44009470 con 0x7f3c4c0b2730 2022-07-21T01:39:09.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c4affd700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c0b2730 0x7f3c4c13e840 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f3c40000bc0 tx=0x7f3c40007a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:09.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c48ff9700 1 -- 172.21.15.3:0/1360180988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c4000ce20 con 0x7f3c4c0b2730 2022-07-21T01:39:09.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c48ff9700 1 -- 172.21.15.3:0/1360180988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c400072e0 con 0x7f3c4c0b2730 2022-07-21T01:39:09.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4c145fd0 con 0x7f3c4c0b2730 2022-07-21T01:39:09.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.445+0000 7f3c48ff9700 1 -- 172.21.15.3:0/1360180988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c40005800 con 0x7f3c4c0b2730 2022-07-21T01:39:09.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.449+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4c0a2e50 con 0x7f3c4c0b2730 2022-07-21T01:39:09.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.449+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4c05fe90 con 0x7f3c4c0b2730 2022-07-21T01:39:09.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.449+0000 7f3c48ff9700 1 -- 172.21.15.3:0/1360180988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3c4001a060 con 0x7f3c4c0b2730 2022-07-21T01:39:09.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.449+0000 7f3c48ff9700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c2c0539e0 0x7f3c2c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.449+0000 7f3c48ff9700 1 -- 172.21.15.3:0/1360180988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f3c4006b850 con 0x7f3c4c0b2730 2022-07-21T01:39:09.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.449+0000 7f3c4b7fe700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c2c0539e0 0x7f3c2c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.453+0000 7f3c4b7fe700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c2c0539e0 0x7f3c2c055ea0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f3c3c00b000 tx=0x7f3c3c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:09.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.457+0000 7f3c48ff9700 1 -- 172.21.15.3:0/1360180988 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3c40042690 con 0x7f3c4c0b2730 2022-07-21T01:39:09.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.617+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f3c4c0574e0 con 0x7f3c4c0b2730 2022-07-21T01:39:09.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.629+0000 7f3c48ff9700 1 -- 172.21.15.3:0/1360180988 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 13 and data pool 14 v34) v1 ==== 189+0+0 (secure 0 0 0) 0x7f3c40018070 con 0x7f3c4c0b2730 2022-07-21T01:39:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c2c0539e0 msgr2=0x7f3c2c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c2c0539e0 0x7f3c2c055ea0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f3c3c00b000 tx=0x7f3c3c009040).stop 2022-07-21T01:39:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c0b2730 msgr2=0x7f3c4c13e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c0b2730 0x7f3c4c13e840 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f3c40000bc0 tx=0x7f3c40007a40).stop 2022-07-21T01:39:09.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 shutdown_connections 2022-07-21T01:39:09.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c2c0539e0 0x7f3c2c055ea0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c0b3190 0x7f3c4c13ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c4c0b2730 0x7f3c4c13e840 secure :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0x7f3c40000bc0 tx=0x7f3c40007a40).stop 2022-07-21T01:39:09.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 --2- 172.21.15.3:0/1360180988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c4c066dc0 0x7f3c4c1402c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 >> 172.21.15.3:0/1360180988 conn(0x7f3c4c063870 msgr2=0x7f3c4c0648c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:09.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 shutdown_connections 2022-07-21T01:39:09.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.633+0000 7f3c508fb700 1 -- 172.21.15.3:0/1360180988 wait complete. 2022-07-21T01:39:09.637 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 13 and data pool 14 2022-07-21T01:39:09.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:39:09.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.745+0000 7f237bbc6700 1 Processor -- start 2022-07-21T01:39:09.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 -- start start 2022-07-21T01:39:09.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f23740b1f40 0x7f23740b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2374148530 0x7f23740b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23740b04e0 con 0x7f23740a7250 2022-07-21T01:39:09.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23740a9250 con 0x7f23740b1f40 2022-07-21T01:39:09.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23740b0780 con 0x7f2374148530 2022-07-21T01:39:09.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2379962700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f23740b1f40 0x7f23740b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2379962700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f23740b1f40 0x7f23740b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46856/0 (socket says 172.21.15.3:46856) 2022-07-21T01:39:09.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2379962700 1 -- 172.21.15.3:0/248941028 learned_addr learned my addr 172.21.15.3:0/248941028 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:09.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2379161700 1 --2- 172.21.15.3:0/248941028 >> v2:172.21.15.143:3300/0 conn(0x7f2374148530 0x7f23740b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2379962700 1 -- 172.21.15.3:0/248941028 >> v2:172.21.15.143:3300/0 conn(0x7f2374148530 msgr2=0x7f23740b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:09.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2379962700 1 --2- 172.21.15.3:0/248941028 >> v2:172.21.15.143:3300/0 conn(0x7f2374148530 0x7f23740b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2379962700 1 -- 172.21.15.3:0/248941028 >> v1:172.21.15.143:6789/0 conn(0x7f23740a7250 legacy=0x7f23741448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:09.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2379962700 1 -- 172.21.15.3:0/248941028 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23740b0a20 con 0x7f23740b1f40 2022-07-21T01:39:09.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2379962700 1 --2- 172.21.15.3:0/248941028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740b1f40 0x7f23740b1160 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f23740acff0 tx=0x7f2364009290).ready entity=mon.0 client_cookie=6881918b02efe69f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:09.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2378960700 1 -- 172.21.15.3:0/248941028 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f236400b820 con 0x7f23740b1f40 2022-07-21T01:39:09.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2378960700 1 -- 172.21.15.3:0/248941028 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236400b980 con 0x7f23740b1f40 2022-07-21T01:39:09.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f2378960700 1 -- 172.21.15.3:0/248941028 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f236400bb30 con 0x7f23740b1f40 2022-07-21T01:39:09.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 -- 172.21.15.3:0/248941028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740b1f40 msgr2=0x7f23740b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 --2- 172.21.15.3:0/248941028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740b1f40 0x7f23740b1160 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f23740acff0 tx=0x7f2364009290).stop 2022-07-21T01:39:09.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 -- 172.21.15.3:0/248941028 shutdown_connections 2022-07-21T01:39:09.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 --2- 172.21.15.3:0/248941028 >> v2:172.21.15.143:3300/0 conn(0x7f2374148530 0x7f23740b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 --2- 172.21.15.3:0/248941028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740b1f40 0x7f23740b1160 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 -- 172.21.15.3:0/248941028 >> 172.21.15.3:0/248941028 conn(0x7f237405f840 msgr2=0x7f237405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:09.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 -- 172.21.15.3:0/248941028 shutdown_connections 2022-07-21T01:39:09.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.749+0000 7f237bbc6700 1 -- 172.21.15.3:0/248941028 wait complete. 2022-07-21T01:39:09.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f237bbc6700 1 Processor -- start 2022-07-21T01:39:09.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f237bbc6700 1 -- start start 2022-07-21T01:39:09.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f237bbc6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740a7250 0x7f23740aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f237bbc6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23740b1f40 0x7f2374156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f237bbc6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2374148530 0x7f23741570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f237bbc6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237414a330 con 0x7f23740a7250 2022-07-21T01:39:09.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f237bbc6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237414a5d0 con 0x7f2374148530 2022-07-21T01:39:09.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f237bbc6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237414a870 con 0x7f23740b1f40 2022-07-21T01:39:09.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f2379962700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740a7250 0x7f23740aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f2379962700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740a7250 0x7f23740aeb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46862/0 (socket says 172.21.15.3:46862) 2022-07-21T01:39:09.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.753+0000 7f2379962700 1 -- 172.21.15.3:0/1818131820 learned_addr learned my addr 172.21.15.3:0/1818131820 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:09.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2379161700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23740b1f40 0x7f2374156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f237a163700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2374148530 0x7f23741570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2379962700 1 -- 172.21.15.3:0/1818131820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23740b1f40 msgr2=0x7f2374156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2379962700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23740b1f40 0x7f2374156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2379962700 1 -- 172.21.15.3:0/1818131820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2374148530 msgr2=0x7f23741570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2379962700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2374148530 0x7f23741570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2379962700 1 -- 172.21.15.3:0/1818131820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364009530 con 0x7f23740a7250 2022-07-21T01:39:09.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f237a163700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2374148530 0x7f23741570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:39:09.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2379962700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740a7250 0x7f23740aeb40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f23740acff0 tx=0x7f2364016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:09.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2372ffd700 1 -- 172.21.15.3:0/1818131820 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2364021630 con 0x7f23740a7250 2022-07-21T01:39:09.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2372ffd700 1 -- 172.21.15.3:0/1818131820 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2364021790 con 0x7f23740a7250 2022-07-21T01:39:09.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237414ab10 con 0x7f23740a7250 2022-07-21T01:39:09.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2372ffd700 1 -- 172.21.15.3:0/1818131820 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2364021af0 con 0x7f23740a7250 2022-07-21T01:39:09.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2374149ee0 con 0x7f23740a7250 2022-07-21T01:39:09.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23740b0cc0 con 0x7f23740a7250 2022-07-21T01:39:09.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2372ffd700 1 -- 172.21.15.3:0/1818131820 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2364027020 con 0x7f23740a7250 2022-07-21T01:39:09.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2372ffd700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2354053a10 0x7f2354055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:09.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2372ffd700 1 -- 172.21.15.3:0/1818131820 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f236406d8b0 con 0x7f23740a7250 2022-07-21T01:39:09.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.757+0000 7f2379161700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2354053a10 0x7f2354055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:09.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.761+0000 7f2379161700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2354053a10 0x7f2354055ed0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f236800c000 tx=0x7f2368008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:09.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.761+0000 7f2372ffd700 1 -- 172.21.15.3:0/1818131820 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f23640445a0 con 0x7f23740a7250 2022-07-21T01:39:09.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.885+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23740574e0 con 0x7f23740a7250 2022-07-21T01:39:09.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f2372ffd700 1 -- 172.21.15.3:0/1818131820 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+16287 (secure 0 0 0) 0x7f2364015070 con 0x7f23740a7250 2022-07-21T01:39:09.886 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:09.886 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":78,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:39:09.622184+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:39:07.362717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-21T01:39:08.613529+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:39:09.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2354053a10 msgr2=0x7f2354055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2354053a10 0x7f2354055ed0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f236800c000 tx=0x7f2368008040).stop 2022-07-21T01:39:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740a7250 msgr2=0x7f23740aeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740a7250 0x7f23740aeb40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f23740acff0 tx=0x7f2364016000).stop 2022-07-21T01:39:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 shutdown_connections 2022-07-21T01:39:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2354053a10 0x7f2354055ed0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2374148530 0x7f23741570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23740b1f40 0x7f2374156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 --2- 172.21.15.3:0/1818131820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23740a7250 0x7f23740aeb40 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 >> 172.21.15.3:0/1818131820 conn(0x7f237405f840 msgr2=0x7f23740b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 shutdown_connections 2022-07-21T01:39:09.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:09.889+0000 7f237bbc6700 1 -- 172.21.15.3:0/1818131820 wait complete. 2022-07-21T01:39:09.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:39:10.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.005+0000 7fdc6dbad700 1 Processor -- start 2022-07-21T01:39:10.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.005+0000 7fdc6dbad700 1 -- start start 2022-07-21T01:39:10.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6710 0x7fdc680b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc680b6af0 0x7fdc6812bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc68144240 con 0x7fdc680b4550 2022-07-21T01:39:10.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc68143d00 con 0x7fdc680b6af0 2022-07-21T01:39:10.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc68143fa0 con 0x7fdc680b6710 2022-07-21T01:39:10.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc67fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc680b6af0 0x7fdc6812bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc677fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6710 0x7fdc680b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc66ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdc680b4550 0x7fdc68061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55152/0 (socket says 172.21.15.3:55152) 2022-07-21T01:39:10.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc66ffd700 1 -- 172.21.15.3:0/3604894802 learned_addr learned my addr 172.21.15.3:0/3604894802 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:10.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc667fc700 1 -- 172.21.15.3:0/3604894802 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387428960 0 0) 0x7fdc68144240 con 0x7fdc680b4550 2022-07-21T01:39:10.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc667fc700 1 -- 172.21.15.3:0/3604894802 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc54003260 con 0x7fdc680b4550 2022-07-21T01:39:10.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc677fe700 1 -- 172.21.15.3:0/3604894802 >> v2:172.21.15.3:3300/0 conn(0x7fdc680b6af0 msgr2=0x7fdc6812bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:10.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc677fe700 1 --2- 172.21.15.3:0/3604894802 >> v2:172.21.15.3:3300/0 conn(0x7fdc680b6af0 0x7fdc6812bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc677fe700 1 -- 172.21.15.3:0/3604894802 >> v1:172.21.15.3:6789/0 conn(0x7fdc680b4550 legacy=0x7fdc68061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc677fe700 1 -- 172.21.15.3:0/3604894802 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc68061780 con 0x7fdc680b6710 2022-07-21T01:39:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc677fe700 1 --2- 172.21.15.3:0/3604894802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6710 0x7fdc680b5d80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fdc600032f0 tx=0x7fdc600091d0).ready entity=mon.2 client_cookie=eceae1642d3f180f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc667fc700 1 -- 172.21.15.3:0/3604894802 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc6000b600 con 0x7fdc680b6710 2022-07-21T01:39:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc667fc700 1 -- 172.21.15.3:0/3604894802 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6000b760 con 0x7fdc680b6710 2022-07-21T01:39:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc667fc700 1 -- 172.21.15.3:0/3604894802 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc6000b8d0 con 0x7fdc680b6710 2022-07-21T01:39:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 -- 172.21.15.3:0/3604894802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6710 msgr2=0x7fdc680b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 --2- 172.21.15.3:0/3604894802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6710 0x7fdc680b5d80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fdc600032f0 tx=0x7fdc600091d0).stop 2022-07-21T01:39:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 -- 172.21.15.3:0/3604894802 shutdown_connections 2022-07-21T01:39:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 --2- 172.21.15.3:0/3604894802 >> v2:172.21.15.3:3300/0 conn(0x7fdc680b6af0 0x7fdc6812bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 --2- 172.21.15.3:0/3604894802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6710 0x7fdc680b5d80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 -- 172.21.15.3:0/3604894802 >> 172.21.15.3:0/3604894802 conn(0x7fdc6805c620 msgr2=0x7fdc6805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:10.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 -- 172.21.15.3:0/3604894802 shutdown_connections 2022-07-21T01:39:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.009+0000 7fdc6dbad700 1 -- 172.21.15.3:0/3604894802 wait complete. 2022-07-21T01:39:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 Processor -- start 2022-07-21T01:39:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 -- start start 2022-07-21T01:39:10.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc680b4550 0x7fdc680b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc680b6710 0x7fdc681524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6af0 0x7fdc681529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6812a0c0 con 0x7fdc680b4550 2022-07-21T01:39:10.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680ad7b0 con 0x7fdc680b6710 2022-07-21T01:39:10.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc681562d0 con 0x7fdc680b6af0 2022-07-21T01:39:10.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc680b4550 0x7fdc680b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc680b4550 0x7fdc680b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46876/0 (socket says 172.21.15.3:46876) 2022-07-21T01:39:10.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc677fe700 1 -- 172.21.15.3:0/2881723903 learned_addr learned my addr 172.21.15.3:0/2881723903 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:10.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc67fff700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6af0 0x7fdc681529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc66ffd700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc680b6710 0x7fdc681524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc677fe700 1 -- 172.21.15.3:0/2881723903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6af0 msgr2=0x7fdc681529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc677fe700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6af0 0x7fdc681529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc677fe700 1 -- 172.21.15.3:0/2881723903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc680b6710 msgr2=0x7fdc681524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc677fe700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc680b6710 0x7fdc681524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc677fe700 1 -- 172.21.15.3:0/2881723903 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc60009470 con 0x7fdc680b4550 2022-07-21T01:39:10.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc677fe700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc680b4550 0x7fdc680b3320 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7fdc68156ab0 tx=0x7fdc60016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:10.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc64ff9700 1 -- 172.21.15.3:0/2881723903 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc60011cb0 con 0x7fdc680b4550 2022-07-21T01:39:10.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc64ff9700 1 -- 172.21.15.3:0/2881723903 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc60011e10 con 0x7fdc680b4550 2022-07-21T01:39:10.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc68156540 con 0x7fdc680b4550 2022-07-21T01:39:10.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc64ff9700 1 -- 172.21.15.3:0/2881723903 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc6001f5e0 con 0x7fdc680b4550 2022-07-21T01:39:10.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.013+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc680ad9e0 con 0x7fdc680b4550 2022-07-21T01:39:10.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.017+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc68061a20 con 0x7fdc680b4550 2022-07-21T01:39:10.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.017+0000 7fdc64ff9700 1 -- 172.21.15.3:0/2881723903 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc60026030 con 0x7fdc680b4550 2022-07-21T01:39:10.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.017+0000 7fdc64ff9700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc480539e0 0x7fdc48055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.017+0000 7fdc64ff9700 1 -- 172.21.15.3:0/2881723903 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdc6006cd50 con 0x7fdc680b4550 2022-07-21T01:39:10.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.017+0000 7fdc66ffd700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc480539e0 0x7fdc48055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.017+0000 7fdc66ffd700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc480539e0 0x7fdc48055ea0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fdc5c00b000 tx=0x7fdc5c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:10.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.025+0000 7fdc64ff9700 1 -- 172.21.15.3:0/2881723903 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc60043b10 con 0x7fdc680b4550 2022-07-21T01:39:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.149+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc680574e0 con 0x7fdc680b4550 2022-07-21T01:39:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.149+0000 7fdc64ff9700 1 -- 172.21.15.3:0/2881723903 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+16287 (secure 0 0 0) 0x7fdc60015090 con 0x7fdc680b4550 2022-07-21T01:39:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:10.149 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":78,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:39:09.622184+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:39:07.362717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-21T01:39:08.613529+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:39:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc480539e0 msgr2=0x7fdc48055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc480539e0 0x7fdc48055ea0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fdc5c00b000 tx=0x7fdc5c009040).stop 2022-07-21T01:39:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc680b4550 msgr2=0x7fdc680b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc680b4550 0x7fdc680b3320 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7fdc68156ab0 tx=0x7fdc60016000).stop 2022-07-21T01:39:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 shutdown_connections 2022-07-21T01:39:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc680b6af0 0x7fdc681529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc480539e0 0x7fdc48055ea0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc680b6710 0x7fdc681524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 --2- 172.21.15.3:0/2881723903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc680b4550 0x7fdc680b3320 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 >> 172.21.15.3:0/2881723903 conn(0x7fdc6805c620 msgr2=0x7fdc68142ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:10.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 shutdown_connections 2022-07-21T01:39:10.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.153+0000 7fdc6dbad700 1 -- 172.21.15.3:0/2881723903 wait complete. 2022-07-21T01:39:10.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:39:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef6d6f7700 1 Processor -- start 2022-07-21T01:39:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef6d6f7700 1 -- start start 2022-07-21T01:39:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef6d6f7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef680619b0 0x7fef680b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef6d6f7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fef680b6ee0 0x7fef68143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef6d6f7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef68061d90 con 0x7fef680b5ed0 2022-07-21T01:39:10.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef6d6f7700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef68142f60 con 0x7fef680b6ee0 2022-07-21T01:39:10.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef6d6f7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef680ba5a0 con 0x7fef680619b0 2022-07-21T01:39:10.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef677fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fef680b6ee0 0x7fef68143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef677fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fef680b6ee0 0x7fef68143890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46888/0 (socket says 172.21.15.3:46888) 2022-07-21T01:39:10.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef677fe700 1 -- 172.21.15.3:0/3922596802 learned_addr learned my addr 172.21.15.3:0/3922596802 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:10.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef67fff700 1 --2- 172.21.15.3:0/3922596802 >> v2:172.21.15.143:3300/0 conn(0x7fef680619b0 0x7fef680b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef677fe700 1 -- 172.21.15.3:0/3922596802 >> v2:172.21.15.143:3300/0 conn(0x7fef680619b0 msgr2=0x7fef680b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:10.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef677fe700 1 --2- 172.21.15.3:0/3922596802 >> v2:172.21.15.143:3300/0 conn(0x7fef680619b0 0x7fef680b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef677fe700 1 -- 172.21.15.3:0/3922596802 >> v1:172.21.15.143:6789/0 conn(0x7fef680b5ed0 legacy=0x7fef680b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef677fe700 1 -- 172.21.15.3:0/3922596802 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68143150 con 0x7fef680b6ee0 2022-07-21T01:39:10.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.277+0000 7fef67fff700 1 --2- 172.21.15.3:0/3922596802 >> v2:172.21.15.143:3300/0 conn(0x7fef680619b0 0x7fef680b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:10.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef677fe700 1 --2- 172.21.15.3:0/3922596802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b6ee0 0x7fef68143890 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7fef58002f70 tx=0x7fef58006eb0).ready entity=mon.0 client_cookie=97468463d023d79f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:10.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef66ffd700 1 -- 172.21.15.3:0/3922596802 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef5800d610 con 0x7fef680b6ee0 2022-07-21T01:39:10.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef66ffd700 1 -- 172.21.15.3:0/3922596802 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef58000e90 con 0x7fef680b6ee0 2022-07-21T01:39:10.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef66ffd700 1 -- 172.21.15.3:0/3922596802 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef5800d9a0 con 0x7fef680b6ee0 2022-07-21T01:39:10.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 -- 172.21.15.3:0/3922596802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b6ee0 msgr2=0x7fef68143890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 --2- 172.21.15.3:0/3922596802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b6ee0 0x7fef68143890 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7fef58002f70 tx=0x7fef58006eb0).stop 2022-07-21T01:39:10.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 -- 172.21.15.3:0/3922596802 shutdown_connections 2022-07-21T01:39:10.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 --2- 172.21.15.3:0/3922596802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b6ee0 0x7fef68143890 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 --2- 172.21.15.3:0/3922596802 >> v2:172.21.15.143:3300/0 conn(0x7fef680619b0 0x7fef680b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 -- 172.21.15.3:0/3922596802 >> 172.21.15.3:0/3922596802 conn(0x7fef6806af80 msgr2=0x7fef6805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:10.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 -- 172.21.15.3:0/3922596802 shutdown_connections 2022-07-21T01:39:10.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 -- 172.21.15.3:0/3922596802 wait complete. 2022-07-21T01:39:10.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 Processor -- start 2022-07-21T01:39:10.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.281+0000 7fef6d6f7700 1 -- start start 2022-07-21T01:39:10.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef6d6f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef680619b0 0x7fef680b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef6d6f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b5ed0 0x7fef680b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef6d6f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef680b6ee0 0x7fef6813f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef6d6f7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef68146420 con 0x7fef680b5ed0 2022-07-21T01:39:10.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef6d6f7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef68147170 con 0x7fef680619b0 2022-07-21T01:39:10.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef6d6f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef680aec40 con 0x7fef680b6ee0 2022-07-21T01:39:10.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b5ed0 0x7fef680b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b5ed0 0x7fef680b0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46892/0 (socket says 172.21.15.3:46892) 2022-07-21T01:39:10.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef677fe700 1 -- 172.21.15.3:0/2324301290 learned_addr learned my addr 172.21.15.3:0/2324301290 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:10.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef67fff700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef680619b0 0x7fef680b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef6cef6700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef680b6ee0 0x7fef6813f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef677fe700 1 -- 172.21.15.3:0/2324301290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef680b6ee0 msgr2=0x7fef6813f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef677fe700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef680b6ee0 0x7fef6813f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef677fe700 1 -- 172.21.15.3:0/2324301290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef680619b0 msgr2=0x7fef680b93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef677fe700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef680619b0 0x7fef680b93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef677fe700 1 -- 172.21.15.3:0/2324301290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5800b1e0 con 0x7fef680b5ed0 2022-07-21T01:39:10.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef677fe700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b5ed0 0x7fef680b0340 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7fef68147300 tx=0x7fef58015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:10.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef657fa700 1 -- 172.21.15.3:0/2324301290 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef58006be0 con 0x7fef680b5ed0 2022-07-21T01:39:10.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef657fa700 1 -- 172.21.15.3:0/2324301290 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5800d290 con 0x7fef680b5ed0 2022-07-21T01:39:10.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef68146ed0 con 0x7fef680b5ed0 2022-07-21T01:39:10.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef657fa700 1 -- 172.21.15.3:0/2324301290 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef58011e90 con 0x7fef680b5ed0 2022-07-21T01:39:10.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.285+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef68146880 con 0x7fef680b5ed0 2022-07-21T01:39:10.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.289+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef680574e0 con 0x7fef680b5ed0 2022-07-21T01:39:10.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.289+0000 7fef657fa700 1 -- 172.21.15.3:0/2324301290 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef58025030 con 0x7fef680b5ed0 2022-07-21T01:39:10.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.289+0000 7fef657fa700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef4c053a70 0x7fef4c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.289+0000 7fef657fa700 1 -- 172.21.15.3:0/2324301290 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fef5806d1e0 con 0x7fef680b5ed0 2022-07-21T01:39:10.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.289+0000 7fef67fff700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef4c053a70 0x7fef4c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.289+0000 7fef67fff700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef4c053a70 0x7fef4c055f30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fef5000c000 tx=0x7fef50008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:10.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.297+0000 7fef657fa700 1 -- 172.21.15.3:0/2324301290 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef58043fa0 con 0x7fef680b5ed0 2022-07-21T01:39:10.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.461+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fef6805db60 con 0x7fef680b5ed0 2022-07-21T01:39:10.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.637+0000 7fef657fa700 1 -- 172.21.15.3:0/2324301290 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) v1 ==== 118+0+0 (secure 0 0 0) 0x7fef58023020 con 0x7fef680b5ed0 2022-07-21T01:39:10.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef4c053a70 msgr2=0x7fef4c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef4c053a70 0x7fef4c055f30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fef5000c000 tx=0x7fef50008040).stop 2022-07-21T01:39:10.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b5ed0 msgr2=0x7fef680b0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b5ed0 0x7fef680b0340 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7fef68147300 tx=0x7fef58015000).stop 2022-07-21T01:39:10.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 shutdown_connections 2022-07-21T01:39:10.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef680b6ee0 0x7fef6813f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef680b5ed0 0x7fef680b0340 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef4c053a70 0x7fef4c055f30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 --2- 172.21.15.3:0/2324301290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef680619b0 0x7fef680b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 >> 172.21.15.3:0/2324301290 conn(0x7fef6806af80 msgr2=0x7fef6805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:10.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 shutdown_connections 2022-07-21T01:39:10.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.641+0000 7fef6d6f7700 1 -- 172.21.15.3:0/2324301290 wait complete. 2022-07-21T01:39:10.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:39:10.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7cae962700 1 Processor -- start 2022-07-21T01:39:10.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7cae962700 1 -- start start 2022-07-21T01:39:10.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7cae962700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ca8148530 0x7f7ca80b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7cae962700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca80b04e0 con 0x7f7ca80a7250 2022-07-21T01:39:10.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7cae962700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca80b0780 con 0x7f7ca80b1f40 2022-07-21T01:39:10.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7cae962700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca80a9250 con 0x7f7ca8148530 2022-07-21T01:39:10.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7caceff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7ca80a7250 0x7f7ca81448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55182/0 (socket says 172.21.15.3:55182) 2022-07-21T01:39:10.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7caceff700 1 -- 172.21.15.3:0/429333520 learned_addr learned my addr 172.21.15.3:0/429333520 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:10.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca77fe700 1 --2- 172.21.15.3:0/429333520 >> v2:172.21.15.143:3300/0 conn(0x7f7ca8148530 0x7f7ca80b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca6ffd700 1 -- 172.21.15.3:0/429333520 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510093011 0 0) 0x7f7ca80b04e0 con 0x7f7ca80a7250 2022-07-21T01:39:10.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca6ffd700 1 -- 172.21.15.3:0/429333520 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c88003260 con 0x7f7ca80a7250 2022-07-21T01:39:10.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca77fe700 1 -- 172.21.15.3:0/429333520 >> v1:172.21.15.143:6789/0 conn(0x7f7ca80b1f40 legacy=0x7f7ca80b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca77fe700 1 -- 172.21.15.3:0/429333520 >> v1:172.21.15.3:6789/0 conn(0x7f7ca80a7250 legacy=0x7f7ca81448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca77fe700 1 -- 172.21.15.3:0/429333520 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca80b0a20 con 0x7f7ca8148530 2022-07-21T01:39:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca77fe700 1 --2- 172.21.15.3:0/429333520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca8148530 0x7f7ca80b1160 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7ca81478c0 tx=0x7f7ca00091d0).ready entity=mon.1 client_cookie=1bdfaa2482b83a4b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca6ffd700 1 -- 172.21.15.3:0/429333520 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ca000b680 con 0x7f7ca8148530 2022-07-21T01:39:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca6ffd700 1 -- 172.21.15.3:0/429333520 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca000b7e0 con 0x7f7ca8148530 2022-07-21T01:39:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.765+0000 7f7ca6ffd700 1 -- 172.21.15.3:0/429333520 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ca000b950 con 0x7f7ca8148530 2022-07-21T01:39:10.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.769+0000 7f7cae962700 1 -- 172.21.15.3:0/429333520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca8148530 msgr2=0x7f7ca80b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.769+0000 7f7cae962700 1 --2- 172.21.15.3:0/429333520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca8148530 0x7f7ca80b1160 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7ca81478c0 tx=0x7f7ca00091d0).stop 2022-07-21T01:39:10.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.769+0000 7f7cae962700 1 -- 172.21.15.3:0/429333520 shutdown_connections 2022-07-21T01:39:10.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.769+0000 7f7cae962700 1 --2- 172.21.15.3:0/429333520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca8148530 0x7f7ca80b1160 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.769+0000 7f7cae962700 1 -- 172.21.15.3:0/429333520 >> 172.21.15.3:0/429333520 conn(0x7f7ca805f840 msgr2=0x7f7ca805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:10.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.769+0000 7f7cae962700 1 -- 172.21.15.3:0/429333520 shutdown_connections 2022-07-21T01:39:10.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.769+0000 7f7cae962700 1 -- 172.21.15.3:0/429333520 wait complete. 2022-07-21T01:39:10.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.769+0000 7f7cae962700 1 Processor -- start 2022-07-21T01:39:10.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.769+0000 7f7cae962700 1 -- start start 2022-07-21T01:39:10.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7cae962700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca80a7250 0x7f7ca814b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7cae962700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ca80b1f40 0x7f7ca80aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7cae962700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca8148530 0x7f7ca8143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7cae962700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca814aef0 con 0x7f7ca80b1f40 2022-07-21T01:39:10.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7cae962700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca814ac50 con 0x7f7ca8148530 2022-07-21T01:39:10.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7cae962700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca814a340 con 0x7f7ca80a7250 2022-07-21T01:39:10.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ca80b1f40 0x7f7ca80aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ca80b1f40 0x7f7ca80aeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46904/0 (socket says 172.21.15.3:46904) 2022-07-21T01:39:10.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca77fe700 1 -- 172.21.15.3:0/391355015 learned_addr learned my addr 172.21.15.3:0/391355015 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:10.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7caceff700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca8148530 0x7f7ca8143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca7fff700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca80a7250 0x7f7ca814b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca77fe700 1 -- 172.21.15.3:0/391355015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca80a7250 msgr2=0x7f7ca814b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca77fe700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca80a7250 0x7f7ca814b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca77fe700 1 -- 172.21.15.3:0/391355015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca8148530 msgr2=0x7f7ca8143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca77fe700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca8148530 0x7f7ca8143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca77fe700 1 -- 172.21.15.3:0/391355015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca0009470 con 0x7f7ca80b1f40 2022-07-21T01:39:10.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca77fe700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ca80b1f40 0x7f7ca80aeec0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f7ca81478c0 tx=0x7f7ca0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:10.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca57fa700 1 -- 172.21.15.3:0/391355015 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ca000b680 con 0x7f7ca80b1f40 2022-07-21T01:39:10.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca57fa700 1 -- 172.21.15.3:0/391355015 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca0021630 con 0x7f7ca80b1f40 2022-07-21T01:39:10.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca814a5e0 con 0x7f7ca80b1f40 2022-07-21T01:39:10.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7ca57fa700 1 -- 172.21.15.3:0/391355015 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ca0021990 con 0x7f7ca80b1f40 2022-07-21T01:39:10.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca80a9fa0 con 0x7f7ca80b1f40 2022-07-21T01:39:10.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.773+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca80aa2b0 con 0x7f7ca80b1f40 2022-07-21T01:39:10.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.777+0000 7f7ca57fa700 1 -- 172.21.15.3:0/391355015 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7ca0027020 con 0x7f7ca80b1f40 2022-07-21T01:39:10.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.777+0000 7f7ca57fa700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7c8c053b00 0x7f7c8c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:10.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.777+0000 7f7ca57fa700 1 -- 172.21.15.3:0/391355015 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7ca006d2d0 con 0x7f7ca80b1f40 2022-07-21T01:39:10.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.777+0000 7f7ca7fff700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7c8c053b00 0x7f7c8c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:10.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.777+0000 7f7ca7fff700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7c8c053b00 0x7f7c8c055fc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f7c9000d000 tx=0x7f7c90000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:10.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.781+0000 7f7ca57fa700 1 -- 172.21.15.3:0/391355015 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7ca0044010 con 0x7f7ca80b1f40 2022-07-21T01:39:10.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.949+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ca81444c0 con 0x7f7ca80b1f40 2022-07-21T01:39:10.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.949+0000 7f7ca57fa700 1 -- 172.21.15.3:0/391355015 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2487 (secure 0 0 0) 0x7f7ca00159d0 con 0x7f7ca80b1f40 2022-07-21T01:39:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:10.949 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:39:09.622528+0000","modified":"2022-07-21T01:39:10.626387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6259":{"gid":6259,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/862725488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":862725488},{"type":"v1","addr":"172.21.15.143:6833","nonce":862725488}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-21T01:39:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7c8c053b00 msgr2=0x7f7c8c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7c8c053b00 0x7f7c8c055fc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f7c9000d000 tx=0x7f7c90000bc0).stop 2022-07-21T01:39:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ca80b1f40 msgr2=0x7f7ca80aeec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ca80b1f40 0x7f7ca80aeec0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f7ca81478c0 tx=0x7f7ca0016000).stop 2022-07-21T01:39:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 shutdown_connections 2022-07-21T01:39:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7c8c053b00 0x7f7c8c055fc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca8148530 0x7f7ca8143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ca80b1f40 0x7f7ca80aeec0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 --2- 172.21.15.3:0/391355015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca80a7250 0x7f7ca814b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 >> 172.21.15.3:0/391355015 conn(0x7f7ca805f840 msgr2=0x7f7ca814e170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 shutdown_connections 2022-07-21T01:39:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:10.953+0000 7f7cae962700 1 -- 172.21.15.3:0/391355015 wait complete. 2022-07-21T01:39:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2022-07-21T01:39:10.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:39:11.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 Processor -- start 2022-07-21T01:39:11.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 -- start start 2022-07-21T01:39:11.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b08a0 0x7fcbbc142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcbbc0b3260 0x7fcbbc05c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc143ab0 con 0x7fcbbc0ad000 2022-07-21T01:39:11.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbbc0b0c80 con 0x7fcbbc0b3260 2022-07-21T01:39:11.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0ae810 con 0x7fcbbc0b08a0 2022-07-21T01:39:11.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc123a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcbbc0b3260 0x7fcbbc05c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc123a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcbbc0b3260 0x7fcbbc05c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46914/0 (socket says 172.21.15.3:46914) 2022-07-21T01:39:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc123a700 1 -- 172.21.15.3:0/184799475 learned_addr learned my addr 172.21.15.3:0/184799475 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc1a3b700 1 --2- 172.21.15.3:0/184799475 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b08a0 0x7fcbbc142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc123a700 1 -- 172.21.15.3:0/184799475 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b08a0 msgr2=0x7fcbbc142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc123a700 1 --2- 172.21.15.3:0/184799475 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b08a0 0x7fcbbc142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc123a700 1 -- 172.21.15.3:0/184799475 >> v1:172.21.15.143:6789/0 conn(0x7fcbbc0ad000 legacy=0x7fcbbc1433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc123a700 1 -- 172.21.15.3:0/184799475 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0aeab0 con 0x7fcbbc0b3260 2022-07-21T01:39:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc123a700 1 --2- 172.21.15.3:0/184799475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0b3260 0x7fcbbc05c620 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7fcbb40032f0 tx=0x7fcbb4009290).ready entity=mon.0 client_cookie=d1ad7e10e3ac8b7b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc0a39700 1 -- 172.21.15.3:0/184799475 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb400b8a0 con 0x7fcbbc0b3260 2022-07-21T01:39:11.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc0a39700 1 -- 172.21.15.3:0/184799475 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb400ba00 con 0x7fcbbc0b3260 2022-07-21T01:39:11.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc0a39700 1 -- 172.21.15.3:0/184799475 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb400bb70 con 0x7fcbbc0b3260 2022-07-21T01:39:11.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/184799475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0b3260 msgr2=0x7fcbbc05c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 --2- 172.21.15.3:0/184799475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0b3260 0x7fcbbc05c620 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7fcbb40032f0 tx=0x7fcbb4009290).stop 2022-07-21T01:39:11.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/184799475 shutdown_connections 2022-07-21T01:39:11.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 --2- 172.21.15.3:0/184799475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0b3260 0x7fcbbc05c620 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 --2- 172.21.15.3:0/184799475 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b08a0 0x7fcbbc142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/184799475 >> 172.21.15.3:0/184799475 conn(0x7fcbbc0612d0 msgr2=0x7fcbbc062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:11.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/184799475 shutdown_connections 2022-07-21T01:39:11.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.073+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/184799475 wait complete. 2022-07-21T01:39:11.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc2a3d700 1 Processor -- start 2022-07-21T01:39:11.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc2a3d700 1 -- start start 2022-07-21T01:39:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc2a3d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0ad000 0x7fcbbc13a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc2a3d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbbc0b08a0 0x7fcbbc065990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc2a3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b3260 0x7fcbbc065e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc2a3d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0b4200 con 0x7fcbbc0ad000 2022-07-21T01:39:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc2a3d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0b4650 con 0x7fcbbc0b08a0 2022-07-21T01:39:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc2a3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc063d40 con 0x7fcbbc0b3260 2022-07-21T01:39:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc1a3b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0ad000 0x7fcbbc13a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc1a3b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0ad000 0x7fcbbc13a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46920/0 (socket says 172.21.15.3:46920) 2022-07-21T01:39:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc1a3b700 1 -- 172.21.15.3:0/1412707493 learned_addr learned my addr 172.21.15.3:0/1412707493 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc123a700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbbc0b08a0 0x7fcbbc065990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc223c700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b3260 0x7fcbbc065e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc1a3b700 1 -- 172.21.15.3:0/1412707493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b3260 msgr2=0x7fcbbc065e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc1a3b700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b3260 0x7fcbbc065e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc1a3b700 1 -- 172.21.15.3:0/1412707493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbbc0b08a0 msgr2=0x7fcbbc065990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc1a3b700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbbc0b08a0 0x7fcbbc065990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc1a3b700 1 -- 172.21.15.3:0/1412707493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4009530 con 0x7fcbbc0ad000 2022-07-21T01:39:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.077+0000 7fcbc1a3b700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0ad000 0x7fcbbc13a130 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7fcbb80030b0 tx=0x7fcbb8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbb2ffd700 1 -- 172.21.15.3:0/1412707493 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb8011780 con 0x7fcbbc0ad000 2022-07-21T01:39:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbb2ffd700 1 -- 172.21.15.3:0/1412707493 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb80118e0 con 0x7fcbbc0ad000 2022-07-21T01:39:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbb2ffd700 1 -- 172.21.15.3:0/1412707493 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb8011c40 con 0x7fcbbc0ad000 2022-07-21T01:39:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc063fe0 con 0x7fcbbc0ad000 2022-07-21T01:39:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc064520 con 0x7fcbbc0ad000 2022-07-21T01:39:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbbc0646e0 con 0x7fcbbc0ad000 2022-07-21T01:39:11.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbb2ffd700 1 -- 172.21.15.3:0/1412707493 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcbb8012420 con 0x7fcbbc0ad000 2022-07-21T01:39:11.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbb2ffd700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb9c053950 0x7fcb9c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbb2ffd700 1 -- 172.21.15.3:0/1412707493 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcbb806be80 con 0x7fcbbc0ad000 2022-07-21T01:39:11.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.081+0000 7fcbc123a700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb9c053950 0x7fcb9c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.085+0000 7fcbc123a700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb9c053950 0x7fcb9c055e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fcbb401b000 tx=0x7fcbb4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.089+0000 7fcbb2ffd700 1 -- 172.21.15.3:0/1412707493 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcbb8042ad0 con 0x7fcbbc0ad000 2022-07-21T01:39:11.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.213+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbbc0574e0 con 0x7fcbbc0ad000 2022-07-21T01:39:11.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbb2ffd700 1 -- 172.21.15.3:0/1412707493 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16274 (secure 0 0 0) 0x7fcbb80460f0 con 0x7fcbbc0ad000 2022-07-21T01:39:11.215 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:11.215 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":79,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:39:10.626495+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:39:07.362717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-21T01:39:08.613529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:39:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb9c053950 msgr2=0x7fcb9c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb9c053950 0x7fcb9c055e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fcbb401b000 tx=0x7fcbb4016000).stop 2022-07-21T01:39:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0ad000 msgr2=0x7fcbbc13a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0ad000 0x7fcbbc13a130 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7fcbb80030b0 tx=0x7fcbb8008040).stop 2022-07-21T01:39:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 shutdown_connections 2022-07-21T01:39:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbbc0b3260 0x7fcbbc065e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb9c053950 0x7fcb9c055e10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbbc0b08a0 0x7fcbbc065990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 --2- 172.21.15.3:0/1412707493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbbc0ad000 0x7fcbbc13a130 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 >> 172.21.15.3:0/1412707493 conn(0x7fcbbc0612d0 msgr2=0x7fcbbc142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 shutdown_connections 2022-07-21T01:39:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.217+0000 7fcbc2a3d700 1 -- 172.21.15.3:0/1412707493 wait complete. 2022-07-21T01:39:11.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:39:11.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.345+0000 7f72e4e1e700 1 Processor -- start 2022-07-21T01:39:11.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.345+0000 7f72e4e1e700 1 -- start start 2022-07-21T01:39:11.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.345+0000 7f72e4e1e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72e00ad750 0x7f72e0061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.345+0000 7f72e4e1e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 0x7f72e0140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.345+0000 7f72e4e1e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e00aee80 con 0x7f72e00614b0 2022-07-21T01:39:11.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.345+0000 7f72e4e1e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e00b70b0 con 0x7f72e00ad750 2022-07-21T01:39:11.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.345+0000 7f72e4e1e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00af0b0 con 0x7f72e01405d0 2022-07-21T01:39:11.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72deffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72e00ad750 0x7f72e0061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72dffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 0x7f72e0140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72deffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72e00ad750 0x7f72e0061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46930/0 (socket says 172.21.15.3:46930) 2022-07-21T01:39:11.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72deffd700 1 -- 172.21.15.3:0/997571846 learned_addr learned my addr 172.21.15.3:0/997571846 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:11.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72df7fe700 1 --1- 172.21.15.3:0/997571846 >> v1:172.21.15.143:6789/0 conn(0x7f72e00614b0 0x7f72e00b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:57322/0 (socket says 172.21.15.3:57322) 2022-07-21T01:39:11.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72deffd700 1 -- 172.21.15.3:0/997571846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 msgr2=0x7f72e0140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:11.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72deffd700 1 --2- 172.21.15.3:0/997571846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 0x7f72e0140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72deffd700 1 -- 172.21.15.3:0/997571846 >> v1:172.21.15.143:6789/0 conn(0x7f72e00614b0 legacy=0x7f72e00b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72deffd700 1 -- 172.21.15.3:0/997571846 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00af350 con 0x7f72e00ad750 2022-07-21T01:39:11.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72dffff700 1 --2- 172.21.15.3:0/997571846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 0x7f72e0140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:11.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72deffd700 1 --2- 172.21.15.3:0/997571846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00ad750 0x7f72e0061890 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f72d0002f70 tx=0x7f72d0006eb0).ready entity=mon.0 client_cookie=8c5e8148ef7c15a1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72de7fc700 1 -- 172.21.15.3:0/997571846 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72d000d570 con 0x7f72e00ad750 2022-07-21T01:39:11.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72de7fc700 1 -- 172.21.15.3:0/997571846 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d0000e90 con 0x7f72e00ad750 2022-07-21T01:39:11.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72de7fc700 1 -- 172.21.15.3:0/997571846 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72d000d900 con 0x7f72e00ad750 2022-07-21T01:39:11.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 -- 172.21.15.3:0/997571846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00ad750 msgr2=0x7f72e0061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 --2- 172.21.15.3:0/997571846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00ad750 0x7f72e0061890 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f72d0002f70 tx=0x7f72d0006eb0).stop 2022-07-21T01:39:11.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 -- 172.21.15.3:0/997571846 shutdown_connections 2022-07-21T01:39:11.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 --2- 172.21.15.3:0/997571846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 0x7f72e0140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 --2- 172.21.15.3:0/997571846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00ad750 0x7f72e0061890 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 -- 172.21.15.3:0/997571846 >> 172.21.15.3:0/997571846 conn(0x7f72e0066f70 msgr2=0x7f72e0067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:11.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 -- 172.21.15.3:0/997571846 shutdown_connections 2022-07-21T01:39:11.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 -- 172.21.15.3:0/997571846 wait complete. 2022-07-21T01:39:11.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 Processor -- start 2022-07-21T01:39:11.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.349+0000 7f72e4e1e700 1 -- start start 2022-07-21T01:39:11.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72e4e1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00614b0 0x7f72e005d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72e4e1e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72e00ad750 0x7f72e0152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72e4e1e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 0x7f72e0152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72e4e1e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00aac80 con 0x7f72e00614b0 2022-07-21T01:39:11.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72e4e1e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00aa2b0 con 0x7f72e00ad750 2022-07-21T01:39:11.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72e4e1e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e0152650 con 0x7f72e01405d0 2022-07-21T01:39:11.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72df7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00614b0 0x7f72e005d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72df7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00614b0 0x7f72e005d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46934/0 (socket says 172.21.15.3:46934) 2022-07-21T01:39:11.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72df7fe700 1 -- 172.21.15.3:0/3384202593 learned_addr learned my addr 172.21.15.3:0/3384202593 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:11.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72dffff700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 0x7f72e0152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72deffd700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72e00ad750 0x7f72e0152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72df7fe700 1 -- 172.21.15.3:0/3384202593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 msgr2=0x7f72e0152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72df7fe700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 0x7f72e0152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72df7fe700 1 -- 172.21.15.3:0/3384202593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72e00ad750 msgr2=0x7f72e0152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72df7fe700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72e00ad750 0x7f72e0152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72df7fe700 1 -- 172.21.15.3:0/3384202593 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d8009470 con 0x7f72e00614b0 2022-07-21T01:39:11.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72df7fe700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00614b0 0x7f72e005d100 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f72e00af970 tx=0x7f72c8007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72dcff9700 1 -- 172.21.15.3:0/3384202593 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72c80070d0 con 0x7f72e00614b0 2022-07-21T01:39:11.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72dcff9700 1 -- 172.21.15.3:0/3384202593 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c8007230 con 0x7f72e00614b0 2022-07-21T01:39:11.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d000b1e0 con 0x7f72e00614b0 2022-07-21T01:39:11.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72dcff9700 1 -- 172.21.15.3:0/3384202593 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72c8005550 con 0x7f72e00614b0 2022-07-21T01:39:11.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e01456e0 con 0x7f72e00614b0 2022-07-21T01:39:11.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.353+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e00aa4b0 con 0x7f72e00614b0 2022-07-21T01:39:11.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.357+0000 7f72dcff9700 1 -- 172.21.15.3:0/3384202593 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f72c801a070 con 0x7f72e00614b0 2022-07-21T01:39:11.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.357+0000 7f72dcff9700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f72c4053950 0x7f72c4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.357+0000 7f72dcff9700 1 -- 172.21.15.3:0/3384202593 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f72c806b340 con 0x7f72e00614b0 2022-07-21T01:39:11.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.357+0000 7f72deffd700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f72c4053950 0x7f72c4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.357+0000 7f72deffd700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f72c4053950 0x7f72c4055e10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f72d001c000 tx=0x7f72d0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.361+0000 7f72dcff9700 1 -- 172.21.15.3:0/3384202593 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f72c8042060 con 0x7f72e00614b0 2022-07-21T01:39:11.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.529+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f72e0155c00 con 0x7f72e00614b0 2022-07-21T01:39:11.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72dcff9700 1 -- 172.21.15.3:0/3384202593 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v23) v1 ==== 184+0+0 (secure 0 0 0) 0x7f72c8045680 con 0x7f72e00614b0 2022-07-21T01:39:11.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f72c4053950 msgr2=0x7f72c4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f72c4053950 0x7f72c4055e10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f72d001c000 tx=0x7f72d0015000).stop 2022-07-21T01:39:11.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00614b0 msgr2=0x7f72e005d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00614b0 0x7f72e005d100 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f72e00af970 tx=0x7f72c8007c40).stop 2022-07-21T01:39:11.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 shutdown_connections 2022-07-21T01:39:11.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72e01405d0 0x7f72e0152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f72c4053950 0x7f72c4055e10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72e00ad750 0x7f72e0152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 --2- 172.21.15.3:0/3384202593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72e00614b0 0x7f72e005d100 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 >> 172.21.15.3:0/3384202593 conn(0x7f72e0066f70 msgr2=0x7f72e00b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:11.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 shutdown_connections 2022-07-21T01:39:11.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.537+0000 7f72e4e1e700 1 -- 172.21.15.3:0/3384202593 wait complete. 2022-07-21T01:39:11.540 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:39:11.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:39:11.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa753e26700 1 Processor -- start 2022-07-21T01:39:11.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa753e26700 1 -- start start 2022-07-21T01:39:11.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa753e26700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b4550 0x7fa74c0b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa753e26700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa74c0b6af0 0x7fa74c12bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa753e26700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74c144240 con 0x7fa74c0b6710 2022-07-21T01:39:11.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa753e26700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74c143d00 con 0x7fa74c0b6af0 2022-07-21T01:39:11.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa753e26700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c143fa0 con 0x7fa74c0b4550 2022-07-21T01:39:11.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa7523c3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa74c0b6af0 0x7fa74c12bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa7513c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b4550 0x7fa74c0b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.665+0000 7fa7523c3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa74c0b6af0 0x7fa74c12bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:52392/0 (socket says 172.21.15.3:52392) 2022-07-21T01:39:11.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa7523c3700 1 -- 172.21.15.3:0/2799701730 learned_addr learned my addr 172.21.15.3:0/2799701730 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:11.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa7523c3700 1 -- 172.21.15.3:0/2799701730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b4550 msgr2=0x7fa74c0b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:11.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa7523c3700 1 --2- 172.21.15.3:0/2799701730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b4550 0x7fa74c0b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa7523c3700 1 -- 172.21.15.3:0/2799701730 >> v1:172.21.15.143:6789/0 conn(0x7fa74c0b6710 legacy=0x7fa74c061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa7523c3700 1 -- 172.21.15.3:0/2799701730 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c061780 con 0x7fa74c0b6af0 2022-07-21T01:39:11.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa7523c3700 1 --2- 172.21.15.3:0/2799701730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6af0 0x7fa74c12bb90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fa7440032f0 tx=0x7fa744009290).ready entity=mon.1 client_cookie=aafb02bcdef83e35 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa750bc0700 1 -- 172.21.15.3:0/2799701730 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa74400b8a0 con 0x7fa74c0b6af0 2022-07-21T01:39:11.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa750bc0700 1 -- 172.21.15.3:0/2799701730 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74400ba00 con 0x7fa74c0b6af0 2022-07-21T01:39:11.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa750bc0700 1 -- 172.21.15.3:0/2799701730 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa74400bb70 con 0x7fa74c0b6af0 2022-07-21T01:39:11.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 -- 172.21.15.3:0/2799701730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6af0 msgr2=0x7fa74c12bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 --2- 172.21.15.3:0/2799701730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6af0 0x7fa74c12bb90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fa7440032f0 tx=0x7fa744009290).stop 2022-07-21T01:39:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 -- 172.21.15.3:0/2799701730 shutdown_connections 2022-07-21T01:39:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 --2- 172.21.15.3:0/2799701730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6af0 0x7fa74c12bb90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 --2- 172.21.15.3:0/2799701730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b4550 0x7fa74c0b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 -- 172.21.15.3:0/2799701730 >> 172.21.15.3:0/2799701730 conn(0x7fa74c05c620 msgr2=0x7fa74c05ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:11.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 -- 172.21.15.3:0/2799701730 shutdown_connections 2022-07-21T01:39:11.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 -- 172.21.15.3:0/2799701730 wait complete. 2022-07-21T01:39:11.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 Processor -- start 2022-07-21T01:39:11.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.669+0000 7fa753e26700 1 -- start start 2022-07-21T01:39:11.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa753e26700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa74c0b4550 0x7fa74c128ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa753e26700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6710 0x7fa74c0adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa753e26700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b6af0 0x7fa74c0ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa753e26700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c12ad30 con 0x7fa74c0b4550 2022-07-21T01:39:11.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa753e26700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c12a060 con 0x7fa74c0b6710 2022-07-21T01:39:11.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa753e26700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c12aa90 con 0x7fa74c0b6af0 2022-07-21T01:39:11.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa751bc2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa74c0b4550 0x7fa74c128ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa7513c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6710 0x7fa74c0adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa751bc2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa74c0b4550 0x7fa74c128ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46948/0 (socket says 172.21.15.3:46948) 2022-07-21T01:39:11.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa7523c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b6af0 0x7fa74c0ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa751bc2700 1 -- 172.21.15.3:0/2508620426 learned_addr learned my addr 172.21.15.3:0/2508620426 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:11.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa751bc2700 1 -- 172.21.15.3:0/2508620426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b6af0 msgr2=0x7fa74c0ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa751bc2700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b6af0 0x7fa74c0ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa751bc2700 1 -- 172.21.15.3:0/2508620426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6710 msgr2=0x7fa74c0adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa751bc2700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6710 0x7fa74c0adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa751bc2700 1 -- 172.21.15.3:0/2508620426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa744009530 con 0x7fa74c0b4550 2022-07-21T01:39:11.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa7513c1700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6710 0x7fa74c0adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:11.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa751bc2700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa74c0b4550 0x7fa74c128ac0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7fa74c061100 tx=0x7fa740007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa73effd700 1 -- 172.21.15.3:0/2508620426 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa740007030 con 0x7fa74c0b4550 2022-07-21T01:39:11.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa73effd700 1 -- 172.21.15.3:0/2508620426 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa740007190 con 0x7fa74c0b4550 2022-07-21T01:39:11.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c0b2ea0 con 0x7fa74c0b4550 2022-07-21T01:39:11.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa73effd700 1 -- 172.21.15.3:0/2508620426 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa740006510 con 0x7fa74c0b4550 2022-07-21T01:39:11.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c128490 con 0x7fa74c0b4550 2022-07-21T01:39:11.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.673+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74c0b3340 con 0x7fa74c0b4550 2022-07-21T01:39:11.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.677+0000 7fa73effd700 1 -- 172.21.15.3:0/2508620426 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa740010030 con 0x7fa74c0b4550 2022-07-21T01:39:11.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.677+0000 7fa73effd700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa72c053b00 0x7fa72c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.677+0000 7fa73effd700 1 -- 172.21.15.3:0/2508620426 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa74006bdd0 con 0x7fa74c0b4550 2022-07-21T01:39:11.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.677+0000 7fa7513c1700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa72c053b00 0x7fa72c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.677+0000 7fa7513c1700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa72c053b00 0x7fa72c055fc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fa74800b000 tx=0x7fa748006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.685+0000 7fa73effd700 1 -- 172.21.15.3:0/2508620426 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa740042a00 con 0x7fa74c0b4550 2022-07-21T01:39:11.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.853+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa74c061a20 con 0x7fa74c0b4550 2022-07-21T01:39:11.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa73effd700 1 -- 172.21.15.3:0/2508620426 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v24) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa740046020 con 0x7fa74c0b4550 2022-07-21T01:39:11.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa72c053b00 msgr2=0x7fa72c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa72c053b00 0x7fa72c055fc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fa74800b000 tx=0x7fa748006df0).stop 2022-07-21T01:39:11.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa74c0b4550 msgr2=0x7fa74c128ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa74c0b4550 0x7fa74c128ac0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7fa74c061100 tx=0x7fa740007c10).stop 2022-07-21T01:39:11.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 shutdown_connections 2022-07-21T01:39:11.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa74c0b6af0 0x7fa74c0ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa72c053b00 0x7fa72c055fc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa74c0b6710 0x7fa74c0adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 --2- 172.21.15.3:0/2508620426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa74c0b4550 0x7fa74c128ac0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 >> 172.21.15.3:0/2508620426 conn(0x7fa74c05c620 msgr2=0x7fa74c141120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:11.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 shutdown_connections 2022-07-21T01:39:11.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.865+0000 7fa753e26700 1 -- 172.21.15.3:0/2508620426 wait complete. 2022-07-21T01:39:11.867 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:39:11.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:39:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.981+0000 7f376a546700 1 Processor -- start 2022-07-21T01:39:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 -- start start 2022-07-21T01:39:11.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f37640b2710 0x7f376414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f37640a8c50 0x7f376414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:11.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37640a7250 con 0x7f37640b01a0 2022-07-21T01:39:11.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640a7480 con 0x7f37640b2710 2022-07-21T01:39:11.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640a7720 con 0x7f37640a8c50 2022-07-21T01:39:11.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769544700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f37640b2710 0x7f376414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769544700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f37640b2710 0x7f376414afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46954/0 (socket says 172.21.15.3:46954) 2022-07-21T01:39:11.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769544700 1 -- 172.21.15.3:0/1960548894 learned_addr learned my addr 172.21.15.3:0/1960548894 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:11.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769544700 1 -- 172.21.15.3:0/1960548894 >> v2:172.21.15.143:3300/0 conn(0x7f37640a8c50 msgr2=0x7f376414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:11.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769d45700 1 --2- 172.21.15.3:0/1960548894 >> v2:172.21.15.143:3300/0 conn(0x7f37640a8c50 0x7f376414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:11.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769544700 1 --2- 172.21.15.3:0/1960548894 >> v2:172.21.15.143:3300/0 conn(0x7f37640a8c50 0x7f376414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769544700 1 -- 172.21.15.3:0/1960548894 >> v1:172.21.15.3:6789/0 conn(0x7f37640b01a0 legacy=0x7f37640a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769544700 1 -- 172.21.15.3:0/1960548894 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37640a79c0 con 0x7f37640b2710 2022-07-21T01:39:11.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769d45700 1 --2- 172.21.15.3:0/1960548894 >> v2:172.21.15.143:3300/0 conn(0x7f37640a8c50 0x7f376414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:39:11.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3769544700 1 --2- 172.21.15.3:0/1960548894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b2710 0x7f376414afd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f37540032f0 tx=0x7f3754009290).ready entity=mon.0 client_cookie=168e290d441a1f31 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:11.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3763fff700 1 -- 172.21.15.3:0/1960548894 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f375400b870 con 0x7f37640b2710 2022-07-21T01:39:11.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3763fff700 1 -- 172.21.15.3:0/1960548894 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375400b9d0 con 0x7f37640b2710 2022-07-21T01:39:11.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f3763fff700 1 -- 172.21.15.3:0/1960548894 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f375400bb40 con 0x7f37640b2710 2022-07-21T01:39:11.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 -- 172.21.15.3:0/1960548894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b2710 msgr2=0x7f376414afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:11.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 --2- 172.21.15.3:0/1960548894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b2710 0x7f376414afd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f37540032f0 tx=0x7f3754009290).stop 2022-07-21T01:39:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 -- 172.21.15.3:0/1960548894 shutdown_connections 2022-07-21T01:39:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 --2- 172.21.15.3:0/1960548894 >> v2:172.21.15.143:3300/0 conn(0x7f37640a8c50 0x7f376414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 --2- 172.21.15.3:0/1960548894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b2710 0x7f376414afd0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:11.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 -- 172.21.15.3:0/1960548894 >> 172.21.15.3:0/1960548894 conn(0x7f3764060310 msgr2=0x7f37640b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:11.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 -- 172.21.15.3:0/1960548894 shutdown_connections 2022-07-21T01:39:11.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.985+0000 7f376a546700 1 -- 172.21.15.3:0/1960548894 wait complete. 2022-07-21T01:39:11.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f376a546700 1 Processor -- start 2022-07-21T01:39:11.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f376a546700 1 -- start start 2022-07-21T01:39:12.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f376a546700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37640a8c50 0x7f3764146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f376a546700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b01a0 0x7f37640aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f376a546700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37640b2710 0x7f37640aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f376a546700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640a9510 con 0x7f37640b01a0 2022-07-21T01:39:12.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f376a546700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640aaaf0 con 0x7f37640a8c50 2022-07-21T01:39:12.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f376a546700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640b3a80 con 0x7f37640b2710 2022-07-21T01:39:12.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f3768d43700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b01a0 0x7f37640aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f3768d43700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b01a0 0x7f37640aa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46962/0 (socket says 172.21.15.3:46962) 2022-07-21T01:39:12.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f3768d43700 1 -- 172.21.15.3:0/1008376881 learned_addr learned my addr 172.21.15.3:0/1008376881 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:12.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f3769544700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37640a8c50 0x7f3764146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.989+0000 7f3769d45700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37640b2710 0x7f37640aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f3768d43700 1 -- 172.21.15.3:0/1008376881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37640b2710 msgr2=0x7f37640aaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f3768d43700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37640b2710 0x7f37640aaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f3768d43700 1 -- 172.21.15.3:0/1008376881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37640a8c50 msgr2=0x7f3764146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f3768d43700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37640a8c50 0x7f3764146cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f3768d43700 1 -- 172.21.15.3:0/1008376881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3754009530 con 0x7f37640b01a0 2022-07-21T01:39:12.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f3768d43700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b01a0 0x7f37640aa090 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f376406b950 tx=0x7f3758007bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:12.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f37627fc700 1 -- 172.21.15.3:0/1008376881 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3758007030 con 0x7f37640b01a0 2022-07-21T01:39:12.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f37627fc700 1 -- 172.21.15.3:0/1008376881 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3758007190 con 0x7f37640b01a0 2022-07-21T01:39:12.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37640b3d20 con 0x7f37640b01a0 2022-07-21T01:39:12.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f37627fc700 1 -- 172.21.15.3:0/1008376881 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3758006510 con 0x7f37640b01a0 2022-07-21T01:39:12.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37640b3fc0 con 0x7f37640b01a0 2022-07-21T01:39:12.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37640574e0 con 0x7f37640b01a0 2022-07-21T01:39:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f37627fc700 1 -- 172.21.15.3:0/1008376881 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3758010030 con 0x7f37640b01a0 2022-07-21T01:39:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f37627fc700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3744053a70 0x7f3744055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f37627fc700 1 -- 172.21.15.3:0/1008376881 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f375806bb90 con 0x7f37640b01a0 2022-07-21T01:39:12.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.993+0000 7f3769544700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3744053a70 0x7f3744055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:11.997+0000 7f3769544700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3744053a70 0x7f3744055f30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f375401b000 tx=0x7f3754016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.005+0000 7f37627fc700 1 -- 172.21.15.3:0/1008376881 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3758042950 con 0x7f37640b01a0 2022-07-21T01:39:12.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.165+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37640aa8c0 con 0x7f37640b01a0 2022-07-21T01:39:12.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.165+0000 7f37627fc700 1 -- 172.21.15.3:0/1008376881 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2487 (secure 0 0 0) 0x7f3758045f70 con 0x7f37640b01a0 2022-07-21T01:39:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:12.164 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:39:09.622528+0000","modified":"2022-07-21T01:39:10.626387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6259":{"gid":6259,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/862725488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":862725488},{"type":"v1","addr":"172.21.15.143:6833","nonce":862725488}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-21T01:39:12.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3744053a70 msgr2=0x7f3744055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3744053a70 0x7f3744055f30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f375401b000 tx=0x7f3754016000).stop 2022-07-21T01:39:12.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b01a0 msgr2=0x7f37640aa090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b01a0 0x7f37640aa090 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f376406b950 tx=0x7f3758007bb0).stop 2022-07-21T01:39:12.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 shutdown_connections 2022-07-21T01:39:12.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37640b2710 0x7f37640aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37640b01a0 0x7f37640aa090 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3744053a70 0x7f3744055f30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 --2- 172.21.15.3:0/1008376881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37640a8c50 0x7f3764146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 >> 172.21.15.3:0/1008376881 conn(0x7f3764060310 msgr2=0x7f37641457a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:12.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 shutdown_connections 2022-07-21T01:39:12.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.169+0000 7f376a546700 1 -- 172.21.15.3:0/1008376881 wait complete. 2022-07-21T01:39:12.170 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2022-07-21T01:39:12.486 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:39:09.622528+0000', 'modified': '2022-07-21T01:39:10.626387+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6259': {'gid': 6259, 'name': 'b', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6833/862725488', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 862725488}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 862725488}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:39:12.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:39:12.488 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-21T01:39:12.601 INFO:teuthology.orchestra.run.smithi143.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6259,"want_state":"up:active","state":"up:active","rank_uptime":2.9691155170000001,"mdsmap_epoch":35,"osdmap_epoch":79,"osdmap_epoch_barrier":78,"uptime":8.021175135} 2022-07-21T01:39:12.602 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6259, "mdsmap_epoch": 35, "osdmap_epoch": 79, "osdmap_epoch_barrier": 78, "rank_uptime": 2.969115517, "state": "up:active", "uptime": 8.021175135, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:39:12.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:39:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 Processor -- start 2022-07-21T01:39:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 -- start start 2022-07-21T01:39:12.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde54148530 0x7fde540b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540a7250 0x7fde541448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde540b04e0 con 0x7fde540b1f40 2022-07-21T01:39:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde540a9250 con 0x7fde54148530 2022-07-21T01:39:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde540b0780 con 0x7fde540a7250 2022-07-21T01:39:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5883c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fde540b1f40 0x7fde540b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55248/0 (socket says 172.21.15.3:55248) 2022-07-21T01:39:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5883c700 1 -- 172.21.15.3:0/211789060 learned_addr learned my addr 172.21.15.3:0/211789060 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde53fff700 1 --2- 172.21.15.3:0/211789060 >> v2:172.21.15.143:3300/0 conn(0x7fde54148530 0x7fde540b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5903d700 1 --2- 172.21.15.3:0/211789060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540a7250 0x7fde541448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde537fe700 1 -- 172.21.15.3:0/211789060 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961174437 0 0) 0x7fde540b04e0 con 0x7fde540b1f40 2022-07-21T01:39:12.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde537fe700 1 -- 172.21.15.3:0/211789060 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde38003260 con 0x7fde540b1f40 2022-07-21T01:39:12.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde53fff700 1 -- 172.21.15.3:0/211789060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540a7250 msgr2=0x7fde541448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:12.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde53fff700 1 --2- 172.21.15.3:0/211789060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540a7250 0x7fde541448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde53fff700 1 -- 172.21.15.3:0/211789060 >> v1:172.21.15.3:6789/0 conn(0x7fde540b1f40 legacy=0x7fde540b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde537fe700 1 -- 172.21.15.3:0/211789060 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 516267416 0 0) 0x7fde38003260 con 0x7fde540b1f40 2022-07-21T01:39:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde53fff700 1 -- 172.21.15.3:0/211789060 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde540b0a20 con 0x7fde54148530 2022-07-21T01:39:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde53fff700 1 --2- 172.21.15.3:0/211789060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde54148530 0x7fde540b1160 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fde541478c0 tx=0x7fde440091d0).ready entity=mon.1 client_cookie=d00ee319c287596c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde537fe700 1 -- 172.21.15.3:0/211789060 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde4400b600 con 0x7fde54148530 2022-07-21T01:39:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde537fe700 1 -- 172.21.15.3:0/211789060 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4400b760 con 0x7fde54148530 2022-07-21T01:39:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde537fe700 1 -- 172.21.15.3:0/211789060 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde4400b8d0 con 0x7fde54148530 2022-07-21T01:39:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 -- 172.21.15.3:0/211789060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde54148530 msgr2=0x7fde540b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 --2- 172.21.15.3:0/211789060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde54148530 0x7fde540b1160 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fde541478c0 tx=0x7fde440091d0).stop 2022-07-21T01:39:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 -- 172.21.15.3:0/211789060 shutdown_connections 2022-07-21T01:39:12.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 --2- 172.21.15.3:0/211789060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540a7250 0x7fde541448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 --2- 172.21.15.3:0/211789060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde54148530 0x7fde540b1160 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.717+0000 7fde5aaa0700 1 -- 172.21.15.3:0/211789060 >> 172.21.15.3:0/211789060 conn(0x7fde5405f840 msgr2=0x7fde5405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:12.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 -- 172.21.15.3:0/211789060 shutdown_connections 2022-07-21T01:39:12.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 -- 172.21.15.3:0/211789060 wait complete. 2022-07-21T01:39:12.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 Processor -- start 2022-07-21T01:39:12.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 -- start start 2022-07-21T01:39:12.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540a7250 0x7fde540aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde540b1f40 0x7fde5414b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde54148530 0x7fde5414b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde54149e00 con 0x7fde540b1f40 2022-07-21T01:39:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde540a9930 con 0x7fde540a7250 2022-07-21T01:39:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5aaa0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5414a030 con 0x7fde54148530 2022-07-21T01:39:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde53fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde540b1f40 0x7fde5414b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5883c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540a7250 0x7fde540aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde53fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde540b1f40 0x7fde5414b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46976/0 (socket says 172.21.15.3:46976) 2022-07-21T01:39:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde53fff700 1 -- 172.21.15.3:0/2449541434 learned_addr learned my addr 172.21.15.3:0/2449541434 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.721+0000 7fde5903d700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde54148530 0x7fde5414b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde53fff700 1 -- 172.21.15.3:0/2449541434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde54148530 msgr2=0x7fde5414b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde53fff700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde54148530 0x7fde5414b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde53fff700 1 -- 172.21.15.3:0/2449541434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540a7250 msgr2=0x7fde540aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde53fff700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540a7250 0x7fde540aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde53fff700 1 -- 172.21.15.3:0/2449541434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde44009470 con 0x7fde540b1f40 2022-07-21T01:39:12.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde5883c700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540a7250 0x7fde540aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:12.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde53fff700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde540b1f40 0x7fde5414b4a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fde541478c0 tx=0x7fde44016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:12.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde51ffb700 1 -- 172.21.15.3:0/2449541434 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde4400b8d0 con 0x7fde540b1f40 2022-07-21T01:39:12.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde51ffb700 1 -- 172.21.15.3:0/2449541434 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde440032b0 con 0x7fde540b1f40 2022-07-21T01:39:12.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde540abbb0 con 0x7fde540b1f40 2022-07-21T01:39:12.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde51ffb700 1 -- 172.21.15.3:0/2449541434 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde4400b600 con 0x7fde540b1f40 2022-07-21T01:39:12.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde540a95d0 con 0x7fde540b1f40 2022-07-21T01:39:12.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5414b290 con 0x7fde540b1f40 2022-07-21T01:39:12.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.725+0000 7fde51ffb700 1 -- 172.21.15.3:0/2449541434 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fde44026030 con 0x7fde540b1f40 2022-07-21T01:39:12.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.729+0000 7fde51ffb700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde2c053a70 0x7fde2c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:12.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.729+0000 7fde51ffb700 1 -- 172.21.15.3:0/2449541434 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fde4406cae0 con 0x7fde540b1f40 2022-07-21T01:39:12.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.729+0000 7fde5883c700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde2c053a70 0x7fde2c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:12.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.729+0000 7fde5883c700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde2c053a70 0x7fde2c055f30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fde3c00d000 tx=0x7fde3c001740).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:12.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.733+0000 7fde51ffb700 1 -- 172.21.15.3:0/2449541434 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fde44043790 con 0x7fde540b1f40 2022-07-21T01:39:12.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.861+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde540574e0 con 0x7fde540b1f40 2022-07-21T01:39:12.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde51ffb700 1 -- 172.21.15.3:0/2449541434 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16274 (secure 0 0 0) 0x7fde44024090 con 0x7fde540b1f40 2022-07-21T01:39:12.863 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:12.863 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":79,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:39:10.626495+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:39:07.362717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-21T01:39:08.613529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:39:12.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde2c053a70 msgr2=0x7fde2c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde2c053a70 0x7fde2c055f30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fde3c00d000 tx=0x7fde3c001740).stop 2022-07-21T01:39:12.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde540b1f40 msgr2=0x7fde5414b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:12.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde540b1f40 0x7fde5414b4a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fde541478c0 tx=0x7fde44016000).stop 2022-07-21T01:39:12.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 shutdown_connections 2022-07-21T01:39:12.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde54148530 0x7fde5414b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde540b1f40 0x7fde5414b4a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde2c053a70 0x7fde2c055f30 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 --2- 172.21.15.3:0/2449541434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540a7250 0x7fde540aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:12.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 >> 172.21.15.3:0/2449541434 conn(0x7fde5405f840 msgr2=0x7fde540b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:12.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 shutdown_connections 2022-07-21T01:39:12.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:12.865+0000 7fde5aaa0700 1 -- 172.21.15.3:0/2449541434 wait complete. 2022-07-21T01:39:12.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:39:07.362717+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:39:08.613529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:39:12.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:39:13.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff446dd9700 1 Processor -- start 2022-07-21T01:39:13.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff446dd9700 1 -- start start 2022-07-21T01:39:13.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff446dd9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 0x7ff4400affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff446dd9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4400ad9a0 con 0x7ff4400b5850 2022-07-21T01:39:13.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff446dd9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4400adc40 con 0x7ff4400b2b50 2022-07-21T01:39:13.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff446dd9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400adee0 con 0x7ff4400b1770 2022-07-21T01:39:13.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff445376700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff4400b5850 0x7ff440143630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55266/0 (socket says 172.21.15.3:55266) 2022-07-21T01:39:13.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff445376700 1 -- 172.21.15.3:0/3517881724 learned_addr learned my addr 172.21.15.3:0/3517881724 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:13.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff43ffff700 1 --2- 172.21.15.3:0/3517881724 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 0x7ff4400affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff43f7fe700 1 -- 172.21.15.3:0/3517881724 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587196495 0 0) 0x7ff4400ad9a0 con 0x7ff4400b5850 2022-07-21T01:39:13.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff43f7fe700 1 -- 172.21.15.3:0/3517881724 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff420003260 con 0x7ff4400b5850 2022-07-21T01:39:13.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff43f7fe700 1 -- 172.21.15.3:0/3517881724 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112547549 0 0) 0x7ff4400adc40 con 0x7ff4400b2b50 2022-07-21T01:39:13.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff43ffff700 1 -- 172.21.15.3:0/3517881724 >> v1:172.21.15.143:6789/0 conn(0x7ff4400b2b50 legacy=0x7ff4400af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff43ffff700 1 -- 172.21.15.3:0/3517881724 >> v1:172.21.15.3:6789/0 conn(0x7ff4400b5850 legacy=0x7ff440143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.081+0000 7ff43ffff700 1 -- 172.21.15.3:0/3517881724 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400ae180 con 0x7ff4400b1770 2022-07-21T01:39:13.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff43ffff700 1 --2- 172.21.15.3:0/3517881724 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 0x7ff4400affc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff428002f70 tx=0x7ff428006df0).ready entity=mon.2 client_cookie=51579a51d74bebbd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff43f7fe700 1 -- 172.21.15.3:0/3517881724 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff42800d570 con 0x7ff4400b1770 2022-07-21T01:39:13.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff43f7fe700 1 -- 172.21.15.3:0/3517881724 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff428000e90 con 0x7ff4400b1770 2022-07-21T01:39:13.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff43f7fe700 1 -- 172.21.15.3:0/3517881724 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff42800d8c0 con 0x7ff4400b1770 2022-07-21T01:39:13.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff446dd9700 1 -- 172.21.15.3:0/3517881724 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 msgr2=0x7ff4400affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff446dd9700 1 --2- 172.21.15.3:0/3517881724 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 0x7ff4400affc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff428002f70 tx=0x7ff428006df0).stop 2022-07-21T01:39:13.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff446dd9700 1 -- 172.21.15.3:0/3517881724 shutdown_connections 2022-07-21T01:39:13.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff446dd9700 1 --2- 172.21.15.3:0/3517881724 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 0x7ff4400affc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff446dd9700 1 -- 172.21.15.3:0/3517881724 >> 172.21.15.3:0/3517881724 conn(0x7ff44006a2a0 msgr2=0x7ff44005ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:13.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff446dd9700 1 -- 172.21.15.3:0/3517881724 shutdown_connections 2022-07-21T01:39:13.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.085+0000 7ff446dd9700 1 -- 172.21.15.3:0/3517881724 wait complete. 2022-07-21T01:39:13.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 Processor -- start 2022-07-21T01:39:13.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 -- start start 2022-07-21T01:39:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 0x7ff4401528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b2b50 0x7ff4401530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b5850 0x7ff440141af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44013c4e0 con 0x7ff4400b2b50 2022-07-21T01:39:13.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff440142e40 con 0x7ff4400b5850 2022-07-21T01:39:13.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4401413f0 con 0x7ff4400b1770 2022-07-21T01:39:13.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b2b50 0x7ff4401530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b2b50 0x7ff4401530c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46990/0 (socket says 172.21.15.3:46990) 2022-07-21T01:39:13.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43ffff700 1 -- 172.21.15.3:0/2663358599 learned_addr learned my addr 172.21.15.3:0/2663358599 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:13.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff444b75700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 0x7ff4401528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff445376700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b5850 0x7ff440141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43ffff700 1 -- 172.21.15.3:0/2663358599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 msgr2=0x7ff4401528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43ffff700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 0x7ff4401528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43ffff700 1 -- 172.21.15.3:0/2663358599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b5850 msgr2=0x7ff440141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43ffff700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b5850 0x7ff440141af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43ffff700 1 -- 172.21.15.3:0/2663358599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42800b180 con 0x7ff4400b2b50 2022-07-21T01:39:13.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff445376700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b5850 0x7ff440141af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:13.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43ffff700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b2b50 0x7ff4401530c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7ff428002f70 tx=0x7ff428014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43dffb700 1 -- 172.21.15.3:0/2663358599 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff428013e50 con 0x7ff4400b2b50 2022-07-21T01:39:13.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43dffb700 1 -- 172.21.15.3:0/2663358599 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42800de40 con 0x7ff4400b2b50 2022-07-21T01:39:13.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4400ad9e0 con 0x7ff4400b2b50 2022-07-21T01:39:13.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff43dffb700 1 -- 172.21.15.3:0/2663358599 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff42801e5f0 con 0x7ff4400b2b50 2022-07-21T01:39:13.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.089+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff440141650 con 0x7ff4400b2b50 2022-07-21T01:39:13.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.093+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff440141850 con 0x7ff4400b2b50 2022-07-21T01:39:13.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.093+0000 7ff43dffb700 1 -- 172.21.15.3:0/2663358599 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff428025030 con 0x7ff4400b2b50 2022-07-21T01:39:13.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.093+0000 7ff43dffb700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff424053980 0x7ff424055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.093+0000 7ff43dffb700 1 -- 172.21.15.3:0/2663358599 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff42806cd90 con 0x7ff4400b2b50 2022-07-21T01:39:13.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.093+0000 7ff444b75700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff424053980 0x7ff424055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.093+0000 7ff444b75700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff424053980 0x7ff424055e40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff43000d000 tx=0x7ff430007d80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.101+0000 7ff43dffb700 1 -- 172.21.15.3:0/2663358599 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff440141850 con 0x7ff4400b2b50 2022-07-21T01:39:13.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.225+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff440141850 con 0x7ff4400b2b50 2022-07-21T01:39:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff43dffb700 1 -- 172.21.15.3:0/2663358599 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v79) v1 ==== 110+0+10 (secure 0 0 0) 0x7ff440141850 con 0x7ff4400b2b50 2022-07-21T01:39:13.227 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:39:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff424053980 msgr2=0x7ff424055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff424053980 0x7ff424055e40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff43000d000 tx=0x7ff430007d80).stop 2022-07-21T01:39:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b2b50 msgr2=0x7ff4401530c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b2b50 0x7ff4401530c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7ff428002f70 tx=0x7ff428014000).stop 2022-07-21T01:39:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 shutdown_connections 2022-07-21T01:39:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff424053980 0x7ff424055e40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4400b5850 0x7ff440141af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4400b2b50 0x7ff4401530c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 --2- 172.21.15.3:0/2663358599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4400b1770 0x7ff4401528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 >> 172.21.15.3:0/2663358599 conn(0x7ff44006a2a0 msgr2=0x7ff440060c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 shutdown_connections 2022-07-21T01:39:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.229+0000 7ff446dd9700 1 -- 172.21.15.3:0/2663358599 wait complete. 2022-07-21T01:39:13.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:39:13.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.345+0000 7f3140c08700 1 Processor -- start 2022-07-21T01:39:13.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.345+0000 7f3140c08700 1 -- start start 2022-07-21T01:39:13.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f3140c08700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c146ae0 0x7f313c1403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f3140c08700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313c143cf0 con 0x7f313c0b4820 2022-07-21T01:39:13.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f3140c08700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313c143f90 con 0x7f313c0b4310 2022-07-21T01:39:13.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f3140c08700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c1476a0 con 0x7f313c146ae0 2022-07-21T01:39:13.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313ad9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f313c0b4820 0x7f313c0b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55278/0 (socket says 172.21.15.3:55278) 2022-07-21T01:39:13.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313ad9d700 1 -- 172.21.15.3:0/2885566958 learned_addr learned my addr 172.21.15.3:0/2885566958 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:13.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313a59c700 1 --2- 172.21.15.3:0/2885566958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c146ae0 0x7f313c1403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313959a700 1 -- 172.21.15.3:0/2885566958 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986665531 0 0) 0x7f313c143cf0 con 0x7f313c0b4820 2022-07-21T01:39:13.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313959a700 1 -- 172.21.15.3:0/2885566958 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3120003260 con 0x7f313c0b4820 2022-07-21T01:39:13.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313959a700 1 -- 172.21.15.3:0/2885566958 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666849652 0 0) 0x7f313c143f90 con 0x7f313c0b4310 2022-07-21T01:39:13.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313959a700 1 -- 172.21.15.3:0/2885566958 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f313c143cf0 con 0x7f313c0b4310 2022-07-21T01:39:13.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313a59c700 1 -- 172.21.15.3:0/2885566958 >> v1:172.21.15.143:6789/0 conn(0x7f313c0b4310 legacy=0x7f313c1408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313a59c700 1 -- 172.21.15.3:0/2885566958 >> v1:172.21.15.3:6789/0 conn(0x7f313c0b4820 legacy=0x7f313c0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313959a700 1 -- 172.21.15.3:0/2885566958 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095850523 0 0) 0x7f3120003260 con 0x7f313c0b4820 2022-07-21T01:39:13.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313a59c700 1 -- 172.21.15.3:0/2885566958 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c0b0e20 con 0x7f313c146ae0 2022-07-21T01:39:13.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313a59c700 1 --2- 172.21.15.3:0/2885566958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c146ae0 0x7f313c1403c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3134001490 tx=0x7f312c006df0).ready entity=mon.2 client_cookie=701aaec80f28990c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313959a700 1 -- 172.21.15.3:0/2885566958 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31340019e0 con 0x7f313c146ae0 2022-07-21T01:39:13.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313959a700 1 -- 172.21.15.3:0/2885566958 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c000e90 con 0x7f313c146ae0 2022-07-21T01:39:13.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.349+0000 7f313959a700 1 -- 172.21.15.3:0/2885566958 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f312c012460 con 0x7f313c146ae0 2022-07-21T01:39:13.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 -- 172.21.15.3:0/2885566958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c146ae0 msgr2=0x7f313c1403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 --2- 172.21.15.3:0/2885566958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c146ae0 0x7f313c1403c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3134001490 tx=0x7f312c006df0).stop 2022-07-21T01:39:13.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 -- 172.21.15.3:0/2885566958 shutdown_connections 2022-07-21T01:39:13.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 --2- 172.21.15.3:0/2885566958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c146ae0 0x7f313c1403c0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0x7f3134001490 tx=0x7f312c006df0).stop 2022-07-21T01:39:13.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 -- 172.21.15.3:0/2885566958 >> 172.21.15.3:0/2885566958 conn(0x7f313c05c620 msgr2=0x7f313c05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:13.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 -- 172.21.15.3:0/2885566958 shutdown_connections 2022-07-21T01:39:13.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 -- 172.21.15.3:0/2885566958 wait complete. 2022-07-21T01:39:13.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 Processor -- start 2022-07-21T01:39:13.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 -- start start 2022-07-21T01:39:13.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c0b4310 0x7f313c112b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f313c0b4820 0x7f313c1450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313c156ae0 0x7f313c1530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c152a60 con 0x7f313c0b4820 2022-07-21T01:39:13.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c0afcb0 con 0x7f313c156ae0 2022-07-21T01:39:13.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3140c08700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313c0afee0 con 0x7f313c0b4310 2022-07-21T01:39:13.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3139d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f313c0b4820 0x7f313c1450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3139d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f313c0b4820 0x7f313c1450b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47002/0 (socket says 172.21.15.3:47002) 2022-07-21T01:39:13.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f3139d9b700 1 -- 172.21.15.3:0/1033583836 learned_addr learned my addr 172.21.15.3:0/1033583836 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:13.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f313ad9d700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313c156ae0 0x7f313c1530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.353+0000 7f313a59c700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c0b4310 0x7f313c112b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f3139d9b700 1 -- 172.21.15.3:0/1033583836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c0b4310 msgr2=0x7f313c112b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f3139d9b700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c0b4310 0x7f313c112b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f3139d9b700 1 -- 172.21.15.3:0/1033583836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313c156ae0 msgr2=0x7f313c1530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f3139d9b700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313c156ae0 0x7f313c1530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f3139d9b700 1 -- 172.21.15.3:0/1033583836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c00b180 con 0x7f313c0b4820 2022-07-21T01:39:13.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f313ad9d700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313c156ae0 0x7f313c1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:13.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f3139d9b700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f313c0b4820 0x7f313c1450b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f3130001110 tx=0x7f3130007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f312b7fe700 1 -- 172.21.15.3:0/1033583836 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3130012400 con 0x7f313c0b4820 2022-07-21T01:39:13.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f312b7fe700 1 -- 172.21.15.3:0/1033583836 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3130012560 con 0x7f313c0b4820 2022-07-21T01:39:13.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313c0b0180 con 0x7f313c0b4820 2022-07-21T01:39:13.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f312b7fe700 1 -- 172.21.15.3:0/1033583836 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31300128c0 con 0x7f313c0b4820 2022-07-21T01:39:13.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313c113290 con 0x7f313c0b4820 2022-07-21T01:39:13.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313c152e50 con 0x7f313c0b4820 2022-07-21T01:39:13.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.357+0000 7f312b7fe700 1 -- 172.21.15.3:0/1033583836 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3130019030 con 0x7f313c0b4820 2022-07-21T01:39:13.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.361+0000 7f312b7fe700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3118053a70 0x7f3118055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.361+0000 7f312b7fe700 1 -- 172.21.15.3:0/1033583836 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f313006b530 con 0x7f313c0b4820 2022-07-21T01:39:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.361+0000 7f313a59c700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3118053a70 0x7f3118055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.361+0000 7f313a59c700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3118053a70 0x7f3118055f30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f312c01a000 tx=0x7f312c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.365+0000 7f312b7fe700 1 -- 172.21.15.3:0/1033583836 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f313c152e50 con 0x7f313c0b4820 2022-07-21T01:39:13.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.493+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f313c152e50 con 0x7f313c0b4820 2022-07-21T01:39:13.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.493+0000 7f312b7fe700 1 -- 172.21.15.3:0/1033583836 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v79) v1 ==== 92+0+10 (secure 0 0 0) 0x7f313c152e50 con 0x7f313c0b4820 2022-07-21T01:39:13.494 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:39:13.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3118053a70 msgr2=0x7f3118055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3118053a70 0x7f3118055f30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f312c01a000 tx=0x7f312c014000).stop 2022-07-21T01:39:13.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f313c0b4820 msgr2=0x7f313c1450b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f313c0b4820 0x7f313c1450b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f3130001110 tx=0x7f3130007ab0).stop 2022-07-21T01:39:13.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 shutdown_connections 2022-07-21T01:39:13.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3118053a70 0x7f3118055f30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313c156ae0 0x7f313c1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f313c0b4820 0x7f313c1450b0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 --2- 172.21.15.3:0/1033583836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313c0b4310 0x7f313c112b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 >> 172.21.15.3:0/1033583836 conn(0x7f313c05c620 msgr2=0x7f313c0b3570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:13.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 shutdown_connections 2022-07-21T01:39:13.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.497+0000 7f3140c08700 1 -- 172.21.15.3:0/1033583836 wait complete. 2022-07-21T01:39:13.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:39:13.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fecab700 1 Processor -- start 2022-07-21T01:39:13.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fecab700 1 -- start start 2022-07-21T01:39:13.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fecab700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f27f80b13c0 0x7f27f80b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fecab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f80afd80 0x7f27f8142910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fecab700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f80ac520 con 0x7f27f8142e00 2022-07-21T01:39:13.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fecab700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f80b17a0 con 0x7f27f80b13c0 2022-07-21T01:39:13.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fecab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f80b0450 con 0x7f27f80afd80 2022-07-21T01:39:13.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fdca9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f27f80b13c0 0x7f27f80b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fdca9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f27f80b13c0 0x7f27f80b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47010/0 (socket says 172.21.15.3:47010) 2022-07-21T01:39:13.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fdca9700 1 -- 172.21.15.3:0/3531423053 learned_addr learned my addr 172.21.15.3:0/3531423053 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:13.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.621+0000 7f27fd4a8700 1 --2- 172.21.15.3:0/3531423053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f80afd80 0x7f27f8142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fdca9700 1 -- 172.21.15.3:0/3531423053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f80afd80 msgr2=0x7f27f8142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:13.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fdca9700 1 --2- 172.21.15.3:0/3531423053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f80afd80 0x7f27f8142910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fdca9700 1 -- 172.21.15.3:0/3531423053 >> v1:172.21.15.143:6789/0 conn(0x7f27f8142e00 legacy=0x7f27f80ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fcca7700 1 -- 172.21.15.3:0/3531423053 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288010859 0 0) 0x7f27f80ac520 con 0x7f27f8142e00 2022-07-21T01:39:13.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fdca9700 1 -- 172.21.15.3:0/3531423053 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f80b06f0 con 0x7f27f80b13c0 2022-07-21T01:39:13.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fdca9700 1 --2- 172.21.15.3:0/3531423053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 0x7f27f80b0b20 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f27e8002f70 tx=0x7f27e8006eb0).ready entity=mon.0 client_cookie=4badb37b430f26c7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fcca7700 1 -- 172.21.15.3:0/3531423053 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27e800d570 con 0x7f27f80b13c0 2022-07-21T01:39:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fcca7700 1 -- 172.21.15.3:0/3531423053 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e8000e90 con 0x7f27f80b13c0 2022-07-21T01:39:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fcca7700 1 -- 172.21.15.3:0/3531423053 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27e800d8c0 con 0x7f27f80b13c0 2022-07-21T01:39:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 -- 172.21.15.3:0/3531423053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 msgr2=0x7f27f80b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 --2- 172.21.15.3:0/3531423053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 0x7f27f80b0b20 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f27e8002f70 tx=0x7f27e8006eb0).stop 2022-07-21T01:39:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 -- 172.21.15.3:0/3531423053 shutdown_connections 2022-07-21T01:39:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 --2- 172.21.15.3:0/3531423053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f80afd80 0x7f27f8142910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 --2- 172.21.15.3:0/3531423053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 0x7f27f80b0b20 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 -- 172.21.15.3:0/3531423053 >> 172.21.15.3:0/3531423053 conn(0x7f27f80626f0 msgr2=0x7f27f8062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 -- 172.21.15.3:0/3531423053 shutdown_connections 2022-07-21T01:39:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 -- 172.21.15.3:0/3531423053 wait complete. 2022-07-21T01:39:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 Processor -- start 2022-07-21T01:39:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.625+0000 7f27fecab700 1 -- start start 2022-07-21T01:39:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fecab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f80afd80 0x7f27f80aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fecab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 0x7f27f8156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fecab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f8142e00 0x7f27f8152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fecab700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f8156b50 con 0x7f27f80b13c0 2022-07-21T01:39:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fecab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f813bee0 con 0x7f27f80afd80 2022-07-21T01:39:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fecab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f813c180 con 0x7f27f8142e00 2022-07-21T01:39:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fd4a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 0x7f27f8156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fd4a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 0x7f27f8156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47016/0 (socket says 172.21.15.3:47016) 2022-07-21T01:39:13.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fd4a8700 1 -- 172.21.15.3:0/1035423716 learned_addr learned my addr 172.21.15.3:0/1035423716 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:13.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fdca9700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f80afd80 0x7f27f80aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fe4aa700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f8142e00 0x7f27f8152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fd4a8700 1 -- 172.21.15.3:0/1035423716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f8142e00 msgr2=0x7f27f8152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fd4a8700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f8142e00 0x7f27f8152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fd4a8700 1 -- 172.21.15.3:0/1035423716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f80afd80 msgr2=0x7f27f80aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fd4a8700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f80afd80 0x7f27f80aead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fd4a8700 1 -- 172.21.15.3:0/1035423716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e800b1e0 con 0x7f27f80b13c0 2022-07-21T01:39:13.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fd4a8700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 0x7f27f8156660 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7f27f400c000 tx=0x7f27f4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27e6ffd700 1 -- 172.21.15.3:0/1035423716 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27f4010780 con 0x7f27f80b13c0 2022-07-21T01:39:13.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27e6ffd700 1 -- 172.21.15.3:0/1035423716 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f40108e0 con 0x7f27f80b13c0 2022-07-21T01:39:13.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f813c420 con 0x7f27f80b13c0 2022-07-21T01:39:13.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27e6ffd700 1 -- 172.21.15.3:0/1035423716 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27f4010c40 con 0x7f27f80b13c0 2022-07-21T01:39:13.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f8152680 con 0x7f27f80b13c0 2022-07-21T01:39:13.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f80574e0 con 0x7f27f80b13c0 2022-07-21T01:39:13.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.629+0000 7f27e6ffd700 1 -- 172.21.15.3:0/1035423716 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f27f401c460 con 0x7f27f80b13c0 2022-07-21T01:39:13.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.633+0000 7f27e6ffd700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27d8053b00 0x7f27d8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.633+0000 7f27e6ffd700 1 -- 172.21.15.3:0/1035423716 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f27f406c3c0 con 0x7f27f80b13c0 2022-07-21T01:39:13.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.633+0000 7f27fdca9700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27d8053b00 0x7f27d8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.633+0000 7f27fdca9700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27d8053b00 0x7f27d8055fc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f27e801c000 tx=0x7f27e8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.641+0000 7f27e6ffd700 1 -- 172.21.15.3:0/1035423716 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f27f4043120 con 0x7f27f80b13c0 2022-07-21T01:39:13.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f27f8061020 con 0x7f27f80b13c0 2022-07-21T01:39:13.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27e6ffd700 1 -- 172.21.15.3:0/1035423716 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) v1 ==== 104+0+10 (secure 0 0 0) 0x7f27f4046740 con 0x7f27f80b13c0 2022-07-21T01:39:13.763 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:39:13.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27d8053b00 msgr2=0x7f27d8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27d8053b00 0x7f27d8055fc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f27e801c000 tx=0x7f27e8015000).stop 2022-07-21T01:39:13.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 msgr2=0x7f27f8156660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 0x7f27f8156660 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7f27f400c000 tx=0x7f27f4006df0).stop 2022-07-21T01:39:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 shutdown_connections 2022-07-21T01:39:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f8142e00 0x7f27f8152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27f80b13c0 0x7f27f8156660 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27d8053b00 0x7f27d8055fc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 --2- 172.21.15.3:0/1035423716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f80afd80 0x7f27f80aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 >> 172.21.15.3:0/1035423716 conn(0x7f27f80626f0 msgr2=0x7f27f81402c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 shutdown_connections 2022-07-21T01:39:13.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.765+0000 7f27fecab700 1 -- 172.21.15.3:0/1035423716 wait complete. 2022-07-21T01:39:13.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:39:13.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e7fff700 1 Processor -- start 2022-07-21T01:39:13.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e7fff700 1 -- start start 2022-07-21T01:39:13.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80614b0 0x7fc3e8061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e7fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc3e80ad750 0x7fc3e80b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e7fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80aee80 con 0x7fc3e81405d0 2022-07-21T01:39:13.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e7fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e80b70b0 con 0x7fc3e80ad750 2022-07-21T01:39:13.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80af0b0 con 0x7fc3e80614b0 2022-07-21T01:39:13.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e77fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc3e81405d0 0x7fc3e81466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55306/0 (socket says 172.21.15.3:55306) 2022-07-21T01:39:13.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e77fe700 1 -- 172.21.15.3:0/3547010714 learned_addr learned my addr 172.21.15.3:0/3547010714 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:13.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e67fc700 1 --2- 172.21.15.3:0/3547010714 >> v2:172.21.15.143:3300/0 conn(0x7fc3e80ad750 0x7fc3e80b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e6ffd700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80614b0 0x7fc3e8061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657258986 0 0) 0x7fc3e80aee80 con 0x7fc3e81405d0 2022-07-21T01:39:13.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.893+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3cc003260 con 0x7fc3e81405d0 2022-07-21T01:39:13.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 751024758 0 0) 0x7fc3cc003260 con 0x7fc3e81405d0 2022-07-21T01:39:13.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80614b0 msgr2=0x7fc3e8061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:13.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80614b0 0x7fc3e8061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 >> v2:172.21.15.143:3300/0 conn(0x7fc3e80ad750 msgr2=0x7fc3e80b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:13.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 --2- 172.21.15.3:0/3547010714 >> v2:172.21.15.143:3300/0 conn(0x7fc3e80ad750 0x7fc3e80b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80af350 con 0x7fc3e81405d0 2022-07-21T01:39:13.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fc3e00019e0 con 0x7fc3e81405d0 2022-07-21T01:39:13.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 >> v1:172.21.15.3:6789/0 conn(0x7fc3e81405d0 legacy=0x7fc3e81466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80614b0 0x7fc3cc003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3cc003c80 0x7fc3cc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3cc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80b70b0 con 0x7fc3cc0049f0 2022-07-21T01:39:13.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80af0b0 con 0x7fc3cc003c80 2022-07-21T01:39:13.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc00b750 con 0x7fc3e80614b0 2022-07-21T01:39:13.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e77fe700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3cc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e67fc700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3cc003c80 0x7fc3cc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e6ffd700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80614b0 0x7fc3cc003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e77fe700 1 -- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80614b0 msgr2=0x7fc3cc003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e77fe700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80614b0 0x7fc3cc003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e77fe700 1 -- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3cc003c80 msgr2=0x7fc3cc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e77fe700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3cc003c80 0x7fc3cc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e77fe700 1 -- 172.21.15.3:0/3547010714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc003260 con 0x7fc3cc0049f0 2022-07-21T01:39:13.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e77fe700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3cc004e70 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7fc3cc0054d0 tx=0x7fc3e80af4e0).ready entity=mon.0 client_cookie=9f0f391ac4a1c181 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3e000cd90 con 0x7fc3cc0049f0 2022-07-21T01:39:13.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e0005990 con 0x7fc3cc0049f0 2022-07-21T01:39:13.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e5ffb700 1 -- 172.21.15.3:0/3547010714 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3e0005cf0 con 0x7fc3cc0049f0 2022-07-21T01:39:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e7fff700 1 -- 172.21.15.3:0/3547010714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 msgr2=0x7fc3cc004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e7fff700 1 --2- 172.21.15.3:0/3547010714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3cc004e70 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7fc3cc0054d0 tx=0x7fc3e80af4e0).stop 2022-07-21T01:39:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e77fe700 1 -- 172.21.15.3:0/3547010714 reap_dead start 2022-07-21T01:39:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e7fff700 1 -- 172.21.15.3:0/3547010714 shutdown_connections 2022-07-21T01:39:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e7fff700 1 -- 172.21.15.3:0/3547010714 >> 172.21.15.3:0/3547010714 conn(0x7fc3e8066f70 msgr2=0x7fc3e8067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:13.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e7fff700 1 -- 172.21.15.3:0/3547010714 shutdown_connections 2022-07-21T01:39:13.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e7fff700 1 -- 172.21.15.3:0/3547010714 wait complete. 2022-07-21T01:39:13.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.897+0000 7fc3e7fff700 1 Processor -- start 2022-07-21T01:39:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 -- start start 2022-07-21T01:39:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3e81459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e8156050 0x7fc3e81564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80b7390 0x7fc3e80b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e81569c0 con 0x7fc3cc0049f0 2022-07-21T01:39:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80b92f0 con 0x7fc3e8156050 2022-07-21T01:39:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80abe00 con 0x7fc3e80b7390 2022-07-21T01:39:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3e81459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3e81459a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47034/0 (socket says 172.21.15.3:47034) 2022-07-21T01:39:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e6ffd700 1 -- 172.21.15.3:0/1525582129 learned_addr learned my addr 172.21.15.3:0/1525582129 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e77fe700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80b7390 0x7fc3e80b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e67fc700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e8156050 0x7fc3e81564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e6ffd700 1 -- 172.21.15.3:0/1525582129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80b7390 msgr2=0x7fc3e80b7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e6ffd700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80b7390 0x7fc3e80b7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e6ffd700 1 -- 172.21.15.3:0/1525582129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e8156050 msgr2=0x7fc3e81564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:13.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e6ffd700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e8156050 0x7fc3e81564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:13.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e6ffd700 1 -- 172.21.15.3:0/1525582129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80ac0a0 con 0x7fc3cc0049f0 2022-07-21T01:39:13.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e6ffd700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3e81459a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7fc3dc00b000 tx=0x7fc3dc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3d7fff700 1 -- 172.21.15.3:0/1525582129 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3dc012610 con 0x7fc3cc0049f0 2022-07-21T01:39:13.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3d7fff700 1 -- 172.21.15.3:0/1525582129 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc012770 con 0x7fc3cc0049f0 2022-07-21T01:39:13.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e80b9050 con 0x7fc3cc0049f0 2022-07-21T01:39:13.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3d7fff700 1 -- 172.21.15.3:0/1525582129 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3dc012ad0 con 0x7fc3cc0049f0 2022-07-21T01:39:13.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e80b8c50 con 0x7fc3cc0049f0 2022-07-21T01:39:13.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.901+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e80aa680 con 0x7fc3cc0049f0 2022-07-21T01:39:13.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.905+0000 7fc3d7fff700 1 -- 172.21.15.3:0/1525582129 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc3dc012c30 con 0x7fc3cc0049f0 2022-07-21T01:39:13.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.905+0000 7fc3d7fff700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc3c4053980 0x7fc3c4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:13.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.905+0000 7fc3d7fff700 1 -- 172.21.15.3:0/1525582129 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc3dc06c860 con 0x7fc3cc0049f0 2022-07-21T01:39:13.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.905+0000 7fc3e67fc700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc3c4053980 0x7fc3c4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:13.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.905+0000 7fc3e67fc700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc3c4053980 0x7fc3c4055e40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fc3d800b000 tx=0x7fc3d8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:13.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:13.909+0000 7fc3d7fff700 1 -- 172.21.15.3:0/1525582129 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc3dc043620 con 0x7fc3cc0049f0 2022-07-21T01:39:14.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.033+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3e80af5f0 con 0x7fc3cc0049f0 2022-07-21T01:39:14.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.033+0000 7fc3d7fff700 1 -- 172.21.15.3:0/1525582129 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc3dc046c40 con 0x7fc3cc0049f0 2022-07-21T01:39:14.033 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:39:14.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc3c4053980 msgr2=0x7fc3c4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:14.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc3c4053980 0x7fc3c4055e40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fc3d800b000 tx=0x7fc3d8008040).stop 2022-07-21T01:39:14.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 msgr2=0x7fc3e81459a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:14.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3e81459a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7fc3dc00b000 tx=0x7fc3dc008040).stop 2022-07-21T01:39:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 shutdown_connections 2022-07-21T01:39:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e80b7390 0x7fc3e80b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc3c4053980 0x7fc3c4055e40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e8156050 0x7fc3e81564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 --2- 172.21.15.3:0/1525582129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3cc0049f0 0x7fc3e81459a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 >> 172.21.15.3:0/1525582129 conn(0x7fc3e8066f70 msgr2=0x7fc3e8142290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:14.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 shutdown_connections 2022-07-21T01:39:14.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.037+0000 7fc3e7fff700 1 -- 172.21.15.3:0/1525582129 wait complete. 2022-07-21T01:39:14.046 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:39:14.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:39:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa748de6700 1 Processor -- start 2022-07-21T01:39:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa748de6700 1 -- start start 2022-07-21T01:39:14.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa748de6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa7440b2710 0x7fa74414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa748de6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440b01a0 0x7fa74414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa748de6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7440a7250 con 0x7fa7440afd60 2022-07-21T01:39:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa748de6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7440a7480 con 0x7fa7440b2710 2022-07-21T01:39:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa748de6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440a7720 con 0x7fa7440b01a0 2022-07-21T01:39:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa743fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa7440afd60 0x7fa7440a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55326/0 (socket says 172.21.15.3:55326) 2022-07-21T01:39:14.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa743fff700 1 -- 172.21.15.3:0/444827105 learned_addr learned my addr 172.21.15.3:0/444827105 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:14.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa742ffd700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440b01a0 0x7fa74414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:14.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7437fe700 1 --2- 172.21.15.3:0/444827105 >> v2:172.21.15.143:3300/0 conn(0x7fa7440b2710 0x7fa74414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:14.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315458932 0 0) 0x7fa7440a7250 con 0x7fa7440afd60 2022-07-21T01:39:14.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa730003260 con 0x7fa7440afd60 2022-07-21T01:39:14.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1809173116 0 0) 0x7fa730003260 con 0x7fa7440afd60 2022-07-21T01:39:14.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440b01a0 msgr2=0x7fa74414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:14.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440b01a0 0x7fa74414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 >> v2:172.21.15.143:3300/0 conn(0x7fa7440b2710 msgr2=0x7fa74414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:14.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 --2- 172.21.15.3:0/444827105 >> v2:172.21.15.143:3300/0 conn(0x7fa7440b2710 0x7fa74414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7440a79c0 con 0x7fa7440afd60 2022-07-21T01:39:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fa73c0019e0 con 0x7fa7440afd60 2022-07-21T01:39:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 >> v1:172.21.15.3:6789/0 conn(0x7fa7440afd60 legacy=0x7fa7440a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa730003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440b01a0 0x7fa730004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:14.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300049f0 0x7fa730004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:14.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440a7480 con 0x7fa730003750 2022-07-21T01:39:14.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440a7720 con 0x7fa7300049f0 2022-07-21T01:39:14.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa73000b750 con 0x7fa7440b01a0 2022-07-21T01:39:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa7437fe700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa730003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa742ffd700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440b01a0 0x7fa730004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.153+0000 7fa743fff700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300049f0 0x7fa730004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7437fe700 1 -- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440b01a0 msgr2=0x7fa730004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7437fe700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440b01a0 0x7fa730004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7437fe700 1 -- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300049f0 msgr2=0x7fa730004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:14.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7437fe700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300049f0 0x7fa730004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7437fe700 1 -- 172.21.15.3:0/444827105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa730003260 con 0x7fa730003750 2022-07-21T01:39:14.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa742ffd700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440b01a0 0x7fa730004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7437fe700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa730003b70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7fa73400c000 tx=0x7fa734006df0).ready entity=mon.0 client_cookie=561deceb34e45d82 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa73400e830 con 0x7fa730003750 2022-07-21T01:39:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73400e990 con 0x7fa730003750 2022-07-21T01:39:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7427fc700 1 -- 172.21.15.3:0/444827105 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa73400ecf0 con 0x7fa730003750 2022-07-21T01:39:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 -- 172.21.15.3:0/444827105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 msgr2=0x7fa730003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:14.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 --2- 172.21.15.3:0/444827105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa730003b70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7fa73400c000 tx=0x7fa734006df0).stop 2022-07-21T01:39:14.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa743fff700 1 -- 172.21.15.3:0/444827105 reap_dead start 2022-07-21T01:39:14.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 -- 172.21.15.3:0/444827105 shutdown_connections 2022-07-21T01:39:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 -- 172.21.15.3:0/444827105 >> 172.21.15.3:0/444827105 conn(0x7fa744060310 msgr2=0x7fa7440b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 -- 172.21.15.3:0/444827105 shutdown_connections 2022-07-21T01:39:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 -- 172.21.15.3:0/444827105 wait complete. 2022-07-21T01:39:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 Processor -- start 2022-07-21T01:39:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 -- start start 2022-07-21T01:39:14.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa7440b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:14.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7440aafe0 0x7fa744064bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:14.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440650c0 0x7fa7440a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:14.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa744065540 con 0x7fa730003750 2022-07-21T01:39:14.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440aba30 con 0x7fa7440aafe0 2022-07-21T01:39:14.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa748de6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440ac250 con 0x7fa7440650c0 2022-07-21T01:39:14.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa7440b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:14.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa7440b58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47054/0 (socket says 172.21.15.3:47054) 2022-07-21T01:39:14.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa742ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7440aafe0 0x7fa744064bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:14.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa7437fe700 1 -- 172.21.15.3:0/3710407028 learned_addr learned my addr 172.21.15.3:0/3710407028 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:14.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.157+0000 7fa743fff700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440650c0 0x7fa7440a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:14.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa7437fe700 1 -- 172.21.15.3:0/3710407028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440650c0 msgr2=0x7fa7440a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:14.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa7437fe700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440650c0 0x7fa7440a92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa7437fe700 1 -- 172.21.15.3:0/3710407028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7440aafe0 msgr2=0x7fa744064bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:14.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa7437fe700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7440aafe0 0x7fa744064bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa7437fe700 1 -- 172.21.15.3:0/3710407028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7440ac4f0 con 0x7fa730003750 2022-07-21T01:39:14.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa7437fe700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa7440b58c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7fa73400c000 tx=0x7fa734019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:14.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa740ff9700 1 -- 172.21.15.3:0/3710407028 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa73401b820 con 0x7fa730003750 2022-07-21T01:39:14.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa740ff9700 1 -- 172.21.15.3:0/3710407028 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa734000e90 con 0x7fa730003750 2022-07-21T01:39:14.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7440ab720 con 0x7fa730003750 2022-07-21T01:39:14.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa740ff9700 1 -- 172.21.15.3:0/3710407028 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa73401bb70 con 0x7fa730003750 2022-07-21T01:39:14.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7440aac00 con 0x7fa730003750 2022-07-21T01:39:14.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa740ff9700 1 -- 172.21.15.3:0/3710407028 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa73401bcd0 con 0x7fa730003750 2022-07-21T01:39:14.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7440abc30 con 0x7fa730003750 2022-07-21T01:39:14.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa740ff9700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa724053aa0 0x7fa724055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:14.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa740ff9700 1 -- 172.21.15.3:0/3710407028 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa73406d320 con 0x7fa730003750 2022-07-21T01:39:14.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.161+0000 7fa742ffd700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa724053aa0 0x7fa724055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:14.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.165+0000 7fa742ffd700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa724053aa0 0x7fa724055f60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa73800b000 tx=0x7fa738008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:14.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.173+0000 7fa740ff9700 1 -- 172.21.15.3:0/3710407028 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa734044160 con 0x7fa730003750 2022-07-21T01:39:14.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.333+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7440ac680 con 0x7fa730003750 2022-07-21T01:39:14.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.333+0000 7fa740ff9700 1 -- 172.21.15.3:0/3710407028 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2487 (secure 0 0 0) 0x7fa734047700 con 0x7fa730003750 2022-07-21T01:39:14.333 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:14.334 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:39:09.622528+0000","modified":"2022-07-21T01:39:10.626387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6259":{"gid":6259,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/862725488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":862725488},{"type":"v1","addr":"172.21.15.143:6833","nonce":862725488}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-21T01:39:14.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa724053aa0 msgr2=0x7fa724055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:14.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa724053aa0 0x7fa724055f60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa73800b000 tx=0x7fa738008040).stop 2022-07-21T01:39:14.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 msgr2=0x7fa7440b58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:14.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa7440b58c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7fa73400c000 tx=0x7fa734019000).stop 2022-07-21T01:39:14.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 shutdown_connections 2022-07-21T01:39:14.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7440650c0 0x7fa7440a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa724053aa0 0x7fa724055f60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7440aafe0 0x7fa744064bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 --2- 172.21.15.3:0/3710407028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730003750 0x7fa7440b58c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:14.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 >> 172.21.15.3:0/3710407028 conn(0x7fa744060310 msgr2=0x7fa74405be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:14.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 shutdown_connections 2022-07-21T01:39:14.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:14.337+0000 7fa748de6700 1 -- 172.21.15.3:0/3710407028 wait complete. 2022-07-21T01:39:14.343 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2022-07-21T01:39:14.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:39:09.622528+0000', 'modified': '2022-07-21T01:39:10.626387+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6259': {'gid': 6259, 'name': 'b', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6833/862725488', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 862725488}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 862725488}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:39:14.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:39:14.649 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-21T01:39:14.764 INFO:teuthology.orchestra.run.smithi143.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6259,"want_state":"up:active","state":"up:active","rank_uptime":5.1313296299999998,"mdsmap_epoch":35,"osdmap_epoch":79,"osdmap_epoch_barrier":78,"uptime":10.183389353999999} 2022-07-21T01:39:14.764 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6259, "mdsmap_epoch": 35, "osdmap_epoch": 79, "osdmap_epoch_barrier": 78, "rank_uptime": 5.13132963, "state": "up:active", "uptime": 10.183389354, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:39:14.765 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:39:14.765 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:39:14.766 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:39:14.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:14.767 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:39:14.775 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:39:14.835 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:39:14.836 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:39:14.851 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:39:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:39:14.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:14.855 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:39:14.863 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:39:14.864 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:39:14.908 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:39:14.927 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:39:14.928 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:39:14.942 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:39:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:39:14.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:14.956 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:39:14.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:39:14.984+0000 7f7936472080 -1 init, newargv = 0x55f401870710 newargc=9ceph-fuse[22774]: starting ceph client 2022-07-21T01:39:14.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr: 2022-07-21T01:39:14.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22774]: starting fuse 2022-07-21T01:39:16.091 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:39:16.110 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:39:16.110 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:39:16.122 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:39:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:39:16.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:16.125 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:39:16.173 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:39:16.174 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:39:16.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:16.175 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:39:16.175 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:39:16.176 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:39:16.176 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:39:16.176 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:39:16.177 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:39:16.177 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:39:16.178 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:39:16.178 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:39:16.179 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:39:16.179 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:39:16.180 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:39:16.180 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:39:16.181 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:39:16.181 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:39:16.181 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:39:16.182 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:39:16.182 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:39:16.183 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:39:16.183 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:39:16.184 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:39:16.184 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:39:16.185 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:39:16.232 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22774.asok 2022-07-21T01:39:16.237 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22774.asok 2022-07-21T01:39:16.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:16.238 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22774.asok status 2022-07-21T01:39:16.326 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:39:16.327 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:39:16.327 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:39:16.327 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:39:16.327 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:39:16.327 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:39:16.328 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:39:16.328 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22774", 2022-07-21T01:39:16.328 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:39:16.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:39:16.329 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:39:16.329 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:39:16.329 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 6403, 2022-07-21T01:39:16.329 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:39:16.330 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:39:16.330 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:39:16.330 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 6403 2022-07-21T01:39:16.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:39:16.330 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:39:16.331 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:39:16.331 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:39:16.331 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 752832426 2022-07-21T01:39:16.331 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:39:16.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:39:16.332 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:39:16.332 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:39:16.332 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:39:16.332 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 752832426 2022-07-21T01:39:16.333 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:39:16.333 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.6403 172.21.15.171:0/752832426", 2022-07-21T01:39:16.333 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/752832426", 2022-07-21T01:39:16.333 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:39:16.334 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 35, 2022-07-21T01:39:16.334 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 79, 2022-07-21T01:39:16.334 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:39:16.334 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:39:16.334 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:39:16.335 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:39:16.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:16.336 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:39:16.337 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:39:16.344 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:39:16.345 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:39:16.345 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:16.346 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:39:16.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:39:16.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.477+0000 7feab6506700 1 Processor -- start 2022-07-21T01:39:16.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.477+0000 7feab6506700 1 -- start start 2022-07-21T01:39:16.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.477+0000 7feab6506700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feab00614b0 0x7feab0061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.477+0000 7feab6506700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00aee80 con 0x7feab01405d0 2022-07-21T01:39:16.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.477+0000 7feab6506700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00b70b0 con 0x7feab00ad750 2022-07-21T01:39:16.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.477+0000 7feab6506700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab00af0b0 con 0x7feab00614b0 2022-07-21T01:39:16.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.477+0000 7feab5504700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feab00614b0 0x7feab0061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.477+0000 7feab5d05700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7feab01405d0 0x7feab01466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55346/0 (socket says 172.21.15.3:55346) 2022-07-21T01:39:16.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.477+0000 7feab5d05700 1 -- 172.21.15.3:0/4232899413 learned_addr learned my addr 172.21.15.3:0/4232899413 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:16.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab5504700 1 -- 172.21.15.3:0/4232899413 >> v1:172.21.15.143:6789/0 conn(0x7feab00ad750 legacy=0x7feab00b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab5504700 1 -- 172.21.15.3:0/4232899413 >> v1:172.21.15.3:6789/0 conn(0x7feab01405d0 legacy=0x7feab01466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feaa7fff700 1 -- 172.21.15.3:0/4232899413 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826488568 0 0) 0x7feab00aee80 con 0x7feab01405d0 2022-07-21T01:39:16.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab5504700 1 -- 172.21.15.3:0/4232899413 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab00af350 con 0x7feab00614b0 2022-07-21T01:39:16.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab5504700 1 --2- 172.21.15.3:0/4232899413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00614b0 0x7feab0061890 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7feab00aed30 tx=0x7feaa0006df0).ready entity=mon.0 client_cookie=b8751e39c055352a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:16.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feaa7fff700 1 -- 172.21.15.3:0/4232899413 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaa000d610 con 0x7feab00614b0 2022-07-21T01:39:16.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feaa7fff700 1 -- 172.21.15.3:0/4232899413 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa0000e90 con 0x7feab00614b0 2022-07-21T01:39:16.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feaa7fff700 1 -- 172.21.15.3:0/4232899413 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaa000d960 con 0x7feab00614b0 2022-07-21T01:39:16.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab6506700 1 -- 172.21.15.3:0/4232899413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00614b0 msgr2=0x7feab0061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab6506700 1 --2- 172.21.15.3:0/4232899413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00614b0 0x7feab0061890 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7feab00aed30 tx=0x7feaa0006df0).stop 2022-07-21T01:39:16.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab6506700 1 -- 172.21.15.3:0/4232899413 shutdown_connections 2022-07-21T01:39:16.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab6506700 1 --2- 172.21.15.3:0/4232899413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00614b0 0x7feab0061890 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab6506700 1 -- 172.21.15.3:0/4232899413 >> 172.21.15.3:0/4232899413 conn(0x7feab0066f70 msgr2=0x7feab0067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:16.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab6506700 1 -- 172.21.15.3:0/4232899413 shutdown_connections 2022-07-21T01:39:16.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.481+0000 7feab6506700 1 -- 172.21.15.3:0/4232899413 wait complete. 2022-07-21T01:39:16.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 Processor -- start 2022-07-21T01:39:16.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 -- start start 2022-07-21T01:39:16.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00614b0 0x7feab00ad060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00ad750 0x7feab00b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feab01405d0 0x7feab0152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab00b3e40 con 0x7feab00ad750 2022-07-21T01:39:16.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab00ad550 con 0x7feab00614b0 2022-07-21T01:39:16.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab00ab190 con 0x7feab01405d0 2022-07-21T01:39:16.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab4d03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00ad750 0x7feab00b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab4d03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00ad750 0x7feab00b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47070/0 (socket says 172.21.15.3:47070) 2022-07-21T01:39:16.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab4d03700 1 -- 172.21.15.3:0/3724143474 learned_addr learned my addr 172.21.15.3:0/3724143474 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:16.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab5d05700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feab01405d0 0x7feab0152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab5504700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00614b0 0x7feab00ad060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab4d03700 1 -- 172.21.15.3:0/3724143474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feab01405d0 msgr2=0x7feab0152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab4d03700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feab01405d0 0x7feab0152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab4d03700 1 -- 172.21.15.3:0/3724143474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00614b0 msgr2=0x7feab00ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab4d03700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00614b0 0x7feab00ad060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab4d03700 1 -- 172.21.15.3:0/3724143474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa000b180 con 0x7feab00ad750 2022-07-21T01:39:16.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab4d03700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00ad750 0x7feab00b35b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7feab00af970 tx=0x7feaac007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:16.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feaa67fc700 1 -- 172.21.15.3:0/3724143474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaac00f470 con 0x7feab00ad750 2022-07-21T01:39:16.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feaa67fc700 1 -- 172.21.15.3:0/3724143474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac00f5d0 con 0x7feab00ad750 2022-07-21T01:39:16.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feaa67fc700 1 -- 172.21.15.3:0/3724143474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaac00f930 con 0x7feab00ad750 2022-07-21T01:39:16.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab00ab430 con 0x7feab00ad750 2022-07-21T01:39:16.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.485+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab00b3aa0 con 0x7feab00ad750 2022-07-21T01:39:16.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.489+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab00ab5c0 con 0x7feab00ad750 2022-07-21T01:39:16.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.489+0000 7feaa67fc700 1 -- 172.21.15.3:0/3724143474 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feaac019030 con 0x7feab00ad750 2022-07-21T01:39:16.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.489+0000 7feaa67fc700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea98053a70 0x7fea98055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.489+0000 7feaa67fc700 1 -- 172.21.15.3:0/3724143474 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7feaac06b6f0 con 0x7feab00ad750 2022-07-21T01:39:16.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.489+0000 7feab5504700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea98053a70 0x7fea98055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.489+0000 7feab5504700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea98053a70 0x7fea98055f30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7feaa001c000 tx=0x7feaa0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:16.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.493+0000 7feaa67fc700 1 -- 172.21.15.3:0/3724143474 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feaac042410 con 0x7feab00ad750 2022-07-21T01:39:16.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.621+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feab00af5f0 con 0x7feab00ad750 2022-07-21T01:39:16.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feaa67fc700 1 -- 172.21.15.3:0/3724143474 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16274 (secure 0 0 0) 0x7feaac045a30 con 0x7feab00ad750 2022-07-21T01:39:16.623 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:16.623 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":79,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:39:10.626495+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:39:07.362717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-21T01:39:08.613529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:39:16.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea98053a70 msgr2=0x7fea98055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea98053a70 0x7fea98055f30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7feaa001c000 tx=0x7feaa0015000).stop 2022-07-21T01:39:16.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00ad750 msgr2=0x7feab00b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00ad750 0x7feab00b35b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7feab00af970 tx=0x7feaac007f70).stop 2022-07-21T01:39:16.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 shutdown_connections 2022-07-21T01:39:16.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feab01405d0 0x7feab0152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab00ad750 0x7feab00b35b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea98053a70 0x7fea98055f30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 --2- 172.21.15.3:0/3724143474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00614b0 0x7feab00ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 >> 172.21.15.3:0/3724143474 conn(0x7feab0066f70 msgr2=0x7feab013fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:16.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 shutdown_connections 2022-07-21T01:39:16.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.625+0000 7feab6506700 1 -- 172.21.15.3:0/3724143474 wait complete. 2022-07-21T01:39:16.636 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:39:07.362717+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:39:08.613529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:39:16.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:39:16.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.744+0000 7f08c08c6700 1 Processor -- start 2022-07-21T01:39:16.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.744+0000 7f08c08c6700 1 -- start start 2022-07-21T01:39:16.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.744+0000 7f08c08c6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f08bc063a60 0x7f08bc062050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.744+0000 7f08c08c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc0b1940 0x7f08bc13e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.744+0000 7f08c08c6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc0b10d0 con 0x7f08bc13e5b0 2022-07-21T01:39:16.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.744+0000 7f08c08c6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc063e40 con 0x7f08bc063a60 2022-07-21T01:39:16.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.744+0000 7f08c08c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc060200 con 0x7f08bc0b1940 2022-07-21T01:39:16.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08bad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f08bc063a60 0x7f08bc062050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08bad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f08bc063a60 0x7f08bc062050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47078/0 (socket says 172.21.15.3:47078) 2022-07-21T01:39:16.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08bad9d700 1 -- 172.21.15.3:0/140087585 learned_addr learned my addr 172.21.15.3:0/140087585 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:16.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08b259c700 1 --2- 172.21.15.3:0/140087585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc0b1940 0x7f08bc13e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08bad9d700 1 -- 172.21.15.3:0/140087585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc0b1940 msgr2=0x7f08bc13e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:16.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08bad9d700 1 --2- 172.21.15.3:0/140087585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc0b1940 0x7f08bc13e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08ba59c700 1 -- 172.21.15.3:0/140087585 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326129855 0 0) 0x7f08bc0b10d0 con 0x7f08bc13e5b0 2022-07-21T01:39:16.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08bad9d700 1 -- 172.21.15.3:0/140087585 >> v1:172.21.15.143:6789/0 conn(0x7f08bc13e5b0 legacy=0x7f08bc1414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08bad9d700 1 -- 172.21.15.3:0/140087585 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0b1300 con 0x7f08bc063a60 2022-07-21T01:39:16.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08b259c700 1 --2- 172.21.15.3:0/140087585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc0b1940 0x7f08bc13e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:16.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08bad9d700 1 --2- 172.21.15.3:0/140087585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 0x7f08bc062050 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7f08ac002f70 tx=0x7f08ac006eb0).ready entity=mon.0 client_cookie=c6ee03424c7b139e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:16.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08ba59c700 1 -- 172.21.15.3:0/140087585 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac00d570 con 0x7f08bc063a60 2022-07-21T01:39:16.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08ba59c700 1 -- 172.21.15.3:0/140087585 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac000e90 con 0x7f08bc063a60 2022-07-21T01:39:16.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08ba59c700 1 -- 172.21.15.3:0/140087585 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac00d8c0 con 0x7f08bc063a60 2022-07-21T01:39:16.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 -- 172.21.15.3:0/140087585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 msgr2=0x7f08bc062050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 --2- 172.21.15.3:0/140087585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 0x7f08bc062050 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7f08ac002f70 tx=0x7f08ac006eb0).stop 2022-07-21T01:39:16.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 -- 172.21.15.3:0/140087585 shutdown_connections 2022-07-21T01:39:16.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 --2- 172.21.15.3:0/140087585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc0b1940 0x7f08bc13e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 --2- 172.21.15.3:0/140087585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 0x7f08bc062050 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 -- 172.21.15.3:0/140087585 >> 172.21.15.3:0/140087585 conn(0x7f08bc05c620 msgr2=0x7f08bc05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:16.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 -- 172.21.15.3:0/140087585 shutdown_connections 2022-07-21T01:39:16.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 -- 172.21.15.3:0/140087585 wait complete. 2022-07-21T01:39:16.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 Processor -- start 2022-07-21T01:39:16.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 -- start start 2022-07-21T01:39:16.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.748+0000 7f08c08c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 0x7f08bc136360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08c08c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc0b1940 0x7f08bc145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08c08c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc13e5b0 0x7f08bc134090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08c08c6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc13a190 con 0x7f08bc063a60 2022-07-21T01:39:16.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08c08c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc151180 con 0x7f08bc0b1940 2022-07-21T01:39:16.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08c08c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc1513e0 con 0x7f08bc13e5b0 2022-07-21T01:39:16.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 0x7f08bc136360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 0x7f08bc136360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47084/0 (socket says 172.21.15.3:47084) 2022-07-21T01:39:16.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bad9d700 1 -- 172.21.15.3:0/2241637465 learned_addr learned my addr 172.21.15.3:0/2241637465 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:16.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08b259c700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc0b1940 0x7f08bc145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bb59e700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc13e5b0 0x7f08bc134090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bad9d700 1 -- 172.21.15.3:0/2241637465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc13e5b0 msgr2=0x7f08bc134090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bad9d700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc13e5b0 0x7f08bc134090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bad9d700 1 -- 172.21.15.3:0/2241637465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc0b1940 msgr2=0x7f08bc145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bad9d700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc0b1940 0x7f08bc145ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bad9d700 1 -- 172.21.15.3:0/2241637465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a400b180 con 0x7f08bc063a60 2022-07-21T01:39:16.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08bad9d700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 0x7f08bc136360 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f08ac002f70 tx=0x7f08ac015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:16.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08b8d99700 1 -- 172.21.15.3:0/2241637465 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac0068b0 con 0x7f08bc063a60 2022-07-21T01:39:16.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08b8d99700 1 -- 172.21.15.3:0/2241637465 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac00d8f0 con 0x7f08bc063a60 2022-07-21T01:39:16.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08b8d99700 1 -- 172.21.15.3:0/2241637465 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac011d70 con 0x7f08bc063a60 2022-07-21T01:39:16.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac00b1e0 con 0x7f08bc063a60 2022-07-21T01:39:16.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc139a20 con 0x7f08bc063a60 2022-07-21T01:39:16.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc1515e0 con 0x7f08bc063a60 2022-07-21T01:39:16.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.752+0000 7f08b8d99700 1 -- 172.21.15.3:0/2241637465 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f08ac01a040 con 0x7f08bc063a60 2022-07-21T01:39:16.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.756+0000 7f08b8d99700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0894053b00 0x7f0894055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:16.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.756+0000 7f08b8d99700 1 -- 172.21.15.3:0/2241637465 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f08ac06ce00 con 0x7f08bc063a60 2022-07-21T01:39:16.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.756+0000 7f08b259c700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0894053b00 0x7f0894055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:16.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.756+0000 7f08b259c700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0894053b00 0x7f0894055fc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f08a400b000 tx=0x7f08a4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:16.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.760+0000 7f08b8d99700 1 -- 172.21.15.3:0/2241637465 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08ac043c40 con 0x7f08bc063a60 2022-07-21T01:39:16.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f08bc0b15a0 con 0x7f08bc063a60 2022-07-21T01:39:16.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08b8d99700 1 -- 172.21.15.3:0/2241637465 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v79) v1 ==== 110+0+10 (secure 0 0 0) 0x7f08ac023020 con 0x7f08bc063a60 2022-07-21T01:39:16.884 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:39:16.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0894053b00 msgr2=0x7f0894055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0894053b00 0x7f0894055fc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f08a400b000 tx=0x7f08a4006df0).stop 2022-07-21T01:39:16.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 msgr2=0x7f08bc136360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:16.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 0x7f08bc136360 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f08ac002f70 tx=0x7f08ac015000).stop 2022-07-21T01:39:16.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 shutdown_connections 2022-07-21T01:39:16.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc13e5b0 0x7f08bc134090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0894053b00 0x7f0894055fc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc0b1940 0x7f08bc145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 --2- 172.21.15.3:0/2241637465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08bc063a60 0x7f08bc136360 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:16.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 >> 172.21.15.3:0/2241637465 conn(0x7f08bc05c620 msgr2=0x7f08bc1413e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:16.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 shutdown_connections 2022-07-21T01:39:16.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:16.884+0000 7f08c08c6700 1 -- 172.21.15.3:0/2241637465 wait complete. 2022-07-21T01:39:16.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:39:17.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.008+0000 7f88a4b9c700 1 Processor -- start 2022-07-21T01:39:17.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.008+0000 7f88a4b9c700 1 -- start start 2022-07-21T01:39:17.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f88a4b9c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88a0146ae0 0x7f88a01403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f88a4b9c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a00b4310 0x7f88a01408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f88a4b9c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a0143cf0 con 0x7f88a00b4820 2022-07-21T01:39:17.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f88a4b9c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a01476a0 con 0x7f88a0146ae0 2022-07-21T01:39:17.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f88a4b9c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a0143f90 con 0x7f88a00b4310 2022-07-21T01:39:17.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88a0146ae0 0x7f88a01403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889ed9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f88a00b4820 0x7f88a00b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:57488/0 (socket says 172.21.15.3:57488) 2022-07-21T01:39:17.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a00b4310 0x7f88a01408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889ed9d700 1 -- 172.21.15.3:0/3724688932 learned_addr learned my addr 172.21.15.3:0/3724688932 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:17.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889e59c700 1 -- 172.21.15.3:0/3724688932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a00b4310 msgr2=0x7f88a01408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:17.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889e59c700 1 --2- 172.21.15.3:0/3724688932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a00b4310 0x7f88a01408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889e59c700 1 -- 172.21.15.3:0/3724688932 >> v1:172.21.15.143:6789/0 conn(0x7f88a00b4820 legacy=0x7f88a00b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889e59c700 1 -- 172.21.15.3:0/3724688932 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a00b0e20 con 0x7f88a0146ae0 2022-07-21T01:39:17.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889d59a700 1 -- 172.21.15.3:0/3724688932 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757457923 0 0) 0x7f88a0143cf0 con 0x7f88a00b4820 2022-07-21T01:39:17.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889dd9b700 1 --2- 172.21.15.3:0/3724688932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a00b4310 0x7f88a01408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:17.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889e59c700 1 --2- 172.21.15.3:0/3724688932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a0146ae0 0x7f88a01403c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f8888002f70 tx=0x7f8888006df0).ready entity=mon.1 client_cookie=a7cd2211f9bdc884 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889d59a700 1 -- 172.21.15.3:0/3724688932 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f888800d570 con 0x7f88a0146ae0 2022-07-21T01:39:17.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889d59a700 1 -- 172.21.15.3:0/3724688932 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8888000e90 con 0x7f88a0146ae0 2022-07-21T01:39:17.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.012+0000 7f889d59a700 1 -- 172.21.15.3:0/3724688932 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f888800d8c0 con 0x7f88a0146ae0 2022-07-21T01:39:17.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 -- 172.21.15.3:0/3724688932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a0146ae0 msgr2=0x7f88a01403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 --2- 172.21.15.3:0/3724688932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a0146ae0 0x7f88a01403c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f8888002f70 tx=0x7f8888006df0).stop 2022-07-21T01:39:17.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 -- 172.21.15.3:0/3724688932 shutdown_connections 2022-07-21T01:39:17.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 --2- 172.21.15.3:0/3724688932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a00b4310 0x7f88a01408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 --2- 172.21.15.3:0/3724688932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a0146ae0 0x7f88a01403c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 -- 172.21.15.3:0/3724688932 >> 172.21.15.3:0/3724688932 conn(0x7f88a005c620 msgr2=0x7f88a005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:17.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 -- 172.21.15.3:0/3724688932 shutdown_connections 2022-07-21T01:39:17.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 -- 172.21.15.3:0/3724688932 wait complete. 2022-07-21T01:39:17.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 Processor -- start 2022-07-21T01:39:17.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 -- start start 2022-07-21T01:39:17.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88a00b4310 0x7f88a00b9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00b4820 0x7f88a01522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a0146ae0 0x7f88a0155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a0143520 con 0x7f88a00b4310 2022-07-21T01:39:17.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a01532e0 con 0x7f88a00b4820 2022-07-21T01:39:17.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f88a4b9c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a01558c0 con 0x7f88a0146ae0 2022-07-21T01:39:17.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f889e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88a00b4310 0x7f88a00b9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f889e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88a00b4310 0x7f88a00b9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47096/0 (socket says 172.21.15.3:47096) 2022-07-21T01:39:17.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f889e59c700 1 -- 172.21.15.3:0/442498400 learned_addr learned my addr 172.21.15.3:0/442498400 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f889ed9d700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a0146ae0 0x7f88a0155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f889dd9b700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00b4820 0x7f88a01522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.016+0000 7f889e59c700 1 -- 172.21.15.3:0/442498400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a0146ae0 msgr2=0x7f88a0155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f889e59c700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a0146ae0 0x7f88a0155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f889e59c700 1 -- 172.21.15.3:0/442498400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00b4820 msgr2=0x7f88a01522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f889e59c700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00b4820 0x7f88a01522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f889e59c700 1 -- 172.21.15.3:0/442498400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889000b180 con 0x7f88a00b4310 2022-07-21T01:39:17.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f889e59c700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88a00b4310 0x7f88a00b9c80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f88a00aed70 tx=0x7f8888014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f88977fe700 1 -- 172.21.15.3:0/442498400 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f888800dea0 con 0x7f88a00b4310 2022-07-21T01:39:17.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f88977fe700 1 -- 172.21.15.3:0/442498400 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8888013da0 con 0x7f88a00b4310 2022-07-21T01:39:17.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888800b180 con 0x7f88a00b4310 2022-07-21T01:39:17.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f88977fe700 1 -- 172.21.15.3:0/442498400 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f888801e5f0 con 0x7f88a00b4310 2022-07-21T01:39:17.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a01428e0 con 0x7f88a00b4310 2022-07-21T01:39:17.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a00574e0 con 0x7f88a00b4310 2022-07-21T01:39:17.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f88977fe700 1 -- 172.21.15.3:0/442498400 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8888025030 con 0x7f88a00b4310 2022-07-21T01:39:17.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f88977fe700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88840539e0 0x7f8884055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f88977fe700 1 -- 172.21.15.3:0/442498400 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f888806cee0 con 0x7f88a00b4310 2022-07-21T01:39:17.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.020+0000 7f889dd9b700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88840539e0 0x7f8884055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.024+0000 7f889dd9b700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88840539e0 0x7f8884055ea0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f889000b000 tx=0x7f8890006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.028+0000 7f88977fe700 1 -- 172.21.15.3:0/442498400 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8888043b90 con 0x7f88a00b4310 2022-07-21T01:39:17.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.152+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f88a0069a70 con 0x7f88a00b4310 2022-07-21T01:39:17.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.152+0000 7f88977fe700 1 -- 172.21.15.3:0/442498400 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v79) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8888023070 con 0x7f88a00b4310 2022-07-21T01:39:17.201 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:39:17.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88840539e0 msgr2=0x7f8884055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88840539e0 0x7f8884055ea0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f889000b000 tx=0x7f8890006df0).stop 2022-07-21T01:39:17.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88a00b4310 msgr2=0x7f88a00b9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88a00b4310 0x7f88a00b9c80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f88a00aed70 tx=0x7f8888014000).stop 2022-07-21T01:39:17.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 shutdown_connections 2022-07-21T01:39:17.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88a0146ae0 0x7f88a0155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88840539e0 0x7f8884055ea0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00b4820 0x7f88a01522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 --2- 172.21.15.3:0/442498400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88a00b4310 0x7f88a00b9c80 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 >> 172.21.15.3:0/442498400 conn(0x7f88a005c620 msgr2=0x7f88a00b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:17.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 shutdown_connections 2022-07-21T01:39:17.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.156+0000 7f88a4b9c700 1 -- 172.21.15.3:0/442498400 wait complete. 2022-07-21T01:39:17.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:39:17.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e6f21700 1 Processor -- start 2022-07-21T01:39:17.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e6f21700 1 -- start start 2022-07-21T01:39:17.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e6f21700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f37e00b5b20 0x7f37e00b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e6f21700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e01440f0 con 0x7f37e005fd90 2022-07-21T01:39:17.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e6f21700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e0060170 con 0x7f37e00b66f0 2022-07-21T01:39:17.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e6f21700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e00ba840 con 0x7f37e00b5b20 2022-07-21T01:39:17.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e4cbd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f37e005fd90 0x7f37e00ba130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55384/0 (socket says 172.21.15.3:55384) 2022-07-21T01:39:17.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e4cbd700 1 -- 172.21.15.3:0/2591930878 learned_addr learned my addr 172.21.15.3:0/2591930878 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:17.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e54be700 1 --2- 172.21.15.3:0/2591930878 >> v2:172.21.15.3:3300/0 conn(0x7f37e00b5b20 0x7f37e00b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37df7fe700 1 -- 172.21.15.3:0/2591930878 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829885242 0 0) 0x7f37e01440f0 con 0x7f37e005fd90 2022-07-21T01:39:17.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37df7fe700 1 -- 172.21.15.3:0/2591930878 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37cc003260 con 0x7f37e005fd90 2022-07-21T01:39:17.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e54be700 1 -- 172.21.15.3:0/2591930878 >> v1:172.21.15.143:6789/0 conn(0x7f37e00b66f0 legacy=0x7f37e00b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e54be700 1 -- 172.21.15.3:0/2591930878 >> v1:172.21.15.3:6789/0 conn(0x7f37e005fd90 legacy=0x7f37e00ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.320+0000 7f37e54be700 1 -- 172.21.15.3:0/2591930878 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e0143be0 con 0x7f37e00b5b20 2022-07-21T01:39:17.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e54be700 1 --2- 172.21.15.3:0/2591930878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 0x7f37e00b4340 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f37d80032f0 tx=0x7f37d80091d0).ready entity=mon.0 client_cookie=fb7680e3d439c1fb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37df7fe700 1 -- 172.21.15.3:0/2591930878 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37d800b6c0 con 0x7f37e00b5b20 2022-07-21T01:39:17.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37df7fe700 1 -- 172.21.15.3:0/2591930878 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d800b820 con 0x7f37e00b5b20 2022-07-21T01:39:17.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37df7fe700 1 -- 172.21.15.3:0/2591930878 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37d800b990 con 0x7f37e00b5b20 2022-07-21T01:39:17.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e6f21700 1 -- 172.21.15.3:0/2591930878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 msgr2=0x7f37e00b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e6f21700 1 --2- 172.21.15.3:0/2591930878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 0x7f37e00b4340 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f37d80032f0 tx=0x7f37d80091d0).stop 2022-07-21T01:39:17.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e6f21700 1 -- 172.21.15.3:0/2591930878 shutdown_connections 2022-07-21T01:39:17.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e6f21700 1 --2- 172.21.15.3:0/2591930878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 0x7f37e00b4340 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e6f21700 1 -- 172.21.15.3:0/2591930878 >> 172.21.15.3:0/2591930878 conn(0x7f37e0062530 msgr2=0x7f37e0066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:17.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e6f21700 1 -- 172.21.15.3:0/2591930878 shutdown_connections 2022-07-21T01:39:17.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e6f21700 1 -- 172.21.15.3:0/2591930878 wait complete. 2022-07-21T01:39:17.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e6f21700 1 Processor -- start 2022-07-21T01:39:17.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.324+0000 7f37e6f21700 1 -- start start 2022-07-21T01:39:17.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e6f21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e005fd90 0x7f37e0146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e6f21700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 0x7f37e0142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e6f21700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e00b66f0 0x7f37e00af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e6f21700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e01463f0 con 0x7f37e00b5b20 2022-07-21T01:39:17.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e6f21700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e00af5f0 con 0x7f37e00b66f0 2022-07-21T01:39:17.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e6f21700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e0142b70 con 0x7f37e005fd90 2022-07-21T01:39:17.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37dffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 0x7f37e0142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37dffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 0x7f37e0142310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47110/0 (socket says 172.21.15.3:47110) 2022-07-21T01:39:17.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37dffff700 1 -- 172.21.15.3:0/1813261164 learned_addr learned my addr 172.21.15.3:0/1813261164 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:17.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e4cbd700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e005fd90 0x7f37e0146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e54be700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e00b66f0 0x7f37e00af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37dffff700 1 -- 172.21.15.3:0/1813261164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e005fd90 msgr2=0x7f37e0146af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37dffff700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e005fd90 0x7f37e0146af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37dffff700 1 -- 172.21.15.3:0/1813261164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e00b66f0 msgr2=0x7f37e00af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37dffff700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e00b66f0 0x7f37e00af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37dffff700 1 -- 172.21.15.3:0/1813261164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d8009470 con 0x7f37e00b5b20 2022-07-21T01:39:17.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37dffff700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 0x7f37e0142310 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f37e00b1b70 tx=0x7f37d4007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37ddffb700 1 -- 172.21.15.3:0/1813261164 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37d4012400 con 0x7f37e00b5b20 2022-07-21T01:39:17.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37ddffb700 1 -- 172.21.15.3:0/1813261164 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d4012560 con 0x7f37e00b5b20 2022-07-21T01:39:17.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e00aee60 con 0x7f37e00b5b20 2022-07-21T01:39:17.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37ddffb700 1 -- 172.21.15.3:0/1813261164 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37d40128c0 con 0x7f37e00b5b20 2022-07-21T01:39:17.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.328+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e00ae950 con 0x7f37e00b5b20 2022-07-21T01:39:17.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.332+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e0143e50 con 0x7f37e00b5b20 2022-07-21T01:39:17.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.332+0000 7f37ddffb700 1 -- 172.21.15.3:0/1813261164 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f37d4019030 con 0x7f37e00b5b20 2022-07-21T01:39:17.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.332+0000 7f37ddffb700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f37c0053b00 0x7f37c0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.332+0000 7f37ddffb700 1 -- 172.21.15.3:0/1813261164 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f37d406b2f0 con 0x7f37e00b5b20 2022-07-21T01:39:17.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.332+0000 7f37e4cbd700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f37c0053b00 0x7f37c0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.332+0000 7f37e4cbd700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f37c0053b00 0x7f37c0055fc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f37d000d000 tx=0x7f37d0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.336+0000 7f37ddffb700 1 -- 172.21.15.3:0/1813261164 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f37d40420b0 con 0x7f37e00b5b20 2022-07-21T01:39:17.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f37e00574e0 con 0x7f37e00b5b20 2022-07-21T01:39:17.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37ddffb700 1 -- 172.21.15.3:0/1813261164 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) v1 ==== 104+0+10 (secure 0 0 0) 0x7f37d40456d0 con 0x7f37e00b5b20 2022-07-21T01:39:17.464 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:39:17.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f37c0053b00 msgr2=0x7f37c0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f37c0053b00 0x7f37c0055fc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f37d000d000 tx=0x7f37d0009040).stop 2022-07-21T01:39:17.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 msgr2=0x7f37e0142310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 0x7f37e0142310 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f37e00b1b70 tx=0x7f37d4007c50).stop 2022-07-21T01:39:17.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 shutdown_connections 2022-07-21T01:39:17.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f37c0053b00 0x7f37c0055fc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e00b66f0 0x7f37e00af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37e00b5b20 0x7f37e0142310 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 --2- 172.21.15.3:0/1813261164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e005fd90 0x7f37e0146af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 >> 172.21.15.3:0/1813261164 conn(0x7f37e0062530 msgr2=0x7f37e012c230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:17.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 shutdown_connections 2022-07-21T01:39:17.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.464+0000 7f37e6f21700 1 -- 172.21.15.3:0/1813261164 wait complete. 2022-07-21T01:39:17.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:39:17.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52d56700 1 Processor -- start 2022-07-21T01:39:17.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52d56700 1 -- start start 2022-07-21T01:39:17.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52d56700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda4c0614b0 0x7fda4c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52d56700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c1405d0 0x7fda4c140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52d56700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4c0aee80 con 0x7fda4c0ad750 2022-07-21T01:39:17.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52d56700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4c0b70b0 con 0x7fda4c0614b0 2022-07-21T01:39:17.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52d56700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0af0b0 con 0x7fda4c1405d0 2022-07-21T01:39:17.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52555700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c1405d0 0x7fda4c140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda51d54700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda4c0614b0 0x7fda4c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52555700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c1405d0 0x7fda4c140a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:54034/0 (socket says 172.21.15.3:54034) 2022-07-21T01:39:17.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52555700 1 -- 172.21.15.3:0/3518264605 learned_addr learned my addr 172.21.15.3:0/3518264605 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:17.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda51553700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fda4c0ad750 0x7fda4c0b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:57516/0 (socket says 172.21.15.3:57516) 2022-07-21T01:39:17.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52555700 1 -- 172.21.15.3:0/3518264605 >> v2:172.21.15.143:3300/0 conn(0x7fda4c0614b0 msgr2=0x7fda4c061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:17.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52555700 1 --2- 172.21.15.3:0/3518264605 >> v2:172.21.15.143:3300/0 conn(0x7fda4c0614b0 0x7fda4c061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52555700 1 -- 172.21.15.3:0/3518264605 >> v1:172.21.15.143:6789/0 conn(0x7fda4c0ad750 legacy=0x7fda4c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52555700 1 -- 172.21.15.3:0/3518264605 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c0af350 con 0x7fda4c1405d0 2022-07-21T01:39:17.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.588+0000 7fda52555700 1 --2- 172.21.15.3:0/3518264605 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c1405d0 0x7fda4c140a60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fda4c0aed30 tx=0x7fda44006eb0).ready entity=mon.2 client_cookie=b89df12630113331 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda50d52700 1 -- 172.21.15.3:0/3518264605 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda4400e540 con 0x7fda4c1405d0 2022-07-21T01:39:17.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda50d52700 1 -- 172.21.15.3:0/3518264605 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda44000e90 con 0x7fda4c1405d0 2022-07-21T01:39:17.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda50d52700 1 -- 172.21.15.3:0/3518264605 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda4400e8d0 con 0x7fda4c1405d0 2022-07-21T01:39:17.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 -- 172.21.15.3:0/3518264605 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c1405d0 msgr2=0x7fda4c140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 --2- 172.21.15.3:0/3518264605 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c1405d0 0x7fda4c140a60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fda4c0aed30 tx=0x7fda44006eb0).stop 2022-07-21T01:39:17.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 -- 172.21.15.3:0/3518264605 shutdown_connections 2022-07-21T01:39:17.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 --2- 172.21.15.3:0/3518264605 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c1405d0 0x7fda4c140a60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 --2- 172.21.15.3:0/3518264605 >> v2:172.21.15.143:3300/0 conn(0x7fda4c0614b0 0x7fda4c061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 -- 172.21.15.3:0/3518264605 >> 172.21.15.3:0/3518264605 conn(0x7fda4c066f70 msgr2=0x7fda4c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:17.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 -- 172.21.15.3:0/3518264605 shutdown_connections 2022-07-21T01:39:17.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 -- 172.21.15.3:0/3518264605 wait complete. 2022-07-21T01:39:17.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 Processor -- start 2022-07-21T01:39:17.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.592+0000 7fda52d56700 1 -- start start 2022-07-21T01:39:17.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52d56700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c0614b0 0x7fda4c0afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52d56700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda4c0ad750 0x7fda4c1416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52d56700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda4c1405d0 0x7fda4c0ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52d56700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c143bc0 con 0x7fda4c0ad750 2022-07-21T01:39:17.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52d56700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c143df0 con 0x7fda4c1405d0 2022-07-21T01:39:17.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52d56700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c153160 con 0x7fda4c0614b0 2022-07-21T01:39:17.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51553700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda4c0ad750 0x7fda4c1416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51d54700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c0614b0 0x7fda4c0afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51553700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda4c0ad750 0x7fda4c1416c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47126/0 (socket says 172.21.15.3:47126) 2022-07-21T01:39:17.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52555700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda4c1405d0 0x7fda4c0ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51553700 1 -- 172.21.15.3:0/775547498 learned_addr learned my addr 172.21.15.3:0/775547498 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:17.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51553700 1 -- 172.21.15.3:0/775547498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c0614b0 msgr2=0x7fda4c0afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51553700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c0614b0 0x7fda4c0afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51553700 1 -- 172.21.15.3:0/775547498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda4c1405d0 msgr2=0x7fda4c0ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51553700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda4c1405d0 0x7fda4c0ac650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51553700 1 -- 172.21.15.3:0/775547498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4400b1e0 con 0x7fda4c0ad750 2022-07-21T01:39:17.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda51553700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda4c0ad750 0x7fda4c1416c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7fda48001550 tx=0x7fda48007f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda3affd700 1 -- 172.21.15.3:0/775547498 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda48005ae0 con 0x7fda4c0ad750 2022-07-21T01:39:17.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c153400 con 0x7fda4c0ad750 2022-07-21T01:39:17.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda3affd700 1 -- 172.21.15.3:0/775547498 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48005c40 con 0x7fda4c0ad750 2022-07-21T01:39:17.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda3affd700 1 -- 172.21.15.3:0/775547498 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda48007110 con 0x7fda4c0ad750 2022-07-21T01:39:17.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c141bb0 con 0x7fda4c0ad750 2022-07-21T01:39:17.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c0af5f0 con 0x7fda4c0ad750 2022-07-21T01:39:17.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.596+0000 7fda3affd700 1 -- 172.21.15.3:0/775547498 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fda48010030 con 0x7fda4c0ad750 2022-07-21T01:39:17.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.600+0000 7fda3affd700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda2c053980 0x7fda2c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.600+0000 7fda3affd700 1 -- 172.21.15.3:0/775547498 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fda4806c110 con 0x7fda4c0ad750 2022-07-21T01:39:17.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.600+0000 7fda51d54700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda2c053980 0x7fda2c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.600+0000 7fda51d54700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda2c053980 0x7fda2c055e40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fda3c00b000 tx=0x7fda3c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.604+0000 7fda3affd700 1 -- 172.21.15.3:0/775547498 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fda48042e30 con 0x7fda4c0ad750 2022-07-21T01:39:17.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.728+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fda4c0574e0 con 0x7fda4c0ad750 2022-07-21T01:39:17.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda3affd700 1 -- 172.21.15.3:0/775547498 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) v1 ==== 100+0+10 (secure 0 0 0) 0x7fda48046450 con 0x7fda4c0ad750 2022-07-21T01:39:17.731 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:39:17.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda2c053980 msgr2=0x7fda2c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda2c053980 0x7fda2c055e40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fda3c00b000 tx=0x7fda3c006df0).stop 2022-07-21T01:39:17.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda4c0ad750 msgr2=0x7fda4c1416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda4c0ad750 0x7fda4c1416c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7fda48001550 tx=0x7fda48007f20).stop 2022-07-21T01:39:17.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 shutdown_connections 2022-07-21T01:39:17.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fda2c053980 0x7fda2c055e40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda4c1405d0 0x7fda4c0ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda4c0ad750 0x7fda4c1416c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 --2- 172.21.15.3:0/775547498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda4c0614b0 0x7fda4c0afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 >> 172.21.15.3:0/775547498 conn(0x7fda4c066f70 msgr2=0x7fda4c0b8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:17.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 shutdown_connections 2022-07-21T01:39:17.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.732+0000 7fda52d56700 1 -- 172.21.15.3:0/775547498 wait complete. 2022-07-21T01:39:17.744 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:39:17.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:39:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa736559700 1 Processor -- start 2022-07-21T01:39:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa736559700 1 -- start start 2022-07-21T01:39:17.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa736559700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300b6a90 0x7fa7300acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa736559700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa730060aa0 0x7fa7300aa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa736559700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa7300aa540 0x7fa7300aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa736559700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7300b6e70 con 0x7fa730060aa0 2022-07-21T01:39:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa736559700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa730061370 con 0x7fa7300aa540 2022-07-21T01:39:17.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa736559700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa730061610 con 0x7fa7300b6a90 2022-07-21T01:39:17.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa727fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa730060aa0 0x7fa7300aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa727fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa730060aa0 0x7fa7300aa050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47132/0 (socket says 172.21.15.3:47132) 2022-07-21T01:39:17.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa727fff700 1 -- 172.21.15.3:0/4159783330 learned_addr learned my addr 172.21.15.3:0/4159783330 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:17.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa72ffff700 1 --2- 172.21.15.3:0/4159783330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300b6a90 0x7fa7300acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.848+0000 7fa734af6700 1 --2- 172.21.15.3:0/4159783330 >> v2:172.21.15.143:3300/0 conn(0x7fa7300aa540 0x7fa7300aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa727fff700 1 -- 172.21.15.3:0/4159783330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300b6a90 msgr2=0x7fa7300acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:17.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa727fff700 1 --2- 172.21.15.3:0/4159783330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300b6a90 0x7fa7300acee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa727fff700 1 -- 172.21.15.3:0/4159783330 >> v2:172.21.15.143:3300/0 conn(0x7fa7300aa540 msgr2=0x7fa7300aa9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:17.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa727fff700 1 --2- 172.21.15.3:0/4159783330 >> v2:172.21.15.143:3300/0 conn(0x7fa7300aa540 0x7fa7300aa9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa727fff700 1 -- 172.21.15.3:0/4159783330 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7300618b0 con 0x7fa730060aa0 2022-07-21T01:39:17.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa727fff700 1 --2- 172.21.15.3:0/4159783330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 0x7fa7300aa050 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7fa718002f70 tx=0x7fa718006eb0).ready entity=mon.0 client_cookie=71c85a356a1530ec server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa72f7fe700 1 -- 172.21.15.3:0/4159783330 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa71800d570 con 0x7fa730060aa0 2022-07-21T01:39:17.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa72f7fe700 1 -- 172.21.15.3:0/4159783330 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa718000e90 con 0x7fa730060aa0 2022-07-21T01:39:17.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa72f7fe700 1 -- 172.21.15.3:0/4159783330 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa71800d900 con 0x7fa730060aa0 2022-07-21T01:39:17.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 -- 172.21.15.3:0/4159783330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 msgr2=0x7fa7300aa050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 --2- 172.21.15.3:0/4159783330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 0x7fa7300aa050 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7fa718002f70 tx=0x7fa718006eb0).stop 2022-07-21T01:39:17.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 -- 172.21.15.3:0/4159783330 shutdown_connections 2022-07-21T01:39:17.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 --2- 172.21.15.3:0/4159783330 >> v2:172.21.15.143:3300/0 conn(0x7fa7300aa540 0x7fa7300aa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 --2- 172.21.15.3:0/4159783330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 0x7fa7300aa050 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 --2- 172.21.15.3:0/4159783330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300b6a90 0x7fa7300acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 -- 172.21.15.3:0/4159783330 >> 172.21.15.3:0/4159783330 conn(0x7fa7300b3610 msgr2=0x7fa7300b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:17.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 -- 172.21.15.3:0/4159783330 shutdown_connections 2022-07-21T01:39:17.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 -- 172.21.15.3:0/4159783330 wait complete. 2022-07-21T01:39:17.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 Processor -- start 2022-07-21T01:39:17.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.852+0000 7fa736559700 1 -- start start 2022-07-21T01:39:17.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa736559700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 0x7fa730063dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa736559700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300aa540 0x7fa730140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa736559700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300b6a90 0x7fa730064550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa736559700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa730137ff0 con 0x7fa730060aa0 2022-07-21T01:39:17.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa736559700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa730138310 con 0x7fa7300b6a90 2022-07-21T01:39:17.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa736559700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa730151040 con 0x7fa7300aa540 2022-07-21T01:39:17.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 0x7fa730063dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 0x7fa730063dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47136/0 (socket says 172.21.15.3:47136) 2022-07-21T01:39:17.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72ffff700 1 -- 172.21.15.3:0/79330135 learned_addr learned my addr 172.21.15.3:0/79330135 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:17.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa727fff700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300aa540 0x7fa730140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa734af6700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300b6a90 0x7fa730064550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72ffff700 1 -- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300aa540 msgr2=0x7fa730140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72ffff700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300aa540 0x7fa730140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72ffff700 1 -- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300b6a90 msgr2=0x7fa730064550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:17.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72ffff700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300b6a90 0x7fa730064550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:17.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72ffff700 1 -- 172.21.15.3:0/79330135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71800b1e0 con 0x7fa730060aa0 2022-07-21T01:39:17.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa727fff700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300aa540 0x7fa730140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:17.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa734af6700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300b6a90 0x7fa730064550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:17.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72ffff700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 0x7fa730063dd0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7fa72000c000 tx=0x7fa720006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72dffb700 1 -- 172.21.15.3:0/79330135 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa72000c830 con 0x7fa730060aa0 2022-07-21T01:39:17.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72dffb700 1 -- 172.21.15.3:0/79330135 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72000c990 con 0x7fa730060aa0 2022-07-21T01:39:17.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7301512e0 con 0x7fa730060aa0 2022-07-21T01:39:17.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa72dffb700 1 -- 172.21.15.3:0/79330135 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa72000ccf0 con 0x7fa730060aa0 2022-07-21T01:39:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7300631c0 con 0x7fa730060aa0 2022-07-21T01:39:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.856+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa730063bd0 con 0x7fa730060aa0 2022-07-21T01:39:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.860+0000 7fa72dffb700 1 -- 172.21.15.3:0/79330135 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa72001c460 con 0x7fa730060aa0 2022-07-21T01:39:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.860+0000 7fa72dffb700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa714053950 0x7fa714055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.860+0000 7fa72dffb700 1 -- 172.21.15.3:0/79330135 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa72006c510 con 0x7fa730060aa0 2022-07-21T01:39:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.860+0000 7fa727fff700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa714053950 0x7fa714055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.860+0000 7fa727fff700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa714053950 0x7fa714055e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fa71801a000 tx=0x7fa718015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:17.868+0000 7fa72dffb700 1 -- 172.21.15.3:0/79330135 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa720043270 con 0x7fa730060aa0 2022-07-21T01:39:18.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.032+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa730061b50 con 0x7fa730060aa0 2022-07-21T01:39:18.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.032+0000 7fa72dffb700 1 -- 172.21.15.3:0/79330135 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2487 (secure 0 0 0) 0x7fa720046890 con 0x7fa730060aa0 2022-07-21T01:39:18.033 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:18.033 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:39:09.622528+0000","modified":"2022-07-21T01:39:10.626387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6259":{"gid":6259,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/862725488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":862725488},{"type":"v1","addr":"172.21.15.143:6833","nonce":862725488}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-21T01:39:18.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa714053950 msgr2=0x7fa714055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:18.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa714053950 0x7fa714055e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fa71801a000 tx=0x7fa718015000).stop 2022-07-21T01:39:18.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 msgr2=0x7fa730063dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:18.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 0x7fa730063dd0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7fa72000c000 tx=0x7fa720006df0).stop 2022-07-21T01:39:18.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 shutdown_connections 2022-07-21T01:39:18.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa714053950 0x7fa714055e10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:18.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7300b6a90 0x7fa730064550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:18.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7300aa540 0x7fa730140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:18.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 --2- 172.21.15.3:0/79330135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa730060aa0 0x7fa730063dd0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:18.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 >> 172.21.15.3:0/79330135 conn(0x7fa7300b3610 msgr2=0x7fa7301396a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:18.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 shutdown_connections 2022-07-21T01:39:18.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:18.036+0000 7fa736559700 1 -- 172.21.15.3:0/79330135 wait complete. 2022-07-21T01:39:18.043 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2022-07-21T01:39:18.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:39:09.622528+0000', 'modified': '2022-07-21T01:39:10.626387+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6259': {'gid': 6259, 'name': 'b', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6833/862725488', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 862725488}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 862725488}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:39:18.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:39:18.351 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-21T01:39:18.468 INFO:teuthology.orchestra.run.smithi143.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6259,"want_state":"up:active","state":"up:active","rank_uptime":8.8344127659999998,"mdsmap_epoch":35,"osdmap_epoch":79,"osdmap_epoch_barrier":78,"uptime":13.886473383} 2022-07-21T01:39:18.468 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6259, "mdsmap_epoch": 35, "osdmap_epoch": 79, "osdmap_epoch_barrier": 78, "rank_uptime": 8.834412766, "state": "up:active", "uptime": 13.886473383, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:39:18.469 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:39:18.469 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:39:18.470 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:39:18.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:18.471 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:39:18.480 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:39:18.496 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:39:18.497 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:39:18.556 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:39:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:39:18.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:18.560 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:39:18.566 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:39:18.567 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:39:18.611 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:39:18.629 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:39:18.630 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:39:18.640 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:39:18.642 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:39:18.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:18.643 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:39:18.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:39:18.686+0000 7feadde30080 -1 init, newargv = 0x55c7b91369c0 newargc=9ceph-fuse[22433]: starting ceph client 2022-07-21T01:39:18.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr: 2022-07-21T01:39:18.696 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22433]: starting fuse 2022-07-21T01:39:19.765 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:39:19.789 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:39:19.790 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:39:19.848 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:39:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:39:19.852 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:19.852 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:39:19.863 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:39:19.864 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:39:19.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:19.866 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:39:19.867 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:39:19.868 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:39:19.868 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:39:19.869 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:39:19.870 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:39:19.871 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:39:19.872 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:39:19.873 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:39:19.874 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:39:19.875 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:39:19.875 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:39:19.876 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:39:19.877 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:39:19.878 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:39:19.878 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:39:19.879 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:39:19.880 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:39:19.880 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:39:19.881 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:39:19.882 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:39:19.883 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:39:19.884 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:39:19.953 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.22433.asok 2022-07-21T01:39:19.957 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22433.asok 2022-07-21T01:39:19.958 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:19.958 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22433.asok status 2022-07-21T01:39:20.052 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:39:20.053 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:39:20.054 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:39:20.054 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:39:20.054 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:39:20.054 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:39:20.055 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:39:20.055 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "22433", 2022-07-21T01:39:20.055 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:39:20.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:39:20.056 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:39:20.056 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:39:20.056 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 6451, 2022-07-21T01:39:20.056 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:39:20.057 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:39:20.057 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:39:20.057 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 6451 2022-07-21T01:39:20.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:39:20.058 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:39:20.058 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:39:20.058 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:39:20.058 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3316948089 2022-07-21T01:39:20.059 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:39:20.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:39:20.059 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:39:20.059 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:39:20.059 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:39:20.060 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3316948089 2022-07-21T01:39:20.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:39:20.060 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.6451 172.21.15.174:0/3316948089", 2022-07-21T01:39:20.060 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/3316948089", 2022-07-21T01:39:20.061 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:39:20.061 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 35, 2022-07-21T01:39:20.061 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 79, 2022-07-21T01:39:20.061 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:39:20.061 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:39:20.062 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:39:20.062 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:39:20.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:20.064 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:39:20.064 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:39:20.074 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:39:20.075 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:39:20.076 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:20.077 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:39:20.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:39:20.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.220+0000 7fcb16cf2700 1 Processor -- start 2022-07-21T01:39:20.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.220+0000 7fcb16cf2700 1 -- start start 2022-07-21T01:39:20.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1160 0x7fcb101448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb100b04e0 con 0x7fcb10148530 2022-07-21T01:39:20.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb100b0780 con 0x7fcb100b1f40 2022-07-21T01:39:20.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100a9250 con 0x7fcb100b1160 2022-07-21T01:39:20.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcb10148530 0x7fcb100a7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55424/0 (socket says 172.21.15.3:55424) 2022-07-21T01:39:20.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0ffff700 1 -- 172.21.15.3:0/1291468022 learned_addr learned my addr 172.21.15.3:0/1291468022 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:20.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb1528f700 1 --2- 172.21.15.3:0/1291468022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1160 0x7fcb101448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0f7fe700 1 -- 172.21.15.3:0/1291468022 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851003220 0 0) 0x7fcb100b04e0 con 0x7fcb10148530 2022-07-21T01:39:20.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0f7fe700 1 -- 172.21.15.3:0/1291468022 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcafc003260 con 0x7fcb10148530 2022-07-21T01:39:20.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0f7fe700 1 -- 172.21.15.3:0/1291468022 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981984919 0 0) 0x7fcb100b0780 con 0x7fcb100b1f40 2022-07-21T01:39:20.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0f7fe700 1 -- 172.21.15.3:0/1291468022 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb100b04e0 con 0x7fcb100b1f40 2022-07-21T01:39:20.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0f7fe700 1 -- 172.21.15.3:0/1291468022 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447155762 0 0) 0x7fcafc003260 con 0x7fcb10148530 2022-07-21T01:39:20.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb1528f700 1 -- 172.21.15.3:0/1291468022 >> v1:172.21.15.143:6789/0 conn(0x7fcb100b1f40 legacy=0x7fcb100b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb1528f700 1 -- 172.21.15.3:0/1291468022 >> v1:172.21.15.3:6789/0 conn(0x7fcb10148530 legacy=0x7fcb100a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb1528f700 1 -- 172.21.15.3:0/1291468022 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100b0a20 con 0x7fcb100b1160 2022-07-21T01:39:20.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb1528f700 1 --2- 172.21.15.3:0/1291468022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1160 0x7fcb101448b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fcb101478c0 tx=0x7fcb08006df0).ready entity=mon.2 client_cookie=596836fb2c43f982 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:20.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0f7fe700 1 -- 172.21.15.3:0/1291468022 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb000019e0 con 0x7fcb100b1160 2022-07-21T01:39:20.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0f7fe700 1 -- 172.21.15.3:0/1291468022 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb08000e90 con 0x7fcb100b1160 2022-07-21T01:39:20.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb0f7fe700 1 -- 172.21.15.3:0/1291468022 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb08012460 con 0x7fcb100b1160 2022-07-21T01:39:20.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 -- 172.21.15.3:0/1291468022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1160 msgr2=0x7fcb101448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 --2- 172.21.15.3:0/1291468022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1160 0x7fcb101448b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fcb101478c0 tx=0x7fcb08006df0).stop 2022-07-21T01:39:20.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 -- 172.21.15.3:0/1291468022 shutdown_connections 2022-07-21T01:39:20.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 --2- 172.21.15.3:0/1291468022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1160 0x7fcb101448b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 -- 172.21.15.3:0/1291468022 >> 172.21.15.3:0/1291468022 conn(0x7fcb1005f840 msgr2=0x7fcb1005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:20.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 -- 172.21.15.3:0/1291468022 shutdown_connections 2022-07-21T01:39:20.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.224+0000 7fcb16cf2700 1 -- 172.21.15.3:0/1291468022 wait complete. 2022-07-21T01:39:20.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb16cf2700 1 Processor -- start 2022-07-21T01:39:20.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb16cf2700 1 -- start start 2022-07-21T01:39:20.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb16cf2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb100b1160 0x7fcb1014b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb16cf2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1f40 0x7fcb100a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb16cf2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb10148530 0x7fcb100a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb16cf2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100ab3c0 con 0x7fcb10148530 2022-07-21T01:39:20.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb16cf2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100ab660 con 0x7fcb100b1160 2022-07-21T01:39:20.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb16cf2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100ab900 con 0x7fcb100b1f40 2022-07-21T01:39:20.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb1528f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb10148530 0x7fcb100a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb1528f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb10148530 0x7fcb100a9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47152/0 (socket says 172.21.15.3:47152) 2022-07-21T01:39:20.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb1528f700 1 -- 172.21.15.3:0/669669238 learned_addr learned my addr 172.21.15.3:0/669669238 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:20.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb0ffff700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1f40 0x7fcb100a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb14a8e700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb100b1160 0x7fcb1014b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb1528f700 1 -- 172.21.15.3:0/669669238 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1f40 msgr2=0x7fcb100a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb1528f700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1f40 0x7fcb100a95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb1528f700 1 -- 172.21.15.3:0/669669238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb100b1160 msgr2=0x7fcb1014b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb1528f700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb100b1160 0x7fcb1014b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb1528f700 1 -- 172.21.15.3:0/669669238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0800b180 con 0x7fcb10148530 2022-07-21T01:39:20.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb14a8e700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb100b1160 0x7fcb1014b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:39:20.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb1528f700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb10148530 0x7fcb100a9ac0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fcb101478c0 tx=0x7fcb08014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:20.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb0dffb700 1 -- 172.21.15.3:0/669669238 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb0800d7d0 con 0x7fcb10148530 2022-07-21T01:39:20.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb0dffb700 1 -- 172.21.15.3:0/669669238 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb08017070 con 0x7fcb10148530 2022-07-21T01:39:20.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb100abba0 con 0x7fcb10148530 2022-07-21T01:39:20.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.228+0000 7fcb0dffb700 1 -- 172.21.15.3:0/669669238 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb0800db20 con 0x7fcb10148530 2022-07-21T01:39:20.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.232+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb100ac020 con 0x7fcb10148530 2022-07-21T01:39:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.232+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb100574e0 con 0x7fcb10148530 2022-07-21T01:39:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.232+0000 7fcb0dffb700 1 -- 172.21.15.3:0/669669238 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb08025020 con 0x7fcb10148530 2022-07-21T01:39:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.232+0000 7fcb0dffb700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcaf0053a70 0x7fcaf0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.232+0000 7fcb0dffb700 1 -- 172.21.15.3:0/669669238 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcb0806d160 con 0x7fcb10148530 2022-07-21T01:39:20.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.232+0000 7fcb14a8e700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcaf0053a70 0x7fcaf0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.232+0000 7fcb14a8e700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcaf0053a70 0x7fcaf0055f30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fcb0400b000 tx=0x7fcb04009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:20.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.240+0000 7fcb0dffb700 1 -- 172.21.15.3:0/669669238 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb08043f20 con 0x7fcb10148530 2022-07-21T01:39:20.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.408+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb100b3fc0 con 0x7fcb10148530 2022-07-21T01:39:20.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.408+0000 7fcb0dffb700 1 -- 172.21.15.3:0/669669238 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2487 (secure 0 0 0) 0x7fcb08047540 con 0x7fcb10148530 2022-07-21T01:39:20.410 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:20.410 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:39:09.622528+0000","modified":"2022-07-21T01:39:10.626387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6259":{"gid":6259,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/862725488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":862725488},{"type":"v1","addr":"172.21.15.143:6833","nonce":862725488}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-21T01:39:20.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcaf0053a70 msgr2=0x7fcaf0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcaf0053a70 0x7fcaf0055f30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fcb0400b000 tx=0x7fcb04009040).stop 2022-07-21T01:39:20.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb10148530 msgr2=0x7fcb100a9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb10148530 0x7fcb100a9ac0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fcb101478c0 tx=0x7fcb08014000).stop 2022-07-21T01:39:20.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 shutdown_connections 2022-07-21T01:39:20.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb10148530 0x7fcb100a9ac0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb100b1f40 0x7fcb100a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcaf0053a70 0x7fcaf0055f30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 --2- 172.21.15.3:0/669669238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb100b1160 0x7fcb1014b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 >> 172.21.15.3:0/669669238 conn(0x7fcb1005f840 msgr2=0x7fcb100b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 shutdown_connections 2022-07-21T01:39:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.412+0000 7fcb16cf2700 1 -- 172.21.15.3:0/669669238 wait complete. 2022-07-21T01:39:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2022-07-21T01:39:20.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:39:20.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.520+0000 7f51538ed700 1 Processor -- start 2022-07-21T01:39:20.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.520+0000 7f51538ed700 1 -- start start 2022-07-21T01:39:20.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f514c0b6a90 0x7f514c0acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f514c060aa0 0x7f514c0aa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c0b6e70 con 0x7f514c05c620 2022-07-21T01:39:20.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c061370 con 0x7f514c0b6a90 2022-07-21T01:39:20.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c061610 con 0x7f514c060aa0 2022-07-21T01:39:20.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5151689700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f514c0b6a90 0x7f514c0acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5151e8a700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f514c05c620 0x7f514c0aa540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55442/0 (socket says 172.21.15.3:55442) 2022-07-21T01:39:20.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5151e8a700 1 -- 172.21.15.3:0/814574845 learned_addr learned my addr 172.21.15.3:0/814574845 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:39:20.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5150e88700 1 --2- 172.21.15.3:0/814574845 >> v2:172.21.15.143:3300/0 conn(0x7f514c060aa0 0x7f514c0aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f513bfff700 1 -- 172.21.15.3:0/814574845 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625654383 0 0) 0x7f514c0b6e70 con 0x7f514c05c620 2022-07-21T01:39:20.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f513bfff700 1 -- 172.21.15.3:0/814574845 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5134003260 con 0x7f514c05c620 2022-07-21T01:39:20.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5150e88700 1 -- 172.21.15.3:0/814574845 >> v2:172.21.15.3:3300/0 conn(0x7f514c0b6a90 msgr2=0x7f514c0acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:39:20.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5150e88700 1 --2- 172.21.15.3:0/814574845 >> v2:172.21.15.3:3300/0 conn(0x7f514c0b6a90 0x7f514c0acee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5150e88700 1 -- 172.21.15.3:0/814574845 >> v1:172.21.15.3:6789/0 conn(0x7f514c05c620 legacy=0x7f514c0aa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5150e88700 1 -- 172.21.15.3:0/814574845 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c0618b0 con 0x7f514c060aa0 2022-07-21T01:39:20.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5151689700 1 --2- 172.21.15.3:0/814574845 >> v2:172.21.15.3:3300/0 conn(0x7f514c0b6a90 0x7f514c0acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:39:20.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f5150e88700 1 --2- 172.21.15.3:0/814574845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c060aa0 0x7f514c0aa050 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f513c002f70 tx=0x7f513c006df0).ready entity=mon.1 client_cookie=2418b59b1415856 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:20.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f513bfff700 1 -- 172.21.15.3:0/814574845 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f513c00d4f0 con 0x7f514c060aa0 2022-07-21T01:39:20.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f513bfff700 1 -- 172.21.15.3:0/814574845 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c000e90 con 0x7f514c060aa0 2022-07-21T01:39:20.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f513bfff700 1 -- 172.21.15.3:0/814574845 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f513c00d840 con 0x7f514c060aa0 2022-07-21T01:39:20.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 -- 172.21.15.3:0/814574845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c060aa0 msgr2=0x7f514c0aa050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 --2- 172.21.15.3:0/814574845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c060aa0 0x7f514c0aa050 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f513c002f70 tx=0x7f513c006df0).stop 2022-07-21T01:39:20.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 -- 172.21.15.3:0/814574845 shutdown_connections 2022-07-21T01:39:20.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 --2- 172.21.15.3:0/814574845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c060aa0 0x7f514c0aa050 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0x7f513c002f70 tx=0x7f513c006df0).stop 2022-07-21T01:39:20.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 --2- 172.21.15.3:0/814574845 >> v2:172.21.15.3:3300/0 conn(0x7f514c0b6a90 0x7f514c0acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 -- 172.21.15.3:0/814574845 >> 172.21.15.3:0/814574845 conn(0x7f514c0b3610 msgr2=0x7f514c0b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:20.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 -- 172.21.15.3:0/814574845 shutdown_connections 2022-07-21T01:39:20.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.524+0000 7f51538ed700 1 -- 172.21.15.3:0/814574845 wait complete. 2022-07-21T01:39:20.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 Processor -- start 2022-07-21T01:39:20.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 -- start start 2022-07-21T01:39:20.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c05c620 0x7f514c153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c0b6a90 0x7f514c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c1432d0 0x7f514c0adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c1429b0 con 0x7f514c0b6a90 2022-07-21T01:39:20.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c063260 con 0x7f514c1432d0 2022-07-21T01:39:20.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c1542d0 con 0x7f514c05c620 2022-07-21T01:39:20.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5150e88700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c0b6a90 0x7f514c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5150e88700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c0b6a90 0x7f514c0afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47166/0 (socket says 172.21.15.3:47166) 2022-07-21T01:39:20.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5150e88700 1 -- 172.21.15.3:0/1400462196 learned_addr learned my addr 172.21.15.3:0/1400462196 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:39:20.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5151e8a700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c1432d0 0x7f514c0adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5151689700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c05c620 0x7f514c153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5150e88700 1 -- 172.21.15.3:0/1400462196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c05c620 msgr2=0x7f514c153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5150e88700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c05c620 0x7f514c153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5150e88700 1 -- 172.21.15.3:0/1400462196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c1432d0 msgr2=0x7f514c0adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5150e88700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c1432d0 0x7f514c0adc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5150e88700 1 -- 172.21.15.3:0/1400462196 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514800b180 con 0x7f514c0b6a90 2022-07-21T01:39:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5151689700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c05c620 0x7f514c153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:39:20.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f5150e88700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c0b6a90 0x7f514c0afc60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f513c002f70 tx=0x7f513c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:20.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f513a7fc700 1 -- 172.21.15.3:0/1400462196 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f513c016070 con 0x7f514c0b6a90 2022-07-21T01:39:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f513a7fc700 1 -- 172.21.15.3:0/1400462196 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c013a20 con 0x7f514c0b6a90 2022-07-21T01:39:20.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c00b180 con 0x7f514c0b6a90 2022-07-21T01:39:20.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f513a7fc700 1 -- 172.21.15.3:0/1400462196 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f513c013d80 con 0x7f514c0b6a90 2022-07-21T01:39:20.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.528+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c0b0150 con 0x7f514c0b6a90 2022-07-21T01:39:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.532+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c143070 con 0x7f514c0b6a90 2022-07-21T01:39:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.532+0000 7f513a7fc700 1 -- 172.21.15.3:0/1400462196 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f513c00f030 con 0x7f514c0b6a90 2022-07-21T01:39:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.532+0000 7f513a7fc700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f512c053aa0 0x7f512c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:39:20.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.532+0000 7f513a7fc700 1 -- 172.21.15.3:0/1400462196 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f513c06ccb0 con 0x7f514c0b6a90 2022-07-21T01:39:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.532+0000 7f5151689700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f512c053aa0 0x7f512c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:39:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.532+0000 7f5151689700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f512c053aa0 0x7f512c055f60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f514800b000 tx=0x7f5148008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:39:20.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.536+0000 7f513a7fc700 1 -- 172.21.15.3:0/1400462196 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f513c0439d0 con 0x7f514c0b6a90 2022-07-21T01:39:20.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.696+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f514c061b50 con 0x7f514c0b6a90 2022-07-21T01:39:20.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.696+0000 7f513a7fc700 1 -- 172.21.15.3:0/1400462196 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2487 (secure 0 0 0) 0x7f513c023020 con 0x7f514c0b6a90 2022-07-21T01:39:20.697 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:39:20.697 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:39:09.622528+0000","modified":"2022-07-21T01:39:10.626387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6259":{"gid":6259,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/862725488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":862725488},{"type":"v1","addr":"172.21.15.143:6833","nonce":862725488}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-21T01:39:20.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f512c053aa0 msgr2=0x7f512c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f512c053aa0 0x7f512c055f60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f514800b000 tx=0x7f5148008040).stop 2022-07-21T01:39:20.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c0b6a90 msgr2=0x7f514c0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:39:20.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c0b6a90 0x7f514c0afc60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f513c002f70 tx=0x7f513c014000).stop 2022-07-21T01:39:20.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 shutdown_connections 2022-07-21T01:39:20.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f512c053aa0 0x7f512c055f60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c1432d0 0x7f514c0adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c0b6a90 0x7f514c0afc60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 --2- 172.21.15.3:0/1400462196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c05c620 0x7f514c153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:39:20.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 >> 172.21.15.3:0/1400462196 conn(0x7f514c0b3610 msgr2=0x7f514c0b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:39:20.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 shutdown_connections 2022-07-21T01:39:20.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:39:20.700+0000 7f51538ed700 1 -- 172.21.15.3:0/1400462196 wait complete. 2022-07-21T01:39:20.704 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2022-07-21T01:39:20.711 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_cap_revoke_eviction_timeout 30.0 2022-07-21T01:39:20.827 INFO:teuthology.orchestra.run.smithi143.stdout:{"success":"mds_cap_revoke_eviction_timeout = '30.000000' "} 2022-07-21T01:39:20.828 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_cap_revoke_eviction_timeout = '30.000000' " } 2022-07-21T01:39:20.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:20.829 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:39:20.830 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:39:20.839 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:39:20.843 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:39:20.845 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-21T01:39:20.845 DEBUG:teuthology.orchestra.run.smithi171:> import time 2022-07-21T01:39:20.846 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:39:20.847 DEBUG:teuthology.orchestra.run.smithi171:> with open("/home/ubuntu/cephtest/mnt.0/background_file", '"'"'w'"'"') as f: 2022-07-21T01:39:20.848 DEBUG:teuthology.orchestra.run.smithi171:> f.write('"'"'content'"'"') 2022-07-21T01:39:20.848 DEBUG:teuthology.orchestra.run.smithi171:> f.flush() 2022-07-21T01:39:20.849 DEBUG:teuthology.orchestra.run.smithi171:> f.write('"'"'content2'"'"') 2022-07-21T01:39:20.850 DEBUG:teuthology.orchestra.run.smithi171:> while True: 2022-07-21T01:39:20.851 DEBUG:teuthology.orchestra.run.smithi171:> time.sleep(1) 2022-07-21T01:39:20.851 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:39:20.857 DEBUG:teuthology.orchestra.run.smithi171:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-07-21T01:39:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-07-21T01:39:20.881 INFO:teuthology.orchestra.run.smithi171.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/background_file': No such file or directory 2022-07-21T01:39:21.882 DEBUG:teuthology.orchestra.run.smithi171:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-07-21T01:39:21.899 INFO:teuthology.orchestra.run.smithi171.stdout:/home/ubuntu/cephtest/mnt.0/background_file 2022-07-21T01:39:21.900 DEBUG:tasks.cephfs.mount:File background_file became visible from 0 after 1s 2022-07-21T01:39:21.901 DEBUG:teuthology.orchestra.run.smithi174:> sudo ls /home/ubuntu/cephtest/mnt.1/background_file 2022-07-21T01:39:21.942 INFO:teuthology.orchestra.run.smithi174.stdout:/home/ubuntu/cephtest/mnt.1/background_file 2022-07-21T01:39:21.944 DEBUG:tasks.cephfs.mount:File background_file became visible from 1 after 0s 2022-07-21T01:39:21.945 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:39:21.962 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:39:21.963 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:39:21.964 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:39:21.973 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:39:21.974 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:39:21.975 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-21T01:39:21.975 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:39:21.976 DEBUG:teuthology.orchestra.run.smithi174:> import time 2022-07-21T01:39:21.976 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:39:21.977 DEBUG:teuthology.orchestra.run.smithi174:> fd = os.open("/home/ubuntu/cephtest/mnt.1/background_file", os.O_RDWR | os.O_CREAT, 0o644) 2022-07-21T01:39:21.977 DEBUG:teuthology.orchestra.run.smithi174:> try: 2022-07-21T01:39:21.977 DEBUG:teuthology.orchestra.run.smithi174:> while True: 2022-07-21T01:39:21.978 DEBUG:teuthology.orchestra.run.smithi174:> os.write(fd, b'"'"'content'"'"') 2022-07-21T01:39:21.978 DEBUG:teuthology.orchestra.run.smithi174:> time.sleep(1) 2022-07-21T01:39:21.979 DEBUG:teuthology.orchestra.run.smithi174:> if not False: 2022-07-21T01:39:21.979 DEBUG:teuthology.orchestra.run.smithi174:> break 2022-07-21T01:39:21.979 DEBUG:teuthology.orchestra.run.smithi174:> except IOError as e: 2022-07-21T01:39:21.980 DEBUG:teuthology.orchestra.run.smithi174:> pass 2022-07-21T01:39:21.980 DEBUG:teuthology.orchestra.run.smithi174:> os.close(fd) 2022-07-21T01:39:21.980 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:39:32.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:39:32.043 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:39:32.046 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:39:32.049 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:40:00.744 INFO:tasks.cephfs.test_misc:cap_waiter waited 38.72602367401123s 2022-07-21T01:40:00.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist ls --format=json 2022-07-21T01:40:00.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.858+0000 7f5fade1f700 1 Processor -- start 2022-07-21T01:40:00.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.858+0000 7f5fade1f700 1 -- start start 2022-07-21T01:40:00.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.858+0000 7f5fade1f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b1410 0x7f5fa80b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:00.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.858+0000 7f5fade1f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa8140e80 0x7f5fa80b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:00.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5fa80a8e00 0x7f5fa80ae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:00.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa81404a0 con 0x7f5fa80a8e00 2022-07-21T01:40:00.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa8140740 con 0x7f5fa80b1410 2022-07-21T01:40:00.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa81409e0 con 0x7f5fa8140e80 2022-07-21T01:40:00.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5fa80a8e00 0x7f5fa80ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:00.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5fa80a8e00 0x7f5fa80ae840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47176/0 (socket says 172.21.15.3:47176) 2022-07-21T01:40:00.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa7fff700 1 -- 172.21.15.3:0/815596247 learned_addr learned my addr 172.21.15.3:0/815596247 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:00.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5f9ffff700 1 --2- 172.21.15.3:0/815596247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa8140e80 0x7f5fa80b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:00.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa77fe700 1 --2- 172.21.15.3:0/815596247 >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b1410 0x7f5fa80b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:00.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa7fff700 1 -- 172.21.15.3:0/815596247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa8140e80 msgr2=0x7f5fa80b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:00.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa7fff700 1 --2- 172.21.15.3:0/815596247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa8140e80 0x7f5fa80b22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:00.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa7fff700 1 -- 172.21.15.3:0/815596247 >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b1410 msgr2=0x7f5fa80b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:00.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa7fff700 1 --2- 172.21.15.3:0/815596247 >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b1410 0x7f5fa80b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:00.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa7fff700 1 -- 172.21.15.3:0/815596247 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa8140190 con 0x7f5fa80a8e00 2022-07-21T01:40:00.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5f9ffff700 1 --2- 172.21.15.3:0/815596247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa8140e80 0x7f5fa80b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:00.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa7fff700 1 --2- 172.21.15.3:0/815596247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 0x7f5fa80ae840 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f5fa00032f0 tx=0x7f5fa0009290).ready entity=mon.0 client_cookie=5fbf353985837994 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:00.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa6ffd700 1 -- 172.21.15.3:0/815596247 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fa000b870 con 0x7f5fa80a8e00 2022-07-21T01:40:00.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa6ffd700 1 -- 172.21.15.3:0/815596247 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa000b9d0 con 0x7f5fa80a8e00 2022-07-21T01:40:00.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fa6ffd700 1 -- 172.21.15.3:0/815596247 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fa000bb40 con 0x7f5fa80a8e00 2022-07-21T01:40:00.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 -- 172.21.15.3:0/815596247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 msgr2=0x7f5fa80ae840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:00.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 --2- 172.21.15.3:0/815596247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 0x7f5fa80ae840 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f5fa00032f0 tx=0x7f5fa0009290).stop 2022-07-21T01:40:00.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 -- 172.21.15.3:0/815596247 shutdown_connections 2022-07-21T01:40:00.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 --2- 172.21.15.3:0/815596247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 0x7f5fa80ae840 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:00.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 --2- 172.21.15.3:0/815596247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa8140e80 0x7f5fa80b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:00.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 --2- 172.21.15.3:0/815596247 >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b1410 0x7f5fa80b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:00.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 -- 172.21.15.3:0/815596247 >> 172.21.15.3:0/815596247 conn(0x7f5fa805f840 msgr2=0x7f5fa805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:00.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 -- 172.21.15.3:0/815596247 shutdown_connections 2022-07-21T01:40:00.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.862+0000 7f5fade1f700 1 -- 172.21.15.3:0/815596247 wait complete. 2022-07-21T01:40:00.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 Processor -- start 2022-07-21T01:40:00.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 -- start start 2022-07-21T01:40:00.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 0x7f5fa80ac920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:00.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b1410 0x7f5fa8066f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:00.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa8140e80 0x7f5fa813ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:00.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa8134b10 con 0x7f5fa80a8e00 2022-07-21T01:40:00.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80ab300 con 0x7f5fa8140e80 2022-07-21T01:40:00.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80ab5a0 con 0x7f5fa80b1410 2022-07-21T01:40:00.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 0x7f5fa80ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:00.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 0x7f5fa80ac920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47178/0 (socket says 172.21.15.3:47178) 2022-07-21T01:40:00.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa77fe700 1 -- 172.21.15.3:0/2566343902 learned_addr learned my addr 172.21.15.3:0/2566343902 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:00.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5f9ffff700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b1410 0x7f5fa8066f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:00.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa7fff700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa8140e80 0x7f5fa813ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:00.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa77fe700 1 -- 172.21.15.3:0/2566343902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b1410 msgr2=0x7f5fa8066f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:00.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa77fe700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b1410 0x7f5fa8066f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:00.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa77fe700 1 -- 172.21.15.3:0/2566343902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa8140e80 msgr2=0x7f5fa813ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:00.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa77fe700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa8140e80 0x7f5fa813ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:00.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa77fe700 1 -- 172.21.15.3:0/2566343902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0009530 con 0x7f5fa80a8e00 2022-07-21T01:40:00.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa77fe700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 0x7f5fa80ac920 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f5f980030b0 tx=0x7f5f98008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:00.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa57fa700 1 -- 172.21.15.3:0/2566343902 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f98011780 con 0x7f5fa80a8e00 2022-07-21T01:40:00.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa57fa700 1 -- 172.21.15.3:0/2566343902 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f980118e0 con 0x7f5fa80a8e00 2022-07-21T01:40:00.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa80ab840 con 0x7f5fa80a8e00 2022-07-21T01:40:00.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fa57fa700 1 -- 172.21.15.3:0/2566343902 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f98011c40 con 0x7f5fa80a8e00 2022-07-21T01:40:00.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.866+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa80673f0 con 0x7f5fa80a8e00 2022-07-21T01:40:00.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.870+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa80abce0 con 0x7f5fa80a8e00 2022-07-21T01:40:00.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.870+0000 7f5fa57fa700 1 -- 172.21.15.3:0/2566343902 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f9801c460 con 0x7f5fa80a8e00 2022-07-21T01:40:00.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.870+0000 7f5fa57fa700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f8c053980 0x7f5f8c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:00.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.870+0000 7f5fa57fa700 1 -- 172.21.15.3:0/2566343902 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f5f9806c120 con 0x7f5fa80a8e00 2022-07-21T01:40:00.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.870+0000 7f5f9ffff700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f8c053980 0x7f5f8c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:00.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.874+0000 7f5f9ffff700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f8c053980 0x7f5f8c055e40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5f9000c000 tx=0x7f5f90008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:00.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:00.878+0000 7f5fa57fa700 1 -- 172.21.15.3:0/2566343902 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5fa80abce0 con 0x7f5fa80a8e00 2022-07-21T01:40:01.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.014+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist ls", "format": "json"} v 0) v1 -- 0x7f5fa80abce0 con 0x7f5fa80a8e00 2022-07-21T01:40:01.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fa57fa700 1 -- 172.21.15.3:0/2566343902 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist ls", "format": "json"}]=0 listed 1 entries v80) v1 ==== 98+0+80 (secure 0 0 0) 0x7f5fa80abce0 con 0x7f5fa80a8e00 2022-07-21T01:40:01.020 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:01.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f8c053980 msgr2=0x7f5f8c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f8c053980 0x7f5f8c055e40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5f9000c000 tx=0x7f5f90008040).stop 2022-07-21T01:40:01.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 msgr2=0x7f5fa80ac920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 0x7f5fa80ac920 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f5f980030b0 tx=0x7f5f98008040).stop 2022-07-21T01:40:01.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 shutdown_connections 2022-07-21T01:40:01.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f8c053980 0x7f5f8c055e40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa8140e80 0x7f5fa813ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b1410 0x7f5fa8066f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 --2- 172.21.15.3:0/2566343902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80a8e00 0x7f5fa80ac920 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 >> 172.21.15.3:0/2566343902 conn(0x7f5fa805f840 msgr2=0x7f5fa80b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:01.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 shutdown_connections 2022-07-21T01:40:01.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.018+0000 7f5fade1f700 1 -- 172.21.15.3:0/2566343902 wait complete. 2022-07-21T01:40:01.029 INFO:teuthology.orchestra.run.smithi003.stderr:listed 1 entries 2022-07-21T01:40:01.039 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr":"172.21.15.171:0/752832426","until":"2022-07-21T02:39:54.585234+0000"}] 2022-07-21T01:40:01.055 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-21T01:40:01.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:01.057 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:01.057 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:01.064 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-21T01:40:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:40:01.066 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:01.067 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:40:01.067 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:40:01.067 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:01.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:01.082 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:01.082 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:01.129 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:40:01.130 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:40:01.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:40:01.131 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:01.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:01.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.250+0000 7f15a1a7b700 1 Processor -- start 2022-07-21T01:40:01.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.250+0000 7f15a1a7b700 1 -- start start 2022-07-21T01:40:01.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.250+0000 7f15a1a7b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f159c0af450 0x7f159c0af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.250+0000 7f15a1a7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0ad4b0 0x7f159c1432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.250+0000 7f15a1a7b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c0aff30 con 0x7f159c1437d0 2022-07-21T01:40:01.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.250+0000 7f15a1a7b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159c05c7a0 con 0x7f159c0af450 2022-07-21T01:40:01.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.250+0000 7f15a1a7b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c140990 con 0x7f159c0ad4b0 2022-07-21T01:40:01.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a0a79700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f159c0af450 0x7f159c0af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a127a700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f159c1437d0 0x7f159c0a8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55470/0 (socket says 172.21.15.3:55470) 2022-07-21T01:40:01.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a127a700 1 -- 172.21.15.3:0/4251759772 learned_addr learned my addr 172.21.15.3:0/4251759772 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:01.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f159bfff700 1 --2- 172.21.15.3:0/4251759772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0ad4b0 0x7f159c1432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f159b7fe700 1 -- 172.21.15.3:0/4251759772 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052785426 0 0) 0x7f159c0aff30 con 0x7f159c1437d0 2022-07-21T01:40:01.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f159b7fe700 1 -- 172.21.15.3:0/4251759772 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1588003260 con 0x7f159c1437d0 2022-07-21T01:40:01.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a0a79700 1 -- 172.21.15.3:0/4251759772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0ad4b0 msgr2=0x7f159c1432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:01.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a0a79700 1 --2- 172.21.15.3:0/4251759772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0ad4b0 0x7f159c1432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a0a79700 1 -- 172.21.15.3:0/4251759772 >> v1:172.21.15.3:6789/0 conn(0x7f159c1437d0 legacy=0x7f159c0a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a0a79700 1 -- 172.21.15.3:0/4251759772 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c0a9550 con 0x7f159c0af450 2022-07-21T01:40:01.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a0a79700 1 --2- 172.21.15.3:0/4251759772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0af450 0x7f159c0af830 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f159c13fdc0 tx=0x7f158c006df0).ready entity=mon.0 client_cookie=9ca7dc4b2e42bc8f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:01.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f159b7fe700 1 -- 172.21.15.3:0/4251759772 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f158c00d570 con 0x7f159c0af450 2022-07-21T01:40:01.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f159b7fe700 1 -- 172.21.15.3:0/4251759772 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c000e90 con 0x7f159c0af450 2022-07-21T01:40:01.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f159b7fe700 1 -- 172.21.15.3:0/4251759772 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f158c00d8c0 con 0x7f159c0af450 2022-07-21T01:40:01.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a1a7b700 1 -- 172.21.15.3:0/4251759772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0af450 msgr2=0x7f159c0af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a1a7b700 1 --2- 172.21.15.3:0/4251759772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0af450 0x7f159c0af830 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f159c13fdc0 tx=0x7f158c006df0).stop 2022-07-21T01:40:01.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a1a7b700 1 -- 172.21.15.3:0/4251759772 shutdown_connections 2022-07-21T01:40:01.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a1a7b700 1 --2- 172.21.15.3:0/4251759772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0ad4b0 0x7f159c1432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a1a7b700 1 --2- 172.21.15.3:0/4251759772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0af450 0x7f159c0af830 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a1a7b700 1 -- 172.21.15.3:0/4251759772 >> 172.21.15.3:0/4251759772 conn(0x7f159c0612d0 msgr2=0x7f159c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:01.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a1a7b700 1 -- 172.21.15.3:0/4251759772 shutdown_connections 2022-07-21T01:40:01.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.254+0000 7f15a1a7b700 1 -- 172.21.15.3:0/4251759772 wait complete. 2022-07-21T01:40:01.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 Processor -- start 2022-07-21T01:40:01.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 -- start start 2022-07-21T01:40:01.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0ad4b0 0x7f159c064820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0af450 0x7f159c067430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c1437d0 0x7f159c067920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c0b5400 con 0x7f159c0ad4b0 2022-07-21T01:40:01.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c0ac730 con 0x7f159c1437d0 2022-07-21T01:40:01.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c0b4830 con 0x7f159c0af450 2022-07-21T01:40:01.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a0a79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0ad4b0 0x7f159c064820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a0a79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0ad4b0 0x7f159c064820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47192/0 (socket says 172.21.15.3:47192) 2022-07-21T01:40:01.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a0a79700 1 -- 172.21.15.3:0/704924101 learned_addr learned my addr 172.21.15.3:0/704924101 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:01.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f159bfff700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0af450 0x7f159c067430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a127a700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c1437d0 0x7f159c067920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a0a79700 1 -- 172.21.15.3:0/704924101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0af450 msgr2=0x7f159c067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a0a79700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0af450 0x7f159c067430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a0a79700 1 -- 172.21.15.3:0/704924101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c1437d0 msgr2=0x7f159c067920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a0a79700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c1437d0 0x7f159c067920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a0a79700 1 -- 172.21.15.3:0/704924101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c00b180 con 0x7f159c0ad4b0 2022-07-21T01:40:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f159bfff700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0af450 0x7f159c067430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a0a79700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0ad4b0 0x7f159c064820 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7f159c13fdc0 tx=0x7f158c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f1599ffb700 1 -- 172.21.15.3:0/704924101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f158c016070 con 0x7f159c0ad4b0 2022-07-21T01:40:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f1599ffb700 1 -- 172.21.15.3:0/704924101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c013a30 con 0x7f159c0ad4b0 2022-07-21T01:40:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159c067e10 con 0x7f159c0ad4b0 2022-07-21T01:40:01.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f1599ffb700 1 -- 172.21.15.3:0/704924101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f158c013d90 con 0x7f159c0ad4b0 2022-07-21T01:40:01.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.258+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159c0b3b90 con 0x7f159c0ad4b0 2022-07-21T01:40:01.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.262+0000 7f1599ffb700 1 -- 172.21.15.3:0/704924101 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f158c00f030 con 0x7f159c0ad4b0 2022-07-21T01:40:01.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.262+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159c065050 con 0x7f159c0ad4b0 2022-07-21T01:40:01.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.262+0000 7f1599ffb700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f157c053a70 0x7f157c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.262+0000 7f1599ffb700 1 -- 172.21.15.3:0/704924101 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f158c06cf00 con 0x7f159c0ad4b0 2022-07-21T01:40:01.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.262+0000 7f159bfff700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f157c053a70 0x7f157c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.262+0000 7f159bfff700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f157c053a70 0x7f157c055f30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f159000c000 tx=0x7f1590008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:01.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.266+0000 7f1599ffb700 1 -- 172.21.15.3:0/704924101 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f158c043ba0 con 0x7f159c0ad4b0 2022-07-21T01:40:01.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.394+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f159c14ebc0 con 0x7f159c0ad4b0 2022-07-21T01:40:01.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.394+0000 7f1599ffb700 1 -- 172.21.15.3:0/704924101 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16335 (secure 0 0 0) 0x7f158c023090 con 0x7f159c0ad4b0 2022-07-21T01:40:01.399 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:01.399 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":80,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:39:54.719898+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:39:07.362717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-21T01:39:08.613529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.171:0/752832426":"2022-07-21T02:39:54.585234+0000"},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:01.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f157c053a70 msgr2=0x7f157c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f157c053a70 0x7f157c055f30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f159000c000 tx=0x7f1590008040).stop 2022-07-21T01:40:01.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0ad4b0 msgr2=0x7f159c064820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0ad4b0 0x7f159c064820 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7f159c13fdc0 tx=0x7f158c014000).stop 2022-07-21T01:40:01.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 shutdown_connections 2022-07-21T01:40:01.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f157c053a70 0x7f157c055f30 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c1437d0 0x7f159c067920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c0af450 0x7f159c067430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 --2- 172.21.15.3:0/704924101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c0ad4b0 0x7f159c064820 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 >> 172.21.15.3:0/704924101 conn(0x7f159c0612d0 msgr2=0x7f159c0af830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:01.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 shutdown_connections 2022-07-21T01:40:01.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.398+0000 7f15a1a7b700 1 -- 172.21.15.3:0/704924101 wait complete. 2022-07-21T01:40:01.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:39:07.362717+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:39:08.613529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:40:01.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:40:01.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f48970db700 1 Processor -- start 2022-07-21T01:40:01.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f48970db700 1 -- start start 2022-07-21T01:40:01.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f48970db700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f489013e5f0 0x7f4890064a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f48970db700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f48900b3a30 0x7f48900b76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f48970db700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4890065530 con 0x7f48900b7bb0 2022-07-21T01:40:01.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f48970db700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4890066160 con 0x7f48900b3a30 2022-07-21T01:40:01.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f48970db700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48900b3260 con 0x7f489013e5f0 2022-07-21T01:40:01.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f488ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f48900b3a30 0x7f48900b76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f488ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f48900b3a30 0x7f48900b76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47202/0 (socket says 172.21.15.3:47202) 2022-07-21T01:40:01.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f488ffff700 1 -- 172.21.15.3:0/1260533827 learned_addr learned my addr 172.21.15.3:0/1260533827 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:01.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f4894e77700 1 --2- 172.21.15.3:0/1260533827 >> v2:172.21.15.143:3300/0 conn(0x7f489013e5f0 0x7f4890064a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f488ffff700 1 -- 172.21.15.3:0/1260533827 >> v2:172.21.15.143:3300/0 conn(0x7f489013e5f0 msgr2=0x7f4890064a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:01.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f488ffff700 1 --2- 172.21.15.3:0/1260533827 >> v2:172.21.15.143:3300/0 conn(0x7f489013e5f0 0x7f4890064a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f488ffff700 1 -- 172.21.15.3:0/1260533827 >> v1:172.21.15.143:6789/0 conn(0x7f48900b7bb0 legacy=0x7f4890066f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:01.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f488ffff700 1 -- 172.21.15.3:0/1260533827 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4890064ff0 con 0x7f48900b3a30 2022-07-21T01:40:01.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f4894e77700 1 --2- 172.21.15.3:0/1260533827 >> v2:172.21.15.143:3300/0 conn(0x7f489013e5f0 0x7f4890064a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:40:01.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f488ffff700 1 --2- 172.21.15.3:0/1260533827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 0x7f48900b76c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f4890067930 tx=0x7f4878006eb0).ready entity=mon.0 client_cookie=f00ba588d916bcf7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:01.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.522+0000 7f488f7fe700 1 -- 172.21.15.3:0/1260533827 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f487800d5c0 con 0x7f48900b3a30 2022-07-21T01:40:01.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f488f7fe700 1 -- 172.21.15.3:0/1260533827 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4878000e90 con 0x7f48900b3a30 2022-07-21T01:40:01.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f488f7fe700 1 -- 172.21.15.3:0/1260533827 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f487800d910 con 0x7f48900b3a30 2022-07-21T01:40:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 -- 172.21.15.3:0/1260533827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 msgr2=0x7f48900b76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 --2- 172.21.15.3:0/1260533827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 0x7f48900b76c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f4890067930 tx=0x7f4878006eb0).stop 2022-07-21T01:40:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 -- 172.21.15.3:0/1260533827 shutdown_connections 2022-07-21T01:40:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 --2- 172.21.15.3:0/1260533827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 0x7f48900b76c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 --2- 172.21.15.3:0/1260533827 >> v2:172.21.15.143:3300/0 conn(0x7f489013e5f0 0x7f4890064a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 -- 172.21.15.3:0/1260533827 >> 172.21.15.3:0/1260533827 conn(0x7f4890062b60 msgr2=0x7f4890062f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 -- 172.21.15.3:0/1260533827 shutdown_connections 2022-07-21T01:40:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 -- 172.21.15.3:0/1260533827 wait complete. 2022-07-21T01:40:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 Processor -- start 2022-07-21T01:40:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.526+0000 7f48970db700 1 -- start start 2022-07-21T01:40:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f48970db700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 0x7f489009ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f48970db700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f489013e5f0 0x7f48900aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f48970db700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48900ab9d0 0x7f48900abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f48970db700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48900adba0 con 0x7f48900b3a30 2022-07-21T01:40:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f48970db700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890153d20 con 0x7f48900ab9d0 2022-07-21T01:40:01.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f48970db700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4890153fc0 con 0x7f489013e5f0 2022-07-21T01:40:01.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4894e77700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 0x7f489009ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4894e77700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 0x7f489009ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47206/0 (socket says 172.21.15.3:47206) 2022-07-21T01:40:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4894e77700 1 -- 172.21.15.3:0/4148981471 learned_addr learned my addr 172.21.15.3:0/4148981471 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f488ffff700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f489013e5f0 0x7f48900aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4895678700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48900ab9d0 0x7f48900abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4894e77700 1 -- 172.21.15.3:0/4148981471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f489013e5f0 msgr2=0x7f48900aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4894e77700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f489013e5f0 0x7f48900aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4894e77700 1 -- 172.21.15.3:0/4148981471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48900ab9d0 msgr2=0x7f48900abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4894e77700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48900ab9d0 0x7f48900abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4894e77700 1 -- 172.21.15.3:0/4148981471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487800b1e0 con 0x7f48900b3a30 2022-07-21T01:40:01.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f4894e77700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 0x7f489009ed80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f4880007000 tx=0x7f4880008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:01.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f488dffb700 1 -- 172.21.15.3:0/4148981471 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4880010580 con 0x7f48900b3a30 2022-07-21T01:40:01.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4890154260 con 0x7f48900b3a30 2022-07-21T01:40:01.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f488dffb700 1 -- 172.21.15.3:0/4148981471 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48800106e0 con 0x7f48900b3a30 2022-07-21T01:40:01.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f488dffb700 1 -- 172.21.15.3:0/4148981471 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4880010a40 con 0x7f48900b3a30 2022-07-21T01:40:01.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48900ac350 con 0x7f48900b3a30 2022-07-21T01:40:01.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f488dffb700 1 -- 172.21.15.3:0/4148981471 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4880010ba0 con 0x7f48900b3a30 2022-07-21T01:40:01.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f488dffb700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4874053950 0x7f4874055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f488dffb700 1 -- 172.21.15.3:0/4148981471 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f488006bfc0 con 0x7f48900b3a30 2022-07-21T01:40:01.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f488ffff700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4874053950 0x7f4874055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4890065290 con 0x7f48900b3a30 2022-07-21T01:40:01.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.530+0000 7f488ffff700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4874053950 0x7f4874055e10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f487801a000 tx=0x7f4878015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:01.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.538+0000 7f488dffb700 1 -- 172.21.15.3:0/4148981471 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4880042cc0 con 0x7f48900b3a30 2022-07-21T01:40:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f48900574e0 con 0x7f48900b3a30 2022-07-21T01:40:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f488dffb700 1 -- 172.21.15.3:0/4148981471 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v80) v1 ==== 110+0+10 (secure 0 0 0) 0x7f48800462e0 con 0x7f48900b3a30 2022-07-21T01:40:01.669 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:40:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4874053950 msgr2=0x7f4874055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4874053950 0x7f4874055e10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f487801a000 tx=0x7f4878015000).stop 2022-07-21T01:40:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 msgr2=0x7f489009ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 0x7f489009ed80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f4880007000 tx=0x7f4880008040).stop 2022-07-21T01:40:01.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 shutdown_connections 2022-07-21T01:40:01.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4874053950 0x7f4874055e10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48900ab9d0 0x7f48900abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f489013e5f0 0x7f48900aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 --2- 172.21.15.3:0/4148981471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48900b3a30 0x7f489009ed80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 >> 172.21.15.3:0/4148981471 conn(0x7f4890062b60 msgr2=0x7f489013fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:01.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 shutdown_connections 2022-07-21T01:40:01.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.666+0000 7f48970db700 1 -- 172.21.15.3:0/4148981471 wait complete. 2022-07-21T01:40:01.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:40:01.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9ab59e700 1 Processor -- start 2022-07-21T01:40:01.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9ab59e700 1 -- start start 2022-07-21T01:40:01.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9ab59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac0b31d0 0x7fb9ac066020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9ab59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb9ac1402d0 0x7fb9ac140780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9ab59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac067030 con 0x7fb9ac0a9da0 2022-07-21T01:40:01.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9ab59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ac064a70 con 0x7fb9ac1402d0 2022-07-21T01:40:01.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9ab59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac061f60 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9aad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb9ac1402d0 0x7fb9ac140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9aad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb9ac1402d0 0x7fb9ac140780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47218/0 (socket says 172.21.15.3:47218) 2022-07-21T01:40:01.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9aad9d700 1 -- 172.21.15.3:0/420131333 learned_addr learned my addr 172.21.15.3:0/420131333 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:01.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.786+0000 7fb9aa59c700 1 --2- 172.21.15.3:0/420131333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac0b31d0 0x7fb9ac066020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9aad9d700 1 -- 172.21.15.3:0/420131333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac0b31d0 msgr2=0x7fb9ac066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9aad9d700 1 --2- 172.21.15.3:0/420131333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac0b31d0 0x7fb9ac066020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9aad9d700 1 -- 172.21.15.3:0/420131333 >> v1:172.21.15.143:6789/0 conn(0x7fb9ac0a9da0 legacy=0x7fb9ac060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9aad9d700 1 -- 172.21.15.3:0/420131333 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac0b35b0 con 0x7fb9ac1402d0 2022-07-21T01:40:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9aad9d700 1 --2- 172.21.15.3:0/420131333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac1402d0 0x7fb9ac140780 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7fb99c002f70 tx=0x7fb99c006f40).ready entity=mon.0 client_cookie=9640c79314dc4f28 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9a959a700 1 -- 172.21.15.3:0/420131333 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb99c00da80 con 0x7fb9ac1402d0 2022-07-21T01:40:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9a959a700 1 -- 172.21.15.3:0/420131333 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99c000e90 con 0x7fb9ac1402d0 2022-07-21T01:40:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9a959a700 1 -- 172.21.15.3:0/420131333 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb99c00de10 con 0x7fb9ac1402d0 2022-07-21T01:40:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 -- 172.21.15.3:0/420131333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac1402d0 msgr2=0x7fb9ac140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 --2- 172.21.15.3:0/420131333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac1402d0 0x7fb9ac140780 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7fb99c002f70 tx=0x7fb99c006f40).stop 2022-07-21T01:40:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 -- 172.21.15.3:0/420131333 shutdown_connections 2022-07-21T01:40:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 --2- 172.21.15.3:0/420131333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac1402d0 0x7fb9ac140780 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 --2- 172.21.15.3:0/420131333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac0b31d0 0x7fb9ac066020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 -- 172.21.15.3:0/420131333 >> 172.21.15.3:0/420131333 conn(0x7fb9ac063870 msgr2=0x7fb9ac063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 -- 172.21.15.3:0/420131333 shutdown_connections 2022-07-21T01:40:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 -- 172.21.15.3:0/420131333 wait complete. 2022-07-21T01:40:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 Processor -- start 2022-07-21T01:40:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.790+0000 7fb9ab59e700 1 -- start start 2022-07-21T01:40:01.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9ab59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9ac0a9da0 0x7fb9ac0b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9ab59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac0b31d0 0x7fb9ac13c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9ab59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac1402d0 0x7fb9ac13cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9ab59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac13bce0 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9ab59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac0b4ca0 con 0x7fb9ac0a9da0 2022-07-21T01:40:01.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9ab59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac0b4f00 con 0x7fb9ac1402d0 2022-07-21T01:40:01.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac0b31d0 0x7fb9ac13c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac0b31d0 0x7fb9ac13c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47222/0 (socket says 172.21.15.3:47222) 2022-07-21T01:40:01.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a9d9b700 1 -- 172.21.15.3:0/886534108 learned_addr learned my addr 172.21.15.3:0/886534108 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:01.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9aad9d700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac1402d0 0x7fb9ac13cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9aa59c700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9ac0a9da0 0x7fb9ac0b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a9d9b700 1 -- 172.21.15.3:0/886534108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac1402d0 msgr2=0x7fb9ac13cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a9d9b700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac1402d0 0x7fb9ac13cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a9d9b700 1 -- 172.21.15.3:0/886534108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9ac0a9da0 msgr2=0x7fb9ac0b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a9d9b700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9ac0a9da0 0x7fb9ac0b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a9d9b700 1 -- 172.21.15.3:0/886534108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c00b460 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a9d9b700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac0b31d0 0x7fb9ac13c7b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fb9ac065790 tx=0x7fb994007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:01.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a37fe700 1 -- 172.21.15.3:0/886534108 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb994012400 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a37fe700 1 -- 172.21.15.3:0/886534108 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb994012560 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ac0b4290 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9a37fe700 1 -- 172.21.15.3:0/886534108 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9940128c0 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ac0b47d0 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.794+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ac066af0 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.798+0000 7fb9a37fe700 1 -- 172.21.15.3:0/886534108 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb994019030 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.798+0000 7fb9a37fe700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb990053a70 0x7fb990055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:01.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.798+0000 7fb9a37fe700 1 -- 172.21.15.3:0/886534108 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7fb99406b490 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.798+0000 7fb9aa59c700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb990053a70 0x7fb990055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:01.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.798+0000 7fb9aa59c700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb990053a70 0x7fb990055f30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fb9a400b000 tx=0x7fb9a40091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:01.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.806+0000 7fb9a37fe700 1 -- 172.21.15.3:0/886534108 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb994042120 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.930+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb9ac0574e0 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9a37fe700 1 -- 172.21.15.3:0/886534108 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v80) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb994045740 con 0x7fb9ac0b31d0 2022-07-21T01:40:01.935 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:01.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb990053a70 msgr2=0x7fb990055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb990053a70 0x7fb990055f30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fb9a400b000 tx=0x7fb9a40091d0).stop 2022-07-21T01:40:01.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac0b31d0 msgr2=0x7fb9ac13c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:01.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac0b31d0 0x7fb9ac13c7b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fb9ac065790 tx=0x7fb994007c20).stop 2022-07-21T01:40:01.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 shutdown_connections 2022-07-21T01:40:01.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9ac1402d0 0x7fb9ac13cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9ac0b31d0 0x7fb9ac13c7b0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb990053a70 0x7fb990055f30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 --2- 172.21.15.3:0/886534108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9ac0a9da0 0x7fb9ac0b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:01.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 >> 172.21.15.3:0/886534108 conn(0x7fb9ac063870 msgr2=0x7fb9ac142dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:01.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 shutdown_connections 2022-07-21T01:40:01.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:01.934+0000 7fb9ab59e700 1 -- 172.21.15.3:0/886534108 wait complete. 2022-07-21T01:40:01.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:40:02.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.050+0000 7f1eecc08700 1 Processor -- start 2022-07-21T01:40:02.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.050+0000 7f1eecc08700 1 -- start start 2022-07-21T01:40:02.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ee805fd90 0x7f1ee80b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1ee80b66f0 0x7f1ee80ba130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee81440f0 con 0x7f1ee80b5b20 2022-07-21T01:40:02.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee8060170 con 0x7f1ee805fd90 2022-07-21T01:40:02.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee8143be0 con 0x7f1ee80b66f0 2022-07-21T01:40:02.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee659c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ee805fd90 0x7f1ee80b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee6d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1ee80b5b20 0x7f1ee80b4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55510/0 (socket says 172.21.15.3:55510) 2022-07-21T01:40:02.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee6d9d700 1 -- 172.21.15.3:0/2884900363 learned_addr learned my addr 172.21.15.3:0/2884900363 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:02.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee5d9b700 1 --2- 172.21.15.3:0/2884900363 >> v2:172.21.15.143:3300/0 conn(0x7f1ee80b66f0 0x7f1ee80ba130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee559a700 1 -- 172.21.15.3:0/2884900363 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078424588 0 0) 0x7f1ee81440f0 con 0x7f1ee80b5b20 2022-07-21T01:40:02.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee559a700 1 -- 172.21.15.3:0/2884900363 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ecc003260 con 0x7f1ee80b5b20 2022-07-21T01:40:02.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee659c700 1 -- 172.21.15.3:0/2884900363 >> v2:172.21.15.143:3300/0 conn(0x7f1ee80b66f0 msgr2=0x7f1ee80ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:02.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee659c700 1 --2- 172.21.15.3:0/2884900363 >> v2:172.21.15.143:3300/0 conn(0x7f1ee80b66f0 0x7f1ee80ba130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee659c700 1 -- 172.21.15.3:0/2884900363 >> v1:172.21.15.3:6789/0 conn(0x7f1ee80b5b20 legacy=0x7f1ee80b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee659c700 1 -- 172.21.15.3:0/2884900363 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee8143e50 con 0x7f1ee805fd90 2022-07-21T01:40:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee659c700 1 --2- 172.21.15.3:0/2884900363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 0x7f1ee80b4340 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f1ed8002f70 tx=0x7f1ed8006df0).ready entity=mon.0 client_cookie=7ff9ff0febcab624 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee559a700 1 -- 172.21.15.3:0/2884900363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed800d4f0 con 0x7f1ee805fd90 2022-07-21T01:40:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee559a700 1 -- 172.21.15.3:0/2884900363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed8000e90 con 0x7f1ee805fd90 2022-07-21T01:40:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1ee559a700 1 -- 172.21.15.3:0/2884900363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed800d840 con 0x7f1ee805fd90 2022-07-21T01:40:02.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 -- 172.21.15.3:0/2884900363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 msgr2=0x7f1ee80b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 --2- 172.21.15.3:0/2884900363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 0x7f1ee80b4340 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f1ed8002f70 tx=0x7f1ed8006df0).stop 2022-07-21T01:40:02.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 -- 172.21.15.3:0/2884900363 shutdown_connections 2022-07-21T01:40:02.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 --2- 172.21.15.3:0/2884900363 >> v2:172.21.15.143:3300/0 conn(0x7f1ee80b66f0 0x7f1ee80ba130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 --2- 172.21.15.3:0/2884900363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 0x7f1ee80b4340 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 -- 172.21.15.3:0/2884900363 >> 172.21.15.3:0/2884900363 conn(0x7f1ee8062530 msgr2=0x7f1ee8066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 -- 172.21.15.3:0/2884900363 shutdown_connections 2022-07-21T01:40:02.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 -- 172.21.15.3:0/2884900363 wait complete. 2022-07-21T01:40:02.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.054+0000 7f1eecc08700 1 Processor -- start 2022-07-21T01:40:02.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 -- start start 2022-07-21T01:40:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 0x7f1ee8142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b5b20 0x7f1ee80af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b66f0 0x7f1ee80afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee8145240 con 0x7f1ee805fd90 2022-07-21T01:40:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee80af460 con 0x7f1ee80b66f0 2022-07-21T01:40:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee8145490 con 0x7f1ee80b5b20 2022-07-21T01:40:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 0x7f1ee8142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 0x7f1ee8142310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47238/0 (socket says 172.21.15.3:47238) 2022-07-21T01:40:02.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee659c700 1 -- 172.21.15.3:0/1565794964 learned_addr learned my addr 172.21.15.3:0/1565794964 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:02.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee6d9d700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b66f0 0x7f1ee80afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee5d9b700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b5b20 0x7f1ee80af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee659c700 1 -- 172.21.15.3:0/1565794964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b5b20 msgr2=0x7f1ee80af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee659c700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b5b20 0x7f1ee80af7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee659c700 1 -- 172.21.15.3:0/1565794964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b66f0 msgr2=0x7f1ee80afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee659c700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b66f0 0x7f1ee80afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee659c700 1 -- 172.21.15.3:0/1565794964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed800b180 con 0x7f1ee805fd90 2022-07-21T01:40:02.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1ee659c700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 0x7f1ee8142310 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f1ed8002f70 tx=0x7f1ed8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:02.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1edf7fe700 1 -- 172.21.15.3:0/1565794964 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed8013de0 con 0x7f1ee805fd90 2022-07-21T01:40:02.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1edf7fe700 1 -- 172.21.15.3:0/1565794964 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed800de60 con 0x7f1ee805fd90 2022-07-21T01:40:02.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee813d210 con 0x7f1ee805fd90 2022-07-21T01:40:02.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1edf7fe700 1 -- 172.21.15.3:0/1565794964 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed801e5f0 con 0x7f1ee805fd90 2022-07-21T01:40:02.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee80b04a0 con 0x7f1ee805fd90 2022-07-21T01:40:02.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.058+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee813d3d0 con 0x7f1ee805fd90 2022-07-21T01:40:02.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.062+0000 7f1edf7fe700 1 -- 172.21.15.3:0/1565794964 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ed8025030 con 0x7f1ee805fd90 2022-07-21T01:40:02.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.062+0000 7f1edf7fe700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ec0053b30 0x7f1ec0055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.062+0000 7f1edf7fe700 1 -- 172.21.15.3:0/1565794964 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f1ed806cba0 con 0x7f1ee805fd90 2022-07-21T01:40:02.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.062+0000 7f1ee5d9b700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ec0053b30 0x7f1ec0055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.062+0000 7f1ee5d9b700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ec0053b30 0x7f1ec0055ff0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1ed000b000 tx=0x7f1ed0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:02.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.070+0000 7f1edf7fe700 1 -- 172.21.15.3:0/1565794964 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ed8043940 con 0x7f1ee805fd90 2022-07-21T01:40:02.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:40:02.100 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:40:02.103 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:40:02.105 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:40:02.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.198+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ee80af080 con 0x7f1ee805fd90 2022-07-21T01:40:02.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.198+0000 7f1edf7fe700 1 -- 172.21.15.3:0/1565794964 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1ed8023020 con 0x7f1ee805fd90 2022-07-21T01:40:02.203 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:02.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ec0053b30 msgr2=0x7f1ec0055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ec0053b30 0x7f1ec0055ff0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1ed000b000 tx=0x7f1ed0006df0).stop 2022-07-21T01:40:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 msgr2=0x7f1ee8142310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 0x7f1ee8142310 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f1ed8002f70 tx=0x7f1ed8014000).stop 2022-07-21T01:40:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 shutdown_connections 2022-07-21T01:40:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ec0053b30 0x7f1ec0055ff0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ee80b66f0 0x7f1ee80afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ee80b5b20 0x7f1ee80af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 --2- 172.21.15.3:0/1565794964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ee805fd90 0x7f1ee8142310 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 >> 172.21.15.3:0/1565794964 conn(0x7f1ee8062530 msgr2=0x7f1ee80b80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:02.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 shutdown_connections 2022-07-21T01:40:02.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.202+0000 7f1eecc08700 1 -- 172.21.15.3:0/1565794964 wait complete. 2022-07-21T01:40:02.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:40:02.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.322+0000 7ff26256c700 1 Processor -- start 2022-07-21T01:40:02.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.322+0000 7ff26256c700 1 -- start start 2022-07-21T01:40:02.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.322+0000 7ff26256c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0b3190 0x7ff25c060ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.322+0000 7ff26256c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff25c0b2730 0x7ff25c061190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.322+0000 7ff26256c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff25c0666b0 0x7ff25c066b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.322+0000 7ff26256c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c062190 con 0x7ff25c0666b0 2022-07-21T01:40:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.322+0000 7ff26256c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c0654c0 con 0x7ff25c0b2730 2022-07-21T01:40:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.322+0000 7ff26256c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0b3570 con 0x7ff25c0b3190 2022-07-21T01:40:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff261d6b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff25c0666b0 0x7ff25c066b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff261d6b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff25c0666b0 0x7ff25c066b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47244/0 (socket says 172.21.15.3:47244) 2022-07-21T01:40:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff261d6b700 1 -- 172.21.15.3:0/715419793 learned_addr learned my addr 172.21.15.3:0/715419793 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff260d69700 1 --2- 172.21.15.3:0/715419793 >> v2:172.21.15.143:3300/0 conn(0x7ff25c0b2730 0x7ff25c061190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26156a700 1 --2- 172.21.15.3:0/715419793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0b3190 0x7ff25c060ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff261d6b700 1 -- 172.21.15.3:0/715419793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0b3190 msgr2=0x7ff25c060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:02.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff261d6b700 1 --2- 172.21.15.3:0/715419793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0b3190 0x7ff25c060ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff261d6b700 1 -- 172.21.15.3:0/715419793 >> v2:172.21.15.143:3300/0 conn(0x7ff25c0b2730 msgr2=0x7ff25c061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:02.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff261d6b700 1 --2- 172.21.15.3:0/715419793 >> v2:172.21.15.143:3300/0 conn(0x7ff25c0b2730 0x7ff25c061190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff261d6b700 1 -- 172.21.15.3:0/715419793 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c05fbf0 con 0x7ff25c0666b0 2022-07-21T01:40:02.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff260d69700 1 --2- 172.21.15.3:0/715419793 >> v2:172.21.15.143:3300/0 conn(0x7ff25c0b2730 0x7ff25c061190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:02.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff261d6b700 1 --2- 172.21.15.3:0/715419793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0666b0 0x7ff25c066b20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7ff2540032f0 tx=0x7ff254009290).ready entity=mon.0 client_cookie=136569930057130a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:02.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff247fff700 1 -- 172.21.15.3:0/715419793 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff25400b870 con 0x7ff25c0666b0 2022-07-21T01:40:02.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff247fff700 1 -- 172.21.15.3:0/715419793 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25400b9d0 con 0x7ff25c0666b0 2022-07-21T01:40:02.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff247fff700 1 -- 172.21.15.3:0/715419793 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff25400bb40 con 0x7ff25c0666b0 2022-07-21T01:40:02.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 -- 172.21.15.3:0/715419793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0666b0 msgr2=0x7ff25c066b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 --2- 172.21.15.3:0/715419793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0666b0 0x7ff25c066b20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7ff2540032f0 tx=0x7ff254009290).stop 2022-07-21T01:40:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 -- 172.21.15.3:0/715419793 shutdown_connections 2022-07-21T01:40:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 --2- 172.21.15.3:0/715419793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0666b0 0x7ff25c066b20 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 --2- 172.21.15.3:0/715419793 >> v2:172.21.15.143:3300/0 conn(0x7ff25c0b2730 0x7ff25c061190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 --2- 172.21.15.3:0/715419793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0b3190 0x7ff25c060ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 -- 172.21.15.3:0/715419793 >> 172.21.15.3:0/715419793 conn(0x7ff25c063870 msgr2=0x7ff25c063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 -- 172.21.15.3:0/715419793 shutdown_connections 2022-07-21T01:40:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 -- 172.21.15.3:0/715419793 wait complete. 2022-07-21T01:40:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 Processor -- start 2022-07-21T01:40:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.326+0000 7ff26256c700 1 -- start start 2022-07-21T01:40:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26256c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0666b0 0x7ff25c152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26256c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff25c0b2730 0x7ff25c0b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26256c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0b3190 0x7ff25c0a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26256c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c153290 con 0x7ff25c0b3190 2022-07-21T01:40:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26256c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0ac2f0 con 0x7ff25c0b2730 2022-07-21T01:40:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26256c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0ac520 con 0x7ff25c0666b0 2022-07-21T01:40:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff261d6b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0b3190 0x7ff25c0a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff261d6b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0b3190 0x7ff25c0a93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47252/0 (socket says 172.21.15.3:47252) 2022-07-21T01:40:02.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26156a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0666b0 0x7ff25c152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff261d6b700 1 -- 172.21.15.3:0/3041786826 learned_addr learned my addr 172.21.15.3:0/3041786826 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:02.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff260d69700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff25c0b2730 0x7ff25c0b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff261d6b700 1 -- 172.21.15.3:0/3041786826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0666b0 msgr2=0x7ff25c152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff261d6b700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0666b0 0x7ff25c152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff261d6b700 1 -- 172.21.15.3:0/3041786826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff25c0b2730 msgr2=0x7ff25c0b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff261d6b700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff25c0b2730 0x7ff25c0b01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff261d6b700 1 -- 172.21.15.3:0/3041786826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff254009530 con 0x7ff25c0b3190 2022-07-21T01:40:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff260d69700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff25c0b2730 0x7ff25c0b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff261d6b700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0b3190 0x7ff25c0a93a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7ff25c1533d0 tx=0x7ff254016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff24effd700 1 -- 172.21.15.3:0/3041786826 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff25400b870 con 0x7ff25c0b3190 2022-07-21T01:40:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff24effd700 1 -- 172.21.15.3:0/3041786826 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff254020430 con 0x7ff25c0b3190 2022-07-21T01:40:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff24effd700 1 -- 172.21.15.3:0/3041786826 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff254020790 con 0x7ff25c0b3190 2022-07-21T01:40:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c0ac7c0 con 0x7ff25c0b3190 2022-07-21T01:40:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c0b06b0 con 0x7ff25c0b3190 2022-07-21T01:40:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c05fe90 con 0x7ff25c0b3190 2022-07-21T01:40:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff24effd700 1 -- 172.21.15.3:0/3041786826 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff254026020 con 0x7ff25c0b3190 2022-07-21T01:40:02.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff24effd700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff2380539e0 0x7ff238055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff24effd700 1 -- 172.21.15.3:0/3041786826 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7ff25406cec0 con 0x7ff25c0b3190 2022-07-21T01:40:02.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.330+0000 7ff26156a700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff2380539e0 0x7ff238055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.334+0000 7ff26156a700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff2380539e0 0x7ff238055ea0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7ff25800c000 tx=0x7ff258008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:02.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.338+0000 7ff24effd700 1 -- 172.21.15.3:0/3041786826 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff254043c60 con 0x7ff25c0b3190 2022-07-21T01:40:02.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.462+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff25c0574e0 con 0x7ff25c0b3190 2022-07-21T01:40:02.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff24effd700 1 -- 172.21.15.3:0/3041786826 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff254015070 con 0x7ff25c0b3190 2022-07-21T01:40:02.468 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:02.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff2380539e0 msgr2=0x7ff238055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff2380539e0 0x7ff238055ea0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7ff25800c000 tx=0x7ff258008040).stop 2022-07-21T01:40:02.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0b3190 msgr2=0x7ff25c0a93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0b3190 0x7ff25c0a93a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7ff25c1533d0 tx=0x7ff254016000).stop 2022-07-21T01:40:02.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 shutdown_connections 2022-07-21T01:40:02.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff25c0b3190 0x7ff25c0a93a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff2380539e0 0x7ff238055ea0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff25c0b2730 0x7ff25c0b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 --2- 172.21.15.3:0/3041786826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff25c0666b0 0x7ff25c152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 >> 172.21.15.3:0/3041786826 conn(0x7ff25c063870 msgr2=0x7ff25c066a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 shutdown_connections 2022-07-21T01:40:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.466+0000 7ff26256c700 1 -- 172.21.15.3:0/3041786826 wait complete. 2022-07-21T01:40:02.485 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:40:02.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:02.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.590+0000 7f29508fe700 1 Processor -- start 2022-07-21T01:40:02.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.590+0000 7f29508fe700 1 -- start start 2022-07-21T01:40:02.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.590+0000 7f29508fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f294c0ad670 0x7f294c0b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.590+0000 7f29508fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 0x7f294c13fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.590+0000 7f29508fe700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294c0b0c40 con 0x7f294c0af120 2022-07-21T01:40:02.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.590+0000 7f29508fe700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294c0b0ee0 con 0x7f294c0ad670 2022-07-21T01:40:02.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.590+0000 7f29508fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c0ae9e0 con 0x7f294c13fa40 2022-07-21T01:40:02.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294259c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f294c0af120 0x7f294c0b12a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:57652/0 (socket says 172.21.15.3:57652) 2022-07-21T01:40:02.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294259c700 1 -- 172.21.15.3:0/1856915136 learned_addr learned my addr 172.21.15.3:0/1856915136 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:02.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294ad9d700 1 --2- 172.21.15.3:0/1856915136 >> v2:172.21.15.143:3300/0 conn(0x7f294c0ad670 0x7f294c0b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294b59e700 1 --2- 172.21.15.3:0/1856915136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 0x7f294c13fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294a59c700 1 -- 172.21.15.3:0/1856915136 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582497390 0 0) 0x7f294c0b0c40 con 0x7f294c0af120 2022-07-21T01:40:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294a59c700 1 -- 172.21.15.3:0/1856915136 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f292c003260 con 0x7f294c0af120 2022-07-21T01:40:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294b59e700 1 -- 172.21.15.3:0/1856915136 >> v2:172.21.15.143:3300/0 conn(0x7f294c0ad670 msgr2=0x7f294c0b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294b59e700 1 --2- 172.21.15.3:0/1856915136 >> v2:172.21.15.143:3300/0 conn(0x7f294c0ad670 0x7f294c0b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294b59e700 1 -- 172.21.15.3:0/1856915136 >> v1:172.21.15.143:6789/0 conn(0x7f294c0af120 legacy=0x7f294c0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294b59e700 1 -- 172.21.15.3:0/1856915136 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c0aec80 con 0x7f294c13fa40 2022-07-21T01:40:02.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294ad9d700 1 --2- 172.21.15.3:0/1856915136 >> v2:172.21.15.143:3300/0 conn(0x7f294c0ad670 0x7f294c0b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:02.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294b59e700 1 --2- 172.21.15.3:0/1856915136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 0x7f294c13fed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f29440032f0 tx=0x7f29440091d0).ready entity=mon.2 client_cookie=749732d892bb3beb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:02.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294a59c700 1 -- 172.21.15.3:0/1856915136 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f294400b600 con 0x7f294c13fa40 2022-07-21T01:40:02.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294a59c700 1 -- 172.21.15.3:0/1856915136 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294400b760 con 0x7f294c13fa40 2022-07-21T01:40:02.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f294a59c700 1 -- 172.21.15.3:0/1856915136 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f294400b8d0 con 0x7f294c13fa40 2022-07-21T01:40:02.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f29508fe700 1 -- 172.21.15.3:0/1856915136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 msgr2=0x7f294c13fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f29508fe700 1 --2- 172.21.15.3:0/1856915136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 0x7f294c13fed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f29440032f0 tx=0x7f29440091d0).stop 2022-07-21T01:40:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f29508fe700 1 -- 172.21.15.3:0/1856915136 shutdown_connections 2022-07-21T01:40:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f29508fe700 1 --2- 172.21.15.3:0/1856915136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 0x7f294c13fed0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f29508fe700 1 --2- 172.21.15.3:0/1856915136 >> v2:172.21.15.143:3300/0 conn(0x7f294c0ad670 0x7f294c0b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f29508fe700 1 -- 172.21.15.3:0/1856915136 >> 172.21.15.3:0/1856915136 conn(0x7f294c061360 msgr2=0x7f294c062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:02.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f29508fe700 1 -- 172.21.15.3:0/1856915136 shutdown_connections 2022-07-21T01:40:02.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f29508fe700 1 -- 172.21.15.3:0/1856915136 wait complete. 2022-07-21T01:40:02.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.594+0000 7f29508fe700 1 Processor -- start 2022-07-21T01:40:02.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f29508fe700 1 -- start start 2022-07-21T01:40:02.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f29508fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f294c0ad670 0x7f294c0b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f29508fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f294c0af120 0x7f294c153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f29508fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 0x7f294c13a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f29508fe700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c067690 con 0x7f294c0af120 2022-07-21T01:40:02.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f29508fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c0678c0 con 0x7f294c0ad670 2022-07-21T01:40:02.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f29508fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c063dc0 con 0x7f294c13fa40 2022-07-21T01:40:02.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f294c0af120 0x7f294c153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f294c0af120 0x7f294c153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47264/0 (socket says 172.21.15.3:47264) 2022-07-21T01:40:02.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294259c700 1 -- 172.21.15.3:0/627138488 learned_addr learned my addr 172.21.15.3:0/627138488 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:02.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294ad9d700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f294c0ad670 0x7f294c0b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294b59e700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 0x7f294c13a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294259c700 1 -- 172.21.15.3:0/627138488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 msgr2=0x7f294c13a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294259c700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 0x7f294c13a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294259c700 1 -- 172.21.15.3:0/627138488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f294c0ad670 msgr2=0x7f294c0b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294259c700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f294c0ad670 0x7f294c0b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294259c700 1 -- 172.21.15.3:0/627138488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c00b180 con 0x7f294c0af120 2022-07-21T01:40:02.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f294259c700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f294c0af120 0x7f294c153f30 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7f2934001110 tx=0x7f2934007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:02.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f2948d99700 1 -- 172.21.15.3:0/627138488 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f293400ce80 con 0x7f294c0af120 2022-07-21T01:40:02.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f2948d99700 1 -- 172.21.15.3:0/627138488 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29340070d0 con 0x7f294c0af120 2022-07-21T01:40:02.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2944009470 con 0x7f294c0af120 2022-07-21T01:40:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f2948d99700 1 -- 172.21.15.3:0/627138488 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2934007430 con 0x7f294c0af120 2022-07-21T01:40:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.598+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294c13a540 con 0x7f294c0af120 2022-07-21T01:40:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.602+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294c067ac0 con 0x7f294c0af120 2022-07-21T01:40:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.602+0000 7f2948d99700 1 -- 172.21.15.3:0/627138488 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2934010070 con 0x7f294c0af120 2022-07-21T01:40:02.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.602+0000 7f2948d99700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2930053950 0x7f2930055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:02.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.602+0000 7f2948d99700 1 -- 172.21.15.3:0/627138488 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f293406b740 con 0x7f294c0af120 2022-07-21T01:40:02.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.602+0000 7f294ad9d700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2930053950 0x7f2930055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:02.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.602+0000 7f294ad9d700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2930053950 0x7f2930055e10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f293c00b000 tx=0x7f293c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:02.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.606+0000 7f2948d99700 1 -- 172.21.15.3:0/627138488 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2934042370 con 0x7f294c0af120 2022-07-21T01:40:02.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.766+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f294c0af590 con 0x7f294c0af120 2022-07-21T01:40:02.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f2948d99700 1 -- 172.21.15.3:0/627138488 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2487 (secure 0 0 0) 0x7f2934045990 con 0x7f294c0af120 2022-07-21T01:40:02.772 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:02.773 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:39:09.622528+0000","modified":"2022-07-21T01:39:10.626387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6259":{"gid":6259,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/862725488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":862725488},{"type":"v1","addr":"172.21.15.143:6833","nonce":862725488}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-21T01:40:02.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2930053950 msgr2=0x7f2930055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2930053950 0x7f2930055e10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f293c00b000 tx=0x7f293c006df0).stop 2022-07-21T01:40:02.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f294c0af120 msgr2=0x7f294c153f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:02.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f294c0af120 0x7f294c153f30 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7f2934001110 tx=0x7f2934007aa0).stop 2022-07-21T01:40:02.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 shutdown_connections 2022-07-21T01:40:02.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f294c13fa40 0x7f294c13a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f294c0af120 0x7f294c153f30 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2930053950 0x7f2930055e10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 --2- 172.21.15.3:0/627138488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f294c0ad670 0x7f294c0b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:02.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 >> 172.21.15.3:0/627138488 conn(0x7f294c061360 msgr2=0x7f294c143090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:02.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 shutdown_connections 2022-07-21T01:40:02.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:02.770+0000 7f29508fe700 1 -- 172.21.15.3:0/627138488 wait complete. 2022-07-21T01:40:02.779 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2022-07-21T01:40:03.088 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:39:09.622528+0000', 'modified': '2022-07-21T01:39:10.626387+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6259': {'gid': 6259, 'name': 'b', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6833/862725488', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 862725488}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 862725488}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:40:03.089 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:40:03.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-21T01:40:03.205 INFO:teuthology.orchestra.run.smithi143.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6259,"want_state":"up:active","state":"up:active","rank_uptime":53.566950640999998,"mdsmap_epoch":35,"osdmap_epoch":80,"osdmap_epoch_barrier":80,"uptime":58.619013596000002} 2022-07-21T01:40:03.205 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6259, "mdsmap_epoch": 35, "osdmap_epoch": 80, "osdmap_epoch_barrier": 80, "rank_uptime": 53.566950641, "state": "up:active", "uptime": 58.619013596, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:40:03.206 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:40:03.207 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:40:03.207 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:40:03.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:03.208 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:03.216 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:40:03.234 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:03.235 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:03.245 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:03.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:03.247 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:40:03.297 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:40:03.297 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:03.300 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:40:03.317 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:03.318 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:03.328 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:03.331 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:03.332 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:40:03.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:40:03.375+0000 7fc62f0d1080 -1 init, newargv = 0x55da0c5405f0 newargc=9ceph-fuse[22873]: starting ceph client 2022-07-21T01:40:03.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr: 2022-07-21T01:40:03.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22873]: starting fuse 2022-07-21T01:40:04.481 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:40:04.500 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:04.501 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:04.514 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:04.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:04.516 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:40:04.565 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:40:04.565 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:40:04.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:04.566 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:40:04.567 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:40:04.567 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:40:04.568 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:40:04.568 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:40:04.569 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:04.569 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:40:04.569 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:40:04.570 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:40:04.570 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:04.571 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:40:04.571 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:40:04.572 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:40:04.572 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:40:04.573 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:04.573 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:40:04.573 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:40:04.574 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:40:04.574 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:40:04.575 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:40:04.575 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:04.576 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:40:04.576 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:40:04.621 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22873.asok 2022-07-21T01:40:04.626 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22873.asok 2022-07-21T01:40:04.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:04.627 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22873.asok status 2022-07-21T01:40:04.715 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:40:04.715 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:40:04.715 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:40:04.716 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:40:04.716 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:40:04.716 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:40:04.717 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:40:04.717 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22873", 2022-07-21T01:40:04.717 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:40:04.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:04.718 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:40:04.718 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:40:04.718 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 5837, 2022-07-21T01:40:04.719 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:40:04.719 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:40:04.719 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:40:04.719 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 5837 2022-07-21T01:40:04.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:04.720 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:40:04.720 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:40:04.720 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:40:04.721 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 1904387501 2022-07-21T01:40:04.721 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:40:04.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:04.722 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:40:04.722 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:40:04.722 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:40:04.722 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 1904387501 2022-07-21T01:40:04.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:04.723 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.5837 172.21.15.171:0/1904387501", 2022-07-21T01:40:04.723 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/1904387501", 2022-07-21T01:40:04.723 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:40:04.724 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 35, 2022-07-21T01:40:04.724 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 80, 2022-07-21T01:40:04.724 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:40:04.724 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:40:04.725 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:40:04.725 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:40:04.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:04.727 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:04.727 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:04.735 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:40:04.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:04.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:04.737 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:04.753 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:40:04.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:04.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:04.795 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:04.814 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:04.836 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:40:04.836 INFO:tasks.cephfs.mount:Terminating background process 2022-07-21T01:40:04.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:04.837 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:04.838 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:04.846 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:40:04.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:04.848 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:40:04.848 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:40:04.848 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:04.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22873]: fuse finished with error 0 and tester_r 0 2022-07-21T01:40:04.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:04.866 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:04.867 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:04.913 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:40:04.915 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:40:04.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:40:04.970 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:04.979 INFO:tasks.cephfs.mount:Terminating background process 2022-07-21T01:40:04.979 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:04.980 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:04.980 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:04.989 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:40:04.991 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:40:04.991 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:40:04.992 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:40:04.993 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:05.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22433]: fuse finished with error 0 and tester_r 0 2022-07-21T01:40:05.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:05.011 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:05.012 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:05.065 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:40:05.067 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:40:05.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:40:05.186 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:05.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:40:05.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 Processor -- start 2022-07-21T01:40:05.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 -- start start 2022-07-21T01:40:05.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8a2c0b1f40 0x7f8a2c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c148530 0x7f8a2c0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c0b04e0 con 0x7f8a2c0a7250 2022-07-21T01:40:05.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2c0a9250 con 0x7f8a2c0b1f40 2022-07-21T01:40:05.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0b0780 con 0x7f8a2c148530 2022-07-21T01:40:05.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a2a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c148530 0x7f8a2c0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a2a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c148530 0x7f8a2c0b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:54186/0 (socket says 172.21.15.3:54186) 2022-07-21T01:40:05.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a2a7fc700 1 -- 172.21.15.3:0/2352819087 learned_addr learned my addr 172.21.15.3:0/2352819087 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:05.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a2affd700 1 --2- 172.21.15.3:0/2352819087 >> v2:172.21.15.143:3300/0 conn(0x7f8a2c0b1f40 0x7f8a2c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a2a7fc700 1 -- 172.21.15.3:0/2352819087 >> v2:172.21.15.143:3300/0 conn(0x7f8a2c0b1f40 msgr2=0x7f8a2c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:05.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a2a7fc700 1 --2- 172.21.15.3:0/2352819087 >> v2:172.21.15.143:3300/0 conn(0x7f8a2c0b1f40 0x7f8a2c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a2a7fc700 1 -- 172.21.15.3:0/2352819087 >> v1:172.21.15.143:6789/0 conn(0x7f8a2c0a7250 legacy=0x7f8a2c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a29ffb700 1 -- 172.21.15.3:0/2352819087 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065956078 0 0) 0x7f8a2c0b04e0 con 0x7f8a2c0a7250 2022-07-21T01:40:05.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a2a7fc700 1 -- 172.21.15.3:0/2352819087 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c0b0a20 con 0x7f8a2c148530 2022-07-21T01:40:05.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a2a7fc700 1 --2- 172.21.15.3:0/2352819087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c148530 0x7f8a2c0b2320 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8a1c002f70 tx=0x7f8a1c006eb0).ready entity=mon.2 client_cookie=76f39a240a4cbcaa server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:05.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a29ffb700 1 -- 172.21.15.3:0/2352819087 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1c00d570 con 0x7f8a2c148530 2022-07-21T01:40:05.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a29ffb700 1 -- 172.21.15.3:0/2352819087 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c000e90 con 0x7f8a2c148530 2022-07-21T01:40:05.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a29ffb700 1 -- 172.21.15.3:0/2352819087 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1c00d8c0 con 0x7f8a2c148530 2022-07-21T01:40:05.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 -- 172.21.15.3:0/2352819087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c148530 msgr2=0x7f8a2c0b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 --2- 172.21.15.3:0/2352819087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c148530 0x7f8a2c0b2320 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8a1c002f70 tx=0x7f8a1c006eb0).stop 2022-07-21T01:40:05.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 -- 172.21.15.3:0/2352819087 shutdown_connections 2022-07-21T01:40:05.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 --2- 172.21.15.3:0/2352819087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c148530 0x7f8a2c0b2320 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 --2- 172.21.15.3:0/2352819087 >> v2:172.21.15.143:3300/0 conn(0x7f8a2c0b1f40 0x7f8a2c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 -- 172.21.15.3:0/2352819087 >> 172.21.15.3:0/2352819087 conn(0x7f8a2c05f840 msgr2=0x7f8a2c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:05.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 -- 172.21.15.3:0/2352819087 shutdown_connections 2022-07-21T01:40:05.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.306+0000 7f8a314b0700 1 -- 172.21.15.3:0/2352819087 wait complete. 2022-07-21T01:40:05.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 Processor -- start 2022-07-21T01:40:05.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 -- start start 2022-07-21T01:40:05.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2c0a7250 0x7f8a2c0ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c0b1f40 0x7f8a2c1439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c148530 0x7f8a2c143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c14b2d0 con 0x7f8a2c148530 2022-07-21T01:40:05.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0a9ab0 con 0x7f8a2c0a7250 2022-07-21T01:40:05.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0a9d50 con 0x7f8a2c0b1f40 2022-07-21T01:40:05.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c0b1f40 0x7f8a2c1439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c148530 0x7f8a2c143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c0b1f40 0x7f8a2c1439d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:54194/0 (socket says 172.21.15.3:54194) 2022-07-21T01:40:05.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2c0a7250 0x7f8a2c0ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2a7fc700 1 -- 172.21.15.3:0/21521357 learned_addr learned my addr 172.21.15.3:0/21521357 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:05.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c148530 0x7f8a2c143ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47280/0 (socket says 172.21.15.3:47280) 2022-07-21T01:40:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2b7fe700 1 -- 172.21.15.3:0/21521357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c0b1f40 msgr2=0x7f8a2c1439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2b7fe700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c0b1f40 0x7f8a2c1439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2b7fe700 1 -- 172.21.15.3:0/21521357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2c0a7250 msgr2=0x7f8a2c0ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2b7fe700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2c0a7250 0x7f8a2c0ae300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2b7fe700 1 -- 172.21.15.3:0/21521357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c00b1e0 con 0x7f8a2c148530 2022-07-21T01:40:05.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a2b7fe700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c148530 0x7f8a2c143ec0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f8a24001250 tx=0x7f8a24007d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:05.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a1bfff700 1 -- 172.21.15.3:0/21521357 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a2400f470 con 0x7f8a2c148530 2022-07-21T01:40:05.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a1bfff700 1 -- 172.21.15.3:0/21521357 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2400f5d0 con 0x7f8a2c148530 2022-07-21T01:40:05.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2c0ab980 con 0x7f8a2c148530 2022-07-21T01:40:05.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a1bfff700 1 -- 172.21.15.3:0/21521357 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a2400f930 con 0x7f8a2c148530 2022-07-21T01:40:05.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.310+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2c0ae9d0 con 0x7f8a2c148530 2022-07-21T01:40:05.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.314+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2c0a9910 con 0x7f8a2c148530 2022-07-21T01:40:05.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.314+0000 7f8a1bfff700 1 -- 172.21.15.3:0/21521357 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8a24019030 con 0x7f8a2c148530 2022-07-21T01:40:05.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.314+0000 7f8a1bfff700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.314+0000 7f8a1bfff700 1 -- 172.21.15.3:0/21521357 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f8a2406b6f0 con 0x7f8a2c148530 2022-07-21T01:40:05.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.314+0000 7f8a2affd700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.314+0000 7f8a2affd700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f8a2000b000 tx=0x7f8a20006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:05.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.322+0000 7f8a1bfff700 1 -- 172.21.15.3:0/21521357 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a2c0a9910 con 0x7f8a2c148530 2022-07-21T01:40:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.446+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f8a2c0a9910 con 0x7f8a2c148530 2022-07-21T01:40:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a1bfff700 1 -- 172.21.15.3:0/21521357 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v80) v1 ==== 81+0+34116 (secure 0 0 0) 0x7f8a24029560 con 0x7f8a2c148530 2022-07-21T01:40:05.453 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:05.453 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:40:05.454 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 80, 2022-07-21T01:40:05.454 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:40:05.454 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:40:05.454 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:39:54.719898+0000", 2022-07-21T01:40:05.455 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:05.455 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:05.455 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:40:05.455 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:40:05.455 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:40:05.456 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:40:05.456 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:40:05.456 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:40:05.456 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:40:05.457 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:05.457 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:40:05.457 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:40:05.457 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:40:05.457 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:40:05.458 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:40:05.458 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 14, 2022-07-21T01:40:05.458 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:40:05.458 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:40:05.459 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:40:05.459 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:40:05.459 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:40:05.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.459 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:40:05.460 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:40:05.460 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:40:05.460 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:05.460 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:05.461 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:05.461 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:05.461 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:05.461 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:05.461 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:05.462 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:05.462 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:40:05.462 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:40:05.462 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:40:05.463 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:40:05.463 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:40:05.463 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:05.463 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:05.463 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:05.464 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:05.464 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:05.464 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:05.464 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:05.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.465 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:40:05.465 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:05.465 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:05.465 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:05.466 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:05.466 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:05.466 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:05.466 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:05.467 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:05.467 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:05.467 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:05.467 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:05.467 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:05.468 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:05.468 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:05.468 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:05.468 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:05.469 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:05.469 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:05.469 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:05.469 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:05.469 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:05.470 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:05.470 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:05.470 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:05.470 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:05.471 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:05.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.471 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:05.471 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:05.471 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:05.472 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:05.472 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:05.472 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:05.472 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:05.472 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:05.473 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:05.473 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:05.473 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:05.473 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:40:05.474 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:40:05.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.474 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:05.474 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:40:05.475 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.475 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:40:05.475 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:40:05.476 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:40:05.476 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:05.476 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:05.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:05.477 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:05.477 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:05.477 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:05.477 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:05.477 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:05.478 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:05.478 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:05.478 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:05.478 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:05.479 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:05.479 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:05.479 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:05.479 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:05.480 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:05.480 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:05.480 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:05.480 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:05.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.481 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:40:05.481 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:05.481 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:05.482 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:05.482 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:05.482 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:05.482 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:05.483 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:05.483 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:05.483 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:05.483 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:05.483 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:05.484 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:05.484 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:05.484 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:05.484 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:05.485 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:05.485 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:05.485 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:05.485 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:05.486 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:05.486 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:05.486 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:05.486 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:05.487 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:05.487 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:05.487 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:05.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.488 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:05.488 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:05.488 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:05.488 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:05.488 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:05.489 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:05.489 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:05.489 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:05.489 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:05.490 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:05.490 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:05.490 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:40:05.490 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:05.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:40:05.491 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.491 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 13, 2022-07-21T01:40:05.492 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:40:05.492 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:39:07.362717+0000", 2022-07-21T01:40:05.492 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:05.492 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:05.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:05.493 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:05.493 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:05.493 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:05.493 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:05.494 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:05.494 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:05.494 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:05.494 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:05.494 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:05.495 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:05.495 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:05.495 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:05.495 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:05.496 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:05.496 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:05.496 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:05.496 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:05.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.497 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "78", 2022-07-21T01:40:05.497 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:05.497 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:05.497 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:05.498 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:05.498 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:05.498 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:05.498 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:05.498 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:05.499 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:05.499 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:05.499 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:05.499 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:05.500 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:05.500 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:05.500 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:05.500 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:05.501 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:05.501 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:05.501 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:05.501 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:05.502 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:05.502 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:05.502 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:05.502 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:05.503 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:05.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:05.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.503 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:05.505 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:05.505 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:05.505 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:05.505 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:05.505 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:05.506 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:05.506 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:40:05.506 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:40:05.506 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:40:05.507 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:40:05.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.507 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:05.507 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:40:05.507 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:40:05.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.509 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 14, 2022-07-21T01:40:05.509 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:40:05.509 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:39:08.613529+0000", 2022-07-21T01:40:05.509 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:05.509 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:05.510 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:05.510 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:05.510 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:05.510 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:05.511 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:05.511 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:05.511 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:05.511 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:05.511 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:05.512 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:05.512 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:05.512 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:05.512 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:05.513 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:05.513 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:05.513 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:05.513 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:05.513 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:05.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.514 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "79", 2022-07-21T01:40:05.514 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:05.514 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:05.515 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:05.515 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:05.515 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:05.515 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:05.515 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:05.516 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:05.516 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:05.516 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:05.516 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:05.517 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:05.517 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:05.517 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:05.517 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:05.517 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:05.518 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:05.518 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:05.518 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:05.518 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:05.519 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:05.519 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:05.519 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:05.519 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:05.520 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:05.520 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:05.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.520 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:05.520 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:05.521 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:05.521 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:05.521 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:05.521 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:05.522 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:05.522 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:05.522 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:05.522 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:05.522 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:05.523 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:40:05.523 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:05.523 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:40:05.523 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:40:05.524 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.524 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.524 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.524 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:05.524 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:40:05.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.525 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:05.525 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:40:05.525 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:05.525 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:05.526 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:05.526 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:05.526 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:05.526 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:05.527 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:05.527 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:05.527 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:05.527 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:05.527 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:05.528 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.528 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:40:05.529 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:05.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.529 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.529 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:40:05.530 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:05.530 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.530 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.531 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:05.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.531 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:40:05.532 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:05.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.532 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.532 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:40:05.533 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:05.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.533 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.534 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:05.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:40:05.535 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:05.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.535 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:40:05.536 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:05.536 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.536 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.537 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:05.537 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.537 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.538 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:40:05.538 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:05.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.539 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:40:05.539 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:05.539 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.540 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.540 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:40:05.541 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:40:05.541 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:40:05.541 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:40:05.541 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:05.542 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:05.542 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:05.542 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.543 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:05.543 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:40:05.543 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:05.544 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:05.544 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:05.544 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:05.544 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:05.544 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:05.545 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:05.545 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 65, 2022-07-21T01:40:05.545 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:05.545 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:05.546 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:05.546 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.546 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.547 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:40:05.547 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:05.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.548 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:40:05.548 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:05.548 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.549 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.549 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:05.549 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.550 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:40:05.550 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:05.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.551 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.551 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:40:05.552 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:05.552 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.552 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.553 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:05.553 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.553 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:40:05.554 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:05.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.555 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.555 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:40:05.555 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:05.555 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.556 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.556 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:05.556 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.557 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:40:05.558 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:05.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.559 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:40:05.559 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:05.559 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.559 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.560 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:40:05.560 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:40:05.560 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:40:05.561 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:40:05.561 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:05.561 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:05.561 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:05.562 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.562 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:05.563 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:40:05.563 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:05.563 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:05.563 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:05.564 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:05.564 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:05.564 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:05.564 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:05.564 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:05.572 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.573 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.573 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:40:05.573 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:05.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.574 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.574 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:40:05.575 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:05.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.575 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.576 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:05.576 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.577 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.577 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:40:05.577 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:05.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.578 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:40:05.578 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:05.579 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.579 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.579 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:05.580 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.580 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.580 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:40:05.581 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:05.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.582 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:40:05.582 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:05.582 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.582 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.583 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:05.583 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.584 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.584 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:40:05.584 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:05.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.585 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.585 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:40:05.586 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:05.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.586 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.587 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:40:05.587 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:40:05.587 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:40:05.588 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:40:05.588 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:05.588 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:05.588 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:05.589 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.590 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:05.590 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:40:05.590 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:05.590 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:05.591 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:05.591 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:05.591 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:05.591 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:05.592 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:05.592 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:05.592 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:05.592 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:05.593 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:05.593 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.594 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:40:05.594 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:05.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.595 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.595 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:40:05.595 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:05.595 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.596 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.596 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:05.596 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.597 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.597 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:40:05.597 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:05.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.598 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.598 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:40:05.599 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:05.599 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.599 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.600 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:05.600 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.600 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.601 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:40:05.601 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:05.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.602 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.602 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:40:05.602 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:05.602 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.603 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.603 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:05.603 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.604 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.604 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:40:05.604 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:05.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.605 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:40:05.606 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:05.606 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.606 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.607 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:40:05.607 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:40:05.607 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:40:05.608 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:40:05.608 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:05.608 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:05.608 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:05.609 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.609 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:05.610 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:40:05.610 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:05.610 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:05.610 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:05.611 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:05.611 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:05.611 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:05.611 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:05.612 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 76, 2022-07-21T01:40:05.612 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:05.612 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:05.612 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:05.613 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.613 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.613 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:40:05.613 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:05.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.614 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.614 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:40:05.615 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:05.615 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.615 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.616 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:05.616 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.616 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.617 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:40:05.617 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:05.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.618 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.618 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:40:05.618 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:05.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.619 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.619 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:05.619 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.620 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.620 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:40:05.620 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:05.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.621 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.621 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:40:05.622 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:05.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.622 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.622 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:05.623 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.623 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:40:05.624 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:05.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.624 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.625 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:40:05.625 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:05.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.625 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.626 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:40:05.626 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:40:05.626 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:40:05.627 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:40:05.627 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:05.627 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:05.627 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:05.628 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.628 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:05.629 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:40:05.629 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:05.629 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:05.629 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:05.630 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:05.630 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:05.630 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:05.630 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:05.631 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:05.631 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:05.631 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:05.631 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:05.632 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.632 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.632 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:40:05.632 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:05.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.633 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.633 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:40:05.634 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:05.634 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.634 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.635 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:05.635 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.636 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:40:05.636 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:05.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.637 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:40:05.637 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:05.637 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.637 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.638 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:05.638 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.638 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.639 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:40:05.639 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:05.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.639 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.640 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:40:05.640 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:05.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.640 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.641 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:05.641 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.642 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.642 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:40:05.642 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:05.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.643 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.643 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:40:05.643 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:05.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.644 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.644 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:40:05.644 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:40:05.645 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:40:05.645 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:40:05.645 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:05.645 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:05.646 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:05.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.646 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:05.647 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:40:05.647 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:05.647 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:05.647 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:05.648 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:05.648 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:05.648 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:05.648 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:05.648 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:05.649 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:05.649 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:05.649 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:05.649 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.650 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.650 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:40:05.650 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:05.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.651 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:40:05.651 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:05.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.652 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.652 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:05.652 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.653 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.653 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:40:05.653 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:05.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.654 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:40:05.654 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:05.655 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.655 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.655 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:05.655 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.656 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.656 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:40:05.656 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:05.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.657 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.657 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:40:05.657 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:05.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.658 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.658 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:05.658 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.659 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.659 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:40:05.659 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:05.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.660 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.660 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:40:05.660 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:05.661 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.661 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.661 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:40:05.662 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:40:05.662 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:40:05.662 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:40:05.662 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:05.662 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:05.663 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:05.663 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.664 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:05.664 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:40:05.664 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:05.664 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:05.664 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:05.665 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:05.665 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:05.665 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:05.665 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:05.666 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 76, 2022-07-21T01:40:05.666 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:05.666 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:05.666 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:05.666 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.667 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:40:05.667 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:05.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.668 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.668 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:40:05.668 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:05.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.669 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.669 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:05.669 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.670 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.670 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:40:05.670 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:05.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.671 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.671 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:40:05.672 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:05.672 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.672 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.672 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:05.673 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.673 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:40:05.674 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:05.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.674 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:40:05.675 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:05.675 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.675 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.676 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:05.676 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:05.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.676 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:05.676 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:40:05.677 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:05.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.677 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:05.677 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:40:05.678 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:05.678 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.678 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.679 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:40:05.679 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:40:05.679 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:40:05.679 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:40:05.679 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:05.680 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:05.680 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:05.680 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:05.680 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.681 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:05.681 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:40:05.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.681 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:05.681 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:05.682 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:05.682 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:05.682 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:05.682 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:05.683 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:40:05.683 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:05.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.683 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:05.684 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:05.684 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:05.684 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:05.684 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:05.684 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:05.685 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:40:05.685 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:05.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.686 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:05.686 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:05.686 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:05.686 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:05.686 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:05.687 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:05.687 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:40:05.687 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:05.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.688 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:05.688 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:05.688 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:05.688 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:05.689 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:05.689 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:05.689 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:40:05.689 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:05.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.690 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:05.690 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:05.690 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:05.691 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:05.691 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:05.691 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:05.691 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:40:05.691 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:05.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.692 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:05.692 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:05.693 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:05.693 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:05.693 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:05.693 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:05.693 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:40:05.694 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:05.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.694 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:05.694 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:05.695 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:05.695 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:05.695 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:05.695 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:05.696 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:40:05.696 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:05.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:05.696 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:05.697 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:05.697 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:05.697 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:05.697 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:05.698 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:05.698 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:40:05.698 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:05.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.698 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:05.699 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:40:05.699 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:40:05.699 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:40:05.699 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:40:05.700 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:40:05.700 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.171:0/752832426": "2022-07-21T02:39:54.585234+0000" 2022-07-21T01:40:05.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.700 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:40:05.700 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:40:05.701 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:40:05.701 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:05.701 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:40:05.701 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:40:05.701 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:05.702 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:40:05.702 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:05.702 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:05.702 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:40:05.703 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:40:05.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.703 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:40:05.703 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:05.703 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:05.704 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:40:05.704 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:05.704 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:05.704 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:40:05.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:05.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:05.705 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:40:05.705 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:40:05.705 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:40:05.706 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:40:05.706 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:40:05.706 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:40:05.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 msgr2=0x7f8a08055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f8a2000b000 tx=0x7f8a20006df0).stop 2022-07-21T01:40:05.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c148530 msgr2=0x7f8a2c143ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c148530 0x7f8a2c143ec0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f8a24001250 tx=0x7f8a24007d30).stop 2022-07-21T01:40:05.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 shutdown_connections 2022-07-21T01:40:05.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a2c148530 0x7f8a2c143ec0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2c0b1f40 0x7f8a2c1439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 --2- 172.21.15.3:0/21521357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2c0a7250 0x7f8a2c0ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 >> 172.21.15.3:0/21521357 conn(0x7f8a2c05f840 msgr2=0x7f8a2c0b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:05.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 shutdown_connections 2022-07-21T01:40:05.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.450+0000 7f8a314b0700 1 -- 172.21.15.3:0/21521357 wait complete. 2022-07-21T01:40:05.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:05.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.822+0000 7fe54f4d5700 1 Processor -- start 2022-07-21T01:40:05.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.822+0000 7fe54f4d5700 1 -- start start 2022-07-21T01:40:05.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.822+0000 7fe54f4d5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5480b1bc0 0x7fe548066b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.822+0000 7fe54f4d5700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54805f8e0 con 0x7fe548066780 2022-07-21T01:40:05.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.822+0000 7fe54f4d5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5480b1630 con 0x7fe548140e40 2022-07-21T01:40:05.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.822+0000 7fe54f4d5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54805fb10 con 0x7fe5480b1bc0 2022-07-21T01:40:05.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.822+0000 7fe54e4d3700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe548066780 0x7fe54805ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55564/0 (socket says 172.21.15.3:55564) 2022-07-21T01:40:05.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.822+0000 7fe54e4d3700 1 -- 172.21.15.3:0/1626767568 learned_addr learned my addr 172.21.15.3:0/1626767568 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:05.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.822+0000 7fe54dcd2700 1 --2- 172.21.15.3:0/1626767568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5480b1bc0 0x7fe548066b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54d4d1700 1 -- 172.21.15.3:0/1626767568 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915084666 0 0) 0x7fe54805f8e0 con 0x7fe548066780 2022-07-21T01:40:05.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54d4d1700 1 -- 172.21.15.3:0/1626767568 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe530003260 con 0x7fe548066780 2022-07-21T01:40:05.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54d4d1700 1 -- 172.21.15.3:0/1626767568 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010238631 0 0) 0x7fe5480b1630 con 0x7fe548140e40 2022-07-21T01:40:05.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54d4d1700 1 -- 172.21.15.3:0/1626767568 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54805f8e0 con 0x7fe548140e40 2022-07-21T01:40:05.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54d4d1700 1 -- 172.21.15.3:0/1626767568 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1234353679 0 0) 0x7fe530003260 con 0x7fe548066780 2022-07-21T01:40:05.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54dcd2700 1 -- 172.21.15.3:0/1626767568 >> v1:172.21.15.143:6789/0 conn(0x7fe548140e40 legacy=0x7fe54813e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54dcd2700 1 -- 172.21.15.3:0/1626767568 >> v1:172.21.15.3:6789/0 conn(0x7fe548066780 legacy=0x7fe54805ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54dcd2700 1 -- 172.21.15.3:0/1626767568 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480b18d0 con 0x7fe5480b1bc0 2022-07-21T01:40:05.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54dcd2700 1 --2- 172.21.15.3:0/1626767568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5480b1bc0 0x7fe548066b60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fe544001800 tx=0x7fe538006df0).ready entity=mon.2 client_cookie=5907a3ee2519d05d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:05.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54d4d1700 1 -- 172.21.15.3:0/1626767568 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe544001660 con 0x7fe5480b1bc0 2022-07-21T01:40:05.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54d4d1700 1 -- 172.21.15.3:0/1626767568 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe538000e90 con 0x7fe5480b1bc0 2022-07-21T01:40:05.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54d4d1700 1 -- 172.21.15.3:0/1626767568 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe538012460 con 0x7fe5480b1bc0 2022-07-21T01:40:05.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1626767568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5480b1bc0 msgr2=0x7fe548066b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54f4d5700 1 --2- 172.21.15.3:0/1626767568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5480b1bc0 0x7fe548066b60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fe544001800 tx=0x7fe538006df0).stop 2022-07-21T01:40:05.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1626767568 shutdown_connections 2022-07-21T01:40:05.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54f4d5700 1 --2- 172.21.15.3:0/1626767568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5480b1bc0 0x7fe548066b60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1626767568 >> 172.21.15.3:0/1626767568 conn(0x7fe548063870 msgr2=0x7fe548063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:05.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1626767568 shutdown_connections 2022-07-21T01:40:05.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1626767568 wait complete. 2022-07-21T01:40:05.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54f4d5700 1 Processor -- start 2022-07-21T01:40:05.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.826+0000 7fe54f4d5700 1 -- start start 2022-07-21T01:40:05.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54f4d5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe548066780 0x7fe5480b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54f4d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5480b1bc0 0x7fe548153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54f4d5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe548140e40 0x7fe548144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54f4d5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54814ffe0 con 0x7fe5480b1bc0 2022-07-21T01:40:05.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54f4d5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480ac310 con 0x7fe548140e40 2022-07-21T01:40:05.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54f4d5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480ab8a0 con 0x7fe548066780 2022-07-21T01:40:05.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54dcd2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5480b1bc0 0x7fe548153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54dcd2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5480b1bc0 0x7fe548153f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47292/0 (socket says 172.21.15.3:47292) 2022-07-21T01:40:05.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54dcd2700 1 -- 172.21.15.3:0/1715876808 learned_addr learned my addr 172.21.15.3:0/1715876808 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:05.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54e4d3700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe548066780 0x7fe5480b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54ecd4700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe548140e40 0x7fe548144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54dcd2700 1 -- 172.21.15.3:0/1715876808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe548066780 msgr2=0x7fe5480b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54dcd2700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe548066780 0x7fe5480b0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54dcd2700 1 -- 172.21.15.3:0/1715876808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe548140e40 msgr2=0x7fe548144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54dcd2700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe548140e40 0x7fe548144910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54dcd2700 1 -- 172.21.15.3:0/1715876808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53800b180 con 0x7fe5480b1bc0 2022-07-21T01:40:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54e4d3700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe548066780 0x7fe5480b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54dcd2700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5480b1bc0 0x7fe548153f10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7fe544001800 tx=0x7fe538014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe5377fe700 1 -- 172.21.15.3:0/1715876808 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe53800da20 con 0x7fe5480b1bc0 2022-07-21T01:40:05.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe5377fe700 1 -- 172.21.15.3:0/1715876808 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53800db80 con 0x7fe5480b1bc0 2022-07-21T01:40:05.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54809ebc0 con 0x7fe5480b1bc0 2022-07-21T01:40:05.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe5377fe700 1 -- 172.21.15.3:0/1715876808 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe53801e400 con 0x7fe5480b1bc0 2022-07-21T01:40:05.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.830+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5481503d0 con 0x7fe5480b1bc0 2022-07-21T01:40:05.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.834+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54809f060 con 0x7fe5480b1bc0 2022-07-21T01:40:05.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.834+0000 7fe5377fe700 1 -- 172.21.15.3:0/1715876808 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe538025030 con 0x7fe5480b1bc0 2022-07-21T01:40:05.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.834+0000 7fe5377fe700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe528053b00 0x7fe528055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:05.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.834+0000 7fe5377fe700 1 -- 172.21.15.3:0/1715876808 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7fe53806cd30 con 0x7fe5480b1bc0 2022-07-21T01:40:05.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.834+0000 7fe54e4d3700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe528053b00 0x7fe528055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:05.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.834+0000 7fe54e4d3700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe528053b00 0x7fe528055fc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fe54400b000 tx=0x7fe544001660).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:05.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:05.838+0000 7fe5377fe700 1 -- 172.21.15.3:0/1715876808 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe54809f060 con 0x7fe5480b1bc0 2022-07-21T01:40:06.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.002+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe54809f060 con 0x7fe5480b1bc0 2022-07-21T01:40:06.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe5377fe700 1 -- 172.21.15.3:0/1715876808 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2487 (secure 0 0 0) 0x7fe54809f060 con 0x7fe5480b1bc0 2022-07-21T01:40:06.008 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:06.008 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:39:09.622528+0000","modified":"2022-07-21T01:39:10.626387+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6259":{"gid":6259,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/862725488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":862725488},{"type":"v1","addr":"172.21.15.143:6833","nonce":862725488}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-21T01:40:06.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe528053b00 msgr2=0x7fe528055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe528053b00 0x7fe528055fc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fe54400b000 tx=0x7fe544001660).stop 2022-07-21T01:40:06.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5480b1bc0 msgr2=0x7fe548153f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5480b1bc0 0x7fe548153f10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7fe544001800 tx=0x7fe538014000).stop 2022-07-21T01:40:06.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 shutdown_connections 2022-07-21T01:40:06.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe528053b00 0x7fe528055fc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe548140e40 0x7fe548144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5480b1bc0 0x7fe548153f10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 --2- 172.21.15.3:0/1715876808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe548066780 0x7fe5480b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 >> 172.21.15.3:0/1715876808 conn(0x7fe548063870 msgr2=0x7fe54813db10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:06.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 shutdown_connections 2022-07-21T01:40:06.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.006+0000 7fe54f4d5700 1 -- 172.21.15.3:0/1715876808 wait complete. 2022-07-21T01:40:06.013 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2022-07-21T01:40:06.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:40:06.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f11580bc700 1 Processor -- start 2022-07-21T01:40:06.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f11580bc700 1 -- start start 2022-07-21T01:40:06.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f11580bc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11500b1ee0 0x7f11500b0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f11580bc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500b22c0 con 0x7f11500a5ab0 2022-07-21T01:40:06.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f11580bc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500ae040 con 0x7f115005c620 2022-07-21T01:40:06.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f11580bc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11500b2c10 con 0x7f11500b1ee0 2022-07-21T01:40:06.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f1156659700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f11500a5ab0 0x7f11500a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55582/0 (socket says 172.21.15.3:55582) 2022-07-21T01:40:06.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f1156659700 1 -- 172.21.15.3:0/1730110606 learned_addr learned my addr 172.21.15.3:0/1730110606 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:06.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f1155e58700 1 --2- 172.21.15.3:0/1730110606 >> v2:172.21.15.143:3300/0 conn(0x7f11500b1ee0 0x7f11500b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.114+0000 7f1154e56700 1 -- 172.21.15.3:0/1730110606 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95887532 0 0) 0x7f11500b22c0 con 0x7f11500a5ab0 2022-07-21T01:40:06.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1154e56700 1 -- 172.21.15.3:0/1730110606 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1138003260 con 0x7f11500a5ab0 2022-07-21T01:40:06.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1155e58700 1 -- 172.21.15.3:0/1730110606 >> v1:172.21.15.143:6789/0 conn(0x7f115005c620 legacy=0x7f11500ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1155e58700 1 -- 172.21.15.3:0/1730110606 >> v1:172.21.15.3:6789/0 conn(0x7f11500a5ab0 legacy=0x7f11500a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1155e58700 1 -- 172.21.15.3:0/1730110606 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11500b2eb0 con 0x7f11500b1ee0 2022-07-21T01:40:06.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1155e58700 1 --2- 172.21.15.3:0/1730110606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500b1ee0 0x7f11500b0680 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f114c002f70 tx=0x7f114c006df0).ready entity=mon.1 client_cookie=959a9c04dc8a0d72 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:06.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1154e56700 1 -- 172.21.15.3:0/1730110606 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f114c00d610 con 0x7f11500b1ee0 2022-07-21T01:40:06.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1154e56700 1 -- 172.21.15.3:0/1730110606 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c000e90 con 0x7f11500b1ee0 2022-07-21T01:40:06.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1154e56700 1 -- 172.21.15.3:0/1730110606 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f114c00d960 con 0x7f11500b1ee0 2022-07-21T01:40:06.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 -- 172.21.15.3:0/1730110606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500b1ee0 msgr2=0x7f11500b0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 --2- 172.21.15.3:0/1730110606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500b1ee0 0x7f11500b0680 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f114c002f70 tx=0x7f114c006df0).stop 2022-07-21T01:40:06.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 -- 172.21.15.3:0/1730110606 shutdown_connections 2022-07-21T01:40:06.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 --2- 172.21.15.3:0/1730110606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500b1ee0 0x7f11500b0680 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 -- 172.21.15.3:0/1730110606 >> 172.21.15.3:0/1730110606 conn(0x7f115005ceb0 msgr2=0x7f1150063130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:06.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 -- 172.21.15.3:0/1730110606 shutdown_connections 2022-07-21T01:40:06.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 -- 172.21.15.3:0/1730110606 wait complete. 2022-07-21T01:40:06.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 Processor -- start 2022-07-21T01:40:06.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 -- start start 2022-07-21T01:40:06.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115005c620 0x7f1150148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500a5ab0 0x7f1150156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11500b1ee0 0x7f1150152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1150156940 con 0x7f115005c620 2022-07-21T01:40:06.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11500aa030 con 0x7f11500a5ab0 2022-07-21T01:40:06.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f11580bc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11500b4da0 con 0x7f11500b1ee0 2022-07-21T01:40:06.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1155e58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115005c620 0x7f1150148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1155e58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115005c620 0x7f1150148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47306/0 (socket says 172.21.15.3:47306) 2022-07-21T01:40:06.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1155e58700 1 -- 172.21.15.3:0/2890603734 learned_addr learned my addr 172.21.15.3:0/2890603734 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:06.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1155657700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500a5ab0 0x7f1150156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.118+0000 7f1156659700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11500b1ee0 0x7f1150152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f1155657700 1 -- 172.21.15.3:0/2890603734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11500b1ee0 msgr2=0x7f1150152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f1155657700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11500b1ee0 0x7f1150152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f1155657700 1 -- 172.21.15.3:0/2890603734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115005c620 msgr2=0x7f1150148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f1155657700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115005c620 0x7f1150148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f1155657700 1 -- 172.21.15.3:0/2890603734 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c00b180 con 0x7f11500a5ab0 2022-07-21T01:40:06.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f1155657700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500a5ab0 0x7f1150156450 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f11500b55b0 tx=0x7f1140007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:06.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f113effd700 1 -- 172.21.15.3:0/2890603734 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f114000ccc0 con 0x7f11500a5ab0 2022-07-21T01:40:06.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f113effd700 1 -- 172.21.15.3:0/2890603734 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114000ce20 con 0x7f11500a5ab0 2022-07-21T01:40:06.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f113effd700 1 -- 172.21.15.3:0/2890603734 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11400125f0 con 0x7f11500a5ab0 2022-07-21T01:40:06.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11500b5040 con 0x7f11500a5ab0 2022-07-21T01:40:06.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11500b5650 con 0x7f11500a5ab0 2022-07-21T01:40:06.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11500574e0 con 0x7f11500a5ab0 2022-07-21T01:40:06.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.122+0000 7f113effd700 1 -- 172.21.15.3:0/2890603734 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1140019030 con 0x7f11500a5ab0 2022-07-21T01:40:06.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.126+0000 7f113effd700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1130053a70 0x7f1130055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.126+0000 7f113effd700 1 -- 172.21.15.3:0/2890603734 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6463+0+0 (secure 0 0 0) 0x7f114006b270 con 0x7f11500a5ab0 2022-07-21T01:40:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.126+0000 7f1155e58700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1130053a70 0x7f1130055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.126+0000 7f1155e58700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1130053a70 0x7f1130055f30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f114c01a000 tx=0x7f114c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:06.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.134+0000 7f113effd700 1 -- 172.21.15.3:0/2890603734 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1140045e60 con 0x7f11500a5ab0 2022-07-21T01:40:06.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.298+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f11501525c0 con 0x7f11500a5ab0 2022-07-21T01:40:06.783 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:40:06.782+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:40:06.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.794+0000 7f113effd700 1 -- 172.21.15.3:0/2890603734 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) v1 ==== 161+0+0 (secure 0 0 0) 0x7f11400459a0 con 0x7f11500a5ab0 2022-07-21T01:40:06.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1130053a70 msgr2=0x7f1130055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1130053a70 0x7f1130055f30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f114c01a000 tx=0x7f114c014000).stop 2022-07-21T01:40:06.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500a5ab0 msgr2=0x7f1150156450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500a5ab0 0x7f1150156450 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f11500b55b0 tx=0x7f1140007b90).stop 2022-07-21T01:40:06.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 shutdown_connections 2022-07-21T01:40:06.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11500b1ee0 0x7f1150152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1130053a70 0x7f1130055f30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11500a5ab0 0x7f1150156450 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 --2- 172.21.15.3:0/2890603734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115005c620 0x7f1150148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 >> 172.21.15.3:0/2890603734 conn(0x7f115005ceb0 msgr2=0x7f115013dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:06.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 shutdown_connections 2022-07-21T01:40:06.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.798+0000 7f11580bc700 1 -- 172.21.15.3:0/2890603734 wait complete. 2022-07-21T01:40:06.806 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:40:06.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:06.834 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:40:06.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c501d700 1 Processor -- start 2022-07-21T01:40:06.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c501d700 1 -- start start 2022-07-21T01:40:06.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c501d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b2930 0x7fe6c0147600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c501d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe6c005c620 0x7fe6c00a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c501d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c014ba70 con 0x7fe6c00b1ea0 2022-07-21T01:40:06.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c501d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c0145340 con 0x7fe6c005c620 2022-07-21T01:40:06.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c501d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00b2d10 con 0x7fe6c00b2930 2022-07-21T01:40:06.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6bffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b2930 0x7fe6c0147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c481c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe6c005c620 0x7fe6c00a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6bf7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fe6c00b1ea0 0x7fe6c014b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:57708/0 (socket says 172.21.15.3:57708) 2022-07-21T01:40:06.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6bf7fe700 1 -- 172.21.15.3:0/955409273 learned_addr learned my addr 172.21.15.3:0/955409273 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:06.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6beffd700 1 -- 172.21.15.3:0/955409273 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242485959 0 0) 0x7fe6c014ba70 con 0x7fe6c00b1ea0 2022-07-21T01:40:06.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6beffd700 1 -- 172.21.15.3:0/955409273 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a0003260 con 0x7fe6c00b1ea0 2022-07-21T01:40:06.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6bffff700 1 -- 172.21.15.3:0/955409273 >> v2:172.21.15.143:3300/0 conn(0x7fe6c005c620 msgr2=0x7fe6c00a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:06.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6bffff700 1 --2- 172.21.15.3:0/955409273 >> v2:172.21.15.143:3300/0 conn(0x7fe6c005c620 0x7fe6c00a8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6bffff700 1 -- 172.21.15.3:0/955409273 >> v1:172.21.15.143:6789/0 conn(0x7fe6c00b1ea0 legacy=0x7fe6c014b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6bffff700 1 -- 172.21.15.3:0/955409273 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c014ac60 con 0x7fe6c00b2930 2022-07-21T01:40:06.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c481c700 1 --2- 172.21.15.3:0/955409273 >> v2:172.21.15.143:3300/0 conn(0x7fe6c005c620 0x7fe6c00a8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:06.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6bffff700 1 --2- 172.21.15.3:0/955409273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b2930 0x7fe6c0147600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe6a8002f70 tx=0x7fe6a8006df0).ready entity=mon.2 client_cookie=8bfdcc51d6ed60d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:06.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6beffd700 1 -- 172.21.15.3:0/955409273 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a800d4a0 con 0x7fe6c00b2930 2022-07-21T01:40:06.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6beffd700 1 -- 172.21.15.3:0/955409273 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a8000e90 con 0x7fe6c00b2930 2022-07-21T01:40:06.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6beffd700 1 -- 172.21.15.3:0/955409273 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a800d7f0 con 0x7fe6c00b2930 2022-07-21T01:40:06.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.926+0000 7fe6c501d700 1 -- 172.21.15.3:0/955409273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b2930 msgr2=0x7fe6c0147600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 --2- 172.21.15.3:0/955409273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b2930 0x7fe6c0147600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe6a8002f70 tx=0x7fe6a8006df0).stop 2022-07-21T01:40:06.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 -- 172.21.15.3:0/955409273 shutdown_connections 2022-07-21T01:40:06.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 --2- 172.21.15.3:0/955409273 >> v2:172.21.15.143:3300/0 conn(0x7fe6c005c620 0x7fe6c00a8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 --2- 172.21.15.3:0/955409273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b2930 0x7fe6c0147600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 -- 172.21.15.3:0/955409273 >> 172.21.15.3:0/955409273 conn(0x7fe6c00612d0 msgr2=0x7fe6c0062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:06.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 -- 172.21.15.3:0/955409273 shutdown_connections 2022-07-21T01:40:06.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 -- 172.21.15.3:0/955409273 wait complete. 2022-07-21T01:40:06.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 Processor -- start 2022-07-21T01:40:06.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 -- start start 2022-07-21T01:40:06.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c005c620 0x7fe6c00abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b1ea0 0x7fe6c00ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c00b2930 0x7fe6c015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c0156870 con 0x7fe6c00b2930 2022-07-21T01:40:06.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00ac7f0 con 0x7fe6c005c620 2022-07-21T01:40:06.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c501d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00aa380 con 0x7fe6c00b1ea0 2022-07-21T01:40:06.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c481c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c00b2930 0x7fe6c015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c481c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c00b2930 0x7fe6c015a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47322/0 (socket says 172.21.15.3:47322) 2022-07-21T01:40:06.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6c481c700 1 -- 172.21.15.3:0/1388362404 learned_addr learned my addr 172.21.15.3:0/1388362404 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:06.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6bf7fe700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b1ea0 0x7fe6c00ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.930+0000 7fe6bffff700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c005c620 0x7fe6c00abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6c481c700 1 -- 172.21.15.3:0/1388362404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b1ea0 msgr2=0x7fe6c00ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6c481c700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b1ea0 0x7fe6c00ac210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6c481c700 1 -- 172.21.15.3:0/1388362404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c005c620 msgr2=0x7fe6c00abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:06.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6c481c700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c005c620 0x7fe6c00abd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:06.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6c481c700 1 -- 172.21.15.3:0/1388362404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b000b180 con 0x7fe6c00b2930 2022-07-21T01:40:06.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6c481c700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c00b2930 0x7fe6c015a0d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7fe6b000c000 tx=0x7fe6b0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:06.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6bd7fa700 1 -- 172.21.15.3:0/1388362404 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6b0010780 con 0x7fe6c00b2930 2022-07-21T01:40:06.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6bd7fa700 1 -- 172.21.15.3:0/1388362404 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b00108e0 con 0x7fe6c00b2930 2022-07-21T01:40:06.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a800b180 con 0x7fe6c00b2930 2022-07-21T01:40:06.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6bd7fa700 1 -- 172.21.15.3:0/1388362404 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6b0010c40 con 0x7fe6c00b2930 2022-07-21T01:40:06.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c00af820 con 0x7fe6c00b2930 2022-07-21T01:40:06.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c00ac9b0 con 0x7fe6c00b2930 2022-07-21T01:40:06.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6bd7fa700 1 -- 172.21.15.3:0/1388362404 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe6b001c460 con 0x7fe6c00b2930 2022-07-21T01:40:06.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.934+0000 7fe6bd7fa700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6a4053950 0x7fe6a4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:06.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.938+0000 7fe6bd7fa700 1 -- 172.21.15.3:0/1388362404 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6549+0+0 (secure 0 0 0) 0x7fe6b006c530 con 0x7fe6c00b2930 2022-07-21T01:40:06.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.938+0000 7fe6bffff700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6a4053950 0x7fe6a4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:06.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.938+0000 7fe6bffff700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6a4053950 0x7fe6a4055e10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fe6a801a000 tx=0x7fe6a8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:06.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:06.942+0000 7fe6bd7fa700 1 -- 172.21.15.3:0/1388362404 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe6b0043210 con 0x7fe6c00b2930 2022-07-21T01:40:07.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6c00574e0 con 0x7fe6c00b2930 2022-07-21T01:40:07.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6bd7fa700 1 -- 172.21.15.3:0/1388362404 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2126 (secure 0 0 0) 0x7fe6b0046830 con 0x7fe6c00b2930 2022-07-21T01:40:07.114 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:07.115 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":36,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:39:09.622528+0000","modified":"2022-07-21T01:40:06.785435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-21T01:40:07.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6a4053950 msgr2=0x7fe6a4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6a4053950 0x7fe6a4055e10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fe6a801a000 tx=0x7fe6a8014000).stop 2022-07-21T01:40:07.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c00b2930 msgr2=0x7fe6c015a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c00b2930 0x7fe6c015a0d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7fe6b000c000 tx=0x7fe6b0006df0).stop 2022-07-21T01:40:07.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 shutdown_connections 2022-07-21T01:40:07.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c00b2930 0x7fe6c015a0d0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6c00b1ea0 0x7fe6c00ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6a4053950 0x7fe6a4055e10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 --2- 172.21.15.3:0/1388362404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c005c620 0x7fe6c00abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.110+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 >> 172.21.15.3:0/1388362404 conn(0x7fe6c00612d0 msgr2=0x7fe6c014b280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:07.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.114+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 shutdown_connections 2022-07-21T01:40:07.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.114+0000 7fe6c501d700 1 -- 172.21.15.3:0/1388362404 wait complete. 2022-07-21T01:40:07.122 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 36 2022-07-21T01:40:07.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:40:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf89f8700 1 Processor -- start 2022-07-21T01:40:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf89f8700 1 -- start start 2022-07-21T01:40:07.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf89f8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0cf40b1460 0x7f0cf40b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf89f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf41413e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf89f8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf40a7960 con 0x7f0cf40b2050 2022-07-21T01:40:07.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf89f8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf40b18d0 con 0x7f0cf40b1460 2022-07-21T01:40:07.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf89f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf40b07e0 con 0x7f0cf40b2430 2022-07-21T01:40:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf41413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf41413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:54246/0 (socket says 172.21.15.3:54246) 2022-07-21T01:40:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf2d9d700 1 -- 172.21.15.3:0/3782432234 learned_addr learned my addr 172.21.15.3:0/3782432234 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf1d9b700 1 --2- 172.21.15.3:0/3782432234 >> v2:172.21.15.143:3300/0 conn(0x7f0cf40b1460 0x7f0cf40b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf2d9d700 1 -- 172.21.15.3:0/3782432234 >> v2:172.21.15.143:3300/0 conn(0x7f0cf40b1460 msgr2=0x7f0cf40b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf2d9d700 1 --2- 172.21.15.3:0/3782432234 >> v2:172.21.15.143:3300/0 conn(0x7f0cf40b1460 0x7f0cf40b0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf2d9d700 1 -- 172.21.15.3:0/3782432234 >> v1:172.21.15.143:6789/0 conn(0x7f0cf40b2050 legacy=0x7f0cf40a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.234+0000 7f0cf2d9d700 1 -- 172.21.15.3:0/3782432234 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf40b0a80 con 0x7f0cf40b2430 2022-07-21T01:40:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf2d9d700 1 --2- 172.21.15.3:0/3782432234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf41413e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f0cec0032f0 tx=0x7f0cec009290).ready entity=mon.2 client_cookie=91636263e215fd95 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf159a700 1 -- 172.21.15.3:0/3782432234 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cec00b8a0 con 0x7f0cf40b2430 2022-07-21T01:40:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf159a700 1 -- 172.21.15.3:0/3782432234 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec00ba00 con 0x7f0cf40b2430 2022-07-21T01:40:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf159a700 1 -- 172.21.15.3:0/3782432234 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cec00bb70 con 0x7f0cf40b2430 2022-07-21T01:40:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3782432234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 msgr2=0x7f0cf41413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 --2- 172.21.15.3:0/3782432234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf41413e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f0cec0032f0 tx=0x7f0cec009290).stop 2022-07-21T01:40:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3782432234 shutdown_connections 2022-07-21T01:40:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 --2- 172.21.15.3:0/3782432234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf41413e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 --2- 172.21.15.3:0/3782432234 >> v2:172.21.15.143:3300/0 conn(0x7f0cf40b1460 0x7f0cf40b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3782432234 >> 172.21.15.3:0/3782432234 conn(0x7f0cf405f840 msgr2=0x7f0cf405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3782432234 shutdown_connections 2022-07-21T01:40:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3782432234 wait complete. 2022-07-21T01:40:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 Processor -- start 2022-07-21T01:40:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.238+0000 7f0cf89f8700 1 -- start start 2022-07-21T01:40:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf89f8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf40b1460 0x7f0cf40ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf89f8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf40b2050 0x7f0cf4147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf89f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf4148090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf89f8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4157cb0 con 0x7f0cf40b1460 2022-07-21T01:40:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf89f8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf40aeef0 con 0x7f0cf40b2050 2022-07-21T01:40:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf89f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4157060 con 0x7f0cf40b2430 2022-07-21T01:40:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf1d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf40b2050 0x7f0cf4147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf40b1460 0x7f0cf40ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf1d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf40b2050 0x7f0cf4147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:52782/0 (socket says 172.21.15.3:52782) 2022-07-21T01:40:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf4148090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf1d9b700 1 -- 172.21.15.3:0/3929955259 learned_addr learned my addr 172.21.15.3:0/3929955259 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf2d9d700 1 -- 172.21.15.3:0/3929955259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf40b2050 msgr2=0x7f0cf4147ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf2d9d700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf40b2050 0x7f0cf4147ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf2d9d700 1 -- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf40b1460 msgr2=0x7f0cf40ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf2d9d700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf40b1460 0x7f0cf40ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf2d9d700 1 -- 172.21.15.3:0/3929955259 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec009530 con 0x7f0cf40b2430 2022-07-21T01:40:07.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf259c700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf40b1460 0x7f0cf40ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:07.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf1d9b700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf40b2050 0x7f0cf4147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:07.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf2d9d700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf4148090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f0cf40aee10 tx=0x7f0cec016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:07.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0ceb7fe700 1 -- 172.21.15.3:0/3929955259 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cec00b8a0 con 0x7f0cf40b2430 2022-07-21T01:40:07.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0ceb7fe700 1 -- 172.21.15.3:0/3929955259 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec021440 con 0x7f0cf40b2430 2022-07-21T01:40:07.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf4148580 con 0x7f0cf40b2430 2022-07-21T01:40:07.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0ceb7fe700 1 -- 172.21.15.3:0/3929955259 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cec0217a0 con 0x7f0cf40b2430 2022-07-21T01:40:07.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf40af150 con 0x7f0cf40b2430 2022-07-21T01:40:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.242+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf40af350 con 0x7f0cf40b2430 2022-07-21T01:40:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.246+0000 7f0ceb7fe700 1 -- 172.21.15.3:0/3929955259 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0cec028070 con 0x7f0cf40b2430 2022-07-21T01:40:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.246+0000 7f0ceb7fe700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0cd8053b00 0x7f0cd8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.246+0000 7f0ceb7fe700 1 -- 172.21.15.3:0/3929955259 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6549+0+0 (secure 0 0 0) 0x7f0cec06ceb0 con 0x7f0cf40b2430 2022-07-21T01:40:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.246+0000 7f0cf259c700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0cd8053b00 0x7f0cd8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.246+0000 7f0cf259c700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0cd8053b00 0x7f0cd8055fc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f0cdc00d000 tx=0x7f0cdc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.254+0000 7f0ceb7fe700 1 -- 172.21.15.3:0/3929955259 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0cec043bf0 con 0x7f0cf40b2430 2022-07-21T01:40:07.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.418+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0cf40af350 con 0x7f0cf40b2430 2022-07-21T01:40:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.798+0000 7f0ceb7fe700 1 -- 172.21.15.3:0/3929955259 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0cf40af350 con 0x7f0cf40b2430 2022-07-21T01:40:07.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0cd8053b00 msgr2=0x7f0cd8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0cd8053b00 0x7f0cd8055fc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f0cdc00d000 tx=0x7f0cdc009040).stop 2022-07-21T01:40:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 msgr2=0x7f0cf4148090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf4148090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f0cf40aee10 tx=0x7f0cec016000).stop 2022-07-21T01:40:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 shutdown_connections 2022-07-21T01:40:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf40b2430 0x7f0cf4148090 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0cd8053b00 0x7f0cd8055fc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf40b2050 0x7f0cf4147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 --2- 172.21.15.3:0/3929955259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cf40b1460 0x7f0cf40ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 >> 172.21.15.3:0/3929955259 conn(0x7f0cf405f840 msgr2=0x7f0cf413d020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 shutdown_connections 2022-07-21T01:40:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.802+0000 7f0cf89f8700 1 -- 172.21.15.3:0/3929955259 wait complete. 2022-07-21T01:40:07.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:40:07.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.934+0000 7feac5f28700 1 Processor -- start 2022-07-21T01:40:07.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.934+0000 7feac5f28700 1 -- start start 2022-07-21T01:40:07.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.934+0000 7feac5f28700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feac00af910 0x7feac0143f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.934+0000 7feac5f28700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7feac00aa0d0 0x7feac01402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.934+0000 7feac5f28700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac00b3a10 con 0x7feac00b3630 2022-07-21T01:40:07.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.934+0000 7feac5f28700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac01443f0 con 0x7feac00af910 2022-07-21T01:40:07.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.934+0000 7feac5f28700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac0146b40 con 0x7feac00aa0d0 2022-07-21T01:40:07.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabeffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feac00af910 0x7feac0143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabeffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feac00af910 0x7feac0143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47342/0 (socket says 172.21.15.3:47342) 2022-07-21T01:40:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabeffd700 1 -- 172.21.15.3:0/3777111294 learned_addr learned my addr 172.21.15.3:0/3777111294 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabf7fe700 1 --1- 172.21.15.3:0/3777111294 >> v1:172.21.15.3:6789/0 conn(0x7feac00b3630 0x7feac00af200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55620/0 (socket says 172.21.15.3:55620) 2022-07-21T01:40:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabffff700 1 --2- 172.21.15.3:0/3777111294 >> v2:172.21.15.143:3300/0 conn(0x7feac00aa0d0 0x7feac01402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabeffd700 1 -- 172.21.15.3:0/3777111294 >> v2:172.21.15.143:3300/0 conn(0x7feac00aa0d0 msgr2=0x7feac01402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabeffd700 1 --2- 172.21.15.3:0/3777111294 >> v2:172.21.15.143:3300/0 conn(0x7feac00aa0d0 0x7feac01402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabeffd700 1 -- 172.21.15.3:0/3777111294 >> v1:172.21.15.3:6789/0 conn(0x7feac00b3630 legacy=0x7feac00af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabeffd700 1 -- 172.21.15.3:0/3777111294 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac0146de0 con 0x7feac00af910 2022-07-21T01:40:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabeffd700 1 --2- 172.21.15.3:0/3777111294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 0x7feac0143f00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7feaa80032f0 tx=0x7feaa8009290).ready entity=mon.0 client_cookie=daf8297a48f91e02 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabe7fc700 1 -- 172.21.15.3:0/3777111294 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaa800b870 con 0x7feac00af910 2022-07-21T01:40:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabe7fc700 1 -- 172.21.15.3:0/3777111294 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa800b9d0 con 0x7feac00af910 2022-07-21T01:40:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feabe7fc700 1 -- 172.21.15.3:0/3777111294 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaa800bb40 con 0x7feac00af910 2022-07-21T01:40:07.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feac5f28700 1 -- 172.21.15.3:0/3777111294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 msgr2=0x7feac0143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feac5f28700 1 --2- 172.21.15.3:0/3777111294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 0x7feac0143f00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7feaa80032f0 tx=0x7feaa8009290).stop 2022-07-21T01:40:07.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feac5f28700 1 -- 172.21.15.3:0/3777111294 shutdown_connections 2022-07-21T01:40:07.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feac5f28700 1 --2- 172.21.15.3:0/3777111294 >> v2:172.21.15.143:3300/0 conn(0x7feac00aa0d0 0x7feac01402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feac5f28700 1 --2- 172.21.15.3:0/3777111294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 0x7feac0143f00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feac5f28700 1 -- 172.21.15.3:0/3777111294 >> 172.21.15.3:0/3777111294 conn(0x7feac0060950 msgr2=0x7feac0060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:07.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feac5f28700 1 -- 172.21.15.3:0/3777111294 shutdown_connections 2022-07-21T01:40:07.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.938+0000 7feac5f28700 1 -- 172.21.15.3:0/3777111294 wait complete. 2022-07-21T01:40:07.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 Processor -- start 2022-07-21T01:40:07.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 -- start start 2022-07-21T01:40:07.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac00aa0d0 0x7feac0141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 0x7feac00b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac00b3630 0x7feac0148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0145730 con 0x7feac00af910 2022-07-21T01:40:07.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0148f70 con 0x7feac00b3630 2022-07-21T01:40:07.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac0145960 con 0x7feac00aa0d0 2022-07-21T01:40:07.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 0x7feac00b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 0x7feac00b5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47348/0 (socket says 172.21.15.3:47348) 2022-07-21T01:40:07.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabeffd700 1 -- 172.21.15.3:0/482166554 learned_addr learned my addr 172.21.15.3:0/482166554 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:07.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabf7fe700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac00aa0d0 0x7feac0141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabffff700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac00b3630 0x7feac0148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabeffd700 1 -- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac00aa0d0 msgr2=0x7feac0141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabeffd700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac00aa0d0 0x7feac0141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabeffd700 1 -- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac00b3630 msgr2=0x7feac0148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:07.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabeffd700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac00b3630 0x7feac0148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:07.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabeffd700 1 -- 172.21.15.3:0/482166554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa8009530 con 0x7feac00af910 2022-07-21T01:40:07.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabffff700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac00b3630 0x7feac0148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:07.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabf7fe700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac00aa0d0 0x7feac0141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:07.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabeffd700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 0x7feac00b5a00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7feaa80032f0 tx=0x7feaa8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:07.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabcff9700 1 -- 172.21.15.3:0/482166554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaa8006b00 con 0x7feac00af910 2022-07-21T01:40:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabcff9700 1 -- 172.21.15.3:0/482166554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa8021440 con 0x7feac00af910 2022-07-21T01:40:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feabcff9700 1 -- 172.21.15.3:0/482166554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaa80217a0 con 0x7feac00af910 2022-07-21T01:40:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac0145c00 con 0x7feac00af910 2022-07-21T01:40:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.942+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac0146570 con 0x7feac00af910 2022-07-21T01:40:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.946+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac0147080 con 0x7feac00af910 2022-07-21T01:40:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.946+0000 7feabcff9700 1 -- 172.21.15.3:0/482166554 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feaa8027020 con 0x7feac00af910 2022-07-21T01:40:07.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.946+0000 7feabcff9700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feaa4053b00 0x7feaa4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:07.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.946+0000 7feabcff9700 1 -- 172.21.15.3:0/482166554 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6549+0+0 (secure 0 0 0) 0x7feaa806d7e0 con 0x7feac00af910 2022-07-21T01:40:07.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.946+0000 7feabf7fe700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feaa4053b00 0x7feaa4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:07.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.946+0000 7feabf7fe700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feaa4053b00 0x7feaa4055fc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7feab800b000 tx=0x7feab8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:07.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:07.958+0000 7feabcff9700 1 -- 172.21.15.3:0/482166554 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feaa80444c0 con 0x7feac00af910 2022-07-21T01:40:08.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.086+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feac00574e0 con 0x7feac00af910 2022-07-21T01:40:08.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.870+0000 7feabcff9700 1 -- 172.21.15.3:0/482166554 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v82) v1 ==== 185+0+0 (secure 0 0 0) 0x7feaa8047aa0 con 0x7feac00af910 2022-07-21T01:40:08.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feaa4053b00 msgr2=0x7feaa4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feaa4053b00 0x7feaa4055fc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7feab800b000 tx=0x7feab8000bc0).stop 2022-07-21T01:40:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 msgr2=0x7feac00b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 0x7feac00b5a00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7feaa80032f0 tx=0x7feaa8016000).stop 2022-07-21T01:40:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 shutdown_connections 2022-07-21T01:40:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feaa4053b00 0x7feaa4055fc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feac00b3630 0x7feac0148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac00af910 0x7feac00b5a00 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 --2- 172.21.15.3:0/482166554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feac00aa0d0 0x7feac0141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 >> 172.21.15.3:0/482166554 conn(0x7feac0060950 msgr2=0x7feac013cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 shutdown_connections 2022-07-21T01:40:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:08.874+0000 7feac5f28700 1 -- 172.21.15.3:0/482166554 wait complete. 2022-07-21T01:40:08.881 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:40:08.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:40:09.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e47fff700 1 Processor -- start 2022-07-21T01:40:09.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e47fff700 1 -- start start 2022-07-21T01:40:09.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e47fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e480b3190 0x7f4e480b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:09.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e47fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 0x7f4e48147e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:09.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e47fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e480b3570 con 0x7f4e480b1a80 2022-07-21T01:40:09.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e47fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e480b2980 con 0x7f4e480b3190 2022-07-21T01:40:09.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e47fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e480b0e00 con 0x7f4e480a9da0 2022-07-21T01:40:09.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e477fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 0x7f4e48147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:09.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e46ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e480b3190 0x7f4e480b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:09.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e467fc700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4e480b1a80 0x7f4e480656a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:57752/0 (socket says 172.21.15.3:57752) 2022-07-21T01:40:09.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e467fc700 1 -- 172.21.15.3:0/943202308 learned_addr learned my addr 172.21.15.3:0/943202308 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:09.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e477fe700 1 -- 172.21.15.3:0/943202308 >> v2:172.21.15.143:3300/0 conn(0x7f4e480b3190 msgr2=0x7f4e480b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:09.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e477fe700 1 --2- 172.21.15.3:0/943202308 >> v2:172.21.15.143:3300/0 conn(0x7f4e480b3190 0x7f4e480b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:09.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e477fe700 1 -- 172.21.15.3:0/943202308 >> v1:172.21.15.143:6789/0 conn(0x7f4e480b1a80 legacy=0x7f4e480656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:09.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e45ffb700 1 -- 172.21.15.3:0/943202308 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075035905 0 0) 0x7f4e480b3570 con 0x7f4e480b1a80 2022-07-21T01:40:09.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e477fe700 1 -- 172.21.15.3:0/943202308 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e480b10a0 con 0x7f4e480a9da0 2022-07-21T01:40:09.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e46ffd700 1 --2- 172.21.15.3:0/943202308 >> v2:172.21.15.143:3300/0 conn(0x7f4e480b3190 0x7f4e480b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:09.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e477fe700 1 --2- 172.21.15.3:0/943202308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 0x7f4e48147e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4e400032f0 tx=0x7f4e400091d0).ready entity=mon.2 client_cookie=2ec707f55e9ba0bf server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:09.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e45ffb700 1 -- 172.21.15.3:0/943202308 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e4000b6c0 con 0x7f4e480a9da0 2022-07-21T01:40:09.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e45ffb700 1 -- 172.21.15.3:0/943202308 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4000b820 con 0x7f4e480a9da0 2022-07-21T01:40:09.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e45ffb700 1 -- 172.21.15.3:0/943202308 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e4000b990 con 0x7f4e480a9da0 2022-07-21T01:40:09.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.002+0000 7f4e47fff700 1 -- 172.21.15.3:0/943202308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 msgr2=0x7f4e48147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:09.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 --2- 172.21.15.3:0/943202308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 0x7f4e48147e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4e400032f0 tx=0x7f4e400091d0).stop 2022-07-21T01:40:09.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 -- 172.21.15.3:0/943202308 shutdown_connections 2022-07-21T01:40:09.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 --2- 172.21.15.3:0/943202308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 0x7f4e48147e30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:09.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 --2- 172.21.15.3:0/943202308 >> v2:172.21.15.143:3300/0 conn(0x7f4e480b3190 0x7f4e480b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:09.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 -- 172.21.15.3:0/943202308 >> 172.21.15.3:0/943202308 conn(0x7f4e48063870 msgr2=0x7f4e48063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:09.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 -- 172.21.15.3:0/943202308 shutdown_connections 2022-07-21T01:40:09.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 -- 172.21.15.3:0/943202308 wait complete. 2022-07-21T01:40:09.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 Processor -- start 2022-07-21T01:40:09.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 -- start start 2022-07-21T01:40:09.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 0x7f4e48148fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:09.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e480b1a80 0x7f4e48142a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:09.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e480b3190 0x7f4e48142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:09.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e480b48a0 con 0x7f4e480b1a80 2022-07-21T01:40:09.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4814c7b0 con 0x7f4e480b3190 2022-07-21T01:40:09.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e47fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e480b5e70 con 0x7f4e480a9da0 2022-07-21T01:40:09.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e467fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e480b1a80 0x7f4e48142a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:09.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e467fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e480b1a80 0x7f4e48142a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47362/0 (socket says 172.21.15.3:47362) 2022-07-21T01:40:09.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e46ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 0x7f4e48148fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:09.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e467fc700 1 -- 172.21.15.3:0/1344903701 learned_addr learned my addr 172.21.15.3:0/1344903701 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:09.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e477fe700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e480b3190 0x7f4e48142f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:09.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e467fc700 1 -- 172.21.15.3:0/1344903701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 msgr2=0x7f4e48148fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:09.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e467fc700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 0x7f4e48148fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:09.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e467fc700 1 -- 172.21.15.3:0/1344903701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e480b3190 msgr2=0x7f4e48142f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:09.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e467fc700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e480b3190 0x7f4e48142f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:09.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.006+0000 7f4e467fc700 1 -- 172.21.15.3:0/1344903701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3800b180 con 0x7f4e480b1a80 2022-07-21T01:40:09.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e467fc700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e480b1a80 0x7f4e48142a70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f4e480b46b0 tx=0x7f4e3c007ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:09.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e37fff700 1 -- 172.21.15.3:0/1344903701 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e3c0070d0 con 0x7f4e480b1a80 2022-07-21T01:40:09.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e37fff700 1 -- 172.21.15.3:0/1344903701 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c007230 con 0x7f4e480b1a80 2022-07-21T01:40:09.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e40009470 con 0x7f4e480b1a80 2022-07-21T01:40:09.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e37fff700 1 -- 172.21.15.3:0/1344903701 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e3c005610 con 0x7f4e480b1a80 2022-07-21T01:40:09.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e481425d0 con 0x7f4e480b1a80 2022-07-21T01:40:09.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e48142790 con 0x7f4e480b1a80 2022-07-21T01:40:09.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e37fff700 1 -- 172.21.15.3:0/1344903701 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4e3c01a060 con 0x7f4e480b1a80 2022-07-21T01:40:09.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e37fff700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e24053b30 0x7f4e24055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:09.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e37fff700 1 -- 172.21.15.3:0/1344903701 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f4e3c06b130 con 0x7f4e480b1a80 2022-07-21T01:40:09.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.010+0000 7f4e46ffd700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e24053b30 0x7f4e24055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:09.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.014+0000 7f4e46ffd700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e24053b30 0x7f4e24055ff0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f4e3800b000 tx=0x7f4e38006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:09.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.018+0000 7f4e37fff700 1 -- 172.21.15.3:0/1344903701 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4e3c042030 con 0x7f4e480b1a80 2022-07-21T01:40:09.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.142+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4e480b6210 con 0x7f4e480b1a80 2022-07-21T01:40:09.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.878+0000 7f4e37fff700 1 -- 172.21.15.3:0/1344903701 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v83) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4e3c018020 con 0x7f4e480b1a80 2022-07-21T01:40:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e24053b30 msgr2=0x7f4e24055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e24053b30 0x7f4e24055ff0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f4e3800b000 tx=0x7f4e38006df0).stop 2022-07-21T01:40:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e480b1a80 msgr2=0x7f4e48142a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e480b1a80 0x7f4e48142a70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f4e480b46b0 tx=0x7f4e3c007ce0).stop 2022-07-21T01:40:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 shutdown_connections 2022-07-21T01:40:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e24053b30 0x7f4e24055ff0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e480b3190 0x7f4e48142f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:09.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e480b1a80 0x7f4e48142a70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:09.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 --2- 172.21.15.3:0/1344903701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e480a9da0 0x7f4e48148fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:09.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 >> 172.21.15.3:0/1344903701 conn(0x7f4e48063870 msgr2=0x7f4e48060e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:09.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 shutdown_connections 2022-07-21T01:40:09.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:09.886+0000 7f4e47fff700 1 -- 172.21.15.3:0/1344903701 wait complete. 2022-07-21T01:40:09.892 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:40:09.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:40:10.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.010+0000 7f9e04830700 1 Processor -- start 2022-07-21T01:40:10.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.010+0000 7f9e04830700 1 -- start start 2022-07-21T01:40:10.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9e00060eb0 0x7f9e0005f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e000b3dd0 con 0x7f9e000b2340 2022-07-21T01:40:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e000b2720 con 0x7f9e00143e20 2022-07-21T01:40:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e00063f80 con 0x7f9e00060eb0 2022-07-21T01:40:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9e000b2340 0x7f9e000b36c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55648/0 (socket says 172.21.15.3:55648) 2022-07-21T01:40:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfed9d700 1 -- 172.21.15.3:0/1207196101 learned_addr learned my addr 172.21.15.3:0/1207196101 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:10.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfe59c700 1 --2- 172.21.15.3:0/1207196101 >> v2:172.21.15.143:3300/0 conn(0x7f9e00060eb0 0x7f9e0005f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfdd9b700 1 -- 172.21.15.3:0/1207196101 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920898718 0 0) 0x7f9e000b3dd0 con 0x7f9e000b2340 2022-07-21T01:40:10.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfdd9b700 1 -- 172.21.15.3:0/1207196101 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de0003260 con 0x7f9e000b2340 2022-07-21T01:40:10.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfdd9b700 1 -- 172.21.15.3:0/1207196101 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614693246 0 0) 0x7f9e000b2720 con 0x7f9e00143e20 2022-07-21T01:40:10.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfdd9b700 1 -- 172.21.15.3:0/1207196101 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e000b3dd0 con 0x7f9e00143e20 2022-07-21T01:40:10.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfdd9b700 1 -- 172.21.15.3:0/1207196101 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 839752944 0 0) 0x7f9de0003260 con 0x7f9e000b2340 2022-07-21T01:40:10.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfe59c700 1 -- 172.21.15.3:0/1207196101 >> v1:172.21.15.143:6789/0 conn(0x7f9e00143e20 legacy=0x7f9e0012c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfe59c700 1 -- 172.21.15.3:0/1207196101 >> v1:172.21.15.3:6789/0 conn(0x7f9e000b2340 legacy=0x7f9e000b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfe59c700 1 -- 172.21.15.3:0/1207196101 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e000b2fc0 con 0x7f9e00060eb0 2022-07-21T01:40:10.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfe59c700 1 --2- 172.21.15.3:0/1207196101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e00060eb0 0x7f9e0005f840 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f9df0001110 tx=0x7f9de8006df0).ready entity=mon.1 client_cookie=e9e14048df05b224 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:10.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfdd9b700 1 -- 172.21.15.3:0/1207196101 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df0001660 con 0x7f9e00060eb0 2022-07-21T01:40:10.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfdd9b700 1 -- 172.21.15.3:0/1207196101 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de8000e90 con 0x7f9e00060eb0 2022-07-21T01:40:10.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9dfdd9b700 1 -- 172.21.15.3:0/1207196101 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9de8012460 con 0x7f9e00060eb0 2022-07-21T01:40:10.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 -- 172.21.15.3:0/1207196101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e00060eb0 msgr2=0x7f9e0005f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 --2- 172.21.15.3:0/1207196101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e00060eb0 0x7f9e0005f840 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f9df0001110 tx=0x7f9de8006df0).stop 2022-07-21T01:40:10.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 -- 172.21.15.3:0/1207196101 shutdown_connections 2022-07-21T01:40:10.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 --2- 172.21.15.3:0/1207196101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e00060eb0 0x7f9e0005f840 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 -- 172.21.15.3:0/1207196101 >> 172.21.15.3:0/1207196101 conn(0x7f9e0005c620 msgr2=0x7f9e0005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:10.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 -- 172.21.15.3:0/1207196101 shutdown_connections 2022-07-21T01:40:10.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.014+0000 7f9e04830700 1 -- 172.21.15.3:0/1207196101 wait complete. 2022-07-21T01:40:10.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9e04830700 1 Processor -- start 2022-07-21T01:40:10.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9e04830700 1 -- start start 2022-07-21T01:40:10.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9e04830700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e00060eb0 0x7f9e00129fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9e04830700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e000b2340 0x7f9e001526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9e04830700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e00143e20 0x7f9e00152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9e04830700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e001447e0 con 0x7f9e000b2340 2022-07-21T01:40:10.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9e04830700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e00144430 con 0x7f9e00060eb0 2022-07-21T01:40:10.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9e04830700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e000b0110 con 0x7f9e00143e20 2022-07-21T01:40:10.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9dfe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e000b2340 0x7f9e001526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9dfe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e000b2340 0x7f9e001526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47374/0 (socket says 172.21.15.3:47374) 2022-07-21T01:40:10.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9dfe59c700 1 -- 172.21.15.3:0/628438871 learned_addr learned my addr 172.21.15.3:0/628438871 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:10.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9dff59e700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e00143e20 0x7f9e00152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.018+0000 7f9dfed9d700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e00060eb0 0x7f9e00129fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9dff59e700 1 -- 172.21.15.3:0/628438871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e00060eb0 msgr2=0x7f9e00129fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9dff59e700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e00060eb0 0x7f9e00129fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9dff59e700 1 -- 172.21.15.3:0/628438871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e000b2340 msgr2=0x7f9e001526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9dff59e700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e000b2340 0x7f9e001526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9dff59e700 1 -- 172.21.15.3:0/628438871 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de800b180 con 0x7f9e00143e20 2022-07-21T01:40:10.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9dff59e700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e00143e20 0x7f9e00152ba0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9df8001490 tx=0x7f9df8007d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:10.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9df7fff700 1 -- 172.21.15.3:0/628438871 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df800f470 con 0x7f9e00143e20 2022-07-21T01:40:10.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9df7fff700 1 -- 172.21.15.3:0/628438871 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df800f5d0 con 0x7f9e00143e20 2022-07-21T01:40:10.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e00156a10 con 0x7f9e00143e20 2022-07-21T01:40:10.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9df7fff700 1 -- 172.21.15.3:0/628438871 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df800f930 con 0x7f9e00143e20 2022-07-21T01:40:10.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e000b4f00 con 0x7f9e00143e20 2022-07-21T01:40:10.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.022+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e00061d60 con 0x7f9e00143e20 2022-07-21T01:40:10.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.026+0000 7f9df7fff700 1 -- 172.21.15.3:0/628438871 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9df8019030 con 0x7f9e00143e20 2022-07-21T01:40:10.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.026+0000 7f9df7fff700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9de40539e0 0x7f9de4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.026+0000 7f9df7fff700 1 -- 172.21.15.3:0/628438871 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f9df806b850 con 0x7f9e00143e20 2022-07-21T01:40:10.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.026+0000 7f9dfed9d700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9de40539e0 0x7f9de4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.026+0000 7f9dfed9d700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9de40539e0 0x7f9de4055ea0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f9df000d000 tx=0x7f9df0000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:10.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.030+0000 7f9df7fff700 1 -- 172.21.15.3:0/628438871 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9df80428c0 con 0x7f9e00143e20 2022-07-21T01:40:10.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.154+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9e000574e0 con 0x7f9e00143e20 2022-07-21T01:40:10.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9df7fff700 1 -- 172.21.15.3:0/628438871 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9df8045ee0 con 0x7f9e00143e20 2022-07-21T01:40:10.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9de40539e0 msgr2=0x7f9de4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9de40539e0 0x7f9de4055ea0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f9df000d000 tx=0x7f9df0000bc0).stop 2022-07-21T01:40:10.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e00143e20 msgr2=0x7f9e00152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e00143e20 0x7f9e00152ba0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9df8001490 tx=0x7f9df8007d40).stop 2022-07-21T01:40:10.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 shutdown_connections 2022-07-21T01:40:10.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e00143e20 0x7f9e00152ba0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e000b2340 0x7f9e001526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9de40539e0 0x7f9de4055ea0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 --2- 172.21.15.3:0/628438871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e00060eb0 0x7f9e00129fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.166+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 >> 172.21.15.3:0/628438871 conn(0x7f9e0005c620 msgr2=0x7f9e000b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:10.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.170+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 shutdown_connections 2022-07-21T01:40:10.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.170+0000 7f9e04830700 1 -- 172.21.15.3:0/628438871 wait complete. 2022-07-21T01:40:10.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-07-21T01:40:10.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.286+0000 7fd7d0807700 1 Processor -- start 2022-07-21T01:40:10.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.286+0000 7fd7d0807700 1 -- start start 2022-07-21T01:40:10.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0b6ee0 0x7fd7cc0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd7cc0b5ed0 0x7fd7cc0b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7cc061d90 con 0x7fd7cc0619b0 2022-07-21T01:40:10.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7cc143fa0 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc142f60 con 0x7fd7cc0b6ee0 2022-07-21T01:40:10.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7cbfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd7cc0b5ed0 0x7fd7cc0b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7cbfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd7cc0b5ed0 0x7fd7cc0b5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47382/0 (socket says 172.21.15.3:47382) 2022-07-21T01:40:10.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7cbfff700 1 -- 172.21.15.3:0/2062361707 learned_addr learned my addr 172.21.15.3:0/2062361707 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:10.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7caffd700 1 --2- 172.21.15.3:0/2062361707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0b6ee0 0x7fd7cc0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7cbfff700 1 -- 172.21.15.3:0/2062361707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0b6ee0 msgr2=0x7fd7cc0b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:10.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7cbfff700 1 --2- 172.21.15.3:0/2062361707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0b6ee0 0x7fd7cc0b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7cbfff700 1 -- 172.21.15.3:0/2062361707 >> v1:172.21.15.143:6789/0 conn(0x7fd7cc0619b0 legacy=0x7fd7cc143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7cbfff700 1 -- 172.21.15.3:0/2062361707 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc0ba5a0 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7ca7fc700 1 -- 172.21.15.3:0/2062361707 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322866201 0 0) 0x7fd7cc061d90 con 0x7fd7cc0619b0 2022-07-21T01:40:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7cbfff700 1 --2- 172.21.15.3:0/2062361707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 0x7fd7cc0b5190 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fd7c40032f0 tx=0x7fd7c4009290).ready entity=mon.0 client_cookie=31c31e41bae33b6c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7ca7fc700 1 -- 172.21.15.3:0/2062361707 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7c400b8a0 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7ca7fc700 1 -- 172.21.15.3:0/2062361707 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c400ba00 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7ca7fc700 1 -- 172.21.15.3:0/2062361707 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7c400bb70 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 -- 172.21.15.3:0/2062361707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 msgr2=0x7fd7cc0b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 --2- 172.21.15.3:0/2062361707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 0x7fd7cc0b5190 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fd7c40032f0 tx=0x7fd7c4009290).stop 2022-07-21T01:40:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 -- 172.21.15.3:0/2062361707 shutdown_connections 2022-07-21T01:40:10.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 --2- 172.21.15.3:0/2062361707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 0x7fd7cc0b5190 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 --2- 172.21.15.3:0/2062361707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0b6ee0 0x7fd7cc0b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 -- 172.21.15.3:0/2062361707 >> 172.21.15.3:0/2062361707 conn(0x7fd7cc06af80 msgr2=0x7fd7cc05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:10.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 -- 172.21.15.3:0/2062361707 shutdown_connections 2022-07-21T01:40:10.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.290+0000 7fd7d0807700 1 -- 172.21.15.3:0/2062361707 wait complete. 2022-07-21T01:40:10.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 Processor -- start 2022-07-21T01:40:10.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 -- start start 2022-07-21T01:40:10.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0619b0 0x7fd7cc0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 0x7fd7cc146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7cc0b6ee0 0x7fd7cc152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc145630 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc1458d0 con 0x7fd7cc0b6ee0 2022-07-21T01:40:10.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc145b70 con 0x7fd7cc0619b0 2022-07-21T01:40:10.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 0x7fd7cc146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 0x7fd7cc146d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47388/0 (socket says 172.21.15.3:47388) 2022-07-21T01:40:10.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7caffd700 1 -- 172.21.15.3:0/3702893714 learned_addr learned my addr 172.21.15.3:0/3702893714 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:10.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7cb7fe700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0619b0 0x7fd7cc0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7cbfff700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7cc0b6ee0 0x7fd7cc152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7caffd700 1 -- 172.21.15.3:0/3702893714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0619b0 msgr2=0x7fd7cc0afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7caffd700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0619b0 0x7fd7cc0afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7caffd700 1 -- 172.21.15.3:0/3702893714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7cc0b6ee0 msgr2=0x7fd7cc152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7caffd700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7cc0b6ee0 0x7fd7cc152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7caffd700 1 -- 172.21.15.3:0/3702893714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c4009530 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7caffd700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 0x7fd7cc146d60 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fd7b400c000 tx=0x7fd7b4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:10.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7c8ff9700 1 -- 172.21.15.3:0/3702893714 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7b4011750 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7c8ff9700 1 -- 172.21.15.3:0/3702893714 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b40118b0 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7c8ff9700 1 -- 172.21.15.3:0/3702893714 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7b4011c10 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc146040 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.294+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc152b40 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.298+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc152450 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.298+0000 7fd7c8ff9700 1 -- 172.21.15.3:0/3702893714 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd7b4011d70 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.298+0000 7fd7c8ff9700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7b0053b00 0x7fd7b0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:10.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.298+0000 7fd7c8ff9700 1 -- 172.21.15.3:0/3702893714 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fd7b406c1e0 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.298+0000 7fd7cb7fe700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7b0053b00 0x7fd7b0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:10.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.298+0000 7fd7cb7fe700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7b0053b00 0x7fd7b0055fc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fd7bc00d000 tx=0x7fd7bc007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:10.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.306+0000 7fd7c8ff9700 1 -- 172.21.15.3:0/3702893714 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd7b4043140 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.474+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7fd7cc0574e0 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.882+0000 7fd7c8ff9700 1 -- 172.21.15.3:0/3702893714 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v291) v1 ==== 139+0+0 (secure 0 0 0) 0x7fd7b4046760 con 0x7fd7cc0b5ed0 2022-07-21T01:40:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7b0053b00 msgr2=0x7fd7b0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7b0053b00 0x7fd7b0055fc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fd7bc00d000 tx=0x7fd7bc007a20).stop 2022-07-21T01:40:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 msgr2=0x7fd7cc146d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 0x7fd7cc146d60 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fd7b400c000 tx=0x7fd7b4006df0).stop 2022-07-21T01:40:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 shutdown_connections 2022-07-21T01:40:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7b0053b00 0x7fd7b0055fc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7cc0b6ee0 0x7fd7cc152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7cc0b5ed0 0x7fd7cc146d60 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 --2- 172.21.15.3:0/3702893714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7cc0619b0 0x7fd7cc0afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:10.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 >> 172.21.15.3:0/3702893714 conn(0x7fd7cc06af80 msgr2=0x7fd7cc13f580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 shutdown_connections 2022-07-21T01:40:10.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:10.886+0000 7fd7d0807700 1 -- 172.21.15.3:0/3702893714 wait complete. 2022-07-21T01:40:10.904 INFO:tasks.cephfs_test_runner:test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-21T01:40:10.905 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode (tasks.cephfs.test_misc.TestMisc) 2022-07-21T01:40:10.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-07-21T01:40:11.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.010+0000 7fb3ca3cf700 1 Processor -- start 2022-07-21T01:40:11.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.010+0000 7fb3ca3cf700 1 -- start start 2022-07-21T01:40:11.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.010+0000 7fb3ca3cf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb3c40b5ed0 0x7fb3c40b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:11.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.010+0000 7fb3ca3cf700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c4061d90 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.010+0000 7fb3ca3cf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c40ba5a0 con 0x7fb3c40619b0 2022-07-21T01:40:11.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.010+0000 7fb3ca3cf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c4143fa0 con 0x7fb3c40b5ed0 2022-07-21T01:40:11.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.010+0000 7fb3c8bcc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb3c40b6ee0 0x7fb3c40b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55676/0 (socket says 172.21.15.3:55676) 2022-07-21T01:40:11.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.010+0000 7fb3c8bcc700 1 -- 172.21.15.3:0/282643610 learned_addr learned my addr 172.21.15.3:0/282643610 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:11.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.010+0000 7fb3c9bce700 1 --2- 172.21.15.3:0/282643610 >> v2:172.21.15.143:3300/0 conn(0x7fb3c40b5ed0 0x7fb3c40b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:11.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3bbfff700 1 -- 172.21.15.3:0/282643610 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641675487 0 0) 0x7fb3c4061d90 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3bbfff700 1 -- 172.21.15.3:0/282643610 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a8003260 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3c9bce700 1 -- 172.21.15.3:0/282643610 >> v1:172.21.15.143:6789/0 conn(0x7fb3c40619b0 legacy=0x7fb3c4143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:11.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3c9bce700 1 -- 172.21.15.3:0/282643610 >> v1:172.21.15.3:6789/0 conn(0x7fb3c40b6ee0 legacy=0x7fb3c40b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3c9bce700 1 -- 172.21.15.3:0/282643610 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c4143150 con 0x7fb3c40b5ed0 2022-07-21T01:40:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3c9bce700 1 --2- 172.21.15.3:0/282643610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c40b5ed0 0x7fb3c40b58a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fb3bc0032f0 tx=0x7fb3bc0091d0).ready entity=mon.1 client_cookie=ef93831ebcc04a32 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3bbfff700 1 -- 172.21.15.3:0/282643610 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3bc00b6c0 con 0x7fb3c40b5ed0 2022-07-21T01:40:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3bbfff700 1 -- 172.21.15.3:0/282643610 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc00b820 con 0x7fb3c40b5ed0 2022-07-21T01:40:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3bbfff700 1 -- 172.21.15.3:0/282643610 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3bc00b990 con 0x7fb3c40b5ed0 2022-07-21T01:40:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/282643610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c40b5ed0 msgr2=0x7fb3c40b58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3ca3cf700 1 --2- 172.21.15.3:0/282643610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c40b5ed0 0x7fb3c40b58a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fb3bc0032f0 tx=0x7fb3bc0091d0).stop 2022-07-21T01:40:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/282643610 shutdown_connections 2022-07-21T01:40:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3ca3cf700 1 --2- 172.21.15.3:0/282643610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c40b5ed0 0x7fb3c40b58a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/282643610 >> 172.21.15.3:0/282643610 conn(0x7fb3c406af80 msgr2=0x7fb3c405fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/282643610 shutdown_connections 2022-07-21T01:40:11.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/282643610 wait complete. 2022-07-21T01:40:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3ca3cf700 1 Processor -- start 2022-07-21T01:40:11.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.014+0000 7fb3ca3cf700 1 -- start start 2022-07-21T01:40:11.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ca3cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40619b0 0x7fb3c40b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:11.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ca3cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c40b5ed0 0x7fb3c4152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:11.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ca3cf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40b6ee0 0x7fb3c40b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:11.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ca3cf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4152930 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ca3cf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c41565a0 con 0x7fb3c40b5ed0 2022-07-21T01:40:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ca3cf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4156840 con 0x7fb3c40619b0 2022-07-21T01:40:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c9bce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40b6ee0 0x7fb3c40b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:11.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c9bce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40b6ee0 0x7fb3c40b3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47404/0 (socket says 172.21.15.3:47404) 2022-07-21T01:40:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c9bce700 1 -- 172.21.15.3:0/3734764683 learned_addr learned my addr 172.21.15.3:0/3734764683 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c93cd700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40619b0 0x7fb3c40b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c8bcc700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c40b5ed0 0x7fb3c4152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:11.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c9bce700 1 -- 172.21.15.3:0/3734764683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40619b0 msgr2=0x7fb3c40b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:11.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c9bce700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40619b0 0x7fb3c40b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:11.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c9bce700 1 -- 172.21.15.3:0/3734764683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c40b5ed0 msgr2=0x7fb3c4152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:11.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c9bce700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c40b5ed0 0x7fb3c4152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:11.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c9bce700 1 -- 172.21.15.3:0/3734764683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc009470 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3c9bce700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40b6ee0 0x7fb3c40b3170 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fb3bc0032f0 tx=0x7fb3bc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:11.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ba7fc700 1 -- 172.21.15.3:0/3734764683 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3bc00b6c0 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ba7fc700 1 -- 172.21.15.3:0/3734764683 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc021630 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c4146f40 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ba7fc700 1 -- 172.21.15.3:0/3734764683 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3bc021990 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c40b3660 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.018+0000 7fb3ba7fc700 1 -- 172.21.15.3:0/3734764683 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb3bc027020 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.022+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c4112d70 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.022+0000 7fb3ba7fc700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb3ac0539e0 0x7fb3ac055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.022+0000 7fb3c93cd700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb3ac0539e0 0x7fb3ac055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.022+0000 7fb3ba7fc700 1 -- 172.21.15.3:0/3734764683 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5729+0+0 (secure 0 0 0) 0x7fb3bc06d1b0 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.022+0000 7fb3c93cd700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb3ac0539e0 0x7fb3ac055ea0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fb3b400d000 tx=0x7fb3b4000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:11.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.026+0000 7fb3ba7fc700 1 -- 172.21.15.3:0/3734764683 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3c4112d70 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.186+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7fb3c4112d70 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.886+0000 7fb3ba7fc700 1 -- 172.21.15.3:0/3734764683 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v292) v1 ==== 129+0+0 (secure 0 0 0) 0x7fb3c4112d70 con 0x7fb3c40b6ee0 2022-07-21T01:40:11.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb3ac0539e0 msgr2=0x7fb3ac055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:11.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb3ac0539e0 0x7fb3ac055ea0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fb3b400d000 tx=0x7fb3b4000bc0).stop 2022-07-21T01:40:11.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40b6ee0 msgr2=0x7fb3c40b3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:11.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40b6ee0 0x7fb3c40b3170 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fb3bc0032f0 tx=0x7fb3bc016000).stop 2022-07-21T01:40:11.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 shutdown_connections 2022-07-21T01:40:11.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40b6ee0 0x7fb3c40b3170 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:11.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb3ac0539e0 0x7fb3ac055ea0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:11.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c40b5ed0 0x7fb3c4152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:11.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 --2- 172.21.15.3:0/3734764683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40619b0 0x7fb3c40b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:11.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 >> 172.21.15.3:0/3734764683 conn(0x7fb3c406af80 msgr2=0x7fb3c4142170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:11.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 shutdown_connections 2022-07-21T01:40:11.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:11.890+0000 7fb3ca3cf700 1 -- 172.21.15.3:0/3734764683 wait complete. 2022-07-21T01:40:11.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:40:12.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5725425700 1 Processor -- start 2022-07-21T01:40:12.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5725425700 1 -- start start 2022-07-21T01:40:12.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5725425700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b2730 0x7f57200666b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5725425700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5720066ba0 0x7f572014dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5725425700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200654c0 con 0x7f57200b3190 2022-07-21T01:40:12.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5725425700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5720062190 con 0x7f5720066ba0 2022-07-21T01:40:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5725425700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57200b3570 con 0x7f57200b2730 2022-07-21T01:40:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f571ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f57200b3190 0x7f5720060ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55688/0 (socket says 172.21.15.3:55688) 2022-07-21T01:40:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f571ffff700 1 -- 172.21.15.3:0/1370621333 learned_addr learned my addr 172.21.15.3:0/1370621333 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5724c24700 1 --2- 172.21.15.3:0/1370621333 >> v2:172.21.15.143:3300/0 conn(0x7f5720066ba0 0x7f572014dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f571f7fe700 1 --2- 172.21.15.3:0/1370621333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b2730 0x7f57200666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f571effd700 1 -- 172.21.15.3:0/1370621333 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108729405 0 0) 0x7f57200654c0 con 0x7f57200b3190 2022-07-21T01:40:12.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f571effd700 1 -- 172.21.15.3:0/1370621333 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570c003260 con 0x7f57200b3190 2022-07-21T01:40:12.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5724c24700 1 -- 172.21.15.3:0/1370621333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b2730 msgr2=0x7f57200666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:12.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5724c24700 1 --2- 172.21.15.3:0/1370621333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b2730 0x7f57200666b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5724c24700 1 -- 172.21.15.3:0/1370621333 >> v1:172.21.15.3:6789/0 conn(0x7f57200b3190 legacy=0x7f5720060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5724c24700 1 -- 172.21.15.3:0/1370621333 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572005fbf0 con 0x7f5720066ba0 2022-07-21T01:40:12.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f5724c24700 1 --2- 172.21.15.3:0/1370621333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5720066ba0 0x7f572014dab0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f57180032f0 tx=0x7f57180091d0).ready entity=mon.1 client_cookie=39a36a9448c78ad9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:12.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.022+0000 7f571effd700 1 -- 172.21.15.3:0/1370621333 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571800b600 con 0x7f5720066ba0 2022-07-21T01:40:12.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f571effd700 1 -- 172.21.15.3:0/1370621333 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571800b760 con 0x7f5720066ba0 2022-07-21T01:40:12.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f571effd700 1 -- 172.21.15.3:0/1370621333 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571800b8d0 con 0x7f5720066ba0 2022-07-21T01:40:12.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f5725425700 1 -- 172.21.15.3:0/1370621333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5720066ba0 msgr2=0x7f572014dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f5725425700 1 --2- 172.21.15.3:0/1370621333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5720066ba0 0x7f572014dab0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f57180032f0 tx=0x7f57180091d0).stop 2022-07-21T01:40:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f5725425700 1 -- 172.21.15.3:0/1370621333 shutdown_connections 2022-07-21T01:40:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f5725425700 1 --2- 172.21.15.3:0/1370621333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5720066ba0 0x7f572014dab0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f5725425700 1 --2- 172.21.15.3:0/1370621333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b2730 0x7f57200666b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f5725425700 1 -- 172.21.15.3:0/1370621333 >> 172.21.15.3:0/1370621333 conn(0x7f5720063870 msgr2=0x7f5720063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f5725425700 1 -- 172.21.15.3:0/1370621333 shutdown_connections 2022-07-21T01:40:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f5725425700 1 -- 172.21.15.3:0/1370621333 wait complete. 2022-07-21T01:40:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.026+0000 7f5725425700 1 Processor -- start 2022-07-21T01:40:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5725425700 1 -- start start 2022-07-21T01:40:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5725425700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5720066ba0 0x7f57201521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5725425700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57200b2730 0x7f5720140140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5725425700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b3190 0x7f572013ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5725425700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572013f4c0 con 0x7f5720066ba0 2022-07-21T01:40:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5725425700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572013f760 con 0x7f57200b2730 2022-07-21T01:40:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5725425700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572013fa00 con 0x7f57200b3190 2022-07-21T01:40:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5720066ba0 0x7f57201521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5720066ba0 0x7f57201521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47414/0 (socket says 172.21.15.3:47414) 2022-07-21T01:40:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571ffff700 1 -- 172.21.15.3:0/323892760 learned_addr learned my addr 172.21.15.3:0/323892760 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5724c24700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b3190 0x7f572013ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571f7fe700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57200b2730 0x7f5720140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571ffff700 1 -- 172.21.15.3:0/323892760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b3190 msgr2=0x7f572013ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571ffff700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b3190 0x7f572013ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571ffff700 1 -- 172.21.15.3:0/323892760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57200b2730 msgr2=0x7f5720140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571ffff700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57200b2730 0x7f5720140140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571ffff700 1 -- 172.21.15.3:0/323892760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718009470 con 0x7f5720066ba0 2022-07-21T01:40:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571ffff700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5720066ba0 0x7f57201521e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f5710001110 tx=0x7f5710007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571d7fa700 1 -- 172.21.15.3:0/323892760 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571000ce80 con 0x7f5720066ba0 2022-07-21T01:40:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571d7fa700 1 -- 172.21.15.3:0/323892760 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57100070d0 con 0x7f5720066ba0 2022-07-21T01:40:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572013fca0 con 0x7f5720066ba0 2022-07-21T01:40:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f571d7fa700 1 -- 172.21.15.3:0/323892760 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5710007430 con 0x7f5720066ba0 2022-07-21T01:40:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.030+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57200b5270 con 0x7f5720066ba0 2022-07-21T01:40:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.034+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572005fe90 con 0x7f5720066ba0 2022-07-21T01:40:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.034+0000 7f571d7fa700 1 -- 172.21.15.3:0/323892760 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5710010070 con 0x7f5720066ba0 2022-07-21T01:40:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.034+0000 7f571d7fa700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57000539e0 0x7f5700055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.034+0000 7f571d7fa700 1 -- 172.21.15.3:0/323892760 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f571006b4f0 con 0x7f5720066ba0 2022-07-21T01:40:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.034+0000 7f571f7fe700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57000539e0 0x7f5700055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.034+0000 7f571f7fe700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57000539e0 0x7f5700055ea0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f571400b000 tx=0x7f5714006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.038+0000 7f571d7fa700 1 -- 172.21.15.3:0/323892760 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5710042450 con 0x7f5720066ba0 2022-07-21T01:40:12.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.170+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f57200574e0 con 0x7f5720066ba0 2022-07-21T01:40:12.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f571d7fa700 1 -- 172.21.15.3:0/323892760 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5710045a70 con 0x7f5720066ba0 2022-07-21T01:40:12.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57000539e0 msgr2=0x7f5700055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57000539e0 0x7f5700055ea0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f571400b000 tx=0x7f5714006df0).stop 2022-07-21T01:40:12.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5720066ba0 msgr2=0x7f57201521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5720066ba0 0x7f57201521e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f5710001110 tx=0x7f5710007aa0).stop 2022-07-21T01:40:12.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 shutdown_connections 2022-07-21T01:40:12.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57200b3190 0x7f572013ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57000539e0 0x7f5700055ea0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57200b2730 0x7f5720140140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 --2- 172.21.15.3:0/323892760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5720066ba0 0x7f57201521e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 >> 172.21.15.3:0/323892760 conn(0x7f5720063870 msgr2=0x7f5720147720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:12.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 shutdown_connections 2022-07-21T01:40:12.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.178+0000 7f5725425700 1 -- 172.21.15.3:0/323892760 wait complete. 2022-07-21T01:40:12.194 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:40:12.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:12.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:12.234 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:12.255 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:12.276 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:40:12.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:12.277 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:12.278 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:12.304 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:40:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:40:12.306 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:12.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:12.307 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:12.308 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:12.315 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:40:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:40:12.316 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:40:12.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:40:12.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d6339700 1 Processor -- start 2022-07-21T01:40:12.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d6339700 1 -- start start 2022-07-21T01:40:12.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d6339700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d00b6420 0x7f53d0061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d6339700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f53d00b41f0 0x7f53d0061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d6339700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d00b60e0 con 0x7f53d0061fe0 2022-07-21T01:40:12.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d6339700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d00b2100 con 0x7f53d00b41f0 2022-07-21T01:40:12.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d6339700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d00b23a0 con 0x7f53d00b6420 2022-07-21T01:40:12.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d4b36700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f53d00b41f0 0x7f53d0061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d4b36700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f53d00b41f0 0x7f53d0061af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47422/0 (socket says 172.21.15.3:47422) 2022-07-21T01:40:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d4b36700 1 -- 172.21.15.3:0/2097511260 learned_addr learned my addr 172.21.15.3:0/2097511260 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d5337700 1 --2- 172.21.15.3:0/2097511260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d00b6420 0x7f53d0061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d4b36700 1 -- 172.21.15.3:0/2097511260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d00b6420 msgr2=0x7f53d0061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d4b36700 1 --2- 172.21.15.3:0/2097511260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d00b6420 0x7f53d0061600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d4b36700 1 -- 172.21.15.3:0/2097511260 >> v1:172.21.15.143:6789/0 conn(0x7f53d0061fe0 legacy=0x7f53d01465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:12.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d4b36700 1 -- 172.21.15.3:0/2097511260 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d00b2640 con 0x7f53d00b41f0 2022-07-21T01:40:12.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.426+0000 7f53d5337700 1 --2- 172.21.15.3:0/2097511260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d00b6420 0x7f53d0061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:40:12.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d4b36700 1 --2- 172.21.15.3:0/2097511260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 0x7f53d0061af0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f53d006ac70 tx=0x7f53cc006eb0).ready entity=mon.0 client_cookie=56c0e8f1caf88b8a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:12.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53c7fff700 1 -- 172.21.15.3:0/2097511260 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53cc00d570 con 0x7f53d00b41f0 2022-07-21T01:40:12.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53c7fff700 1 -- 172.21.15.3:0/2097511260 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53cc000e90 con 0x7f53d00b41f0 2022-07-21T01:40:12.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53c7fff700 1 -- 172.21.15.3:0/2097511260 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53cc00d8c0 con 0x7f53d00b41f0 2022-07-21T01:40:12.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 -- 172.21.15.3:0/2097511260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 msgr2=0x7f53d0061af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 --2- 172.21.15.3:0/2097511260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 0x7f53d0061af0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f53d006ac70 tx=0x7f53cc006eb0).stop 2022-07-21T01:40:12.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 -- 172.21.15.3:0/2097511260 shutdown_connections 2022-07-21T01:40:12.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 --2- 172.21.15.3:0/2097511260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 0x7f53d0061af0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 --2- 172.21.15.3:0/2097511260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d00b6420 0x7f53d0061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 -- 172.21.15.3:0/2097511260 >> 172.21.15.3:0/2097511260 conn(0x7f53d005c620 msgr2=0x7f53d005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:12.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 -- 172.21.15.3:0/2097511260 shutdown_connections 2022-07-21T01:40:12.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 -- 172.21.15.3:0/2097511260 wait complete. 2022-07-21T01:40:12.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 Processor -- start 2022-07-21T01:40:12.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.430+0000 7f53d6339700 1 -- start start 2022-07-21T01:40:12.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d6339700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d0061fe0 0x7f53d0145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d6339700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 0x7f53d0143980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d6339700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d00b6420 0x7f53d0152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d6339700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d0140e90 con 0x7f53d00b41f0 2022-07-21T01:40:12.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d6339700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d01410c0 con 0x7f53d00b6420 2022-07-21T01:40:12.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d6339700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d0141360 con 0x7f53d0061fe0 2022-07-21T01:40:12.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d4b36700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 0x7f53d0143980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d4b36700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 0x7f53d0143980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47430/0 (socket says 172.21.15.3:47430) 2022-07-21T01:40:12.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d4b36700 1 -- 172.21.15.3:0/184533867 learned_addr learned my addr 172.21.15.3:0/184533867 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:12.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d5337700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d0061fe0 0x7f53d0145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d5b38700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d00b6420 0x7f53d0152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d4b36700 1 -- 172.21.15.3:0/184533867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d0061fe0 msgr2=0x7f53d0145df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d4b36700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d0061fe0 0x7f53d0145df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d4b36700 1 -- 172.21.15.3:0/184533867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d00b6420 msgr2=0x7f53d0152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d4b36700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d00b6420 0x7f53d0152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d4b36700 1 -- 172.21.15.3:0/184533867 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc00b1e0 con 0x7f53d00b41f0 2022-07-21T01:40:12.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d4b36700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 0x7f53d0143980 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f53d006ac70 tx=0x7f53cc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:12.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53c67fc700 1 -- 172.21.15.3:0/184533867 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53cc00bec0 con 0x7f53d00b41f0 2022-07-21T01:40:12.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53c67fc700 1 -- 172.21.15.3:0/184533867 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53cc002f50 con 0x7f53d00b41f0 2022-07-21T01:40:12.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d0141600 con 0x7f53d00b41f0 2022-07-21T01:40:12.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53c67fc700 1 -- 172.21.15.3:0/184533867 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53cc01d7c0 con 0x7f53d00b41f0 2022-07-21T01:40:12.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d0143e70 con 0x7f53d00b41f0 2022-07-21T01:40:12.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.434+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d00b28e0 con 0x7f53d00b41f0 2022-07-21T01:40:12.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.438+0000 7f53c67fc700 1 -- 172.21.15.3:0/184533867 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f53cc025030 con 0x7f53d00b41f0 2022-07-21T01:40:12.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.438+0000 7f53c67fc700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f53b0053950 0x7f53b0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.438+0000 7f53c67fc700 1 -- 172.21.15.3:0/184533867 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f53cc06d280 con 0x7f53d00b41f0 2022-07-21T01:40:12.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.438+0000 7f53d5337700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f53b0053950 0x7f53b0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.438+0000 7f53d5337700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f53b0053950 0x7f53b0055e10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f53c000b000 tx=0x7f53c0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:12.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.442+0000 7f53c67fc700 1 -- 172.21.15.3:0/184533867 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f53cc0442f0 con 0x7f53d00b41f0 2022-07-21T01:40:12.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.570+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f53d00b60e0 con 0x7f53d00b41f0 2022-07-21T01:40:12.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.570+0000 7f53c67fc700 1 -- 172.21.15.3:0/184533867 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v83) v1 ==== 81+0+29292 (secure 0 0 0) 0x7f53d00b60e0 con 0x7f53d00b41f0 2022-07-21T01:40:12.575 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:12.575 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:40:12.575 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 83, 2022-07-21T01:40:12.575 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:40:12.576 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:40:12.576 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:40:09.870205+0000", 2022-07-21T01:40:12.576 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:12.576 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:12.577 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:40:12.577 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:40:12.577 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:40:12.578 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:40:12.578 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:40:12.578 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:40:12.578 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:40:12.579 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:12.579 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:40:12.579 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:40:12.579 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:40:12.580 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:40:12.580 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:40:12.580 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 14, 2022-07-21T01:40:12.580 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:40:12.581 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:40:12.581 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:40:12.581 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:40:12.581 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:40:12.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.582 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:40:12.582 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:40:12.583 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:40:12.583 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:12.583 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:12.583 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:12.584 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:12.584 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:12.584 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:12.584 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:12.585 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:12.585 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:40:12.585 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:40:12.585 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:40:12.586 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:40:12.586 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:40:12.586 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:12.587 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:12.587 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:12.587 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:12.587 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:12.588 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:12.588 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:12.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.588 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:40:12.589 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:12.589 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:12.589 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:12.589 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:12.590 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:12.590 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:12.590 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:12.590 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:12.591 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:12.591 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:12.591 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:12.592 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:12.592 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:12.592 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:12.592 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:12.593 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:12.593 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:12.593 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:12.593 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:12.594 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:12.594 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:12.594 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:12.594 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:12.595 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:12.595 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:12.595 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:12.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.596 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:12.596 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:12.596 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:12.596 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:12.597 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:12.597 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:12.597 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:12.597 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:12.598 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:12.598 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:12.598 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:12.598 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:40:12.599 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:40:12.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.599 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:12.599 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:40:12.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.600 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:40:12.601 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:40:12.601 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:40:12.601 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:12.601 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:12.602 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:12.602 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:12.602 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:12.602 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:12.603 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:12.603 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:12.603 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:12.604 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:12.604 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:12.604 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:12.604 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:12.605 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:12.605 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:12.605 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:12.605 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:12.606 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:12.606 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:12.606 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:12.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.607 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:40:12.607 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:12.607 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:12.607 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:12.608 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:12.608 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:12.608 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:12.608 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:12.609 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:12.609 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:12.609 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:12.609 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:12.610 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:12.610 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:12.610 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:12.610 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:12.611 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:12.611 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:12.611 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:12.611 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:12.612 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:12.612 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:12.612 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:12.612 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:12.613 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:12.613 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:12.613 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:12.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.614 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:12.614 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:12.614 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:12.615 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:12.615 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:12.615 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:12.615 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:12.616 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:12.616 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:12.616 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:12.616 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:12.617 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:40:12.617 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:12.617 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:40:12.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.618 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:12.618 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:40:12.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.619 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:12.619 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:40:12.619 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:12.620 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:12.620 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:12.620 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:12.620 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:12.621 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:12.621 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:12.621 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:12.622 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:12.622 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:12.622 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:12.622 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.623 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:40:12.624 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:12.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.624 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.625 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:40:12.625 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:12.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.626 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.626 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:12.626 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.627 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:40:12.628 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:12.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.628 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.629 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:40:12.629 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:12.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.630 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.630 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:12.631 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.631 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:40:12.632 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:12.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.632 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.633 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:40:12.633 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:12.633 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.633 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.634 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:12.634 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.635 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:40:12.635 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:12.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.636 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:40:12.636 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:12.637 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.637 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.637 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:40:12.638 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:40:12.638 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:40:12.638 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:40:12.638 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:12.639 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:12.639 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:12.639 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.640 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:12.640 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:40:12.640 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:12.641 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:12.641 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:12.641 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:12.641 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:12.642 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:12.642 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:12.642 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 65, 2022-07-21T01:40:12.642 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:12.643 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:12.643 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:12.643 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.644 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.644 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:40:12.644 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:12.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.645 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.645 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:40:12.645 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:12.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.647 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:12.647 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.647 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:40:12.648 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:12.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.649 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:40:12.649 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:12.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.649 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.650 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:12.650 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.651 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:40:12.651 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:12.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.652 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.652 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:40:12.653 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:12.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.653 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:12.654 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.654 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:40:12.655 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:12.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.656 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.656 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:40:12.656 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:12.656 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.657 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.657 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:40:12.657 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:40:12.658 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:40:12.658 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:40:12.658 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:12.658 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:12.659 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:12.659 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.660 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:12.660 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:40:12.660 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:12.660 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:12.661 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:12.661 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:12.661 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:12.661 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:12.662 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:12.662 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:12.662 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:12.662 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:12.663 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:12.663 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.664 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:40:12.664 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:12.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.665 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.665 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:40:12.665 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:12.665 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.666 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.666 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:12.666 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.667 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:40:12.667 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:12.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.668 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.668 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:40:12.669 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:12.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.669 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.670 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:12.670 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.670 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.671 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:40:12.671 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:12.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.672 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.672 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:40:12.672 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:12.672 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.673 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.673 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:12.673 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.674 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:40:12.675 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:12.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.675 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.675 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:40:12.676 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:12.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.676 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.677 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:40:12.677 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:40:12.677 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:40:12.678 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:40:12.678 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:12.678 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:12.678 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:12.679 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.679 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:12.680 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:40:12.680 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:12.680 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:12.680 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:12.681 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:12.681 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:12.681 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:12.681 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:12.681 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:12.682 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:12.682 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:12.682 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:12.683 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.683 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.683 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:40:12.683 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:12.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.684 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.685 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:40:12.685 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:12.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.685 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.686 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:12.686 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.687 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.687 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:40:12.687 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:12.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.688 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.688 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:40:12.688 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:12.689 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.689 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.689 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:12.690 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.690 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.690 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:40:12.691 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:12.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.691 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.692 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:40:12.692 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:12.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.692 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.693 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:12.693 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.694 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.694 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:40:12.694 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:12.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.695 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.695 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:40:12.695 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:12.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.696 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.696 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:40:12.697 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:40:12.697 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:40:12.697 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:40:12.697 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:12.697 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:12.698 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:12.698 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.699 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:12.699 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:40:12.699 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:12.699 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:12.700 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:12.700 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:12.700 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:12.700 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:12.701 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:12.701 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 76, 2022-07-21T01:40:12.701 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:12.701 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:12.702 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:12.702 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.702 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.703 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:40:12.703 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:12.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.704 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.704 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:40:12.704 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:12.704 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.705 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.705 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:12.705 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.706 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.706 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:40:12.706 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:12.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.707 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.707 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:40:12.708 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:12.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.708 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.709 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:12.709 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.709 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.710 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:40:12.710 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:12.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.710 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.711 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:40:12.711 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:12.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.711 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.712 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:12.712 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.712 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.713 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:40:12.713 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:12.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.714 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.714 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:40:12.714 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:12.714 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.715 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.715 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:40:12.715 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:40:12.716 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:40:12.716 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:40:12.716 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:12.716 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:12.717 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:12.717 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.718 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:12.718 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:40:12.718 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:12.718 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:12.719 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:12.719 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:12.719 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:12.719 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:12.720 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:12.720 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:12.720 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:12.720 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:12.721 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:12.721 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.721 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.722 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:40:12.722 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:12.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.723 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:40:12.723 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:12.723 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.724 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.724 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:12.724 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.725 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.725 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:40:12.725 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:12.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.726 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.726 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:40:12.726 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:12.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.727 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.727 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:12.728 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.728 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.728 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:40:12.729 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:12.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.729 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.730 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:40:12.730 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:12.730 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.731 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.731 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:12.731 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.732 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.732 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:40:12.732 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:12.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.733 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.733 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:40:12.733 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:12.734 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.734 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.735 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:40:12.735 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:40:12.735 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:40:12.735 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:40:12.736 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:12.736 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:12.736 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:12.736 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.737 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:12.737 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:40:12.738 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:12.738 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:12.738 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:12.738 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:12.738 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:12.739 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:12.739 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:12.739 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 77, 2022-07-21T01:40:12.739 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:12.740 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:12.740 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:12.740 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.741 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.741 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:40:12.741 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:12.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.742 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.742 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:40:12.742 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:12.743 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.743 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.743 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:12.744 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.744 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.744 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:40:12.745 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:12.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.745 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.746 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:40:12.746 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:12.746 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.747 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.747 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:12.747 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.748 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.748 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:40:12.748 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:12.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.749 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.749 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:40:12.749 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:12.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.750 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.751 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:12.751 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.751 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.752 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:40:12.752 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:12.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.753 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.753 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:40:12.753 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:12.753 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.754 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.754 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:40:12.754 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:40:12.755 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:40:12.755 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:40:12.755 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:12.755 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:12.756 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:12.756 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.757 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:12.757 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:40:12.757 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:12.757 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:12.758 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:12.758 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:12.758 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:12.758 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:12.759 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:12.759 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 76, 2022-07-21T01:40:12.759 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:12.759 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:12.760 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:12.760 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.761 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:40:12.761 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:12.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.762 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.762 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:40:12.762 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:12.762 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.763 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.763 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:12.764 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.764 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.764 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:40:12.765 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:12.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.765 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.766 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:40:12.766 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:12.766 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.766 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.767 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:12.767 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.768 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:40:12.768 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:12.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.769 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.769 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:40:12.769 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:12.770 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.770 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.770 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:12.771 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:12.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.771 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:12.771 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:40:12.772 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:12.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.772 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:12.773 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:40:12.773 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:12.773 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.773 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.774 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:40:12.774 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:40:12.775 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:40:12.775 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:40:12.775 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:12.775 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:12.776 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:12.776 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:12.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.776 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:12.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:40:12.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:12.777 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:12.778 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:12.778 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:12.778 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:12.778 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:12.779 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:40:12.779 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:12.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.780 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:12.780 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:12.780 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:12.780 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:12.781 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:12.781 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:12.781 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:40:12.781 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:12.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.782 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:12.782 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:12.783 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:12.783 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:12.783 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:12.783 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:40:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.784 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:12.785 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:40:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:12.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:12.787 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:12.788 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:40:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.789 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:12.790 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:40:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:12.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:12.792 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:40:12.793 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:12.794 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:12.795 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:40:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.796 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:40:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:40:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:40:12.797 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:40:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:40:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.143:6833/862725488": "2022-07-22T01:40:06.303929+0000", 2022-07-21T01:40:12.798 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.143:6832/862725488": "2022-07-22T01:40:06.303929+0000", 2022-07-21T01:40:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.171:0/752832426": "2022-07-21T02:39:54.585234+0000" 2022-07-21T01:40:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:40:12.799 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:40:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:40:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:40:12.800 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:40:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:40:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:12.801 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:40:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:40:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.802 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:40:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:40:12.803 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:40:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:12.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:40:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:40:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:40:12.805 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:40:12.806 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:40:12.806 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:40:12.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f53b0053950 msgr2=0x7f53b0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f53b0053950 0x7f53b0055e10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f53c000b000 tx=0x7f53c0008040).stop 2022-07-21T01:40:12.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 msgr2=0x7f53d0143980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 0x7f53d0143980 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f53d006ac70 tx=0x7f53cc015000).stop 2022-07-21T01:40:12.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 shutdown_connections 2022-07-21T01:40:12.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f53b0053950 0x7f53b0055e10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d00b6420 0x7f53d0152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53d00b41f0 0x7f53d0143980 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 --2- 172.21.15.3:0/184533867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53d0061fe0 0x7f53d0145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 >> 172.21.15.3:0/184533867 conn(0x7f53d005c620 msgr2=0x7f53d00b8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:12.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 shutdown_connections 2022-07-21T01:40:12.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.574+0000 7f53d6339700 1 -- 172.21.15.3:0/184533867 wait complete. 2022-07-21T01:40:12.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:12.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.918+0000 7f672a27a700 1 Processor -- start 2022-07-21T01:40:12.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.918+0000 7f672a27a700 1 -- start start 2022-07-21T01:40:12.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b6420 0x7f6724061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f67240b41f0 0x7f6724061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67240b60e0 con 0x7f6724061fe0 2022-07-21T01:40:12.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67240b2100 con 0x7f67240b41f0 2022-07-21T01:40:12.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67240b23a0 con 0x7f67240b6420 2022-07-21T01:40:12.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6728a77700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f67240b41f0 0x7f6724061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6728a77700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f67240b41f0 0x7f6724061af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47438/0 (socket says 172.21.15.3:47438) 2022-07-21T01:40:12.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6729278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b6420 0x7f6724061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6728a77700 1 -- 172.21.15.3:0/3546385371 learned_addr learned my addr 172.21.15.3:0/3546385371 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:12.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6729a79700 1 --1- 172.21.15.3:0/3546385371 >> v1:172.21.15.143:6789/0 conn(0x7f6724061fe0 0x7f67241465c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:57836/0 (socket says 172.21.15.3:57836) 2022-07-21T01:40:12.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6713fff700 1 -- 172.21.15.3:0/3546385371 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28357099 0 0) 0x7f671c001490 con 0x7f6724061fe0 2022-07-21T01:40:12.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6728a77700 1 -- 172.21.15.3:0/3546385371 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b6420 msgr2=0x7f6724061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:12.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6728a77700 1 --2- 172.21.15.3:0/3546385371 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b6420 0x7f6724061600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6728a77700 1 -- 172.21.15.3:0/3546385371 >> v1:172.21.15.143:6789/0 conn(0x7f6724061fe0 legacy=0x7f67241465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6728a77700 1 -- 172.21.15.3:0/3546385371 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67240b2640 con 0x7f67240b41f0 2022-07-21T01:40:12.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6728a77700 1 --2- 172.21.15.3:0/3546385371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b41f0 0x7f6724061af0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f6714002f70 tx=0x7f6714006eb0).ready entity=mon.0 client_cookie=3f8daabcdd52d61f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:12.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6713fff700 1 -- 172.21.15.3:0/3546385371 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f671400d570 con 0x7f67240b41f0 2022-07-21T01:40:12.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6713fff700 1 -- 172.21.15.3:0/3546385371 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6714000e90 con 0x7f67240b41f0 2022-07-21T01:40:12.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f6713fff700 1 -- 172.21.15.3:0/3546385371 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f671400d8c0 con 0x7f67240b41f0 2022-07-21T01:40:12.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 -- 172.21.15.3:0/3546385371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b41f0 msgr2=0x7f6724061af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 --2- 172.21.15.3:0/3546385371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b41f0 0x7f6724061af0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f6714002f70 tx=0x7f6714006eb0).stop 2022-07-21T01:40:12.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 -- 172.21.15.3:0/3546385371 shutdown_connections 2022-07-21T01:40:12.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 --2- 172.21.15.3:0/3546385371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b41f0 0x7f6724061af0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 --2- 172.21.15.3:0/3546385371 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b6420 0x7f6724061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 -- 172.21.15.3:0/3546385371 >> 172.21.15.3:0/3546385371 conn(0x7f672405c620 msgr2=0x7f672405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:12.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 -- 172.21.15.3:0/3546385371 shutdown_connections 2022-07-21T01:40:12.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.922+0000 7f672a27a700 1 -- 172.21.15.3:0/3546385371 wait complete. 2022-07-21T01:40:12.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f672a27a700 1 Processor -- start 2022-07-21T01:40:12.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f672a27a700 1 -- start start 2022-07-21T01:40:12.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f672a27a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6724061fe0 0x7f67241428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f672a27a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b41f0 0x7f67241432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f672a27a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b6420 0x7f67241437e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f672a27a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6724153670 con 0x7f67240b6420 2022-07-21T01:40:12.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f672a27a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67240b8220 con 0x7f6724061fe0 2022-07-21T01:40:12.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f672a27a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67240b84c0 con 0x7f67240b41f0 2022-07-21T01:40:12.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6724061fe0 0x7f67241428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729a79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b6420 0x7f67241437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6724061fe0 0x7f67241428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:52890/0 (socket says 172.21.15.3:52890) 2022-07-21T01:40:12.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729278700 1 -- 172.21.15.3:0/3469026788 learned_addr learned my addr 172.21.15.3:0/3469026788 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:12.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6728a77700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b41f0 0x7f67241432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729278700 1 -- 172.21.15.3:0/3469026788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b41f0 msgr2=0x7f67241432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729278700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b41f0 0x7f67241432f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729278700 1 -- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b6420 msgr2=0x7f67241437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:12.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729278700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b6420 0x7f67241437e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:12.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729278700 1 -- 172.21.15.3:0/3469026788 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671400b1e0 con 0x7f6724061fe0 2022-07-21T01:40:12.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729a79700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b6420 0x7f67241437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:12.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.926+0000 7f6729278700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6724061fe0 0x7f67241428f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f672000c000 tx=0x7f6720006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:12.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f67127fc700 1 -- 172.21.15.3:0/3469026788 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6720011750 con 0x7f6724061fe0 2022-07-21T01:40:12.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f67127fc700 1 -- 172.21.15.3:0/3469026788 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67200118b0 con 0x7f6724061fe0 2022-07-21T01:40:12.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67240afab0 con 0x7f6724061fe0 2022-07-21T01:40:12.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f67127fc700 1 -- 172.21.15.3:0/3469026788 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6720011c10 con 0x7f6724061fe0 2022-07-21T01:40:12.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6724143e70 con 0x7f6724061fe0 2022-07-21T01:40:12.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672413da40 con 0x7f6724061fe0 2022-07-21T01:40:12.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f67127fc700 1 -- 172.21.15.3:0/3469026788 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6720011d70 con 0x7f6724061fe0 2022-07-21T01:40:12.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f67127fc700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053aa0 0x7f6704055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:12.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f67127fc700 1 -- 172.21.15.3:0/3469026788 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f672006c210 con 0x7f6724061fe0 2022-07-21T01:40:12.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f6728a77700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053aa0 0x7f6704055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:12.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.930+0000 7f6728a77700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053aa0 0x7f6704055f60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f671401c000 tx=0x7f6714015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:12.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:12.942+0000 7f67127fc700 1 -- 172.21.15.3:0/3469026788 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6720043200 con 0x7f6724061fe0 2022-07-21T01:40:13.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.106+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6724143160 con 0x7f6724061fe0 2022-07-21T01:40:13.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.106+0000 7f67127fc700 1 -- 172.21.15.3:0/3469026788 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+1556 (secure 0 0 0) 0x7f6720046820 con 0x7f6724061fe0 2022-07-21T01:40:13.110 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:13.110 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":5858,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/397964915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":397964915},{"type":"v1","addr":"172.21.15.143:6833","nonce":397964915}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":37},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[]} 2022-07-21T01:40:13.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053aa0 msgr2=0x7f6704055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053aa0 0x7f6704055f60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f671401c000 tx=0x7f6714015000).stop 2022-07-21T01:40:13.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6724061fe0 msgr2=0x7f67241428f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6724061fe0 0x7f67241428f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f672000c000 tx=0x7f6720006df0).stop 2022-07-21T01:40:13.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 shutdown_connections 2022-07-21T01:40:13.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67240b6420 0x7f67241437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67240b41f0 0x7f67241432f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6704053aa0 0x7f6704055f60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 --2- 172.21.15.3:0/3469026788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6724061fe0 0x7f67241428f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 >> 172.21.15.3:0/3469026788 conn(0x7f672405c620 msgr2=0x7f672412b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:13.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 shutdown_connections 2022-07-21T01:40:13.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.110+0000 7f672a27a700 1 -- 172.21.15.3:0/3469026788 wait complete. 2022-07-21T01:40:13.115 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 37 2022-07-21T01:40:13.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:13.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fdd4d700 1 Processor -- start 2022-07-21T01:40:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fdd4d700 1 -- start start 2022-07-21T01:40:13.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fdd4d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f47f80b2710 0x7f47f814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fdd4d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f47f80a8c50 0x7f47f814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fdd4d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f80a7250 con 0x7f47f80b01a0 2022-07-21T01:40:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fdd4d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f80a7480 con 0x7f47f80b2710 2022-07-21T01:40:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fdd4d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f80a7720 con 0x7f47f80a8c50 2022-07-21T01:40:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fcd4b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f47f80b2710 0x7f47f814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fcd4b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f47f80b2710 0x7f47f814afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47452/0 (socket says 172.21.15.3:47452) 2022-07-21T01:40:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fcd4b700 1 -- 172.21.15.3:0/1426566007 learned_addr learned my addr 172.21.15.3:0/1426566007 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.222+0000 7f47fd54c700 1 --2- 172.21.15.3:0/1426566007 >> v2:172.21.15.143:3300/0 conn(0x7f47f80a8c50 0x7f47f814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fcd4b700 1 -- 172.21.15.3:0/1426566007 >> v2:172.21.15.143:3300/0 conn(0x7f47f80a8c50 msgr2=0x7f47f814b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fcd4b700 1 --2- 172.21.15.3:0/1426566007 >> v2:172.21.15.143:3300/0 conn(0x7f47f80a8c50 0x7f47f814b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fcd4b700 1 -- 172.21.15.3:0/1426566007 >> v1:172.21.15.143:6789/0 conn(0x7f47f80b01a0 legacy=0x7f47f80a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fcd4b700 1 -- 172.21.15.3:0/1426566007 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f80a79c0 con 0x7f47f80b2710 2022-07-21T01:40:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fcd4b700 1 --2- 172.21.15.3:0/1426566007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80b2710 0x7f47f814afd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f47ec002f70 tx=0x7f47ec006eb0).ready entity=mon.0 client_cookie=3f948ab563ebbd7b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47f77fe700 1 -- 172.21.15.3:0/1426566007 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47ec00d5c0 con 0x7f47f80b2710 2022-07-21T01:40:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47f77fe700 1 -- 172.21.15.3:0/1426566007 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec000e90 con 0x7f47f80b2710 2022-07-21T01:40:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47f77fe700 1 -- 172.21.15.3:0/1426566007 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47ec00d950 con 0x7f47f80b2710 2022-07-21T01:40:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1426566007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80b2710 msgr2=0x7f47f814afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 --2- 172.21.15.3:0/1426566007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80b2710 0x7f47f814afd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f47ec002f70 tx=0x7f47ec006eb0).stop 2022-07-21T01:40:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1426566007 shutdown_connections 2022-07-21T01:40:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 --2- 172.21.15.3:0/1426566007 >> v2:172.21.15.143:3300/0 conn(0x7f47f80a8c50 0x7f47f814b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 --2- 172.21.15.3:0/1426566007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80b2710 0x7f47f814afd0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1426566007 >> 172.21.15.3:0/1426566007 conn(0x7f47f8060310 msgr2=0x7f47f80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1426566007 shutdown_connections 2022-07-21T01:40:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1426566007 wait complete. 2022-07-21T01:40:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 Processor -- start 2022-07-21T01:40:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 -- start start 2022-07-21T01:40:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80a8c50 0x7f47f815a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f80b01a0 0x7f47f81565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47f80b2710 0x7f47f80ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80652d0 con 0x7f47f80a8c50 2022-07-21T01:40:13.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f811e530 con 0x7f47f80b01a0 2022-07-21T01:40:13.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fdd4d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8159c60 con 0x7f47f80b2710 2022-07-21T01:40:13.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fcd4b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80a8c50 0x7f47f815a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fcd4b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80a8c50 0x7f47f815a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47456/0 (socket says 172.21.15.3:47456) 2022-07-21T01:40:13.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fcd4b700 1 -- 172.21.15.3:0/1449679554 learned_addr learned my addr 172.21.15.3:0/1449679554 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:13.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47f7fff700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f80b01a0 0x7f47f81565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.226+0000 7f47fd54c700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47f80b2710 0x7f47f80ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47fcd4b700 1 -- 172.21.15.3:0/1449679554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47f80b2710 msgr2=0x7f47f80ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47fcd4b700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47f80b2710 0x7f47f80ab3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47fcd4b700 1 -- 172.21.15.3:0/1449679554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f80b01a0 msgr2=0x7f47f81565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47fcd4b700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f80b01a0 0x7f47f81565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47fcd4b700 1 -- 172.21.15.3:0/1449679554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec00b1e0 con 0x7f47f80a8c50 2022-07-21T01:40:13.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47fcd4b700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80a8c50 0x7f47f815a640 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f47ec002f70 tx=0x7f47ec015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:13.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47f5ffb700 1 -- 172.21.15.3:0/1449679554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47ec00d940 con 0x7f47f80a8c50 2022-07-21T01:40:13.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47f5ffb700 1 -- 172.21.15.3:0/1449679554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec011b80 con 0x7f47f80a8c50 2022-07-21T01:40:13.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f8159f00 con 0x7f47f80a8c50 2022-07-21T01:40:13.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47f5ffb700 1 -- 172.21.15.3:0/1449679554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47ec01e460 con 0x7f47f80a8c50 2022-07-21T01:40:13.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f81460e0 con 0x7f47f80a8c50 2022-07-21T01:40:13.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f80574e0 con 0x7f47f80a8c50 2022-07-21T01:40:13.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47f5ffb700 1 -- 172.21.15.3:0/1449679554 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f47ec025030 con 0x7f47f80a8c50 2022-07-21T01:40:13.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47f5ffb700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f47d8053950 0x7f47d8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47f5ffb700 1 -- 172.21.15.3:0/1449679554 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f47ec06ce70 con 0x7f47f80a8c50 2022-07-21T01:40:13.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.230+0000 7f47f7fff700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f47d8053950 0x7f47d8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.234+0000 7f47f7fff700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f47d8053950 0x7f47d8055e10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f47e800d000 tx=0x7f47e8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:13.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.238+0000 7f47f5ffb700 1 -- 172.21.15.3:0/1449679554 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f47ec043ee0 con 0x7f47f80a8c50 2022-07-21T01:40:13.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47f8069a70 con 0x7f47f80a8c50 2022-07-21T01:40:13.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47f5ffb700 1 -- 172.21.15.3:0/1449679554 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+1556 (secure 0 0 0) 0x7f47ec023020 con 0x7f47f80a8c50 2022-07-21T01:40:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:13.405 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5621,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3131448734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3131448734},{"type":"v1","addr":"172.21.15.3:6835","nonce":3131448734}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":5858,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/397964915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":397964915},{"type":"v1","addr":"172.21.15.143:6833","nonce":397964915}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":37},{"gid":6240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3989696679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3989696679},{"type":"v1","addr":"172.21.15.3:6837","nonce":3989696679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[]} 2022-07-21T01:40:13.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f47d8053950 msgr2=0x7f47d8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f47d8053950 0x7f47d8055e10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f47e800d000 tx=0x7f47e8000bc0).stop 2022-07-21T01:40:13.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80a8c50 msgr2=0x7f47f815a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80a8c50 0x7f47f815a640 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f47ec002f70 tx=0x7f47ec015000).stop 2022-07-21T01:40:13.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 shutdown_connections 2022-07-21T01:40:13.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47f80b2710 0x7f47f80ab3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f47d8053950 0x7f47d8055e10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f80b01a0 0x7f47f81565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 --2- 172.21.15.3:0/1449679554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f80a8c50 0x7f47f815a640 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 >> 172.21.15.3:0/1449679554 conn(0x7f47f8060310 msgr2=0x7f47f80aed70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:13.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 shutdown_connections 2022-07-21T01:40:13.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.402+0000 7f47fdd4d700 1 -- 172.21.15.3:0/1449679554 wait complete. 2022-07-21T01:40:13.410 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 37 2022-07-21T01:40:13.417 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:40:13.418 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:40:13.418 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:40:13.419 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:40:13.419 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:40:13.419 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:40:13.420 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:40:13.420 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:40:13.420 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:40:13.421 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:40:13.421 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:40:13.421 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:40:13.432 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:40:13.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:40:13.434 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:40:13.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:40:13.436 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:40:13.437 DEBUG:teuthology.parallel:result is None 2022-07-21T01:40:13.438 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:40:13.438 DEBUG:teuthology.parallel:result is None 2022-07-21T01:40:13.439 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:40:13.440 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:40:13.484 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:40:13.485 DEBUG:teuthology.parallel:result is None 2022-07-21T01:40:13.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:40:13.506 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:40:13.508 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:40:13.556 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:40:13.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.586+0000 7f285e5a6700 1 Processor -- start 2022-07-21T01:40:13.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285e5a6700 1 -- start start 2022-07-21T01:40:13.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285e5a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285805fb80 0x7f285813ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285e5a6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f28580b2050 0x7f2858140480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285e5a6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28580afa50 con 0x7f28580ae3c0 2022-07-21T01:40:13.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285e5a6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28580b03c0 con 0x7f28580b2050 2022-07-21T01:40:13.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285e5a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285805ff60 con 0x7f285805fb80 2022-07-21T01:40:13.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285cda3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f28580b2050 0x7f2858140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285cda3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f28580b2050 0x7f2858140480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47490/0 (socket says 172.21.15.3:47490) 2022-07-21T01:40:13.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285cda3700 1 -- 172.21.15.3:0/2740488001 learned_addr learned my addr 172.21.15.3:0/2740488001 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:13.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285dda5700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f28580ae3c0 0x7f28581434e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55772/0 (socket says 172.21.15.3:55772) 2022-07-21T01:40:13.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285d5a4700 1 --2- 172.21.15.3:0/2740488001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285805fb80 0x7f285813ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285cda3700 1 -- 172.21.15.3:0/2740488001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285805fb80 msgr2=0x7f285813ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:13.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285cda3700 1 --2- 172.21.15.3:0/2740488001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285805fb80 0x7f285813ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285cda3700 1 -- 172.21.15.3:0/2740488001 >> v1:172.21.15.3:6789/0 conn(0x7f28580ae3c0 legacy=0x7f28581434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285cda3700 1 -- 172.21.15.3:0/2740488001 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28580ad1c0 con 0x7f28580b2050 2022-07-21T01:40:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f285cda3700 1 --2- 172.21.15.3:0/2740488001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580b2050 0x7f2858140480 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f2848002f70 tx=0x7f2848006eb0).ready entity=mon.0 client_cookie=88215ac0eb1755f8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f2847fff700 1 -- 172.21.15.3:0/2740488001 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f284800d570 con 0x7f28580b2050 2022-07-21T01:40:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f2847fff700 1 -- 172.21.15.3:0/2740488001 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2848000e90 con 0x7f28580b2050 2022-07-21T01:40:13.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.590+0000 7f2847fff700 1 -- 172.21.15.3:0/2740488001 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f284800d900 con 0x7f28580b2050 2022-07-21T01:40:13.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 -- 172.21.15.3:0/2740488001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580b2050 msgr2=0x7f2858140480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 --2- 172.21.15.3:0/2740488001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580b2050 0x7f2858140480 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f2848002f70 tx=0x7f2848006eb0).stop 2022-07-21T01:40:13.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 -- 172.21.15.3:0/2740488001 shutdown_connections 2022-07-21T01:40:13.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 --2- 172.21.15.3:0/2740488001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580b2050 0x7f2858140480 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 --2- 172.21.15.3:0/2740488001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285805fb80 0x7f285813ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 -- 172.21.15.3:0/2740488001 >> 172.21.15.3:0/2740488001 conn(0x7f28580613c0 msgr2=0x7f2858062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:13.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 -- 172.21.15.3:0/2740488001 shutdown_connections 2022-07-21T01:40:13.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 -- 172.21.15.3:0/2740488001 wait complete. 2022-07-21T01:40:13.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 Processor -- start 2022-07-21T01:40:13.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 -- start start 2022-07-21T01:40:13.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285805fb80 0x7f285813e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580ae3c0 0x7f28580659d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28580b2050 0x7f2858065ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28581427c0 con 0x7f28580ae3c0 2022-07-21T01:40:13.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2858143e20 con 0x7f285805fb80 2022-07-21T01:40:13.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285e5a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2858143b80 con 0x7f28580b2050 2022-07-21T01:40:13.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285cda3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580ae3c0 0x7f28580659d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285cda3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580ae3c0 0x7f28580659d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47496/0 (socket says 172.21.15.3:47496) 2022-07-21T01:40:13.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285cda3700 1 -- 172.21.15.3:0/3422076250 learned_addr learned my addr 172.21.15.3:0/3422076250 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:13.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285d5a4700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285805fb80 0x7f285813e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285dda5700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28580b2050 0x7f2858065ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285cda3700 1 -- 172.21.15.3:0/3422076250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28580b2050 msgr2=0x7f2858065ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285cda3700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28580b2050 0x7f2858065ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285cda3700 1 -- 172.21.15.3:0/3422076250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285805fb80 msgr2=0x7f285813e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285cda3700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285805fb80 0x7f285813e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.594+0000 7f285cda3700 1 -- 172.21.15.3:0/3422076250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284800b1e0 con 0x7f28580ae3c0 2022-07-21T01:40:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f285cda3700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580ae3c0 0x7f28580659d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f2848002f70 tx=0x7f2848015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f28467fc700 1 -- 172.21.15.3:0/3422076250 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2848011760 con 0x7f28580ae3c0 2022-07-21T01:40:13.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f28467fc700 1 -- 172.21.15.3:0/3422076250 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28480118c0 con 0x7f28580ae3c0 2022-07-21T01:40:13.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2858142a20 con 0x7f28580ae3c0 2022-07-21T01:40:13.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f28467fc700 1 -- 172.21.15.3:0/3422076250 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2848011c20 con 0x7f28580ae3c0 2022-07-21T01:40:13.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2858139230 con 0x7f28580ae3c0 2022-07-21T01:40:13.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2858142bb0 con 0x7f28580ae3c0 2022-07-21T01:40:13.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f28467fc700 1 -- 172.21.15.3:0/3422076250 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2848025030 con 0x7f28580ae3c0 2022-07-21T01:40:13.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f28467fc700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2838053a70 0x7f2838055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:13.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f28467fc700 1 -- 172.21.15.3:0/3422076250 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5729+0+0 (secure 0 0 0) 0x7f284806ce80 con 0x7f28580ae3c0 2022-07-21T01:40:13.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.598+0000 7f285d5a4700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2838053a70 0x7f2838055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:13.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.602+0000 7f285d5a4700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2838053a70 0x7f2838055f30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f285400b000 tx=0x7f2854006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:13.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.606+0000 7f28467fc700 1 -- 172.21.15.3:0/3422076250 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2848043ef0 con 0x7f28580ae3c0 2022-07-21T01:40:13.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:13.746+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f28580afc80 con 0x7f28580ae3c0 2022-07-21T01:40:14.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f28467fc700 1 -- 172.21.15.3:0/3422076250 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v86) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2848023020 con 0x7f28580ae3c0 2022-07-21T01:40:14.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2838053a70 msgr2=0x7f2838055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:14.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2838053a70 0x7f2838055f30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f285400b000 tx=0x7f2854006df0).stop 2022-07-21T01:40:14.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580ae3c0 msgr2=0x7f28580659d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:14.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580ae3c0 0x7f28580659d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f2848002f70 tx=0x7f2848015000).stop 2022-07-21T01:40:14.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 shutdown_connections 2022-07-21T01:40:14.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28580b2050 0x7f2858065ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28580ae3c0 0x7f28580659d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2838053a70 0x7f2838055f30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 --2- 172.21.15.3:0/3422076250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285805fb80 0x7f285813e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 >> 172.21.15.3:0/3422076250 conn(0x7f28580613c0 msgr2=0x7f28580b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:14.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 shutdown_connections 2022-07-21T01:40:14.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.550+0000 7f285e5a6700 1 -- 172.21.15.3:0/3422076250 wait complete. 2022-07-21T01:40:14.558 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:40:14.571 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:40:14.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:40:14.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.722+0000 7f7e8bca4700 1 Processor -- start 2022-07-21T01:40:14.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.722+0000 7f7e8bca4700 1 -- start start 2022-07-21T01:40:14.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840b20a0 0x7f7e840b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e840aa520 con 0x7f7e840b0250 2022-07-21T01:40:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e840ab3f0 con 0x7f7e840a9d00 2022-07-21T01:40:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840b0630 con 0x7f7e840b20a0 2022-07-21T01:40:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e89a40700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7e840b0250 0x7f7e840aace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55786/0 (socket says 172.21.15.3:55786) 2022-07-21T01:40:14.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e89a40700 1 -- 172.21.15.3:0/996180177 learned_addr learned my addr 172.21.15.3:0/996180177 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:14.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8923f700 1 --2- 172.21.15.3:0/996180177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840b20a0 0x7f7e840b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:14.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e88a3e700 1 -- 172.21.15.3:0/996180177 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160235434 0 0) 0x7f7e840aa520 con 0x7f7e840b0250 2022-07-21T01:40:14.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e88a3e700 1 -- 172.21.15.3:0/996180177 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e6c003260 con 0x7f7e840b0250 2022-07-21T01:40:14.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e88a3e700 1 -- 172.21.15.3:0/996180177 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951663688 0 0) 0x7f7e840ab3f0 con 0x7f7e840a9d00 2022-07-21T01:40:14.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e88a3e700 1 -- 172.21.15.3:0/996180177 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e840aa520 con 0x7f7e840a9d00 2022-07-21T01:40:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8923f700 1 -- 172.21.15.3:0/996180177 >> v1:172.21.15.143:6789/0 conn(0x7f7e840a9d00 legacy=0x7f7e840b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8923f700 1 -- 172.21.15.3:0/996180177 >> v1:172.21.15.3:6789/0 conn(0x7f7e840b0250 legacy=0x7f7e840aace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:14.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8923f700 1 -- 172.21.15.3:0/996180177 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e840aa280 con 0x7f7e840b20a0 2022-07-21T01:40:14.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8923f700 1 --2- 172.21.15.3:0/996180177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840b20a0 0x7f7e840b1ab0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7e80002f70 tx=0x7f7e80006df0).ready entity=mon.2 client_cookie=48767a7412ea1275 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:14.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e88a3e700 1 -- 172.21.15.3:0/996180177 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e8000d5c0 con 0x7f7e840b20a0 2022-07-21T01:40:14.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e88a3e700 1 -- 172.21.15.3:0/996180177 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e80000e90 con 0x7f7e840b20a0 2022-07-21T01:40:14.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e88a3e700 1 -- 172.21.15.3:0/996180177 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e8000d910 con 0x7f7e840b20a0 2022-07-21T01:40:14.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 -- 172.21.15.3:0/996180177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840b20a0 msgr2=0x7f7e840b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:14.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 --2- 172.21.15.3:0/996180177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840b20a0 0x7f7e840b1ab0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7e80002f70 tx=0x7f7e80006df0).stop 2022-07-21T01:40:14.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 -- 172.21.15.3:0/996180177 shutdown_connections 2022-07-21T01:40:14.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 --2- 172.21.15.3:0/996180177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840b20a0 0x7f7e840b1ab0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 -- 172.21.15.3:0/996180177 >> 172.21.15.3:0/996180177 conn(0x7f7e84060950 msgr2=0x7f7e84060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:14.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 -- 172.21.15.3:0/996180177 shutdown_connections 2022-07-21T01:40:14.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.726+0000 7f7e8bca4700 1 -- 172.21.15.3:0/996180177 wait complete. 2022-07-21T01:40:14.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.730+0000 7f7e8bca4700 1 Processor -- start 2022-07-21T01:40:14.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.730+0000 7f7e8bca4700 1 -- start start 2022-07-21T01:40:14.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.730+0000 7f7e8bca4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840a9d00 0x7f7e84063da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:14.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.730+0000 7f7e8bca4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e840b0250 0x7f7e8413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:14.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.730+0000 7f7e8bca4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e840b20a0 0x7f7e8413dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:14.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.730+0000 7f7e8bca4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e84142000 con 0x7f7e840b0250 2022-07-21T01:40:14.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.730+0000 7f7e8bca4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e84142810 con 0x7f7e840b20a0 2022-07-21T01:40:14.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.730+0000 7f7e8bca4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e84142ab0 con 0x7f7e840a9d00 2022-07-21T01:40:14.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.730+0000 7f7e8923f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e840b0250 0x7f7e8413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:14.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8923f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e840b0250 0x7f7e8413d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47514/0 (socket says 172.21.15.3:47514) 2022-07-21T01:40:14.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8923f700 1 -- 172.21.15.3:0/729230176 learned_addr learned my addr 172.21.15.3:0/729230176 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:14.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e89a40700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840a9d00 0x7f7e84063da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:14.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8a241700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e840b20a0 0x7f7e8413dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:14.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8923f700 1 -- 172.21.15.3:0/729230176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840a9d00 msgr2=0x7f7e84063da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:14.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8923f700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840a9d00 0x7f7e84063da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8923f700 1 -- 172.21.15.3:0/729230176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e840b20a0 msgr2=0x7f7e8413dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:14.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8923f700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e840b20a0 0x7f7e8413dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8923f700 1 -- 172.21.15.3:0/729230176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8000b180 con 0x7f7e840b0250 2022-07-21T01:40:14.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8923f700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e840b0250 0x7f7e8413d7b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7f7e80002f70 tx=0x7f7e80014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:14.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e72ffd700 1 -- 172.21.15.3:0/729230176 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e80016070 con 0x7f7e840b0250 2022-07-21T01:40:14.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e72ffd700 1 -- 172.21.15.3:0/729230176 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8000de40 con 0x7f7e840b0250 2022-07-21T01:40:14.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e84142f50 con 0x7f7e840b0250 2022-07-21T01:40:14.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e72ffd700 1 -- 172.21.15.3:0/729230176 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e80013d20 con 0x7f7e840b0250 2022-07-21T01:40:14.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e84141c80 con 0x7f7e840b0250 2022-07-21T01:40:14.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e84142c70 con 0x7f7e840b0250 2022-07-21T01:40:14.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.734+0000 7f7e72ffd700 1 -- 172.21.15.3:0/729230176 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7e80025030 con 0x7f7e840b0250 2022-07-21T01:40:14.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.738+0000 7f7e72ffd700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e64053b00 0x7f7e64055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:14.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.738+0000 7f7e72ffd700 1 -- 172.21.15.3:0/729230176 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f7e8006ca10 con 0x7f7e840b0250 2022-07-21T01:40:14.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.738+0000 7f7e89a40700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e64053b00 0x7f7e64055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:14.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.738+0000 7f7e89a40700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e64053b00 0x7f7e64055fc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7e7400d000 tx=0x7f7e74009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:14.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.746+0000 7f7e72ffd700 1 -- 172.21.15.3:0/729230176 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7e80043b90 con 0x7f7e840b0250 2022-07-21T01:40:14.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.914+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7e840574e0 con 0x7f7e840b0250 2022-07-21T01:40:14.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.914+0000 7f7e72ffd700 1 -- 172.21.15.3:0/729230176 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f7e80023020 con 0x7f7e840b0250 2022-07-21T01:40:14.921 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:14.922 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:40:14.922 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:40:14.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.923 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:40:14.924 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:40:14.925 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:14.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.927 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.927 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.929 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:40:14.929 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:40:14.930 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:14.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.932 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.932 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.934 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:40:14.935 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:40:14.935 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:14.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.938 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.938 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.940 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:40:14.940 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:40:14.941 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.942 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.943 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.944 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:40:14.944 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:40:14.945 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.945 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.946 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.947 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:40:14.947 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:40:14.947 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.948 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.948 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.949 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:40:14.950 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:40:14.950 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.951 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.951 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.952 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:40:14.952 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:40:14.952 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.953 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.953 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.953 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.954 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.954 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:40:14.955 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:40:14.955 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.955 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.955 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.956 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.956 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.957 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:40:14.957 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:40:14.957 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.957 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.958 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.958 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.958 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.959 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:40:14.959 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:40:14.960 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.960 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.960 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.961 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.962 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:40:14.962 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:40:14.962 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.962 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:14.963 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:40:14.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:40:14.963 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.964 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:40:14.964 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:40:14.965 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.965 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:14.965 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:40:14.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:40:14.966 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.967 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:40:14.967 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:40:14.967 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.967 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:40:14.968 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:14.968 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:14.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.969 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.969 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:40:14.970 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:40:14.970 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.970 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:40:14.971 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:40:14.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.971 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:14.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:14.972 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:40:14.972 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:40:14.972 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:14.973 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:40:14.973 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:40:14.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:14.973 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.974 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:14.974 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:14.974 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:40:14.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e64053b00 msgr2=0x7f7e64055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:14.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e64053b00 0x7f7e64055fc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7e7400d000 tx=0x7f7e74009040).stop 2022-07-21T01:40:14.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e840b0250 msgr2=0x7f7e8413d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:14.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e840b0250 0x7f7e8413d7b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7f7e80002f70 tx=0x7f7e80014000).stop 2022-07-21T01:40:14.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 shutdown_connections 2022-07-21T01:40:14.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e64053b00 0x7f7e64055fc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e840b20a0 0x7f7e8413dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e840b0250 0x7f7e8413d7b0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 --2- 172.21.15.3:0/729230176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e840a9d00 0x7f7e84063da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:14.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 >> 172.21.15.3:0/729230176 conn(0x7f7e84060950 msgr2=0x7f7e840b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:14.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 shutdown_connections 2022-07-21T01:40:14.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:14.918+0000 7f7e8bca4700 1 -- 172.21.15.3:0/729230176 wait complete. 2022-07-21T01:40:14.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:15.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.078+0000 7f9365404700 1 Processor -- start 2022-07-21T01:40:15.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.078+0000 7f9365404700 1 -- start start 2022-07-21T01:40:15.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 0x7f9360064bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f93601470d0 0x7f93601403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93600b4f60 con 0x7f93600647d0 2022-07-21T01:40:15.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93600b4a60 con 0x7f93601470d0 2022-07-21T01:40:15.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93600b4cc0 con 0x7f93600b38a0 2022-07-21T01:40:15.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9364c03700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f93601470d0 0x7f93601403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9364c03700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f93601470d0 0x7f93601403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47524/0 (socket says 172.21.15.3:47524) 2022-07-21T01:40:15.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9364c03700 1 -- 172.21.15.3:0/540820714 learned_addr learned my addr 172.21.15.3:0/540820714 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:15.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f935f7fe700 1 --2- 172.21.15.3:0/540820714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 0x7f9360064bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9364c03700 1 -- 172.21.15.3:0/540820714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 msgr2=0x7f9360064bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:15.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9364c03700 1 --2- 172.21.15.3:0/540820714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 0x7f9360064bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9364c03700 1 -- 172.21.15.3:0/540820714 >> v1:172.21.15.143:6789/0 conn(0x7f93600647d0 legacy=0x7f93601436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f935effd700 1 -- 172.21.15.3:0/540820714 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425886824 0 0) 0x7f93600b4f60 con 0x7f93600647d0 2022-07-21T01:40:15.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9364c03700 1 -- 172.21.15.3:0/540820714 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9360143250 con 0x7f93601470d0 2022-07-21T01:40:15.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9364c03700 1 --2- 172.21.15.3:0/540820714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93601470d0 0x7f93601403f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f93580032f0 tx=0x7f9358009290).ready entity=mon.0 client_cookie=9d3cfcf7a9167485 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f935effd700 1 -- 172.21.15.3:0/540820714 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935800b8a0 con 0x7f93601470d0 2022-07-21T01:40:15.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f935effd700 1 -- 172.21.15.3:0/540820714 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935800ba00 con 0x7f93601470d0 2022-07-21T01:40:15.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f935effd700 1 -- 172.21.15.3:0/540820714 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935800bb70 con 0x7f93601470d0 2022-07-21T01:40:15.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 -- 172.21.15.3:0/540820714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93601470d0 msgr2=0x7f93601403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 --2- 172.21.15.3:0/540820714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93601470d0 0x7f93601403f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f93580032f0 tx=0x7f9358009290).stop 2022-07-21T01:40:15.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 -- 172.21.15.3:0/540820714 shutdown_connections 2022-07-21T01:40:15.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 --2- 172.21.15.3:0/540820714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93601470d0 0x7f93601403f0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 --2- 172.21.15.3:0/540820714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 0x7f9360064bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 -- 172.21.15.3:0/540820714 >> 172.21.15.3:0/540820714 conn(0x7f936005f840 msgr2=0x7f936005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:15.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 -- 172.21.15.3:0/540820714 shutdown_connections 2022-07-21T01:40:15.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.082+0000 7f9365404700 1 -- 172.21.15.3:0/540820714 wait complete. 2022-07-21T01:40:15.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 Processor -- start 2022-07-21T01:40:15.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 -- start start 2022-07-21T01:40:15.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93600647d0 0x7f9360145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 0x7f93601524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93601470d0 0x7f93600b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9360146e60 con 0x7f93600647d0 2022-07-21T01:40:15.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936013daf0 con 0x7f93601470d0 2022-07-21T01:40:15.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9360145d10 con 0x7f93600b38a0 2022-07-21T01:40:15.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93600647d0 0x7f9360145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93600647d0 0x7f9360145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47526/0 (socket says 172.21.15.3:47526) 2022-07-21T01:40:15.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935ffff700 1 -- 172.21.15.3:0/61952183 learned_addr learned my addr 172.21.15.3:0/61952183 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:15.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935f7fe700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 0x7f93601524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9364c03700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93601470d0 0x7f93600b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935ffff700 1 -- 172.21.15.3:0/61952183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 msgr2=0x7f93601524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935ffff700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 0x7f93601524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935ffff700 1 -- 172.21.15.3:0/61952183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93601470d0 msgr2=0x7f93600b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935ffff700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93601470d0 0x7f93600b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935ffff700 1 -- 172.21.15.3:0/61952183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9358009530 con 0x7f93600647d0 2022-07-21T01:40:15.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9364c03700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93601470d0 0x7f93600b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:15.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935ffff700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93600647d0 0x7f9360145350 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f9354000ed0 tx=0x7f9354007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935d7fa700 1 -- 172.21.15.3:0/61952183 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935400ce20 con 0x7f93600647d0 2022-07-21T01:40:15.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935d7fa700 1 -- 172.21.15.3:0/61952183 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935400f470 con 0x7f93600647d0 2022-07-21T01:40:15.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93600b6100 con 0x7f93600647d0 2022-07-21T01:40:15.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f935d7fa700 1 -- 172.21.15.3:0/61952183 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935400f7d0 con 0x7f93600647d0 2022-07-21T01:40:15.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.086+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93601529c0 con 0x7f93600647d0 2022-07-21T01:40:15.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.090+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9360146ba0 con 0x7f93600647d0 2022-07-21T01:40:15.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.090+0000 7f935d7fa700 1 -- 172.21.15.3:0/61952183 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9354019030 con 0x7f93600647d0 2022-07-21T01:40:15.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.090+0000 7f935d7fa700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9340053980 0x7f9340055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.090+0000 7f935d7fa700 1 -- 172.21.15.3:0/61952183 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f935406b1a0 con 0x7f93600647d0 2022-07-21T01:40:15.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.090+0000 7f935f7fe700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9340053980 0x7f9340055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.090+0000 7f935f7fe700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9340053980 0x7f9340055e40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f935000b000 tx=0x7f9350006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.098+0000 7f935d7fa700 1 -- 172.21.15.3:0/61952183 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9360146ba0 con 0x7f93600647d0 2022-07-21T01:40:15.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.222+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9360146ba0 con 0x7f93600647d0 2022-07-21T01:40:15.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.222+0000 7f935d7fa700 1 -- 172.21.15.3:0/61952183 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+13415 (secure 0 0 0) 0x7f9360146ba0 con 0x7f93600647d0 2022-07-21T01:40:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":86,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:14.539766+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:15.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9340053980 msgr2=0x7f9340055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9340053980 0x7f9340055e40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f935000b000 tx=0x7f9350006df0).stop 2022-07-21T01:40:15.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93600647d0 msgr2=0x7f9360145350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93600647d0 0x7f9360145350 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f9354000ed0 tx=0x7f9354007ac0).stop 2022-07-21T01:40:15.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 shutdown_connections 2022-07-21T01:40:15.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9340053980 0x7f9340055e40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93601470d0 0x7f93600b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600b38a0 0x7f93601524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 --2- 172.21.15.3:0/61952183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93600647d0 0x7f9360145350 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 >> 172.21.15.3:0/61952183 conn(0x7f936005f840 msgr2=0x7f9360142910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:15.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 shutdown_connections 2022-07-21T01:40:15.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.226+0000 7f9365404700 1 -- 172.21.15.3:0/61952183 wait complete. 2022-07-21T01:40:15.241 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:40:15.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:40:15.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 Processor -- start 2022-07-21T01:40:15.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 -- start start 2022-07-21T01:40:15.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd45813e5f0 0x7fd458064a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd458065530 con 0x7fd4580b3a30 2022-07-21T01:40:15.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4580b3260 con 0x7fd458144530 2022-07-21T01:40:15.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd458066160 con 0x7fd45813e5f0 2022-07-21T01:40:15.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd4567fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd4580b3a30 0x7fd4580b76c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55816/0 (socket says 172.21.15.3:55816) 2022-07-21T01:40:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd456ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd45813e5f0 0x7fd458064a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd4567fc700 1 -- 172.21.15.3:0/2072632621 learned_addr learned my addr 172.21.15.3:0/2072632621 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd455ffb700 1 -- 172.21.15.3:0/2072632621 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637193341 0 0) 0x7fd458065530 con 0x7fd4580b3a30 2022-07-21T01:40:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd455ffb700 1 -- 172.21.15.3:0/2072632621 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd43c003260 con 0x7fd4580b3a30 2022-07-21T01:40:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd456ffd700 1 -- 172.21.15.3:0/2072632621 >> v1:172.21.15.143:6789/0 conn(0x7fd458144530 legacy=0x7fd458066f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd456ffd700 1 -- 172.21.15.3:0/2072632621 >> v1:172.21.15.3:6789/0 conn(0x7fd4580b3a30 legacy=0x7fd4580b76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd456ffd700 1 -- 172.21.15.3:0/2072632621 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4580b7dd0 con 0x7fd45813e5f0 2022-07-21T01:40:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd456ffd700 1 --2- 172.21.15.3:0/2072632621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 0x7fd458064a00 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7fd448002f70 tx=0x7fd448006df0).ready entity=mon.0 client_cookie=be2b9b43c9117dea server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd455ffb700 1 -- 172.21.15.3:0/2072632621 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd44800d610 con 0x7fd45813e5f0 2022-07-21T01:40:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd455ffb700 1 -- 172.21.15.3:0/2072632621 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd448000e90 con 0x7fd45813e5f0 2022-07-21T01:40:15.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd455ffb700 1 -- 172.21.15.3:0/2072632621 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd44800d960 con 0x7fd45813e5f0 2022-07-21T01:40:15.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 -- 172.21.15.3:0/2072632621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 msgr2=0x7fd458064a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 --2- 172.21.15.3:0/2072632621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 0x7fd458064a00 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7fd448002f70 tx=0x7fd448006df0).stop 2022-07-21T01:40:15.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 -- 172.21.15.3:0/2072632621 shutdown_connections 2022-07-21T01:40:15.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 --2- 172.21.15.3:0/2072632621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 0x7fd458064a00 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.350+0000 7fd45d60a700 1 -- 172.21.15.3:0/2072632621 >> 172.21.15.3:0/2072632621 conn(0x7fd458062b60 msgr2=0x7fd458062f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:15.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 -- 172.21.15.3:0/2072632621 shutdown_connections 2022-07-21T01:40:15.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 -- 172.21.15.3:0/2072632621 wait complete. 2022-07-21T01:40:15.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 Processor -- start 2022-07-21T01:40:15.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 -- start start 2022-07-21T01:40:15.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4580b3a30 0x7fd4580ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 0x7fd4580b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd458144530 0x7fd4580b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580ac4e0 con 0x7fd45813e5f0 2022-07-21T01:40:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580ad940 con 0x7fd458144530 2022-07-21T01:40:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580b17d0 con 0x7fd4580b3a30 2022-07-21T01:40:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4567fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 0x7fd4580b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4567fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 0x7fd4580b05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47540/0 (socket says 172.21.15.3:47540) 2022-07-21T01:40:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4567fc700 1 -- 172.21.15.3:0/1871382851 learned_addr learned my addr 172.21.15.3:0/1871382851 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd456ffd700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4580b3a30 0x7fd4580ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4577fe700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd458144530 0x7fd4580b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4567fc700 1 -- 172.21.15.3:0/1871382851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4580b3a30 msgr2=0x7fd4580ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4567fc700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4580b3a30 0x7fd4580ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4567fc700 1 -- 172.21.15.3:0/1871382851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd458144530 msgr2=0x7fd4580b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4567fc700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd458144530 0x7fd4580b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4567fc700 1 -- 172.21.15.3:0/1871382851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44800b180 con 0x7fd45813e5f0 2022-07-21T01:40:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd4567fc700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 0x7fd4580b05b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fd44c001110 tx=0x7fd44c007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd447fff700 1 -- 172.21.15.3:0/1871382851 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd44c0070d0 con 0x7fd45813e5f0 2022-07-21T01:40:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd447fff700 1 -- 172.21.15.3:0/1871382851 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44c007230 con 0x7fd45813e5f0 2022-07-21T01:40:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4580ab4f0 con 0x7fd45813e5f0 2022-07-21T01:40:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd447fff700 1 -- 172.21.15.3:0/1871382851 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd44c005610 con 0x7fd45813e5f0 2022-07-21T01:40:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.354+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4580abed0 con 0x7fd45813e5f0 2022-07-21T01:40:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.358+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd458064ff0 con 0x7fd45813e5f0 2022-07-21T01:40:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.358+0000 7fd447fff700 1 -- 172.21.15.3:0/1871382851 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd44c010070 con 0x7fd45813e5f0 2022-07-21T01:40:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.358+0000 7fd447fff700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd434053a70 0x7fd434055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.358+0000 7fd447fff700 1 -- 172.21.15.3:0/1871382851 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fd44c06b3a0 con 0x7fd45813e5f0 2022-07-21T01:40:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.358+0000 7fd456ffd700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd434053a70 0x7fd434055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.358+0000 7fd456ffd700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd434053a70 0x7fd434055f30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd44801a000 tx=0x7fd448014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.366+0000 7fd447fff700 1 -- 172.21.15.3:0/1871382851 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd44c042390 con 0x7fd45813e5f0 2022-07-21T01:40:15.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.494+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd4580574e0 con 0x7fd45813e5f0 2022-07-21T01:40:15.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.494+0000 7fd447fff700 1 -- 172.21.15.3:0/1871382851 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v86) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd44c018020 con 0x7fd45813e5f0 2022-07-21T01:40:15.498 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:40:15.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd434053a70 msgr2=0x7fd434055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd434053a70 0x7fd434055f30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd44801a000 tx=0x7fd448014000).stop 2022-07-21T01:40:15.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 msgr2=0x7fd4580b05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 0x7fd4580b05b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fd44c001110 tx=0x7fd44c007ad0).stop 2022-07-21T01:40:15.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 shutdown_connections 2022-07-21T01:40:15.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd434053a70 0x7fd434055f30 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd458144530 0x7fd4580b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd45813e5f0 0x7fd4580b05b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 --2- 172.21.15.3:0/1871382851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd4580b3a30 0x7fd4580ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 >> 172.21.15.3:0/1871382851 conn(0x7fd458062b60 msgr2=0x7fd458140880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:15.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 shutdown_connections 2022-07-21T01:40:15.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.498+0000 7fd45d60a700 1 -- 172.21.15.3:0/1871382851 wait complete. 2022-07-21T01:40:15.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:40:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.618+0000 7fb5452c4700 1 Processor -- start 2022-07-21T01:40:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.618+0000 7fb5452c4700 1 -- start start 2022-07-21T01:40:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.618+0000 7fb5452c4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5400614b0 0x7fb540061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.618+0000 7fb5452c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 0x7fb5400b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.618+0000 7fb5452c4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5401405d0 0x7fb540140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.618+0000 7fb5452c4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5400aee80 con 0x7fb5401405d0 2022-07-21T01:40:15.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.618+0000 7fb5452c4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5400b70b0 con 0x7fb5400614b0 2022-07-21T01:40:15.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.618+0000 7fb5452c4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400af0b0 con 0x7fb5400ad750 2022-07-21T01:40:15.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb544ac3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5401405d0 0x7fb540140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb544ac3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5401405d0 0x7fb540140a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47552/0 (socket says 172.21.15.3:47552) 2022-07-21T01:40:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb544ac3700 1 -- 172.21.15.3:0/485357674 learned_addr learned my addr 172.21.15.3:0/485357674 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb53f7fe700 1 --2- 172.21.15.3:0/485357674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 0x7fb5400b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb53ffff700 1 --2- 172.21.15.3:0/485357674 >> v2:172.21.15.143:3300/0 conn(0x7fb5400614b0 0x7fb540061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb544ac3700 1 -- 172.21.15.3:0/485357674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 msgr2=0x7fb5400b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:15.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb544ac3700 1 --2- 172.21.15.3:0/485357674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 0x7fb5400b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb544ac3700 1 -- 172.21.15.3:0/485357674 >> v2:172.21.15.143:3300/0 conn(0x7fb5400614b0 msgr2=0x7fb540061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:15.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb544ac3700 1 --2- 172.21.15.3:0/485357674 >> v2:172.21.15.143:3300/0 conn(0x7fb5400614b0 0x7fb540061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb544ac3700 1 -- 172.21.15.3:0/485357674 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5400af350 con 0x7fb5401405d0 2022-07-21T01:40:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb53ffff700 1 --2- 172.21.15.3:0/485357674 >> v2:172.21.15.143:3300/0 conn(0x7fb5400614b0 0x7fb540061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb53f7fe700 1 --2- 172.21.15.3:0/485357674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 0x7fb5400b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb544ac3700 1 --2- 172.21.15.3:0/485357674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 0x7fb540140a60 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7fb5380032f0 tx=0x7fb538009290).ready entity=mon.0 client_cookie=6cb9509041ccb164 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb53effd700 1 -- 172.21.15.3:0/485357674 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53800b870 con 0x7fb5401405d0 2022-07-21T01:40:15.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb53effd700 1 -- 172.21.15.3:0/485357674 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53800b9d0 con 0x7fb5401405d0 2022-07-21T01:40:15.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb53effd700 1 -- 172.21.15.3:0/485357674 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53800bb40 con 0x7fb5401405d0 2022-07-21T01:40:15.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb5452c4700 1 -- 172.21.15.3:0/485357674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 msgr2=0x7fb540140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb5452c4700 1 --2- 172.21.15.3:0/485357674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 0x7fb540140a60 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7fb5380032f0 tx=0x7fb538009290).stop 2022-07-21T01:40:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb5452c4700 1 -- 172.21.15.3:0/485357674 shutdown_connections 2022-07-21T01:40:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb5452c4700 1 --2- 172.21.15.3:0/485357674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 0x7fb540140a60 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb5452c4700 1 --2- 172.21.15.3:0/485357674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 0x7fb5400b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb5452c4700 1 --2- 172.21.15.3:0/485357674 >> v2:172.21.15.143:3300/0 conn(0x7fb5400614b0 0x7fb540061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb5452c4700 1 -- 172.21.15.3:0/485357674 >> 172.21.15.3:0/485357674 conn(0x7fb540066f70 msgr2=0x7fb540067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:15.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb5452c4700 1 -- 172.21.15.3:0/485357674 shutdown_connections 2022-07-21T01:40:15.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.622+0000 7fb5452c4700 1 -- 172.21.15.3:0/485357674 wait complete. 2022-07-21T01:40:15.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 Processor -- start 2022-07-21T01:40:15.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 -- start start 2022-07-21T01:40:15.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400614b0 0x7fb540148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 0x7fb540153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 0x7fb5400abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400acbc0 con 0x7fb5401405d0 2022-07-21T01:40:15.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400ace20 con 0x7fb5400614b0 2022-07-21T01:40:15.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400ad0c0 con 0x7fb5400ad750 2022-07-21T01:40:15.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb544ac3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 0x7fb5400abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb544ac3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 0x7fb5400abd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47558/0 (socket says 172.21.15.3:47558) 2022-07-21T01:40:15.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb544ac3700 1 -- 172.21.15.3:0/4052294308 learned_addr learned my addr 172.21.15.3:0/4052294308 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:15.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb53f7fe700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 0x7fb540153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb53ffff700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400614b0 0x7fb540148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb544ac3700 1 -- 172.21.15.3:0/4052294308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 msgr2=0x7fb540153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb544ac3700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 0x7fb540153160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb544ac3700 1 -- 172.21.15.3:0/4052294308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400614b0 msgr2=0x7fb540148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb544ac3700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400614b0 0x7fb540148f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb544ac3700 1 -- 172.21.15.3:0/4052294308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb538009530 con 0x7fb5401405d0 2022-07-21T01:40:15.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb544ac3700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 0x7fb5400abd40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7fb5380032f0 tx=0x7fb538016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb53d7fa700 1 -- 172.21.15.3:0/4052294308 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53800b870 con 0x7fb5401405d0 2022-07-21T01:40:15.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb53d7fa700 1 -- 172.21.15.3:0/4052294308 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb538020430 con 0x7fb5401405d0 2022-07-21T01:40:15.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb53d7fa700 1 -- 172.21.15.3:0/4052294308 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb538020790 con 0x7fb5401405d0 2022-07-21T01:40:15.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb540153650 con 0x7fb5401405d0 2022-07-21T01:40:15.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.626+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54014a9c0 con 0x7fb5401405d0 2022-07-21T01:40:15.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.630+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5400af5f0 con 0x7fb5401405d0 2022-07-21T01:40:15.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.630+0000 7fb53d7fa700 1 -- 172.21.15.3:0/4052294308 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb538028070 con 0x7fb5401405d0 2022-07-21T01:40:15.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.630+0000 7fb53d7fa700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb524053950 0x7fb524055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.630+0000 7fb53d7fa700 1 -- 172.21.15.3:0/4052294308 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fb53806c670 con 0x7fb5401405d0 2022-07-21T01:40:15.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.630+0000 7fb53ffff700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb524053950 0x7fb524055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.630+0000 7fb53ffff700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb524053950 0x7fb524055e10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fb52800c000 tx=0x7fb528008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.638+0000 7fb53d7fa700 1 -- 172.21.15.3:0/4052294308 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb538043770 con 0x7fb5401405d0 2022-07-21T01:40:15.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb5400574e0 con 0x7fb5401405d0 2022-07-21T01:40:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb53d7fa700 1 -- 172.21.15.3:0/4052294308 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v86) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb538015020 con 0x7fb5401405d0 2022-07-21T01:40:15.769 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:15.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb524053950 msgr2=0x7fb524055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb524053950 0x7fb524055e10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fb52800c000 tx=0x7fb528008040).stop 2022-07-21T01:40:15.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 msgr2=0x7fb5400abd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 0x7fb5400abd40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7fb5380032f0 tx=0x7fb538016000).stop 2022-07-21T01:40:15.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 shutdown_connections 2022-07-21T01:40:15.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5401405d0 0x7fb5400abd40 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5400ad750 0x7fb540153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb524053950 0x7fb524055e10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.766+0000 7fb5452c4700 1 --2- 172.21.15.3:0/4052294308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400614b0 0x7fb540148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.770+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 >> 172.21.15.3:0/4052294308 conn(0x7fb540066f70 msgr2=0x7fb5400b88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:15.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.770+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 shutdown_connections 2022-07-21T01:40:15.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.770+0000 7fb5452c4700 1 -- 172.21.15.3:0/4052294308 wait complete. 2022-07-21T01:40:15.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:40:15.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b94d0700 1 Processor -- start 2022-07-21T01:40:15.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b94d0700 1 -- start start 2022-07-21T01:40:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b94d0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f81b40b2710 0x7f81b40abca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b94d0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b0470 0x7f81b40ac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b94d0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b40adf10 con 0x7f81b40b13f0 2022-07-21T01:40:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b94d0700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b40ae1b0 con 0x7f81b40b2710 2022-07-21T01:40:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b94d0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40a9520 con 0x7f81b40b0470 2022-07-21T01:40:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b37fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f81b40b13f0 0x7f81b40afd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55842/0 (socket says 172.21.15.3:55842) 2022-07-21T01:40:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b37fe700 1 -- 172.21.15.3:0/439080234 learned_addr learned my addr 172.21.15.3:0/439080234 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f81b40b2710 0x7f81b40abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b8ccf700 1 --2- 172.21.15.3:0/439080234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b0470 0x7f81b40ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b2ffd700 1 -- 172.21.15.3:0/439080234 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536788251 0 0) 0x7f81b40adf10 con 0x7f81b40b13f0 2022-07-21T01:40:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.882+0000 7f81b2ffd700 1 -- 172.21.15.3:0/439080234 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a0003260 con 0x7f81b40b13f0 2022-07-21T01:40:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b3fff700 1 -- 172.21.15.3:0/439080234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b0470 msgr2=0x7f81b40ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b3fff700 1 --2- 172.21.15.3:0/439080234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b0470 0x7f81b40ac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b3fff700 1 -- 172.21.15.3:0/439080234 >> v1:172.21.15.3:6789/0 conn(0x7f81b40b13f0 legacy=0x7f81b40afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b3fff700 1 -- 172.21.15.3:0/439080234 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40a99c0 con 0x7f81b40b2710 2022-07-21T01:40:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b3fff700 1 --2- 172.21.15.3:0/439080234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 0x7f81b40abca0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f81a4002f70 tx=0x7f81a4006df0).ready entity=mon.0 client_cookie=e9a7acda57ef46d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b2ffd700 1 -- 172.21.15.3:0/439080234 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81a400d4a0 con 0x7f81b40b2710 2022-07-21T01:40:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b2ffd700 1 -- 172.21.15.3:0/439080234 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a4000e90 con 0x7f81b40b2710 2022-07-21T01:40:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b2ffd700 1 -- 172.21.15.3:0/439080234 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81a400d7f0 con 0x7f81b40b2710 2022-07-21T01:40:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b94d0700 1 -- 172.21.15.3:0/439080234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 msgr2=0x7f81b40abca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b94d0700 1 --2- 172.21.15.3:0/439080234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 0x7f81b40abca0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f81a4002f70 tx=0x7f81a4006df0).stop 2022-07-21T01:40:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b94d0700 1 -- 172.21.15.3:0/439080234 shutdown_connections 2022-07-21T01:40:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b94d0700 1 --2- 172.21.15.3:0/439080234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b0470 0x7f81b40ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b94d0700 1 --2- 172.21.15.3:0/439080234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 0x7f81b40abca0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b94d0700 1 -- 172.21.15.3:0/439080234 >> 172.21.15.3:0/439080234 conn(0x7f81b4060310 msgr2=0x7f81b40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:15.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b94d0700 1 -- 172.21.15.3:0/439080234 shutdown_connections 2022-07-21T01:40:15.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b94d0700 1 -- 172.21.15.3:0/439080234 wait complete. 2022-07-21T01:40:15.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.886+0000 7f81b94d0700 1 Processor -- start 2022-07-21T01:40:15.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b94d0700 1 -- start start 2022-07-21T01:40:15.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b94d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b40b0470 0x7f81b40ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b94d0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b13f0 0x7f81b4153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b94d0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 0x7f81b4153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b94d0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b4145180 con 0x7f81b40b2710 2022-07-21T01:40:15.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b94d0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b41453b0 con 0x7f81b40b0470 2022-07-21T01:40:15.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b94d0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b4145900 con 0x7f81b40b13f0 2022-07-21T01:40:15.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b8ccf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 0x7f81b4153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b8ccf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 0x7f81b4153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47572/0 (socket says 172.21.15.3:47572) 2022-07-21T01:40:15.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b8ccf700 1 -- 172.21.15.3:0/158624725 learned_addr learned my addr 172.21.15.3:0/158624725 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b37fe700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b13f0 0x7f81b4153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b3fff700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b40b0470 0x7f81b40ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b8ccf700 1 -- 172.21.15.3:0/158624725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b13f0 msgr2=0x7f81b4153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b8ccf700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b13f0 0x7f81b4153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b8ccf700 1 -- 172.21.15.3:0/158624725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b40b0470 msgr2=0x7f81b40ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b8ccf700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b40b0470 0x7f81b40ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b8ccf700 1 -- 172.21.15.3:0/158624725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a400b180 con 0x7f81b40b2710 2022-07-21T01:40:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b8ccf700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 0x7f81b4153df0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f81ac00c000 tx=0x7f81ac0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b17fa700 1 -- 172.21.15.3:0/158624725 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81ac012600 con 0x7f81b40b2710 2022-07-21T01:40:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b17fa700 1 -- 172.21.15.3:0/158624725 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac012760 con 0x7f81b40b2710 2022-07-21T01:40:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b4145ba0 con 0x7f81b40b2710 2022-07-21T01:40:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b17fa700 1 -- 172.21.15.3:0/158624725 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81ac012ac0 con 0x7f81b40b2710 2022-07-21T01:40:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.890+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b40ac100 con 0x7f81b40b2710 2022-07-21T01:40:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.894+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b4144f80 con 0x7f81b40b2710 2022-07-21T01:40:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.894+0000 7f81b17fa700 1 -- 172.21.15.3:0/158624725 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f81ac022070 con 0x7f81b40b2710 2022-07-21T01:40:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.894+0000 7f81b17fa700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8194053a70 0x7f8194055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.894+0000 7f81b17fa700 1 -- 172.21.15.3:0/158624725 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f81ac06c320 con 0x7f81b40b2710 2022-07-21T01:40:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.894+0000 7f81b3fff700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8194053a70 0x7f8194055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.894+0000 7f81b3fff700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8194053a70 0x7f8194055f30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f81a401a000 tx=0x7f81a4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:15.898+0000 7f81b17fa700 1 -- 172.21.15.3:0/158624725 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f81ac043420 con 0x7f81b40b2710 2022-07-21T01:40:16.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.026+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f81b40574e0 con 0x7f81b40b2710 2022-07-21T01:40:16.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.026+0000 7f81b17fa700 1 -- 172.21.15.3:0/158624725 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v86) v1 ==== 84+0+143 (secure 0 0 0) 0x7f81ac046a40 con 0x7f81b40b2710 2022-07-21T01:40:16.030 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:16.031 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:40:16.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:16.033 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:40:16.033 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:40:16.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:16.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:16.034 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:40:16.034 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:40:16.034 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:16.035 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:40:16.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.026+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8194053a70 msgr2=0x7f8194055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.026+0000 7f81b94d0700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8194053a70 0x7f8194055f30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f81a401a000 tx=0x7f81a4014000).stop 2022-07-21T01:40:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.026+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 msgr2=0x7f81b4153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.030+0000 7f81b94d0700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 0x7f81b4153df0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f81ac00c000 tx=0x7f81ac0091d0).stop 2022-07-21T01:40:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.030+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 shutdown_connections 2022-07-21T01:40:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.030+0000 7f81b94d0700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81b40b2710 0x7f81b4153df0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.030+0000 7f81b94d0700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b40b13f0 0x7f81b4153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.030+0000 7f81b94d0700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8194053a70 0x7f8194055f30 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.030+0000 7f81b94d0700 1 --2- 172.21.15.3:0/158624725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b40b0470 0x7f81b40ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.030+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 >> 172.21.15.3:0/158624725 conn(0x7f81b4060310 msgr2=0x7f81b40b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:16.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.030+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 shutdown_connections 2022-07-21T01:40:16.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.030+0000 7f81b94d0700 1 -- 172.21.15.3:0/158624725 wait complete. 2022-07-21T01:40:16.048 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:40:16.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:40:16.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3073fff700 1 Processor -- start 2022-07-21T01:40:16.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3073fff700 1 -- start start 2022-07-21T01:40:16.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3073fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f30740619b0 0x7f30740b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3073fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 0x7f3074143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3073fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f30740b5ed0 0x7f30740b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3073fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3074142f60 con 0x7f30740619b0 2022-07-21T01:40:16.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3073fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3074061d90 con 0x7f30740b5ed0 2022-07-21T01:40:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3073fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30740ba5a0 con 0x7f30740b6ee0 2022-07-21T01:40:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3072ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f30740619b0 0x7f30740b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3072ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f30740619b0 0x7f30740b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47576/0 (socket says 172.21.15.3:47576) 2022-07-21T01:40:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3072ffd700 1 -- 172.21.15.3:0/1847661964 learned_addr learned my addr 172.21.15.3:0/1847661964 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f30737fe700 1 --2- 172.21.15.3:0/1847661964 >> v2:172.21.15.143:3300/0 conn(0x7f30740b5ed0 0x7f30740b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f30727fc700 1 --2- 172.21.15.3:0/1847661964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 0x7f3074143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:16.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3072ffd700 1 -- 172.21.15.3:0/1847661964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 msgr2=0x7f3074143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:16.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3072ffd700 1 --2- 172.21.15.3:0/1847661964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 0x7f3074143890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3072ffd700 1 -- 172.21.15.3:0/1847661964 >> v2:172.21.15.143:3300/0 conn(0x7f30740b5ed0 msgr2=0x7f30740b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:16.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3072ffd700 1 --2- 172.21.15.3:0/1847661964 >> v2:172.21.15.143:3300/0 conn(0x7f30740b5ed0 0x7f30740b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3072ffd700 1 -- 172.21.15.3:0/1847661964 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3074143150 con 0x7f30740619b0 2022-07-21T01:40:16.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f30737fe700 1 --2- 172.21.15.3:0/1847661964 >> v2:172.21.15.143:3300/0 conn(0x7f30740b5ed0 0x7f30740b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:16.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3072ffd700 1 --2- 172.21.15.3:0/1847661964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 0x7f30740b48a0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f3064002f70 tx=0x7f3064006eb0).ready entity=mon.0 client_cookie=c042b75c37206a4b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:16.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3071ffb700 1 -- 172.21.15.3:0/1847661964 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f306400d570 con 0x7f30740619b0 2022-07-21T01:40:16.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3071ffb700 1 -- 172.21.15.3:0/1847661964 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064000e90 con 0x7f30740619b0 2022-07-21T01:40:16.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.158+0000 7f3071ffb700 1 -- 172.21.15.3:0/1847661964 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f306400d900 con 0x7f30740619b0 2022-07-21T01:40:16.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 -- 172.21.15.3:0/1847661964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 msgr2=0x7f30740b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:16.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 --2- 172.21.15.3:0/1847661964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 0x7f30740b48a0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f3064002f70 tx=0x7f3064006eb0).stop 2022-07-21T01:40:16.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 -- 172.21.15.3:0/1847661964 shutdown_connections 2022-07-21T01:40:16.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 --2- 172.21.15.3:0/1847661964 >> v2:172.21.15.143:3300/0 conn(0x7f30740b5ed0 0x7f30740b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 --2- 172.21.15.3:0/1847661964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 0x7f3074143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 --2- 172.21.15.3:0/1847661964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 0x7f30740b48a0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 -- 172.21.15.3:0/1847661964 >> 172.21.15.3:0/1847661964 conn(0x7f307406af80 msgr2=0x7f307405fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:16.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 -- 172.21.15.3:0/1847661964 shutdown_connections 2022-07-21T01:40:16.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 -- 172.21.15.3:0/1847661964 wait complete. 2022-07-21T01:40:16.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 Processor -- start 2022-07-21T01:40:16.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.162+0000 7f3073fff700 1 -- start start 2022-07-21T01:40:16.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3073fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 0x7f30740b90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:16.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3073fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30740b5ed0 0x7f3074144b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:16.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3073fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 0x7f3074145070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:16.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3073fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307413be00 con 0x7f30740619b0 2022-07-21T01:40:16.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3073fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30740ad810 con 0x7f30740b5ed0 2022-07-21T01:40:16.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3073fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30740b9cd0 con 0x7f30740b6ee0 2022-07-21T01:40:16.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3072ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 0x7f30740b90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:16.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3072ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 0x7f30740b90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47584/0 (socket says 172.21.15.3:47584) 2022-07-21T01:40:16.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3072ffd700 1 -- 172.21.15.3:0/1116772073 learned_addr learned my addr 172.21.15.3:0/1116772073 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:16.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f30727fc700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30740b5ed0 0x7f3074144b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f30737fe700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 0x7f3074145070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3072ffd700 1 -- 172.21.15.3:0/1116772073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 msgr2=0x7f3074145070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3072ffd700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 0x7f3074145070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3072ffd700 1 -- 172.21.15.3:0/1116772073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30740b5ed0 msgr2=0x7f3074144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3072ffd700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30740b5ed0 0x7f3074144b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3072ffd700 1 -- 172.21.15.3:0/1116772073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306400b1e0 con 0x7f30740619b0 2022-07-21T01:40:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3072ffd700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 0x7f30740b90d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f3064002f70 tx=0x7f3064015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3063fff700 1 -- 172.21.15.3:0/1116772073 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3064011760 con 0x7f30740619b0 2022-07-21T01:40:16.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3063fff700 1 -- 172.21.15.3:0/1116772073 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30640118c0 con 0x7f30740619b0 2022-07-21T01:40:16.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3063fff700 1 -- 172.21.15.3:0/1116772073 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3064011c20 con 0x7f30740619b0 2022-07-21T01:40:16.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30741402e0 con 0x7f30740619b0 2022-07-21T01:40:16.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.166+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307413edc0 con 0x7f30740619b0 2022-07-21T01:40:16.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.170+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3074144980 con 0x7f30740619b0 2022-07-21T01:40:16.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.170+0000 7f3063fff700 1 -- 172.21.15.3:0/1116772073 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3064025030 con 0x7f30740619b0 2022-07-21T01:40:16.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.170+0000 7f3063fff700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3050053b00 0x7f3050055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:16.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.170+0000 7f3063fff700 1 -- 172.21.15.3:0/1116772073 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f306406cde0 con 0x7f30740619b0 2022-07-21T01:40:16.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.170+0000 7f30727fc700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3050053b00 0x7f3050055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:16.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.170+0000 7f30727fc700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3050053b00 0x7f3050055fc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f306800c000 tx=0x7f3068008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:16.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.174+0000 7f3063fff700 1 -- 172.21.15.3:0/1116772073 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3064043dd0 con 0x7f30740619b0 2022-07-21T01:40:16.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.298+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f30741402e0 con 0x7f30740619b0 2022-07-21T01:40:16.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.930+0000 7f3063fff700 1 -- 172.21.15.3:0/1116772073 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v87) v1 ==== 133+0+0 (secure 0 0 0) 0x7f30741402e0 con 0x7f30740619b0 2022-07-21T01:40:16.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3050053b00 msgr2=0x7f3050055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:16.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3050053b00 0x7f3050055fc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f306800c000 tx=0x7f3068008040).stop 2022-07-21T01:40:16.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 msgr2=0x7f30740b90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:16.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 0x7f30740b90d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f3064002f70 tx=0x7f3064015000).stop 2022-07-21T01:40:16.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 shutdown_connections 2022-07-21T01:40:16.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30740b6ee0 0x7f3074145070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3050053b00 0x7f3050055fc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30740b5ed0 0x7f3074144b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 --2- 172.21.15.3:0/1116772073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30740619b0 0x7f30740b90d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:16.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 >> 172.21.15.3:0/1116772073 conn(0x7f307406af80 msgr2=0x7f30741406b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:16.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 shutdown_connections 2022-07-21T01:40:16.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:16.934+0000 7f3073fff700 1 -- 172.21.15.3:0/1116772073 wait complete. 2022-07-21T01:40:16.940 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:40:16.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:40:17.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a8dfd700 1 Processor -- start 2022-07-21T01:40:17.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a8dfd700 1 -- start start 2022-07-21T01:40:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a8dfd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f45a40b2710 0x7f45a414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a8dfd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a8dfd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a40a7250 con 0x7f45a40afd60 2022-07-21T01:40:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a8dfd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a40a7480 con 0x7f45a40b2710 2022-07-21T01:40:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a8dfd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a40a7720 con 0x7f45a40b01a0 2022-07-21T01:40:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f45a40b2710 0x7f45a414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a3fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f45a40afd60 0x7f45a40a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55872/0 (socket says 172.21.15.3:55872) 2022-07-21T01:40:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a3fff700 1 -- 172.21.15.3:0/869629429 learned_addr learned my addr 172.21.15.3:0/869629429 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a2ffd700 1 --2- 172.21.15.3:0/869629429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a27fc700 1 -- 172.21.15.3:0/869629429 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194075862 0 0) 0x7f45a40a7250 con 0x7f45a40afd60 2022-07-21T01:40:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a27fc700 1 -- 172.21.15.3:0/869629429 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4584003260 con 0x7f45a40afd60 2022-07-21T01:40:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a37fe700 1 -- 172.21.15.3:0/869629429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 msgr2=0x7f45a414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a37fe700 1 --2- 172.21.15.3:0/869629429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a37fe700 1 -- 172.21.15.3:0/869629429 >> v1:172.21.15.3:6789/0 conn(0x7f45a40afd60 legacy=0x7f45a40a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a37fe700 1 -- 172.21.15.3:0/869629429 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40a79c0 con 0x7f45a40b2710 2022-07-21T01:40:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.062+0000 7f45a2ffd700 1 --2- 172.21.15.3:0/869629429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a37fe700 1 --2- 172.21.15.3:0/869629429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40b2710 0x7f45a414afd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f459c0032f0 tx=0x7f459c0091d0).ready entity=mon.0 client_cookie=6a97a38d688864fc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:17.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a27fc700 1 -- 172.21.15.3:0/869629429 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f459c00b6c0 con 0x7f45a40b2710 2022-07-21T01:40:17.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a27fc700 1 -- 172.21.15.3:0/869629429 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c00b820 con 0x7f45a40b2710 2022-07-21T01:40:17.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a27fc700 1 -- 172.21.15.3:0/869629429 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f459c00b990 con 0x7f45a40b2710 2022-07-21T01:40:17.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 -- 172.21.15.3:0/869629429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40b2710 msgr2=0x7f45a414afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 --2- 172.21.15.3:0/869629429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40b2710 0x7f45a414afd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f459c0032f0 tx=0x7f459c0091d0).stop 2022-07-21T01:40:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 -- 172.21.15.3:0/869629429 shutdown_connections 2022-07-21T01:40:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 --2- 172.21.15.3:0/869629429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 --2- 172.21.15.3:0/869629429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40b2710 0x7f45a414afd0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 -- 172.21.15.3:0/869629429 >> 172.21.15.3:0/869629429 conn(0x7f45a4060310 msgr2=0x7f45a40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 -- 172.21.15.3:0/869629429 shutdown_connections 2022-07-21T01:40:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 -- 172.21.15.3:0/869629429 wait complete. 2022-07-21T01:40:17.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 Processor -- start 2022-07-21T01:40:17.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.066+0000 7f45a8dfd700 1 -- start start 2022-07-21T01:40:17.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a8dfd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40afd60 0x7f45a40ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a8dfd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a41431e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a8dfd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45a40b2710 0x7f45a4156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a8dfd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a41565d0 con 0x7f45a40afd60 2022-07-21T01:40:17.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a8dfd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a40ae810 con 0x7f45a40b2710 2022-07-21T01:40:17.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a8dfd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a40a93e0 con 0x7f45a40b01a0 2022-07-21T01:40:17.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40afd60 0x7f45a40ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40afd60 0x7f45a40ac2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47596/0 (socket says 172.21.15.3:47596) 2022-07-21T01:40:17.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a37fe700 1 -- 172.21.15.3:0/716775395 learned_addr learned my addr 172.21.15.3:0/716775395 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:17.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a2ffd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a41431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a3fff700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45a40b2710 0x7f45a4156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a37fe700 1 -- 172.21.15.3:0/716775395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 msgr2=0x7f45a41431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a37fe700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a41431e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a37fe700 1 -- 172.21.15.3:0/716775395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45a40b2710 msgr2=0x7f45a4156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a37fe700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45a40b2710 0x7f45a4156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a37fe700 1 -- 172.21.15.3:0/716775395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c009470 con 0x7f45a40afd60 2022-07-21T01:40:17.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a2ffd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a41431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:17.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a37fe700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40afd60 0x7f45a40ac2f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f459c0032f0 tx=0x7f459c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:17.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a0ff9700 1 -- 172.21.15.3:0/716775395 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f459c021480 con 0x7f45a40afd60 2022-07-21T01:40:17.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a0ff9700 1 -- 172.21.15.3:0/716775395 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c0215e0 con 0x7f45a40afd60 2022-07-21T01:40:17.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a40a9880 con 0x7f45a40afd60 2022-07-21T01:40:17.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a0ff9700 1 -- 172.21.15.3:0/716775395 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f459c021940 con 0x7f45a40afd60 2022-07-21T01:40:17.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a40ae590 con 0x7f45a40afd60 2022-07-21T01:40:17.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a0ff9700 1 -- 172.21.15.3:0/716775395 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f459c00f070 con 0x7f45a40afd60 2022-07-21T01:40:17.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.070+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a40aea10 con 0x7f45a40afd60 2022-07-21T01:40:17.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.074+0000 7f45a0ff9700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4588053a70 0x7f4588055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.074+0000 7f45a0ff9700 1 -- 172.21.15.3:0/716775395 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f459c06d2a0 con 0x7f45a40afd60 2022-07-21T01:40:17.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.074+0000 7f45a2ffd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4588053a70 0x7f4588055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.074+0000 7f45a2ffd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4588053a70 0x7f4588055f30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f458c00c000 tx=0x7f458c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:17.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.078+0000 7f45a0ff9700 1 -- 172.21.15.3:0/716775395 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f459c0442b0 con 0x7f45a40afd60 2022-07-21T01:40:17.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.206+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f45a40a95a0 con 0x7f45a40afd60 2022-07-21T01:40:17.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.614+0000 7f45a0ff9700 1 -- 172.21.15.3:0/716775395 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v88) v1 ==== 125+0+0 (secure 0 0 0) 0x7f459c0150a0 con 0x7f45a40afd60 2022-07-21T01:40:17.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4588053a70 msgr2=0x7f4588055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4588053a70 0x7f4588055f30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f458c00c000 tx=0x7f458c008040).stop 2022-07-21T01:40:17.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40afd60 msgr2=0x7f45a40ac2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40afd60 0x7f45a40ac2f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f459c0032f0 tx=0x7f459c016000).stop 2022-07-21T01:40:17.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 shutdown_connections 2022-07-21T01:40:17.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4588053a70 0x7f4588055f30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45a40b2710 0x7f45a4156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45a40b01a0 0x7f45a41431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 --2- 172.21.15.3:0/716775395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45a40afd60 0x7f45a40ac2f0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 >> 172.21.15.3:0/716775395 conn(0x7f45a4060310 msgr2=0x7f45a4148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:17.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 shutdown_connections 2022-07-21T01:40:17.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.622+0000 7f45a8dfd700 1 -- 172.21.15.3:0/716775395 wait complete. 2022-07-21T01:40:17.629 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:40:17.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:40:17.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.738+0000 7ff8c6284700 1 Processor -- start 2022-07-21T01:40:17.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.738+0000 7ff8c6284700 1 -- start start 2022-07-21T01:40:17.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c00614b0 0x7ff8c0061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8c01405d0 0x7ff8c0140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c00aee80 con 0x7ff8c00ad750 2022-07-21T01:40:17.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c00b70b0 con 0x7ff8c01405d0 2022-07-21T01:40:17.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00af0b0 con 0x7ff8c00614b0 2022-07-21T01:40:17.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c5a83700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff8c01405d0 0x7ff8c0140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c4a81700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff8c00ad750 0x7ff8c00b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55884/0 (socket says 172.21.15.3:55884) 2022-07-21T01:40:17.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c4a81700 1 -- 172.21.15.3:0/1956913967 learned_addr learned my addr 172.21.15.3:0/1956913967 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:17.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c5282700 1 --2- 172.21.15.3:0/1956913967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c00614b0 0x7ff8c0061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c5a83700 1 -- 172.21.15.3:0/1956913967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c00614b0 msgr2=0x7ff8c0061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:17.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c5a83700 1 --2- 172.21.15.3:0/1956913967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c00614b0 0x7ff8c0061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c5a83700 1 -- 172.21.15.3:0/1956913967 >> v1:172.21.15.3:6789/0 conn(0x7ff8c00ad750 legacy=0x7ff8c00b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8affff700 1 -- 172.21.15.3:0/1956913967 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578402729 0 0) 0x7ff8c00aee80 con 0x7ff8c00ad750 2022-07-21T01:40:17.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c5a83700 1 -- 172.21.15.3:0/1956913967 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c00af350 con 0x7ff8c01405d0 2022-07-21T01:40:17.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c5a83700 1 --2- 172.21.15.3:0/1956913967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c01405d0 0x7ff8c0140a60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7ff8b80032f0 tx=0x7ff8b80091d0).ready entity=mon.0 client_cookie=68b4e62c5614c05d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:17.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8affff700 1 -- 172.21.15.3:0/1956913967 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8b8009ea0 con 0x7ff8c01405d0 2022-07-21T01:40:17.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8affff700 1 -- 172.21.15.3:0/1956913967 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b800b8a0 con 0x7ff8c01405d0 2022-07-21T01:40:17.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8affff700 1 -- 172.21.15.3:0/1956913967 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8b800ba10 con 0x7ff8c01405d0 2022-07-21T01:40:17.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 -- 172.21.15.3:0/1956913967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c01405d0 msgr2=0x7ff8c0140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 --2- 172.21.15.3:0/1956913967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c01405d0 0x7ff8c0140a60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7ff8b80032f0 tx=0x7ff8b80091d0).stop 2022-07-21T01:40:17.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 -- 172.21.15.3:0/1956913967 shutdown_connections 2022-07-21T01:40:17.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 --2- 172.21.15.3:0/1956913967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c01405d0 0x7ff8c0140a60 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 --2- 172.21.15.3:0/1956913967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c00614b0 0x7ff8c0061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 -- 172.21.15.3:0/1956913967 >> 172.21.15.3:0/1956913967 conn(0x7ff8c0066f70 msgr2=0x7ff8c0067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:17.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 -- 172.21.15.3:0/1956913967 shutdown_connections 2022-07-21T01:40:17.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.742+0000 7ff8c6284700 1 -- 172.21.15.3:0/1956913967 wait complete. 2022-07-21T01:40:17.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c6284700 1 Processor -- start 2022-07-21T01:40:17.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c6284700 1 -- start start 2022-07-21T01:40:17.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c6284700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c00614b0 0x7ff8c0141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c6284700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c00ad750 0x7ff8c01563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c6284700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c01405d0 0x7ff8c0152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c6284700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00b0cb0 con 0x7ff8c00ad750 2022-07-21T01:40:17.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c6284700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00b0ee0 con 0x7ff8c00614b0 2022-07-21T01:40:17.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c6284700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00b1180 con 0x7ff8c01405d0 2022-07-21T01:40:17.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c4a81700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c00ad750 0x7ff8c01563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c4a81700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c00ad750 0x7ff8c01563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47612/0 (socket says 172.21.15.3:47612) 2022-07-21T01:40:17.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c4a81700 1 -- 172.21.15.3:0/2596126898 learned_addr learned my addr 172.21.15.3:0/2596126898 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:17.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c5a83700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c01405d0 0x7ff8c0152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c5282700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c00614b0 0x7ff8c0141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c4a81700 1 -- 172.21.15.3:0/2596126898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c01405d0 msgr2=0x7ff8c0152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c4a81700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c01405d0 0x7ff8c0152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c4a81700 1 -- 172.21.15.3:0/2596126898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c00614b0 msgr2=0x7ff8c0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c4a81700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c00614b0 0x7ff8c0141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c4a81700 1 -- 172.21.15.3:0/2596126898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b8009470 con 0x7ff8c00ad750 2022-07-21T01:40:17.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.746+0000 7ff8c4a81700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c00ad750 0x7ff8c01563a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7ff8b0000ed0 tx=0x7ff8b0007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:17.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8ae7fc700 1 -- 172.21.15.3:0/2596126898 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8b00070d0 con 0x7ff8c00ad750 2022-07-21T01:40:17.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8ae7fc700 1 -- 172.21.15.3:0/2596126898 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b0007230 con 0x7ff8c00ad750 2022-07-21T01:40:17.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c00b1420 con 0x7ff8c00ad750 2022-07-21T01:40:17.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8ae7fc700 1 -- 172.21.15.3:0/2596126898 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8b0005610 con 0x7ff8c00ad750 2022-07-21T01:40:17.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c00aac10 con 0x7ff8c00ad750 2022-07-21T01:40:17.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c00b15b0 con 0x7ff8c00ad750 2022-07-21T01:40:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8ae7fc700 1 -- 172.21.15.3:0/2596126898 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff8b001a060 con 0x7ff8c00ad750 2022-07-21T01:40:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8ae7fc700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff8a0053a70 0x7ff8a0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8ae7fc700 1 -- 172.21.15.3:0/2596126898 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6304+0+0 (secure 0 0 0) 0x7ff8b006b720 con 0x7ff8c00ad750 2022-07-21T01:40:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8c5282700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff8a0053a70 0x7ff8a0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:17.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.750+0000 7ff8c5282700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff8a0053a70 0x7ff8a0055f30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7ff8bc00c000 tx=0x7ff8bc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:17.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.758+0000 7ff8ae7fc700 1 -- 172.21.15.3:0/2596126898 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff8b0042560 con 0x7ff8c00ad750 2022-07-21T01:40:17.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.922+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7ff8c00af5f0 con 0x7ff8c00ad750 2022-07-21T01:40:17.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.942+0000 7ff8ae7fc700 1 -- 172.21.15.3:0/2596126898 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 15 and data pool 16 v40) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff8b0018020 con 0x7ff8c00ad750 2022-07-21T01:40:17.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff8a0053a70 msgr2=0x7ff8a0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff8a0053a70 0x7ff8a0055f30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7ff8bc00c000 tx=0x7ff8bc008040).stop 2022-07-21T01:40:17.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c00ad750 msgr2=0x7ff8c01563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:17.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c00ad750 0x7ff8c01563a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7ff8b0000ed0 tx=0x7ff8b0007ac0).stop 2022-07-21T01:40:17.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 shutdown_connections 2022-07-21T01:40:17.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c01405d0 0x7ff8c0152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8c00ad750 0x7ff8c01563a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff8a0053a70 0x7ff8a0055f30 secure :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0x7ff8bc00c000 tx=0x7ff8bc008040).stop 2022-07-21T01:40:17.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 --2- 172.21.15.3:0/2596126898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c00614b0 0x7ff8c0141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:17.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 >> 172.21.15.3:0/2596126898 conn(0x7ff8c0066f70 msgr2=0x7ff8c00b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:17.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 shutdown_connections 2022-07-21T01:40:17.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:17.950+0000 7ff8c6284700 1 -- 172.21.15.3:0/2596126898 wait complete. 2022-07-21T01:40:17.961 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 15 and data pool 16 2022-07-21T01:40:17.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:18.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.062+0000 7f92f244f700 1 Processor -- start 2022-07-21T01:40:18.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.062+0000 7f92f244f700 1 -- start start 2022-07-21T01:40:18.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f244f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f92ec0af450 0x7f92ec0af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f244f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ec0aff30 con 0x7f92ec0ad9e0 2022-07-21T01:40:18.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f244f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ec05c7a0 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f244f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92ec140990 con 0x7f92ec0af450 2022-07-21T01:40:18.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f1c4e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f92ec0ad9e0 0x7f92ec0a8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55912/0 (socket says 172.21.15.3:55912) 2022-07-21T01:40:18.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f1c4e700 1 -- 172.21.15.3:0/1548842243 learned_addr learned my addr 172.21.15.3:0/1548842243 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:18.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f144d700 1 --2- 172.21.15.3:0/1548842243 >> v2:172.21.15.143:3300/0 conn(0x7f92ec0af450 0x7f92ec0af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92dffff700 1 -- 172.21.15.3:0/1548842243 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830244464 0 0) 0x7f92ec0aff30 con 0x7f92ec0ad9e0 2022-07-21T01:40:18.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92dffff700 1 -- 172.21.15.3:0/1548842243 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d4003260 con 0x7f92ec0ad9e0 2022-07-21T01:40:18.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f144d700 1 -- 172.21.15.3:0/1548842243 >> v1:172.21.15.143:6789/0 conn(0x7f92ec0ad4b0 legacy=0x7f92ec1432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f144d700 1 -- 172.21.15.3:0/1548842243 >> v1:172.21.15.3:6789/0 conn(0x7f92ec0ad9e0 legacy=0x7f92ec0a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f144d700 1 -- 172.21.15.3:0/1548842243 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec1439f0 con 0x7f92ec0af450 2022-07-21T01:40:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92f144d700 1 --2- 172.21.15.3:0/1548842243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92ec0af450 0x7f92ec0af830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f92ec0ad900 tx=0x7f92e0006df0).ready entity=mon.1 client_cookie=beed6daf2fc55444 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92dffff700 1 -- 172.21.15.3:0/1548842243 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f92e000d610 con 0x7f92ec0af450 2022-07-21T01:40:18.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92dffff700 1 -- 172.21.15.3:0/1548842243 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e0000e90 con 0x7f92ec0af450 2022-07-21T01:40:18.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.066+0000 7f92dffff700 1 -- 172.21.15.3:0/1548842243 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f92e000d960 con 0x7f92ec0af450 2022-07-21T01:40:18.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.070+0000 7f92f244f700 1 -- 172.21.15.3:0/1548842243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92ec0af450 msgr2=0x7f92ec0af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.070+0000 7f92f244f700 1 --2- 172.21.15.3:0/1548842243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92ec0af450 0x7f92ec0af830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f92ec0ad900 tx=0x7f92e0006df0).stop 2022-07-21T01:40:18.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.070+0000 7f92f244f700 1 -- 172.21.15.3:0/1548842243 shutdown_connections 2022-07-21T01:40:18.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.070+0000 7f92f244f700 1 --2- 172.21.15.3:0/1548842243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92ec0af450 0x7f92ec0af830 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.070+0000 7f92f244f700 1 -- 172.21.15.3:0/1548842243 >> 172.21.15.3:0/1548842243 conn(0x7f92ec0612d0 msgr2=0x7f92ec062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:18.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.070+0000 7f92f244f700 1 -- 172.21.15.3:0/1548842243 shutdown_connections 2022-07-21T01:40:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.070+0000 7f92f244f700 1 -- 172.21.15.3:0/1548842243 wait complete. 2022-07-21T01:40:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.070+0000 7f92f244f700 1 Processor -- start 2022-07-21T01:40:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.070+0000 7f92f244f700 1 -- start start 2022-07-21T01:40:18.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f244f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92ec0ad4b0 0x7f92ec0b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f244f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92ec0ad9e0 0x7f92ec0b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f244f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92ec0af450 0x7f92ec150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f244f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec139440 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f244f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec138530 con 0x7f92ec0af450 2022-07-21T01:40:18.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f244f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec13a120 con 0x7f92ec0ad9e0 2022-07-21T01:40:18.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f144d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92ec0ad4b0 0x7f92ec0b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f144d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92ec0ad4b0 0x7f92ec0b6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47634/0 (socket says 172.21.15.3:47634) 2022-07-21T01:40:18.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f144d700 1 -- 172.21.15.3:0/3687687088 learned_addr learned my addr 172.21.15.3:0/3687687088 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:18.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f0c4c700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92ec0ad9e0 0x7f92ec0b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f1c4e700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92ec0af450 0x7f92ec150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f144d700 1 -- 172.21.15.3:0/3687687088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92ec0ad9e0 msgr2=0x7f92ec0b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f144d700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92ec0ad9e0 0x7f92ec0b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f144d700 1 -- 172.21.15.3:0/3687687088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92ec0af450 msgr2=0x7f92ec150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f144d700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92ec0af450 0x7f92ec150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f144d700 1 -- 172.21.15.3:0/3687687088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e000b180 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f144d700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92ec0ad4b0 0x7f92ec0b6050 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f92ec0ad900 tx=0x7f92e0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92de7fc700 1 -- 172.21.15.3:0/3687687088 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f92e0016070 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92de7fc700 1 -- 172.21.15.3:0/3687687088 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e000de90 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ec139670 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92de7fc700 1 -- 172.21.15.3:0/3687687088 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f92e0013da0 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.074+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ec0b6540 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.078+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ec0b4930 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.078+0000 7f92de7fc700 1 -- 172.21.15.3:0/3687687088 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f92e0025030 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.078+0000 7f92de7fc700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f92cc053b00 0x7f92cc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.078+0000 7f92f0c4c700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f92cc053b00 0x7f92cc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.078+0000 7f92de7fc700 1 -- 172.21.15.3:0/3687687088 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f92e006cff0 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.078+0000 7f92f0c4c700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f92cc053b00 0x7f92cc055fc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f92e800d000 tx=0x7f92e8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:18.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.086+0000 7f92de7fc700 1 -- 172.21.15.3:0/3687687088 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f92e0043ca0 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.214+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92ec0574e0 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.214+0000 7f92de7fc700 1 -- 172.21.15.3:0/3687687088 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16287 (secure 0 0 0) 0x7f92e0023020 con 0x7f92ec0ad4b0 2022-07-21T01:40:18.219 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:18.220 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":89,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:17.940363+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:16.360095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:17.262905+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:18.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f92cc053b00 msgr2=0x7f92cc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f92cc053b00 0x7f92cc055fc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f92e800d000 tx=0x7f92e8007b90).stop 2022-07-21T01:40:18.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92ec0ad4b0 msgr2=0x7f92ec0b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92ec0ad4b0 0x7f92ec0b6050 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f92ec0ad900 tx=0x7f92e0014000).stop 2022-07-21T01:40:18.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 shutdown_connections 2022-07-21T01:40:18.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f92cc053b00 0x7f92cc055fc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92ec0af450 0x7f92ec150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92ec0ad9e0 0x7f92ec0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 --2- 172.21.15.3:0/3687687088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92ec0ad4b0 0x7f92ec0b6050 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 >> 172.21.15.3:0/3687687088 conn(0x7f92ec0612d0 msgr2=0x7f92ec0b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:18.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 shutdown_connections 2022-07-21T01:40:18.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.218+0000 7f92f244f700 1 -- 172.21.15.3:0/3687687088 wait complete. 2022-07-21T01:40:18.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:18.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f6b579700 1 Processor -- start 2022-07-21T01:40:18.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f6b579700 1 -- start start 2022-07-21T01:40:18.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f6b579700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 0x7f5f640b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f6b579700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5f64067490 0x7f5f64143a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f6b579700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5f640b4dd0 0x7f5f64143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f6b579700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f640aad40 con 0x7f5f64067490 2022-07-21T01:40:18.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f6b579700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f64060670 con 0x7f5f640b4dd0 2022-07-21T01:40:18.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f6b579700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f64067030 con 0x7f5f64060290 2022-07-21T01:40:18.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f68b14700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5f64067490 0x7f5f64143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f68b14700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5f64067490 0x7f5f64143a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47644/0 (socket says 172.21.15.3:47644) 2022-07-21T01:40:18.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f68b14700 1 -- 172.21.15.3:0/1292316665 learned_addr learned my addr 172.21.15.3:0/1292316665 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:18.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f69315700 1 --2- 172.21.15.3:0/1292316665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 0x7f5f640b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f69b16700 1 --2- 172.21.15.3:0/1292316665 >> v2:172.21.15.143:3300/0 conn(0x7f5f640b4dd0 0x7f5f64143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f68b14700 1 -- 172.21.15.3:0/1292316665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 msgr2=0x7f5f640b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:18.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f68b14700 1 --2- 172.21.15.3:0/1292316665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 0x7f5f640b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f68b14700 1 -- 172.21.15.3:0/1292316665 >> v2:172.21.15.143:3300/0 conn(0x7f5f640b4dd0 msgr2=0x7f5f64143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:18.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f68b14700 1 --2- 172.21.15.3:0/1292316665 >> v2:172.21.15.143:3300/0 conn(0x7f5f640b4dd0 0x7f5f64143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f68b14700 1 -- 172.21.15.3:0/1292316665 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f64144460 con 0x7f5f64067490 2022-07-21T01:40:18.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f68b14700 1 --2- 172.21.15.3:0/1292316665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 0x7f5f64143a80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7f5f4c002f70 tx=0x7f5f4c006eb0).ready entity=mon.0 client_cookie=ae7c253076ed51b0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:18.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f63fff700 1 -- 172.21.15.3:0/1292316665 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4c00d570 con 0x7f5f64067490 2022-07-21T01:40:18.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f63fff700 1 -- 172.21.15.3:0/1292316665 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c000e90 con 0x7f5f64067490 2022-07-21T01:40:18.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.334+0000 7f5f63fff700 1 -- 172.21.15.3:0/1292316665 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4c00d900 con 0x7f5f64067490 2022-07-21T01:40:18.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 -- 172.21.15.3:0/1292316665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 msgr2=0x7f5f64143a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 --2- 172.21.15.3:0/1292316665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 0x7f5f64143a80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7f5f4c002f70 tx=0x7f5f4c006eb0).stop 2022-07-21T01:40:18.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 -- 172.21.15.3:0/1292316665 shutdown_connections 2022-07-21T01:40:18.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 --2- 172.21.15.3:0/1292316665 >> v2:172.21.15.143:3300/0 conn(0x7f5f640b4dd0 0x7f5f64143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 --2- 172.21.15.3:0/1292316665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 0x7f5f64143a80 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 --2- 172.21.15.3:0/1292316665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 0x7f5f640b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 -- 172.21.15.3:0/1292316665 >> 172.21.15.3:0/1292316665 conn(0x7f5f6405d0c0 msgr2=0x7f5f6405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:18.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 -- 172.21.15.3:0/1292316665 shutdown_connections 2022-07-21T01:40:18.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 -- 172.21.15.3:0/1292316665 wait complete. 2022-07-21T01:40:18.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 Processor -- start 2022-07-21T01:40:18.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 -- start start 2022-07-21T01:40:18.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 0x7f5f641474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 0x7f5f641419f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f640b4dd0 0x7f5f64141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640aca50 con 0x7f5f64067490 2022-07-21T01:40:18.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640ad1d0 con 0x7f5f640b4dd0 2022-07-21T01:40:18.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.338+0000 7f5f6b579700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640acca0 con 0x7f5f64060290 2022-07-21T01:40:18.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f68b14700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 0x7f5f641419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f68b14700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 0x7f5f641419f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47650/0 (socket says 172.21.15.3:47650) 2022-07-21T01:40:18.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f68b14700 1 -- 172.21.15.3:0/3653419417 learned_addr learned my addr 172.21.15.3:0/3653419417 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:18.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f69b16700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f640b4dd0 0x7f5f64141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f69315700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 0x7f5f641474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f68b14700 1 -- 172.21.15.3:0/3653419417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 msgr2=0x7f5f641474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f68b14700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 0x7f5f641474a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f68b14700 1 -- 172.21.15.3:0/3653419417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f640b4dd0 msgr2=0x7f5f64141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f68b14700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f640b4dd0 0x7f5f64141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f68b14700 1 -- 172.21.15.3:0/3653419417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c00b1e0 con 0x7f5f64067490 2022-07-21T01:40:18.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f68b14700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 0x7f5f641419f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f5f4c002f70 tx=0x7f5f4c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:18.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f627fc700 1 -- 172.21.15.3:0/3653419417 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4c0117e0 con 0x7f5f64067490 2022-07-21T01:40:18.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f627fc700 1 -- 172.21.15.3:0/3653419417 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c011940 con 0x7f5f64067490 2022-07-21T01:40:18.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f640ad430 con 0x7f5f64067490 2022-07-21T01:40:18.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f627fc700 1 -- 172.21.15.3:0/3653419417 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4c011ca0 con 0x7f5f64067490 2022-07-21T01:40:18.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f641423d0 con 0x7f5f64067490 2022-07-21T01:40:18.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f627fc700 1 -- 172.21.15.3:0/3653419417 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f4c025030 con 0x7f5f64067490 2022-07-21T01:40:18.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f640574e0 con 0x7f5f64067490 2022-07-21T01:40:18.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f627fc700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f48053a70 0x7f5f48055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.342+0000 7f5f627fc700 1 -- 172.21.15.3:0/3653419417 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5f4c06d1a0 con 0x7f5f64067490 2022-07-21T01:40:18.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.346+0000 7f5f69315700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f48053a70 0x7f5f48055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.346+0000 7f5f69315700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f48053a70 0x7f5f48055f30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f5f5c00b000 tx=0x7f5f5c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:18.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.350+0000 7f5f627fc700 1 -- 172.21.15.3:0/3653419417 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f4c043ec0 con 0x7f5f64067490 2022-07-21T01:40:18.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.478+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f640ad7b0 con 0x7f5f64067490 2022-07-21T01:40:18.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f627fc700 1 -- 172.21.15.3:0/3653419417 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16287 (secure 0 0 0) 0x7f5f4c023090 con 0x7f5f64067490 2022-07-21T01:40:18.485 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:18.486 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":89,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:17.940363+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:16.360095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:17.262905+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:18.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f48053a70 msgr2=0x7f5f48055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f48053a70 0x7f5f48055f30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f5f5c00b000 tx=0x7f5f5c008040).stop 2022-07-21T01:40:18.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 msgr2=0x7f5f641419f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 0x7f5f641419f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f5f4c002f70 tx=0x7f5f4c015000).stop 2022-07-21T01:40:18.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 shutdown_connections 2022-07-21T01:40:18.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f48053a70 0x7f5f48055f30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f640b4dd0 0x7f5f64141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f64067490 0x7f5f641419f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 --2- 172.21.15.3:0/3653419417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f64060290 0x7f5f641474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 >> 172.21.15.3:0/3653419417 conn(0x7f5f6405d0c0 msgr2=0x7f5f64144630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:18.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 shutdown_connections 2022-07-21T01:40:18.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.482+0000 7f5f6b579700 1 -- 172.21.15.3:0/3653419417 wait complete. 2022-07-21T01:40:18.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:40:18.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.610+0000 7fc233fff700 1 Processor -- start 2022-07-21T01:40:18.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.610+0000 7fc233fff700 1 -- start start 2022-07-21T01:40:18.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 0x7fc23413ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc2340b2050 0x7fc234140480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2340afa50 con 0x7fc2340ae3c0 2022-07-21T01:40:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2340b03c0 con 0x7fc2340b2050 2022-07-21T01:40:18.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23405ff60 con 0x7fc23405fb80 2022-07-21T01:40:18.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc22bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc2340b2050 0x7fc234140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc22bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc2340b2050 0x7fc234140480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47658/0 (socket says 172.21.15.3:47658) 2022-07-21T01:40:18.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc22bfff700 1 -- 172.21.15.3:0/4197467898 learned_addr learned my addr 172.21.15.3:0/4197467898 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:18.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc232ffd700 1 --2- 172.21.15.3:0/4197467898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 0x7fc23413ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc22bfff700 1 -- 172.21.15.3:0/4197467898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 msgr2=0x7fc23413ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc22bfff700 1 --2- 172.21.15.3:0/4197467898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 0x7fc23413ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc22bfff700 1 -- 172.21.15.3:0/4197467898 >> v1:172.21.15.143:6789/0 conn(0x7fc2340ae3c0 legacy=0x7fc2341434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc22bfff700 1 -- 172.21.15.3:0/4197467898 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340ad1c0 con 0x7fc2340b2050 2022-07-21T01:40:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc232ffd700 1 --2- 172.21.15.3:0/4197467898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 0x7fc23413ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:18.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc22bfff700 1 --2- 172.21.15.3:0/4197467898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340b2050 0x7fc234140480 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fc21c002f70 tx=0x7fc21c006eb0).ready entity=mon.0 client_cookie=c7d31daacb38303e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:18.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc2327fc700 1 -- 172.21.15.3:0/4197467898 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc21c00d5c0 con 0x7fc2340b2050 2022-07-21T01:40:18.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc2327fc700 1 -- 172.21.15.3:0/4197467898 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c000e90 con 0x7fc2340b2050 2022-07-21T01:40:18.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc2327fc700 1 -- 172.21.15.3:0/4197467898 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc21c00d950 con 0x7fc2340b2050 2022-07-21T01:40:18.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 -- 172.21.15.3:0/4197467898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340b2050 msgr2=0x7fc234140480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 --2- 172.21.15.3:0/4197467898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340b2050 0x7fc234140480 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fc21c002f70 tx=0x7fc21c006eb0).stop 2022-07-21T01:40:18.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 -- 172.21.15.3:0/4197467898 shutdown_connections 2022-07-21T01:40:18.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 --2- 172.21.15.3:0/4197467898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340b2050 0x7fc234140480 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 --2- 172.21.15.3:0/4197467898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 0x7fc23413ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.614+0000 7fc233fff700 1 -- 172.21.15.3:0/4197467898 >> 172.21.15.3:0/4197467898 conn(0x7fc2340613c0 msgr2=0x7fc234062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:18.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 -- 172.21.15.3:0/4197467898 shutdown_connections 2022-07-21T01:40:18.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 -- 172.21.15.3:0/4197467898 wait complete. 2022-07-21T01:40:18.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 Processor -- start 2022-07-21T01:40:18.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 -- start start 2022-07-21T01:40:18.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 0x7fc234141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340ae3c0 0x7fc234137e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340b2050 0x7fc234138320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23414fc10 con 0x7fc2340ae3c0 2022-07-21T01:40:18.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc234143f70 con 0x7fc2340b2050 2022-07-21T01:40:18.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc233fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23414fe40 con 0x7fc23405fb80 2022-07-21T01:40:18.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc22bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340ae3c0 0x7fc234137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc22bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340ae3c0 0x7fc234137e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47664/0 (socket says 172.21.15.3:47664) 2022-07-21T01:40:18.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc22bfff700 1 -- 172.21.15.3:0/964601517 learned_addr learned my addr 172.21.15.3:0/964601517 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:18.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc232ffd700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 0x7fc234141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc2337fe700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340b2050 0x7fc234138320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc22bfff700 1 -- 172.21.15.3:0/964601517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 msgr2=0x7fc234141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc22bfff700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 0x7fc234141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc22bfff700 1 -- 172.21.15.3:0/964601517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340b2050 msgr2=0x7fc234138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc22bfff700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340b2050 0x7fc234138320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc22bfff700 1 -- 172.21.15.3:0/964601517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c00b1e0 con 0x7fc2340ae3c0 2022-07-21T01:40:18.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc2337fe700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340b2050 0x7fc234138320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:40:18.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.618+0000 7fc22bfff700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340ae3c0 0x7fc234137e30 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fc21c002f70 tx=0x7fc21c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:18.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.622+0000 7fc230ff9700 1 -- 172.21.15.3:0/964601517 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc21c011950 con 0x7fc2340ae3c0 2022-07-21T01:40:18.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.622+0000 7fc230ff9700 1 -- 172.21.15.3:0/964601517 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c011ab0 con 0x7fc2340ae3c0 2022-07-21T01:40:18.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.622+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23414cd50 con 0x7fc2340ae3c0 2022-07-21T01:40:18.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.622+0000 7fc230ff9700 1 -- 172.21.15.3:0/964601517 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc21c011e10 con 0x7fc2340ae3c0 2022-07-21T01:40:18.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.622+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc234144210 con 0x7fc2340ae3c0 2022-07-21T01:40:18.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.626+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc234150040 con 0x7fc2340ae3c0 2022-07-21T01:40:18.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.626+0000 7fc230ff9700 1 -- 172.21.15.3:0/964601517 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc21c025030 con 0x7fc2340ae3c0 2022-07-21T01:40:18.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.626+0000 7fc230ff9700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc218053a10 0x7fc218055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:18.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.626+0000 7fc230ff9700 1 -- 172.21.15.3:0/964601517 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc21c06d1f0 con 0x7fc2340ae3c0 2022-07-21T01:40:18.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.626+0000 7fc232ffd700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc218053a10 0x7fc218055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:18.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.626+0000 7fc232ffd700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc218053a10 0x7fc218055ed0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fc22400c000 tx=0x7fc224008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:18.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.630+0000 7fc230ff9700 1 -- 172.21.15.3:0/964601517 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc21c043f50 con 0x7fc2340ae3c0 2022-07-21T01:40:18.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.794+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc2340afc80 con 0x7fc2340ae3c0 2022-07-21T01:40:18.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.946+0000 7fc230ff9700 1 -- 172.21.15.3:0/964601517 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc21c023020 con 0x7fc2340ae3c0 2022-07-21T01:40:18.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc218053a10 msgr2=0x7fc218055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc218053a10 0x7fc218055ed0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fc22400c000 tx=0x7fc224008040).stop 2022-07-21T01:40:18.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340ae3c0 msgr2=0x7fc234137e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:18.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340ae3c0 0x7fc234137e30 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fc21c002f70 tx=0x7fc21c015000).stop 2022-07-21T01:40:18.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 shutdown_connections 2022-07-21T01:40:18.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc218053a10 0x7fc218055ed0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340b2050 0x7fc234138320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2340ae3c0 0x7fc234137e30 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 --2- 172.21.15.3:0/964601517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23405fb80 0x7fc234141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:18.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 >> 172.21.15.3:0/964601517 conn(0x7fc2340613c0 msgr2=0x7fc2340b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:18.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 shutdown_connections 2022-07-21T01:40:18.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:18.950+0000 7fc233fff700 1 -- 172.21.15.3:0/964601517 wait complete. 2022-07-21T01:40:18.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:19.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba68f5700 1 Processor -- start 2022-07-21T01:40:19.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba68f5700 1 -- start start 2022-07-21T01:40:19.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba68f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00b2710 0x7fdba014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba68f5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdba00b01a0 0x7fdba014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba68f5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdba00afd60 0x7fdba00a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba68f5700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba00a7250 con 0x7fdba00afd60 2022-07-21T01:40:19.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba68f5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba00a7480 con 0x7fdba00b01a0 2022-07-21T01:40:19.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba68f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00a7720 con 0x7fdba00b2710 2022-07-21T01:40:19.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba60f4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdba00afd60 0x7fdba00a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba60f4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdba00afd60 0x7fdba00a8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47674/0 (socket says 172.21.15.3:47674) 2022-07-21T01:40:19.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba60f4700 1 -- 172.21.15.3:0/149220485 learned_addr learned my addr 172.21.15.3:0/149220485 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:19.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba50f2700 1 --2- 172.21.15.3:0/149220485 >> v2:172.21.15.143:3300/0 conn(0x7fdba00b01a0 0x7fdba014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba58f3700 1 --2- 172.21.15.3:0/149220485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00b2710 0x7fdba014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba60f4700 1 -- 172.21.15.3:0/149220485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00b2710 msgr2=0x7fdba014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:19.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba60f4700 1 --2- 172.21.15.3:0/149220485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00b2710 0x7fdba014afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba60f4700 1 -- 172.21.15.3:0/149220485 >> v2:172.21.15.143:3300/0 conn(0x7fdba00b01a0 msgr2=0x7fdba014b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:19.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba60f4700 1 --2- 172.21.15.3:0/149220485 >> v2:172.21.15.143:3300/0 conn(0x7fdba00b01a0 0x7fdba014b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba60f4700 1 -- 172.21.15.3:0/149220485 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba00a79c0 con 0x7fdba00afd60 2022-07-21T01:40:19.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba58f3700 1 --2- 172.21.15.3:0/149220485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00b2710 0x7fdba014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:19.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba60f4700 1 --2- 172.21.15.3:0/149220485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00afd60 0x7fdba00a8540 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7fdb98002f70 tx=0x7fdb98006eb0).ready entity=mon.0 client_cookie=5c35dd10b7b2ff19 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba48f1700 1 -- 172.21.15.3:0/149220485 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9800d570 con 0x7fdba00afd60 2022-07-21T01:40:19.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba48f1700 1 -- 172.21.15.3:0/149220485 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb98000e90 con 0x7fdba00afd60 2022-07-21T01:40:19.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.074+0000 7fdba48f1700 1 -- 172.21.15.3:0/149220485 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9800d900 con 0x7fdba00afd60 2022-07-21T01:40:19.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 -- 172.21.15.3:0/149220485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00afd60 msgr2=0x7fdba00a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 --2- 172.21.15.3:0/149220485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00afd60 0x7fdba00a8540 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7fdb98002f70 tx=0x7fdb98006eb0).stop 2022-07-21T01:40:19.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 -- 172.21.15.3:0/149220485 shutdown_connections 2022-07-21T01:40:19.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 --2- 172.21.15.3:0/149220485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00afd60 0x7fdba00a8540 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 --2- 172.21.15.3:0/149220485 >> v2:172.21.15.143:3300/0 conn(0x7fdba00b01a0 0x7fdba014b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 --2- 172.21.15.3:0/149220485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00b2710 0x7fdba014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 -- 172.21.15.3:0/149220485 >> 172.21.15.3:0/149220485 conn(0x7fdba0060310 msgr2=0x7fdba00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:19.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 -- 172.21.15.3:0/149220485 shutdown_connections 2022-07-21T01:40:19.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 -- 172.21.15.3:0/149220485 wait complete. 2022-07-21T01:40:19.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 Processor -- start 2022-07-21T01:40:19.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 -- start start 2022-07-21T01:40:19.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.078+0000 7fdba68f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00afd60 0x7fdba00a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba68f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdba00b01a0 0x7fdba0156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba68f5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00b2710 0x7fdba0156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba68f5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba0156ef0 con 0x7fdba00b2710 2022-07-21T01:40:19.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba68f5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba015a710 con 0x7fdba00b01a0 2022-07-21T01:40:19.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba68f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00aecc0 con 0x7fdba00afd60 2022-07-21T01:40:19.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba60f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00b2710 0x7fdba0156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba60f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00b2710 0x7fdba0156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47680/0 (socket says 172.21.15.3:47680) 2022-07-21T01:40:19.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba60f4700 1 -- 172.21.15.3:0/4012567866 learned_addr learned my addr 172.21.15.3:0/4012567866 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:19.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba50f2700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdba00b01a0 0x7fdba0156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba58f3700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00afd60 0x7fdba00a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba60f4700 1 -- 172.21.15.3:0/4012567866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00afd60 msgr2=0x7fdba00a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba60f4700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00afd60 0x7fdba00a95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba60f4700 1 -- 172.21.15.3:0/4012567866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdba00b01a0 msgr2=0x7fdba0156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba60f4700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdba00b01a0 0x7fdba0156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba60f4700 1 -- 172.21.15.3:0/4012567866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9800b1e0 con 0x7fdba00b2710 2022-07-21T01:40:19.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba60f4700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00b2710 0x7fdba0156a00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7fdb98002f70 tx=0x7fdb98015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdb8effd700 1 -- 172.21.15.3:0/4012567866 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb98006be0 con 0x7fdba00b2710 2022-07-21T01:40:19.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdb8effd700 1 -- 172.21.15.3:0/4012567866 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9800de20 con 0x7fdba00b2710 2022-07-21T01:40:19.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba00aef60 con 0x7fdba00b2710 2022-07-21T01:40:19.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdb8effd700 1 -- 172.21.15.3:0/4012567866 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb98011d30 con 0x7fdba00b2710 2022-07-21T01:40:19.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba015a430 con 0x7fdba00b2710 2022-07-21T01:40:19.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.082+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba015a910 con 0x7fdba00b2710 2022-07-21T01:40:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.086+0000 7fdb8effd700 1 -- 172.21.15.3:0/4012567866 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdb98025030 con 0x7fdba00b2710 2022-07-21T01:40:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.086+0000 7fdb8effd700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb80053a70 0x7fdb80055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.086+0000 7fdb8effd700 1 -- 172.21.15.3:0/4012567866 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdb9806d110 con 0x7fdba00b2710 2022-07-21T01:40:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.086+0000 7fdba58f3700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb80053a70 0x7fdb80055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.086+0000 7fdba58f3700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb80053a70 0x7fdb80055f30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fdb9000c000 tx=0x7fdb90008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.090+0000 7fdb8effd700 1 -- 172.21.15.3:0/4012567866 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb98043ed0 con 0x7fdba00b2710 2022-07-21T01:40:19.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.258+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdba00af0f0 con 0x7fdba00b2710 2022-07-21T01:40:19.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.258+0000 7fdb8effd700 1 -- 172.21.15.3:0/4012567866 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2487 (secure 0 0 0) 0x7fdb98023090 con 0x7fdba00b2710 2022-07-21T01:40:19.263 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:19.263 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":41,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/2659690520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2659690520},{"type":"v1","addr":"172.21.15.143:6833","nonce":2659690520}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":6600,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2193393317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2193393317},{"type":"v1","addr":"172.21.15.3:6835","nonce":2193393317}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:17.940781+0000","modified":"2022-07-21T01:40:18.943071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6603},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6603":{"gid":6603,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/539587272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":539587272},{"type":"v1","addr":"172.21.15.3:6837","nonce":539587272}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-21T01:40:19.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb80053a70 msgr2=0x7fdb80055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb80053a70 0x7fdb80055f30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fdb9000c000 tx=0x7fdb90008040).stop 2022-07-21T01:40:19.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00b2710 msgr2=0x7fdba0156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00b2710 0x7fdba0156a00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7fdb98002f70 tx=0x7fdb98015000).stop 2022-07-21T01:40:19.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 shutdown_connections 2022-07-21T01:40:19.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdba00b2710 0x7fdba0156a00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb80053a70 0x7fdb80055f30 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdba00b01a0 0x7fdba0156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 --2- 172.21.15.3:0/4012567866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdba00afd60 0x7fdba00a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 >> 172.21.15.3:0/4012567866 conn(0x7fdba0060310 msgr2=0x7fdba0144c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:19.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 shutdown_connections 2022-07-21T01:40:19.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.262+0000 7fdba68f5700 1 -- 172.21.15.3:0/4012567866 wait complete. 2022-07-21T01:40:19.269 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 41 2022-07-21T01:40:19.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:19.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.382+0000 7fd9d359e700 1 Processor -- start 2022-07-21T01:40:19.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.382+0000 7fd9d359e700 1 -- start start 2022-07-21T01:40:19.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.382+0000 7fd9d359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 0x7fd9d40af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.382+0000 7fd9d359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9d40ad4b0 0x7fd9d41432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.382+0000 7fd9d359e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40aff30 con 0x7fd9d41437d0 2022-07-21T01:40:19.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.382+0000 7fd9d359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d405c7a0 con 0x7fd9d40ad4b0 2022-07-21T01:40:19.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.382+0000 7fd9d359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4140990 con 0x7fd9d40af450 2022-07-21T01:40:19.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d2d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd9d41437d0 0x7fd9d40a8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55966/0 (socket says 172.21.15.3:55966) 2022-07-21T01:40:19.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d2d9d700 1 -- 172.21.15.3:0/3434123530 learned_addr learned my addr 172.21.15.3:0/3434123530 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:19.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9cbfff700 1 --2- 172.21.15.3:0/3434123530 >> v2:172.21.15.143:3300/0 conn(0x7fd9d40ad4b0 0x7fd9d41432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d259c700 1 --2- 172.21.15.3:0/3434123530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 0x7fd9d40af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d1d9b700 1 -- 172.21.15.3:0/3434123530 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957754374 0 0) 0x7fd9d40aff30 con 0x7fd9d41437d0 2022-07-21T01:40:19.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d1d9b700 1 -- 172.21.15.3:0/3434123530 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b4003260 con 0x7fd9d41437d0 2022-07-21T01:40:19.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d259c700 1 -- 172.21.15.3:0/3434123530 >> v2:172.21.15.143:3300/0 conn(0x7fd9d40ad4b0 msgr2=0x7fd9d41432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:19.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d259c700 1 --2- 172.21.15.3:0/3434123530 >> v2:172.21.15.143:3300/0 conn(0x7fd9d40ad4b0 0x7fd9d41432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d259c700 1 -- 172.21.15.3:0/3434123530 >> v1:172.21.15.3:6789/0 conn(0x7fd9d41437d0 legacy=0x7fd9d40a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d259c700 1 -- 172.21.15.3:0/3434123530 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40a9550 con 0x7fd9d40af450 2022-07-21T01:40:19.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d259c700 1 --2- 172.21.15.3:0/3434123530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 0x7fd9d40af830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fd9d413fdc0 tx=0x7fd9c4006df0).ready entity=mon.2 client_cookie=73ca21bd67b3784b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d1d9b700 1 -- 172.21.15.3:0/3434123530 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9c400d570 con 0x7fd9d40af450 2022-07-21T01:40:19.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d1d9b700 1 -- 172.21.15.3:0/3434123530 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c4000e90 con 0x7fd9d40af450 2022-07-21T01:40:19.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d1d9b700 1 -- 172.21.15.3:0/3434123530 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9c400d8c0 con 0x7fd9d40af450 2022-07-21T01:40:19.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d359e700 1 -- 172.21.15.3:0/3434123530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 msgr2=0x7fd9d40af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d359e700 1 --2- 172.21.15.3:0/3434123530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 0x7fd9d40af830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fd9d413fdc0 tx=0x7fd9c4006df0).stop 2022-07-21T01:40:19.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d359e700 1 -- 172.21.15.3:0/3434123530 shutdown_connections 2022-07-21T01:40:19.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d359e700 1 --2- 172.21.15.3:0/3434123530 >> v2:172.21.15.143:3300/0 conn(0x7fd9d40ad4b0 0x7fd9d41432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d359e700 1 --2- 172.21.15.3:0/3434123530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 0x7fd9d40af830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d359e700 1 -- 172.21.15.3:0/3434123530 >> 172.21.15.3:0/3434123530 conn(0x7fd9d40612d0 msgr2=0x7fd9d4062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d359e700 1 -- 172.21.15.3:0/3434123530 shutdown_connections 2022-07-21T01:40:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.386+0000 7fd9d359e700 1 -- 172.21.15.3:0/3434123530 wait complete. 2022-07-21T01:40:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 Processor -- start 2022-07-21T01:40:19.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 -- start start 2022-07-21T01:40:19.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9d40ad4b0 0x7fd9d4137660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 0x7fd9d40b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d41437d0 0x7fd9d40a9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4137bf0 con 0x7fd9d41437d0 2022-07-21T01:40:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4138580 con 0x7fd9d40ad4b0 2022-07-21T01:40:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4138820 con 0x7fd9d40af450 2022-07-21T01:40:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d41437d0 0x7fd9d40a9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d41437d0 0x7fd9d40a9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47694/0 (socket says 172.21.15.3:47694) 2022-07-21T01:40:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d2d9d700 1 -- 172.21.15.3:0/2215207369 learned_addr learned my addr 172.21.15.3:0/2215207369 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d259c700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9d40ad4b0 0x7fd9d4137660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9cbfff700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 0x7fd9d40b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d2d9d700 1 -- 172.21.15.3:0/2215207369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 msgr2=0x7fd9d40b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d2d9d700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 0x7fd9d40b5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d2d9d700 1 -- 172.21.15.3:0/2215207369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9d40ad4b0 msgr2=0x7fd9d4137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d2d9d700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9d40ad4b0 0x7fd9d4137660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d2d9d700 1 -- 172.21.15.3:0/2215207369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c400b180 con 0x7fd9d41437d0 2022-07-21T01:40:19.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d2d9d700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d41437d0 0x7fd9d40a9d80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7fd9cc001490 tx=0x7fd9cc007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9cb7fe700 1 -- 172.21.15.3:0/2215207369 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9cc005c70 con 0x7fd9d41437d0 2022-07-21T01:40:19.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9cb7fe700 1 -- 172.21.15.3:0/2215207369 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc005dd0 con 0x7fd9d41437d0 2022-07-21T01:40:19.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d4138ac0 con 0x7fd9d41437d0 2022-07-21T01:40:19.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9cb7fe700 1 -- 172.21.15.3:0/2215207369 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9cc0072c0 con 0x7fd9d41437d0 2022-07-21T01:40:19.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.390+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d4137f80 con 0x7fd9d41437d0 2022-07-21T01:40:19.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.394+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d40a9ac0 con 0x7fd9d41437d0 2022-07-21T01:40:19.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.394+0000 7fd9cb7fe700 1 -- 172.21.15.3:0/2215207369 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd9cc01a060 con 0x7fd9d41437d0 2022-07-21T01:40:19.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.394+0000 7fd9cb7fe700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd9b80539e0 0x7fd9b8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.394+0000 7fd9cb7fe700 1 -- 172.21.15.3:0/2215207369 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd9cc06b670 con 0x7fd9d41437d0 2022-07-21T01:40:19.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.394+0000 7fd9d259c700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd9b80539e0 0x7fd9b8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.394+0000 7fd9d259c700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd9b80539e0 0x7fd9b8055ea0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd9c401a000 tx=0x7fd9c4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.402+0000 7fd9cb7fe700 1 -- 172.21.15.3:0/2215207369 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd9d40a9ac0 con 0x7fd9d41437d0 2022-07-21T01:40:19.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.530+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9d40a9ac0 con 0x7fd9d41437d0 2022-07-21T01:40:19.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.530+0000 7fd9cb7fe700 1 -- 172.21.15.3:0/2215207369 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16274 (secure 0 0 0) 0x7fd9d40a9ac0 con 0x7fd9d41437d0 2022-07-21T01:40:19.534 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:19.535 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:18.943662+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:16.360095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:17.262905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:19.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd9b80539e0 msgr2=0x7fd9b8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd9b80539e0 0x7fd9b8055ea0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd9c401a000 tx=0x7fd9c4014000).stop 2022-07-21T01:40:19.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d41437d0 msgr2=0x7fd9d40a9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d41437d0 0x7fd9d40a9d80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7fd9cc001490 tx=0x7fd9cc007dc0).stop 2022-07-21T01:40:19.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 shutdown_connections 2022-07-21T01:40:19.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d41437d0 0x7fd9d40a9d80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9d40af450 0x7fd9d40b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd9b80539e0 0x7fd9b8055ea0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 --2- 172.21.15.3:0/2215207369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9d40ad4b0 0x7fd9d4137660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 >> 172.21.15.3:0/2215207369 conn(0x7fd9d40612d0 msgr2=0x7fd9d40a8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:19.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 shutdown_connections 2022-07-21T01:40:19.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.534+0000 7fd9d359e700 1 -- 172.21.15.3:0/2215207369 wait complete. 2022-07-21T01:40:19.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:40:19.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc201f1700 1 Processor -- start 2022-07-21T01:40:19.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc201f1700 1 -- start start 2022-07-21T01:40:19.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc201f1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdc180b3630 0x7fdc18143f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc201f1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc180aa0d0 0x7fdc180af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc201f1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180af6f0 0x7fdc1805f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc201f1700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc181443f0 con 0x7fdc180aa0d0 2022-07-21T01:40:19.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc201f1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc180b3a10 con 0x7fdc180b3630 2022-07-21T01:40:19.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc201f1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180afb80 con 0x7fdc180af6f0 2022-07-21T01:40:19.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1d78c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc180aa0d0 0x7fdc180af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1d78c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc180aa0d0 0x7fdc180af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47700/0 (socket says 172.21.15.3:47700) 2022-07-21T01:40:19.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1d78c700 1 -- 172.21.15.3:0/555452434 learned_addr learned my addr 172.21.15.3:0/555452434 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:19.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1df8d700 1 --2- 172.21.15.3:0/555452434 >> v2:172.21.15.143:3300/0 conn(0x7fdc180b3630 0x7fdc18143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1e78e700 1 --2- 172.21.15.3:0/555452434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180af6f0 0x7fdc1805f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1d78c700 1 -- 172.21.15.3:0/555452434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180af6f0 msgr2=0x7fdc1805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:19.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1d78c700 1 --2- 172.21.15.3:0/555452434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180af6f0 0x7fdc1805f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1d78c700 1 -- 172.21.15.3:0/555452434 >> v2:172.21.15.143:3300/0 conn(0x7fdc180b3630 msgr2=0x7fdc18143f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:19.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1d78c700 1 --2- 172.21.15.3:0/555452434 >> v2:172.21.15.143:3300/0 conn(0x7fdc180b3630 0x7fdc18143f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1d78c700 1 -- 172.21.15.3:0/555452434 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc18146b40 con 0x7fdc180aa0d0 2022-07-21T01:40:19.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.662+0000 7fdc1d78c700 1 --2- 172.21.15.3:0/555452434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180aa0d0 0x7fdc180af200 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fdc08002f70 tx=0x7fdc08006eb0).ready entity=mon.0 client_cookie=f6880b04e0fa8d76 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc1cf8b700 1 -- 172.21.15.3:0/555452434 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc0800d570 con 0x7fdc180aa0d0 2022-07-21T01:40:19.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc1cf8b700 1 -- 172.21.15.3:0/555452434 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc08000e90 con 0x7fdc180aa0d0 2022-07-21T01:40:19.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc1cf8b700 1 -- 172.21.15.3:0/555452434 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc0800d900 con 0x7fdc180aa0d0 2022-07-21T01:40:19.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 -- 172.21.15.3:0/555452434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180aa0d0 msgr2=0x7fdc180af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 --2- 172.21.15.3:0/555452434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180aa0d0 0x7fdc180af200 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fdc08002f70 tx=0x7fdc08006eb0).stop 2022-07-21T01:40:19.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 -- 172.21.15.3:0/555452434 shutdown_connections 2022-07-21T01:40:19.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 --2- 172.21.15.3:0/555452434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180af6f0 0x7fdc1805f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 --2- 172.21.15.3:0/555452434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180aa0d0 0x7fdc180af200 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 --2- 172.21.15.3:0/555452434 >> v2:172.21.15.143:3300/0 conn(0x7fdc180b3630 0x7fdc18143f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 -- 172.21.15.3:0/555452434 >> 172.21.15.3:0/555452434 conn(0x7fdc18060950 msgr2=0x7fdc18060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:19.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 -- 172.21.15.3:0/555452434 shutdown_connections 2022-07-21T01:40:19.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 -- 172.21.15.3:0/555452434 wait complete. 2022-07-21T01:40:19.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.666+0000 7fdc201f1700 1 Processor -- start 2022-07-21T01:40:19.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc201f1700 1 -- start start 2022-07-21T01:40:19.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc201f1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc180aa0d0 0x7fdc180b6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc201f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180af6f0 0x7fdc18156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc201f1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180b3630 0x7fdc1813b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc201f1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc18155b60 con 0x7fdc180af6f0 2022-07-21T01:40:19.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc201f1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc18148320 con 0x7fdc180aa0d0 2022-07-21T01:40:19.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc201f1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc181485c0 con 0x7fdc180b3630 2022-07-21T01:40:19.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1d78c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180af6f0 0x7fdc18156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1d78c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180af6f0 0x7fdc18156230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47706/0 (socket says 172.21.15.3:47706) 2022-07-21T01:40:19.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1d78c700 1 -- 172.21.15.3:0/3818464126 learned_addr learned my addr 172.21.15.3:0/3818464126 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:19.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1df8d700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc180aa0d0 0x7fdc180b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1e78e700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180b3630 0x7fdc1813b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1d78c700 1 -- 172.21.15.3:0/3818464126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180b3630 msgr2=0x7fdc1813b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1d78c700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180b3630 0x7fdc1813b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1d78c700 1 -- 172.21.15.3:0/3818464126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc180aa0d0 msgr2=0x7fdc180b6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1d78c700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc180aa0d0 0x7fdc180b6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1d78c700 1 -- 172.21.15.3:0/3818464126 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0800b1e0 con 0x7fdc180af6f0 2022-07-21T01:40:19.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc1d78c700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180af6f0 0x7fdc18156230 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7fdc08002f70 tx=0x7fdc08015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc06ffd700 1 -- 172.21.15.3:0/3818464126 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc08011710 con 0x7fdc180af6f0 2022-07-21T01:40:19.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc06ffd700 1 -- 172.21.15.3:0/3818464126 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc08011870 con 0x7fdc180af6f0 2022-07-21T01:40:19.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc18155d90 con 0x7fdc180af6f0 2022-07-21T01:40:19.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc06ffd700 1 -- 172.21.15.3:0/3818464126 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc08011bd0 con 0x7fdc180af6f0 2022-07-21T01:40:19.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.670+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc180b5890 con 0x7fdc180af6f0 2022-07-21T01:40:19.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.674+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc18148780 con 0x7fdc180af6f0 2022-07-21T01:40:19.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.674+0000 7fdc06ffd700 1 -- 172.21.15.3:0/3818464126 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc08025030 con 0x7fdc180af6f0 2022-07-21T01:40:19.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.674+0000 7fdc06ffd700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbf8053950 0x7fdbf8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.674+0000 7fdc06ffd700 1 -- 172.21.15.3:0/3818464126 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdc0806d120 con 0x7fdc180af6f0 2022-07-21T01:40:19.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.674+0000 7fdc1df8d700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbf8053950 0x7fdbf8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.674+0000 7fdc1df8d700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbf8053950 0x7fdbf8055e10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fdc1400b000 tx=0x7fdc14008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.682+0000 7fdc06ffd700 1 -- 172.21.15.3:0/3818464126 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc08043ee0 con 0x7fdc180af6f0 2022-07-21T01:40:19.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.846+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fdc18155f20 con 0x7fdc180af6f0 2022-07-21T01:40:19.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc06ffd700 1 -- 172.21.15.3:0/3818464126 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v25) v1 ==== 184+0+0 (secure 0 0 0) 0x7fdc080230e0 con 0x7fdc180af6f0 2022-07-21T01:40:19.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbf8053950 msgr2=0x7fdbf8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbf8053950 0x7fdbf8055e10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fdc1400b000 tx=0x7fdc14008040).stop 2022-07-21T01:40:19.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180af6f0 msgr2=0x7fdc18156230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180af6f0 0x7fdc18156230 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7fdc08002f70 tx=0x7fdc08015000).stop 2022-07-21T01:40:19.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 shutdown_connections 2022-07-21T01:40:19.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc180b3630 0x7fdc1813b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc180af6f0 0x7fdc18156230 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbf8053950 0x7fdbf8055e10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 --2- 172.21.15.3:0/3818464126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc180aa0d0 0x7fdc180b6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.854+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 >> 172.21.15.3:0/3818464126 conn(0x7fdc18060950 msgr2=0x7fdc1813e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:19.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.858+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 shutdown_connections 2022-07-21T01:40:19.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.858+0000 7fdc201f1700 1 -- 172.21.15.3:0/3818464126 wait complete. 2022-07-21T01:40:19.862 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:40:19.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:40:19.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637e048700 1 Processor -- start 2022-07-21T01:40:19.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637e048700 1 -- start start 2022-07-21T01:40:19.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637e048700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f63780b2710 0x7f637814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637e048700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780b01a0 0x7f637814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637e048700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63780a7250 con 0x7f63780afd60 2022-07-21T01:40:19.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637e048700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63780a7480 con 0x7f63780b2710 2022-07-21T01:40:19.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637e048700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63780a7720 con 0x7f63780b01a0 2022-07-21T01:40:19.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637d046700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f63780b2710 0x7f637814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637d046700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f63780b2710 0x7f637814afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47712/0 (socket says 172.21.15.3:47712) 2022-07-21T01:40:19.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637d046700 1 -- 172.21.15.3:0/135254602 learned_addr learned my addr 172.21.15.3:0/135254602 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:19.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637c845700 1 --2- 172.21.15.3:0/135254602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780b01a0 0x7f637814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637d046700 1 -- 172.21.15.3:0/135254602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780b01a0 msgr2=0x7f637814b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:19.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637d046700 1 --2- 172.21.15.3:0/135254602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780b01a0 0x7f637814b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637d046700 1 -- 172.21.15.3:0/135254602 >> v1:172.21.15.143:6789/0 conn(0x7f63780afd60 legacy=0x7f63780a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637d046700 1 -- 172.21.15.3:0/135254602 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63780a79c0 con 0x7f63780b2710 2022-07-21T01:40:19.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f637d046700 1 --2- 172.21.15.3:0/135254602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 0x7f637814afd0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f636c002f70 tx=0x7f636c006eb0).ready entity=mon.0 client_cookie=52f1cb9235122b87 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:19.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f636bfff700 1 -- 172.21.15.3:0/135254602 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f636c00d5c0 con 0x7f63780b2710 2022-07-21T01:40:19.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f636bfff700 1 -- 172.21.15.3:0/135254602 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c000e90 con 0x7f63780b2710 2022-07-21T01:40:19.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.982+0000 7f636bfff700 1 -- 172.21.15.3:0/135254602 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f636c00d950 con 0x7f63780b2710 2022-07-21T01:40:19.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 -- 172.21.15.3:0/135254602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 msgr2=0x7f637814afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:19.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 --2- 172.21.15.3:0/135254602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 0x7f637814afd0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f636c002f70 tx=0x7f636c006eb0).stop 2022-07-21T01:40:19.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 -- 172.21.15.3:0/135254602 shutdown_connections 2022-07-21T01:40:19.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 --2- 172.21.15.3:0/135254602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780b01a0 0x7f637814b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 --2- 172.21.15.3:0/135254602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 0x7f637814afd0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:19.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 -- 172.21.15.3:0/135254602 >> 172.21.15.3:0/135254602 conn(0x7f6378060310 msgr2=0x7f63780b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:19.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 -- 172.21.15.3:0/135254602 shutdown_connections 2022-07-21T01:40:19.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 -- 172.21.15.3:0/135254602 wait complete. 2022-07-21T01:40:19.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 Processor -- start 2022-07-21T01:40:19.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 -- start start 2022-07-21T01:40:19.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780afd60 0x7f63780b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63780b01a0 0x7f63780b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 0x7f6378144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:19.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63780af640 con 0x7f63780b2710 2022-07-21T01:40:19.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63780b4e60 con 0x7f63780b01a0 2022-07-21T01:40:19.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637e048700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63780ac020 con 0x7f63780afd60 2022-07-21T01:40:19.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637d847700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 0x7f6378144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:19.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637d847700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 0x7f6378144ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47722/0 (socket says 172.21.15.3:47722) 2022-07-21T01:40:19.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637d847700 1 -- 172.21.15.3:0/643604710 learned_addr learned my addr 172.21.15.3:0/643604710 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:19.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637c845700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63780b01a0 0x7f63780b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.986+0000 7f637d046700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780afd60 0x7f63780b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f637d847700 1 -- 172.21.15.3:0/643604710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780afd60 msgr2=0x7f63780b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f637d847700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780afd60 0x7f63780b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f637d847700 1 -- 172.21.15.3:0/643604710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63780b01a0 msgr2=0x7f63780b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f637d847700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63780b01a0 0x7f63780b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f637d847700 1 -- 172.21.15.3:0/643604710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c00b1e0 con 0x7f63780b2710 2022-07-21T01:40:20.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f637d847700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 0x7f6378144ee0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f6378061c10 tx=0x7f6370007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:20.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f636a7fc700 1 -- 172.21.15.3:0/643604710 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f637000f470 con 0x7f63780b2710 2022-07-21T01:40:20.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f636a7fc700 1 -- 172.21.15.3:0/643604710 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637000f5d0 con 0x7f63780b2710 2022-07-21T01:40:20.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f636a7fc700 1 -- 172.21.15.3:0/643604710 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f637000f930 con 0x7f63780b2710 2022-07-21T01:40:20.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63780ac2c0 con 0x7f63780b2710 2022-07-21T01:40:20.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63781453d0 con 0x7f63780b2710 2022-07-21T01:40:20.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63780574e0 con 0x7f63780b2710 2022-07-21T01:40:20.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f636a7fc700 1 -- 172.21.15.3:0/643604710 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6370019030 con 0x7f63780b2710 2022-07-21T01:40:20.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.990+0000 7f636a7fc700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6360053b30 0x7f6360055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.994+0000 7f637d046700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6360053b30 0x7f6360055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.994+0000 7f636a7fc700 1 -- 172.21.15.3:0/643604710 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f637006b740 con 0x7f63780b2710 2022-07-21T01:40:20.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.994+0000 7f637d046700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6360053b30 0x7f6360055ff0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f636c01c000 tx=0x7f636c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:20.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:19.998+0000 7f636a7fc700 1 -- 172.21.15.3:0/643604710 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f63700423f0 con 0x7f63780b2710 2022-07-21T01:40:20.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.166+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6378069a70 con 0x7f63780b2710 2022-07-21T01:40:20.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.174+0000 7f636a7fc700 1 -- 172.21.15.3:0/643604710 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v26) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6370045a10 con 0x7f63780b2710 2022-07-21T01:40:20.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6360053b30 msgr2=0x7f6360055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6360053b30 0x7f6360055ff0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f636c01c000 tx=0x7f636c015000).stop 2022-07-21T01:40:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 msgr2=0x7f6378144ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 0x7f6378144ee0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f6378061c10 tx=0x7f6370007f40).stop 2022-07-21T01:40:20.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 shutdown_connections 2022-07-21T01:40:20.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f63780b2710 0x7f6378144ee0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6360053b30 0x7f6360055ff0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63780b01a0 0x7f63780b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 --2- 172.21.15.3:0/643604710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63780afd60 0x7f63780b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 >> 172.21.15.3:0/643604710 conn(0x7f6378060310 msgr2=0x7f63781461a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:20.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 shutdown_connections 2022-07-21T01:40:20.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.178+0000 7f637e048700 1 -- 172.21.15.3:0/643604710 wait complete. 2022-07-21T01:40:20.184 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:40:20.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:20.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.294+0000 7f49067b4700 1 Processor -- start 2022-07-21T01:40:20.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.294+0000 7f49067b4700 1 -- start start 2022-07-21T01:40:20.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f49000b2710 0x7f490014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49000a8c50 0x7f490014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49000a7250 con 0x7f49000b01a0 2022-07-21T01:40:20.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49000a7480 con 0x7f49000b2710 2022-07-21T01:40:20.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49000a7720 con 0x7f49000a8c50 2022-07-21T01:40:20.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4904fb1700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f49000b01a0 0x7f49000a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56006/0 (socket says 172.21.15.3:56006) 2022-07-21T01:40:20.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4904fb1700 1 -- 172.21.15.3:0/620087957 learned_addr learned my addr 172.21.15.3:0/620087957 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:20.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49057b2700 1 --2- 172.21.15.3:0/620087957 >> v2:172.21.15.143:3300/0 conn(0x7f49000b2710 0x7f490014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4905fb3700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49000a8c50 0x7f490014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852900348 0 0) 0x7f49000a7250 con 0x7f49000b01a0 2022-07-21T01:40:20.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e4003260 con 0x7f49000b01a0 2022-07-21T01:40:20.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1140924401 0 0) 0x7f48e4003260 con 0x7f49000b01a0 2022-07-21T01:40:20.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49000a8c50 msgr2=0x7f490014b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:20.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49000a8c50 0x7f490014b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 >> v2:172.21.15.143:3300/0 conn(0x7f49000b2710 msgr2=0x7f490014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:20.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 --2- 172.21.15.3:0/620087957 >> v2:172.21.15.143:3300/0 conn(0x7f49000b2710 0x7f490014afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49000a79c0 con 0x7f49000b01a0 2022-07-21T01:40:20.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f48fc001660 con 0x7f49000b01a0 2022-07-21T01:40:20.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 >> v1:172.21.15.3:6789/0 conn(0x7f49000b01a0 legacy=0x7f49000a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e4003750 0x7f48e4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49000a8c50 0x7f48e4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48e40049f0 0x7f48e4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49000a7480 con 0x7f48e40049f0 2022-07-21T01:40:20.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49000a7720 con 0x7f48e4003750 2022-07-21T01:40:20.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e400b750 con 0x7f49000a8c50 2022-07-21T01:40:20.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4905fb3700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48e40049f0 0x7f48e4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4904fb1700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49000a8c50 0x7f48e4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49057b2700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e4003750 0x7f48e4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4905fb3700 1 -- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49000a8c50 msgr2=0x7f48e4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:20.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4905fb3700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49000a8c50 0x7f48e4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4905fb3700 1 -- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e4003750 msgr2=0x7f48e4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:20.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4905fb3700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e4003750 0x7f48e4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4905fb3700 1 -- 172.21.15.3:0/620087957 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e4003260 con 0x7f48e40049f0 2022-07-21T01:40:20.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49057b2700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e4003750 0x7f48e4003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:40:20.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4905fb3700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48e40049f0 0x7f48e4004e70 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f48f800c000 tx=0x7f48f80091d0).ready entity=mon.0 client_cookie=15232cb41c0b5f5d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:20.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48f800e7b0 con 0x7f48e40049f0 2022-07-21T01:40:20.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f800e910 con 0x7f48e40049f0 2022-07-21T01:40:20.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f48f7fff700 1 -- 172.21.15.3:0/620087957 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48f800ec70 con 0x7f48e40049f0 2022-07-21T01:40:20.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 -- 172.21.15.3:0/620087957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48e40049f0 msgr2=0x7f48e4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 --2- 172.21.15.3:0/620087957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48e40049f0 0x7f48e4004e70 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f48f800c000 tx=0x7f48f80091d0).stop 2022-07-21T01:40:20.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f4905fb3700 1 -- 172.21.15.3:0/620087957 reap_dead start 2022-07-21T01:40:20.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 -- 172.21.15.3:0/620087957 shutdown_connections 2022-07-21T01:40:20.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 -- 172.21.15.3:0/620087957 >> 172.21.15.3:0/620087957 conn(0x7f4900060310 msgr2=0x7f49000b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:20.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 -- 172.21.15.3:0/620087957 shutdown_connections 2022-07-21T01:40:20.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.298+0000 7f49067b4700 1 -- 172.21.15.3:0/620087957 wait complete. 2022-07-21T01:40:20.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 Processor -- start 2022-07-21T01:40:20.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 -- start start 2022-07-21T01:40:20.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40049f0 0x7f49000aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49000ab6b0 0x7f4900159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f490015a480 0x7f4900156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490005c290 con 0x7f49000ab6b0 2022-07-21T01:40:20.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490005c530 con 0x7f48e40049f0 2022-07-21T01:40:20.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490005c7d0 con 0x7f490015a480 2022-07-21T01:40:20.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4904fb1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49000ab6b0 0x7f4900159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4904fb1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49000ab6b0 0x7f4900159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47740/0 (socket says 172.21.15.3:47740) 2022-07-21T01:40:20.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4904fb1700 1 -- 172.21.15.3:0/9906293 learned_addr learned my addr 172.21.15.3:0/9906293 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49057b2700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40049f0 0x7f49000aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4905fb3700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f490015a480 0x7f4900156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4904fb1700 1 -- 172.21.15.3:0/9906293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f490015a480 msgr2=0x7f4900156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4904fb1700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f490015a480 0x7f4900156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4904fb1700 1 -- 172.21.15.3:0/9906293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40049f0 msgr2=0x7f49000aaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4904fb1700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40049f0 0x7f49000aaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4904fb1700 1 -- 172.21.15.3:0/9906293 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490015a930 con 0x7f49000ab6b0 2022-07-21T01:40:20.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f4904fb1700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49000ab6b0 0x7f4900159f90 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f48fc00b000 tx=0x7f49000b0640).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:20.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f48f67fc700 1 -- 172.21.15.3:0/9906293 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48fc015630 con 0x7f49000ab6b0 2022-07-21T01:40:20.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f48f67fc700 1 -- 172.21.15.3:0/9906293 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc010040 con 0x7f49000ab6b0 2022-07-21T01:40:20.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4900065ec0 con 0x7f49000ab6b0 2022-07-21T01:40:20.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f48f67fc700 1 -- 172.21.15.3:0/9906293 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48fc015980 con 0x7f49000ab6b0 2022-07-21T01:40:20.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.302+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4900156520 con 0x7f49000ab6b0 2022-07-21T01:40:20.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.306+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49000574e0 con 0x7f49000ab6b0 2022-07-21T01:40:20.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.306+0000 7f48f67fc700 1 -- 172.21.15.3:0/9906293 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f48fc015ae0 con 0x7f49000ab6b0 2022-07-21T01:40:20.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.306+0000 7f48f67fc700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48e8053aa0 0x7f48e8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:20.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.306+0000 7f49057b2700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48e8053aa0 0x7f48e8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:20.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.306+0000 7f48f67fc700 1 -- 172.21.15.3:0/9906293 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f48fc06c0e0 con 0x7f49000ab6b0 2022-07-21T01:40:20.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.306+0000 7f49057b2700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48e8053aa0 0x7f48e8055f60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f48f0009000 tx=0x7f48f0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:20.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.314+0000 7f48f67fc700 1 -- 172.21.15.3:0/9906293 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f48fc042d90 con 0x7f49000ab6b0 2022-07-21T01:40:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.474+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4900067de0 con 0x7f49000ab6b0 2022-07-21T01:40:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.474+0000 7f48f67fc700 1 -- 172.21.15.3:0/9906293 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2487 (secure 0 0 0) 0x7f48fc0463b0 con 0x7f49000ab6b0 2022-07-21T01:40:20.478 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:20.478 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":41,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/2659690520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2659690520},{"type":"v1","addr":"172.21.15.143:6833","nonce":2659690520}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":6600,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2193393317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2193393317},{"type":"v1","addr":"172.21.15.3:6835","nonce":2193393317}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:17.940781+0000","modified":"2022-07-21T01:40:18.943071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6603},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6603":{"gid":6603,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/539587272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":539587272},{"type":"v1","addr":"172.21.15.3:6837","nonce":539587272}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-21T01:40:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48e8053aa0 msgr2=0x7f48e8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48e8053aa0 0x7f48e8055f60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f48f0009000 tx=0x7f48f0008040).stop 2022-07-21T01:40:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49000ab6b0 msgr2=0x7f4900159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49000ab6b0 0x7f4900159f90 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f48fc00b000 tx=0x7f49000b0640).stop 2022-07-21T01:40:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 shutdown_connections 2022-07-21T01:40:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f490015a480 0x7f4900156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49000ab6b0 0x7f4900159f90 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48e8053aa0 0x7f48e8055f60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 --2- 172.21.15.3:0/9906293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40049f0 0x7f49000aaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 >> 172.21.15.3:0/9906293 conn(0x7f4900060310 msgr2=0x7f4900148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 shutdown_connections 2022-07-21T01:40:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:20.478+0000 7f49067b4700 1 -- 172.21.15.3:0/9906293 wait complete. 2022-07-21T01:40:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 41 2022-07-21T01:40:20.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:40:17.940781+0000', 'modified': '2022-07-21T01:40:18.943071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6603}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6603': {'gid': 6603, 'name': 'a', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/539587272', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 539587272}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 539587272}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:40:20.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:40:20.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:40:20.912 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6603,"want_state":"up:active","state":"up:active","rank_uptime":2.9601597740000001,"mdsmap_epoch":41,"osdmap_epoch":90,"osdmap_epoch_barrier":89,"uptime":7.3909113470000003} 2022-07-21T01:40:20.913 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6603, "mdsmap_epoch": 41, "osdmap_epoch": 90, "osdmap_epoch_barrier": 89, "rank_uptime": 2.960159774, "state": "up:active", "uptime": 7.390911347, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:40:20.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:21.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5facea9700 1 Processor -- start 2022-07-21T01:40:21.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5facea9700 1 -- start start 2022-07-21T01:40:21.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5facea9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b08a0 0x7f5fa8142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5facea9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5fa80b3260 0x7f5fa81433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5facea9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80ad000 0x7f5fa805f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5facea9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa80b0c80 con 0x7f5fa80b3260 2022-07-21T01:40:21.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5facea9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa80ae810 con 0x7f5fa80b08a0 2022-07-21T01:40:21.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5facea9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80aeab0 con 0x7f5fa80ad000 2022-07-21T01:40:21.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5fa6ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5fa80b3260 0x7f5fa81433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5fa6ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5fa80b3260 0x7f5fa81433a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47748/0 (socket says 172.21.15.3:47748) 2022-07-21T01:40:21.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5fa77fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b08a0 0x7f5fa8142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5fa6ffd700 1 -- 172.21.15.3:0/2549155715 learned_addr learned my addr 172.21.15.3:0/2549155715 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.014+0000 7f5fa7fff700 1 --2- 172.21.15.3:0/2549155715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80ad000 0x7f5fa805f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa6ffd700 1 -- 172.21.15.3:0/2549155715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80ad000 msgr2=0x7f5fa805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa6ffd700 1 --2- 172.21.15.3:0/2549155715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80ad000 0x7f5fa805f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa6ffd700 1 -- 172.21.15.3:0/2549155715 >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b08a0 msgr2=0x7f5fa8142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:21.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa6ffd700 1 --2- 172.21.15.3:0/2549155715 >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b08a0 0x7f5fa8142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa6ffd700 1 -- 172.21.15.3:0/2549155715 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa80aed50 con 0x7f5fa80b3260 2022-07-21T01:40:21.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa7fff700 1 --2- 172.21.15.3:0/2549155715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80ad000 0x7f5fa805f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:21.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa6ffd700 1 --2- 172.21.15.3:0/2549155715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80b3260 0x7f5fa81433a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f5f9c002f70 tx=0x7f5f9c006f40).ready entity=mon.0 client_cookie=9f9bff1676a2af44 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa67fc700 1 -- 172.21.15.3:0/2549155715 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f9c010810 con 0x7f5fa80b3260 2022-07-21T01:40:21.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa67fc700 1 -- 172.21.15.3:0/2549155715 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9c000e90 con 0x7f5fa80b3260 2022-07-21T01:40:21.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5fa67fc700 1 -- 172.21.15.3:0/2549155715 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f9c010ba0 con 0x7f5fa80b3260 2022-07-21T01:40:21.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 -- 172.21.15.3:0/2549155715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80b3260 msgr2=0x7f5fa81433a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 --2- 172.21.15.3:0/2549155715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80b3260 0x7f5fa81433a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f5f9c002f70 tx=0x7f5f9c006f40).stop 2022-07-21T01:40:21.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 -- 172.21.15.3:0/2549155715 shutdown_connections 2022-07-21T01:40:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 --2- 172.21.15.3:0/2549155715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80ad000 0x7f5fa805f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 --2- 172.21.15.3:0/2549155715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80b3260 0x7f5fa81433a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 --2- 172.21.15.3:0/2549155715 >> v2:172.21.15.143:3300/0 conn(0x7f5fa80b08a0 0x7f5fa8142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 -- 172.21.15.3:0/2549155715 >> 172.21.15.3:0/2549155715 conn(0x7f5fa80612d0 msgr2=0x7f5fa8062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:21.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 -- 172.21.15.3:0/2549155715 shutdown_connections 2022-07-21T01:40:21.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 -- 172.21.15.3:0/2549155715 wait complete. 2022-07-21T01:40:21.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 Processor -- start 2022-07-21T01:40:21.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.018+0000 7f5facea9700 1 -- start start 2022-07-21T01:40:21.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5facea9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80ad000 0x7f5fa81412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5facea9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa80b08a0 0x7f5fa8151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5facea9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b3260 0x7f5fa81520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5facea9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa8067c30 con 0x7f5fa80ad000 2022-07-21T01:40:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5facea9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa8151780 con 0x7f5fa80b08a0 2022-07-21T01:40:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5facea9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80676f0 con 0x7f5fa80b3260 2022-07-21T01:40:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80ad000 0x7f5fa81412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80ad000 0x7f5fa81412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47752/0 (socket says 172.21.15.3:47752) 2022-07-21T01:40:21.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa77fe700 1 -- 172.21.15.3:0/1818219979 learned_addr learned my addr 172.21.15.3:0/1818219979 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:21.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa6ffd700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa80b08a0 0x7f5fa8151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa7fff700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b3260 0x7f5fa81520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa77fe700 1 -- 172.21.15.3:0/1818219979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b3260 msgr2=0x7f5fa81520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa77fe700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b3260 0x7f5fa81520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa77fe700 1 -- 172.21.15.3:0/1818219979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa80b08a0 msgr2=0x7f5fa8151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa77fe700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa80b08a0 0x7f5fa8151be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa77fe700 1 -- 172.21.15.3:0/1818219979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0009470 con 0x7f5fa80ad000 2022-07-21T01:40:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa77fe700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80ad000 0x7f5fa81412f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f5f9800c000 tx=0x7f5f98006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa4ff9700 1 -- 172.21.15.3:0/1818219979 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f98010780 con 0x7f5fa80ad000 2022-07-21T01:40:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa4ff9700 1 -- 172.21.15.3:0/1818219979 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f980108e0 con 0x7f5fa80ad000 2022-07-21T01:40:21.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c00a360 con 0x7f5fa80ad000 2022-07-21T01:40:21.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5fa4ff9700 1 -- 172.21.15.3:0/1818219979 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f98010c40 con 0x7f5fa80ad000 2022-07-21T01:40:21.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.022+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa80abb60 con 0x7f5fa80ad000 2022-07-21T01:40:21.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.026+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa8151590 con 0x7f5fa80ad000 2022-07-21T01:40:21.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.026+0000 7f5fa4ff9700 1 -- 172.21.15.3:0/1818219979 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f9801c460 con 0x7f5fa80ad000 2022-07-21T01:40:21.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.026+0000 7f5fa4ff9700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f88053a70 0x7f5f88055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.026+0000 7f5fa4ff9700 1 -- 172.21.15.3:0/1818219979 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5f9806c680 con 0x7f5fa80ad000 2022-07-21T01:40:21.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.026+0000 7f5fa6ffd700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f88053a70 0x7f5f88055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.026+0000 7f5fa6ffd700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f88053a70 0x7f5f88055f30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f5f9c01d000 tx=0x7f5fa80a95b0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.030+0000 7f5fa4ff9700 1 -- 172.21.15.3:0/1818219979 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f98043250 con 0x7f5fa80ad000 2022-07-21T01:40:21.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.158+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fa80574e0 con 0x7f5fa80ad000 2022-07-21T01:40:21.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.158+0000 7f5fa4ff9700 1 -- 172.21.15.3:0/1818219979 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16274 (secure 0 0 0) 0x7f5f98046870 con 0x7f5fa80ad000 2022-07-21T01:40:21.163 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:21.164 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:18.943662+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:16.360095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:17.262905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:21.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f88053a70 msgr2=0x7f5f88055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f88053a70 0x7f5f88055f30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f5f9c01d000 tx=0x7f5fa80a95b0).stop 2022-07-21T01:40:21.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80ad000 msgr2=0x7f5fa81412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80ad000 0x7f5fa81412f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f5f9800c000 tx=0x7f5f98006df0).stop 2022-07-21T01:40:21.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 shutdown_connections 2022-07-21T01:40:21.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fa80b3260 0x7f5fa81520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f88053a70 0x7f5f88055f30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fa80b08a0 0x7f5fa8151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 --2- 172.21.15.3:0/1818219979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa80ad000 0x7f5fa81412f0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 >> 172.21.15.3:0/1818219979 conn(0x7f5fa80612d0 msgr2=0x7f5fa8142210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:21.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 shutdown_connections 2022-07-21T01:40:21.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.162+0000 7f5facea9700 1 -- 172.21.15.3:0/1818219979 wait complete. 2022-07-21T01:40:21.176 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:40:16.360095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:40:17.262905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:40:21.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:40:21.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4dce96700 1 Processor -- start 2022-07-21T01:40:21.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4dce96700 1 -- start start 2022-07-21T01:40:21.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4dce96700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe4d80b5ed0 0x7fe4d80b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4dce96700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d8061d90 con 0x7fe4d80619b0 2022-07-21T01:40:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4dce96700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d80ba5a0 con 0x7fe4d80b6ee0 2022-07-21T01:40:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4dce96700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d8143fa0 con 0x7fe4d80b5ed0 2022-07-21T01:40:21.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4d7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe4d80b5ed0 0x7fe4d80b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4d77fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe4d80619b0 0x7fe4d8143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56040/0 (socket says 172.21.15.3:56040) 2022-07-21T01:40:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4d77fe700 1 -- 172.21.15.3:0/1399744971 learned_addr learned my addr 172.21.15.3:0/1399744971 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.282+0000 7fe4d7fff700 1 -- 172.21.15.3:0/1399744971 >> v1:172.21.15.143:6789/0 conn(0x7fe4d80b6ee0 legacy=0x7fe4d80b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4d7fff700 1 -- 172.21.15.3:0/1399744971 >> v1:172.21.15.3:6789/0 conn(0x7fe4d80619b0 legacy=0x7fe4d8143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4d7fff700 1 -- 172.21.15.3:0/1399744971 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d8143150 con 0x7fe4d80b5ed0 2022-07-21T01:40:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4d7fff700 1 --2- 172.21.15.3:0/1399744971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80b5ed0 0x7fe4d80b58a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7fe4d00032f0 tx=0x7fe4d00091d0).ready entity=mon.0 client_cookie=f2d4b99b0c7bc306 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4d67fc700 1 -- 172.21.15.3:0/1399744971 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d0009ea0 con 0x7fe4d80b5ed0 2022-07-21T01:40:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4d67fc700 1 -- 172.21.15.3:0/1399744971 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d000b8a0 con 0x7fe4d80b5ed0 2022-07-21T01:40:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4d67fc700 1 -- 172.21.15.3:0/1399744971 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d000ba10 con 0x7fe4d80b5ed0 2022-07-21T01:40:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4dce96700 1 -- 172.21.15.3:0/1399744971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80b5ed0 msgr2=0x7fe4d80b58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4dce96700 1 --2- 172.21.15.3:0/1399744971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80b5ed0 0x7fe4d80b58a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7fe4d00032f0 tx=0x7fe4d00091d0).stop 2022-07-21T01:40:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4dce96700 1 -- 172.21.15.3:0/1399744971 shutdown_connections 2022-07-21T01:40:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4dce96700 1 --2- 172.21.15.3:0/1399744971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80b5ed0 0x7fe4d80b58a0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4dce96700 1 -- 172.21.15.3:0/1399744971 >> 172.21.15.3:0/1399744971 conn(0x7fe4d806af80 msgr2=0x7fe4d805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4dce96700 1 -- 172.21.15.3:0/1399744971 shutdown_connections 2022-07-21T01:40:21.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4dce96700 1 -- 172.21.15.3:0/1399744971 wait complete. 2022-07-21T01:40:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4dce96700 1 Processor -- start 2022-07-21T01:40:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.286+0000 7fe4dce96700 1 -- start start 2022-07-21T01:40:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4dce96700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80619b0 0x7fe4d8112d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4dce96700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4d80b5ed0 0x7fe4d81526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4dce96700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4d80b6ee0 0x7fe4d8152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4dce96700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d80af0c0 con 0x7fe4d80619b0 2022-07-21T01:40:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4dce96700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d80af360 con 0x7fe4d80b6ee0 2022-07-21T01:40:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4dce96700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d8146b20 con 0x7fe4d80b5ed0 2022-07-21T01:40:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80619b0 0x7fe4d8112d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80619b0 0x7fe4d8112d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47766/0 (socket says 172.21.15.3:47766) 2022-07-21T01:40:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d77fe700 1 -- 172.21.15.3:0/1007712312 learned_addr learned my addr 172.21.15.3:0/1007712312 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d6ffd700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4d80b5ed0 0x7fe4d81526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d7fff700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4d80b6ee0 0x7fe4d8152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d77fe700 1 -- 172.21.15.3:0/1007712312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4d80b5ed0 msgr2=0x7fe4d81526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d77fe700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4d80b5ed0 0x7fe4d81526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d77fe700 1 -- 172.21.15.3:0/1007712312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4d80b6ee0 msgr2=0x7fe4d8152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d77fe700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4d80b6ee0 0x7fe4d8152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d77fe700 1 -- 172.21.15.3:0/1007712312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d0009470 con 0x7fe4d80619b0 2022-07-21T01:40:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d77fe700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80619b0 0x7fe4d8112d70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7fe4d80b41b0 tx=0x7fe4c8007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d4ff9700 1 -- 172.21.15.3:0/1007712312 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4c800ce30 con 0x7fe4d80619b0 2022-07-21T01:40:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d4ff9700 1 -- 172.21.15.3:0/1007712312 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c80072e0 con 0x7fe4d80619b0 2022-07-21T01:40:21.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d4ff9700 1 -- 172.21.15.3:0/1007712312 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4c80057c0 con 0x7fe4d80619b0 2022-07-21T01:40:21.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d8146fc0 con 0x7fe4d80619b0 2022-07-21T01:40:21.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d8145de0 con 0x7fe4d80619b0 2022-07-21T01:40:21.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d8146ce0 con 0x7fe4d80619b0 2022-07-21T01:40:21.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d4ff9700 1 -- 172.21.15.3:0/1007712312 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe4c8010070 con 0x7fe4d80619b0 2022-07-21T01:40:21.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d4ff9700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4bc053b30 0x7fe4bc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.290+0000 7fe4d4ff9700 1 -- 172.21.15.3:0/1007712312 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe4c806b690 con 0x7fe4d80619b0 2022-07-21T01:40:21.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.294+0000 7fe4d6ffd700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4bc053b30 0x7fe4bc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.294+0000 7fe4d6ffd700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4bc053b30 0x7fe4bc055ff0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fe4c000b000 tx=0x7fe4c0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.298+0000 7fe4d4ff9700 1 -- 172.21.15.3:0/1007712312 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4c80423b0 con 0x7fe4d80619b0 2022-07-21T01:40:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.426+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe4d80574e0 con 0x7fe4d80619b0 2022-07-21T01:40:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.426+0000 7fe4d4ff9700 1 -- 172.21.15.3:0/1007712312 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v90) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe4c8018070 con 0x7fe4d80619b0 2022-07-21T01:40:21.430 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:40:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4bc053b30 msgr2=0x7fe4bc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4bc053b30 0x7fe4bc055ff0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fe4c000b000 tx=0x7fe4c0009040).stop 2022-07-21T01:40:21.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80619b0 msgr2=0x7fe4d8112d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80619b0 0x7fe4d8112d70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7fe4d80b41b0 tx=0x7fe4c8007c30).stop 2022-07-21T01:40:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 shutdown_connections 2022-07-21T01:40:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4bc053b30 0x7fe4bc055ff0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4d80b6ee0 0x7fe4d8152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4d80b5ed0 0x7fe4d81526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 --2- 172.21.15.3:0/1007712312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4d80619b0 0x7fe4d8112d70 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 >> 172.21.15.3:0/1007712312 conn(0x7fe4d806af80 msgr2=0x7fe4d805ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:21.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 shutdown_connections 2022-07-21T01:40:21.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.430+0000 7fe4dce96700 1 -- 172.21.15.3:0/1007712312 wait complete. 2022-07-21T01:40:21.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:40:21.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c6527f700 1 Processor -- start 2022-07-21T01:40:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c6527f700 1 -- start start 2022-07-21T01:40:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c6527f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600ad750 0x7f4c60061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c6527f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c600aee80 con 0x7f4c601405d0 2022-07-21T01:40:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c6527f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c600b70b0 con 0x7f4c600614b0 2022-07-21T01:40:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c6527f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600af0b0 con 0x7f4c600ad750 2022-07-21T01:40:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c64a7e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4c601405d0 0x7f4c601466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56058/0 (socket says 172.21.15.3:56058) 2022-07-21T01:40:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c64a7e700 1 -- 172.21.15.3:0/1453451636 learned_addr learned my addr 172.21.15.3:0/1453451636 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c5f7fe700 1 --2- 172.21.15.3:0/1453451636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600ad750 0x7f4c60061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c5effd700 1 -- 172.21.15.3:0/1453451636 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128356458 0 0) 0x7f4c600aee80 con 0x7f4c601405d0 2022-07-21T01:40:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c5effd700 1 -- 172.21.15.3:0/1453451636 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c40003260 con 0x7f4c601405d0 2022-07-21T01:40:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c5effd700 1 -- 172.21.15.3:0/1453451636 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346147827 0 0) 0x7f4c600b70b0 con 0x7f4c600614b0 2022-07-21T01:40:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c5effd700 1 -- 172.21.15.3:0/1453451636 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c600aee80 con 0x7f4c600614b0 2022-07-21T01:40:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c5f7fe700 1 -- 172.21.15.3:0/1453451636 >> v1:172.21.15.143:6789/0 conn(0x7f4c600614b0 legacy=0x7f4c600b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c5f7fe700 1 -- 172.21.15.3:0/1453451636 >> v1:172.21.15.3:6789/0 conn(0x7f4c601405d0 legacy=0x7f4c601466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c5f7fe700 1 -- 172.21.15.3:0/1453451636 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c600af350 con 0x7f4c600ad750 2022-07-21T01:40:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.550+0000 7f4c5f7fe700 1 --2- 172.21.15.3:0/1453451636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600ad750 0x7f4c60061890 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f4c48002f70 tx=0x7f4c48006df0).ready entity=mon.2 client_cookie=f226abaa4909a94f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c5effd700 1 -- 172.21.15.3:0/1453451636 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c4800d5c0 con 0x7f4c600ad750 2022-07-21T01:40:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c5effd700 1 -- 172.21.15.3:0/1453451636 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c48000e90 con 0x7f4c600ad750 2022-07-21T01:40:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c5effd700 1 -- 172.21.15.3:0/1453451636 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c4800d910 con 0x7f4c600ad750 2022-07-21T01:40:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 -- 172.21.15.3:0/1453451636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600ad750 msgr2=0x7f4c60061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 --2- 172.21.15.3:0/1453451636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600ad750 0x7f4c60061890 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f4c48002f70 tx=0x7f4c48006df0).stop 2022-07-21T01:40:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 -- 172.21.15.3:0/1453451636 shutdown_connections 2022-07-21T01:40:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 --2- 172.21.15.3:0/1453451636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600ad750 0x7f4c60061890 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 -- 172.21.15.3:0/1453451636 >> 172.21.15.3:0/1453451636 conn(0x7f4c60066f70 msgr2=0x7f4c60067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:21.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 -- 172.21.15.3:0/1453451636 shutdown_connections 2022-07-21T01:40:21.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 -- 172.21.15.3:0/1453451636 wait complete. 2022-07-21T01:40:21.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 Processor -- start 2022-07-21T01:40:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 -- start start 2022-07-21T01:40:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600614b0 0x7f4c60142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c600ad750 0x7f4c601458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601405d0 0x7f4c60145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600b0840 con 0x7f4c601405d0 2022-07-21T01:40:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600b0ae0 con 0x7f4c600ad750 2022-07-21T01:40:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c6527f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600b0d80 con 0x7f4c600614b0 2022-07-21T01:40:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c64a7e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601405d0 0x7f4c60145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c64a7e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601405d0 0x7f4c60145dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47784/0 (socket says 172.21.15.3:47784) 2022-07-21T01:40:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c64a7e700 1 -- 172.21.15.3:0/388793806 learned_addr learned my addr 172.21.15.3:0/388793806 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c5f7fe700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c600ad750 0x7f4c601458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.554+0000 7f4c5ffff700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600614b0 0x7f4c60142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c64a7e700 1 -- 172.21.15.3:0/388793806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600614b0 msgr2=0x7f4c60142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c64a7e700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600614b0 0x7f4c60142600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c64a7e700 1 -- 172.21.15.3:0/388793806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c600ad750 msgr2=0x7f4c601458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c64a7e700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c600ad750 0x7f4c601458d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c64a7e700 1 -- 172.21.15.3:0/388793806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4800b180 con 0x7f4c601405d0 2022-07-21T01:40:21.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c64a7e700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601405d0 0x7f4c60145dc0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f4c58001110 tx=0x7f4c58007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c5d7fa700 1 -- 172.21.15.3:0/388793806 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c5800cdb0 con 0x7f4c601405d0 2022-07-21T01:40:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c60145430 con 0x7f4c601405d0 2022-07-21T01:40:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c5d7fa700 1 -- 172.21.15.3:0/388793806 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c580070d0 con 0x7f4c601405d0 2022-07-21T01:40:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c5d7fa700 1 -- 172.21.15.3:0/388793806 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c58007430 con 0x7f4c601405d0 2022-07-21T01:40:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c601559f0 con 0x7f4c601405d0 2022-07-21T01:40:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c600b0460 con 0x7f4c601405d0 2022-07-21T01:40:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c5d7fa700 1 -- 172.21.15.3:0/388793806 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4c58010070 con 0x7f4c601405d0 2022-07-21T01:40:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c5d7fa700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c44053b00 0x7f4c44055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.558+0000 7f4c5d7fa700 1 -- 172.21.15.3:0/388793806 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4c5806b6b0 con 0x7f4c601405d0 2022-07-21T01:40:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.562+0000 7f4c5ffff700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c44053b00 0x7f4c44055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.562+0000 7f4c5ffff700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c44053b00 0x7f4c44055fc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f4c5000d000 tx=0x7f4c50007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.562+0000 7f4c5d7fa700 1 -- 172.21.15.3:0/388793806 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4c580423d0 con 0x7f4c601405d0 2022-07-21T01:40:21.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.690+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4c601455c0 con 0x7f4c601405d0 2022-07-21T01:40:21.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.690+0000 7f4c5d7fa700 1 -- 172.21.15.3:0/388793806 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v90) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4c580459f0 con 0x7f4c601405d0 2022-07-21T01:40:21.702 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c44053b00 msgr2=0x7f4c44055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c44053b00 0x7f4c44055fc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f4c5000d000 tx=0x7f4c50007da0).stop 2022-07-21T01:40:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601405d0 msgr2=0x7f4c60145dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601405d0 0x7f4c60145dc0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f4c58001110 tx=0x7f4c58007a80).stop 2022-07-21T01:40:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 shutdown_connections 2022-07-21T01:40:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601405d0 0x7f4c60145dc0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4c44053b00 0x7f4c44055fc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c600ad750 0x7f4c601458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 --2- 172.21.15.3:0/388793806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c600614b0 0x7f4c60142600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 >> 172.21.15.3:0/388793806 conn(0x7f4c60066f70 msgr2=0x7f4c60146880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:21.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 shutdown_connections 2022-07-21T01:40:21.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.694+0000 7f4c6527f700 1 -- 172.21.15.3:0/388793806 wait complete. 2022-07-21T01:40:21.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:40:21.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd748b49700 1 Processor -- start 2022-07-21T01:40:21.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd748b49700 1 -- start start 2022-07-21T01:40:21.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd748b49700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd7440af6e0 0x7fd744147730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd748b49700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7440b5a10 con 0x7fd7440b5630 2022-07-21T01:40:21.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd748b49700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7440ac990 con 0x7fd7440b07a0 2022-07-21T01:40:21.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd748b49700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd744145470 con 0x7fd7440af6e0 2022-07-21T01:40:21.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd7437fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd7440b5630 0x7fd744067730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56070/0 (socket says 172.21.15.3:56070) 2022-07-21T01:40:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd7437fe700 1 -- 172.21.15.3:0/313931504 learned_addr learned my addr 172.21.15.3:0/313931504 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd743fff700 1 --2- 172.21.15.3:0/313931504 >> v2:172.21.15.143:3300/0 conn(0x7fd7440af6e0 0x7fd744147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd7427fc700 1 -- 172.21.15.3:0/313931504 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597568514 0 0) 0x7fd7440b5a10 con 0x7fd7440b5630 2022-07-21T01:40:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd7427fc700 1 -- 172.21.15.3:0/313931504 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd730003260 con 0x7fd7440b5630 2022-07-21T01:40:21.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd7427fc700 1 -- 172.21.15.3:0/313931504 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 602950155 0 0) 0x7fd730003260 con 0x7fd7440b5630 2022-07-21T01:40:21.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd743fff700 1 -- 172.21.15.3:0/313931504 >> v1:172.21.15.143:6789/0 conn(0x7fd7440b07a0 legacy=0x7fd7440acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd743fff700 1 -- 172.21.15.3:0/313931504 >> v1:172.21.15.3:6789/0 conn(0x7fd7440b5630 legacy=0x7fd744067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.814+0000 7fd743fff700 1 -- 172.21.15.3:0/313931504 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7440b1bc0 con 0x7fd7440af6e0 2022-07-21T01:40:21.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd743fff700 1 --2- 172.21.15.3:0/313931504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7440af6e0 0x7fd744147730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fd738001800 tx=0x7fd73c006df0).ready entity=mon.1 client_cookie=b2b8fa7bd1a8a05c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd7427fc700 1 -- 172.21.15.3:0/313931504 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd738001660 con 0x7fd7440af6e0 2022-07-21T01:40:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd7427fc700 1 -- 172.21.15.3:0/313931504 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c000e90 con 0x7fd7440af6e0 2022-07-21T01:40:21.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd7427fc700 1 -- 172.21.15.3:0/313931504 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd73c00d4f0 con 0x7fd7440af6e0 2022-07-21T01:40:21.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 -- 172.21.15.3:0/313931504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7440af6e0 msgr2=0x7fd744147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 --2- 172.21.15.3:0/313931504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7440af6e0 0x7fd744147730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fd738001800 tx=0x7fd73c006df0).stop 2022-07-21T01:40:21.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 -- 172.21.15.3:0/313931504 shutdown_connections 2022-07-21T01:40:21.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 --2- 172.21.15.3:0/313931504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7440af6e0 0x7fd744147730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 -- 172.21.15.3:0/313931504 >> 172.21.15.3:0/313931504 conn(0x7fd7440626f0 msgr2=0x7fd744062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:21.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 -- 172.21.15.3:0/313931504 shutdown_connections 2022-07-21T01:40:21.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 -- 172.21.15.3:0/313931504 wait complete. 2022-07-21T01:40:21.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 Processor -- start 2022-07-21T01:40:21.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 -- start start 2022-07-21T01:40:21.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7440af6e0 0x7fd744149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7440b07a0 0x7fd7440649e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7440b5630 0x7fd744066a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7440ae180 con 0x7fd7440b07a0 2022-07-21T01:40:21.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd744156f10 con 0x7fd7440b5630 2022-07-21T01:40:21.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd748b49700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd744157100 con 0x7fd7440af6e0 2022-07-21T01:40:21.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.818+0000 7fd742ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7440b07a0 0x7fd7440649e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd742ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7440b07a0 0x7fd7440649e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47794/0 (socket says 172.21.15.3:47794) 2022-07-21T01:40:21.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd742ffd700 1 -- 172.21.15.3:0/964359216 learned_addr learned my addr 172.21.15.3:0/964359216 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:21.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd743fff700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7440b5630 0x7fd744066a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd7437fe700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7440af6e0 0x7fd744149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd742ffd700 1 -- 172.21.15.3:0/964359216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7440af6e0 msgr2=0x7fd744149bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd742ffd700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7440af6e0 0x7fd744149bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd742ffd700 1 -- 172.21.15.3:0/964359216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7440b5630 msgr2=0x7fd744066a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd742ffd700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7440b5630 0x7fd744066a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd742ffd700 1 -- 172.21.15.3:0/964359216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c00b180 con 0x7fd7440b07a0 2022-07-21T01:40:21.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd742ffd700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7440b07a0 0x7fd7440649e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7fd744069fc0 tx=0x7fd734007f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd740ff9700 1 -- 172.21.15.3:0/964359216 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd734006aa0 con 0x7fd7440b07a0 2022-07-21T01:40:21.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd740ff9700 1 -- 172.21.15.3:0/964359216 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd734006c00 con 0x7fd7440b07a0 2022-07-21T01:40:21.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7441573a0 con 0x7fd7440b07a0 2022-07-21T01:40:21.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd740ff9700 1 -- 172.21.15.3:0/964359216 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd734007030 con 0x7fd7440b07a0 2022-07-21T01:40:21.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74414a0b0 con 0x7fd7440b07a0 2022-07-21T01:40:21.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd744065070 con 0x7fd7440b07a0 2022-07-21T01:40:21.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd740ff9700 1 -- 172.21.15.3:0/964359216 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd73401b060 con 0x7fd7440b07a0 2022-07-21T01:40:21.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.822+0000 7fd740ff9700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd724053aa0 0x7fd724055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:21.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.826+0000 7fd740ff9700 1 -- 172.21.15.3:0/964359216 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd73406bda0 con 0x7fd7440b07a0 2022-07-21T01:40:21.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.826+0000 7fd7437fe700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd724053aa0 0x7fd724055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:21.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.826+0000 7fd7437fe700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd724053aa0 0x7fd724055f60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fd73800d000 tx=0x7fd738000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:21.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.830+0000 7fd740ff9700 1 -- 172.21.15.3:0/964359216 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd744065070 con 0x7fd7440b07a0 2022-07-21T01:40:21.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.954+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd744065070 con 0x7fd7440b07a0 2022-07-21T01:40:21.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.954+0000 7fd740ff9700 1 -- 172.21.15.3:0/964359216 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd744065070 con 0x7fd7440b07a0 2022-07-21T01:40:21.960 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:21.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd724053aa0 msgr2=0x7fd724055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd724053aa0 0x7fd724055f60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fd73800d000 tx=0x7fd738000bc0).stop 2022-07-21T01:40:21.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7440b07a0 msgr2=0x7fd7440649e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:21.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7440b07a0 0x7fd7440649e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7fd744069fc0 tx=0x7fd734007f60).stop 2022-07-21T01:40:21.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 shutdown_connections 2022-07-21T01:40:21.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd724053aa0 0x7fd724055f60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7440b5630 0x7fd744066a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7440b07a0 0x7fd7440649e0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 --2- 172.21.15.3:0/964359216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7440af6e0 0x7fd744149bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:21.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 >> 172.21.15.3:0/964359216 conn(0x7fd7440626f0 msgr2=0x7fd744148d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:21.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 shutdown_connections 2022-07-21T01:40:21.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:21.958+0000 7fd748b49700 1 -- 172.21.15.3:0/964359216 wait complete. 2022-07-21T01:40:21.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:40:22.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc30759e700 1 Processor -- start 2022-07-21T01:40:22.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc30759e700 1 -- start start 2022-07-21T01:40:22.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc30759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc3080614b0 0x7fc308061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc30759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc3081405d0 0x7fc308140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc30759e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3080aee80 con 0x7fc3080ad750 2022-07-21T01:40:22.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc30759e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3080b70b0 con 0x7fc3081405d0 2022-07-21T01:40:22.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc30759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3080af0b0 con 0x7fc3080614b0 2022-07-21T01:40:22.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc305d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc3080ad750 0x7fc3080b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56082/0 (socket says 172.21.15.3:56082) 2022-07-21T01:40:22.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc305d9b700 1 -- 172.21.15.3:0/3512011455 learned_addr learned my addr 172.21.15.3:0/3512011455 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:22.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc306d9d700 1 --2- 172.21.15.3:0/3512011455 >> v2:172.21.15.3:3300/0 conn(0x7fc3081405d0 0x7fc308140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.094+0000 7fc30659c700 1 --2- 172.21.15.3:0/3512011455 >> v2:172.21.15.143:3300/0 conn(0x7fc3080614b0 0x7fc308061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956782770 0 0) 0x7fc3080aee80 con 0x7fc3080ad750 2022-07-21T01:40:22.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e8003260 con 0x7fc3080ad750 2022-07-21T01:40:22.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1851126366 0 0) 0x7fc2e8003260 con 0x7fc3080ad750 2022-07-21T01:40:22.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 >> v2:172.21.15.143:3300/0 conn(0x7fc3080614b0 msgr2=0x7fc308061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:22.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 --2- 172.21.15.3:0/3512011455 >> v2:172.21.15.143:3300/0 conn(0x7fc3080614b0 0x7fc308061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 >> v2:172.21.15.3:3300/0 conn(0x7fc3081405d0 msgr2=0x7fc308140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:22.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 --2- 172.21.15.3:0/3512011455 >> v2:172.21.15.3:3300/0 conn(0x7fc3081405d0 0x7fc308140a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3080af350 con 0x7fc3080ad750 2022-07-21T01:40:22.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc306d9d700 1 --2- 172.21.15.3:0/3512011455 >> v2:172.21.15.3:3300/0 conn(0x7fc3081405d0 0x7fc308140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:22.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fc2f0001660 con 0x7fc3080ad750 2022-07-21T01:40:22.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 >> v1:172.21.15.3:6789/0 conn(0x7fc3080ad750 legacy=0x7fc3080b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2e8003750 0x7fc2e8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2e8004500 0x7fc2e8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e8004e70 0x7fc2e800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080b70b0 con 0x7fc2e8004500 2022-07-21T01:40:22.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080af0b0 con 0x7fc2e8004e70 2022-07-21T01:40:22.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e800bc40 con 0x7fc2e8003750 2022-07-21T01:40:22.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc305d9b700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2e8004500 0x7fc2e8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30659c700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2e8003750 0x7fc2e8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc306d9d700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e8004e70 0x7fc2e800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc305d9b700 1 -- 172.21.15.3:0/3512011455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2e8003750 msgr2=0x7fc2e8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:22.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc305d9b700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2e8003750 0x7fc2e8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc305d9b700 1 -- 172.21.15.3:0/3512011455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e8004e70 msgr2=0x7fc2e800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc305d9b700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e8004e70 0x7fc2e800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc306d9d700 1 -- 172.21.15.3:0/3512011455 reap_dead start 2022-07-21T01:40:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc305d9b700 1 -- 172.21.15.3:0/3512011455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e8003260 con 0x7fc2e8004500 2022-07-21T01:40:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc305d9b700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2e8004500 0x7fc2e8004980 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7fc3080aed30 tx=0x7fc3080af4e0).ready entity=mon.0 client_cookie=c144144cb4145a58 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:22.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2f00070d0 con 0x7fc2e8004500 2022-07-21T01:40:22.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f0007230 con 0x7fc2e8004500 2022-07-21T01:40:22.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30559a700 1 -- 172.21.15.3:0/3512011455 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2f000c7e0 con 0x7fc2e8004500 2022-07-21T01:40:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30759e700 1 -- 172.21.15.3:0/3512011455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2e8004500 msgr2=0x7fc2e8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30759e700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2e8004500 0x7fc2e8004980 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7fc3080aed30 tx=0x7fc3080af4e0).stop 2022-07-21T01:40:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30759e700 1 -- 172.21.15.3:0/3512011455 shutdown_connections 2022-07-21T01:40:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30759e700 1 --2- 172.21.15.3:0/3512011455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2e8004500 0x7fc2e8004980 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30759e700 1 -- 172.21.15.3:0/3512011455 >> 172.21.15.3:0/3512011455 conn(0x7fc308066f70 msgr2=0x7fc308067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:22.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30759e700 1 -- 172.21.15.3:0/3512011455 shutdown_connections 2022-07-21T01:40:22.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.098+0000 7fc30759e700 1 -- 172.21.15.3:0/3512011455 wait complete. 2022-07-21T01:40:22.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 Processor -- start 2022-07-21T01:40:22.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 -- start start 2022-07-21T01:40:22.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e8004500 0x7fc3080b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc308144900 0x7fc308144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc308142180 0x7fc308142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080b0c70 con 0x7fc308144900 2022-07-21T01:40:22.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080b0200 con 0x7fc2e8004500 2022-07-21T01:40:22.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080b2220 con 0x7fc308142180 2022-07-21T01:40:22.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc305d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc308144900 0x7fc308144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc305d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc308144900 0x7fc308144d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47816/0 (socket says 172.21.15.3:47816) 2022-07-21T01:40:22.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc305d9b700 1 -- 172.21.15.3:0/1971899555 learned_addr learned my addr 172.21.15.3:0/1971899555 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:22.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30659c700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e8004500 0x7fc3080b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc306d9d700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc308142180 0x7fc308142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc305d9b700 1 -- 172.21.15.3:0/1971899555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc308142180 msgr2=0x7fc308142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc305d9b700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc308142180 0x7fc308142600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc305d9b700 1 -- 172.21.15.3:0/1971899555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e8004500 msgr2=0x7fc3080b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc305d9b700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e8004500 0x7fc3080b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc305d9b700 1 -- 172.21.15.3:0/1971899555 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3080b07d0 con 0x7fc308144900 2022-07-21T01:40:22.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc305d9b700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc308144900 0x7fc308144d80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fc3080aed30 tx=0x7fc2f0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:22.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc2ff7fe700 1 -- 172.21.15.3:0/1971899555 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2f00070d0 con 0x7fc308144900 2022-07-21T01:40:22.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc2ff7fe700 1 -- 172.21.15.3:0/1971899555 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f0004520 con 0x7fc308144900 2022-07-21T01:40:22.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3080aff30 con 0x7fc308144900 2022-07-21T01:40:22.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc2ff7fe700 1 -- 172.21.15.3:0/1971899555 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2f0004880 con 0x7fc308144900 2022-07-21T01:40:22.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.102+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3081559d0 con 0x7fc308144900 2022-07-21T01:40:22.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.106+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3080b0960 con 0x7fc308144900 2022-07-21T01:40:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.106+0000 7fc2ff7fe700 1 -- 172.21.15.3:0/1971899555 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc2f0026070 con 0x7fc308144900 2022-07-21T01:40:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.106+0000 7fc2ff7fe700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc2ec0539e0 0x7fc2ec055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.106+0000 7fc2ff7fe700 1 -- 172.21.15.3:0/1971899555 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc2f006c450 con 0x7fc308144900 2022-07-21T01:40:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.106+0000 7fc30659c700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc2ec0539e0 0x7fc2ec055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.106+0000 7fc30659c700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc2ec0539e0 0x7fc2ec055ea0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fc2f800b000 tx=0x7fc2f8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:22.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.114+0000 7fc2ff7fe700 1 -- 172.21.15.3:0/1971899555 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2f0043210 con 0x7fc308144900 2022-07-21T01:40:22.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.238+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3080b23e0 con 0x7fc308144900 2022-07-21T01:40:22.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.238+0000 7fc2ff7fe700 1 -- 172.21.15.3:0/1971899555 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc2f0046830 con 0x7fc308144900 2022-07-21T01:40:22.243 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:22.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc2ec0539e0 msgr2=0x7fc2ec055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc2ec0539e0 0x7fc2ec055ea0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fc2f800b000 tx=0x7fc2f8006df0).stop 2022-07-21T01:40:22.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc308144900 msgr2=0x7fc308144d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc308144900 0x7fc308144d80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fc3080aed30 tx=0x7fc2f0015000).stop 2022-07-21T01:40:22.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 shutdown_connections 2022-07-21T01:40:22.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc308142180 0x7fc308142600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc308144900 0x7fc308144d80 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc2ec0539e0 0x7fc2ec055ea0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 --2- 172.21.15.3:0/1971899555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e8004500 0x7fc3080b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 >> 172.21.15.3:0/1971899555 conn(0x7fc308066f70 msgr2=0x7fc308147a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:22.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 shutdown_connections 2022-07-21T01:40:22.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.242+0000 7fc30759e700 1 -- 172.21.15.3:0/1971899555 wait complete. 2022-07-21T01:40:22.258 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:40:22.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:22.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f2904bea700 1 Processor -- start 2022-07-21T01:40:22.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f2904bea700 1 -- start start 2022-07-21T01:40:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f2904bea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f29000b7840 0x7f2900065720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f2904bea700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f290014b040 0x7f290014b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f2904bea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000b2c20 con 0x7f29000b3070 2022-07-21T01:40:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f2904bea700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2900061830 con 0x7f290014b040 2022-07-21T01:40:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f2904bea700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2900065c10 con 0x7f29000b7840 2022-07-21T01:40:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f28fffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f290014b040 0x7f290014b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f28fffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f290014b040 0x7f290014b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47824/0 (socket says 172.21.15.3:47824) 2022-07-21T01:40:22.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f28fffff700 1 -- 172.21.15.3:0/33129930 learned_addr learned my addr 172.21.15.3:0/33129930 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:22.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f28ff7fe700 1 --2- 172.21.15.3:0/33129930 >> v2:172.21.15.143:3300/0 conn(0x7f29000b7840 0x7f2900065720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.366+0000 7f28feffd700 1 -- 172.21.15.3:0/33129930 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585485039 0 0) 0x7f29000b2c20 con 0x7f29000b3070 2022-07-21T01:40:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f28feffd700 1 -- 172.21.15.3:0/33129930 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e0003260 con 0x7f29000b3070 2022-07-21T01:40:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f28fffff700 1 -- 172.21.15.3:0/33129930 >> v2:172.21.15.143:3300/0 conn(0x7f29000b7840 msgr2=0x7f2900065720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f28fffff700 1 --2- 172.21.15.3:0/33129930 >> v2:172.21.15.143:3300/0 conn(0x7f29000b7840 0x7f2900065720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f28fffff700 1 -- 172.21.15.3:0/33129930 >> v1:172.21.15.143:6789/0 conn(0x7f29000b3070 legacy=0x7f2900061120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f28fffff700 1 -- 172.21.15.3:0/33129930 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000b7c20 con 0x7f290014b040 2022-07-21T01:40:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f28fffff700 1 --2- 172.21.15.3:0/33129930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 0x7f290014b480 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f28f80032f0 tx=0x7f28f8009290).ready entity=mon.0 client_cookie=ba50b5f12871f89d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:22.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f28feffd700 1 -- 172.21.15.3:0/33129930 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28f800b680 con 0x7f290014b040 2022-07-21T01:40:22.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f28feffd700 1 -- 172.21.15.3:0/33129930 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f800b7e0 con 0x7f290014b040 2022-07-21T01:40:22.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f28feffd700 1 -- 172.21.15.3:0/33129930 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28f800b950 con 0x7f290014b040 2022-07-21T01:40:22.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 -- 172.21.15.3:0/33129930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 msgr2=0x7f290014b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 --2- 172.21.15.3:0/33129930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 0x7f290014b480 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f28f80032f0 tx=0x7f28f8009290).stop 2022-07-21T01:40:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 -- 172.21.15.3:0/33129930 shutdown_connections 2022-07-21T01:40:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 --2- 172.21.15.3:0/33129930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 0x7f290014b480 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 --2- 172.21.15.3:0/33129930 >> v2:172.21.15.143:3300/0 conn(0x7f29000b7840 0x7f2900065720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 -- 172.21.15.3:0/33129930 >> 172.21.15.3:0/33129930 conn(0x7f290005f840 msgr2=0x7f290005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:22.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 -- 172.21.15.3:0/33129930 shutdown_connections 2022-07-21T01:40:22.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 -- 172.21.15.3:0/33129930 wait complete. 2022-07-21T01:40:22.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 Processor -- start 2022-07-21T01:40:22.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.370+0000 7f2904bea700 1 -- start start 2022-07-21T01:40:22.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f2904bea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29000b3070 0x7f29000aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f2904bea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29000b7840 0x7f290009ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f2904bea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 0x7f2900156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f2904bea700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000afaf0 con 0x7f290014b040 2022-07-21T01:40:22.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f2904bea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000afd90 con 0x7f29000b7840 2022-07-21T01:40:22.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f2904bea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29000aeb00 con 0x7f29000b3070 2022-07-21T01:40:22.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 0x7f2900156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 0x7f2900156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47832/0 (socket says 172.21.15.3:47832) 2022-07-21T01:40:22.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fffff700 1 -- 172.21.15.3:0/1404287418 learned_addr learned my addr 172.21.15.3:0/1404287418 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:22.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28ff7fe700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29000b3070 0x7f29000aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28f7fff700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29000b7840 0x7f290009ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fffff700 1 -- 172.21.15.3:0/1404287418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29000b3070 msgr2=0x7f29000aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fffff700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29000b3070 0x7f29000aefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fffff700 1 -- 172.21.15.3:0/1404287418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29000b7840 msgr2=0x7f290009ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fffff700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29000b7840 0x7f290009ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fffff700 1 -- 172.21.15.3:0/1404287418 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8009530 con 0x7f290014b040 2022-07-21T01:40:22.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28f7fff700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29000b7840 0x7f290009ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:22.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fffff700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 0x7f2900156980 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f28f80032f0 tx=0x7f28f8017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:22.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fd7fa700 1 -- 172.21.15.3:0/1404287418 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28f801f070 con 0x7f290014b040 2022-07-21T01:40:22.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fd7fa700 1 -- 172.21.15.3:0/1404287418 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f801b440 con 0x7f290014b040 2022-07-21T01:40:22.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29000ad600 con 0x7f290014b040 2022-07-21T01:40:22.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fd7fa700 1 -- 172.21.15.3:0/1404287418 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28f800b950 con 0x7f290014b040 2022-07-21T01:40:22.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2900156e70 con 0x7f290014b040 2022-07-21T01:40:22.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f28fd7fa700 1 -- 172.21.15.3:0/1404287418 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f28f800f070 con 0x7f290014b040 2022-07-21T01:40:22.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.374+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29000aff90 con 0x7f290014b040 2022-07-21T01:40:22.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.378+0000 7f28fd7fa700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28e4053a70 0x7f28e4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:22.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.378+0000 7f28fd7fa700 1 -- 172.21.15.3:0/1404287418 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f28f806d770 con 0x7f290014b040 2022-07-21T01:40:22.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.378+0000 7f28ff7fe700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28e4053a70 0x7f28e4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:22.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.378+0000 7f28ff7fe700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28e4053a70 0x7f28e4055f30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f28f0006000 tx=0x7f28f0006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:22.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.382+0000 7f28fd7fa700 1 -- 172.21.15.3:0/1404287418 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f28f8044420 con 0x7f290014b040 2022-07-21T01:40:22.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.542+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29000aecc0 con 0x7f290014b040 2022-07-21T01:40:22.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f28fd7fa700 1 -- 172.21.15.3:0/1404287418 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2487 (secure 0 0 0) 0x7f28f8047a40 con 0x7f290014b040 2022-07-21T01:40:22.549 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:22.549 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":41,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/2659690520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2659690520},{"type":"v1","addr":"172.21.15.143:6833","nonce":2659690520}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":6600,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2193393317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2193393317},{"type":"v1","addr":"172.21.15.3:6835","nonce":2193393317}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:17.940781+0000","modified":"2022-07-21T01:40:18.943071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6603},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6603":{"gid":6603,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/539587272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":539587272},{"type":"v1","addr":"172.21.15.3:6837","nonce":539587272}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-21T01:40:22.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28e4053a70 msgr2=0x7f28e4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28e4053a70 0x7f28e4055f30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f28f0006000 tx=0x7f28f0006d90).stop 2022-07-21T01:40:22.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 msgr2=0x7f2900156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:22.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 0x7f2900156980 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f28f80032f0 tx=0x7f28f8017000).stop 2022-07-21T01:40:22.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 shutdown_connections 2022-07-21T01:40:22.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f290014b040 0x7f2900156980 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28e4053a70 0x7f28e4055f30 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29000b7840 0x7f290009ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 --2- 172.21.15.3:0/1404287418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29000b3070 0x7f29000aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:22.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 >> 172.21.15.3:0/1404287418 conn(0x7f290005f840 msgr2=0x7f2900066550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:22.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 shutdown_connections 2022-07-21T01:40:22.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:22.546+0000 7f2904bea700 1 -- 172.21.15.3:0/1404287418 wait complete. 2022-07-21T01:40:22.554 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 41 2022-07-21T01:40:22.864 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:40:17.940781+0000', 'modified': '2022-07-21T01:40:18.943071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6603}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6603': {'gid': 6603, 'name': 'a', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/539587272', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 539587272}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 539587272}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:40:22.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:40:22.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:40:22.989 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6603,"want_state":"up:active","state":"up:active","rank_uptime":5.0363528950000003,"mdsmap_epoch":41,"osdmap_epoch":90,"osdmap_epoch_barrier":89,"uptime":9.4671044779999995} 2022-07-21T01:40:22.990 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6603, "mdsmap_epoch": 41, "osdmap_epoch": 90, "osdmap_epoch_barrier": 89, "rank_uptime": 5.036352895, "state": "up:active", "uptime": 9.467104478, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:40:22.991 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:40:22.991 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:40:22.992 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:40:22.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:22.993 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:23.007 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:40:23.077 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:23.078 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:23.143 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:23.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:23.148 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:40:23.203 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:40:23.204 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:23.216 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:40:23.240 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:23.240 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:23.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:40:23.286+0000 7f83aa57c080 -1 init, newargv = 0x558c4bacc980 newargc=9ceph-fuse[ 2022-07-21T01:40:23.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:22953]: starting ceph client 2022-07-21T01:40:23.301 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:23.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22953]: starting fuse 2022-07-21T01:40:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:23.304 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:23.305 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:40:23.319 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:40:23.320 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:40:23.320 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:23.321 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:40:23.322 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:40:23.323 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:40:23.323 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:40:23.324 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:40:23.325 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:23.326 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:40:23.326 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:40:23.327 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:40:23.327 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:23.328 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:40:23.328 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:40:23.329 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:40:23.330 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:40:23.330 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:23.331 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:40:23.331 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:40:23.332 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:40:23.332 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:40:23.333 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:40:23.334 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:23.334 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:40:23.335 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:40:23.405 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.22953.asok 2022-07-21T01:40:23.410 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22953.asok 2022-07-21T01:40:23.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:23.411 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22953.asok status 2022-07-21T01:40:23.498 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:40:23.499 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:40:23.499 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:40:23.499 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:40:23.499 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:40:23.500 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:40:23.500 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:40:23.500 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "22953", 2022-07-21T01:40:23.500 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:40:23.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:23.501 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:40:23.501 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:40:23.502 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 6775, 2022-07-21T01:40:23.502 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:40:23.503 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:40:23.503 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:40:23.504 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 6775 2022-07-21T01:40:23.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:23.505 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:40:23.506 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:40:23.506 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:40:23.506 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2620924110 2022-07-21T01:40:23.507 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:40:23.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:23.507 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:40:23.508 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:40:23.508 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:40:23.508 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2620924110 2022-07-21T01:40:23.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:23.509 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.6775 172.21.15.171:0/2620924110", 2022-07-21T01:40:23.509 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/2620924110", 2022-07-21T01:40:23.509 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:40:23.510 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 41, 2022-07-21T01:40:23.510 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 90, 2022-07-21T01:40:23.510 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:40:23.511 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:40:23.511 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:40:23.511 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:40:23.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:23.514 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:23.514 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:23.523 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:40:23.525 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:23.526 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:23.526 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:23.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f55963c9700 1 Processor -- start 2022-07-21T01:40:23.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f55963c9700 1 -- start start 2022-07-21T01:40:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f55963c9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55900b08a0 0x7f5590142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f55963c9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5590143ab0 con 0x7f55900b3260 2022-07-21T01:40:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f55963c9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55900b0c80 con 0x7f55900ad000 2022-07-21T01:40:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f55963c9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55900ae810 con 0x7f55900b08a0 2022-07-21T01:40:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f5594bc6700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f55900b3260 0x7f55901433a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56118/0 (socket says 172.21.15.3:56118) 2022-07-21T01:40:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f5594bc6700 1 -- 172.21.15.3:0/4109263832 learned_addr learned my addr 172.21.15.3:0/4109263832 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f55953c7700 1 --2- 172.21.15.3:0/4109263832 >> v2:172.21.15.143:3300/0 conn(0x7f55900b08a0 0x7f5590142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f557ffff700 1 -- 172.21.15.3:0/4109263832 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983765889 0 0) 0x7f5590143ab0 con 0x7f55900b3260 2022-07-21T01:40:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.702+0000 7f557ffff700 1 -- 172.21.15.3:0/4109263832 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5578003260 con 0x7f55900b3260 2022-07-21T01:40:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55953c7700 1 -- 172.21.15.3:0/4109263832 >> v1:172.21.15.143:6789/0 conn(0x7f55900ad000 legacy=0x7f559013fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f557ffff700 1 -- 172.21.15.3:0/4109263832 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180310631 0 0) 0x7f55900b0c80 con 0x7f55900ad000 2022-07-21T01:40:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55953c7700 1 -- 172.21.15.3:0/4109263832 >> v1:172.21.15.3:6789/0 conn(0x7f55900b3260 legacy=0x7f55901433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55953c7700 1 -- 172.21.15.3:0/4109263832 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55900aeab0 con 0x7f55900b08a0 2022-07-21T01:40:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55953c7700 1 --2- 172.21.15.3:0/4109263832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55900b08a0 0x7f5590142eb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f5580001800 tx=0x7f558c006e80).ready entity=mon.1 client_cookie=a48c784572290a91 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f557ffff700 1 -- 172.21.15.3:0/4109263832 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f558c014070 con 0x7f55900b08a0 2022-07-21T01:40:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f557ffff700 1 -- 172.21.15.3:0/4109263832 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c000e90 con 0x7f55900b08a0 2022-07-21T01:40:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f557ffff700 1 -- 172.21.15.3:0/4109263832 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5580001660 con 0x7f55900b08a0 2022-07-21T01:40:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55963c9700 1 -- 172.21.15.3:0/4109263832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55900b08a0 msgr2=0x7f5590142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55963c9700 1 --2- 172.21.15.3:0/4109263832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55900b08a0 0x7f5590142eb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f5580001800 tx=0x7f558c006e80).stop 2022-07-21T01:40:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55963c9700 1 -- 172.21.15.3:0/4109263832 shutdown_connections 2022-07-21T01:40:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55963c9700 1 --2- 172.21.15.3:0/4109263832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55900b08a0 0x7f5590142eb0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55963c9700 1 -- 172.21.15.3:0/4109263832 >> 172.21.15.3:0/4109263832 conn(0x7f55900612d0 msgr2=0x7f5590062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55963c9700 1 -- 172.21.15.3:0/4109263832 shutdown_connections 2022-07-21T01:40:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.706+0000 7f55963c9700 1 -- 172.21.15.3:0/4109263832 wait complete. 2022-07-21T01:40:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f55963c9700 1 Processor -- start 2022-07-21T01:40:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f55963c9700 1 -- start start 2022-07-21T01:40:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f55963c9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55900ad000 0x7f55900b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f55963c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55900b08a0 0x7f55900b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f55963c9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55900b3260 0x7f559013cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f55963c9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55900b39f0 con 0x7f55900b3260 2022-07-21T01:40:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f55963c9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55901437d0 con 0x7f55900ad000 2022-07-21T01:40:23.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f55963c9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5590143a70 con 0x7f55900b08a0 2022-07-21T01:40:23.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f5595bc8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55900b3260 0x7f559013cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f5595bc8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55900b3260 0x7f559013cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47846/0 (socket says 172.21.15.3:47846) 2022-07-21T01:40:23.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f5595bc8700 1 -- 172.21.15.3:0/666248976 learned_addr learned my addr 172.21.15.3:0/666248976 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f55953c7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55900ad000 0x7f55900b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f5594bc6700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55900b08a0 0x7f55900b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f5595bc8700 1 -- 172.21.15.3:0/666248976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55900b08a0 msgr2=0x7f55900b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.710+0000 7f5595bc8700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55900b08a0 0x7f55900b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f5595bc8700 1 -- 172.21.15.3:0/666248976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55900ad000 msgr2=0x7f55900b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f5595bc8700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55900ad000 0x7f55900b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f5595bc8700 1 -- 172.21.15.3:0/666248976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c00c310 con 0x7f55900b3260 2022-07-21T01:40:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f5595bc8700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55900b3260 0x7f559013cc30 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f5590146d50 tx=0x7f5588007d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f557e7fc700 1 -- 172.21.15.3:0/666248976 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f558800f470 con 0x7f55900b3260 2022-07-21T01:40:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f557e7fc700 1 -- 172.21.15.3:0/666248976 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558800f5d0 con 0x7f55900b3260 2022-07-21T01:40:23.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55900b3c50 con 0x7f55900b3260 2022-07-21T01:40:23.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f557e7fc700 1 -- 172.21.15.3:0/666248976 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f558800f930 con 0x7f55900b3260 2022-07-21T01:40:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55900b6210 con 0x7f55900b3260 2022-07-21T01:40:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f557e7fc700 1 -- 172.21.15.3:0/666248976 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5588010070 con 0x7f55900b3260 2022-07-21T01:40:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55900b3de0 con 0x7f55900b3260 2022-07-21T01:40:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f557e7fc700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5570053a70 0x7f5570055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f557e7fc700 1 -- 172.21.15.3:0/666248976 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f558806b320 con 0x7f55900b3260 2022-07-21T01:40:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.714+0000 7f55953c7700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5570053a70 0x7f5570055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.718+0000 7f55953c7700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5570053a70 0x7f5570055f30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f558c01a000 tx=0x7f55900ae730).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:23.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.726+0000 7f557e7fc700 1 -- 172.21.15.3:0/666248976 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5588042080 con 0x7f55900b3260 2022-07-21T01:40:23.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.850+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55900574e0 con 0x7f55900b3260 2022-07-21T01:40:23.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.850+0000 7f557e7fc700 1 -- 172.21.15.3:0/666248976 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16274 (secure 0 0 0) 0x7f55880456a0 con 0x7f55900b3260 2022-07-21T01:40:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:23.856 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:18.943662+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:16.360095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:17.262905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:23.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5570053a70 msgr2=0x7f5570055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5570053a70 0x7f5570055f30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f558c01a000 tx=0x7f55900ae730).stop 2022-07-21T01:40:23.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55900b3260 msgr2=0x7f559013cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55900b3260 0x7f559013cc30 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f5590146d50 tx=0x7f5588007d60).stop 2022-07-21T01:40:23.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 shutdown_connections 2022-07-21T01:40:23.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55900b3260 0x7f559013cc30 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55900b08a0 0x7f55900b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5570053a70 0x7f5570055f30 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 --2- 172.21.15.3:0/666248976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55900ad000 0x7f55900b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 >> 172.21.15.3:0/666248976 conn(0x7f55900612d0 msgr2=0x7f559005c620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:23.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 shutdown_connections 2022-07-21T01:40:23.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.854+0000 7f55963c9700 1 -- 172.21.15.3:0/666248976 wait complete. 2022-07-21T01:40:23.870 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:40:16.360095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:40:17.262905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:40:23.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:40:23.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc457fff700 1 Processor -- start 2022-07-21T01:40:23.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc457fff700 1 -- start start 2022-07-21T01:40:23.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc457fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc4580b2340 0x7fc45805f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc457fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc458060eb0 0x7fc4580b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc457fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4580b2720 con 0x7fc458143e20 2022-07-21T01:40:23.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc457fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc458063f80 con 0x7fc4580b2340 2022-07-21T01:40:23.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc457fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4580b2fc0 con 0x7fc458060eb0 2022-07-21T01:40:23.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc4577fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc458143e20 0x7fc45812c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56134/0 (socket says 172.21.15.3:56134) 2022-07-21T01:40:23.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc4577fe700 1 -- 172.21.15.3:0/1562953548 learned_addr learned my addr 172.21.15.3:0/1562953548 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:23.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc4567fc700 1 --2- 172.21.15.3:0/1562953548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc458060eb0 0x7fc4580b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.970+0000 7fc456ffd700 1 --2- 172.21.15.3:0/1562953548 >> v2:172.21.15.143:3300/0 conn(0x7fc4580b2340 0x7fc45805f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc455ffb700 1 -- 172.21.15.3:0/1562953548 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141297231 0 0) 0x7fc4580b2720 con 0x7fc458143e20 2022-07-21T01:40:23.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc455ffb700 1 -- 172.21.15.3:0/1562953548 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c003260 con 0x7fc458143e20 2022-07-21T01:40:23.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc4567fc700 1 -- 172.21.15.3:0/1562953548 >> v2:172.21.15.143:3300/0 conn(0x7fc4580b2340 msgr2=0x7fc45805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:23.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc4567fc700 1 --2- 172.21.15.3:0/1562953548 >> v2:172.21.15.143:3300/0 conn(0x7fc4580b2340 0x7fc45805f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc4567fc700 1 -- 172.21.15.3:0/1562953548 >> v1:172.21.15.3:6789/0 conn(0x7fc458143e20 legacy=0x7fc45812c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc4567fc700 1 -- 172.21.15.3:0/1562953548 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc458063a40 con 0x7fc458060eb0 2022-07-21T01:40:23.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc4567fc700 1 --2- 172.21.15.3:0/1562953548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc458060eb0 0x7fc4580b36c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fc44c002f70 tx=0x7fc44c006df0).ready entity=mon.2 client_cookie=b7e8555d0cda0db7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:23.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc455ffb700 1 -- 172.21.15.3:0/1562953548 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44c00d4a0 con 0x7fc458060eb0 2022-07-21T01:40:23.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc455ffb700 1 -- 172.21.15.3:0/1562953548 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c000e90 con 0x7fc458060eb0 2022-07-21T01:40:23.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc455ffb700 1 -- 172.21.15.3:0/1562953548 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44c00d7f0 con 0x7fc458060eb0 2022-07-21T01:40:23.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 -- 172.21.15.3:0/1562953548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc458060eb0 msgr2=0x7fc4580b36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 --2- 172.21.15.3:0/1562953548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc458060eb0 0x7fc4580b36c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fc44c002f70 tx=0x7fc44c006df0).stop 2022-07-21T01:40:23.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 -- 172.21.15.3:0/1562953548 shutdown_connections 2022-07-21T01:40:23.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 --2- 172.21.15.3:0/1562953548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc458060eb0 0x7fc4580b36c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 --2- 172.21.15.3:0/1562953548 >> v2:172.21.15.143:3300/0 conn(0x7fc4580b2340 0x7fc45805f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 -- 172.21.15.3:0/1562953548 >> 172.21.15.3:0/1562953548 conn(0x7fc45805c620 msgr2=0x7fc45805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:23.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 -- 172.21.15.3:0/1562953548 shutdown_connections 2022-07-21T01:40:23.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 -- 172.21.15.3:0/1562953548 wait complete. 2022-07-21T01:40:23.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 Processor -- start 2022-07-21T01:40:23.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.974+0000 7fc457fff700 1 -- start start 2022-07-21T01:40:23.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc457fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc458060eb0 0x7fc45812b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc457fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4580b2340 0x7fc458065470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc457fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc458143e20 0x7fc4581534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc457fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458150920 con 0x7fc458060eb0 2022-07-21T01:40:23.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc457fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458152bc0 con 0x7fc458143e20 2022-07-21T01:40:23.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc457fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc458067350 con 0x7fc4580b2340 2022-07-21T01:40:23.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc456ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc458060eb0 0x7fc45812b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc456ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc458060eb0 0x7fc45812b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47856/0 (socket says 172.21.15.3:47856) 2022-07-21T01:40:23.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc456ffd700 1 -- 172.21.15.3:0/3508850460 learned_addr learned my addr 172.21.15.3:0/3508850460 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:23.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc4567fc700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4580b2340 0x7fc458065470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc4577fe700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc458143e20 0x7fc4581534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc456ffd700 1 -- 172.21.15.3:0/3508850460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4580b2340 msgr2=0x7fc458065470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc456ffd700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4580b2340 0x7fc458065470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc456ffd700 1 -- 172.21.15.3:0/3508850460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc458143e20 msgr2=0x7fc4581534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:23.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc456ffd700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc458143e20 0x7fc4581534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:23.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc456ffd700 1 -- 172.21.15.3:0/3508850460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c00b180 con 0x7fc458060eb0 2022-07-21T01:40:23.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc456ffd700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc458060eb0 0x7fc45812b5e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7fc44800c000 tx=0x7fc448006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:23.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc447fff700 1 -- 172.21.15.3:0/3508850460 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc448010780 con 0x7fc458060eb0 2022-07-21T01:40:23.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc447fff700 1 -- 172.21.15.3:0/3508850460 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4480108e0 con 0x7fc458060eb0 2022-07-21T01:40:23.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc458152db0 con 0x7fc458060eb0 2022-07-21T01:40:23.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc447fff700 1 -- 172.21.15.3:0/3508850460 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc448010c40 con 0x7fc458060eb0 2022-07-21T01:40:23.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc458153010 con 0x7fc458060eb0 2022-07-21T01:40:23.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc458061d60 con 0x7fc458060eb0 2022-07-21T01:40:23.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc447fff700 1 -- 172.21.15.3:0/3508850460 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc44801c460 con 0x7fc458060eb0 2022-07-21T01:40:23.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc447fff700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc434053b00 0x7fc434055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:23.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc447fff700 1 -- 172.21.15.3:0/3508850460 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc44806c540 con 0x7fc458060eb0 2022-07-21T01:40:23.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc4567fc700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc434053b00 0x7fc434055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:23.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.978+0000 7fc4567fc700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc434053b00 0x7fc434055fc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fc44c01a000 tx=0x7fc44c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:23.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:23.986+0000 7fc447fff700 1 -- 172.21.15.3:0/3508850460 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc448043200 con 0x7fc458060eb0 2022-07-21T01:40:24.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.106+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc4580574e0 con 0x7fc458060eb0 2022-07-21T01:40:24.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.106+0000 7fc447fff700 1 -- 172.21.15.3:0/3508850460 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v90) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc448046820 con 0x7fc458060eb0 2022-07-21T01:40:24.113 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:40:24.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc434053b00 msgr2=0x7fc434055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc434053b00 0x7fc434055fc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fc44c01a000 tx=0x7fc44c014000).stop 2022-07-21T01:40:24.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc458060eb0 msgr2=0x7fc45812b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc458060eb0 0x7fc45812b5e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7fc44800c000 tx=0x7fc448006df0).stop 2022-07-21T01:40:24.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 shutdown_connections 2022-07-21T01:40:24.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc434053b00 0x7fc434055fc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc458143e20 0x7fc4581534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4580b2340 0x7fc458065470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 --2- 172.21.15.3:0/3508850460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc458060eb0 0x7fc45812b5e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 >> 172.21.15.3:0/3508850460 conn(0x7fc45805c620 msgr2=0x7fc4580b64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:24.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 shutdown_connections 2022-07-21T01:40:24.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.110+0000 7fc457fff700 1 -- 172.21.15.3:0/3508850460 wait complete. 2022-07-21T01:40:24.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:40:24.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.242+0000 7fbfe6881700 1 Processor -- start 2022-07-21T01:40:24.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.242+0000 7fbfe6881700 1 -- start start 2022-07-21T01:40:24.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.242+0000 7fbfe6881700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbfe00614b0 0x7fbfe0061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.242+0000 7fbfe6881700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe00aee80 con 0x7fbfe01405d0 2022-07-21T01:40:24.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.242+0000 7fbfe6881700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe00b70b0 con 0x7fbfe00ad750 2022-07-21T01:40:24.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.242+0000 7fbfe6881700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe00af0b0 con 0x7fbfe00614b0 2022-07-21T01:40:24.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.242+0000 7fbfe6080700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbfe01405d0 0x7fbfe01466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56148/0 (socket says 172.21.15.3:56148) 2022-07-21T01:40:24.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.242+0000 7fbfe6080700 1 -- 172.21.15.3:0/1659886648 learned_addr learned my addr 172.21.15.3:0/1659886648 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:24.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe587f700 1 --2- 172.21.15.3:0/1659886648 >> v2:172.21.15.143:3300/0 conn(0x7fbfe00614b0 0x7fbfe0061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636559612 0 0) 0x7fbfe00aee80 con 0x7fbfe01405d0 2022-07-21T01:40:24.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc8003260 con 0x7fbfe01405d0 2022-07-21T01:40:24.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616139305 0 0) 0x7fbfc8003260 con 0x7fbfe01405d0 2022-07-21T01:40:24.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 >> v2:172.21.15.143:3300/0 conn(0x7fbfe00614b0 msgr2=0x7fbfe0061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:24.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 --2- 172.21.15.3:0/1659886648 >> v2:172.21.15.143:3300/0 conn(0x7fbfe00614b0 0x7fbfe0061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 >> v1:172.21.15.143:6789/0 conn(0x7fbfe00ad750 legacy=0x7fbfe00b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe00af350 con 0x7fbfe01405d0 2022-07-21T01:40:24.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fbfd80019e0 con 0x7fbfe01405d0 2022-07-21T01:40:24.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 >> v1:172.21.15.3:6789/0 conn(0x7fbfe01405d0 legacy=0x7fbfe01466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfc8003750 0x7fbfc8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfc8004500 0x7fbfc8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc8004e70 0x7fbfc800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00af0b0 con 0x7fbfc8004e70 2022-07-21T01:40:24.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc800bd20 con 0x7fbfc8004500 2022-07-21T01:40:24.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc800bee0 con 0x7fbfc8003750 2022-07-21T01:40:24.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6080700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc8004e70 0x7fbfc800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe587f700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfc8003750 0x7fbfc8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe507e700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfc8004500 0x7fbfc8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6080700 1 -- 172.21.15.3:0/1659886648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfc8003750 msgr2=0x7fbfc8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:24.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6080700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfc8003750 0x7fbfc8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6080700 1 -- 172.21.15.3:0/1659886648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfc8004500 msgr2=0x7fbfc8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:24.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6080700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfc8004500 0x7fbfc8004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6080700 1 -- 172.21.15.3:0/1659886648 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc8003260 con 0x7fbfc8004e70 2022-07-21T01:40:24.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6080700 1 -- 172.21.15.3:0/1659886648 reap_dead start 2022-07-21T01:40:24.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6080700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc8004e70 0x7fbfc800b750 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7fbfd80016f0 tx=0x7fbfe00af4e0).ready entity=mon.0 client_cookie=5e36ac61b6056191 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:24.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd800cca0 con 0x7fbfc8004e70 2022-07-21T01:40:24.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd800ce00 con 0x7fbfc8004e70 2022-07-21T01:40:24.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe487d700 1 -- 172.21.15.3:0/1659886648 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd8005b80 con 0x7fbfc8004e70 2022-07-21T01:40:24.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6881700 1 -- 172.21.15.3:0/1659886648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc8004e70 msgr2=0x7fbfc800b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6881700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc8004e70 0x7fbfc800b750 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7fbfd80016f0 tx=0x7fbfe00af4e0).stop 2022-07-21T01:40:24.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6881700 1 -- 172.21.15.3:0/1659886648 shutdown_connections 2022-07-21T01:40:24.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6881700 1 --2- 172.21.15.3:0/1659886648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc8004e70 0x7fbfc800b750 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6881700 1 -- 172.21.15.3:0/1659886648 >> 172.21.15.3:0/1659886648 conn(0x7fbfe0066f70 msgr2=0x7fbfe0067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:24.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6881700 1 -- 172.21.15.3:0/1659886648 shutdown_connections 2022-07-21T01:40:24.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.246+0000 7fbfe6881700 1 -- 172.21.15.3:0/1659886648 wait complete. 2022-07-21T01:40:24.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6881700 1 Processor -- start 2022-07-21T01:40:24.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6881700 1 -- start start 2022-07-21T01:40:24.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6881700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfc8004e70 0x7fbfe00b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6881700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe00b3900 0x7fbfe013f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6881700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe013fd80 0x7fbfe00b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6881700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0153b30 con 0x7fbfe00b3900 2022-07-21T01:40:24.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6881700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0152eb0 con 0x7fbfc8004e70 2022-07-21T01:40:24.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6881700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0153150 con 0x7fbfe013fd80 2022-07-21T01:40:24.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe507e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe00b3900 0x7fbfe013f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe507e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe00b3900 0x7fbfe013f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47878/0 (socket says 172.21.15.3:47878) 2022-07-21T01:40:24.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe507e700 1 -- 172.21.15.3:0/434774689 learned_addr learned my addr 172.21.15.3:0/434774689 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:24.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe587f700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfc8004e70 0x7fbfe00b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6080700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe013fd80 0x7fbfe00b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe507e700 1 -- 172.21.15.3:0/434774689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe013fd80 msgr2=0x7fbfe00b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe507e700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe013fd80 0x7fbfe00b1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe507e700 1 -- 172.21.15.3:0/434774689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfc8004e70 msgr2=0x7fbfe00b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe507e700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfc8004e70 0x7fbfe00b80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe507e700 1 -- 172.21.15.3:0/434774689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe01533f0 con 0x7fbfe00b3900 2022-07-21T01:40:24.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe507e700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe00b3900 0x7fbfe013f890 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fbfd000d000 tx=0x7fbfd0007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:24.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfceffd700 1 -- 172.21.15.3:0/434774689 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd0002e60 con 0x7fbfe00b3900 2022-07-21T01:40:24.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfceffd700 1 -- 172.21.15.3:0/434774689 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd00114c0 con 0x7fbfe00b3900 2022-07-21T01:40:24.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe0153690 con 0x7fbfe00b3900 2022-07-21T01:40:24.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.250+0000 7fbfceffd700 1 -- 172.21.15.3:0/434774689 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd0011820 con 0x7fbfe00b3900 2022-07-21T01:40:24.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.254+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe0152ba0 con 0x7fbfe00b3900 2022-07-21T01:40:24.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.254+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe0140190 con 0x7fbfe00b3900 2022-07-21T01:40:24.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.254+0000 7fbfceffd700 1 -- 172.21.15.3:0/434774689 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbfd001a030 con 0x7fbfe00b3900 2022-07-21T01:40:24.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.254+0000 7fbfceffd700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbfc0053a70 0x7fbfc0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.254+0000 7fbfceffd700 1 -- 172.21.15.3:0/434774689 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fbfd006b870 con 0x7fbfe00b3900 2022-07-21T01:40:24.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.254+0000 7fbfe587f700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbfc0053a70 0x7fbfc0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.254+0000 7fbfe587f700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbfc0053a70 0x7fbfc0055f30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fbfdc00b000 tx=0x7fbfdc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:24.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.262+0000 7fbfceffd700 1 -- 172.21.15.3:0/434774689 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbfd00426e0 con 0x7fbfe00b3900 2022-07-21T01:40:24.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.386+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbfe00574e0 con 0x7fbfe00b3900 2022-07-21T01:40:24.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.386+0000 7fbfceffd700 1 -- 172.21.15.3:0/434774689 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v90) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbfd0018070 con 0x7fbfe00b3900 2022-07-21T01:40:24.392 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbfc0053a70 msgr2=0x7fbfc0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbfc0053a70 0x7fbfc0055f30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fbfdc00b000 tx=0x7fbfdc006df0).stop 2022-07-21T01:40:24.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe00b3900 msgr2=0x7fbfe013f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe00b3900 0x7fbfe013f890 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fbfd000d000 tx=0x7fbfd0007a00).stop 2022-07-21T01:40:24.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 shutdown_connections 2022-07-21T01:40:24.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe013fd80 0x7fbfe00b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe00b3900 0x7fbfe013f890 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbfc0053a70 0x7fbfc0055f30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 --2- 172.21.15.3:0/434774689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfc8004e70 0x7fbfe00b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 >> 172.21.15.3:0/434774689 conn(0x7fbfe0066f70 msgr2=0x7fbfe01466b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 shutdown_connections 2022-07-21T01:40:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.390+0000 7fbfe6881700 1 -- 172.21.15.3:0/434774689 wait complete. 2022-07-21T01:40:24.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:40:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f55489b0700 1 Processor -- start 2022-07-21T01:40:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f55489b0700 1 -- start start 2022-07-21T01:40:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f55489b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5544147720 0x7f5544147b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f55489b0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55440b42c0 0x7f5544061670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f55489b0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55440b21c0 con 0x7f5544061b60 2022-07-21T01:40:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f55489b0700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55440b2460 con 0x7f55440b42c0 2022-07-21T01:40:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f55489b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55440b2700 con 0x7f5544147720 2022-07-21T01:40:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f554259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55440b42c0 0x7f5544061670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f554259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f55440b42c0 0x7f5544061670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47884/0 (socket says 172.21.15.3:47884) 2022-07-21T01:40:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f554259c700 1 -- 172.21.15.3:0/1142317309 learned_addr learned my addr 172.21.15.3:0/1142317309 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f554359e700 1 --1- 172.21.15.3:0/1142317309 >> v1:172.21.15.3:6789/0 conn(0x7f5544061b60 0x7f55440b97b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56168/0 (socket says 172.21.15.3:56168) 2022-07-21T01:40:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f5542d9d700 1 --2- 172.21.15.3:0/1142317309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5544147720 0x7f5544147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f554259c700 1 -- 172.21.15.3:0/1142317309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5544147720 msgr2=0x7f5544147b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f554259c700 1 --2- 172.21.15.3:0/1142317309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5544147720 0x7f5544147b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f554259c700 1 -- 172.21.15.3:0/1142317309 >> v1:172.21.15.3:6789/0 conn(0x7f5544061b60 legacy=0x7f55440b97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f554259c700 1 -- 172.21.15.3:0/1142317309 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5544147ff0 con 0x7f55440b42c0 2022-07-21T01:40:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f5541d9b700 1 -- 172.21.15.3:0/1142317309 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548846748 0 0) 0x7f55440b21c0 con 0x7f5544061b60 2022-07-21T01:40:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f5542d9d700 1 --2- 172.21.15.3:0/1142317309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5544147720 0x7f5544147b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:40:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.522+0000 7f554259c700 1 --2- 172.21.15.3:0/1142317309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55440b42c0 0x7f5544061670 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f55300032f0 tx=0x7f5530009290).ready entity=mon.0 client_cookie=727fd7598428ca10 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f5541d9b700 1 -- 172.21.15.3:0/1142317309 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553000b8a0 con 0x7f55440b42c0 2022-07-21T01:40:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f5541d9b700 1 -- 172.21.15.3:0/1142317309 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553000ba00 con 0x7f55440b42c0 2022-07-21T01:40:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f5541d9b700 1 -- 172.21.15.3:0/1142317309 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553000bb70 con 0x7f55440b42c0 2022-07-21T01:40:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 -- 172.21.15.3:0/1142317309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55440b42c0 msgr2=0x7f5544061670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 --2- 172.21.15.3:0/1142317309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55440b42c0 0x7f5544061670 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f55300032f0 tx=0x7f5530009290).stop 2022-07-21T01:40:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 -- 172.21.15.3:0/1142317309 shutdown_connections 2022-07-21T01:40:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 --2- 172.21.15.3:0/1142317309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55440b42c0 0x7f5544061670 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 --2- 172.21.15.3:0/1142317309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5544147720 0x7f5544147b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 -- 172.21.15.3:0/1142317309 >> 172.21.15.3:0/1142317309 conn(0x7f554406af80 msgr2=0x7f554405fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 -- 172.21.15.3:0/1142317309 shutdown_connections 2022-07-21T01:40:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 -- 172.21.15.3:0/1142317309 wait complete. 2022-07-21T01:40:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 Processor -- start 2022-07-21T01:40:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 -- start start 2022-07-21T01:40:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55440b42c0 0x7f55440aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5544147720 0x7f55440af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5544144840 0x7f55440afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55440af6a0 con 0x7f5544147720 2022-07-21T01:40:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55441570e0 con 0x7f5544144840 2022-07-21T01:40:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f55489b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5544157330 con 0x7f55440b42c0 2022-07-21T01:40:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5544147720 0x7f55440af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5544147720 0x7f55440af1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47892/0 (socket says 172.21.15.3:47892) 2022-07-21T01:40:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554259c700 1 -- 172.21.15.3:0/105425059 learned_addr learned my addr 172.21.15.3:0/105425059 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f5542d9d700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55440b42c0 0x7f55440aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554359e700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5544144840 0x7f55440afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554259c700 1 -- 172.21.15.3:0/105425059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55440b42c0 msgr2=0x7f55440aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554259c700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55440b42c0 0x7f55440aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554259c700 1 -- 172.21.15.3:0/105425059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5544144840 msgr2=0x7f55440afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554259c700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5544144840 0x7f55440afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554259c700 1 -- 172.21.15.3:0/105425059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5530009530 con 0x7f5544147720 2022-07-21T01:40:24.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f5542d9d700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55440b42c0 0x7f55440aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:24.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.526+0000 7f554259c700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5544147720 0x7f55440af1b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f55440ae8d0 tx=0x7f5530016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:24.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f553bfff700 1 -- 172.21.15.3:0/105425059 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5530021630 con 0x7f5544147720 2022-07-21T01:40:24.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f553bfff700 1 -- 172.21.15.3:0/105425059 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5530021790 con 0x7f5544147720 2022-07-21T01:40:24.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55441575d0 con 0x7f5544147720 2022-07-21T01:40:24.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f553bfff700 1 -- 172.21.15.3:0/105425059 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5530021af0 con 0x7f5544147720 2022-07-21T01:40:24.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55440b0bd0 con 0x7f5544147720 2022-07-21T01:40:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55441482f0 con 0x7f5544147720 2022-07-21T01:40:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f553bfff700 1 -- 172.21.15.3:0/105425059 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5530027020 con 0x7f5544147720 2022-07-21T01:40:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f553bfff700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5520053b00 0x7f5520055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f553bfff700 1 -- 172.21.15.3:0/105425059 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f553006d8f0 con 0x7f5544147720 2022-07-21T01:40:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f5542d9d700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5520053b00 0x7f5520055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.530+0000 7f5542d9d700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5520053b00 0x7f5520055fc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f553400b000 tx=0x7f5534008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.538+0000 7f553bfff700 1 -- 172.21.15.3:0/105425059 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5530044760 con 0x7f5544147720 2022-07-21T01:40:24.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.662+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f55440574e0 con 0x7f5544147720 2022-07-21T01:40:24.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.662+0000 7f553bfff700 1 -- 172.21.15.3:0/105425059 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5530015020 con 0x7f5544147720 2022-07-21T01:40:24.667 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:24.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.662+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5520053b00 msgr2=0x7f5520055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.662+0000 7f55489b0700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5520053b00 0x7f5520055fc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f553400b000 tx=0x7f5534008040).stop 2022-07-21T01:40:24.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.662+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5544147720 msgr2=0x7f55440af1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.662+0000 7f55489b0700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5544147720 0x7f55440af1b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f55440ae8d0 tx=0x7f5530016000).stop 2022-07-21T01:40:24.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.666+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 shutdown_connections 2022-07-21T01:40:24.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.666+0000 7f55489b0700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5520053b00 0x7f5520055fc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.666+0000 7f55489b0700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5544144840 0x7f55440afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.666+0000 7f55489b0700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5544147720 0x7f55440af1b0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.666+0000 7f55489b0700 1 --2- 172.21.15.3:0/105425059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55440b42c0 0x7f55440aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.666+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 >> 172.21.15.3:0/105425059 conn(0x7f554406af80 msgr2=0x7f55441466c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:24.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.666+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 shutdown_connections 2022-07-21T01:40:24.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.666+0000 7f55489b0700 1 -- 172.21.15.3:0/105425059 wait complete. 2022-07-21T01:40:24.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:40:24.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a62b0700 1 Processor -- start 2022-07-21T01:40:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a62b0700 1 -- start start 2022-07-21T01:40:24.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a62b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a00b6420 0x7f03a0061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a62b0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f03a00b41f0 0x7f03a0061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a62b0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a00b60e0 con 0x7f03a0061fe0 2022-07-21T01:40:24.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a62b0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a00b2100 con 0x7f03a00b41f0 2022-07-21T01:40:24.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a62b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a00b23a0 con 0x7f03a00b6420 2022-07-21T01:40:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a5aaf700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f03a0061fe0 0x7f03a01465c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56180/0 (socket says 172.21.15.3:56180) 2022-07-21T01:40:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a5aaf700 1 -- 172.21.15.3:0/2484455521 learned_addr learned my addr 172.21.15.3:0/2484455521 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a4aad700 1 --2- 172.21.15.3:0/2484455521 >> v2:172.21.15.143:3300/0 conn(0x7f03a00b41f0 0x7f03a0061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a52ae700 1 --2- 172.21.15.3:0/2484455521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a00b6420 0x7f03a0061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f0393fff700 1 -- 172.21.15.3:0/2484455521 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461163219 0 0) 0x7f0398001490 con 0x7f03a0061fe0 2022-07-21T01:40:24.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f0393fff700 1 -- 172.21.15.3:0/2484455521 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0384003260 con 0x7f03a0061fe0 2022-07-21T01:40:24.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a52ae700 1 -- 172.21.15.3:0/2484455521 >> v2:172.21.15.143:3300/0 conn(0x7f03a00b41f0 msgr2=0x7f03a0061af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:24.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a52ae700 1 --2- 172.21.15.3:0/2484455521 >> v2:172.21.15.143:3300/0 conn(0x7f03a00b41f0 0x7f03a0061af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a52ae700 1 -- 172.21.15.3:0/2484455521 >> v1:172.21.15.3:6789/0 conn(0x7f03a0061fe0 legacy=0x7f03a01465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.782+0000 7f03a52ae700 1 -- 172.21.15.3:0/2484455521 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a00b2640 con 0x7f03a00b6420 2022-07-21T01:40:24.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a52ae700 1 --2- 172.21.15.3:0/2484455521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a00b6420 0x7f03a0061600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f039c002f70 tx=0x7f039c006df0).ready entity=mon.2 client_cookie=14b6c8a205813bba server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:24.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f0393fff700 1 -- 172.21.15.3:0/2484455521 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f039c00d4a0 con 0x7f03a00b6420 2022-07-21T01:40:24.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f0393fff700 1 -- 172.21.15.3:0/2484455521 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c000e90 con 0x7f03a00b6420 2022-07-21T01:40:24.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f0393fff700 1 -- 172.21.15.3:0/2484455521 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f039c00d7f0 con 0x7f03a00b6420 2022-07-21T01:40:24.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 -- 172.21.15.3:0/2484455521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a00b6420 msgr2=0x7f03a0061600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 --2- 172.21.15.3:0/2484455521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a00b6420 0x7f03a0061600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f039c002f70 tx=0x7f039c006df0).stop 2022-07-21T01:40:24.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 -- 172.21.15.3:0/2484455521 shutdown_connections 2022-07-21T01:40:24.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 --2- 172.21.15.3:0/2484455521 >> v2:172.21.15.143:3300/0 conn(0x7f03a00b41f0 0x7f03a0061af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 --2- 172.21.15.3:0/2484455521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a00b6420 0x7f03a0061600 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 -- 172.21.15.3:0/2484455521 >> 172.21.15.3:0/2484455521 conn(0x7f03a005c620 msgr2=0x7f03a005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:24.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 -- 172.21.15.3:0/2484455521 shutdown_connections 2022-07-21T01:40:24.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 -- 172.21.15.3:0/2484455521 wait complete. 2022-07-21T01:40:24.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 Processor -- start 2022-07-21T01:40:24.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 -- start start 2022-07-21T01:40:24.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a0061fe0 0x7f03a0143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a00b41f0 0x7f03a0152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03a00b6420 0x7f03a01533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a0144570 con 0x7f03a00b6420 2022-07-21T01:40:24.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a00aeeb0 con 0x7f03a00b41f0 2022-07-21T01:40:24.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a62b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a00b02a0 con 0x7f03a0061fe0 2022-07-21T01:40:24.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a5aaf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03a00b6420 0x7f03a01533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a5aaf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03a00b6420 0x7f03a01533a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47904/0 (socket says 172.21.15.3:47904) 2022-07-21T01:40:24.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a5aaf700 1 -- 172.21.15.3:0/194034410 learned_addr learned my addr 172.21.15.3:0/194034410 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:24.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a4aad700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a00b41f0 0x7f03a0152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a52ae700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a0061fe0 0x7f03a0143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a5aaf700 1 -- 172.21.15.3:0/194034410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a0061fe0 msgr2=0x7f03a0143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a5aaf700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a0061fe0 0x7f03a0143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a5aaf700 1 -- 172.21.15.3:0/194034410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a00b41f0 msgr2=0x7f03a0152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a5aaf700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a00b41f0 0x7f03a0152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a5aaf700 1 -- 172.21.15.3:0/194034410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c00b180 con 0x7f03a00b6420 2022-07-21T01:40:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a52ae700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a0061fe0 0x7f03a0143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.786+0000 7f03a5aaf700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03a00b6420 0x7f03a01533a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f03a0143940 tx=0x7f0398007c60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:24.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03927fc700 1 -- 172.21.15.3:0/194034410 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0398005ba0 con 0x7f03a00b6420 2022-07-21T01:40:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03927fc700 1 -- 172.21.15.3:0/194034410 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0398005d00 con 0x7f03a00b6420 2022-07-21T01:40:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a0152700 con 0x7f03a00b6420 2022-07-21T01:40:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03927fc700 1 -- 172.21.15.3:0/194034410 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03980072c0 con 0x7f03a00b6420 2022-07-21T01:40:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a00af160 con 0x7f03a00b6420 2022-07-21T01:40:24.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03927fc700 1 -- 172.21.15.3:0/194034410 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0398010070 con 0x7f03a00b6420 2022-07-21T01:40:24.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03927fc700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0388053b30 0x7f0388055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:24.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03927fc700 1 -- 172.21.15.3:0/194034410 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f039806b960 con 0x7f03a00b6420 2022-07-21T01:40:24.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a00b60e0 con 0x7f03a00b6420 2022-07-21T01:40:24.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03a52ae700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0388053b30 0x7f0388055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:24.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.790+0000 7f03a52ae700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0388053b30 0x7f0388055ff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f039c018000 tx=0x7f039c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:24.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.794+0000 7f03927fc700 1 -- 172.21.15.3:0/194034410 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f03a00b60e0 con 0x7f03a00b6420 2022-07-21T01:40:24.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.922+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03a00b0410 con 0x7f03a00b6420 2022-07-21T01:40:24.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.922+0000 7f03927fc700 1 -- 172.21.15.3:0/194034410 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+10 (secure 0 0 0) 0x7f03a00b0410 con 0x7f03a00b6420 2022-07-21T01:40:24.928 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:24.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0388053b30 msgr2=0x7f0388055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0388053b30 0x7f0388055ff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f039c018000 tx=0x7f039c014000).stop 2022-07-21T01:40:24.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03a00b6420 msgr2=0x7f03a01533a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:24.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03a00b6420 0x7f03a01533a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f03a0143940 tx=0x7f0398007c60).stop 2022-07-21T01:40:24.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 shutdown_connections 2022-07-21T01:40:24.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03a00b6420 0x7f03a01533a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0388053b30 0x7f0388055ff0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a00b41f0 0x7f03a0152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 --2- 172.21.15.3:0/194034410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03a0061fe0 0x7f03a0143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:24.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 >> 172.21.15.3:0/194034410 conn(0x7f03a005c620 msgr2=0x7f03a005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:24.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 shutdown_connections 2022-07-21T01:40:24.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:24.926+0000 7f03a62b0700 1 -- 172.21.15.3:0/194034410 wait complete. 2022-07-21T01:40:24.941 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:40:24.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:25.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f5e13700 1 Processor -- start 2022-07-21T01:40:25.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f5e13700 1 -- start start 2022-07-21T01:40:25.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f5e13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f00b6420 0x7f77f0061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:25.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f5e13700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f77f00b41f0 0x7f77f0061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:25.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f5e13700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f00b60e0 con 0x7f77f0061fe0 2022-07-21T01:40:25.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f5e13700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f00b2100 con 0x7f77f00b41f0 2022-07-21T01:40:25.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f5e13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f00b23a0 con 0x7f77f00b6420 2022-07-21T01:40:25.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f4e11700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f00b6420 0x7f77f0061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:25.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77effff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f77f00b41f0 0x7f77f0061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:25.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f4e11700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f00b6420 0x7f77f0061600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:54826/0 (socket says 172.21.15.3:54826) 2022-07-21T01:40:25.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f5612700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f77f0061fe0 0x7f77f01465c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:58308/0 (socket says 172.21.15.3:58308) 2022-07-21T01:40:25.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77f4e11700 1 -- 172.21.15.3:0/3819798425 learned_addr learned my addr 172.21.15.3:0/3819798425 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:25.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.046+0000 7f77effff700 1 --2- 172.21.15.3:0/3819798425 >> v2:172.21.15.143:3300/0 conn(0x7f77f00b41f0 0x7f77f0061af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:53360/0 (socket says 172.21.15.3:53360) 2022-07-21T01:40:25.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f4e11700 1 -- 172.21.15.3:0/3819798425 >> v2:172.21.15.143:3300/0 conn(0x7f77f00b41f0 msgr2=0x7f77f0061af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:25.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f4e11700 1 --2- 172.21.15.3:0/3819798425 >> v2:172.21.15.143:3300/0 conn(0x7f77f00b41f0 0x7f77f0061af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:25.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f4e11700 1 -- 172.21.15.3:0/3819798425 >> v1:172.21.15.143:6789/0 conn(0x7f77f0061fe0 legacy=0x7f77f01465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:25.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f4e11700 1 -- 172.21.15.3:0/3819798425 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f00b2640 con 0x7f77f00b6420 2022-07-21T01:40:25.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f4e11700 1 --2- 172.21.15.3:0/3819798425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f00b6420 0x7f77f0061600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f77e00032f0 tx=0x7f77e0009290).ready entity=mon.2 client_cookie=250a6ad189acbe5a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:25.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77ef7fe700 1 -- 172.21.15.3:0/3819798425 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77e000b8a0 con 0x7f77f00b6420 2022-07-21T01:40:25.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77ef7fe700 1 -- 172.21.15.3:0/3819798425 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e000ba00 con 0x7f77f00b6420 2022-07-21T01:40:25.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77ef7fe700 1 -- 172.21.15.3:0/3819798425 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77e000bb70 con 0x7f77f00b6420 2022-07-21T01:40:25.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 -- 172.21.15.3:0/3819798425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f00b6420 msgr2=0x7f77f0061600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:25.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 --2- 172.21.15.3:0/3819798425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f00b6420 0x7f77f0061600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f77e00032f0 tx=0x7f77e0009290).stop 2022-07-21T01:40:25.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 -- 172.21.15.3:0/3819798425 shutdown_connections 2022-07-21T01:40:25.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 --2- 172.21.15.3:0/3819798425 >> v2:172.21.15.143:3300/0 conn(0x7f77f00b41f0 0x7f77f0061af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:25.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 --2- 172.21.15.3:0/3819798425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f00b6420 0x7f77f0061600 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:25.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 -- 172.21.15.3:0/3819798425 >> 172.21.15.3:0/3819798425 conn(0x7f77f005c620 msgr2=0x7f77f005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:25.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 -- 172.21.15.3:0/3819798425 shutdown_connections 2022-07-21T01:40:25.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 -- 172.21.15.3:0/3819798425 wait complete. 2022-07-21T01:40:25.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 Processor -- start 2022-07-21T01:40:25.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 -- start start 2022-07-21T01:40:25.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f0061fe0 0x7f77f00af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:25.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f00b41f0 0x7f77f0146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:25.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77f00b6420 0x7f77f0156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:25.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f0141dc0 con 0x7f77f00b41f0 2022-07-21T01:40:25.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f013d700 con 0x7f77f00b6420 2022-07-21T01:40:25.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5e13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f0143b30 con 0x7f77f0061fe0 2022-07-21T01:40:25.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f00b41f0 0x7f77f0146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f4e11700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f0061fe0 0x7f77f00af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f00b41f0 0x7f77f0146a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47918/0 (socket says 172.21.15.3:47918) 2022-07-21T01:40:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77effff700 1 -- 172.21.15.3:0/2905477113 learned_addr learned my addr 172.21.15.3:0/2905477113 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.050+0000 7f77f5612700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77f00b6420 0x7f77f0156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77effff700 1 -- 172.21.15.3:0/2905477113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f0061fe0 msgr2=0x7f77f00af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77effff700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f0061fe0 0x7f77f00af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:25.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77effff700 1 -- 172.21.15.3:0/2905477113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77f00b6420 msgr2=0x7f77f0156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:25.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77effff700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77f00b6420 0x7f77f0156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:25.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77effff700 1 -- 172.21.15.3:0/2905477113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e0009530 con 0x7f77f00b41f0 2022-07-21T01:40:25.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77effff700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f00b41f0 0x7f77f0146a00 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f77d800c000 tx=0x7f77d8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:25.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77edffb700 1 -- 172.21.15.3:0/2905477113 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77d8010590 con 0x7f77f00b41f0 2022-07-21T01:40:25.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77edffb700 1 -- 172.21.15.3:0/2905477113 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d80106f0 con 0x7f77f00b41f0 2022-07-21T01:40:25.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f0143fd0 con 0x7f77f00b41f0 2022-07-21T01:40:25.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77edffb700 1 -- 172.21.15.3:0/2905477113 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77d8010a50 con 0x7f77f00b41f0 2022-07-21T01:40:25.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f0155d80 con 0x7f77f00b41f0 2022-07-21T01:40:25.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f00b28e0 con 0x7f77f00b41f0 2022-07-21T01:40:25.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.054+0000 7f77edffb700 1 -- 172.21.15.3:0/2905477113 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f77d8010bb0 con 0x7f77f00b41f0 2022-07-21T01:40:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.058+0000 7f77edffb700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f77c8053a70 0x7f77c8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.058+0000 7f77edffb700 1 -- 172.21.15.3:0/2905477113 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f77d806bfe0 con 0x7f77f00b41f0 2022-07-21T01:40:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.058+0000 7f77f4e11700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f77c8053a70 0x7f77c8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.058+0000 7f77f4e11700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f77c8053a70 0x7f77c8055f30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f77e001b000 tx=0x7f77e0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.066+0000 7f77edffb700 1 -- 172.21.15.3:0/2905477113 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f77d8042d40 con 0x7f77f00b41f0 2022-07-21T01:40:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.226+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77f0143cf0 con 0x7f77f00b41f0 2022-07-21T01:40:25.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77edffb700 1 -- 172.21.15.3:0/2905477113 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2487 (secure 0 0 0) 0x7f77d8046360 con 0x7f77f00b41f0 2022-07-21T01:40:25.235 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:25.235 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":41,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/2659690520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2659690520},{"type":"v1","addr":"172.21.15.143:6833","nonce":2659690520}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":6600,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2193393317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2193393317},{"type":"v1","addr":"172.21.15.3:6835","nonce":2193393317}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:17.940781+0000","modified":"2022-07-21T01:40:18.943071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6603},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6603":{"gid":6603,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/539587272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":539587272},{"type":"v1","addr":"172.21.15.3:6837","nonce":539587272}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-21T01:40:25.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f77c8053a70 msgr2=0x7f77c8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:25.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f77c8053a70 0x7f77c8055f30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f77e001b000 tx=0x7f77e0016000).stop 2022-07-21T01:40:25.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f00b41f0 msgr2=0x7f77f0146a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:25.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f00b41f0 0x7f77f0146a00 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f77d800c000 tx=0x7f77d8006df0).stop 2022-07-21T01:40:25.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 shutdown_connections 2022-07-21T01:40:25.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f77c8053a70 0x7f77c8055f30 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:25.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77f00b6420 0x7f77f0156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:25.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f00b41f0 0x7f77f0146a00 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:25.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 --2- 172.21.15.3:0/2905477113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77f0061fe0 0x7f77f00af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:25.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 >> 172.21.15.3:0/2905477113 conn(0x7f77f005c620 msgr2=0x7f77f00b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:25.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 shutdown_connections 2022-07-21T01:40:25.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:25.230+0000 7f77f5e13700 1 -- 172.21.15.3:0/2905477113 wait complete. 2022-07-21T01:40:25.245 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 41 2022-07-21T01:40:25.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:40:17.940781+0000', 'modified': '2022-07-21T01:40:18.943071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6603}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6603': {'gid': 6603, 'name': 'a', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/539587272', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 539587272}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 539587272}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:40:25.551 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:40:25.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:40:25.673 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6603,"want_state":"up:active","state":"up:active","rank_uptime":7.7212739160000003,"mdsmap_epoch":41,"osdmap_epoch":90,"osdmap_epoch_barrier":89,"uptime":12.152027849} 2022-07-21T01:40:25.674 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6603, "mdsmap_epoch": 41, "osdmap_epoch": 90, "osdmap_epoch_barrier": 89, "rank_uptime": 7.721273916, "state": "up:active", "uptime": 12.152027849, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:40:25.675 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:40:25.676 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:40:25.676 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:40:25.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:25.677 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:25.687 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:40:25.749 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:25.750 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:25.763 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:25.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:25.765 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:40:25.770 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:40:25.771 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:25.814 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:40:25.831 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:25.832 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:25.843 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:25.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:25.847 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:40:25.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:40:25.886+0000 7f722d063080 -1 init, newargv = 0x559199db2c60 newargc=9ceph-fuse[22543]: starting ceph client 2022-07-21T01:40:25.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr: 2022-07-21T01:40:25.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22543]: starting fuse 2022-07-21T01:40:26.995 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:40:27.014 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:27.015 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:27.025 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:27.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:27.027 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:40:27.078 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:40:27.078 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:40:27.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:27.079 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:40:27.080 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:40:27.080 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:40:27.080 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:40:27.081 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:40:27.081 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:40:27.082 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:40:27.082 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:40:27.083 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:40:27.083 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:40:27.083 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:40:27.084 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:40:27.084 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:40:27.085 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:40:27.085 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:40:27.086 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:40:27.086 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:40:27.086 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:40:27.087 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:40:27.087 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:40:27.088 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:40:27.088 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:40:27.089 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:40:27.134 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.22543.asok 2022-07-21T01:40:27.138 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22543.asok 2022-07-21T01:40:27.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:27.139 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22543.asok status 2022-07-21T01:40:27.229 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:40:27.230 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:40:27.230 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:40:27.230 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:40:27.230 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:40:27.231 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:40:27.231 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:40:27.231 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "22543", 2022-07-21T01:40:27.231 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:40:27.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:40:27.232 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:40:27.232 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:40:27.232 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 6823, 2022-07-21T01:40:27.233 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:40:27.233 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:40:27.233 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:40:27.233 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 6823 2022-07-21T01:40:27.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:40:27.234 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:40:27.234 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:40:27.234 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:40:27.235 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3482325625 2022-07-21T01:40:27.235 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:40:27.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:40:27.235 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:40:27.235 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:40:27.236 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:40:27.236 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3482325625 2022-07-21T01:40:27.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:40:27.236 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.6823 172.21.15.174:0/3482325625", 2022-07-21T01:40:27.237 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/3482325625", 2022-07-21T01:40:27.237 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:40:27.237 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 41, 2022-07-21T01:40:27.237 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 90, 2022-07-21T01:40:27.238 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:40:27.238 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:40:27.238 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:40:27.238 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:40:27.240 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:27.240 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:27.241 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:27.247 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:40:27.248 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:40:27.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:27.249 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:27.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af359e700 1 Processor -- start 2022-07-21T01:40:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af359e700 1 -- start start 2022-07-21T01:40:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40b36e0 0x7f5af4141a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af359e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af4064ef0 con 0x7f5af4144e60 2022-07-21T01:40:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af4066690 con 0x7f5af4064b10 2022-07-21T01:40:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af413ebf0 con 0x7f5af40b36e0 2022-07-21T01:40:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af2d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5af4144e60 0x7f5af4066ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56208/0 (socket says 172.21.15.3:56208) 2022-07-21T01:40:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af2d9d700 1 -- 172.21.15.3:0/1249638671 learned_addr learned my addr 172.21.15.3:0/1249638671 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af1d9b700 1 --2- 172.21.15.3:0/1249638671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40b36e0 0x7f5af4141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af159a700 1 -- 172.21.15.3:0/1249638671 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025765384 0 0) 0x7f5af4064ef0 con 0x7f5af4144e60 2022-07-21T01:40:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af159a700 1 -- 172.21.15.3:0/1249638671 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad4003260 con 0x7f5af4144e60 2022-07-21T01:40:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af159a700 1 -- 172.21.15.3:0/1249638671 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699587209 0 0) 0x7f5af4066690 con 0x7f5af4064b10 2022-07-21T01:40:27.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af159a700 1 -- 172.21.15.3:0/1249638671 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af4064ef0 con 0x7f5af4064b10 2022-07-21T01:40:27.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af1d9b700 1 -- 172.21.15.3:0/1249638671 >> v1:172.21.15.143:6789/0 conn(0x7f5af4064b10 legacy=0x7f5af4141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:27.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af1d9b700 1 -- 172.21.15.3:0/1249638671 >> v1:172.21.15.3:6789/0 conn(0x7f5af4144e60 legacy=0x7f5af4066ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:27.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af159a700 1 -- 172.21.15.3:0/1249638671 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3729621758 0 0) 0x7f5ad4003260 con 0x7f5af4144e60 2022-07-21T01:40:27.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af1d9b700 1 -- 172.21.15.3:0/1249638671 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af40b4850 con 0x7f5af40b36e0 2022-07-21T01:40:27.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af1d9b700 1 --2- 172.21.15.3:0/1249638671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40b36e0 0x7f5af4141a80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5aec001110 tx=0x7f5ae40091d0).ready entity=mon.2 client_cookie=3682d1f1333fff86 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:27.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af159a700 1 -- 172.21.15.3:0/1249638671 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5aec001680 con 0x7f5af40b36e0 2022-07-21T01:40:27.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af159a700 1 -- 172.21.15.3:0/1249638671 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae400f510 con 0x7f5af40b36e0 2022-07-21T01:40:27.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.373+0000 7f5af159a700 1 -- 172.21.15.3:0/1249638671 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ae400f680 con 0x7f5af40b36e0 2022-07-21T01:40:27.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- 172.21.15.3:0/1249638671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40b36e0 msgr2=0x7f5af4141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:27.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 --2- 172.21.15.3:0/1249638671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40b36e0 0x7f5af4141a80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5aec001110 tx=0x7f5ae40091d0).stop 2022-07-21T01:40:27.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- 172.21.15.3:0/1249638671 shutdown_connections 2022-07-21T01:40:27.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 --2- 172.21.15.3:0/1249638671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40b36e0 0x7f5af4141a80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:27.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- 172.21.15.3:0/1249638671 >> 172.21.15.3:0/1249638671 conn(0x7f5af405f840 msgr2=0x7f5af405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:27.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- 172.21.15.3:0/1249638671 shutdown_connections 2022-07-21T01:40:27.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- 172.21.15.3:0/1249638671 wait complete. 2022-07-21T01:40:27.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 Processor -- start 2022-07-21T01:40:27.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- start start 2022-07-21T01:40:27.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af4064b10 0x7f5af4140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:27.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40b36e0 0x7f5af4150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:27.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4144e60 0x7f5af4153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:27.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af41502c0 con 0x7f5af40b36e0 2022-07-21T01:40:27.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af40aec30 con 0x7f5af4144e60 2022-07-21T01:40:27.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af4143870 con 0x7f5af4064b10 2022-07-21T01:40:27.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40b36e0 0x7f5af4150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:27.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40b36e0 0x7f5af4150bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47932/0 (socket says 172.21.15.3:47932) 2022-07-21T01:40:27.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af1d9b700 1 -- 172.21.15.3:0/2827526403 learned_addr learned my addr 172.21.15.3:0/2827526403 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:27.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af259c700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af4064b10 0x7f5af4140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:27.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af2d9d700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4144e60 0x7f5af4153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:27.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af1d9b700 1 -- 172.21.15.3:0/2827526403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af4064b10 msgr2=0x7f5af4140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:27.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af1d9b700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af4064b10 0x7f5af4140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:27.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af1d9b700 1 -- 172.21.15.3:0/2827526403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4144e60 msgr2=0x7f5af4153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af1d9b700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4144e60 0x7f5af4153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af1d9b700 1 -- 172.21.15.3:0/2827526403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4009470 con 0x7f5af40b36e0 2022-07-21T01:40:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af2d9d700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4144e60 0x7f5af4153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af1d9b700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40b36e0 0x7f5af4150bb0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7f5af40ae8b0 tx=0x7f5ae4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5aeb7fe700 1 -- 172.21.15.3:0/2827526403 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ae4011bf0 con 0x7f5af40b36e0 2022-07-21T01:40:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5aeb7fe700 1 -- 172.21.15.3:0/2827526403 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae4011d50 con 0x7f5af40b36e0 2022-07-21T01:40:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af41504b0 con 0x7f5af40b36e0 2022-07-21T01:40:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5aeb7fe700 1 -- 172.21.15.3:0/2827526403 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ae4021630 con 0x7f5af40b36e0 2022-07-21T01:40:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.377+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af4150710 con 0x7f5af40b36e0 2022-07-21T01:40:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.381+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af41508d0 con 0x7f5af40b36e0 2022-07-21T01:40:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.381+0000 7f5aeb7fe700 1 -- 172.21.15.3:0/2827526403 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5ae4027020 con 0x7f5af40b36e0 2022-07-21T01:40:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.381+0000 7f5aeb7fe700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5ad8053aa0 0x7f5ad8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.381+0000 7f5af259c700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5ad8053aa0 0x7f5ad8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.381+0000 7f5aeb7fe700 1 -- 172.21.15.3:0/2827526403 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5ae406d380 con 0x7f5af40b36e0 2022-07-21T01:40:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.381+0000 7f5af259c700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5ad8053aa0 0x7f5ad8055f60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f5adc00d000 tx=0x7f5adc007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.385+0000 7f5aeb7fe700 1 -- 172.21.15.3:0/2827526403 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5ae4044030 con 0x7f5af40b36e0 2022-07-21T01:40:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.545+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5af40574e0 con 0x7f5af40b36e0 2022-07-21T01:40:27.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5aeb7fe700 1 -- 172.21.15.3:0/2827526403 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2487 (secure 0 0 0) 0x7f5ae4015090 con 0x7f5af40b36e0 2022-07-21T01:40:27.554 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:27.555 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":41,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/2659690520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2659690520},{"type":"v1","addr":"172.21.15.143:6833","nonce":2659690520}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":6600,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2193393317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2193393317},{"type":"v1","addr":"172.21.15.3:6835","nonce":2193393317}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:17.940781+0000","modified":"2022-07-21T01:40:18.943071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6603},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6603":{"gid":6603,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/539587272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":539587272},{"type":"v1","addr":"172.21.15.3:6837","nonce":539587272}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-21T01:40:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5ad8053aa0 msgr2=0x7f5ad8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5ad8053aa0 0x7f5ad8055f60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f5adc00d000 tx=0x7f5adc007a20).stop 2022-07-21T01:40:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40b36e0 msgr2=0x7f5af4150bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40b36e0 0x7f5af4150bb0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7f5af40ae8b0 tx=0x7f5ae4016000).stop 2022-07-21T01:40:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 shutdown_connections 2022-07-21T01:40:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5ad8053aa0 0x7f5ad8055f60 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4144e60 0x7f5af4153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5af40b36e0 0x7f5af4150bb0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 --2- 172.21.15.3:0/2827526403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af4064b10 0x7f5af4140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 >> 172.21.15.3:0/2827526403 conn(0x7f5af405f840 msgr2=0x7f5af405cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 shutdown_connections 2022-07-21T01:40:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:27.549+0000 7f5af359e700 1 -- 172.21.15.3:0/2827526403 wait complete. 2022-07-21T01:40:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 41 2022-07-21T01:40:27.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump inode 1 2022-07-21T01:40:27.672 INFO:teuthology.orchestra.run.smithi003.stdout:{"path":"/","ino":1,"rdev":0,"ctime":"2022-07-21T01:40:23.586324+0000","btime":"2022-07-21T01:40:17.945448+0000","mode":17407,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":16,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":0,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-07-21T01:40:17.945448+0000","atime":"0.000000","time_warp_seq":0,"change_attr":1,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-07-21T01:40:23.586324+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-07-21T01:40:17.945448+0000"},"version":3,"file_data_version":0,"xattr_version":1,"backtrace_version":3,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"request":1,"lock":3,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"authpin":1},"nref":9,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":1,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"xlockdone","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":1,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{"gather_set":[],"state":"sync","is_leased":false,"num_rdlocks":1,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent"],"client_caps":[{"client_id":6775,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2},{"client_id":6823,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[]} 2022-07-21T01:40:27.673 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "2022-07-21T01:40:17.945448+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "0.000000", "auth_pins": 2, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "state": "xlockdone", "xlock_by": {} }, "backtrace_version": 3, "btime": "2022-07-21T01:40:17.945448+0000", "change_attr": 1, "client_caps": [ { "client_id": 6775, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" }, { "client_id": 6823, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-07-21T01:40:23.586324+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 16, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 17407, "mtime": "2022-07-21T01:40:17.945448+0000", "nestlock": {}, "nlink": 1, "nref": 9, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/", "pins": { "authpin": 1, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 3, "request": 1 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2022-07-21T01:40:23.586324+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": { "gather_set": [], "is_leased": false, "num_rdlocks": 1, "num_wrlocks": 0, "num_xlocks": 0, "state": "sync", "xlock_by": {} }, "states": [ "auth", "dirty", "dirtyparent" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 0, "truncate_size": 18446744073709551615, "uid": 0, "version": 3, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } 2022-07-21T01:40:27.674 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:40:27.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:27.693 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:27.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:27.735 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:27.757 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:40:27.757 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:27.758 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:27.758 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:27.787 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:40:27.789 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:27.790 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:40:27.791 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:40:27.792 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:27.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[22953]: fuse finished with error 0 and tester_r 0 2022-07-21T01:40:27.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:27.812 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:27.813 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:27.819 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:40:27.820 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:40:27.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:40:27.891 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:27.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:27.906 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:27.907 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:27.916 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:40:27.918 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:40:27.919 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:40:27.920 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:40:27.920 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:27.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22543]: fuse finished with error 0 and tester_r 0 2022-07-21T01:40:27.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:27.940 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:27.941 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:27.990 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:40:27.991 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:40:28.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:40:28.086 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:28.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:40:28.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd84359e700 1 Processor -- start 2022-07-21T01:40:28.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd84359e700 1 -- start start 2022-07-21T01:40:28.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd84359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405d240 0x7fd8440b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd84359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd84413f520 0x7fd8440b2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd84359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84405cd40 con 0x7fd84405c960 2022-07-21T01:40:28.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd84359e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8440ad6d0 con 0x7fd84413f520 2022-07-21T01:40:28.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd84359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd844067f00 con 0x7fd84405d240 2022-07-21T01:40:28.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd842d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd84413f520 0x7fd8440b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd842d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd84413f520 0x7fd8440b2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47942/0 (socket says 172.21.15.3:47942) 2022-07-21T01:40:28.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.205+0000 7fd842d9d700 1 -- 172.21.15.3:0/3991229248 learned_addr learned my addr 172.21.15.3:0/3991229248 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:28.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd83bfff700 1 --2- 172.21.15.3:0/3991229248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405d240 0x7fd8440b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd842d9d700 1 -- 172.21.15.3:0/3991229248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405d240 msgr2=0x7fd8440b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:28.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd842d9d700 1 --2- 172.21.15.3:0/3991229248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405d240 0x7fd8440b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd842d9d700 1 -- 172.21.15.3:0/3991229248 >> v1:172.21.15.143:6789/0 conn(0x7fd84405c960 legacy=0x7fd8440acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd842d9d700 1 -- 172.21.15.3:0/3991229248 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8440b1920 con 0x7fd84413f520 2022-07-21T01:40:28.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd842d9d700 1 --2- 172.21.15.3:0/3991229248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 0x7fd8440b2570 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fd83c0032f0 tx=0x7fd83c009290).ready entity=mon.0 client_cookie=d7744e8b4bd38451 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:28.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd841d9b700 1 -- 172.21.15.3:0/3991229248 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd83c00b8a0 con 0x7fd84413f520 2022-07-21T01:40:28.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd841d9b700 1 -- 172.21.15.3:0/3991229248 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c00ba00 con 0x7fd84413f520 2022-07-21T01:40:28.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd841d9b700 1 -- 172.21.15.3:0/3991229248 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd83c00bb70 con 0x7fd84413f520 2022-07-21T01:40:28.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 -- 172.21.15.3:0/3991229248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 msgr2=0x7fd8440b2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 --2- 172.21.15.3:0/3991229248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 0x7fd8440b2570 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fd83c0032f0 tx=0x7fd83c009290).stop 2022-07-21T01:40:28.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 -- 172.21.15.3:0/3991229248 shutdown_connections 2022-07-21T01:40:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 --2- 172.21.15.3:0/3991229248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 0x7fd8440b2570 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 --2- 172.21.15.3:0/3991229248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405d240 0x7fd8440b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 -- 172.21.15.3:0/3991229248 >> 172.21.15.3:0/3991229248 conn(0x7fd844060310 msgr2=0x7fd8440b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 -- 172.21.15.3:0/3991229248 shutdown_connections 2022-07-21T01:40:28.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 -- 172.21.15.3:0/3991229248 wait complete. 2022-07-21T01:40:28.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 Processor -- start 2022-07-21T01:40:28.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.209+0000 7fd84359e700 1 -- start start 2022-07-21T01:40:28.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405c960 0x7fd8440634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84405d240 0x7fd844140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 0x7fd8441542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440ab0b0 con 0x7fd84413f520 2022-07-21T01:40:28.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd844142b70 con 0x7fd84405d240 2022-07-21T01:40:28.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440abce0 con 0x7fd84405c960 2022-07-21T01:40:28.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd842d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 0x7fd8441542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd842d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 0x7fd8441542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47948/0 (socket says 172.21.15.3:47948) 2022-07-21T01:40:28.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd842d9d700 1 -- 172.21.15.3:0/1336731718 learned_addr learned my addr 172.21.15.3:0/1336731718 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:28.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84259c700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405c960 0x7fd8440634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd83bfff700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84405d240 0x7fd844140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd842d9d700 1 -- 172.21.15.3:0/1336731718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405c960 msgr2=0x7fd8440634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd842d9d700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405c960 0x7fd8440634a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd842d9d700 1 -- 172.21.15.3:0/1336731718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84405d240 msgr2=0x7fd844140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd842d9d700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84405d240 0x7fd844140ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd842d9d700 1 -- 172.21.15.3:0/1336731718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c009530 con 0x7fd84413f520 2022-07-21T01:40:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd83bfff700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84405d240 0x7fd844140ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd842d9d700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 0x7fd8441542e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7fd8441435a0 tx=0x7fd83c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:28.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd83b7fe700 1 -- 172.21.15.3:0/1336731718 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd83c00b8a0 con 0x7fd84413f520 2022-07-21T01:40:28.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd83b7fe700 1 -- 172.21.15.3:0/1336731718 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c021440 con 0x7fd84413f520 2022-07-21T01:40:28.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd844067000 con 0x7fd84413f520 2022-07-21T01:40:28.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd83b7fe700 1 -- 172.21.15.3:0/1336731718 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd83c0217a0 con 0x7fd84413f520 2022-07-21T01:40:28.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd844133160 con 0x7fd84413f520 2022-07-21T01:40:28.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.213+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd844067190 con 0x7fd84413f520 2022-07-21T01:40:28.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.217+0000 7fd83b7fe700 1 -- 172.21.15.3:0/1336731718 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd83c00f070 con 0x7fd84413f520 2022-07-21T01:40:28.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.217+0000 7fd83b7fe700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd828053a70 0x7fd828055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.217+0000 7fd83b7fe700 1 -- 172.21.15.3:0/1336731718 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd83c06d890 con 0x7fd84413f520 2022-07-21T01:40:28.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.217+0000 7fd84259c700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd828053a70 0x7fd828055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.217+0000 7fd84259c700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd828053a70 0x7fd828055f30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fd83400d000 tx=0x7fd834007b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:28.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.221+0000 7fd83b7fe700 1 -- 172.21.15.3:0/1336731718 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd83c044650 con 0x7fd84413f520 2022-07-21T01:40:28.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.349+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd844063190 con 0x7fd84413f520 2022-07-21T01:40:28.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.353+0000 7fd83b7fe700 1 -- 172.21.15.3:0/1336731718 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v90) v1 ==== 81+0+34040 (secure 0 0 0) 0x7fd83c015020 con 0x7fd84413f520 2022-07-21T01:40:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:28.359 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:40:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 90, 2022-07-21T01:40:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:40:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:40:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:40:18.943662+0000", 2022-07-21T01:40:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:40:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:40:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:40:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:40:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:40:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:40:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:40:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:40:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:40:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:40:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:40:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:40:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 16, 2022-07-21T01:40:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:40:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:40:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:40:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:40:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:40:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:40:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:40:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:40:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:40:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:40:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:40:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:40:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:40:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:40:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:40:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:40:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:40:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:40:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:40:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:40:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:40:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:40:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:40:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 15, 2022-07-21T01:40:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:40:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:40:16.360095+0000", 2022-07-21T01:40:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "89", 2022-07-21T01:40:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:28.424 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:28.424 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:28.424 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:28.424 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:28.425 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:28.425 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:28.425 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:40:28.425 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:40:28.425 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:40:28.426 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:40:28.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.426 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:28.426 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:40:28.427 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:40:28.427 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.427 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.428 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 16, 2022-07-21T01:40:28.428 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:40:28.428 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:40:17.262905+0000", 2022-07-21T01:40:28.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:28.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:28.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:28.429 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:28.429 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:28.430 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:28.430 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:28.430 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:28.430 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:28.431 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:28.431 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:28.431 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:28.432 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:28.432 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:28.432 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:28.432 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:28.433 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:28.433 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:28.433 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:28.434 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:28.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.434 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "90", 2022-07-21T01:40:28.434 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:28.435 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:28.435 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:28.435 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:28.435 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:28.436 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:28.436 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:28.436 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:28.436 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:28.436 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:28.437 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:28.437 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:28.437 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:28.437 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:28.438 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:28.438 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:28.438 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:28.438 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:28.439 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:28.439 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:28.439 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:28.439 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:28.439 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:28.440 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:28.440 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:28.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:28.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.441 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:28.441 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:28.441 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:28.441 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:28.441 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:28.442 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:28.442 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:28.442 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:28.442 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:28.443 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:28.443 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:28.443 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:40:28.443 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:28.444 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:40:28.444 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:40:28.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.445 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:28.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:40:28.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:28.446 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:40:28.446 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:28.446 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:28.446 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:28.447 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:28.447 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:28.447 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:28.447 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:28.447 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:28.448 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:28.448 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:28.448 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:28.448 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.449 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.449 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:40:28.449 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:28.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.450 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:40:28.450 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:28.451 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.451 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.451 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:28.451 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.452 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.452 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:40:28.452 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:28.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.453 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.453 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:40:28.453 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:28.454 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.454 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.454 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:28.455 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.455 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.455 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:40:28.456 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:28.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.456 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:40:28.457 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:28.457 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.457 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.458 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:28.458 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.458 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:40:28.459 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:28.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:40:28.460 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:28.460 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.460 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.461 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:40:28.461 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:40:28.461 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:40:28.462 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:40:28.462 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:28.462 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:28.462 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:28.462 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:28.463 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:40:28.464 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:28.464 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:28.464 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:28.464 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:28.464 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:28.465 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:28.465 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:28.465 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 87, 2022-07-21T01:40:28.466 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:28.466 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:28.466 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:28.466 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.467 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.467 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:40:28.468 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:28.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.468 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.469 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:40:28.469 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:28.469 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.469 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.470 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:28.470 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.470 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.471 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:40:28.471 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:28.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:40:28.472 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:28.472 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.472 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.473 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:28.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.474 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:40:28.474 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:28.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.475 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:40:28.475 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:28.475 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.476 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.476 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:28.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.477 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:40:28.477 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:28.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:40:28.478 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:28.479 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.479 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.479 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:40:28.479 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:40:28.480 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:40:28.480 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:40:28.480 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:28.480 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:28.481 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:28.481 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:28.482 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:40:28.482 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:28.482 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:28.482 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:28.483 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:28.483 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:28.483 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:28.483 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:28.484 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:28.484 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:28.484 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:28.484 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:28.484 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.485 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:40:28.485 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:28.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.486 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:40:28.486 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:28.487 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.487 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.487 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:28.488 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.488 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.488 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:40:28.489 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:28.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.489 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.489 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:40:28.490 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:28.490 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.490 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.491 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:28.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.491 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:40:28.492 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:28.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.493 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:40:28.493 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:28.493 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.493 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.494 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:28.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.495 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:40:28.495 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:28.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.496 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.496 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:40:28.496 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:28.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.497 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.497 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:40:28.497 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:40:28.498 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:40:28.498 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:40:28.498 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:28.498 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:28.498 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:28.499 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.499 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:28.500 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:40:28.500 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:28.500 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:28.500 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:28.501 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:28.501 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:28.501 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:28.501 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:28.501 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:28.502 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:28.502 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:28.502 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:28.502 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:40:28.503 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:28.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.504 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:40:28.504 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:28.505 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.505 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.505 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:28.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.506 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:40:28.507 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:28.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:40:28.508 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:28.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.508 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.509 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:28.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.509 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:40:28.510 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:28.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.510 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:40:28.511 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:28.511 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.511 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.512 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:28.512 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.513 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.513 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:40:28.513 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:28.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.514 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.514 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:40:28.514 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:28.514 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.515 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.515 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:40:28.515 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:40:28.516 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:40:28.516 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:40:28.516 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:28.516 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:28.516 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:28.517 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.517 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:28.518 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:40:28.518 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:28.518 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:28.518 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:28.518 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:28.519 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:28.519 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:28.519 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:28.519 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:28.520 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:28.520 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:28.520 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:28.520 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.521 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.521 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:40:28.521 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:28.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.522 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.522 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:40:28.522 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:28.523 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.523 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.523 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:28.523 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.524 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.524 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:40:28.524 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:28.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.525 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.525 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:40:28.525 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:28.526 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.526 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.526 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:28.527 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.527 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:40:28.528 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:28.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.528 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:40:28.529 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:28.529 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.529 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.530 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:28.530 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.530 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.530 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:40:28.531 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:28.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.531 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.532 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:40:28.532 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:28.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.532 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.533 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:40:28.533 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:40:28.533 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:40:28.533 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:40:28.534 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:28.534 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:28.534 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:28.534 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.535 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:28.535 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:40:28.535 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:28.536 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:28.536 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:28.536 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:28.536 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:28.536 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:28.537 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:28.537 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:28.537 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:28.537 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:28.538 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:28.538 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.539 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:40:28.539 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:28.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.539 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.540 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:40:28.540 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:28.540 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.540 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.541 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:28.541 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.542 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.542 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:40:28.542 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:28.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.543 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:40:28.543 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:28.543 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.544 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.544 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:28.544 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.545 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.545 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:40:28.545 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:28.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.546 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.546 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:40:28.546 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:28.547 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.547 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.547 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:28.548 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.548 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:40:28.548 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:28.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.549 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.549 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:40:28.550 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:28.550 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.550 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.551 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:40:28.551 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:40:28.551 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:40:28.551 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:40:28.551 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:28.552 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:28.552 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:28.552 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.553 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:28.553 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:40:28.553 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:28.554 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:28.554 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:28.554 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:28.555 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:28.555 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:28.555 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:28.555 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:28.555 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:28.556 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:28.556 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:28.556 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.557 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:40:28.557 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:28.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.558 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:40:28.558 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:28.559 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.559 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.559 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:28.559 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.560 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.560 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:40:28.560 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:28.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.561 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.561 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:40:28.562 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:28.562 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.562 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.562 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:28.563 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.563 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.563 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:40:28.564 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:28.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.564 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.564 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:40:28.565 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:28.565 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.565 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.566 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:28.566 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.566 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.567 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:40:28.567 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:28.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.568 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.568 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:40:28.568 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:28.568 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.568 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.569 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:40:28.569 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:40:28.569 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:40:28.570 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:40:28.570 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:28.570 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:28.570 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:28.571 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.571 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:28.571 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:40:28.572 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:28.572 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:28.572 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:28.572 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:28.572 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:28.573 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:28.573 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:28.573 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 76, 2022-07-21T01:40:28.573 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:28.574 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:28.574 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:28.574 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.575 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:40:28.575 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:28.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.576 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.576 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:40:28.576 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:28.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.577 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.577 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:28.577 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.578 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:40:28.578 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:28.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.579 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.579 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:40:28.579 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:28.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.580 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.580 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:28.580 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.581 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:40:28.581 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:28.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.582 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.582 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:40:28.582 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:28.583 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.583 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.583 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:28.584 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:28.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.584 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:28.584 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:40:28.584 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:28.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.585 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:28.585 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:40:28.586 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:28.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.586 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.587 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:40:28.587 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:40:28.587 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:40:28.588 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:40:28.588 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:28.588 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:28.588 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:28.588 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:28.589 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.589 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:28.589 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:40:28.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.590 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:28.590 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:28.591 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:28.591 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:28.591 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:28.591 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:28.592 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:40:28.592 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:28.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.593 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:28.593 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:28.594 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:28.594 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:28.594 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:28.594 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:28.595 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:40:28.595 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:28.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.596 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:28.596 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:28.596 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:28.596 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:28.597 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:28.597 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:28.597 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:40:28.597 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:28.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.598 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:28.598 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:28.599 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:28.599 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:28.599 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:28.599 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:28.600 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:40:28.600 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:28.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.601 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:28.601 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:28.601 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:28.602 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:28.602 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:28.602 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:28.602 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:40:28.603 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:28.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.603 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:28.604 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:28.604 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:28.604 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:28.605 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:28.605 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:28.605 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:40:28.605 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:28.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.606 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:28.606 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:28.607 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:28.607 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:28.607 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:28.607 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:28.608 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:40:28.608 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:28.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:28.609 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:28.609 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:28.609 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:28.609 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:28.610 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:28.610 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:28.610 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:40:28.610 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:28.611 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.611 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:28.611 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:40:28.611 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:40:28.612 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:40:28.612 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:40:28.612 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:40:28.612 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:40:28.613 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:40:28.613 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:40:28.613 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:28.613 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:40:28.614 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:40:28.614 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:28.614 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:40:28.614 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:28.615 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:28.615 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:40:28.615 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:40:28.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.615 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:40:28.616 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:28.616 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:28.616 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:40:28.616 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:28.617 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:28.617 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:40:28.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:28.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:28.618 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:40:28.618 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:40:28.618 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:40:28.618 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:40:28.619 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:40:28.619 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:40:28.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd828053a70 msgr2=0x7fd828055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd828053a70 0x7fd828055f30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fd83400d000 tx=0x7fd834007b50).stop 2022-07-21T01:40:28.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 msgr2=0x7fd8441542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 0x7fd8441542e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7fd8441435a0 tx=0x7fd83c016000).stop 2022-07-21T01:40:28.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 shutdown_connections 2022-07-21T01:40:28.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd84413f520 0x7fd8441542e0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd828053a70 0x7fd828055f30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84405d240 0x7fd844140ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 --2- 172.21.15.3:0/1336731718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84405c960 0x7fd8440634a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 >> 172.21.15.3:0/1336731718 conn(0x7fd844060310 msgr2=0x7fd8440b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:28.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 shutdown_connections 2022-07-21T01:40:28.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.357+0000 7fd84359e700 1 -- 172.21.15.3:0/1336731718 wait complete. 2022-07-21T01:40:28.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:28.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6774384700 1 Processor -- start 2022-07-21T01:40:28.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6774384700 1 -- start start 2022-07-21T01:40:28.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6774384700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 0x7f676c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6774384700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f676c148530 0x7f676c0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6774384700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f676c0b04e0 con 0x7f676c0a7250 2022-07-21T01:40:28.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6774384700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c0a9250 con 0x7f676c148530 2022-07-21T01:40:28.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6774384700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0b0780 con 0x7f676c0b1f40 2022-07-21T01:40:28.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6772921700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f676c0a7250 0x7f676c1448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56236/0 (socket says 172.21.15.3:56236) 2022-07-21T01:40:28.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677191f700 1 --2- 172.21.15.3:0/1817227580 >> v2:172.21.15.3:3300/0 conn(0x7f676c148530 0x7f676c0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6772921700 1 -- 172.21.15.3:0/1817227580 learned_addr learned my addr 172.21.15.3:0/1817227580 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:28.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6772120700 1 --2- 172.21.15.3:0/1817227580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 0x7f676c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677111e700 1 -- 172.21.15.3:0/1817227580 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861202924 0 0) 0x7f676c0b04e0 con 0x7f676c0a7250 2022-07-21T01:40:28.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677111e700 1 -- 172.21.15.3:0/1817227580 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6754003260 con 0x7f676c0a7250 2022-07-21T01:40:28.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677191f700 1 -- 172.21.15.3:0/1817227580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 msgr2=0x7f676c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:28.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677191f700 1 --2- 172.21.15.3:0/1817227580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 0x7f676c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677191f700 1 -- 172.21.15.3:0/1817227580 >> v1:172.21.15.3:6789/0 conn(0x7f676c0a7250 legacy=0x7f676c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677191f700 1 -- 172.21.15.3:0/1817227580 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c0b0a20 con 0x7f676c148530 2022-07-21T01:40:28.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677191f700 1 --2- 172.21.15.3:0/1817227580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c148530 0x7f676c0b2320 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f6760002f70 tx=0x7f6760006df0).ready entity=mon.0 client_cookie=5f9f941d4ced51b6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:28.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677111e700 1 -- 172.21.15.3:0/1817227580 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f676000d4a0 con 0x7f676c148530 2022-07-21T01:40:28.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677111e700 1 -- 172.21.15.3:0/1817227580 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6760000e90 con 0x7f676c148530 2022-07-21T01:40:28.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f677111e700 1 -- 172.21.15.3:0/1817227580 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f676000d7f0 con 0x7f676c148530 2022-07-21T01:40:28.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6774384700 1 -- 172.21.15.3:0/1817227580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c148530 msgr2=0x7f676c0b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.733+0000 7f6774384700 1 --2- 172.21.15.3:0/1817227580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c148530 0x7f676c0b2320 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f6760002f70 tx=0x7f6760006df0).stop 2022-07-21T01:40:28.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 -- 172.21.15.3:0/1817227580 shutdown_connections 2022-07-21T01:40:28.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 --2- 172.21.15.3:0/1817227580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c148530 0x7f676c0b2320 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 --2- 172.21.15.3:0/1817227580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 0x7f676c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 -- 172.21.15.3:0/1817227580 >> 172.21.15.3:0/1817227580 conn(0x7f676c05f840 msgr2=0x7f676c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:28.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 -- 172.21.15.3:0/1817227580 shutdown_connections 2022-07-21T01:40:28.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 -- 172.21.15.3:0/1817227580 wait complete. 2022-07-21T01:40:28.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 Processor -- start 2022-07-21T01:40:28.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 -- start start 2022-07-21T01:40:28.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c0a7250 0x7f676c0a9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 0x7f676c15a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f676c148530 0x7f676c15a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c14bca0 con 0x7f676c0a7250 2022-07-21T01:40:28.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c14ba00 con 0x7f676c148530 2022-07-21T01:40:28.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.737+0000 7f6774384700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c14af70 con 0x7f676c0b1f40 2022-07-21T01:40:28.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772120700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c0a7250 0x7f676c0a9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772120700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c0a7250 0x7f676c0a9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47958/0 (socket says 172.21.15.3:47958) 2022-07-21T01:40:28.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772120700 1 -- 172.21.15.3:0/1141489143 learned_addr learned my addr 172.21.15.3:0/1141489143 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:28.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772921700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f676c148530 0x7f676c15a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f677191f700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 0x7f676c15a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772120700 1 -- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 msgr2=0x7f676c15a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772120700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 0x7f676c15a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772120700 1 -- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f676c148530 msgr2=0x7f676c15a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772120700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f676c148530 0x7f676c15a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772120700 1 -- 172.21.15.3:0/1141489143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676000b180 con 0x7f676c0a7250 2022-07-21T01:40:28.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772921700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f676c148530 0x7f676c15a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:28.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f677191f700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 0x7f676c15a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:28.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6772120700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c0a7250 0x7f676c0a9910 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f676400c000 tx=0x7f67640091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:28.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f675f7fe700 1 -- 172.21.15.3:0/1141489143 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f676400c850 con 0x7f676c0a7250 2022-07-21T01:40:28.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f675f7fe700 1 -- 172.21.15.3:0/1141489143 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676400c9b0 con 0x7f676c0a7250 2022-07-21T01:40:28.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c14a130 con 0x7f676c0a7250 2022-07-21T01:40:28.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f675f7fe700 1 -- 172.21.15.3:0/1141489143 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f676400cd10 con 0x7f676c0a7250 2022-07-21T01:40:28.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c149f70 con 0x7f676c0a7250 2022-07-21T01:40:28.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c0b0cc0 con 0x7f676c0a7250 2022-07-21T01:40:28.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.741+0000 7f675f7fe700 1 -- 172.21.15.3:0/1141489143 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6764022070 con 0x7f676c0a7250 2022-07-21T01:40:28.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.745+0000 7f675f7fe700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f674c0539e0 0x7f674c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:28.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.745+0000 7f675f7fe700 1 -- 172.21.15.3:0/1141489143 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f676406c6e0 con 0x7f676c0a7250 2022-07-21T01:40:28.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.745+0000 7f677191f700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f674c0539e0 0x7f674c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:28.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.745+0000 7f677191f700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f674c0539e0 0x7f674c055ea0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f6760018000 tx=0x7f6760014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:28.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.749+0000 7f675f7fe700 1 -- 172.21.15.3:0/1141489143 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f67640434a0 con 0x7f676c0a7250 2022-07-21T01:40:28.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.909+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f676c149f70 con 0x7f676c0a7250 2022-07-21T01:40:28.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.909+0000 7f675f7fe700 1 -- 172.21.15.3:0/1141489143 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2487 (secure 0 0 0) 0x7f676c149f70 con 0x7f676c0a7250 2022-07-21T01:40:28.917 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:28.918 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":41,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/2659690520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2659690520},{"type":"v1","addr":"172.21.15.143:6833","nonce":2659690520}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":6600,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2193393317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2193393317},{"type":"v1","addr":"172.21.15.3:6835","nonce":2193393317}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:17.940781+0000","modified":"2022-07-21T01:40:18.943071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6603},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6603":{"gid":6603,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/539587272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":539587272},{"type":"v1","addr":"172.21.15.3:6837","nonce":539587272}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-21T01:40:28.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f674c0539e0 msgr2=0x7f674c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f674c0539e0 0x7f674c055ea0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f6760018000 tx=0x7f6760014000).stop 2022-07-21T01:40:28.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c0a7250 msgr2=0x7f676c0a9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:28.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c0a7250 0x7f676c0a9910 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f676400c000 tx=0x7f67640091d0).stop 2022-07-21T01:40:28.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 shutdown_connections 2022-07-21T01:40:28.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f674c0539e0 0x7f674c055ea0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f676c148530 0x7f676c15a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f676c0b1f40 0x7f676c15a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 --2- 172.21.15.3:0/1141489143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f676c0a7250 0x7f676c0a9910 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:28.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 >> 172.21.15.3:0/1141489143 conn(0x7f676c05f840 msgr2=0x7f676c1470b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:28.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 shutdown_connections 2022-07-21T01:40:28.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:28.913+0000 7f6774384700 1 -- 172.21.15.3:0/1141489143 wait complete. 2022-07-21T01:40:28.926 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 41 2022-07-21T01:40:28.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:40:29.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 Processor -- start 2022-07-21T01:40:29.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 -- start start 2022-07-21T01:40:29.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fca940ad3f0 0x7fca940ab660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca940afff0 0x7fca94147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca94148090 0x7fca940ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca940abb50 con 0x7fca940ad3f0 2022-07-21T01:40:29.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca940ad7d0 con 0x7fca940afff0 2022-07-21T01:40:29.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca94147650 con 0x7fca94148090 2022-07-21T01:40:29.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fca940ad3f0 0x7fca940ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fca940ad3f0 0x7fca940ab660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47966/0 (socket says 172.21.15.3:47966) 2022-07-21T01:40:29.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9259c700 1 -- 172.21.15.3:0/168821151 learned_addr learned my addr 172.21.15.3:0/168821151 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:29.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca91d9b700 1 --2- 172.21.15.3:0/168821151 >> v2:172.21.15.143:3300/0 conn(0x7fca940afff0 0x7fca94147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca92d9d700 1 --2- 172.21.15.3:0/168821151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca94148090 0x7fca940ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9259c700 1 -- 172.21.15.3:0/168821151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca94148090 msgr2=0x7fca940ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:29.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9259c700 1 --2- 172.21.15.3:0/168821151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca94148090 0x7fca940ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9259c700 1 -- 172.21.15.3:0/168821151 >> v2:172.21.15.143:3300/0 conn(0x7fca940afff0 msgr2=0x7fca94147ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9259c700 1 --2- 172.21.15.3:0/168821151 >> v2:172.21.15.143:3300/0 conn(0x7fca940afff0 0x7fca94147ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9259c700 1 -- 172.21.15.3:0/168821151 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca940b0610 con 0x7fca940ad3f0 2022-07-21T01:40:29.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca91d9b700 1 --2- 172.21.15.3:0/168821151 >> v2:172.21.15.143:3300/0 conn(0x7fca940afff0 0x7fca94147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:29.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9259c700 1 --2- 172.21.15.3:0/168821151 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca940ad3f0 0x7fca940ab660 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7fca7c002f70 tx=0x7fca7c006eb0).ready entity=mon.0 client_cookie=7c00475d7c9b40f7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:29.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9159a700 1 -- 172.21.15.3:0/168821151 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca7c00d570 con 0x7fca940ad3f0 2022-07-21T01:40:29.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9159a700 1 -- 172.21.15.3:0/168821151 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c000e90 con 0x7fca940ad3f0 2022-07-21T01:40:29.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9159a700 1 -- 172.21.15.3:0/168821151 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca7c00d900 con 0x7fca940ad3f0 2022-07-21T01:40:29.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 -- 172.21.15.3:0/168821151 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca940ad3f0 msgr2=0x7fca940ab660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 --2- 172.21.15.3:0/168821151 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca940ad3f0 0x7fca940ab660 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7fca7c002f70 tx=0x7fca7c006eb0).stop 2022-07-21T01:40:29.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 -- 172.21.15.3:0/168821151 shutdown_connections 2022-07-21T01:40:29.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.045+0000 7fca9359e700 1 --2- 172.21.15.3:0/168821151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca94148090 0x7fca940ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 --2- 172.21.15.3:0/168821151 >> v2:172.21.15.143:3300/0 conn(0x7fca940afff0 0x7fca94147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 --2- 172.21.15.3:0/168821151 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca940ad3f0 0x7fca940ab660 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 -- 172.21.15.3:0/168821151 >> 172.21.15.3:0/168821151 conn(0x7fca940613c0 msgr2=0x7fca94062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:29.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 -- 172.21.15.3:0/168821151 shutdown_connections 2022-07-21T01:40:29.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 -- 172.21.15.3:0/168821151 wait complete. 2022-07-21T01:40:29.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 Processor -- start 2022-07-21T01:40:29.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 -- start start 2022-07-21T01:40:29.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca940ad3f0 0x7fca940b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca940afff0 0x7fca940b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca94148090 0x7fca940b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca940b5790 con 0x7fca94148090 2022-07-21T01:40:29.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca94156cb0 con 0x7fca940ad3f0 2022-07-21T01:40:29.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca94156f50 con 0x7fca940afff0 2022-07-21T01:40:29.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca92d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca94148090 0x7fca940b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca92d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca94148090 0x7fca940b5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47976/0 (socket says 172.21.15.3:47976) 2022-07-21T01:40:29.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca92d9d700 1 -- 172.21.15.3:0/3615247314 learned_addr learned my addr 172.21.15.3:0/3615247314 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:29.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca9259c700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca940ad3f0 0x7fca940b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca91d9b700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca940afff0 0x7fca940b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca92d9d700 1 -- 172.21.15.3:0/3615247314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca940afff0 msgr2=0x7fca940b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca92d9d700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca940afff0 0x7fca940b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca92d9d700 1 -- 172.21.15.3:0/3615247314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca940ad3f0 msgr2=0x7fca940b4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca92d9d700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca940ad3f0 0x7fca940b4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.049+0000 7fca92d9d700 1 -- 172.21.15.3:0/3615247314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c00b1e0 con 0x7fca94148090 2022-07-21T01:40:29.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca92d9d700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca94148090 0x7fca940b5910 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7fca8400a000 tx=0x7fca84008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:29.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca8b7fe700 1 -- 172.21.15.3:0/3615247314 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca84011780 con 0x7fca94148090 2022-07-21T01:40:29.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca8b7fe700 1 -- 172.21.15.3:0/3615247314 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca840118e0 con 0x7fca94148090 2022-07-21T01:40:29.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca941571f0 con 0x7fca94148090 2022-07-21T01:40:29.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca8b7fe700 1 -- 172.21.15.3:0/3615247314 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca84011c40 con 0x7fca94148090 2022-07-21T01:40:29.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca941421d0 con 0x7fca94148090 2022-07-21T01:40:29.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca940b08b0 con 0x7fca94148090 2022-07-21T01:40:29.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca8b7fe700 1 -- 172.21.15.3:0/3615247314 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fca84012420 con 0x7fca94148090 2022-07-21T01:40:29.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca8b7fe700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca78053950 0x7fca78055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca8b7fe700 1 -- 172.21.15.3:0/3615247314 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fca8406bd70 con 0x7fca94148090 2022-07-21T01:40:29.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca9259c700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca78053950 0x7fca78055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.053+0000 7fca9259c700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca78053950 0x7fca78055e10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fca7c01c000 tx=0x7fca7c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:29.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.061+0000 7fca8b7fe700 1 -- 172.21.15.3:0/3615247314 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fca84042ad0 con 0x7fca94148090 2022-07-21T01:40:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.221+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fca940574e0 con 0x7fca94148090 2022-07-21T01:40:29.619 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:40:29.613+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:40:29.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.629+0000 7fca8b7fe700 1 -- 172.21.15.3:0/3615247314 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7fca840460f0 con 0x7fca94148090 2022-07-21T01:40:29.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca78053950 msgr2=0x7fca78055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca78053950 0x7fca78055e10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fca7c01c000 tx=0x7fca7c015000).stop 2022-07-21T01:40:29.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca94148090 msgr2=0x7fca940b5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca94148090 0x7fca940b5910 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7fca8400a000 tx=0x7fca84008040).stop 2022-07-21T01:40:29.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 shutdown_connections 2022-07-21T01:40:29.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca94148090 0x7fca940b5910 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca940afff0 0x7fca940b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca78053950 0x7fca78055e10 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 --2- 172.21.15.3:0/3615247314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca940ad3f0 0x7fca940b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 >> 172.21.15.3:0/3615247314 conn(0x7fca940613c0 msgr2=0x7fca941412a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:29.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 shutdown_connections 2022-07-21T01:40:29.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.633+0000 7fca9359e700 1 -- 172.21.15.3:0/3615247314 wait complete. 2022-07-21T01:40:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:40:29.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:29.661 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:40:29.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024eff6700 1 Processor -- start 2022-07-21T01:40:29.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024eff6700 1 -- start start 2022-07-21T01:40:29.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024eff6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480ad670 0x7f02480b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024eff6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f02480af120 0x7f02480b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024eff6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02480b0c40 con 0x7f024805c620 2022-07-21T01:40:29.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024eff6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02480b0ee0 con 0x7f02480af120 2022-07-21T01:40:29.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024eff6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02480ae9e0 con 0x7f02480ad670 2022-07-21T01:40:29.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024e7f5700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f024805c620 0x7f024805f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56276/0 (socket says 172.21.15.3:56276) 2022-07-21T01:40:29.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024e7f5700 1 -- 172.21.15.3:0/2522372208 learned_addr learned my addr 172.21.15.3:0/2522372208 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:29.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024dff4700 1 --2- 172.21.15.3:0/2522372208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480ad670 0x7f02480b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024d7f3700 1 --2- 172.21.15.3:0/2522372208 >> v2:172.21.15.143:3300/0 conn(0x7f02480af120 0x7f02480b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024cff2700 1 -- 172.21.15.3:0/2522372208 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822966831 0 0) 0x7f02480b0c40 con 0x7f024805c620 2022-07-21T01:40:29.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.761+0000 7f024cff2700 1 -- 172.21.15.3:0/2522372208 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0230003260 con 0x7f024805c620 2022-07-21T01:40:29.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024dff4700 1 -- 172.21.15.3:0/2522372208 >> v2:172.21.15.143:3300/0 conn(0x7f02480af120 msgr2=0x7f02480b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:29.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024dff4700 1 --2- 172.21.15.3:0/2522372208 >> v2:172.21.15.143:3300/0 conn(0x7f02480af120 0x7f02480b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024dff4700 1 -- 172.21.15.3:0/2522372208 >> v1:172.21.15.3:6789/0 conn(0x7f024805c620 legacy=0x7f024805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024dff4700 1 -- 172.21.15.3:0/2522372208 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02480aec80 con 0x7f02480ad670 2022-07-21T01:40:29.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024dff4700 1 --2- 172.21.15.3:0/2522372208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480ad670 0x7f02480b0570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f02440032f0 tx=0x7f02440091d0).ready entity=mon.2 client_cookie=2e7ee3a75f2d37e9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024cff2700 1 -- 172.21.15.3:0/2522372208 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f024400b600 con 0x7f02480ad670 2022-07-21T01:40:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024cff2700 1 -- 172.21.15.3:0/2522372208 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024400b760 con 0x7f02480ad670 2022-07-21T01:40:29.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024cff2700 1 -- 172.21.15.3:0/2522372208 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f024400b8d0 con 0x7f02480ad670 2022-07-21T01:40:29.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 -- 172.21.15.3:0/2522372208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480ad670 msgr2=0x7f02480b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 --2- 172.21.15.3:0/2522372208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480ad670 0x7f02480b0570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f02440032f0 tx=0x7f02440091d0).stop 2022-07-21T01:40:29.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 -- 172.21.15.3:0/2522372208 shutdown_connections 2022-07-21T01:40:29.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 --2- 172.21.15.3:0/2522372208 >> v2:172.21.15.143:3300/0 conn(0x7f02480af120 0x7f02480b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 --2- 172.21.15.3:0/2522372208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480ad670 0x7f02480b0570 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 -- 172.21.15.3:0/2522372208 >> 172.21.15.3:0/2522372208 conn(0x7f0248061360 msgr2=0x7f0248062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:29.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 -- 172.21.15.3:0/2522372208 shutdown_connections 2022-07-21T01:40:29.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 -- 172.21.15.3:0/2522372208 wait complete. 2022-07-21T01:40:29.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 Processor -- start 2022-07-21T01:40:29.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.765+0000 7f024eff6700 1 -- start start 2022-07-21T01:40:29.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024eff6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f024805c620 0x7f024813f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024eff6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02480ad670 0x7f024813e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024eff6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480af120 0x7f024813e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024eff6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0248066ce0 con 0x7f024805c620 2022-07-21T01:40:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024eff6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0248142650 con 0x7f02480ad670 2022-07-21T01:40:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024eff6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02481428f0 con 0x7f02480af120 2022-07-21T01:40:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024dff4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f024805c620 0x7f024813f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024dff4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f024805c620 0x7f024813f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47998/0 (socket says 172.21.15.3:47998) 2022-07-21T01:40:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024dff4700 1 -- 172.21.15.3:0/2125761689 learned_addr learned my addr 172.21.15.3:0/2125761689 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024d7f3700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02480ad670 0x7f024813e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024e7f5700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480af120 0x7f024813e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024dff4700 1 -- 172.21.15.3:0/2125761689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480af120 msgr2=0x7f024813e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024dff4700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480af120 0x7f024813e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024dff4700 1 -- 172.21.15.3:0/2125761689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02480ad670 msgr2=0x7f024813e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024dff4700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02480ad670 0x7f024813e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024dff4700 1 -- 172.21.15.3:0/2125761689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0244009470 con 0x7f024805c620 2022-07-21T01:40:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024dff4700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f024805c620 0x7f024813f620 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f02440032f0 tx=0x7f0244016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:29.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f0236ffd700 1 -- 172.21.15.3:0/2125761689 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0244011b60 con 0x7f024805c620 2022-07-21T01:40:29.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f0236ffd700 1 -- 172.21.15.3:0/2125761689 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0244011cc0 con 0x7f024805c620 2022-07-21T01:40:29.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0248142b90 con 0x7f024805c620 2022-07-21T01:40:29.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f0236ffd700 1 -- 172.21.15.3:0/2125761689 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f024401f5e0 con 0x7f024805c620 2022-07-21T01:40:29.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0248066910 con 0x7f024805c620 2022-07-21T01:40:29.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.769+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02480b34e0 con 0x7f024805c620 2022-07-21T01:40:29.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.773+0000 7f0236ffd700 1 -- 172.21.15.3:0/2125761689 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0244026030 con 0x7f024805c620 2022-07-21T01:40:29.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.773+0000 7f0236ffd700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0228053950 0x7f0228055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.773+0000 7f024d7f3700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0228053950 0x7f0228055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.773+0000 7f0236ffd700 1 -- 172.21.15.3:0/2125761689 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f024406cbe0 con 0x7f024805c620 2022-07-21T01:40:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.773+0000 7f024d7f3700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0228053950 0x7f0228055e10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f023800b000 tx=0x7f0238006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.781+0000 7f0236ffd700 1 -- 172.21.15.3:0/2125761689 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02440438f0 con 0x7f024805c620 2022-07-21T01:40:29.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.945+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02480574e0 con 0x7f024805c620 2022-07-21T01:40:29.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.945+0000 7f0236ffd700 1 -- 172.21.15.3:0/2125761689 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+2132 (secure 0 0 0) 0x7f0244024870 con 0x7f024805c620 2022-07-21T01:40:29.952 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:29.953 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":42,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/2659690520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2659690520},{"type":"v1","addr":"172.21.15.143:6833","nonce":2659690520}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":6600,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2193393317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2193393317},{"type":"v1","addr":"172.21.15.3:6835","nonce":2193393317}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:17.940781+0000","modified":"2022-07-21T01:40:29.617765+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-21T01:40:29.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.945+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0228053950 msgr2=0x7f0228055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.945+0000 7f024eff6700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0228053950 0x7f0228055e10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f023800b000 tx=0x7f0238006df0).stop 2022-07-21T01:40:29.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.945+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f024805c620 msgr2=0x7f024813f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:29.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.945+0000 7f024eff6700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f024805c620 0x7f024813f620 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f02440032f0 tx=0x7f0244016000).stop 2022-07-21T01:40:29.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.945+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 shutdown_connections 2022-07-21T01:40:29.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.945+0000 7f024eff6700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02480af120 0x7f024813e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.945+0000 7f024eff6700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0228053950 0x7f0228055e10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.949+0000 7f024eff6700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02480ad670 0x7f024813e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.949+0000 7f024eff6700 1 --2- 172.21.15.3:0/2125761689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f024805c620 0x7f024813f620 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:29.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.949+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 >> 172.21.15.3:0/2125761689 conn(0x7f0248061360 msgr2=0x7f0248142f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:29.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.949+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 shutdown_connections 2022-07-21T01:40:29.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:29.949+0000 7f024eff6700 1 -- 172.21.15.3:0/2125761689 wait complete. 2022-07-21T01:40:29.959 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 42 2022-07-21T01:40:29.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:40:30.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4408fc5700 1 Processor -- start 2022-07-21T01:40:30.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4408fc5700 1 -- start start 2022-07-21T01:40:30.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4408fc5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4404146ae0 0x7f44041403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4408fc5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f44040b4820 0x7f44040b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4408fc5700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4404143cf0 con 0x7f44040b4310 2022-07-21T01:40:30.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4408fc5700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44041476a0 con 0x7f4404146ae0 2022-07-21T01:40:30.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4408fc5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4404143f90 con 0x7f44040b4820 2022-07-21T01:40:30.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f440259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4404146ae0 0x7f44041403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4401d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f44040b4310 0x7f44041408b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56286/0 (socket says 172.21.15.3:56286) 2022-07-21T01:40:30.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4401d9b700 1 -- 172.21.15.3:0/3118174670 learned_addr learned my addr 172.21.15.3:0/3118174670 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:30.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.069+0000 7f4402d9d700 1 --2- 172.21.15.3:0/3118174670 >> v2:172.21.15.143:3300/0 conn(0x7f44040b4820 0x7f44040b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440159a700 1 -- 172.21.15.3:0/3118174670 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015947301 0 0) 0x7f4404143cf0 con 0x7f44040b4310 2022-07-21T01:40:30.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440159a700 1 -- 172.21.15.3:0/3118174670 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e8003260 con 0x7f44040b4310 2022-07-21T01:40:30.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440259c700 1 -- 172.21.15.3:0/3118174670 >> v2:172.21.15.143:3300/0 conn(0x7f44040b4820 msgr2=0x7f44040b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:30.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440259c700 1 --2- 172.21.15.3:0/3118174670 >> v2:172.21.15.143:3300/0 conn(0x7f44040b4820 0x7f44040b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440259c700 1 -- 172.21.15.3:0/3118174670 >> v1:172.21.15.3:6789/0 conn(0x7f44040b4310 legacy=0x7f44041408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440259c700 1 -- 172.21.15.3:0/3118174670 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44040b0e20 con 0x7f4404146ae0 2022-07-21T01:40:30.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440259c700 1 --2- 172.21.15.3:0/3118174670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4404146ae0 0x7f44041403c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f43f4002f70 tx=0x7f43f4006df0).ready entity=mon.0 client_cookie=424dd98c00d4d114 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:30.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440159a700 1 -- 172.21.15.3:0/3118174670 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f400d4f0 con 0x7f4404146ae0 2022-07-21T01:40:30.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440159a700 1 -- 172.21.15.3:0/3118174670 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f4000e90 con 0x7f4404146ae0 2022-07-21T01:40:30.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f440159a700 1 -- 172.21.15.3:0/3118174670 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f400d840 con 0x7f4404146ae0 2022-07-21T01:40:30.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 -- 172.21.15.3:0/3118174670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4404146ae0 msgr2=0x7f44041403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 --2- 172.21.15.3:0/3118174670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4404146ae0 0x7f44041403c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7f43f4002f70 tx=0x7f43f4006df0).stop 2022-07-21T01:40:30.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 -- 172.21.15.3:0/3118174670 shutdown_connections 2022-07-21T01:40:30.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 --2- 172.21.15.3:0/3118174670 >> v2:172.21.15.143:3300/0 conn(0x7f44040b4820 0x7f44040b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 --2- 172.21.15.3:0/3118174670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4404146ae0 0x7f44041403c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 -- 172.21.15.3:0/3118174670 >> 172.21.15.3:0/3118174670 conn(0x7f440405c620 msgr2=0x7f440405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 -- 172.21.15.3:0/3118174670 shutdown_connections 2022-07-21T01:40:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 -- 172.21.15.3:0/3118174670 wait complete. 2022-07-21T01:40:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 Processor -- start 2022-07-21T01:40:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.073+0000 7f4408fc5700 1 -- start start 2022-07-21T01:40:30.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4408fc5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44040b4310 0x7f4404141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4408fc5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040b4820 0x7f44040b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4408fc5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4404146ae0 0x7f4404152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4408fc5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44040ba520 con 0x7f44040b4820 2022-07-21T01:40:30.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4408fc5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44040b7570 con 0x7f4404146ae0 2022-07-21T01:40:30.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4408fc5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44040b7810 con 0x7f44040b4310 2022-07-21T01:40:30.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4401d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040b4820 0x7f44040b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4401d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040b4820 0x7f44040b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48014/0 (socket says 172.21.15.3:48014) 2022-07-21T01:40:30.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4401d9b700 1 -- 172.21.15.3:0/1828929322 learned_addr learned my addr 172.21.15.3:0/1828929322 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:30.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f440259c700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44040b4310 0x7f4404141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4402d9d700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4404146ae0 0x7f4404152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4401d9b700 1 -- 172.21.15.3:0/1828929322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44040b4310 msgr2=0x7f4404141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4401d9b700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44040b4310 0x7f4404141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4401d9b700 1 -- 172.21.15.3:0/1828929322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4404146ae0 msgr2=0x7f4404152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4401d9b700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4404146ae0 0x7f4404152370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4401d9b700 1 -- 172.21.15.3:0/1828929322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f400b180 con 0x7f44040b4820 2022-07-21T01:40:30.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f440259c700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44040b4310 0x7f4404141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:30.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4401d9b700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040b4820 0x7f44040b8260 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f43f8001110 tx=0x7f43f8007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:30.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f43f37fe700 1 -- 172.21.15.3:0/1828929322 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f80070d0 con 0x7f44040b4820 2022-07-21T01:40:30.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f43f37fe700 1 -- 172.21.15.3:0/1828929322 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f8007230 con 0x7f44040b4820 2022-07-21T01:40:30.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44040b7ab0 con 0x7f44040b4820 2022-07-21T01:40:30.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f43f37fe700 1 -- 172.21.15.3:0/1828929322 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f8005610 con 0x7f44040b4820 2022-07-21T01:40:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.077+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4404142c90 con 0x7f44040b4820 2022-07-21T01:40:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.081+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44040574e0 con 0x7f44040b4820 2022-07-21T01:40:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.081+0000 7f43f37fe700 1 -- 172.21.15.3:0/1828929322 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f43f801a070 con 0x7f44040b4820 2022-07-21T01:40:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.081+0000 7f43f37fe700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43e0053b00 0x7f43e0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.081+0000 7f43f37fe700 1 -- 172.21.15.3:0/1828929322 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f43f806b350 con 0x7f44040b4820 2022-07-21T01:40:30.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.081+0000 7f440259c700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43e0053b00 0x7f43e0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.081+0000 7f440259c700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43e0053b00 0x7f43e0055fc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f43f4018000 tx=0x7f43f4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:30.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.085+0000 7f43f37fe700 1 -- 172.21.15.3:0/1828929322 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43f8042060 con 0x7f44040b4820 2022-07-21T01:40:30.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.245+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f44040b7c40 con 0x7f44040b4820 2022-07-21T01:40:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.637+0000 7f43f37fe700 1 -- 172.21.15.3:0/1828929322 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f43f8018020 con 0x7f44040b4820 2022-07-21T01:40:30.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43e0053b00 msgr2=0x7f43e0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43e0053b00 0x7f43e0055fc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f43f4018000 tx=0x7f43f4014000).stop 2022-07-21T01:40:30.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040b4820 msgr2=0x7f44040b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040b4820 0x7f44040b8260 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f43f8001110 tx=0x7f43f8007a60).stop 2022-07-21T01:40:30.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 shutdown_connections 2022-07-21T01:40:30.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43e0053b00 0x7f43e0055fc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4404146ae0 0x7f4404152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040b4820 0x7f44040b8260 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 --2- 172.21.15.3:0/1828929322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44040b4310 0x7f4404141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 >> 172.21.15.3:0/1828929322 conn(0x7f440405c620 msgr2=0x7f44040b6670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:30.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 shutdown_connections 2022-07-21T01:40:30.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.641+0000 7f4408fc5700 1 -- 172.21.15.3:0/1828929322 wait complete. 2022-07-21T01:40:30.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:40:30.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e6a7eb700 1 Processor -- start 2022-07-21T01:40:30.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e6a7eb700 1 -- start start 2022-07-21T01:40:30.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e6a7eb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64060290 0x7f2e640617e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e6a7eb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2e64065b00 0x7f2e64065f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e6a7eb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e64060670 con 0x7f2e64066760 2022-07-21T01:40:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e6a7eb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e640672b0 con 0x7f2e64065b00 2022-07-21T01:40:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e6a7eb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64060e00 con 0x7f2e64060290 2022-07-21T01:40:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e63fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64060290 0x7f2e640617e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e637fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2e64065b00 0x7f2e64065f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e68d88700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f2e64066760 0x7f2e64066ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:58416/0 (socket says 172.21.15.3:58416) 2022-07-21T01:40:30.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.777+0000 7f2e68d88700 1 -- 172.21.15.3:0/4184772340 learned_addr learned my addr 172.21.15.3:0/4184772340 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:30.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e637fe700 1 -- 172.21.15.3:0/4184772340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64060290 msgr2=0x7f2e640617e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:30.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e637fe700 1 --2- 172.21.15.3:0/4184772340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64060290 0x7f2e640617e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e637fe700 1 -- 172.21.15.3:0/4184772340 >> v1:172.21.15.143:6789/0 conn(0x7f2e64066760 legacy=0x7f2e64066ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e62ffd700 1 -- 172.21.15.3:0/4184772340 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629947909 0 0) 0x7f2e64060670 con 0x7f2e64066760 2022-07-21T01:40:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e637fe700 1 -- 172.21.15.3:0/4184772340 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e640610a0 con 0x7f2e64065b00 2022-07-21T01:40:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e63fff700 1 --2- 172.21.15.3:0/4184772340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64060290 0x7f2e640617e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:30.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e637fe700 1 --2- 172.21.15.3:0/4184772340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e64065b00 0x7f2e64065f80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f2e54002f70 tx=0x7f2e54006df0).ready entity=mon.1 client_cookie=f03ff91a2efa1816 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e62ffd700 1 -- 172.21.15.3:0/4184772340 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e5400d570 con 0x7f2e64065b00 2022-07-21T01:40:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e62ffd700 1 -- 172.21.15.3:0/4184772340 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e54000e90 con 0x7f2e64065b00 2022-07-21T01:40:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e62ffd700 1 -- 172.21.15.3:0/4184772340 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e5400d8c0 con 0x7f2e64065b00 2022-07-21T01:40:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/4184772340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e64065b00 msgr2=0x7f2e64065f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 --2- 172.21.15.3:0/4184772340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e64065b00 0x7f2e64065f80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f2e54002f70 tx=0x7f2e54006df0).stop 2022-07-21T01:40:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/4184772340 shutdown_connections 2022-07-21T01:40:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 --2- 172.21.15.3:0/4184772340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e64065b00 0x7f2e64065f80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 --2- 172.21.15.3:0/4184772340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64060290 0x7f2e640617e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/4184772340 >> 172.21.15.3:0/4184772340 conn(0x7f2e6405d0c0 msgr2=0x7f2e6405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/4184772340 shutdown_connections 2022-07-21T01:40:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/4184772340 wait complete. 2022-07-21T01:40:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 Processor -- start 2022-07-21T01:40:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.781+0000 7f2e6a7eb700 1 -- start start 2022-07-21T01:40:30.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e6a7eb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e64060290 0x7f2e64146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e6a7eb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64065b00 0x7f2e641565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e6a7eb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e64066760 0x7f2e64156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e6a7eb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64156160 con 0x7f2e64060290 2022-07-21T01:40:30.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e6a7eb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640a5450 con 0x7f2e64066760 2022-07-21T01:40:30.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e6a7eb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640a4f10 con 0x7f2e64065b00 2022-07-21T01:40:30.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e63fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e64060290 0x7f2e64146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e63fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e64060290 0x7f2e64146ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48026/0 (socket says 172.21.15.3:48026) 2022-07-21T01:40:30.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e63fff700 1 -- 172.21.15.3:0/2553096025 learned_addr learned my addr 172.21.15.3:0/2553096025 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:30.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e637fe700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64065b00 0x7f2e641565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e68d88700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e64066760 0x7f2e64156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e63fff700 1 -- 172.21.15.3:0/2553096025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64065b00 msgr2=0x7f2e641565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e63fff700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64065b00 0x7f2e641565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e63fff700 1 -- 172.21.15.3:0/2553096025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e64066760 msgr2=0x7f2e64156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:30.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e63fff700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e64066760 0x7f2e64156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:30.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e63fff700 1 -- 172.21.15.3:0/2553096025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5800b180 con 0x7f2e64060290 2022-07-21T01:40:30.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e63fff700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e64060290 0x7f2e64146ff0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f2e5800c000 tx=0x7f2e58006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:30.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e617fa700 1 -- 172.21.15.3:0/2553096025 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e5800c7e0 con 0x7f2e64060290 2022-07-21T01:40:30.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e617fa700 1 -- 172.21.15.3:0/2553096025 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e5800c940 con 0x7f2e64060290 2022-07-21T01:40:30.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5400b180 con 0x7f2e64060290 2022-07-21T01:40:30.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e617fa700 1 -- 172.21.15.3:0/2553096025 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e5800cca0 con 0x7f2e64060290 2022-07-21T01:40:30.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e640a9510 con 0x7f2e64060290 2022-07-21T01:40:30.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.785+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e640a6480 con 0x7f2e64060290 2022-07-21T01:40:30.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.789+0000 7f2e617fa700 1 -- 172.21.15.3:0/2553096025 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2e5801c460 con 0x7f2e64060290 2022-07-21T01:40:30.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.789+0000 7f2e617fa700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e44053b00 0x7f2e44055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:30.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.789+0000 7f2e637fe700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e44053b00 0x7f2e44055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:30.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.789+0000 7f2e617fa700 1 -- 172.21.15.3:0/2553096025 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f2e5806c650 con 0x7f2e64060290 2022-07-21T01:40:30.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.789+0000 7f2e637fe700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e44053b00 0x7f2e44055fc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f2e5401a000 tx=0x7f2e54014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:30.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.793+0000 7f2e617fa700 1 -- 172.21.15.3:0/2553096025 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2e58043360 con 0x7f2e64060290 2022-07-21T01:40:30.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:30.921+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2e640574e0 con 0x7f2e64060290 2022-07-21T01:40:31.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.641+0000 7f2e617fa700 1 -- 172.21.15.3:0/2553096025 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v92) v1 ==== 185+0+0 (secure 0 0 0) 0x7f2e58046980 con 0x7f2e64060290 2022-07-21T01:40:31.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e44053b00 msgr2=0x7f2e44055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:31.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e44053b00 0x7f2e44055fc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f2e5401a000 tx=0x7f2e54014000).stop 2022-07-21T01:40:31.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e64060290 msgr2=0x7f2e64146ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:31.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e64060290 0x7f2e64146ff0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f2e5800c000 tx=0x7f2e58006df0).stop 2022-07-21T01:40:31.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 shutdown_connections 2022-07-21T01:40:31.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2e44053b00 0x7f2e44055fc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e64066760 0x7f2e64156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e64065b00 0x7f2e641565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 --2- 172.21.15.3:0/2553096025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e64060290 0x7f2e64146ff0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 >> 172.21.15.3:0/2553096025 conn(0x7f2e6405d0c0 msgr2=0x7f2e64142e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:31.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 shutdown_connections 2022-07-21T01:40:31.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.645+0000 7f2e6a7eb700 1 -- 172.21.15.3:0/2553096025 wait complete. 2022-07-21T01:40:31.653 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:40:31.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:40:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.769+0000 7fe408aac700 1 Processor -- start 2022-07-21T01:40:31.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.769+0000 7fe408aac700 1 -- start start 2022-07-21T01:40:31.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe408aac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe4040ad4b0 0x7fe4040af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:31.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe408aac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4040ad9e0 0x7fe4040a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:31.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe408aac700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4040aff30 con 0x7fe4040af450 2022-07-21T01:40:31.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe408aac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40405c7a0 con 0x7fe4040ad4b0 2022-07-21T01:40:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe408aac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe404140990 con 0x7fe4040ad9e0 2022-07-21T01:40:31.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe4037fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe4040af450 0x7fe4041432e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56314/0 (socket says 172.21.15.3:56314) 2022-07-21T01:40:31.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe4037fe700 1 -- 172.21.15.3:0/3566355227 learned_addr learned my addr 172.21.15.3:0/3566355227 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:31.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe403fff700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4040ad9e0 0x7fe4040a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:31.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe3fbfff700 1 --2- 172.21.15.3:0/3566355227 >> v2:172.21.15.143:3300/0 conn(0x7fe4040ad4b0 0x7fe4040af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:31.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626828316 0 0) 0x7fe4040aff30 con 0x7fe4040af450 2022-07-21T01:40:31.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e4003260 con 0x7fe4040af450 2022-07-21T01:40:31.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3535637134 0 0) 0x7fe3e4003260 con 0x7fe4040af450 2022-07-21T01:40:31.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4040ad9e0 msgr2=0x7fe4040a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:31.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4040ad9e0 0x7fe4040a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 >> v2:172.21.15.143:3300/0 conn(0x7fe4040ad4b0 msgr2=0x7fe4040af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:31.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 --2- 172.21.15.3:0/3566355227 >> v2:172.21.15.143:3300/0 conn(0x7fe4040ad4b0 0x7fe4040af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4041439f0 con 0x7fe4040af450 2022-07-21T01:40:31.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fe3f4001660 con 0x7fe4040af450 2022-07-21T01:40:31.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe403fff700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4040ad9e0 0x7fe4040a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:31.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 >> v1:172.21.15.3:6789/0 conn(0x7fe4040af450 legacy=0x7fe4041432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:31.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e4003750 0x7fe3e4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:31.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4040ad9e0 0x7fe3e4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:31.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e40049f0 0x7fe3e4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:31.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40405c7a0 con 0x7fe3e40049f0 2022-07-21T01:40:31.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe404140990 con 0x7fe3e4003750 2022-07-21T01:40:31.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e400b750 con 0x7fe4040ad9e0 2022-07-21T01:40:31.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe403fff700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e40049f0 0x7fe3e4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:31.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe4037fe700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e4003750 0x7fe3e4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:31.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe3fbfff700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4040ad9e0 0x7fe3e4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:31.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe403fff700 1 -- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4040ad9e0 msgr2=0x7fe3e4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:31.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe403fff700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4040ad9e0 0x7fe3e4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe403fff700 1 -- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e4003750 msgr2=0x7fe3e4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:31.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe403fff700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e4003750 0x7fe3e4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe403fff700 1 -- 172.21.15.3:0/3566355227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc009470 con 0x7fe3e40049f0 2022-07-21T01:40:31.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe403fff700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e40049f0 0x7fe3e4004e70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7fe3fc00c000 tx=0x7fe3fc0091d0).ready entity=mon.0 client_cookie=6b8413612a19f391 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:31.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3fc00e920 con 0x7fe3e40049f0 2022-07-21T01:40:31.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc00ea80 con 0x7fe3e40049f0 2022-07-21T01:40:31.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.773+0000 7fe402ffd700 1 -- 172.21.15.3:0/3566355227 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3fc00ede0 con 0x7fe3e40049f0 2022-07-21T01:40:31.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 -- 172.21.15.3:0/3566355227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e40049f0 msgr2=0x7fe3e4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:31.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 --2- 172.21.15.3:0/3566355227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e40049f0 0x7fe3e4004e70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7fe3fc00c000 tx=0x7fe3fc0091d0).stop 2022-07-21T01:40:31.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe403fff700 1 -- 172.21.15.3:0/3566355227 reap_dead start 2022-07-21T01:40:31.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 -- 172.21.15.3:0/3566355227 shutdown_connections 2022-07-21T01:40:31.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 -- 172.21.15.3:0/3566355227 >> 172.21.15.3:0/3566355227 conn(0x7fe4040612d0 msgr2=0x7fe404062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:31.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 -- 172.21.15.3:0/3566355227 shutdown_connections 2022-07-21T01:40:31.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 -- 172.21.15.3:0/3566355227 wait complete. 2022-07-21T01:40:31.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 Processor -- start 2022-07-21T01:40:31.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 -- start start 2022-07-21T01:40:31.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e40049f0 0x7fe4040a8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:31.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe40414a8c0 0x7fe4040abae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:31.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4040abfd0 0x7fe4040ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:31.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040b5180 con 0x7fe4040abfd0 2022-07-21T01:40:31.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040ac970 con 0x7fe40414a8c0 2022-07-21T01:40:31.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe408aac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040639a0 con 0x7fe3e40049f0 2022-07-21T01:40:31.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe403fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4040abfd0 0x7fe4040ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:31.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe4037fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e40049f0 0x7fe4040a8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:31.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe3fbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe40414a8c0 0x7fe4040abae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:31.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe4037fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e40049f0 0x7fe4040a8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:54962/0 (socket says 172.21.15.3:54962) 2022-07-21T01:40:31.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe4037fe700 1 -- 172.21.15.3:0/2297890422 learned_addr learned my addr 172.21.15.3:0/2297890422 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:31.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.777+0000 7fe403fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4040abfd0 0x7fe4040ac480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48050/0 (socket says 172.21.15.3:48050) 2022-07-21T01:40:31.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe403fff700 1 -- 172.21.15.3:0/2297890422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e40049f0 msgr2=0x7fe4040a8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:31.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe403fff700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e40049f0 0x7fe4040a8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe403fff700 1 -- 172.21.15.3:0/2297890422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe40414a8c0 msgr2=0x7fe4040abae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:31.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe403fff700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe40414a8c0 0x7fe4040abae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:31.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe403fff700 1 -- 172.21.15.3:0/2297890422 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe404067250 con 0x7fe4040abfd0 2022-07-21T01:40:31.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe403fff700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4040abfd0 0x7fe4040ac480 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7fe4040651f0 tx=0x7fe3fc01a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:31.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe4017fa700 1 -- 172.21.15.3:0/2297890422 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3fc00e920 con 0x7fe4040abfd0 2022-07-21T01:40:31.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe4017fa700 1 -- 172.21.15.3:0/2297890422 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc01b450 con 0x7fe4040abfd0 2022-07-21T01:40:31.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe408aac700 1 -- 172.21.15.3:0/2297890422 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4040674f0 con 0x7fe4040abfd0 2022-07-21T01:40:31.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe4017fa700 1 -- 172.21.15.3:0/2297890422 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3fc01b5c0 con 0x7fe4040abfd0 2022-07-21T01:40:31.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe408aac700 1 -- 172.21.15.3:0/2297890422 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe404067790 con 0x7fe4040abfd0 2022-07-21T01:40:31.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe408aac700 1 -- 172.21.15.3:0/2297890422 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4040574e0 con 0x7fe4040abfd0 2022-07-21T01:40:31.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe4017fa700 1 -- 172.21.15.3:0/2297890422 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe3fc01b720 con 0x7fe4040abfd0 2022-07-21T01:40:31.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.781+0000 7fe4017fa700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3e8053aa0 0x7fe3e8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:31.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.785+0000 7fe4017fa700 1 -- 172.21.15.3:0/2297890422 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6068+0+0 (secure 0 0 0) 0x7fe3fc06d8c0 con 0x7fe4040abfd0 2022-07-21T01:40:31.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.785+0000 7fe4037fe700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3e8053aa0 0x7fe3e8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:31.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.785+0000 7fe4037fe700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3e8053aa0 0x7fe3e8055f60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fe3f4005000 tx=0x7fe4040af030).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:31.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.789+0000 7fe4017fa700 1 -- 172.21.15.3:0/2297890422 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe3fc0445f0 con 0x7fe4040abfd0 2022-07-21T01:40:31.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:31.913+0000 7fe408aac700 1 -- 172.21.15.3:0/2297890422 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe404061b20 con 0x7fe4040abfd0 2022-07-21T01:40:32.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:40:32.196 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:40:32.199 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:40:32.201 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:40:32.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.653+0000 7fe4017fa700 1 -- 172.21.15.3:0/2297890422 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v93) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe3fc016020 con 0x7fe4040abfd0 2022-07-21T01:40:32.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 -- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3e8053aa0 msgr2=0x7fe3e8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:32.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3e8053aa0 0x7fe3e8055f60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fe3f4005000 tx=0x7fe4040af030).stop 2022-07-21T01:40:32.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 -- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4040abfd0 msgr2=0x7fe4040ac480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:32.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4040abfd0 0x7fe4040ac480 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7fe4040651f0 tx=0x7fe3fc01a000).stop 2022-07-21T01:40:32.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 -- 172.21.15.3:0/2297890422 shutdown_connections 2022-07-21T01:40:32.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4040abfd0 0x7fe4040ac480 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3e8053aa0 0x7fe3e8055f60 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe40414a8c0 0x7fe4040abae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 --2- 172.21.15.3:0/2297890422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e40049f0 0x7fe4040a8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 -- 172.21.15.3:0/2297890422 >> 172.21.15.3:0/2297890422 conn(0x7fe4040612d0 msgr2=0x7fe404143120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:32.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 -- 172.21.15.3:0/2297890422 shutdown_connections 2022-07-21T01:40:32.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.657+0000 7fe3f8ff9700 1 -- 172.21.15.3:0/2297890422 wait complete. 2022-07-21T01:40:32.667 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:40:32.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:40:32.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd5a74700 1 Processor -- start 2022-07-21T01:40:32.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd5a74700 1 -- start start 2022-07-21T01:40:32.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd5a74700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3bd00619b0 0x7f3bd00b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:32.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd5a74700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3bd00b6ee0 0x7f3bd0143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:32.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd5a74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b5ed0 0x7f3bd00b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:32.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd5a74700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd0142f60 con 0x7f3bd00b6ee0 2022-07-21T01:40:32.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd5a74700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd0061d90 con 0x7f3bd00619b0 2022-07-21T01:40:32.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd5a74700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00ba5a0 con 0x7f3bd00b5ed0 2022-07-21T01:40:32.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bcffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3bd00b6ee0 0x7f3bd0143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:32.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd4a72700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3bd00619b0 0x7f3bd00b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:32.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd5273700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b5ed0 0x7f3bd00b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:32.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd4a72700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3bd00619b0 0x7f3bd00b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:53502/0 (socket says 172.21.15.3:53502) 2022-07-21T01:40:32.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bd4a72700 1 -- 172.21.15.3:0/3576844994 learned_addr learned my addr 172.21.15.3:0/3576844994 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:32.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.785+0000 7f3bcffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3bd00b6ee0 0x7f3bd0143890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48056/0 (socket says 172.21.15.3:48056) 2022-07-21T01:40:32.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5273700 1 -- 172.21.15.3:0/3576844994 >> v2:172.21.15.143:3300/0 conn(0x7f3bd00619b0 msgr2=0x7f3bd00b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:32.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5273700 1 --2- 172.21.15.3:0/3576844994 >> v2:172.21.15.143:3300/0 conn(0x7f3bd00619b0 0x7f3bd00b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5273700 1 -- 172.21.15.3:0/3576844994 >> v2:172.21.15.3:3300/0 conn(0x7f3bd00b6ee0 msgr2=0x7f3bd0143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:32.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5273700 1 --2- 172.21.15.3:0/3576844994 >> v2:172.21.15.3:3300/0 conn(0x7f3bd00b6ee0 0x7f3bd0143890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5273700 1 -- 172.21.15.3:0/3576844994 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd0143150 con 0x7f3bd00b5ed0 2022-07-21T01:40:32.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd4a72700 1 --2- 172.21.15.3:0/3576844994 >> v2:172.21.15.143:3300/0 conn(0x7f3bd00619b0 0x7f3bd00b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:32.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5273700 1 --2- 172.21.15.3:0/3576844994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b5ed0 0x7f3bd00b5190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f3bc80032f0 tx=0x7f3bc80091d0).ready entity=mon.2 client_cookie=64e405f558f7c78d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:32.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bcf7fe700 1 -- 172.21.15.3:0/3576844994 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc800b600 con 0x7f3bd00b5ed0 2022-07-21T01:40:32.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bcf7fe700 1 -- 172.21.15.3:0/3576844994 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc800b760 con 0x7f3bd00b5ed0 2022-07-21T01:40:32.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bcf7fe700 1 -- 172.21.15.3:0/3576844994 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc800b8d0 con 0x7f3bd00b5ed0 2022-07-21T01:40:32.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 -- 172.21.15.3:0/3576844994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b5ed0 msgr2=0x7f3bd00b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:32.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/3576844994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b5ed0 0x7f3bd00b5190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f3bc80032f0 tx=0x7f3bc80091d0).stop 2022-07-21T01:40:32.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 -- 172.21.15.3:0/3576844994 shutdown_connections 2022-07-21T01:40:32.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/3576844994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b5ed0 0x7f3bd00b5190 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/3576844994 >> v2:172.21.15.3:3300/0 conn(0x7f3bd00b6ee0 0x7f3bd0143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/3576844994 >> v2:172.21.15.143:3300/0 conn(0x7f3bd00619b0 0x7f3bd00b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 -- 172.21.15.3:0/3576844994 >> 172.21.15.3:0/3576844994 conn(0x7f3bd006af80 msgr2=0x7f3bd005fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:32.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 -- 172.21.15.3:0/3576844994 shutdown_connections 2022-07-21T01:40:32.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 -- 172.21.15.3:0/3576844994 wait complete. 2022-07-21T01:40:32.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 Processor -- start 2022-07-21T01:40:32.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.789+0000 7f3bd5a74700 1 -- start start 2022-07-21T01:40:32.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd5a74700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bd00619b0 0x7f3bd013e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:32.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd5a74700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00b5ed0 0x7f3bd01449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:32.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd5a74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b6ee0 0x7f3bd0144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:32.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd5a74700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd0146140 con 0x7f3bd00619b0 2022-07-21T01:40:32.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd5a74700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00b9330 con 0x7f3bd00b5ed0 2022-07-21T01:40:32.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd5a74700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00b90c0 con 0x7f3bd00b6ee0 2022-07-21T01:40:32.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd4a72700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bd00619b0 0x7f3bd013e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:32.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd4a72700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bd00619b0 0x7f3bd013e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48060/0 (socket says 172.21.15.3:48060) 2022-07-21T01:40:32.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd4a72700 1 -- 172.21.15.3:0/1860248529 learned_addr learned my addr 172.21.15.3:0/1860248529 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:32.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd5273700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b6ee0 0x7f3bd0144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:32.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bcffff700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00b5ed0 0x7f3bd01449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:32.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd4a72700 1 -- 172.21.15.3:0/1860248529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b6ee0 msgr2=0x7f3bd0144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:32.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd4a72700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b6ee0 0x7f3bd0144e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd4a72700 1 -- 172.21.15.3:0/1860248529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00b5ed0 msgr2=0x7f3bd01449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:32.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd4a72700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00b5ed0 0x7f3bd01449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd4a72700 1 -- 172.21.15.3:0/1860248529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc000b1e0 con 0x7f3bd00619b0 2022-07-21T01:40:32.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd4a72700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bd00619b0 0x7f3bd013e150 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f3bc000c000 tx=0x7f3bc0006eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:32.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bcdffb700 1 -- 172.21.15.3:0/1860248529 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc0012600 con 0x7f3bd00619b0 2022-07-21T01:40:32.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bcdffb700 1 -- 172.21.15.3:0/1860248529 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc0012760 con 0x7f3bd00619b0 2022-07-21T01:40:32.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc8009470 con 0x7f3bd00619b0 2022-07-21T01:40:32.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bcdffb700 1 -- 172.21.15.3:0/1860248529 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc0012ac0 con 0x7f3bd00619b0 2022-07-21T01:40:32.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.793+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd00b8530 con 0x7f3bd00619b0 2022-07-21T01:40:32.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.797+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd00574e0 con 0x7f3bd00619b0 2022-07-21T01:40:32.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.797+0000 7f3bcdffb700 1 -- 172.21.15.3:0/1860248529 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3bc0022070 con 0x7f3bd00619b0 2022-07-21T01:40:32.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.797+0000 7f3bcdffb700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3bb00539e0 0x7f3bb0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:32.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.797+0000 7f3bcdffb700 1 -- 172.21.15.3:0/1860248529 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f3bc006c330 con 0x7f3bd00619b0 2022-07-21T01:40:32.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.797+0000 7f3bcffff700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3bb00539e0 0x7f3bb0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:32.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.797+0000 7f3bcffff700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3bb00539e0 0x7f3bb0055ea0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f3bc400b000 tx=0x7f3bc4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:32.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.805+0000 7f3bcdffb700 1 -- 172.21.15.3:0/1860248529 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3bc0043240 con 0x7f3bd00619b0 2022-07-21T01:40:32.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.929+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3bd005db60 con 0x7f3bd00619b0 2022-07-21T01:40:32.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bcdffb700 1 -- 172.21.15.3:0/1860248529 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3bc0046860 con 0x7f3bd00619b0 2022-07-21T01:40:32.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3bb00539e0 msgr2=0x7f3bb0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:32.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3bb00539e0 0x7f3bb0055ea0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f3bc400b000 tx=0x7f3bc4006df0).stop 2022-07-21T01:40:32.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bd00619b0 msgr2=0x7f3bd013e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:32.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bd00619b0 0x7f3bd013e150 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f3bc000c000 tx=0x7f3bc0006eb0).stop 2022-07-21T01:40:32.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 shutdown_connections 2022-07-21T01:40:32.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00b6ee0 0x7f3bd0144e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3bb00539e0 0x7f3bb0055ea0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00b5ed0 0x7f3bd01449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 --2- 172.21.15.3:0/1860248529 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bd00619b0 0x7f3bd013e150 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:32.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.937+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 >> 172.21.15.3:0/1860248529 conn(0x7f3bd006af80 msgr2=0x7f3bd0141e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:32.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.941+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 shutdown_connections 2022-07-21T01:40:32.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:32.941+0000 7f3bd5a74700 1 -- 172.21.15.3:0/1860248529 wait complete. 2022-07-21T01:40:32.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-07-21T01:40:33.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 Processor -- start 2022-07-21T01:40:33.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 -- start start 2022-07-21T01:40:33.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0e0c0b5ed0 0x7f0e0c0b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c061d90 con 0x7f0e0c0619b0 2022-07-21T01:40:33.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0ba5a0 con 0x7f0e0c0b6ee0 2022-07-21T01:40:33.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0c143fa0 con 0x7f0e0c0b5ed0 2022-07-21T01:40:33.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e11b49700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0e0c0619b0 0x7f0e0c143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56348/0 (socket says 172.21.15.3:56348) 2022-07-21T01:40:33.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e11b49700 1 -- 172.21.15.3:0/4078581814 learned_addr learned my addr 172.21.15.3:0/4078581814 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:33.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e1234a700 1 --2- 172.21.15.3:0/4078581814 >> v2:172.21.15.3:3300/0 conn(0x7f0e0c0b5ed0 0x7f0e0c0b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e1234a700 1 -- 172.21.15.3:0/4078581814 >> v1:172.21.15.143:6789/0 conn(0x7f0e0c0b6ee0 legacy=0x7f0e0c0b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e1234a700 1 -- 172.21.15.3:0/4078581814 >> v1:172.21.15.3:6789/0 conn(0x7f0e0c0619b0 legacy=0x7f0e0c143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e1234a700 1 -- 172.21.15.3:0/4078581814 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c143150 con 0x7f0e0c0b5ed0 2022-07-21T01:40:33.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e1234a700 1 --2- 172.21.15.3:0/4078581814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0b5ed0 0x7f0e0c0b58a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f0e040032f0 tx=0x7f0e040091d0).ready entity=mon.0 client_cookie=a4071dae8e419562 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:33.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e10b47700 1 -- 172.21.15.3:0/4078581814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e04009ea0 con 0x7f0e0c0b5ed0 2022-07-21T01:40:33.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e10b47700 1 -- 172.21.15.3:0/4078581814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0400b8a0 con 0x7f0e0c0b5ed0 2022-07-21T01:40:33.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e10b47700 1 -- 172.21.15.3:0/4078581814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e0400ba10 con 0x7f0e0c0b5ed0 2022-07-21T01:40:33.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 -- 172.21.15.3:0/4078581814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0b5ed0 msgr2=0x7f0e0c0b58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 --2- 172.21.15.3:0/4078581814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0b5ed0 0x7f0e0c0b58a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f0e040032f0 tx=0x7f0e040091d0).stop 2022-07-21T01:40:33.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 -- 172.21.15.3:0/4078581814 shutdown_connections 2022-07-21T01:40:33.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.053+0000 7f0e12b4b700 1 --2- 172.21.15.3:0/4078581814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0b5ed0 0x7f0e0c0b58a0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 -- 172.21.15.3:0/4078581814 >> 172.21.15.3:0/4078581814 conn(0x7f0e0c06af80 msgr2=0x7f0e0c05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:33.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 -- 172.21.15.3:0/4078581814 shutdown_connections 2022-07-21T01:40:33.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 -- 172.21.15.3:0/4078581814 wait complete. 2022-07-21T01:40:33.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 Processor -- start 2022-07-21T01:40:33.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 -- start start 2022-07-21T01:40:33.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0619b0 0x7f0e0c0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0c0b5ed0 0x7f0e0c156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e0c0b6ee0 0x7f0e0c152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0af930 con 0x7f0e0c0619b0 2022-07-21T01:40:33.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c112b90 con 0x7f0e0c0b5ed0 2022-07-21T01:40:33.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e12b4b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1569f0 con 0x7f0e0c0b6ee0 2022-07-21T01:40:33.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11b49700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0619b0 0x7f0e0c0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11b49700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0619b0 0x7f0e0c0b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48076/0 (socket says 172.21.15.3:48076) 2022-07-21T01:40:33.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11b49700 1 -- 172.21.15.3:0/2860863219 learned_addr learned my addr 172.21.15.3:0/2860863219 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:33.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11348700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0c0b5ed0 0x7f0e0c156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e1234a700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e0c0b6ee0 0x7f0e0c152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11b49700 1 -- 172.21.15.3:0/2860863219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e0c0b6ee0 msgr2=0x7f0e0c152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11b49700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e0c0b6ee0 0x7f0e0c152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11b49700 1 -- 172.21.15.3:0/2860863219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0c0b5ed0 msgr2=0x7f0e0c156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11b49700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0c0b5ed0 0x7f0e0c156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11b49700 1 -- 172.21.15.3:0/2860863219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e04009470 con 0x7f0e0c0619b0 2022-07-21T01:40:33.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.057+0000 7f0e11b49700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0619b0 0x7f0e0c0b04b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f0e0c0b41b0 tx=0x7f0e08007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:33.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.061+0000 7f0dfaffd700 1 -- 172.21.15.3:0/2860863219 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e080070d0 con 0x7f0e0c0619b0 2022-07-21T01:40:33.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.061+0000 7f0dfaffd700 1 -- 172.21.15.3:0/2860863219 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e08007230 con 0x7f0e0c0619b0 2022-07-21T01:40:33.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.061+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c146f20 con 0x7f0e0c0619b0 2022-07-21T01:40:33.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.061+0000 7f0dfaffd700 1 -- 172.21.15.3:0/2860863219 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e080055d0 con 0x7f0e0c0619b0 2022-07-21T01:40:33.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.061+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c146360 con 0x7f0e0c0619b0 2022-07-21T01:40:33.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.061+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c1473c0 con 0x7f0e0c0619b0 2022-07-21T01:40:33.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.061+0000 7f0dfaffd700 1 -- 172.21.15.3:0/2860863219 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e08010070 con 0x7f0e0c0619b0 2022-07-21T01:40:33.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.065+0000 7f0dfaffd700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0dec053a70 0x7f0dec055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.065+0000 7f0e11348700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0dec053a70 0x7f0dec055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.065+0000 7f0dfaffd700 1 -- 172.21.15.3:0/2860863219 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f0e0806b2f0 con 0x7f0e0c0619b0 2022-07-21T01:40:33.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.065+0000 7f0e11348700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0dec053a70 0x7f0dec055f30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0dfc00d000 tx=0x7f0dfc007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:33.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.069+0000 7f0dfaffd700 1 -- 172.21.15.3:0/2860863219 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e08068e90 con 0x7f0e0c0619b0 2022-07-21T01:40:33.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.241+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7f0e0c146520 con 0x7f0e0c0619b0 2022-07-21T01:40:33.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.657+0000 7f0dfaffd700 1 -- 172.21.15.3:0/2860863219 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v313) v1 ==== 126+0+0 (secure 0 0 0) 0x7f0e08042280 con 0x7f0e0c0619b0 2022-07-21T01:40:33.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0dec053a70 msgr2=0x7f0dec055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0dec053a70 0x7f0dec055f30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0dfc00d000 tx=0x7f0dfc007b90).stop 2022-07-21T01:40:33.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0619b0 msgr2=0x7f0e0c0b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0619b0 0x7f0e0c0b04b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f0e0c0b41b0 tx=0x7f0e08007c30).stop 2022-07-21T01:40:33.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 shutdown_connections 2022-07-21T01:40:33.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e0c0b6ee0 0x7f0e0c152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0dec053a70 0x7f0dec055f30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0c0b5ed0 0x7f0e0c156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 --2- 172.21.15.3:0/2860863219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e0c0619b0 0x7f0e0c0b04b0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 >> 172.21.15.3:0/2860863219 conn(0x7f0e0c06af80 msgr2=0x7f0e0c1405e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:33.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 shutdown_connections 2022-07-21T01:40:33.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.661+0000 7f0e12b4b700 1 -- 172.21.15.3:0/2860863219 wait complete. 2022-07-21T01:40:33.680 INFO:tasks.cephfs_test_runner:test_dump_inode (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-21T01:40:33.681 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) 2022-07-21T01:40:33.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-07-21T01:40:33.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.789+0000 7f3b28876700 1 Processor -- start 2022-07-21T01:40:33.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.789+0000 7f3b28876700 1 -- start start 2022-07-21T01:40:33.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.789+0000 7f3b28876700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240ab2f0 0x7f3b24067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b24065af0 con 0x7f3b240b4ee0 2022-07-21T01:40:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240650a0 con 0x7f3b240aaf10 2022-07-21T01:40:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240b3e70 con 0x7f3b240ab2f0 2022-07-21T01:40:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b23fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3b240b4ee0 0x7f3b240b76b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56366/0 (socket says 172.21.15.3:56366) 2022-07-21T01:40:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b23fff700 1 -- 172.21.15.3:0/17976237 learned_addr learned my addr 172.21.15.3:0/17976237 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b1affd700 1 --2- 172.21.15.3:0/17976237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240ab2f0 0x7f3b24067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b22ffd700 1 -- 172.21.15.3:0/17976237 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563174165 0 0) 0x7f3b24065af0 con 0x7f3b240b4ee0 2022-07-21T01:40:33.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b22ffd700 1 -- 172.21.15.3:0/17976237 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b04003260 con 0x7f3b240b4ee0 2022-07-21T01:40:33.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b22ffd700 1 -- 172.21.15.3:0/17976237 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190119738 0 0) 0x7f3b240650a0 con 0x7f3b240aaf10 2022-07-21T01:40:33.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b1affd700 1 -- 172.21.15.3:0/17976237 >> v1:172.21.15.143:6789/0 conn(0x7f3b240aaf10 legacy=0x7f3b24064990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b1affd700 1 -- 172.21.15.3:0/17976237 >> v1:172.21.15.3:6789/0 conn(0x7f3b240b4ee0 legacy=0x7f3b240b76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b1affd700 1 -- 172.21.15.3:0/17976237 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b24066ff0 con 0x7f3b240ab2f0 2022-07-21T01:40:33.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b1affd700 1 --2- 172.21.15.3:0/17976237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240ab2f0 0x7f3b24067490 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f3b0c002f70 tx=0x7f3b0c006d90).ready entity=mon.2 client_cookie=3131aff72568f828 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:33.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b22ffd700 1 -- 172.21.15.3:0/17976237 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b0c00d4a0 con 0x7f3b240ab2f0 2022-07-21T01:40:33.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b22ffd700 1 -- 172.21.15.3:0/17976237 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0c000e90 con 0x7f3b240ab2f0 2022-07-21T01:40:33.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b22ffd700 1 -- 172.21.15.3:0/17976237 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b0c00d7f0 con 0x7f3b240ab2f0 2022-07-21T01:40:33.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 -- 172.21.15.3:0/17976237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240ab2f0 msgr2=0x7f3b24067490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 --2- 172.21.15.3:0/17976237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240ab2f0 0x7f3b24067490 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f3b0c002f70 tx=0x7f3b0c006d90).stop 2022-07-21T01:40:33.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 -- 172.21.15.3:0/17976237 shutdown_connections 2022-07-21T01:40:33.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 --2- 172.21.15.3:0/17976237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240ab2f0 0x7f3b24067490 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 -- 172.21.15.3:0/17976237 >> 172.21.15.3:0/17976237 conn(0x7f3b24062d70 msgr2=0x7f3b2405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:33.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 -- 172.21.15.3:0/17976237 shutdown_connections 2022-07-21T01:40:33.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.793+0000 7f3b28876700 1 -- 172.21.15.3:0/17976237 wait complete. 2022-07-21T01:40:33.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 Processor -- start 2022-07-21T01:40:33.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 -- start start 2022-07-21T01:40:33.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240aaf10 0x7f3b240af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b240ab2f0 0x7f3b24150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b240b4ee0 0x7f3b24153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240ac5c0 con 0x7f3b240b4ee0 2022-07-21T01:40:33.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240aca90 con 0x7f3b240ab2f0 2022-07-21T01:40:33.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24150530 con 0x7f3b240aaf10 2022-07-21T01:40:33.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b23fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b240b4ee0 0x7f3b24153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b23fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b240b4ee0 0x7f3b24153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48090/0 (socket says 172.21.15.3:48090) 2022-07-21T01:40:33.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b23fff700 1 -- 172.21.15.3:0/1801800276 learned_addr learned my addr 172.21.15.3:0/1801800276 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:33.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b1affd700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b240ab2f0 0x7f3b24150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b237fe700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240aaf10 0x7f3b240af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b23fff700 1 -- 172.21.15.3:0/1801800276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240aaf10 msgr2=0x7f3b240af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b23fff700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240aaf10 0x7f3b240af720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b23fff700 1 -- 172.21.15.3:0/1801800276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b240ab2f0 msgr2=0x7f3b24150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:33.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b23fff700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b240ab2f0 0x7f3b24150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:33.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b23fff700 1 -- 172.21.15.3:0/1801800276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c00b120 con 0x7f3b240b4ee0 2022-07-21T01:40:33.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b237fe700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240aaf10 0x7f3b240af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:33.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b23fff700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b240b4ee0 0x7f3b24153de0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f3b1c001490 tx=0x7f3b1c007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:33.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b217fa700 1 -- 172.21.15.3:0/1801800276 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b1c005ba0 con 0x7f3b240b4ee0 2022-07-21T01:40:33.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b217fa700 1 -- 172.21.15.3:0/1801800276 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c005d00 con 0x7f3b240b4ee0 2022-07-21T01:40:33.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b240ac7f0 con 0x7f3b240b4ee0 2022-07-21T01:40:33.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b217fa700 1 -- 172.21.15.3:0/1801800276 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b1c0072c0 con 0x7f3b240b4ee0 2022-07-21T01:40:33.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.797+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b24140400 con 0x7f3b240b4ee0 2022-07-21T01:40:33.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.801+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b240abc40 con 0x7f3b240b4ee0 2022-07-21T01:40:33.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.801+0000 7f3b217fa700 1 -- 172.21.15.3:0/1801800276 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3b1c010070 con 0x7f3b240b4ee0 2022-07-21T01:40:33.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.801+0000 7f3b217fa700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b08053a70 0x7f3b08055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:33.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.801+0000 7f3b237fe700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b08053a70 0x7f3b08055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:33.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.801+0000 7f3b217fa700 1 -- 172.21.15.3:0/1801800276 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f3b1c06b590 con 0x7f3b240b4ee0 2022-07-21T01:40:33.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.801+0000 7f3b237fe700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b08053a70 0x7f3b08055f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f3b1400b000 tx=0x7f3b14009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:33.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.809+0000 7f3b217fa700 1 -- 172.21.15.3:0/1801800276 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3b1c042630 con 0x7f3b240b4ee0 2022-07-21T01:40:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:33.981+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7f3b240574e0 con 0x7f3b240b4ee0 2022-07-21T01:40:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.665+0000 7f3b217fa700 1 -- 172.21.15.3:0/1801800276 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v314) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3b1c045c50 con 0x7f3b240b4ee0 2022-07-21T01:40:34.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b08053a70 msgr2=0x7f3b08055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:34.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b08053a70 0x7f3b08055f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f3b1400b000 tx=0x7f3b14009040).stop 2022-07-21T01:40:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b240b4ee0 msgr2=0x7f3b24153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b240b4ee0 0x7f3b24153de0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f3b1c001490 tx=0x7f3b1c007d40).stop 2022-07-21T01:40:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 shutdown_connections 2022-07-21T01:40:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b240b4ee0 0x7f3b24153de0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b08053a70 0x7f3b08055f30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b240ab2f0 0x7f3b24150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 --2- 172.21.15.3:0/1801800276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b240aaf10 0x7f3b240af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 >> 172.21.15.3:0/1801800276 conn(0x7f3b24062d70 msgr2=0x7f3b240b7680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:34.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 shutdown_connections 2022-07-21T01:40:34.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.669+0000 7f3b28876700 1 -- 172.21.15.3:0/1801800276 wait complete. 2022-07-21T01:40:34.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:40:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.797+0000 7f0ef288d700 1 Processor -- start 2022-07-21T01:40:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.797+0000 7f0ef288d700 1 -- start start 2022-07-21T01:40:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0eec0b6420 0x7f0eec061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b41f0 0x7f0eec061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0eec061fe0 0x7f0eec1465c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eec0b60e0 con 0x7f0eec0b6420 2022-07-21T01:40:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eec0b2100 con 0x7f0eec061fe0 2022-07-21T01:40:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec0b23a0 con 0x7f0eec0b41f0 2022-07-21T01:40:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef188b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0eec0b6420 0x7f0eec061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef188b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0eec0b6420 0x7f0eec061600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48098/0 (socket says 172.21.15.3:48098) 2022-07-21T01:40:34.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef188b700 1 -- 172.21.15.3:0/1452288510 learned_addr learned my addr 172.21.15.3:0/1452288510 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:34.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef208c700 1 --2- 172.21.15.3:0/1452288510 >> v2:172.21.15.143:3300/0 conn(0x7f0eec061fe0 0x7f0eec1465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:34.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef108a700 1 --2- 172.21.15.3:0/1452288510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b41f0 0x7f0eec061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:34.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef188b700 1 -- 172.21.15.3:0/1452288510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b41f0 msgr2=0x7f0eec061af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef188b700 1 --2- 172.21.15.3:0/1452288510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b41f0 0x7f0eec061af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef188b700 1 -- 172.21.15.3:0/1452288510 >> v2:172.21.15.143:3300/0 conn(0x7f0eec061fe0 msgr2=0x7f0eec1465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef188b700 1 --2- 172.21.15.3:0/1452288510 >> v2:172.21.15.143:3300/0 conn(0x7f0eec061fe0 0x7f0eec1465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef188b700 1 -- 172.21.15.3:0/1452288510 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec0b2640 con 0x7f0eec0b6420 2022-07-21T01:40:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef188b700 1 --2- 172.21.15.3:0/1452288510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec0b6420 0x7f0eec061600 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f0ee8002f70 tx=0x7f0ee8006eb0).ready entity=mon.0 client_cookie=e7af63c038c1d549 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef0889700 1 -- 172.21.15.3:0/1452288510 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ee800d570 con 0x7f0eec0b6420 2022-07-21T01:40:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef0889700 1 -- 172.21.15.3:0/1452288510 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee8000e90 con 0x7f0eec0b6420 2022-07-21T01:40:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef0889700 1 -- 172.21.15.3:0/1452288510 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ee800d900 con 0x7f0eec0b6420 2022-07-21T01:40:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 -- 172.21.15.3:0/1452288510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec0b6420 msgr2=0x7f0eec061600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 --2- 172.21.15.3:0/1452288510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec0b6420 0x7f0eec061600 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f0ee8002f70 tx=0x7f0ee8006eb0).stop 2022-07-21T01:40:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 -- 172.21.15.3:0/1452288510 shutdown_connections 2022-07-21T01:40:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 --2- 172.21.15.3:0/1452288510 >> v2:172.21.15.143:3300/0 conn(0x7f0eec061fe0 0x7f0eec1465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 --2- 172.21.15.3:0/1452288510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b41f0 0x7f0eec061af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 --2- 172.21.15.3:0/1452288510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec0b6420 0x7f0eec061600 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 -- 172.21.15.3:0/1452288510 >> 172.21.15.3:0/1452288510 conn(0x7f0eec05c620 msgr2=0x7f0eec05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 -- 172.21.15.3:0/1452288510 shutdown_connections 2022-07-21T01:40:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.801+0000 7f0ef288d700 1 -- 172.21.15.3:0/1452288510 wait complete. 2022-07-21T01:40:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef288d700 1 Processor -- start 2022-07-21T01:40:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef288d700 1 -- start start 2022-07-21T01:40:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef288d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec061fe0 0x7f0eec0b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef288d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0eec0b41f0 0x7f0eec0b7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef288d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b6420 0x7f0eec12aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef288d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec142f30 con 0x7f0eec061fe0 2022-07-21T01:40:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef288d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec0b0750 con 0x7f0eec0b41f0 2022-07-21T01:40:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef288d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec143850 con 0x7f0eec0b6420 2022-07-21T01:40:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef188b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec061fe0 0x7f0eec0b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef188b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec061fe0 0x7f0eec0b0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48104/0 (socket says 172.21.15.3:48104) 2022-07-21T01:40:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef188b700 1 -- 172.21.15.3:0/2592090496 learned_addr learned my addr 172.21.15.3:0/2592090496 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef208c700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b6420 0x7f0eec12aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef108a700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0eec0b41f0 0x7f0eec0b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef188b700 1 -- 172.21.15.3:0/2592090496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b6420 msgr2=0x7f0eec12aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef188b700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b6420 0x7f0eec12aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef188b700 1 -- 172.21.15.3:0/2592090496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0eec0b41f0 msgr2=0x7f0eec0b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef188b700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0eec0b41f0 0x7f0eec0b7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef188b700 1 -- 172.21.15.3:0/2592090496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee800b1e0 con 0x7f0eec061fe0 2022-07-21T01:40:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef208c700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b6420 0x7f0eec12aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef188b700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec061fe0 0x7f0eec0b0140 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f0ee8002f70 tx=0x7f0ee8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0edeffd700 1 -- 172.21.15.3:0/2592090496 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ee80117e0 con 0x7f0eec061fe0 2022-07-21T01:40:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0edeffd700 1 -- 172.21.15.3:0/2592090496 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee8011940 con 0x7f0eec061fe0 2022-07-21T01:40:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.805+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eec143af0 con 0x7f0eec061fe0 2022-07-21T01:40:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.809+0000 7f0edeffd700 1 -- 172.21.15.3:0/2592090496 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ee8011ca0 con 0x7f0eec061fe0 2022-07-21T01:40:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.809+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eec144230 con 0x7f0eec061fe0 2022-07-21T01:40:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.809+0000 7f0edeffd700 1 -- 172.21.15.3:0/2592090496 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0ee8025030 con 0x7f0eec061fe0 2022-07-21T01:40:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.809+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eec0b28e0 con 0x7f0eec061fe0 2022-07-21T01:40:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.809+0000 7f0edeffd700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ecc0539e0 0x7f0ecc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.809+0000 7f0edeffd700 1 -- 172.21.15.3:0/2592090496 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f0ee806cf60 con 0x7f0eec061fe0 2022-07-21T01:40:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.809+0000 7f0ef108a700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ecc0539e0 0x7f0ecc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.809+0000 7f0ef108a700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ecc0539e0 0x7f0ecc055ea0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f0ee000b000 tx=0x7f0ee0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.817+0000 7f0edeffd700 1 -- 172.21.15.3:0/2592090496 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0ee8043f60 con 0x7f0eec061fe0 2022-07-21T01:40:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.941+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0eec0574e0 con 0x7f0eec061fe0 2022-07-21T01:40:34.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.949+0000 7f0edeffd700 1 -- 172.21.15.3:0/2592090496 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0ee8023020 con 0x7f0eec061fe0 2022-07-21T01:40:34.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ecc0539e0 msgr2=0x7f0ecc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:34.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ecc0539e0 0x7f0ecc055ea0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f0ee000b000 tx=0x7f0ee0006df0).stop 2022-07-21T01:40:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec061fe0 msgr2=0x7f0eec0b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec061fe0 0x7f0eec0b0140 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f0ee8002f70 tx=0x7f0ee8015000).stop 2022-07-21T01:40:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 shutdown_connections 2022-07-21T01:40:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0eec0b6420 0x7f0eec12aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ecc0539e0 0x7f0ecc055ea0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0eec0b41f0 0x7f0eec0b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 --2- 172.21.15.3:0/2592090496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eec061fe0 0x7f0eec0b0140 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 >> 172.21.15.3:0/2592090496 conn(0x7f0eec05c620 msgr2=0x7f0eec0b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 shutdown_connections 2022-07-21T01:40:34.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:34.953+0000 7f0ef288d700 1 -- 172.21.15.3:0/2592090496 wait complete. 2022-07-21T01:40:34.971 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:40:34.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:34.986 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:35.008 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:35.029 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:35.050 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:40:35.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:35.052 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:35.052 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:35.078 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:40:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:40:35.080 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:35.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:35.081 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:35.081 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:40:35.090 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:40:35.090 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:40:35.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:40:35.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d8d75700 1 Processor -- start 2022-07-21T01:40:35.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d8d75700 1 -- start start 2022-07-21T01:40:35.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d8d75700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86d40b3190 0x7f86d40b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d8d75700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86d40a9da0 0x7f86d4147e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d8d75700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d40b3570 con 0x7f86d40b1a80 2022-07-21T01:40:35.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d8d75700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d40b2980 con 0x7f86d40a9da0 2022-07-21T01:40:35.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d8d75700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d40b0e00 con 0x7f86d40b3190 2022-07-21T01:40:35.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86d40a9da0 0x7f86d4147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86d40a9da0 0x7f86d4147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48116/0 (socket says 172.21.15.3:48116) 2022-07-21T01:40:35.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d3fff700 1 -- 172.21.15.3:0/852518549 learned_addr learned my addr 172.21.15.3:0/852518549 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:35.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d37fe700 1 --2- 172.21.15.3:0/852518549 >> v2:172.21.15.143:3300/0 conn(0x7f86d40b3190 0x7f86d40b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d3fff700 1 -- 172.21.15.3:0/852518549 >> v2:172.21.15.143:3300/0 conn(0x7f86d40b3190 msgr2=0x7f86d40b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d3fff700 1 --2- 172.21.15.3:0/852518549 >> v2:172.21.15.143:3300/0 conn(0x7f86d40b3190 0x7f86d40b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d3fff700 1 -- 172.21.15.3:0/852518549 >> v1:172.21.15.143:6789/0 conn(0x7f86d40b1a80 legacy=0x7f86d40656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d3fff700 1 -- 172.21.15.3:0/852518549 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d40b10a0 con 0x7f86d40a9da0 2022-07-21T01:40:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d3fff700 1 --2- 172.21.15.3:0/852518549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40a9da0 0x7f86d4147e30 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7f86cc0032f0 tx=0x7f86cc009290).ready entity=mon.0 client_cookie=b936c79291f87305 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.197+0000 7f86d27fc700 1 -- 172.21.15.3:0/852518549 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86cc00b870 con 0x7f86d40a9da0 2022-07-21T01:40:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d27fc700 1 -- 172.21.15.3:0/852518549 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc00b9d0 con 0x7f86d40a9da0 2022-07-21T01:40:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d27fc700 1 -- 172.21.15.3:0/852518549 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86cc00bb40 con 0x7f86d40a9da0 2022-07-21T01:40:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 -- 172.21.15.3:0/852518549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40a9da0 msgr2=0x7f86d4147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 --2- 172.21.15.3:0/852518549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40a9da0 0x7f86d4147e30 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7f86cc0032f0 tx=0x7f86cc009290).stop 2022-07-21T01:40:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 -- 172.21.15.3:0/852518549 shutdown_connections 2022-07-21T01:40:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 --2- 172.21.15.3:0/852518549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40a9da0 0x7f86d4147e30 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 --2- 172.21.15.3:0/852518549 >> v2:172.21.15.143:3300/0 conn(0x7f86d40b3190 0x7f86d40b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 -- 172.21.15.3:0/852518549 >> 172.21.15.3:0/852518549 conn(0x7f86d4063870 msgr2=0x7f86d4063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 -- 172.21.15.3:0/852518549 shutdown_connections 2022-07-21T01:40:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 -- 172.21.15.3:0/852518549 wait complete. 2022-07-21T01:40:35.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 Processor -- start 2022-07-21T01:40:35.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.201+0000 7f86d8d75700 1 -- start start 2022-07-21T01:40:35.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d8d75700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86d40a9da0 0x7f86d41493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d8d75700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40b1a80 0x7f86d40ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d8d75700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86d40b3190 0x7f86d40abd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d8d75700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d40b5ad0 con 0x7f86d40b1a80 2022-07-21T01:40:35.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d8d75700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d414aad0 con 0x7f86d40a9da0 2022-07-21T01:40:35.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d8d75700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d414a080 con 0x7f86d40b3190 2022-07-21T01:40:35.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40b1a80 0x7f86d40ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40b1a80 0x7f86d40ab870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48120/0 (socket says 172.21.15.3:48120) 2022-07-21T01:40:35.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d2ffd700 1 -- 172.21.15.3:0/574209066 learned_addr learned my addr 172.21.15.3:0/574209066 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:35.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d37fe700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86d40a9da0 0x7f86d41493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d3fff700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86d40b3190 0x7f86d40abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d2ffd700 1 -- 172.21.15.3:0/574209066 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86d40b3190 msgr2=0x7f86d40abd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d2ffd700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86d40b3190 0x7f86d40abd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d2ffd700 1 -- 172.21.15.3:0/574209066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86d40a9da0 msgr2=0x7f86d41493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d2ffd700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86d40a9da0 0x7f86d41493a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d2ffd700 1 -- 172.21.15.3:0/574209066 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc009530 con 0x7f86d40b1a80 2022-07-21T01:40:35.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d2ffd700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40b1a80 0x7f86d40ab870 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7f86d40634e0 tx=0x7f86c8007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:35.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d0ff9700 1 -- 172.21.15.3:0/574209066 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86c8007030 con 0x7f86d40b1a80 2022-07-21T01:40:35.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d0ff9700 1 -- 172.21.15.3:0/574209066 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c8007190 con 0x7f86d40b1a80 2022-07-21T01:40:35.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d40ab3d0 con 0x7f86d40b1a80 2022-07-21T01:40:35.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d0ff9700 1 -- 172.21.15.3:0/574209066 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86c8006510 con 0x7f86d40b1a80 2022-07-21T01:40:35.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d4145200 con 0x7f86d40b1a80 2022-07-21T01:40:35.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.205+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d40ab560 con 0x7f86d40b1a80 2022-07-21T01:40:35.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.209+0000 7f86d0ff9700 1 -- 172.21.15.3:0/574209066 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f86c8010030 con 0x7f86d40b1a80 2022-07-21T01:40:35.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.209+0000 7f86d0ff9700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86b4053aa0 0x7f86b4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.209+0000 7f86d0ff9700 1 -- 172.21.15.3:0/574209066 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f86c806b940 con 0x7f86d40b1a80 2022-07-21T01:40:35.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.209+0000 7f86d37fe700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86b4053aa0 0x7f86b4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.209+0000 7f86d37fe700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86b4053aa0 0x7f86b4055f60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f86c400b000 tx=0x7f86c4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:35.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.217+0000 7f86d0ff9700 1 -- 172.21.15.3:0/574209066 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f86c80429e0 con 0x7f86d40b1a80 2022-07-21T01:40:35.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.345+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f86d40574e0 con 0x7f86d40b1a80 2022-07-21T01:40:35.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d0ff9700 1 -- 172.21.15.3:0/574209066 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v93) v1 ==== 81+0+29216 (secure 0 0 0) 0x7f86c8046000 con 0x7f86d40b1a80 2022-07-21T01:40:35.353 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:35.354 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:40:35.354 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 93, 2022-07-21T01:40:35.355 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:40:35.355 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:40:35.356 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:40:32.641885+0000", 2022-07-21T01:40:35.356 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:35.356 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:35.356 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:40:35.357 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:40:35.357 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:40:35.357 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:40:35.357 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:40:35.358 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:40:35.358 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:40:35.358 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:35.359 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:40:35.359 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:40:35.359 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:40:35.360 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:40:35.360 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:40:35.360 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 16, 2022-07-21T01:40:35.360 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:40:35.361 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:40:35.361 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:40:35.361 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:40:35.361 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:40:35.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.362 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:40:35.363 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:40:35.363 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:40:35.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:35.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:35.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:35.364 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:35.364 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:35.365 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:35.365 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:35.365 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:35.365 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:40:35.366 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:40:35.366 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:40:35.366 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:40:35.366 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:40:35.367 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:35.367 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:35.367 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:35.367 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:35.367 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:35.368 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:35.368 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:35.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.369 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:40:35.369 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:35.370 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:35.370 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:35.370 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:35.371 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:35.371 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:35.371 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:35.371 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:35.371 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:35.372 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:35.372 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:35.372 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:35.372 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:35.373 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:35.373 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:35.373 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:35.373 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:35.374 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:35.374 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:35.375 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:35.375 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:35.375 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:35.375 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:35.376 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:35.376 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:35.376 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:35.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.377 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:35.377 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:35.377 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:35.377 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:35.378 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:35.378 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:35.379 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:35.379 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:35.379 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:35.380 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:35.380 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:35.380 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:40:35.380 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:40:35.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.381 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:35.381 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:40:35.381 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.382 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:40:35.382 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:40:35.383 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:40:35.383 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:35.383 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:35.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:35.384 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:35.384 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:35.385 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:35.385 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:35.385 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:35.385 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:35.386 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:35.386 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:35.386 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:35.387 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:35.387 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:35.387 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:35.388 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:35.388 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:35.388 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:35.388 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:35.389 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:35.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.389 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:40:35.389 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:35.390 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:35.390 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:35.391 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:35.391 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:35.391 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:35.392 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:35.392 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:35.392 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:35.392 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:35.393 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:35.393 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:35.393 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:35.394 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:35.394 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:35.394 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:35.394 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:35.395 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:35.395 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:35.395 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:35.395 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:35.396 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:35.396 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:35.396 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:35.397 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:35.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:35.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.398 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:35.398 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:35.398 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:35.399 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:35.399 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:35.399 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:35.399 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:35.400 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:35.400 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:35.400 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:35.401 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:35.401 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:40:35.401 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:35.401 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:40:35.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.402 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:35.403 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:40:35.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.403 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:35.404 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:40:35.404 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:35.404 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:35.404 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:35.405 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:35.405 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:35.405 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:35.405 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:35.406 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:35.406 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:35.406 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:35.406 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:35.407 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.407 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.408 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:40:35.408 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:35.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.409 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:40:35.410 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:35.410 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.411 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.411 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:35.411 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.412 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.412 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:40:35.413 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:35.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.413 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.414 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:40:35.414 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:35.414 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.414 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.415 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:35.415 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.416 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:40:35.416 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:35.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.417 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.417 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:40:35.417 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:35.418 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.418 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.418 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:35.419 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.420 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.420 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:40:35.420 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:35.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.422 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:40:35.423 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:35.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.424 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.424 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:40:35.425 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:40:35.425 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:40:35.426 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:40:35.426 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:35.426 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:35.427 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:35.427 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.428 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:35.429 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:40:35.429 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:35.430 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:35.430 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:35.430 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:35.431 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:35.431 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:35.431 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:35.431 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 87, 2022-07-21T01:40:35.432 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:35.432 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:35.432 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:35.433 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.434 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:40:35.434 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:35.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.435 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:40:35.436 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:35.436 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.436 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.437 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:35.437 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.438 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:40:35.438 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:35.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.439 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.439 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:40:35.440 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:35.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.441 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.441 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:35.442 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.442 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:40:35.443 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:35.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.444 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:40:35.444 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:35.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.445 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.445 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:35.445 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.446 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.446 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:40:35.446 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:35.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.447 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.447 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:40:35.447 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:35.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.448 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.448 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:40:35.449 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:40:35.449 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:40:35.450 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:40:35.450 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:35.451 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:35.451 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:35.452 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.453 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:35.454 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:40:35.454 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:35.455 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:35.455 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:35.456 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:35.456 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:35.456 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:35.457 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:35.457 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:35.458 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:35.458 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:35.459 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:35.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:40:35.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:35.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.462 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:40:35.462 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:35.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.463 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:35.463 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.464 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:40:35.464 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:35.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.465 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.465 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:40:35.466 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:35.466 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.466 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.467 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:35.468 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.469 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.469 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:40:35.470 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:35.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.471 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.471 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:40:35.471 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:35.472 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.472 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.473 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:35.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:40:35.475 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:35.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:40:35.476 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:35.476 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.477 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.477 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:40:35.477 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:40:35.477 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:40:35.478 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:40:35.478 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:35.478 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:35.478 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:35.479 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:35.479 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:40:35.480 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:35.480 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:35.480 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:35.480 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:35.480 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:35.481 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:35.481 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:35.481 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:35.481 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:35.482 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:35.482 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:35.482 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.482 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.483 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:40:35.483 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:35.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.483 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.484 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:40:35.484 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:35.484 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.484 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.485 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:35.485 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.486 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:40:35.486 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:35.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.487 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:40:35.487 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:35.487 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.487 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.488 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:35.488 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.488 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.489 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:40:35.489 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:35.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.489 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.490 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:40:35.490 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:35.490 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.490 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.491 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:35.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.491 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.492 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:40:35.492 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:35.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.493 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:40:35.493 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:35.493 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.493 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.494 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:40:35.494 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:40:35.494 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:40:35.495 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:40:35.495 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:35.495 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:35.495 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:35.495 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.496 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:35.496 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:40:35.496 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:35.497 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:35.497 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:35.497 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:35.497 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:35.498 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:35.498 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:35.498 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:35.498 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:35.498 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:35.499 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:35.499 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.499 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.500 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:40:35.500 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:35.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.500 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.501 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:40:35.501 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:35.501 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.501 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.502 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:35.502 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.502 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:40:35.503 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:35.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:40:35.504 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:35.504 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.504 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.505 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:35.505 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.505 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:40:35.506 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:35.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.506 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:40:35.507 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:35.507 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.507 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.508 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:35.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:40:35.509 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:35.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.509 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.510 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:40:35.510 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:35.510 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.510 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.511 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:40:35.511 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:40:35.511 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:40:35.511 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:40:35.512 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:35.512 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:35.512 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:35.512 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.513 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:35.513 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:40:35.513 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:35.513 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:35.514 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:35.514 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:35.514 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:35.514 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:35.515 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:35.515 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:35.515 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:35.515 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:35.515 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:35.516 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.516 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.517 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:40:35.517 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:35.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.518 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.518 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:40:35.518 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:35.518 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.519 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.519 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:35.519 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.520 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.520 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:40:35.520 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:35.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.521 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.521 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:40:35.521 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:35.521 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.522 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.522 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:35.522 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.523 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.523 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:40:35.523 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:35.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.524 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.524 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:40:35.524 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:35.524 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.525 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.525 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:35.525 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:40:35.526 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:35.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.527 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:40:35.527 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:35.528 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.528 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.528 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:40:35.528 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:40:35.529 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:40:35.529 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:40:35.529 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:35.529 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:35.530 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:35.530 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.530 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:35.531 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:40:35.531 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:35.531 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:35.531 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:35.531 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:35.532 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:35.532 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:35.532 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:35.532 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:35.532 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:35.533 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:35.533 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:35.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:40:35.534 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:35.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.535 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.535 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:40:35.535 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:35.535 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.536 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.536 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:35.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.537 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.537 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:40:35.537 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:35.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.538 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:40:35.538 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:35.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.539 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.539 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:35.539 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.540 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.540 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:40:35.540 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:35.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.541 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.541 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:40:35.541 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:35.541 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.542 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.542 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:35.542 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.543 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:40:35.543 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:35.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.544 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.544 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:40:35.544 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:35.544 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.545 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.545 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:40:35.545 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:40:35.545 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:40:35.546 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:40:35.546 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:35.546 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:35.546 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:35.546 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.547 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:35.547 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:40:35.548 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:35.548 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:35.548 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:35.548 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:35.548 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:35.549 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:35.549 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:35.549 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 76, 2022-07-21T01:40:35.549 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:35.549 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:35.550 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:35.550 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.551 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:40:35.551 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:35.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.551 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.552 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:40:35.552 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:35.552 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.552 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.553 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:35.553 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.553 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:40:35.554 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:35.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.554 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.555 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:40:35.555 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:35.555 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.555 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.556 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:35.556 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.556 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.556 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:40:35.557 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:35.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.558 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:40:35.558 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:35.558 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.558 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.559 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:35.559 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:35.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.559 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:35.560 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:40:35.560 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:35.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.560 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:35.561 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:40:35.561 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:35.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.561 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.562 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:40:35.562 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:40:35.562 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:40:35.562 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:40:35.563 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:35.563 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:35.563 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:35.563 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:35.563 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.564 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:35.564 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:40:35.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.564 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:35.564 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:35.565 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:35.565 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:35.565 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:35.565 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:35.566 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:40:35.566 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:35.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.566 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:35.567 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:35.567 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:35.567 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:35.567 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:35.567 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:35.568 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:40:35.568 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:35.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.569 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:35.569 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:35.569 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:35.569 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:35.569 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:35.570 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:35.570 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:40:35.570 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:35.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.571 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:35.571 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:35.571 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:35.571 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:35.572 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:35.572 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:35.572 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:40:35.572 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:35.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.573 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:35.573 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:35.573 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:35.573 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:35.574 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:35.574 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:35.574 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:40:35.574 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:35.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.575 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:35.575 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:35.575 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:35.576 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:35.576 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:35.576 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:35.576 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:40:35.576 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:35.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.577 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:35.577 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:35.578 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:35.578 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:35.578 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:35.578 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:35.578 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:40:35.579 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:35.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:35.579 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:35.580 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:35.580 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:35.580 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:35.580 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:35.580 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:35.581 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:40:35.581 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:35.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.581 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:35.581 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:40:35.582 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:40:35.582 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:40:35.582 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:40:35.582 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:40:35.583 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6837/539587272": "2022-07-22T01:40:29.227554+0000", 2022-07-21T01:40:35.583 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6836/539587272": "2022-07-22T01:40:29.227554+0000" 2022-07-21T01:40:35.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.583 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:40:35.583 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:40:35.584 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:40:35.584 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:35.584 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:40:35.584 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:40:35.584 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:35.585 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:40:35.585 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:35.585 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:35.585 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:40:35.585 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:40:35.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.586 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:40:35.586 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:35.586 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:35.587 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:40:35.587 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:35.587 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:35.587 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:40:35.587 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:35.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:35.588 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:40:35.588 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:40:35.588 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:40:35.588 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:40:35.589 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:40:35.589 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:40:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86b4053aa0 msgr2=0x7f86b4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86b4053aa0 0x7f86b4055f60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f86c400b000 tx=0x7f86c4006df0).stop 2022-07-21T01:40:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40b1a80 msgr2=0x7f86d40ab870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40b1a80 0x7f86d40ab870 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7f86d40634e0 tx=0x7f86c8007c10).stop 2022-07-21T01:40:35.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 shutdown_connections 2022-07-21T01:40:35.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86d40b3190 0x7f86d40abd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86d40b1a80 0x7f86d40ab870 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f86b4053aa0 0x7f86b4055f60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 --2- 172.21.15.3:0/574209066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86d40a9da0 0x7f86d41493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 >> 172.21.15.3:0/574209066 conn(0x7f86d4063870 msgr2=0x7f86d4060010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 shutdown_connections 2022-07-21T01:40:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.349+0000 7f86d8d75700 1 -- 172.21.15.3:0/574209066 wait complete. 2022-07-21T01:40:35.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:35.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 Processor -- start 2022-07-21T01:40:35.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 -- start start 2022-07-21T01:40:35.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f80c80b5d80 0x7f80c80b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c8144240 con 0x7f80c80b6710 2022-07-21T01:40:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c80b6fe0 con 0x7f80c80b4550 2022-07-21T01:40:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c8143d00 con 0x7f80c80b5d80 2022-07-21T01:40:35.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c6ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f80c80b6710 0x7f80c8061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56406/0 (socket says 172.21.15.3:56406) 2022-07-21T01:40:35.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f80c80b5d80 0x7f80c80b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c6ffd700 1 -- 172.21.15.3:0/3421881681 learned_addr learned my addr 172.21.15.3:0/3421881681 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:35.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c77fe700 1 -- 172.21.15.3:0/3421881681 >> v1:172.21.15.143:6789/0 conn(0x7f80c80b4550 legacy=0x7f80c812bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c77fe700 1 -- 172.21.15.3:0/3421881681 >> v1:172.21.15.3:6789/0 conn(0x7f80c80b6710 legacy=0x7f80c8061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c77fe700 1 -- 172.21.15.3:0/3421881681 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c8143fa0 con 0x7f80c80b5d80 2022-07-21T01:40:35.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c77fe700 1 --2- 172.21.15.3:0/3421881681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b5d80 0x7f80c80b6af0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f80b8002f70 tx=0x7f80b8006df0).ready entity=mon.0 client_cookie=76310f4ac5038e29 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:35.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c5ffb700 1 -- 172.21.15.3:0/3421881681 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80b800d610 con 0x7f80c80b5d80 2022-07-21T01:40:35.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c5ffb700 1 -- 172.21.15.3:0/3421881681 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b8000e90 con 0x7f80c80b5d80 2022-07-21T01:40:35.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80c5ffb700 1 -- 172.21.15.3:0/3421881681 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80b800d960 con 0x7f80c80b5d80 2022-07-21T01:40:35.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 -- 172.21.15.3:0/3421881681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b5d80 msgr2=0x7f80c80b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 --2- 172.21.15.3:0/3421881681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b5d80 0x7f80c80b6af0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f80b8002f70 tx=0x7f80b8006df0).stop 2022-07-21T01:40:35.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 -- 172.21.15.3:0/3421881681 shutdown_connections 2022-07-21T01:40:35.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 --2- 172.21.15.3:0/3421881681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b5d80 0x7f80c80b6af0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 -- 172.21.15.3:0/3421881681 >> 172.21.15.3:0/3421881681 conn(0x7f80c805c620 msgr2=0x7f80c805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:35.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.701+0000 7f80cd7e4700 1 -- 172.21.15.3:0/3421881681 shutdown_connections 2022-07-21T01:40:35.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.705+0000 7f80cd7e4700 1 -- 172.21.15.3:0/3421881681 wait complete. 2022-07-21T01:40:35.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.705+0000 7f80cd7e4700 1 Processor -- start 2022-07-21T01:40:35.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.705+0000 7f80cd7e4700 1 -- start start 2022-07-21T01:40:35.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80cd7e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80c80b4550 0x7f80c8156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80cd7e4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80c80b5d80 0x7f80c81256d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80cd7e4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b6710 0x7f80c80b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80cd7e4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c80b3eb0 con 0x7f80c80b6710 2022-07-21T01:40:35.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80cd7e4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c80b3c10 con 0x7f80c80b4550 2022-07-21T01:40:35.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80cd7e4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c8155cd0 con 0x7f80c80b5d80 2022-07-21T01:40:35.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b6710 0x7f80c80b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b6710 0x7f80c80b1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48136/0 (socket says 172.21.15.3:48136) 2022-07-21T01:40:35.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c77fe700 1 -- 172.21.15.3:0/2282486541 learned_addr learned my addr 172.21.15.3:0/2282486541 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:35.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c6ffd700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80c80b4550 0x7f80c8156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c67fc700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80c80b5d80 0x7f80c81256d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c77fe700 1 -- 172.21.15.3:0/2282486541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80c80b5d80 msgr2=0x7f80c81256d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c77fe700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80c80b5d80 0x7f80c81256d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c77fe700 1 -- 172.21.15.3:0/2282486541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80c80b4550 msgr2=0x7f80c8156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c77fe700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80c80b4550 0x7f80c8156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c77fe700 1 -- 172.21.15.3:0/2282486541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b800b180 con 0x7f80c80b6710 2022-07-21T01:40:35.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c6ffd700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80c80b4550 0x7f80c8156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:35.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80c77fe700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b6710 0x7f80c80b1e50 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f80b8002f70 tx=0x7f80b8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:35.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80bffff700 1 -- 172.21.15.3:0/2282486541 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80b800d990 con 0x7f80c80b6710 2022-07-21T01:40:35.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80bffff700 1 -- 172.21.15.3:0/2282486541 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b8011ce0 con 0x7f80c80b6710 2022-07-21T01:40:35.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c8155f70 con 0x7f80c80b6710 2022-07-21T01:40:35.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80bffff700 1 -- 172.21.15.3:0/2282486541 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80b801e650 con 0x7f80c80b6710 2022-07-21T01:40:35.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c80ae320 con 0x7f80c80b6710 2022-07-21T01:40:35.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c8156100 con 0x7f80c80b6710 2022-07-21T01:40:35.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.709+0000 7f80bffff700 1 -- 172.21.15.3:0/2282486541 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f80b8025030 con 0x7f80c80b6710 2022-07-21T01:40:35.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.713+0000 7f80bffff700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f80a0053b30 0x7f80a0055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:35.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.713+0000 7f80bffff700 1 -- 172.21.15.3:0/2282486541 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f80b806d140 con 0x7f80c80b6710 2022-07-21T01:40:35.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.713+0000 7f80c6ffd700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f80a0053b30 0x7f80a0055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:35.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.713+0000 7f80c6ffd700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f80a0053b30 0x7f80a0055ff0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f80b000b000 tx=0x7f80b0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:35.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.717+0000 7f80bffff700 1 -- 172.21.15.3:0/2282486541 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f80b80441e0 con 0x7f80c80b6710 2022-07-21T01:40:35.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80c8061780 con 0x7f80c80b6710 2022-07-21T01:40:35.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80bffff700 1 -- 172.21.15.3:0/2282486541 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+1559 (secure 0 0 0) 0x7f80b8023090 con 0x7f80c80b6710 2022-07-21T01:40:35.886 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:35.886 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/2659690520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2659690520},{"type":"v1","addr":"172.21.15.143:6833","nonce":2659690520}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":6600,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2193393317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2193393317},{"type":"v1","addr":"172.21.15.3:6835","nonce":2193393317}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38},{"gid":6855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/2481359857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2481359857},{"type":"v1","addr":"172.21.15.3:6837","nonce":2481359857}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[]} 2022-07-21T01:40:35.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f80a0053b30 msgr2=0x7f80a0055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f80a0053b30 0x7f80a0055ff0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f80b000b000 tx=0x7f80b0009040).stop 2022-07-21T01:40:35.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b6710 msgr2=0x7f80c80b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:35.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b6710 0x7f80c80b1e50 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f80b8002f70 tx=0x7f80b8015000).stop 2022-07-21T01:40:35.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 shutdown_connections 2022-07-21T01:40:35.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c80b6710 0x7f80c80b1e50 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80c80b5d80 0x7f80c81256d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f80a0053b30 0x7f80a0055ff0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 --2- 172.21.15.3:0/2282486541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80c80b4550 0x7f80c8156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:35.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 >> 172.21.15.3:0/2282486541 conn(0x7f80c805c620 msgr2=0x7f80c812bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:35.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 shutdown_connections 2022-07-21T01:40:35.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.881+0000 7f80cd7e4700 1 -- 172.21.15.3:0/2282486541 wait complete. 2022-07-21T01:40:35.892 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 44 2022-07-21T01:40:35.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:35.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.993+0000 7f1cd4ab8700 1 Processor -- start 2022-07-21T01:40:36.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.993+0000 7f1cd4ab8700 1 -- start start 2022-07-21T01:40:36.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b6710 0x7f1cd00b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1cd00b6af0 0x7f1cd012bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd0144240 con 0x7f1cd00b4550 2022-07-21T01:40:36.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd0143d00 con 0x7f1cd00b6af0 2022-07-21T01:40:36.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd0143fa0 con 0x7f1cd00b6710 2022-07-21T01:40:36.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cced9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1cd00b6af0 0x7f1cd012bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cced9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1cd00b6af0 0x7f1cd012bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48142/0 (socket says 172.21.15.3:48142) 2022-07-21T01:40:36.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cced9d700 1 -- 172.21.15.3:0/1336944566 learned_addr learned my addr 172.21.15.3:0/1336944566 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:36.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cce59c700 1 --2- 172.21.15.3:0/1336944566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b6710 0x7f1cd00b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cced9d700 1 -- 172.21.15.3:0/1336944566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b6710 msgr2=0x7f1cd00b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:36.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cced9d700 1 --2- 172.21.15.3:0/1336944566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b6710 0x7f1cd00b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1ccd59a700 1 -- 172.21.15.3:0/1336944566 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716621695 0 0) 0x7f1cd0144240 con 0x7f1cd00b4550 2022-07-21T01:40:36.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cced9d700 1 -- 172.21.15.3:0/1336944566 >> v1:172.21.15.143:6789/0 conn(0x7f1cd00b4550 legacy=0x7f1cd0061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cced9d700 1 -- 172.21.15.3:0/1336944566 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd0061780 con 0x7f1cd00b6af0 2022-07-21T01:40:36.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cce59c700 1 --2- 172.21.15.3:0/1336944566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b6710 0x7f1cd00b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:36.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cced9d700 1 --2- 172.21.15.3:0/1336944566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 0x7f1cd012bb90 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f1cc8002f70 tx=0x7f1cc8006eb0).ready entity=mon.0 client_cookie=99bcee741ea4f46b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:36.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1ccd59a700 1 -- 172.21.15.3:0/1336944566 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cc800d5c0 con 0x7f1cd00b6af0 2022-07-21T01:40:36.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1ccd59a700 1 -- 172.21.15.3:0/1336944566 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc8000e90 con 0x7f1cd00b6af0 2022-07-21T01:40:36.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1ccd59a700 1 -- 172.21.15.3:0/1336944566 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cc800d910 con 0x7f1cd00b6af0 2022-07-21T01:40:36.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1336944566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 msgr2=0x7f1cd012bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 --2- 172.21.15.3:0/1336944566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 0x7f1cd012bb90 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f1cc8002f70 tx=0x7f1cc8006eb0).stop 2022-07-21T01:40:36.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1336944566 shutdown_connections 2022-07-21T01:40:36.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 --2- 172.21.15.3:0/1336944566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 0x7f1cd012bb90 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 --2- 172.21.15.3:0/1336944566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b6710 0x7f1cd00b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1336944566 >> 172.21.15.3:0/1336944566 conn(0x7f1cd005c620 msgr2=0x7f1cd005ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:36.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1336944566 shutdown_connections 2022-07-21T01:40:36.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:35.997+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1336944566 wait complete. 2022-07-21T01:40:36.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 Processor -- start 2022-07-21T01:40:36.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 -- start start 2022-07-21T01:40:36.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b4550 0x7f1cd0141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cd00b6710 0x7f1cd00b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 0x7f1cd0155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd0152660 con 0x7f1cd00b6af0 2022-07-21T01:40:36.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd0152900 con 0x7f1cd00b6710 2022-07-21T01:40:36.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd0152ba0 con 0x7f1cd00b4550 2022-07-21T01:40:36.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cced9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 0x7f1cd0155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cced9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 0x7f1cd0155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48150/0 (socket says 172.21.15.3:48150) 2022-07-21T01:40:36.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cced9d700 1 -- 172.21.15.3:0/1656358981 learned_addr learned my addr 172.21.15.3:0/1656358981 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:36.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1ccdd9b700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cd00b6710 0x7f1cd00b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cce59c700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b4550 0x7f1cd0141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cced9d700 1 -- 172.21.15.3:0/1656358981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b4550 msgr2=0x7f1cd0141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cced9d700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b4550 0x7f1cd0141950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cced9d700 1 -- 172.21.15.3:0/1656358981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cd00b6710 msgr2=0x7f1cd00b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cced9d700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cd00b6710 0x7f1cd00b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cced9d700 1 -- 172.21.15.3:0/1656358981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc800b1e0 con 0x7f1cd00b6af0 2022-07-21T01:40:36.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1ccdd9b700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cd00b6710 0x7f1cd00b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:36.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cced9d700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 0x7f1cd0155a90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f1cc8002f70 tx=0x7f1cc8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:36.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cbf7fe700 1 -- 172.21.15.3:0/1656358981 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cc8006be0 con 0x7f1cd00b6af0 2022-07-21T01:40:36.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cbf7fe700 1 -- 172.21.15.3:0/1656358981 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc800d940 con 0x7f1cd00b6af0 2022-07-21T01:40:36.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd0152e40 con 0x7f1cd00b6af0 2022-07-21T01:40:36.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cbf7fe700 1 -- 172.21.15.3:0/1656358981 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cc801e460 con 0x7f1cd00b6af0 2022-07-21T01:40:36.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd0127280 con 0x7f1cd00b6af0 2022-07-21T01:40:36.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.001+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd0152fd0 con 0x7f1cd00b6af0 2022-07-21T01:40:36.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.005+0000 7f1cbf7fe700 1 -- 172.21.15.3:0/1656358981 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1cc8025030 con 0x7f1cd00b6af0 2022-07-21T01:40:36.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.005+0000 7f1cbf7fe700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1cac053b00 0x7f1cac055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.005+0000 7f1cbf7fe700 1 -- 172.21.15.3:0/1656358981 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f1cc806d050 con 0x7f1cd00b6af0 2022-07-21T01:40:36.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.005+0000 7f1cce59c700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1cac053b00 0x7f1cac055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.005+0000 7f1cce59c700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1cac053b00 0x7f1cac055fc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f1cc000c000 tx=0x7f1cc0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:36.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.013+0000 7f1cbf7fe700 1 -- 172.21.15.3:0/1656358981 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1cc80440f0 con 0x7f1cd00b6af0 2022-07-21T01:40:36.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.181+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cd0061a20 con 0x7f1cd00b6af0 2022-07-21T01:40:36.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.181+0000 7f1cbf7fe700 1 -- 172.21.15.3:0/1656358981 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+1559 (secure 0 0 0) 0x7f1cc8023070 con 0x7f1cd00b6af0 2022-07-21T01:40:36.188 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:36.189 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5921,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/2659690520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2659690520},{"type":"v1","addr":"172.21.15.143:6833","nonce":2659690520}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":6600,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2193393317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2193393317},{"type":"v1","addr":"172.21.15.3:6835","nonce":2193393317}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38},{"gid":6855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/2481359857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2481359857},{"type":"v1","addr":"172.21.15.3:6837","nonce":2481359857}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[]} 2022-07-21T01:40:36.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1cac053b00 msgr2=0x7f1cac055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1cac053b00 0x7f1cac055fc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f1cc000c000 tx=0x7f1cc0008040).stop 2022-07-21T01:40:36.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 msgr2=0x7f1cd0155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 0x7f1cd0155a90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f1cc8002f70 tx=0x7f1cc8015000).stop 2022-07-21T01:40:36.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 shutdown_connections 2022-07-21T01:40:36.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cd00b6af0 0x7f1cd0155a90 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1cac053b00 0x7f1cac055fc0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cd00b6710 0x7f1cd00b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 --2- 172.21.15.3:0/1656358981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1cd00b4550 0x7f1cd0141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 >> 172.21.15.3:0/1656358981 conn(0x7f1cd005c620 msgr2=0x7f1cd0128510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:36.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 shutdown_connections 2022-07-21T01:40:36.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.185+0000 7f1cd4ab8700 1 -- 172.21.15.3:0/1656358981 wait complete. 2022-07-21T01:40:36.194 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 44 2022-07-21T01:40:36.200 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:40:36.201 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:40:36.201 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:40:36.202 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:40:36.202 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:40:36.202 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:40:36.203 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:40:36.203 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:40:36.204 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:40:36.204 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:40:36.205 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:40:36.205 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:40:36.216 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:40:36.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:40:36.218 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:40:36.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:40:36.220 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:40:36.220 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:40:36.222 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:40:36.223 DEBUG:teuthology.parallel:result is None 2022-07-21T01:40:36.261 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:40:36.262 DEBUG:teuthology.parallel:result is None 2022-07-21T01:40:36.263 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:40:36.264 DEBUG:teuthology.parallel:result is None 2022-07-21T01:40:36.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:40:36.293 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:40:36.329 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:40:36.349 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:40:36.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.369+0000 7f84df59e700 1 Processor -- start 2022-07-21T01:40:36.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.369+0000 7f84df59e700 1 -- start start 2022-07-21T01:40:36.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84e00b5490 0x7f84e013e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00af0b0 con 0x7f84e00b15f0 2022-07-21T01:40:36.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00ac490 con 0x7f84e013e730 2022-07-21T01:40:36.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e00af2e0 con 0x7f84e00b5490 2022-07-21T01:40:36.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84de59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84e00b5490 0x7f84e013e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84ddd9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f84e00b15f0 0x7f84e01411d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56460/0 (socket says 172.21.15.3:56460) 2022-07-21T01:40:36.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84ddd9b700 1 -- 172.21.15.3:0/3667540665 learned_addr learned my addr 172.21.15.3:0/3667540665 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:36.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84de59c700 1 -- 172.21.15.3:0/3667540665 >> v1:172.21.15.143:6789/0 conn(0x7f84e013e730 legacy=0x7f84e01418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84de59c700 1 -- 172.21.15.3:0/3667540665 >> v1:172.21.15.3:6789/0 conn(0x7f84e00b15f0 legacy=0x7f84e01411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84de59c700 1 -- 172.21.15.3:0/3667540665 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e00af580 con 0x7f84e00b5490 2022-07-21T01:40:36.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84de59c700 1 --2- 172.21.15.3:0/3667540665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b5490 0x7f84e013e240 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f84d0002f70 tx=0x7f84d0006df0).ready entity=mon.0 client_cookie=7c9e68b91565077 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84dd59a700 1 -- 172.21.15.3:0/3667540665 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84d000d610 con 0x7f84e00b5490 2022-07-21T01:40:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84dd59a700 1 -- 172.21.15.3:0/3667540665 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d0000e90 con 0x7f84e00b5490 2022-07-21T01:40:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84dd59a700 1 -- 172.21.15.3:0/3667540665 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84d000d960 con 0x7f84e00b5490 2022-07-21T01:40:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 -- 172.21.15.3:0/3667540665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b5490 msgr2=0x7f84e013e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 --2- 172.21.15.3:0/3667540665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b5490 0x7f84e013e240 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f84d0002f70 tx=0x7f84d0006df0).stop 2022-07-21T01:40:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 -- 172.21.15.3:0/3667540665 shutdown_connections 2022-07-21T01:40:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 --2- 172.21.15.3:0/3667540665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b5490 0x7f84e013e240 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 -- 172.21.15.3:0/3667540665 >> 172.21.15.3:0/3667540665 conn(0x7f84e00626f0 msgr2=0x7f84e0062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 -- 172.21.15.3:0/3667540665 shutdown_connections 2022-07-21T01:40:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 -- 172.21.15.3:0/3667540665 wait complete. 2022-07-21T01:40:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.373+0000 7f84df59e700 1 Processor -- start 2022-07-21T01:40:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84df59e700 1 -- start start 2022-07-21T01:40:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84df59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b15f0 0x7f84e013c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84df59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84e00b5490 0x7f84e0156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84df59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e013e730 0x7f84e0156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84df59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00ae260 con 0x7f84e00b15f0 2022-07-21T01:40:36.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84df59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00ad120 con 0x7f84e00b5490 2022-07-21T01:40:36.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84df59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00ad380 con 0x7f84e013e730 2022-07-21T01:40:36.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84de59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b15f0 0x7f84e013c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84de59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b15f0 0x7f84e013c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48184/0 (socket says 172.21.15.3:48184) 2022-07-21T01:40:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84de59c700 1 -- 172.21.15.3:0/707597541 learned_addr learned my addr 172.21.15.3:0/707597541 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84ded9d700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e013e730 0x7f84e0156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84ddd9b700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84e00b5490 0x7f84e0156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84de59c700 1 -- 172.21.15.3:0/707597541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e013e730 msgr2=0x7f84e0156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84de59c700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e013e730 0x7f84e0156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84de59c700 1 -- 172.21.15.3:0/707597541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84e00b5490 msgr2=0x7f84e0156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:36.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84de59c700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84e00b5490 0x7f84e0156360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:36.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84de59c700 1 -- 172.21.15.3:0/707597541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d000b180 con 0x7f84e00b15f0 2022-07-21T01:40:36.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84de59c700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b15f0 0x7f84e013c570 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7f84d0002f70 tx=0x7f84d0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:36.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84cf7fe700 1 -- 172.21.15.3:0/707597541 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84d000d990 con 0x7f84e00b15f0 2022-07-21T01:40:36.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84cf7fe700 1 -- 172.21.15.3:0/707597541 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d0011b40 con 0x7f84e00b15f0 2022-07-21T01:40:36.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e013ca60 con 0x7f84e00b15f0 2022-07-21T01:40:36.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84cf7fe700 1 -- 172.21.15.3:0/707597541 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84d0011ea0 con 0x7f84e00b15f0 2022-07-21T01:40:36.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.377+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e013cff0 con 0x7f84e00b15f0 2022-07-21T01:40:36.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.381+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e00afd80 con 0x7f84e00b15f0 2022-07-21T01:40:36.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.381+0000 7f84cf7fe700 1 -- 172.21.15.3:0/707597541 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f84d0025030 con 0x7f84e00b15f0 2022-07-21T01:40:36.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.381+0000 7f84cf7fe700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84bc053b00 0x7f84bc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:36.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.381+0000 7f84cf7fe700 1 -- 172.21.15.3:0/707597541 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f84d006d190 con 0x7f84e00b15f0 2022-07-21T01:40:36.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.381+0000 7f84ddd9b700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84bc053b00 0x7f84bc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:36.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.381+0000 7f84ddd9b700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84bc053b00 0x7f84bc055fc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f84d400d000 tx=0x7f84d4007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:36.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.385+0000 7f84cf7fe700 1 -- 172.21.15.3:0/707597541 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f84d0044120 con 0x7f84e00b15f0 2022-07-21T01:40:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:36.521+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f84e0061020 con 0x7f84e00b15f0 2022-07-21T01:40:37.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.377+0000 7f84cf7fe700 1 -- 172.21.15.3:0/707597541 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v97) v1 ==== 97+0+0 (secure 0 0 0) 0x7f84d0023080 con 0x7f84e00b15f0 2022-07-21T01:40:37.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84bc053b00 msgr2=0x7f84bc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84bc053b00 0x7f84bc055fc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f84d400d000 tx=0x7f84d4007be0).stop 2022-07-21T01:40:37.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b15f0 msgr2=0x7f84e013c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b15f0 0x7f84e013c570 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7f84d0002f70 tx=0x7f84d0015000).stop 2022-07-21T01:40:37.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 shutdown_connections 2022-07-21T01:40:37.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e013e730 0x7f84e0156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84bc053b00 0x7f84bc055fc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84e00b5490 0x7f84e0156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 --2- 172.21.15.3:0/707597541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e00b15f0 0x7f84e013c570 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 >> 172.21.15.3:0/707597541 conn(0x7f84e00626f0 msgr2=0x7f84e0140170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:37.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 shutdown_connections 2022-07-21T01:40:37.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.381+0000 7f84df59e700 1 -- 172.21.15.3:0/707597541 wait complete. 2022-07-21T01:40:37.390 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:40:37.403 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:40:37.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:40:37.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 Processor -- start 2022-07-21T01:40:37.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- start start 2022-07-21T01:40:37.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00617e0 0x7f35d0066760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d0060670 con 0x7f35d0060290 2022-07-21T01:40:37.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d0060e00 con 0x7f35d0066c50 2022-07-21T01:40:37.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00610a0 con 0x7f35d00617e0 2022-07-21T01:40:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d4a86700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f35d0060290 0x7f35d0065b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56476/0 (socket says 172.21.15.3:56476) 2022-07-21T01:40:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d4a86700 1 -- 172.21.15.3:0/1259047866 learned_addr learned my addr 172.21.15.3:0/1259047866 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cffff700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00617e0 0x7f35d0066760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501581552 0 0) 0x7f35d0060670 con 0x7f35d0060290 2022-07-21T01:40:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc003260 con 0x7f35d0060290 2022-07-21T01:40:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618637231 0 0) 0x7f35d0060e00 con 0x7f35d0066c50 2022-07-21T01:40:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d0060670 con 0x7f35d0066c50 2022-07-21T01:40:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1865973626 0 0) 0x7f35bc003260 con 0x7f35d0060290 2022-07-21T01:40:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00617e0 msgr2=0x7f35d0066760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00617e0 0x7f35d0066760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 >> v1:172.21.15.143:6789/0 conn(0x7f35d0066c50 legacy=0x7f35d014db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d0061340 con 0x7f35d0060290 2022-07-21T01:40:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f35c8001a00 con 0x7f35d0060290 2022-07-21T01:40:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 >> v1:172.21.15.3:6789/0 conn(0x7f35d0060290 legacy=0x7f35d0065b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc003910 0x7f35bc003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00617e0 0x7f35bc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0049f0 0x7f35bc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00610a0 con 0x7f35bc0049f0 2022-07-21T01:40:37.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc00b7b0 con 0x7f35bc003910 2022-07-21T01:40:37.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc00b970 con 0x7f35d00617e0 2022-07-21T01:40:37.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d5287700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0049f0 0x7f35bc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d4a86700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc003910 0x7f35bc003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cffff700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00617e0 0x7f35bc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d5287700 1 -- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00617e0 msgr2=0x7f35bc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:37.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d5287700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00617e0 0x7f35bc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d5287700 1 -- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc003910 msgr2=0x7f35bc003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:37.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d5287700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35bc003910 0x7f35bc003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d5287700 1 -- 172.21.15.3:0/1259047866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc003260 con 0x7f35bc0049f0 2022-07-21T01:40:37.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d5287700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0049f0 0x7f35bc004e70 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f35c0001110 tx=0x7f35c0007a00).ready entity=mon.0 client_cookie=d008ba7099f2243 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:37.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35c0007d10 con 0x7f35bc0049f0 2022-07-21T01:40:37.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c0007030 con 0x7f35bc0049f0 2022-07-21T01:40:37.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35cf7fe700 1 -- 172.21.15.3:0/1259047866 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35c0007390 con 0x7f35bc0049f0 2022-07-21T01:40:37.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- 172.21.15.3:0/1259047866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0049f0 msgr2=0x7f35bc004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 --2- 172.21.15.3:0/1259047866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35bc0049f0 0x7f35bc004e70 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f35c0001110 tx=0x7f35c0007a00).stop 2022-07-21T01:40:37.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d5287700 1 -- 172.21.15.3:0/1259047866 reap_dead start 2022-07-21T01:40:37.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- 172.21.15.3:0/1259047866 shutdown_connections 2022-07-21T01:40:37.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- 172.21.15.3:0/1259047866 >> 172.21.15.3:0/1259047866 conn(0x7f35d005d0c0 msgr2=0x7f35d005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:37.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- 172.21.15.3:0/1259047866 shutdown_connections 2022-07-21T01:40:37.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- 172.21.15.3:0/1259047866 wait complete. 2022-07-21T01:40:37.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 Processor -- start 2022-07-21T01:40:37.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.513+0000 7f35d6cea700 1 -- start start 2022-07-21T01:40:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d6cea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00b0f50 0x7f35d0153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d6cea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d00a59f0 0x7f35d00a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d6cea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d00a2f30 0x7f35d00a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d6cea700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0150500 con 0x7f35d00a59f0 2022-07-21T01:40:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d6cea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00a6340 con 0x7f35d00a2f30 2022-07-21T01:40:37.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d6cea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00b1330 con 0x7f35d00b0f50 2022-07-21T01:40:37.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d00a59f0 0x7f35d00a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d4a86700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00b0f50 0x7f35d0153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d00a59f0 0x7f35d00a5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48212/0 (socket says 172.21.15.3:48212) 2022-07-21T01:40:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cffff700 1 -- 172.21.15.3:0/388872910 learned_addr learned my addr 172.21.15.3:0/388872910 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d5287700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d00a2f30 0x7f35d00a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cffff700 1 -- 172.21.15.3:0/388872910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00b0f50 msgr2=0x7f35d0153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cffff700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00b0f50 0x7f35d0153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cffff700 1 -- 172.21.15.3:0/388872910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d00a2f30 msgr2=0x7f35d00a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cffff700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d00a2f30 0x7f35d00a33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cffff700 1 -- 172.21.15.3:0/388872910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d00a7780 con 0x7f35d00a59f0 2022-07-21T01:40:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cffff700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d00a59f0 0x7f35d00a5e50 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f35c400b000 tx=0x7f35c4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cdffb700 1 -- 172.21.15.3:0/388872910 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35c400f760 con 0x7f35d00a59f0 2022-07-21T01:40:37.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cdffb700 1 -- 172.21.15.3:0/388872910 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c400f8c0 con 0x7f35d00a59f0 2022-07-21T01:40:37.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d00a7a20 con 0x7f35d00a59f0 2022-07-21T01:40:37.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35cdffb700 1 -- 172.21.15.3:0/388872910 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35c400fc20 con 0x7f35d00a59f0 2022-07-21T01:40:37.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.517+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d00a7f60 con 0x7f35d00a59f0 2022-07-21T01:40:37.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.521+0000 7f35cdffb700 1 -- 172.21.15.3:0/388872910 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f35c4014440 con 0x7f35d00a59f0 2022-07-21T01:40:37.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.521+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d00b6f30 con 0x7f35d00a59f0 2022-07-21T01:40:37.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.521+0000 7f35cdffb700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b30 0x7f35b0055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.521+0000 7f35cdffb700 1 -- 172.21.15.3:0/388872910 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f35c406c630 con 0x7f35d00a59f0 2022-07-21T01:40:37.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.521+0000 7f35d4a86700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b30 0x7f35b0055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.521+0000 7f35d4a86700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b30 0x7f35b0055ff0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f35c8005000 tx=0x7f35d0064d40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.529+0000 7f35cdffb700 1 -- 172.21.15.3:0/388872910 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35c40436d0 con 0x7f35d00a59f0 2022-07-21T01:40:37.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f35d00a80f0 con 0x7f35d00a59f0 2022-07-21T01:40:37.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35cdffb700 1 -- 172.21.15.3:0/388872910 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v26) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f35c4018020 con 0x7f35d00a59f0 2022-07-21T01:40:37.694 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:37.694 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:40:37.694 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:40:37.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.695 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:40:37.695 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:40:37.695 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.695 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:37.696 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.696 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.696 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.697 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:40:37.697 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:40:37.698 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.698 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:37.698 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.698 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.699 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.699 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.699 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:40:37.700 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:40:37.700 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.700 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:37.700 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.701 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.701 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.702 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:40:37.702 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:40:37.702 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.702 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.703 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.703 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.704 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:40:37.704 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:40:37.704 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.704 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.705 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.705 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.706 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:40:37.706 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:40:37.706 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.707 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.707 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.707 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.707 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.708 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:40:37.708 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:40:37.708 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.709 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.709 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.709 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.709 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.710 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:40:37.710 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:40:37.710 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.711 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.711 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.711 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.712 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:40:37.712 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:40:37.712 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.713 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.713 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.713 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.713 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.714 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:40:37.714 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:40:37.714 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.715 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.715 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.715 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.715 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.716 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:40:37.716 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:40:37.716 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.717 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.717 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.717 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.717 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.718 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:40:37.718 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:40:37.718 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.719 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:37.719 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:40:37.719 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:40:37.719 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.720 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:40:37.720 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:40:37.720 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:40:37.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:40:37.721 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:40:37.721 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.722 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:40:37.722 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:40:37.722 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:40:37.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:40:37.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:40:37.723 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.724 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.724 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:40:37.725 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:40:37.725 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:40:37.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:40:37.726 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.726 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:37.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:37.726 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:40:37.727 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:40:37.727 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:40:37.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:40:37.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:40:37.728 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:40:37.728 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.728 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:37.728 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:37.729 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:40:37.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b30 msgr2=0x7f35b0055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b30 0x7f35b0055ff0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f35c8005000 tx=0x7f35d0064d40).stop 2022-07-21T01:40:37.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d00a59f0 msgr2=0x7f35d00a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d00a59f0 0x7f35d00a5e50 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f35c400b000 tx=0x7f35c4006df0).stop 2022-07-21T01:40:37.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 shutdown_connections 2022-07-21T01:40:37.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b30 0x7f35b0055ff0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d00a2f30 0x7f35d00a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d00a59f0 0x7f35d00a5e50 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 --2- 172.21.15.3:0/388872910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35d00b0f50 0x7f35d0153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 >> 172.21.15.3:0/388872910 conn(0x7f35d005d0c0 msgr2=0x7f35d00b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:37.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 shutdown_connections 2022-07-21T01:40:37.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.689+0000 7f35d6cea700 1 -- 172.21.15.3:0/388872910 wait complete. 2022-07-21T01:40:37.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:37.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f72932700 1 Processor -- start 2022-07-21T01:40:37.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f72932700 1 -- start start 2022-07-21T01:40:37.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f72932700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f6c0b6420 0x7f3f6c061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f72932700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 0x7f3f6c1434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f72932700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c0b60e0 con 0x7f3f6c0b41f0 2022-07-21T01:40:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f72932700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6c0b2100 con 0x7f3f6c0b6420 2022-07-21T01:40:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f72932700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0b23a0 con 0x7f3f6c142ff0 2022-07-21T01:40:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f7112f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3f6c0b41f0 0x7f3f6c061af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56496/0 (socket says 172.21.15.3:56496) 2022-07-21T01:40:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f7112f700 1 -- 172.21.15.3:0/2934598974 learned_addr learned my addr 172.21.15.3:0/2934598974 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f71930700 1 --2- 172.21.15.3:0/2934598974 >> v2:172.21.15.143:3300/0 conn(0x7f3f6c0b6420 0x7f3f6c061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f72131700 1 --2- 172.21.15.3:0/2934598974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 0x7f3f6c1434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f7092e700 1 -- 172.21.15.3:0/2934598974 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257518057 0 0) 0x7f3f60001110 con 0x7f3f6c0b41f0 2022-07-21T01:40:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f7092e700 1 -- 172.21.15.3:0/2934598974 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f54003260 con 0x7f3f6c0b41f0 2022-07-21T01:40:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f71930700 1 -- 172.21.15.3:0/2934598974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 msgr2=0x7f3f6c1434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f71930700 1 --2- 172.21.15.3:0/2934598974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 0x7f3f6c1434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f71930700 1 -- 172.21.15.3:0/2934598974 >> v1:172.21.15.3:6789/0 conn(0x7f3f6c0b41f0 legacy=0x7f3f6c061af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f71930700 1 -- 172.21.15.3:0/2934598974 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c0b2640 con 0x7f3f6c0b6420 2022-07-21T01:40:37.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f71930700 1 --2- 172.21.15.3:0/2934598974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b6420 0x7f3f6c061600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f3f68002f70 tx=0x7f3f68006df0).ready entity=mon.1 client_cookie=a3d4af84b14e3f0a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:37.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f7092e700 1 -- 172.21.15.3:0/2934598974 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6800d4a0 con 0x7f3f6c0b6420 2022-07-21T01:40:37.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f7092e700 1 -- 172.21.15.3:0/2934598974 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f68000e90 con 0x7f3f6c0b6420 2022-07-21T01:40:37.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.841+0000 7f3f7092e700 1 -- 172.21.15.3:0/2934598974 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6800d7f0 con 0x7f3f6c0b6420 2022-07-21T01:40:37.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 -- 172.21.15.3:0/2934598974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b6420 msgr2=0x7f3f6c061600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 --2- 172.21.15.3:0/2934598974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b6420 0x7f3f6c061600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f3f68002f70 tx=0x7f3f68006df0).stop 2022-07-21T01:40:37.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 -- 172.21.15.3:0/2934598974 shutdown_connections 2022-07-21T01:40:37.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 --2- 172.21.15.3:0/2934598974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 0x7f3f6c1434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 --2- 172.21.15.3:0/2934598974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b6420 0x7f3f6c061600 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 -- 172.21.15.3:0/2934598974 >> 172.21.15.3:0/2934598974 conn(0x7f3f6c05c620 msgr2=0x7f3f6c05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:37.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 -- 172.21.15.3:0/2934598974 shutdown_connections 2022-07-21T01:40:37.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 -- 172.21.15.3:0/2934598974 wait complete. 2022-07-21T01:40:37.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 Processor -- start 2022-07-21T01:40:37.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 -- start start 2022-07-21T01:40:37.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b41f0 0x7f3f6c0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0b6420 0x7f3f6c1405c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 0x7f3f6c0ae750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0afd10 con 0x7f3f6c0b6420 2022-07-21T01:40:37.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c153720 con 0x7f3f6c0b41f0 2022-07-21T01:40:37.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72932700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0b0680 con 0x7f3f6c142ff0 2022-07-21T01:40:37.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f7112f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0b6420 0x7f3f6c1405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f71930700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b41f0 0x7f3f6c0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f71930700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b41f0 0x7f3f6c0aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:53670/0 (socket says 172.21.15.3:53670) 2022-07-21T01:40:37.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f7112f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0b6420 0x7f3f6c1405c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48224/0 (socket says 172.21.15.3:48224) 2022-07-21T01:40:37.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f71930700 1 -- 172.21.15.3:0/2090031347 learned_addr learned my addr 172.21.15.3:0/2090031347 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:37.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72131700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 0x7f3f6c0ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f7112f700 1 -- 172.21.15.3:0/2090031347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 msgr2=0x7f3f6c0ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f7112f700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 0x7f3f6c0ae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f7112f700 1 -- 172.21.15.3:0/2090031347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b41f0 msgr2=0x7f3f6c0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f7112f700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b41f0 0x7f3f6c0aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f7112f700 1 -- 172.21.15.3:0/2090031347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6800b180 con 0x7f3f6c0b6420 2022-07-21T01:40:37.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f72131700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 0x7f3f6c0ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:37.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.845+0000 7f3f7112f700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0b6420 0x7f3f6c1405c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f3f600012a0 tx=0x7f3f600078a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:37.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f5effd700 1 -- 172.21.15.3:0/2090031347 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f600070d0 con 0x7f3f6c0b6420 2022-07-21T01:40:37.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f5effd700 1 -- 172.21.15.3:0/2090031347 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f60007230 con 0x7f3f6c0b6420 2022-07-21T01:40:37.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f5effd700 1 -- 172.21.15.3:0/2090031347 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f60005610 con 0x7f3f6c0b6420 2022-07-21T01:40:37.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c0b0920 con 0x7f3f6c0b6420 2022-07-21T01:40:37.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c140ab0 con 0x7f3f6c0b6420 2022-07-21T01:40:37.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c0b0ab0 con 0x7f3f6c0b6420 2022-07-21T01:40:37.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f5effd700 1 -- 172.21.15.3:0/2090031347 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3f60010070 con 0x7f3f6c0b6420 2022-07-21T01:40:37.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f5effd700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f4c053980 0x7f3f4c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:37.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f5effd700 1 -- 172.21.15.3:0/2090031347 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f3f6006b0d0 con 0x7f3f6c0b6420 2022-07-21T01:40:37.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.849+0000 7f3f71930700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f4c053980 0x7f3f4c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:37.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.853+0000 7f3f71930700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f4c053980 0x7f3f4c055e40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3f6801a000 tx=0x7f3f68014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:37.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.857+0000 7f3f5effd700 1 -- 172.21.15.3:0/2090031347 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f600421d0 con 0x7f3f6c0b6420 2022-07-21T01:40:37.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.977+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f6c0b60e0 con 0x7f3f6c0b6420 2022-07-21T01:40:37.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.981+0000 7f3f5effd700 1 -- 172.21.15.3:0/2090031347 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+13415 (secure 0 0 0) 0x7f3f6c0b60e0 con 0x7f3f6c0b6420 2022-07-21T01:40:37.982 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:37.983 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":97,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:37.374462+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:37.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.981+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f4c053980 msgr2=0x7f3f4c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.981+0000 7f3f72932700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f4c053980 0x7f3f4c055e40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f3f6801a000 tx=0x7f3f68014000).stop 2022-07-21T01:40:37.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.981+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0b6420 msgr2=0x7f3f6c1405c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:37.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.981+0000 7f3f72932700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0b6420 0x7f3f6c1405c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f3f600012a0 tx=0x7f3f600078a0).stop 2022-07-21T01:40:37.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.985+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 shutdown_connections 2022-07-21T01:40:37.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.985+0000 7f3f72932700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f6c142ff0 0x7f3f6c0ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.985+0000 7f3f72932700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0b6420 0x7f3f6c1405c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.985+0000 7f3f72932700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3f4c053980 0x7f3f4c055e40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.985+0000 7f3f72932700 1 --2- 172.21.15.3:0/2090031347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f6c0b41f0 0x7f3f6c0aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:37.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.985+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 >> 172.21.15.3:0/2090031347 conn(0x7f3f6c05c620 msgr2=0x7f3f6c140e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:37.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.985+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 shutdown_connections 2022-07-21T01:40:37.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:37.985+0000 7f3f72932700 1 -- 172.21.15.3:0/2090031347 wait complete. 2022-07-21T01:40:37.996 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:40:37.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:40:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.105+0000 7fe9fea31700 1 Processor -- start 2022-07-21T01:40:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.105+0000 7fe9fea31700 1 -- start start 2022-07-21T01:40:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe9f8064cd0 0x7fe9f8143a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b3ad0 0x7fe9f8143f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f80b2400 con 0x7fe9f80b2e80 2022-07-21T01:40:38.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f8144410 con 0x7fe9f8064cd0 2022-07-21T01:40:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f80623c0 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fd22e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe9f80b2e80 0x7fe9f80b33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56510/0 (socket says 172.21.15.3:56510) 2022-07-21T01:40:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fd22e700 1 -- 172.21.15.3:0/2886944711 learned_addr learned my addr 172.21.15.3:0/2886944711 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fda2f700 1 --2- 172.21.15.3:0/2886944711 >> v2:172.21.15.143:3300/0 conn(0x7fe9f8064cd0 0x7fe9f8143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fe230700 1 --2- 172.21.15.3:0/2886944711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b3ad0 0x7fe9f8143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fca2d700 1 -- 172.21.15.3:0/2886944711 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060645325 0 0) 0x7fe9f80b2400 con 0x7fe9f80b2e80 2022-07-21T01:40:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fca2d700 1 -- 172.21.15.3:0/2886944711 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e0003260 con 0x7fe9f80b2e80 2022-07-21T01:40:38.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fe230700 1 -- 172.21.15.3:0/2886944711 >> v2:172.21.15.143:3300/0 conn(0x7fe9f8064cd0 msgr2=0x7fe9f8143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:38.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fe230700 1 --2- 172.21.15.3:0/2886944711 >> v2:172.21.15.143:3300/0 conn(0x7fe9f8064cd0 0x7fe9f8143a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fe230700 1 -- 172.21.15.3:0/2886944711 >> v1:172.21.15.3:6789/0 conn(0x7fe9f80b2e80 legacy=0x7fe9f80b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fe230700 1 -- 172.21.15.3:0/2886944711 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f80b3f20 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fe230700 1 --2- 172.21.15.3:0/2886944711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b3ad0 0x7fe9f8143f20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe9f00032f0 tx=0x7fe9f00091d0).ready entity=mon.2 client_cookie=76d19619986631fe server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fca2d700 1 -- 172.21.15.3:0/2886944711 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f000b600 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fca2d700 1 -- 172.21.15.3:0/2886944711 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f000b760 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fca2d700 1 -- 172.21.15.3:0/2886944711 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f000b8d0 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 -- 172.21.15.3:0/2886944711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b3ad0 msgr2=0x7fe9f8143f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 --2- 172.21.15.3:0/2886944711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b3ad0 0x7fe9f8143f20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe9f00032f0 tx=0x7fe9f00091d0).stop 2022-07-21T01:40:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 -- 172.21.15.3:0/2886944711 shutdown_connections 2022-07-21T01:40:38.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 --2- 172.21.15.3:0/2886944711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b3ad0 0x7fe9f8143f20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 --2- 172.21.15.3:0/2886944711 >> v2:172.21.15.143:3300/0 conn(0x7fe9f8064cd0 0x7fe9f8143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 -- 172.21.15.3:0/2886944711 >> 172.21.15.3:0/2886944711 conn(0x7fe9f8062d70 msgr2=0x7fe9f805f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:38.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 -- 172.21.15.3:0/2886944711 shutdown_connections 2022-07-21T01:40:38.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 -- 172.21.15.3:0/2886944711 wait complete. 2022-07-21T01:40:38.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 Processor -- start 2022-07-21T01:40:38.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.109+0000 7fe9fea31700 1 -- start start 2022-07-21T01:40:38.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fea31700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9f8064cd0 0x7fe9f8149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fea31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b2e80 0x7fe9f80b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fea31700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9f80b3ad0 0x7fe9f80b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fea31700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f80af990 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fea31700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f80af490 con 0x7fe9f8064cd0 2022-07-21T01:40:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fea31700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f80aeef0 con 0x7fe9f80b2e80 2022-07-21T01:40:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fe230700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9f80b3ad0 0x7fe9f80b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fe230700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9f80b3ad0 0x7fe9f80b0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48238/0 (socket says 172.21.15.3:48238) 2022-07-21T01:40:38.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fe230700 1 -- 172.21.15.3:0/1008371378 learned_addr learned my addr 172.21.15.3:0/1008371378 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:38.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fda2f700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9f8064cd0 0x7fe9f8149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fd22e700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b2e80 0x7fe9f80b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fe230700 1 -- 172.21.15.3:0/1008371378 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b2e80 msgr2=0x7fe9f80b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fe230700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b2e80 0x7fe9f80b0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fe230700 1 -- 172.21.15.3:0/1008371378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9f8064cd0 msgr2=0x7fe9f8149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fe230700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9f8064cd0 0x7fe9f8149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fe230700 1 -- 172.21.15.3:0/1008371378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f0009470 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fe230700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9f80b3ad0 0x7fe9f80b0160 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fe9f00032f0 tx=0x7fe9f0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9eaffd700 1 -- 172.21.15.3:0/1008371378 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f000b8d0 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f80af6f0 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9eaffd700 1 -- 172.21.15.3:0/1008371378 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f001f3f0 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f80aebb0 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.113+0000 7fe9eaffd700 1 -- 172.21.15.3:0/1008371378 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f000b600 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.117+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f80574e0 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.117+0000 7fe9eaffd700 1 -- 172.21.15.3:0/1008371378 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe9f0026030 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.117+0000 7fe9eaffd700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe9d8053b00 0x7fe9d8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.117+0000 7fe9eaffd700 1 -- 172.21.15.3:0/1008371378 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fe9f006c7f0 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.117+0000 7fe9fda2f700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe9d8053b00 0x7fe9d8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.117+0000 7fe9fda2f700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe9d8053b00 0x7fe9d8055fc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fe9f400b000 tx=0x7fe9f4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.125+0000 7fe9eaffd700 1 -- 172.21.15.3:0/1008371378 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe9f0043850 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe9f805db60 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9eaffd700 1 -- 172.21.15.3:0/1008371378 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v97) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe9f0024020 con 0x7fe9f80b3ad0 2022-07-21T01:40:38.254 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:40:38.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe9d8053b00 msgr2=0x7fe9d8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe9d8053b00 0x7fe9d8055fc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fe9f400b000 tx=0x7fe9f4006df0).stop 2022-07-21T01:40:38.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9f80b3ad0 msgr2=0x7fe9f80b0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9f80b3ad0 0x7fe9f80b0160 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fe9f00032f0 tx=0x7fe9f0016000).stop 2022-07-21T01:40:38.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 shutdown_connections 2022-07-21T01:40:38.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9f80b3ad0 0x7fe9f80b0160 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9f80b2e80 0x7fe9f80b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe9d8053b00 0x7fe9d8055fc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 --2- 172.21.15.3:0/1008371378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9f8064cd0 0x7fe9f8149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 >> 172.21.15.3:0/1008371378 conn(0x7fe9f8062d70 msgr2=0x7fe9f8146860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:38.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 shutdown_connections 2022-07-21T01:40:38.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.253+0000 7fe9fea31700 1 -- 172.21.15.3:0/1008371378 wait complete. 2022-07-21T01:40:38.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:40:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.377+0000 7f9ec8829700 1 Processor -- start 2022-07-21T01:40:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.377+0000 7f9ec8829700 1 -- start start 2022-07-21T01:40:38.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.377+0000 7f9ec8829700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9ec40b1180 0x7f9ec4141920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.377+0000 7f9ec8829700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec40aac20 con 0x7f9ec40b28b0 2022-07-21T01:40:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.377+0000 7f9ec8829700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec40b2c90 con 0x7f9ec413e8d0 2022-07-21T01:40:38.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.377+0000 7f9ec8829700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec4140e20 con 0x7f9ec40b1180 2022-07-21T01:40:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.377+0000 7f9ec2d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9ec40b28b0 0x7f9ec413e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56526/0 (socket says 172.21.15.3:56526) 2022-07-21T01:40:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.377+0000 7f9ec2d9d700 1 -- 172.21.15.3:0/541412296 learned_addr learned my addr 172.21.15.3:0/541412296 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:38.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.377+0000 7f9ec359e700 1 --2- 172.21.15.3:0/541412296 >> v2:172.21.15.143:3300/0 conn(0x7f9ec40b1180 0x7f9ec4141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.381+0000 7f9ec359e700 1 -- 172.21.15.3:0/541412296 >> v1:172.21.15.143:6789/0 conn(0x7f9ec413e8d0 legacy=0x7f9ec4141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.381+0000 7f9ec359e700 1 -- 172.21.15.3:0/541412296 >> v1:172.21.15.3:6789/0 conn(0x7f9ec40b28b0 legacy=0x7f9ec413e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.381+0000 7f9ec359e700 1 -- 172.21.15.3:0/541412296 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec40aa440 con 0x7f9ec40b1180 2022-07-21T01:40:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.381+0000 7f9ec359e700 1 --2- 172.21.15.3:0/541412296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40b1180 0x7f9ec4141920 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f9ebc0032f0 tx=0x7f9ebc0091d0).ready entity=mon.1 client_cookie=273b9b3dd7ac03d1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.381+0000 7f9ec259c700 1 -- 172.21.15.3:0/541412296 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ebc009ea0 con 0x7f9ec40b1180 2022-07-21T01:40:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.381+0000 7f9ec259c700 1 -- 172.21.15.3:0/541412296 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ebc00b8a0 con 0x7f9ec40b1180 2022-07-21T01:40:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.381+0000 7f9ec259c700 1 -- 172.21.15.3:0/541412296 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ebc00ba10 con 0x7f9ec40b1180 2022-07-21T01:40:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.385+0000 7f9ec8829700 1 -- 172.21.15.3:0/541412296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40b1180 msgr2=0x7f9ec4141920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.385+0000 7f9ec8829700 1 --2- 172.21.15.3:0/541412296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40b1180 0x7f9ec4141920 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f9ebc0032f0 tx=0x7f9ebc0091d0).stop 2022-07-21T01:40:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.385+0000 7f9ec8829700 1 -- 172.21.15.3:0/541412296 shutdown_connections 2022-07-21T01:40:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.385+0000 7f9ec8829700 1 --2- 172.21.15.3:0/541412296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40b1180 0x7f9ec4141920 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.385+0000 7f9ec8829700 1 -- 172.21.15.3:0/541412296 >> 172.21.15.3:0/541412296 conn(0x7f9ec4060310 msgr2=0x7f9ec40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.385+0000 7f9ec8829700 1 -- 172.21.15.3:0/541412296 shutdown_connections 2022-07-21T01:40:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.385+0000 7f9ec8829700 1 -- 172.21.15.3:0/541412296 wait complete. 2022-07-21T01:40:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.385+0000 7f9ec8829700 1 Processor -- start 2022-07-21T01:40:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.385+0000 7f9ec8829700 1 -- start start 2022-07-21T01:40:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec8829700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40b1180 0x7f9ec413f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec8829700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40b28b0 0x7f9ec4141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec8829700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4144d50 0x7f9ec41395b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec8829700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec4144bd0 con 0x7f9ec40b1180 2022-07-21T01:40:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec8829700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec40ad1a0 con 0x7f9ec4144d50 2022-07-21T01:40:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec8829700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec4142ad0 con 0x7f9ec40b28b0 2022-07-21T01:40:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40b1180 0x7f9ec413f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40b1180 0x7f9ec413f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48250/0 (socket says 172.21.15.3:48250) 2022-07-21T01:40:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec2d9d700 1 -- 172.21.15.3:0/3824578601 learned_addr learned my addr 172.21.15.3:0/3824578601 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec359e700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4144d50 0x7f9ec41395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9eba59c700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40b28b0 0x7f9ec4141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec2d9d700 1 -- 172.21.15.3:0/3824578601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40b28b0 msgr2=0x7f9ec4141fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec2d9d700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40b28b0 0x7f9ec4141fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec2d9d700 1 -- 172.21.15.3:0/3824578601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4144d50 msgr2=0x7f9ec41395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec2d9d700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4144d50 0x7f9ec41395b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec2d9d700 1 -- 172.21.15.3:0/3824578601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc009470 con 0x7f9ec40b1180 2022-07-21T01:40:38.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec2d9d700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40b1180 0x7f9ec413f000 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f9ec4061c10 tx=0x7f9eb4007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec0d99700 1 -- 172.21.15.3:0/3824578601 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9eb40070d0 con 0x7f9ec40b1180 2022-07-21T01:40:38.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec0d99700 1 -- 172.21.15.3:0/3824578601 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb4007230 con 0x7f9ec40b1180 2022-07-21T01:40:38.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec40ae1b0 con 0x7f9ec40b1180 2022-07-21T01:40:38.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec0d99700 1 -- 172.21.15.3:0/3824578601 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9eb4005610 con 0x7f9ec40b1180 2022-07-21T01:40:38.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.389+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec413f6d0 con 0x7f9ec40b1180 2022-07-21T01:40:38.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.393+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec40ae340 con 0x7f9ec40b1180 2022-07-21T01:40:38.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.393+0000 7f9ec0d99700 1 -- 172.21.15.3:0/3824578601 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9eb4010070 con 0x7f9ec40b1180 2022-07-21T01:40:38.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.393+0000 7f9ec0d99700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ea8053a70 0x7f9ea8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.393+0000 7f9ec0d99700 1 -- 172.21.15.3:0/3824578601 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f9eb406b010 con 0x7f9ec40b1180 2022-07-21T01:40:38.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.393+0000 7f9eba59c700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ea8053a70 0x7f9ea8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.393+0000 7f9eba59c700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ea8053a70 0x7f9ea8055f30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f9eac00d000 tx=0x7f9eac003040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.397+0000 7f9ec0d99700 1 -- 172.21.15.3:0/3824578601 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9eb4042110 con 0x7f9ec40b1180 2022-07-21T01:40:38.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.521+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9ec40aac20 con 0x7f9ec40b1180 2022-07-21T01:40:38.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.521+0000 7f9ec0d99700 1 -- 172.21.15.3:0/3824578601 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v97) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9ec40aac20 con 0x7f9ec40b1180 2022-07-21T01:40:38.522 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:38.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ea8053a70 msgr2=0x7f9ea8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ea8053a70 0x7f9ea8055f30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f9eac00d000 tx=0x7f9eac003040).stop 2022-07-21T01:40:38.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40b1180 msgr2=0x7f9ec413f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40b1180 0x7f9ec413f000 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f9ec4061c10 tx=0x7f9eb4007c00).stop 2022-07-21T01:40:38.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 shutdown_connections 2022-07-21T01:40:38.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ea8053a70 0x7f9ea8055f30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4144d50 0x7f9ec41395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40b28b0 0x7f9ec4141fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 --2- 172.21.15.3:0/3824578601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40b1180 0x7f9ec413f000 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 >> 172.21.15.3:0/3824578601 conn(0x7f9ec4060310 msgr2=0x7f9ec4140470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:38.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 shutdown_connections 2022-07-21T01:40:38.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.525+0000 7f9ec8829700 1 -- 172.21.15.3:0/3824578601 wait complete. 2022-07-21T01:40:38.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:40:38.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.645+0000 7fc84e73c700 1 Processor -- start 2022-07-21T01:40:38.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.645+0000 7fc84e73c700 1 -- start start 2022-07-21T01:40:38.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.645+0000 7fc84e73c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b0270 0x7fc848143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.645+0000 7fc84e73c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc8480ac880 0x7fc8481434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.645+0000 7fc84e73c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc848067a10 con 0x7fc8480b2710 2022-07-21T01:40:38.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.645+0000 7fc84e73c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8480ad740 con 0x7fc8480ac880 2022-07-21T01:40:38.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.645+0000 7fc84e73c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc848067cb0 con 0x7fc8480b0270 2022-07-21T01:40:38.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84cf39700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b0270 0x7fc848143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84d73a700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fc8480b2710 0x7fc8481401e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:58654/0 (socket says 172.21.15.3:58654) 2022-07-21T01:40:38.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84d73a700 1 -- 172.21.15.3:0/1083666891 learned_addr learned my addr 172.21.15.3:0/1083666891 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:38.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84df3b700 1 --2- 172.21.15.3:0/1083666891 >> v2:172.21.15.143:3300/0 conn(0x7fc8480ac880 0x7fc8481434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc83ffff700 1 -- 172.21.15.3:0/1083666891 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558455648 0 0) 0x7fc848067a10 con 0x7fc8480b2710 2022-07-21T01:40:38.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84cf39700 1 -- 172.21.15.3:0/1083666891 >> v2:172.21.15.143:3300/0 conn(0x7fc8480ac880 msgr2=0x7fc8481434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:38.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84cf39700 1 --2- 172.21.15.3:0/1083666891 >> v2:172.21.15.143:3300/0 conn(0x7fc8480ac880 0x7fc8481434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84cf39700 1 -- 172.21.15.3:0/1083666891 >> v1:172.21.15.143:6789/0 conn(0x7fc8480b2710 legacy=0x7fc8481401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84cf39700 1 -- 172.21.15.3:0/1083666891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc848147160 con 0x7fc8480b0270 2022-07-21T01:40:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84df3b700 1 --2- 172.21.15.3:0/1083666891 >> v2:172.21.15.143:3300/0 conn(0x7fc8480ac880 0x7fc8481434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84cf39700 1 --2- 172.21.15.3:0/1083666891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b0270 0x7fc848143000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc844002f70 tx=0x7fc844006df0).ready entity=mon.2 client_cookie=1c49218b194e737e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc83ffff700 1 -- 172.21.15.3:0/1083666891 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc84400d570 con 0x7fc8480b0270 2022-07-21T01:40:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc83ffff700 1 -- 172.21.15.3:0/1083666891 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc844000e90 con 0x7fc8480b0270 2022-07-21T01:40:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc83ffff700 1 -- 172.21.15.3:0/1083666891 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc84400d8c0 con 0x7fc8480b0270 2022-07-21T01:40:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 -- 172.21.15.3:0/1083666891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b0270 msgr2=0x7fc848143000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 --2- 172.21.15.3:0/1083666891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b0270 0x7fc848143000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc844002f70 tx=0x7fc844006df0).stop 2022-07-21T01:40:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 -- 172.21.15.3:0/1083666891 shutdown_connections 2022-07-21T01:40:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 --2- 172.21.15.3:0/1083666891 >> v2:172.21.15.143:3300/0 conn(0x7fc8480ac880 0x7fc8481434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 --2- 172.21.15.3:0/1083666891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b0270 0x7fc848143000 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 -- 172.21.15.3:0/1083666891 >> 172.21.15.3:0/1083666891 conn(0x7fc848060310 msgr2=0x7fc8480b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:38.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 -- 172.21.15.3:0/1083666891 shutdown_connections 2022-07-21T01:40:38.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 -- 172.21.15.3:0/1083666891 wait complete. 2022-07-21T01:40:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 Processor -- start 2022-07-21T01:40:38.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.649+0000 7fc84e73c700 1 -- start start 2022-07-21T01:40:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84e73c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8480ac880 0x7fc8480b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84e73c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8480b0270 0x7fc8481560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84e73c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b2710 0x7fc8480aa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84e73c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8481559d0 con 0x7fc8480ac880 2022-07-21T01:40:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84e73c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc848140ea0 con 0x7fc8480b0270 2022-07-21T01:40:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84e73c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc848155c00 con 0x7fc8480b2710 2022-07-21T01:40:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84d73a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8480ac880 0x7fc8480b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84d73a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8480ac880 0x7fc8480b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48262/0 (socket says 172.21.15.3:48262) 2022-07-21T01:40:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84d73a700 1 -- 172.21.15.3:0/1439428423 learned_addr learned my addr 172.21.15.3:0/1439428423 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84cf39700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8480b0270 0x7fc8481560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84df3b700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b2710 0x7fc8480aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84d73a700 1 -- 172.21.15.3:0/1439428423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b2710 msgr2=0x7fc8480aa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84d73a700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b2710 0x7fc8480aa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84d73a700 1 -- 172.21.15.3:0/1439428423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8480b0270 msgr2=0x7fc8481560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84d73a700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8480b0270 0x7fc8481560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84d73a700 1 -- 172.21.15.3:0/1439428423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc840009470 con 0x7fc8480ac880 2022-07-21T01:40:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84cf39700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8480b0270 0x7fc8481560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84d73a700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8480ac880 0x7fc8480b43e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7fc838000ed0 tx=0x7fc838007ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc83e7fc700 1 -- 172.21.15.3:0/1439428423 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc83800ce20 con 0x7fc8480ac880 2022-07-21T01:40:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc83e7fc700 1 -- 172.21.15.3:0/1439428423 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8380072e0 con 0x7fc8480ac880 2022-07-21T01:40:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84400b180 con 0x7fc8480ac880 2022-07-21T01:40:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc83e7fc700 1 -- 172.21.15.3:0/1439428423 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc838005610 con 0x7fc8480ac880 2022-07-21T01:40:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc848141140 con 0x7fc8480ac880 2022-07-21T01:40:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.653+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc848155dc0 con 0x7fc8480ac880 2022-07-21T01:40:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.657+0000 7fc83e7fc700 1 -- 172.21.15.3:0/1439428423 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc838010070 con 0x7fc8480ac880 2022-07-21T01:40:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.657+0000 7fc83e7fc700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc8300539e0 0x7fc830055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.657+0000 7fc83e7fc700 1 -- 172.21.15.3:0/1439428423 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fc83806b510 con 0x7fc8480ac880 2022-07-21T01:40:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.657+0000 7fc84cf39700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc8300539e0 0x7fc830055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.657+0000 7fc84cf39700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc8300539e0 0x7fc830055ea0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fc844018000 tx=0x7fc844014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.661+0000 7fc83e7fc700 1 -- 172.21.15.3:0/1439428423 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc838042570 con 0x7fc8480ac880 2022-07-21T01:40:38.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.789+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc84813b0a0 con 0x7fc8480ac880 2022-07-21T01:40:38.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.789+0000 7fc83e7fc700 1 -- 172.21.15.3:0/1439428423 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v97) v1 ==== 84+0+143 (secure 0 0 0) 0x7fc838045b90 con 0x7fc8480ac880 2022-07-21T01:40:38.791 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:38.791 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:40:38.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:38.792 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:40:38.792 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:40:38.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:38.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:38.793 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:40:38.793 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:40:38.794 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:38.794 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:40:38.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.789+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc8300539e0 msgr2=0x7fc830055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.789+0000 7fc84e73c700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc8300539e0 0x7fc830055ea0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fc844018000 tx=0x7fc844014000).stop 2022-07-21T01:40:38.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8480ac880 msgr2=0x7fc8480b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8480ac880 0x7fc8480b43e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7fc838000ed0 tx=0x7fc838007ce0).stop 2022-07-21T01:40:38.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 shutdown_connections 2022-07-21T01:40:38.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8480b2710 0x7fc8480aa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc8300539e0 0x7fc830055ea0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8480b0270 0x7fc8481560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 --2- 172.21.15.3:0/1439428423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8480ac880 0x7fc8480b43e0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 >> 172.21.15.3:0/1439428423 conn(0x7fc848060310 msgr2=0x7fc8481404c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:38.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 shutdown_connections 2022-07-21T01:40:38.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.793+0000 7fc84e73c700 1 -- 172.21.15.3:0/1439428423 wait complete. 2022-07-21T01:40:38.804 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:40:38.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:40:38.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eba856700 1 Processor -- start 2022-07-21T01:40:38.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eba856700 1 -- start start 2022-07-21T01:40:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eba856700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7eb40b6ee0 0x7f7eb40b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eba856700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb4061d90 con 0x7f7eb40619b0 2022-07-21T01:40:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eba856700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb40ba5a0 con 0x7f7eb40b5ed0 2022-07-21T01:40:38.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eba856700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb4143fa0 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eb9854700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7eb40619b0 0x7f7eb4143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56550/0 (socket says 172.21.15.3:56550) 2022-07-21T01:40:38.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eb9854700 1 -- 172.21.15.3:0/120866306 learned_addr learned my addr 172.21.15.3:0/120866306 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:38.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eb9053700 1 --2- 172.21.15.3:0/120866306 >> v2:172.21.15.143:3300/0 conn(0x7f7eb40b6ee0 0x7f7eb40b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eb8852700 1 -- 172.21.15.3:0/120866306 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137154999 0 0) 0x7f7eb4061d90 con 0x7f7eb40619b0 2022-07-21T01:40:38.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eb8852700 1 -- 172.21.15.3:0/120866306 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e9c003260 con 0x7f7eb40619b0 2022-07-21T01:40:38.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.913+0000 7f7eb9053700 1 -- 172.21.15.3:0/120866306 >> v1:172.21.15.143:6789/0 conn(0x7f7eb40b5ed0 legacy=0x7f7eb40b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eb9053700 1 -- 172.21.15.3:0/120866306 >> v1:172.21.15.3:6789/0 conn(0x7f7eb40619b0 legacy=0x7f7eb4143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eb9053700 1 -- 172.21.15.3:0/120866306 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4143150 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eb9053700 1 --2- 172.21.15.3:0/120866306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7eb40b6ee0 0x7f7eb40b48a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f7eb0002f70 tx=0x7f7eb0006df0).ready entity=mon.1 client_cookie=407a13921b4ea6f0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eb8852700 1 -- 172.21.15.3:0/120866306 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb000d610 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eb8852700 1 -- 172.21.15.3:0/120866306 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb0000e90 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eb8852700 1 -- 172.21.15.3:0/120866306 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb000d960 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eba856700 1 -- 172.21.15.3:0/120866306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7eb40b6ee0 msgr2=0x7f7eb40b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eba856700 1 --2- 172.21.15.3:0/120866306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7eb40b6ee0 0x7f7eb40b48a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f7eb0002f70 tx=0x7f7eb0006df0).stop 2022-07-21T01:40:38.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eba856700 1 -- 172.21.15.3:0/120866306 shutdown_connections 2022-07-21T01:40:38.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eba856700 1 --2- 172.21.15.3:0/120866306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7eb40b6ee0 0x7f7eb40b48a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eba856700 1 -- 172.21.15.3:0/120866306 >> 172.21.15.3:0/120866306 conn(0x7f7eb406af80 msgr2=0x7f7eb405fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:38.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eba856700 1 -- 172.21.15.3:0/120866306 shutdown_connections 2022-07-21T01:40:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eba856700 1 -- 172.21.15.3:0/120866306 wait complete. 2022-07-21T01:40:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eba856700 1 Processor -- start 2022-07-21T01:40:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.917+0000 7f7eba856700 1 -- start start 2022-07-21T01:40:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba856700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7eb40619b0 0x7f7eb4112bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba856700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7eb40b5ed0 0x7f7eb414f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba856700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7eb40b6ee0 0x7f7eb414fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba856700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb413cb50 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba856700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb4153270 con 0x7f7eb40619b0 2022-07-21T01:40:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba856700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb40b02f0 con 0x7f7eb40b5ed0 2022-07-21T01:40:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba055700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7eb40b6ee0 0x7f7eb414fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eb9854700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7eb40619b0 0x7f7eb4112bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba055700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7eb40b6ee0 0x7f7eb414fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48280/0 (socket says 172.21.15.3:48280) 2022-07-21T01:40:38.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eb9053700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7eb40b5ed0 0x7f7eb414f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba055700 1 -- 172.21.15.3:0/616769288 learned_addr learned my addr 172.21.15.3:0/616769288 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba055700 1 -- 172.21.15.3:0/616769288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7eb40b5ed0 msgr2=0x7f7eb414f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba055700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7eb40b5ed0 0x7f7eb414f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba055700 1 -- 172.21.15.3:0/616769288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7eb40619b0 msgr2=0x7f7eb4112bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:38.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba055700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7eb40619b0 0x7f7eb4112bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:38.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba055700 1 -- 172.21.15.3:0/616769288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb000b180 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba055700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7eb40b6ee0 0x7f7eb414fb60 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f7eb4062290 tx=0x7f7eac007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7ea2ffd700 1 -- 172.21.15.3:0/616769288 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eac00f470 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7ea2ffd700 1 -- 172.21.15.3:0/616769288 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac00f5d0 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb40b0590 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7ea2ffd700 1 -- 172.21.15.3:0/616769288 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eac00f930 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb4152f30 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb40b0720 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.921+0000 7f7ea2ffd700 1 -- 172.21.15.3:0/616769288 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7eac019030 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.925+0000 7f7ea2ffd700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e94053a70 0x7f7e94055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.925+0000 7f7ea2ffd700 1 -- 172.21.15.3:0/616769288 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f7eac06b3e0 con 0x7f7eb40b6ee0 2022-07-21T01:40:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.925+0000 7f7eb9854700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e94053a70 0x7f7e94055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.925+0000 7f7eb9854700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e94053a70 0x7f7e94055f30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f7ea400d000 tx=0x7f7ea4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:38.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:38.929+0000 7f7ea2ffd700 1 -- 172.21.15.3:0/616769288 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7eac042370 con 0x7f7eb40b6ee0 2022-07-21T01:40:39.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.053+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f7eb4153430 con 0x7f7eb40b6ee0 2022-07-21T01:40:39.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.701+0000 7f7ea2ffd700 1 -- 172.21.15.3:0/616769288 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v98) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7eac045990 con 0x7f7eb40b6ee0 2022-07-21T01:40:39.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e94053a70 msgr2=0x7f7e94055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e94053a70 0x7f7e94055f30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f7ea400d000 tx=0x7f7ea4009040).stop 2022-07-21T01:40:39.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7eb40b6ee0 msgr2=0x7f7eb414fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:39.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7eb40b6ee0 0x7f7eb414fb60 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f7eb4062290 tx=0x7f7eac007f40).stop 2022-07-21T01:40:39.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 shutdown_connections 2022-07-21T01:40:39.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7eb40b6ee0 0x7f7eb414fb60 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:39.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7eb40b5ed0 0x7f7eb414f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:39.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7e94053a70 0x7f7e94055f30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:39.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 --2- 172.21.15.3:0/616769288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7eb40619b0 0x7f7eb4112bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:39.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 >> 172.21.15.3:0/616769288 conn(0x7f7eb406af80 msgr2=0x7f7eb405ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:39.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 shutdown_connections 2022-07-21T01:40:39.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.705+0000 7f7eba856700 1 -- 172.21.15.3:0/616769288 wait complete. 2022-07-21T01:40:39.715 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:40:39.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:40:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.833+0000 7f908a9f3700 1 Processor -- start 2022-07-21T01:40:39.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.833+0000 7f908a9f3700 1 -- start start 2022-07-21T01:40:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4310 0x7f9084140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:39.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084143cf0 con 0x7f9084146ae0 2022-07-21T01:40:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084143f90 con 0x7f90840b4820 2022-07-21T01:40:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90841476a0 con 0x7f90840b4310 2022-07-21T01:40:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f9083fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9084146ae0 0x7f90841403c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56564/0 (socket says 172.21.15.3:56564) 2022-07-21T01:40:39.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f9083fff700 1 -- 172.21.15.3:0/4256846832 learned_addr learned my addr 172.21.15.3:0/4256846832 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f90837fe700 1 --2- 172.21.15.3:0/4256846832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4310 0x7f9084140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f9082ffd700 1 -- 172.21.15.3:0/4256846832 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445586577 0 0) 0x7f9084143cf0 con 0x7f9084146ae0 2022-07-21T01:40:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f9082ffd700 1 -- 172.21.15.3:0/4256846832 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9064003260 con 0x7f9084146ae0 2022-07-21T01:40:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f9082ffd700 1 -- 172.21.15.3:0/4256846832 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702949858 0 0) 0x7f9084143f90 con 0x7f90840b4820 2022-07-21T01:40:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f90837fe700 1 -- 172.21.15.3:0/4256846832 >> v1:172.21.15.143:6789/0 conn(0x7f90840b4820 legacy=0x7f90840b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f90837fe700 1 -- 172.21.15.3:0/4256846832 >> v1:172.21.15.3:6789/0 conn(0x7f9084146ae0 legacy=0x7f90841403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f90837fe700 1 -- 172.21.15.3:0/4256846832 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90840b0e20 con 0x7f90840b4310 2022-07-21T01:40:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f90837fe700 1 --2- 172.21.15.3:0/4256846832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4310 0x7f9084140ad0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9074002f70 tx=0x7f9074006df0).ready entity=mon.2 client_cookie=7cf5b509daac3b50 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f9082ffd700 1 -- 172.21.15.3:0/4256846832 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f907400d570 con 0x7f90840b4310 2022-07-21T01:40:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f9082ffd700 1 -- 172.21.15.3:0/4256846832 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9074000e90 con 0x7f90840b4310 2022-07-21T01:40:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f9082ffd700 1 -- 172.21.15.3:0/4256846832 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f907400d8c0 con 0x7f90840b4310 2022-07-21T01:40:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 -- 172.21.15.3:0/4256846832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4310 msgr2=0x7f9084140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 --2- 172.21.15.3:0/4256846832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4310 0x7f9084140ad0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9074002f70 tx=0x7f9074006df0).stop 2022-07-21T01:40:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 -- 172.21.15.3:0/4256846832 shutdown_connections 2022-07-21T01:40:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 --2- 172.21.15.3:0/4256846832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4310 0x7f9084140ad0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 -- 172.21.15.3:0/4256846832 >> 172.21.15.3:0/4256846832 conn(0x7f908405c620 msgr2=0x7f908405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 -- 172.21.15.3:0/4256846832 shutdown_connections 2022-07-21T01:40:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.837+0000 7f908a9f3700 1 -- 172.21.15.3:0/4256846832 wait complete. 2022-07-21T01:40:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 Processor -- start 2022-07-21T01:40:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 -- start start 2022-07-21T01:40:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90840b4310 0x7f90840b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4820 0x7f908413b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9084146ae0 0x7f908413b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9084148480 con 0x7f9084146ae0 2022-07-21T01:40:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90841486b0 con 0x7f90840b4310 2022-07-21T01:40:39.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90840b9db0 con 0x7f90840b4820 2022-07-21T01:40:39.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f9088f90700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9084146ae0 0x7f908413b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:39.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f9088f90700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9084146ae0 0x7f908413b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48294/0 (socket says 172.21.15.3:48294) 2022-07-21T01:40:39.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f9088f90700 1 -- 172.21.15.3:0/1587053187 learned_addr learned my addr 172.21.15.3:0/1587053187 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90837fe700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4820 0x7f908413b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f9083fff700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90840b4310 0x7f90840b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90837fe700 1 -- 172.21.15.3:0/1587053187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90840b4310 msgr2=0x7f90840b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90837fe700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90840b4310 0x7f90840b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90837fe700 1 -- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9084146ae0 msgr2=0x7f908413b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90837fe700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9084146ae0 0x7f908413b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90837fe700 1 -- 172.21.15.3:0/1587053187 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907400b180 con 0x7f90840b4820 2022-07-21T01:40:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f9088f90700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9084146ae0 0x7f908413b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90837fe700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4820 0x7f908413b010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f90840b9d00 tx=0x7f9074014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90817fa700 1 -- 172.21.15.3:0/1587053187 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9074016070 con 0x7f90840b4820 2022-07-21T01:40:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90817fa700 1 -- 172.21.15.3:0/1587053187 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90740137d0 con 0x7f90840b4820 2022-07-21T01:40:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90840ba050 con 0x7f90840b4820 2022-07-21T01:40:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f90817fa700 1 -- 172.21.15.3:0/1587053187 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9074013b30 con 0x7f90840b4820 2022-07-21T01:40:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.841+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9084147860 con 0x7f90840b4820 2022-07-21T01:40:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.845+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90840574e0 con 0x7f90840b4820 2022-07-21T01:40:39.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.845+0000 7f90817fa700 1 -- 172.21.15.3:0/1587053187 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9074025070 con 0x7f90840b4820 2022-07-21T01:40:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.845+0000 7f90817fa700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90680539e0 0x7f9068055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:39.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.845+0000 7f90817fa700 1 -- 172.21.15.3:0/1587053187 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f907406cbe0 con 0x7f90840b4820 2022-07-21T01:40:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.845+0000 7f9083fff700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90680539e0 0x7f9068055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.845+0000 7f9083fff700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90680539e0 0x7f9068055ea0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f906c00d000 tx=0x7f906c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.853+0000 7f90817fa700 1 -- 172.21.15.3:0/1587053187 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9074043b70 con 0x7f90840b4820 2022-07-21T01:40:39.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:39.985+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f90840ba7b0 con 0x7f90840b4820 2022-07-21T01:40:40.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.709+0000 7f90817fa700 1 -- 172.21.15.3:0/1587053187 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v99) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9074023030 con 0x7f90840b4820 2022-07-21T01:40:40.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.713+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90680539e0 msgr2=0x7f9068055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:40.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.713+0000 7f908a9f3700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90680539e0 0x7f9068055ea0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f906c00d000 tx=0x7f906c009040).stop 2022-07-21T01:40:40.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.713+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4820 msgr2=0x7f908413b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:40.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.713+0000 7f908a9f3700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4820 0x7f908413b010 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f90840b9d00 tx=0x7f9074014000).stop 2022-07-21T01:40:40.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.717+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 shutdown_connections 2022-07-21T01:40:40.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.717+0000 7f908a9f3700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9084146ae0 0x7f908413b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:40.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.717+0000 7f908a9f3700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90840b4820 0x7f908413b010 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:40.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.717+0000 7f908a9f3700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90680539e0 0x7f9068055ea0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:40.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.717+0000 7f908a9f3700 1 --2- 172.21.15.3:0/1587053187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90840b4310 0x7f90840b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:40.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.717+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 >> 172.21.15.3:0/1587053187 conn(0x7f908405c620 msgr2=0x7f90840b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:40.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.717+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 shutdown_connections 2022-07-21T01:40:40.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.717+0000 7f908a9f3700 1 -- 172.21.15.3:0/1587053187 wait complete. 2022-07-21T01:40:40.720 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:40:40.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:40:40.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0609602700 1 Processor -- start 2022-07-21T01:40:40.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0609602700 1 -- start start 2022-07-21T01:40:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0609602700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06040b3190 0x7f0604060ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:40.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0609602700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040b2730 0x7f0604061190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:40.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0609602700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06040654c0 con 0x7f06040666b0 2022-07-21T01:40:40.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0609602700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0604062190 con 0x7f06040b3190 2022-07-21T01:40:40.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0609602700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06040b3570 con 0x7f06040b2730 2022-07-21T01:40:40.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0608e01700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f06040666b0 0x7f0604066b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56582/0 (socket says 172.21.15.3:56582) 2022-07-21T01:40:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0608e01700 1 -- 172.21.15.3:0/772469326 learned_addr learned my addr 172.21.15.3:0/772469326 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f0603fff700 1 --2- 172.21.15.3:0/772469326 >> v2:172.21.15.3:3300/0 conn(0x7f06040b3190 0x7f0604060ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.841+0000 7f05fb7fe700 1 --2- 172.21.15.3:0/772469326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040b2730 0x7f0604061190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:40.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f06037fe700 1 -- 172.21.15.3:0/772469326 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502064803 0 0) 0x7f06040654c0 con 0x7f06040666b0 2022-07-21T01:40:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f06037fe700 1 -- 172.21.15.3:0/772469326 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e4003260 con 0x7f06040666b0 2022-07-21T01:40:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f05fb7fe700 1 -- 172.21.15.3:0/772469326 >> v2:172.21.15.3:3300/0 conn(0x7f06040b3190 msgr2=0x7f0604060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f05fb7fe700 1 --2- 172.21.15.3:0/772469326 >> v2:172.21.15.3:3300/0 conn(0x7f06040b3190 0x7f0604060ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:40.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f05fb7fe700 1 -- 172.21.15.3:0/772469326 >> v1:172.21.15.3:6789/0 conn(0x7f06040666b0 legacy=0x7f0604066b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f06037fe700 1 -- 172.21.15.3:0/772469326 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3257445372 0 0) 0x7f05e4003260 con 0x7f06040666b0 2022-07-21T01:40:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f05fb7fe700 1 -- 172.21.15.3:0/772469326 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060405fbf0 con 0x7f06040b2730 2022-07-21T01:40:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f0603fff700 1 --2- 172.21.15.3:0/772469326 >> v2:172.21.15.3:3300/0 conn(0x7f06040b3190 0x7f0604060ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f05fb7fe700 1 --2- 172.21.15.3:0/772469326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040b2730 0x7f0604061190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f05fc001b80 tx=0x7f05ec006df0).ready entity=mon.2 client_cookie=2877dae607a9a54b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:40.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f06037fe700 1 -- 172.21.15.3:0/772469326 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05fc0019e0 con 0x7f06040b2730 2022-07-21T01:40:40.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f06037fe700 1 -- 172.21.15.3:0/772469326 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec000e90 con 0x7f06040b2730 2022-07-21T01:40:40.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f06037fe700 1 -- 172.21.15.3:0/772469326 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05ec012460 con 0x7f06040b2730 2022-07-21T01:40:40.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f0609602700 1 -- 172.21.15.3:0/772469326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040b2730 msgr2=0x7f0604061190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:40.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f0609602700 1 --2- 172.21.15.3:0/772469326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040b2730 0x7f0604061190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f05fc001b80 tx=0x7f05ec006df0).stop 2022-07-21T01:40:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f0609602700 1 -- 172.21.15.3:0/772469326 shutdown_connections 2022-07-21T01:40:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f0609602700 1 --2- 172.21.15.3:0/772469326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040b2730 0x7f0604061190 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f0609602700 1 --2- 172.21.15.3:0/772469326 >> v2:172.21.15.3:3300/0 conn(0x7f06040b3190 0x7f0604060ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f0609602700 1 -- 172.21.15.3:0/772469326 >> 172.21.15.3:0/772469326 conn(0x7f0604063870 msgr2=0x7f0604063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f0609602700 1 -- 172.21.15.3:0/772469326 shutdown_connections 2022-07-21T01:40:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.845+0000 7f0609602700 1 -- 172.21.15.3:0/772469326 wait complete. 2022-07-21T01:40:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 Processor -- start 2022-07-21T01:40:40.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 -- start start 2022-07-21T01:40:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040666b0 0x7f0604148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040b2730 0x7f06040ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06040b3190 0x7f0604152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06040b0b40 con 0x7f06040b2730 2022-07-21T01:40:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06040ac2e0 con 0x7f06040b3190 2022-07-21T01:40:40.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06040ac580 con 0x7f06040666b0 2022-07-21T01:40:40.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f05fb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040b2730 0x7f06040ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:40.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f05fb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040b2730 0x7f06040ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48306/0 (socket says 172.21.15.3:48306) 2022-07-21T01:40:40.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f05fb7fe700 1 -- 172.21.15.3:0/2533995381 learned_addr learned my addr 172.21.15.3:0/2533995381 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0608e01700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06040b3190 0x7f0604152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0603fff700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040666b0 0x7f0604148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f05fb7fe700 1 -- 172.21.15.3:0/2533995381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040666b0 msgr2=0x7f0604148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:40.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f05fb7fe700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040666b0 0x7f0604148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f05fb7fe700 1 -- 172.21.15.3:0/2533995381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06040b3190 msgr2=0x7f0604152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f05fb7fe700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06040b3190 0x7f0604152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f05fb7fe700 1 -- 172.21.15.3:0/2533995381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f400b180 con 0x7f06040b2730 2022-07-21T01:40:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f05fb7fe700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040b2730 0x7f06040ace00 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f05fc001b80 tx=0x7f05ec014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:40.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0601ffb700 1 -- 172.21.15.3:0/2533995381 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05ec016070 con 0x7f06040b2730 2022-07-21T01:40:40.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0601ffb700 1 -- 172.21.15.3:0/2533995381 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec00bec0 con 0x7f06040b2730 2022-07-21T01:40:40.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ec00b180 con 0x7f06040b2730 2022-07-21T01:40:40.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0601ffb700 1 -- 172.21.15.3:0/2533995381 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05ec00dd40 con 0x7f06040b2730 2022-07-21T01:40:40.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.849+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0604152180 con 0x7f06040b2730 2022-07-21T01:40:40.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.853+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060405f440 con 0x7f06040b2730 2022-07-21T01:40:40.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.853+0000 7f0601ffb700 1 -- 172.21.15.3:0/2533995381 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f05ec025030 con 0x7f06040b2730 2022-07-21T01:40:40.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.853+0000 7f0601ffb700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05e8053a70 0x7f05e8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:40.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.853+0000 7f0601ffb700 1 -- 172.21.15.3:0/2533995381 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f05ec06d040 con 0x7f06040b2730 2022-07-21T01:40:40.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.853+0000 7f0603fff700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05e8053a70 0x7f05e8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:40.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.853+0000 7f0603fff700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05e8053a70 0x7f05e8055f30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f05f400b000 tx=0x7f05f4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:40.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:40.861+0000 7f0601ffb700 1 -- 172.21.15.3:0/2533995381 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05ec043e80 con 0x7f06040b2730 2022-07-21T01:40:41.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.033+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f0604152340 con 0x7f06040b2730 2022-07-21T01:40:41.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.041+0000 7f0601ffb700 1 -- 172.21.15.3:0/2533995381 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 17 and data pool 18 v46) v1 ==== 189+0+0 (secure 0 0 0) 0x7f05ec023070 con 0x7f06040b2730 2022-07-21T01:40:41.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05e8053a70 msgr2=0x7f05e8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05e8053a70 0x7f05e8055f30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f05f400b000 tx=0x7f05f4006df0).stop 2022-07-21T01:40:41.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040b2730 msgr2=0x7f06040ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040b2730 0x7f06040ace00 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f05fc001b80 tx=0x7f05ec014000).stop 2022-07-21T01:40:41.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 shutdown_connections 2022-07-21T01:40:41.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f05e8053a70 0x7f05e8055f30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06040b3190 0x7f0604152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06040b2730 0x7f06040ace00 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 --2- 172.21.15.3:0/2533995381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06040666b0 0x7f0604148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 >> 172.21.15.3:0/2533995381 conn(0x7f0604063870 msgr2=0x7f0604064930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:41.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 shutdown_connections 2022-07-21T01:40:41.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.049+0000 7f0609602700 1 -- 172.21.15.3:0/2533995381 wait complete. 2022-07-21T01:40:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 17 and data pool 18 2022-07-21T01:40:41.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.153+0000 7fc193fff700 1 Processor -- start 2022-07-21T01:40:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.153+0000 7fc193fff700 1 -- start start 2022-07-21T01:40:41.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc194140c10 0x7fc1940b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1941467f0 con 0x7fc1940b6510 2022-07-21T01:40:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc194062140 con 0x7fc194140c10 2022-07-21T01:40:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1940656b0 con 0x7fc1940b2aa0 2022-07-21T01:40:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc18a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc1937fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc194140c10 0x7fc1940b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc18a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940b0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:55238/0 (socket says 172.21.15.3:55238) 2022-07-21T01:40:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc18a7fc700 1 -- 172.21.15.3:0/3255170969 learned_addr learned my addr 172.21.15.3:0/3255170969 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc192ffd700 1 --1- 172.21.15.3:0/3255170969 >> v1:172.21.15.143:6789/0 conn(0x7fc1940b6510 0x7fc194140500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:58716/0 (socket says 172.21.15.3:58716) 2022-07-21T01:40:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc18a7fc700 1 -- 172.21.15.3:0/3255170969 >> v2:172.21.15.143:3300/0 conn(0x7fc194140c10 msgr2=0x7fc1940b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc18a7fc700 1 --2- 172.21.15.3:0/3255170969 >> v2:172.21.15.143:3300/0 conn(0x7fc194140c10 0x7fc1940b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc18a7fc700 1 -- 172.21.15.3:0/3255170969 >> v1:172.21.15.143:6789/0 conn(0x7fc1940b6510 legacy=0x7fc194140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc18a7fc700 1 -- 172.21.15.3:0/3255170969 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1941476a0 con 0x7fc1940b2aa0 2022-07-21T01:40:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc1937fe700 1 --2- 172.21.15.3:0/3255170969 >> v2:172.21.15.143:3300/0 conn(0x7fc194140c10 0x7fc1940b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc18a7fc700 1 --2- 172.21.15.3:0/3255170969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940b0e60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc17c002f70 tx=0x7fc17c006eb0).ready entity=mon.2 client_cookie=2644f9cbdbc401e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc1927fc700 1 -- 172.21.15.3:0/3255170969 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc17c00d570 con 0x7fc1940b2aa0 2022-07-21T01:40:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc1927fc700 1 -- 172.21.15.3:0/3255170969 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c000e90 con 0x7fc1940b2aa0 2022-07-21T01:40:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc1927fc700 1 -- 172.21.15.3:0/3255170969 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc17c00d900 con 0x7fc1940b2aa0 2022-07-21T01:40:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 -- 172.21.15.3:0/3255170969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 msgr2=0x7fc1940b0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 --2- 172.21.15.3:0/3255170969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940b0e60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc17c002f70 tx=0x7fc17c006eb0).stop 2022-07-21T01:40:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 -- 172.21.15.3:0/3255170969 shutdown_connections 2022-07-21T01:40:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 --2- 172.21.15.3:0/3255170969 >> v2:172.21.15.143:3300/0 conn(0x7fc194140c10 0x7fc1940b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 --2- 172.21.15.3:0/3255170969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940b0e60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 -- 172.21.15.3:0/3255170969 >> 172.21.15.3:0/3255170969 conn(0x7fc194063800 msgr2=0x7fc194063be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 -- 172.21.15.3:0/3255170969 shutdown_connections 2022-07-21T01:40:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.157+0000 7fc193fff700 1 -- 172.21.15.3:0/3255170969 wait complete. 2022-07-21T01:40:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc193fff700 1 Processor -- start 2022-07-21T01:40:41.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc193fff700 1 -- start start 2022-07-21T01:40:41.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc193fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc193fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1940b6510 0x7fc194153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc193fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc194140c10 0x7fc1940afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc193fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc194144020 con 0x7fc194140c10 2022-07-21T01:40:41.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc193fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1940a3e80 con 0x7fc1940b6510 2022-07-21T01:40:41.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc193fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941534f0 con 0x7fc1940b2aa0 2022-07-21T01:40:41.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc1937fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc194140c10 0x7fc1940afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc1937fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc194140c10 0x7fc1940afea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48330/0 (socket says 172.21.15.3:48330) 2022-07-21T01:40:41.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc1937fe700 1 -- 172.21.15.3:0/2786427861 learned_addr learned my addr 172.21.15.3:0/2786427861 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:41.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc18a7fc700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1940b6510 0x7fc194153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.161+0000 7fc192ffd700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc18a7fc700 1 -- 172.21.15.3:0/2786427861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 msgr2=0x7fc1940ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc18a7fc700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc18a7fc700 1 -- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc194140c10 msgr2=0x7fc1940afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc18a7fc700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc194140c10 0x7fc1940afea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc18a7fc700 1 -- 172.21.15.3:0/2786427861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18c009470 con 0x7fc1940b6510 2022-07-21T01:40:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc1937fe700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc194140c10 0x7fc1940afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:41.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc18a7fc700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1940b6510 0x7fc194153000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fc17c002f70 tx=0x7fc17c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:41.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc190ff9700 1 -- 172.21.15.3:0/2786427861 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc17c006b40 con 0x7fc1940b6510 2022-07-21T01:40:41.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc190ff9700 1 -- 172.21.15.3:0/2786427861 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c00d8f0 con 0x7fc1940b6510 2022-07-21T01:40:41.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c00b1e0 con 0x7fc1940b6510 2022-07-21T01:40:41.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc190ff9700 1 -- 172.21.15.3:0/2786427861 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc17c011d70 con 0x7fc1940b6510 2022-07-21T01:40:41.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1940a4380 con 0x7fc1940b6510 2022-07-21T01:40:41.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.165+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1940ae100 con 0x7fc1940b6510 2022-07-21T01:40:41.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.169+0000 7fc190ff9700 1 -- 172.21.15.3:0/2786427861 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc17c01a040 con 0x7fc1940b6510 2022-07-21T01:40:41.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.169+0000 7fc190ff9700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc178053aa0 0x7fc178055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.169+0000 7fc190ff9700 1 -- 172.21.15.3:0/2786427861 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc17c06ce30 con 0x7fc1940b6510 2022-07-21T01:40:41.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.169+0000 7fc192ffd700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc178053aa0 0x7fc178055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.169+0000 7fc192ffd700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc178053aa0 0x7fc178055f60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc18400d000 tx=0x7fc184007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:41.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.177+0000 7fc190ff9700 1 -- 172.21.15.3:0/2786427861 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc17c043b50 con 0x7fc1940b6510 2022-07-21T01:40:41.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.305+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1940a47c0 con 0x7fc1940b6510 2022-07-21T01:40:41.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.309+0000 7fc190ff9700 1 -- 172.21.15.3:0/2786427861 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16290 (secure 0 0 0) 0x7fc17c023090 con 0x7fc1940b6510 2022-07-21T01:40:41.309 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:41.309 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":100,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:41.034259+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:39.112065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:40.040729+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:41.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc178053aa0 msgr2=0x7fc178055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc178053aa0 0x7fc178055f60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc18400d000 tx=0x7fc184007b80).stop 2022-07-21T01:40:41.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1940b6510 msgr2=0x7fc194153000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1940b6510 0x7fc194153000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fc17c002f70 tx=0x7fc17c015000).stop 2022-07-21T01:40:41.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 shutdown_connections 2022-07-21T01:40:41.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc194140c10 0x7fc1940afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc178053aa0 0x7fc178055f60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1940b6510 0x7fc194153000 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 --2- 172.21.15.3:0/2786427861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1940b2aa0 0x7fc1940ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 >> 172.21.15.3:0/2786427861 conn(0x7fc194063800 msgr2=0x7fc194066b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:41.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 shutdown_connections 2022-07-21T01:40:41.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.313+0000 7fc193fff700 1 -- 172.21.15.3:0/2786427861 wait complete. 2022-07-21T01:40:41.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a6d517700 1 Processor -- start 2022-07-21T01:40:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a6d517700 1 -- start start 2022-07-21T01:40:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a6d517700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0a680b3190 0x7f0a680b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a6d517700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680b3570 con 0x7f0a680b1a80 2022-07-21T01:40:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a6d517700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680b0e00 con 0x7f0a680a9da0 2022-07-21T01:40:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a6d517700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a680b2980 con 0x7f0a680b3190 2022-07-21T01:40:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a677fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0a680b1a80 0x7f0a680656a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56614/0 (socket says 172.21.15.3:56614) 2022-07-21T01:40:41.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a677fe700 1 -- 172.21.15.3:0/73144280 learned_addr learned my addr 172.21.15.3:0/73144280 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:41.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a67fff700 1 --2- 172.21.15.3:0/73144280 >> v2:172.21.15.3:3300/0 conn(0x7f0a680b3190 0x7f0a680b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a66ffd700 1 -- 172.21.15.3:0/73144280 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922485933 0 0) 0x7f0a680b3570 con 0x7f0a680b1a80 2022-07-21T01:40:41.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a66ffd700 1 -- 172.21.15.3:0/73144280 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a48003260 con 0x7f0a680b1a80 2022-07-21T01:40:41.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a66ffd700 1 -- 172.21.15.3:0/73144280 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830748350 0 0) 0x7f0a680b0e00 con 0x7f0a680a9da0 2022-07-21T01:40:41.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a67fff700 1 -- 172.21.15.3:0/73144280 >> v1:172.21.15.143:6789/0 conn(0x7f0a680a9da0 legacy=0x7f0a68147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a67fff700 1 -- 172.21.15.3:0/73144280 >> v1:172.21.15.3:6789/0 conn(0x7f0a680b1a80 legacy=0x7f0a680656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.429+0000 7f0a67fff700 1 -- 172.21.15.3:0/73144280 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a680b10a0 con 0x7f0a680b3190 2022-07-21T01:40:41.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a67fff700 1 --2- 172.21.15.3:0/73144280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b3190 0x7f0a680b2490 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f0a58002f70 tx=0x7f0a58006df0).ready entity=mon.0 client_cookie=14f7ff03772ed8ac server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:41.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a66ffd700 1 -- 172.21.15.3:0/73144280 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a5800d610 con 0x7f0a680b3190 2022-07-21T01:40:41.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a66ffd700 1 -- 172.21.15.3:0/73144280 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a58000e90 con 0x7f0a680b3190 2022-07-21T01:40:41.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a66ffd700 1 -- 172.21.15.3:0/73144280 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a5800d960 con 0x7f0a680b3190 2022-07-21T01:40:41.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a6d517700 1 -- 172.21.15.3:0/73144280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b3190 msgr2=0x7f0a680b2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a6d517700 1 --2- 172.21.15.3:0/73144280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b3190 0x7f0a680b2490 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f0a58002f70 tx=0x7f0a58006df0).stop 2022-07-21T01:40:41.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a6d517700 1 -- 172.21.15.3:0/73144280 shutdown_connections 2022-07-21T01:40:41.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a6d517700 1 --2- 172.21.15.3:0/73144280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b3190 0x7f0a680b2490 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a6d517700 1 -- 172.21.15.3:0/73144280 >> 172.21.15.3:0/73144280 conn(0x7f0a68063870 msgr2=0x7f0a68063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:41.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a6d517700 1 -- 172.21.15.3:0/73144280 shutdown_connections 2022-07-21T01:40:41.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a6d517700 1 -- 172.21.15.3:0/73144280 wait complete. 2022-07-21T01:40:41.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.433+0000 7f0a6d517700 1 Processor -- start 2022-07-21T01:40:41.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6d517700 1 -- start start 2022-07-21T01:40:41.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6d517700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a680a9da0 0x7f0a68159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6d517700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b1a80 0x7f0a681443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6d517700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a680b3190 0x7f0a6814c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6d517700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a681421a0 con 0x7f0a680b1a80 2022-07-21T01:40:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6d517700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6814d250 con 0x7f0a680b3190 2022-07-21T01:40:41.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6d517700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6814d4f0 con 0x7f0a680a9da0 2022-07-21T01:40:41.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b1a80 0x7f0a681443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b1a80 0x7f0a681443e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48342/0 (socket says 172.21.15.3:48342) 2022-07-21T01:40:41.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a677fe700 1 -- 172.21.15.3:0/1074855095 learned_addr learned my addr 172.21.15.3:0/1074855095 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:41.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a67fff700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a680a9da0 0x7f0a68159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6cd16700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a680b3190 0x7f0a6814c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a677fe700 1 -- 172.21.15.3:0/1074855095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a680a9da0 msgr2=0x7f0a68159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a677fe700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a680a9da0 0x7f0a68159cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a677fe700 1 -- 172.21.15.3:0/1074855095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a680b3190 msgr2=0x7f0a6814c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a677fe700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a680b3190 0x7f0a6814c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a677fe700 1 -- 172.21.15.3:0/1074855095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5800b180 con 0x7f0a680b1a80 2022-07-21T01:40:41.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a677fe700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b1a80 0x7f0a681443e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f0a50001490 tx=0x7f0a50007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:41.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a657fa700 1 -- 172.21.15.3:0/1074855095 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a50005ba0 con 0x7f0a680b1a80 2022-07-21T01:40:41.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a657fa700 1 -- 172.21.15.3:0/1074855095 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a50005d00 con 0x7f0a680b1a80 2022-07-21T01:40:41.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6814d790 con 0x7f0a680b1a80 2022-07-21T01:40:41.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a657fa700 1 -- 172.21.15.3:0/1074855095 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a500072c0 con 0x7f0a680b1a80 2022-07-21T01:40:41.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.437+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a68144040 con 0x7f0a680b1a80 2022-07-21T01:40:41.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.441+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a680574e0 con 0x7f0a680b1a80 2022-07-21T01:40:41.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.441+0000 7f0a657fa700 1 -- 172.21.15.3:0/1074855095 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0a5001a070 con 0x7f0a680b1a80 2022-07-21T01:40:41.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.441+0000 7f0a657fa700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0a4c053a10 0x7f0a4c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.441+0000 7f0a657fa700 1 -- 172.21.15.3:0/1074855095 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0a5006b2a0 con 0x7f0a680b1a80 2022-07-21T01:40:41.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.441+0000 7f0a67fff700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0a4c053a10 0x7f0a4c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.441+0000 7f0a67fff700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0a4c053a10 0x7f0a4c055ed0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0a58018000 tx=0x7f0a58014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:41.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.445+0000 7f0a657fa700 1 -- 172.21.15.3:0/1074855095 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0a50042060 con 0x7f0a680b1a80 2022-07-21T01:40:41.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.573+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a68064130 con 0x7f0a680b1a80 2022-07-21T01:40:41.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.573+0000 7f0a657fa700 1 -- 172.21.15.3:0/1074855095 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16290 (secure 0 0 0) 0x7f0a50045680 con 0x7f0a680b1a80 2022-07-21T01:40:41.576 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:41.577 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":100,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:41.034259+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:39.112065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:40.040729+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:41.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0a4c053a10 msgr2=0x7f0a4c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0a4c053a10 0x7f0a4c055ed0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0a58018000 tx=0x7f0a58014000).stop 2022-07-21T01:40:41.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b1a80 msgr2=0x7f0a681443e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b1a80 0x7f0a681443e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f0a50001490 tx=0x7f0a50007e20).stop 2022-07-21T01:40:41.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 shutdown_connections 2022-07-21T01:40:41.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0a4c053a10 0x7f0a4c055ed0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a680b3190 0x7f0a6814c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a680b1a80 0x7f0a681443e0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 --2- 172.21.15.3:0/1074855095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a680a9da0 0x7f0a68159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 >> 172.21.15.3:0/1074855095 conn(0x7f0a68063870 msgr2=0x7f0a680648c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:41.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 shutdown_connections 2022-07-21T01:40:41.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.577+0000 7f0a6d517700 1 -- 172.21.15.3:0/1074855095 wait complete. 2022-07-21T01:40:41.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:40:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdda6ec700 1 Processor -- start 2022-07-21T01:40:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdda6ec700 1 -- start start 2022-07-21T01:40:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdda6ec700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efdd40b5490 0x7efdd413e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdda6ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd413e730 0x7efdd41418e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdda6ec700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd40af0b0 con 0x7efdd40b15f0 2022-07-21T01:40:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdda6ec700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd40ac490 con 0x7efdd40b5490 2022-07-21T01:40:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdda6ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd40af2e0 con 0x7efdd413e730 2022-07-21T01:40:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdd96ea700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7efdd40b5490 0x7efdd413e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdd8ee9700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efdd40b15f0 0x7efdd41411d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56628/0 (socket says 172.21.15.3:56628) 2022-07-21T01:40:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdd8ee9700 1 -- 172.21.15.3:0/499537231 learned_addr learned my addr 172.21.15.3:0/499537231 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.697+0000 7efdd9eeb700 1 --2- 172.21.15.3:0/499537231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd413e730 0x7efdd41418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd3fff700 1 -- 172.21.15.3:0/499537231 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945243232 0 0) 0x7efdd40af0b0 con 0x7efdd40b15f0 2022-07-21T01:40:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd3fff700 1 -- 172.21.15.3:0/499537231 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc0003260 con 0x7efdd40b15f0 2022-07-21T01:40:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd3fff700 1 -- 172.21.15.3:0/499537231 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4232493309 0 0) 0x7efdc0003260 con 0x7efdd40b15f0 2022-07-21T01:40:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd96ea700 1 -- 172.21.15.3:0/499537231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd413e730 msgr2=0x7efdd41418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd96ea700 1 --2- 172.21.15.3:0/499537231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd413e730 0x7efdd41418e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd96ea700 1 -- 172.21.15.3:0/499537231 >> v1:172.21.15.3:6789/0 conn(0x7efdd40b15f0 legacy=0x7efdd41411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd96ea700 1 -- 172.21.15.3:0/499537231 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd40af580 con 0x7efdd40b5490 2022-07-21T01:40:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd9eeb700 1 --2- 172.21.15.3:0/499537231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd413e730 0x7efdd41418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd96ea700 1 --2- 172.21.15.3:0/499537231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40b5490 0x7efdd413e240 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7efdc4001b80 tx=0x7efdc8006df0).ready entity=mon.0 client_cookie=ba52f3b9ceb56e14 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd3fff700 1 -- 172.21.15.3:0/499537231 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdc40019e0 con 0x7efdd40b5490 2022-07-21T01:40:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd3fff700 1 -- 172.21.15.3:0/499537231 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc8000e90 con 0x7efdd40b5490 2022-07-21T01:40:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdd3fff700 1 -- 172.21.15.3:0/499537231 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdc8012460 con 0x7efdd40b5490 2022-07-21T01:40:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 -- 172.21.15.3:0/499537231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40b5490 msgr2=0x7efdd413e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 --2- 172.21.15.3:0/499537231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40b5490 0x7efdd413e240 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7efdc4001b80 tx=0x7efdc8006df0).stop 2022-07-21T01:40:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 -- 172.21.15.3:0/499537231 shutdown_connections 2022-07-21T01:40:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 --2- 172.21.15.3:0/499537231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd413e730 0x7efdd41418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 --2- 172.21.15.3:0/499537231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40b5490 0x7efdd413e240 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 -- 172.21.15.3:0/499537231 >> 172.21.15.3:0/499537231 conn(0x7efdd40626f0 msgr2=0x7efdd4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 -- 172.21.15.3:0/499537231 shutdown_connections 2022-07-21T01:40:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 -- 172.21.15.3:0/499537231 wait complete. 2022-07-21T01:40:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 Processor -- start 2022-07-21T01:40:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.701+0000 7efdda6ec700 1 -- start start 2022-07-21T01:40:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdda6ec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdd40b15f0 0x7efdd4145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdda6ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd40b5490 0x7efdd4144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdda6ec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd413e730 0x7efdd4142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdda6ec700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd413b510 con 0x7efdd413e730 2022-07-21T01:40:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdda6ec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd413b7b0 con 0x7efdd40b15f0 2022-07-21T01:40:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdda6ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd413ba50 con 0x7efdd40b5490 2022-07-21T01:40:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd9eeb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd413e730 0x7efdd4142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd8ee9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd40b5490 0x7efdd4144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd9eeb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd413e730 0x7efdd4142800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48358/0 (socket says 172.21.15.3:48358) 2022-07-21T01:40:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd9eeb700 1 -- 172.21.15.3:0/3864816081 learned_addr learned my addr 172.21.15.3:0/3864816081 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd96ea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdd40b15f0 0x7efdd4145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd9eeb700 1 -- 172.21.15.3:0/3864816081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd40b5490 msgr2=0x7efdd4144050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd9eeb700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd40b5490 0x7efdd4144050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd9eeb700 1 -- 172.21.15.3:0/3864816081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdd40b15f0 msgr2=0x7efdd4145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd9eeb700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdd40b15f0 0x7efdd4145090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd9eeb700 1 -- 172.21.15.3:0/3864816081 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc00b180 con 0x7efdd413e730 2022-07-21T01:40:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd8ee9700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd40b5490 0x7efdd4144050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd96ea700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdd40b15f0 0x7efdd4145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd9eeb700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd413e730 0x7efdd4142800 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7efdcc00c000 tx=0x7efdcc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd27fc700 1 -- 172.21.15.3:0/3864816081 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdcc010780 con 0x7efdd413e730 2022-07-21T01:40:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd27fc700 1 -- 172.21.15.3:0/3864816081 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdcc0108e0 con 0x7efdd413e730 2022-07-21T01:40:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc800b180 con 0x7efdd413e730 2022-07-21T01:40:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdd27fc700 1 -- 172.21.15.3:0/3864816081 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdcc010c40 con 0x7efdd413e730 2022-07-21T01:40:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.705+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd4142cf0 con 0x7efdd413e730 2022-07-21T01:40:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.709+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd40673f0 con 0x7efdd413e730 2022-07-21T01:40:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.709+0000 7efdd27fc700 1 -- 172.21.15.3:0/3864816081 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efdcc01c460 con 0x7efdd413e730 2022-07-21T01:40:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.709+0000 7efdd27fc700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efdb4053b00 0x7efdb4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.709+0000 7efdd27fc700 1 -- 172.21.15.3:0/3864816081 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6420+0+0 (secure 0 0 0) 0x7efdcc06c6a0 con 0x7efdd413e730 2022-07-21T01:40:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.709+0000 7efdd96ea700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efdb4053b00 0x7efdb4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.709+0000 7efdd96ea700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efdb4053b00 0x7efdb4055fc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7efdc801a000 tx=0x7efdc8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.713+0000 7efdd27fc700 1 -- 172.21.15.3:0/3864816081 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efdcc0432f0 con 0x7efdd413e730 2022-07-21T01:40:41.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:41.881+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7efdd413bef0 con 0x7efdd413e730 2022-07-21T01:40:42.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.725+0000 7efdd27fc700 1 -- 172.21.15.3:0/3864816081 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7efdcc046910 con 0x7efdd413e730 2022-07-21T01:40:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efdb4053b00 msgr2=0x7efdb4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efdb4053b00 0x7efdb4055fc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7efdc801a000 tx=0x7efdc8014000).stop 2022-07-21T01:40:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd413e730 msgr2=0x7efdd4142800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd413e730 0x7efdd4142800 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7efdcc00c000 tx=0x7efdcc006df0).stop 2022-07-21T01:40:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 shutdown_connections 2022-07-21T01:40:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd413e730 0x7efdd4142800 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdd40b5490 0x7efdd4144050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efdb4053b00 0x7efdb4055fc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 --2- 172.21.15.3:0/3864816081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdd40b15f0 0x7efdd4145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 >> 172.21.15.3:0/3864816081 conn(0x7efdd40626f0 msgr2=0x7efdd413c980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 shutdown_connections 2022-07-21T01:40:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.729+0000 7efdda6ec700 1 -- 172.21.15.3:0/3864816081 wait complete. 2022-07-21T01:40:42.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:42.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.857+0000 7fe6d4bcf700 1 Processor -- start 2022-07-21T01:40:42.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.857+0000 7fe6d4bcf700 1 -- start start 2022-07-21T01:40:42.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.857+0000 7fe6d4bcf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00b3630 0x7fe6d0143f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:42.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.857+0000 7fe6d4bcf700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d00b3a10 con 0x7fe6d00aa0d0 2022-07-21T01:40:42.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.857+0000 7fe6d4bcf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d0146b40 con 0x7fe6d00af910 2022-07-21T01:40:42.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.857+0000 7fe6d4bcf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01443f0 con 0x7fe6d00b3630 2022-07-21T01:40:42.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.857+0000 7fe6cdd9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe6d00aa0d0 0x7fe6d00af200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56644/0 (socket says 172.21.15.3:56644) 2022-07-21T01:40:42.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.857+0000 7fe6cdd9b700 1 -- 172.21.15.3:0/720898998 learned_addr learned my addr 172.21.15.3:0/720898998 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:42.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6ce59c700 1 --2- 172.21.15.3:0/720898998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00b3630 0x7fe6d0143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:42.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6cd59a700 1 -- 172.21.15.3:0/720898998 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88616524 0 0) 0x7fe6d00b3a10 con 0x7fe6d00aa0d0 2022-07-21T01:40:42.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6cd59a700 1 -- 172.21.15.3:0/720898998 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b4003260 con 0x7fe6d00aa0d0 2022-07-21T01:40:42.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6cd59a700 1 -- 172.21.15.3:0/720898998 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484308992 0 0) 0x7fe6d0146b40 con 0x7fe6d00af910 2022-07-21T01:40:42.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6ce59c700 1 -- 172.21.15.3:0/720898998 >> v1:172.21.15.143:6789/0 conn(0x7fe6d00af910 legacy=0x7fe6d01402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:42.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6ce59c700 1 -- 172.21.15.3:0/720898998 >> v1:172.21.15.3:6789/0 conn(0x7fe6d00aa0d0 legacy=0x7fe6d00af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:42.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6ce59c700 1 -- 172.21.15.3:0/720898998 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d0146de0 con 0x7fe6d00b3630 2022-07-21T01:40:42.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6ce59c700 1 --2- 172.21.15.3:0/720898998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00b3630 0x7fe6d0143f00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe6c0002f70 tx=0x7fe6c0006df0).ready entity=mon.2 client_cookie=8c3ddef0b2f9de6a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:42.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6cd59a700 1 -- 172.21.15.3:0/720898998 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6c000d570 con 0x7fe6d00b3630 2022-07-21T01:40:42.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6cd59a700 1 -- 172.21.15.3:0/720898998 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c0000e90 con 0x7fe6d00b3630 2022-07-21T01:40:42.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6cd59a700 1 -- 172.21.15.3:0/720898998 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6c000d8c0 con 0x7fe6d00b3630 2022-07-21T01:40:42.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/720898998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00b3630 msgr2=0x7fe6d0143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:42.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6d4bcf700 1 --2- 172.21.15.3:0/720898998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00b3630 0x7fe6d0143f00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe6c0002f70 tx=0x7fe6c0006df0).stop 2022-07-21T01:40:42.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/720898998 shutdown_connections 2022-07-21T01:40:42.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6d4bcf700 1 --2- 172.21.15.3:0/720898998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00b3630 0x7fe6d0143f00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:42.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/720898998 >> 172.21.15.3:0/720898998 conn(0x7fe6d0060950 msgr2=0x7fe6d0060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:42.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/720898998 shutdown_connections 2022-07-21T01:40:42.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.861+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/720898998 wait complete. 2022-07-21T01:40:42.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 Processor -- start 2022-07-21T01:40:42.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 -- start start 2022-07-21T01:40:42.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00aa0d0 0x7fe6d01491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:42.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6d00af910 0x7fe6d013ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:42.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6d00b3630 0x7fe6d013b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:42.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d0148760 con 0x7fe6d00af910 2022-07-21T01:40:42.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d0153120 con 0x7fe6d00b3630 2022-07-21T01:40:42.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01533c0 con 0x7fe6d00aa0d0 2022-07-21T01:40:42.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6cdd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6d00af910 0x7fe6d013ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:42.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6cdd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6d00af910 0x7fe6d013ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48372/0 (socket says 172.21.15.3:48372) 2022-07-21T01:40:42.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6cdd9b700 1 -- 172.21.15.3:0/341277206 learned_addr learned my addr 172.21.15.3:0/341277206 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:42.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6ce59c700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00aa0d0 0x7fe6d01491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:42.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6ced9d700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6d00b3630 0x7fe6d013b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:42.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6cdd9b700 1 -- 172.21.15.3:0/341277206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00aa0d0 msgr2=0x7fe6d01491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:42.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6cdd9b700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00aa0d0 0x7fe6d01491f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:42.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6cdd9b700 1 -- 172.21.15.3:0/341277206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6d00b3630 msgr2=0x7fe6d013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:42.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6cdd9b700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6d00b3630 0x7fe6d013b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:42.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6cdd9b700 1 -- 172.21.15.3:0/341277206 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c000b180 con 0x7fe6d00af910 2022-07-21T01:40:42.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6cdd9b700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6d00af910 0x7fe6d013ab10 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7fe6c4001110 tx=0x7fe6c4007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:42.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6bf7fe700 1 -- 172.21.15.3:0/341277206 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6c4007110 con 0x7fe6d00af910 2022-07-21T01:40:42.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6bf7fe700 1 -- 172.21.15.3:0/341277206 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c4007270 con 0x7fe6d00af910 2022-07-21T01:40:42.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d01414a0 con 0x7fe6d00af910 2022-07-21T01:40:42.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6bf7fe700 1 -- 172.21.15.3:0/341277206 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6c4005030 con 0x7fe6d00af910 2022-07-21T01:40:42.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.865+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d013b4f0 con 0x7fe6d00af910 2022-07-21T01:40:42.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.869+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d0153580 con 0x7fe6d00af910 2022-07-21T01:40:42.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.869+0000 7fe6bf7fe700 1 -- 172.21.15.3:0/341277206 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe6c401b070 con 0x7fe6d00af910 2022-07-21T01:40:42.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.869+0000 7fe6bf7fe700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6ac053b00 0x7fe6ac055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:42.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.869+0000 7fe6bf7fe700 1 -- 172.21.15.3:0/341277206 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe6c406b600 con 0x7fe6d00af910 2022-07-21T01:40:42.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.869+0000 7fe6ce59c700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6ac053b00 0x7fe6ac055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:42.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.869+0000 7fe6ce59c700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6ac053b00 0x7fe6ac055fc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fe6c0018000 tx=0x7fe6c0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:42.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:42.877+0000 7fe6bf7fe700 1 -- 172.21.15.3:0/341277206 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe6c4042230 con 0x7fe6d00af910 2022-07-21T01:40:43.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.037+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6d0141630 con 0x7fe6d00af910 2022-07-21T01:40:43.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.037+0000 7fe6bf7fe700 1 -- 172.21.15.3:0/341277206 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2484 (secure 0 0 0) 0x7fe6c4045850 con 0x7fe6d00af910 2022-07-21T01:40:43.039 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:43.040 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/485774043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":485774043},{"type":"v1","addr":"172.21.15.143:6833","nonce":485774043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":6936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/756498280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":756498280},{"type":"v1","addr":"172.21.15.3:6835","nonce":756498280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:41.034617+0000","modified":"2022-07-21T01:40:42.719494+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6942},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6942":{"gid":6942,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1935398064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1935398064},{"type":"v1","addr":"172.21.15.3:6837","nonce":1935398064}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-21T01:40:43.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6ac053b00 msgr2=0x7fe6ac055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6ac053b00 0x7fe6ac055fc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fe6c0018000 tx=0x7fe6c0014000).stop 2022-07-21T01:40:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6d00af910 msgr2=0x7fe6d013ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6d00af910 0x7fe6d013ab10 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7fe6c4001110 tx=0x7fe6c4007a80).stop 2022-07-21T01:40:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 shutdown_connections 2022-07-21T01:40:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe6ac053b00 0x7fe6ac055fc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6d00b3630 0x7fe6d013b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6d00af910 0x7fe6d013ab10 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 --2- 172.21.15.3:0/341277206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6d00aa0d0 0x7fe6d01491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 >> 172.21.15.3:0/341277206 conn(0x7fe6d0060950 msgr2=0x7fe6d00afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 shutdown_connections 2022-07-21T01:40:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.041+0000 7fe6d4bcf700 1 -- 172.21.15.3:0/341277206 wait complete. 2022-07-21T01:40:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 48 2022-07-21T01:40:43.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:43.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.153+0000 7f6a9ac17700 1 Processor -- start 2022-07-21T01:40:43.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.153+0000 7f6a9ac17700 1 -- start start 2022-07-21T01:40:43.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 0x7f6a940b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a940b04e0 con 0x7f6a940a7250 2022-07-21T01:40:43.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a940b0780 con 0x7f6a940b1f40 2022-07-21T01:40:43.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940a9250 con 0x7f6a94148530 2022-07-21T01:40:43.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a991b4700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6a940a7250 0x7f6a941448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56660/0 (socket says 172.21.15.3:56660) 2022-07-21T01:40:43.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a991b4700 1 -- 172.21.15.3:0/972137087 learned_addr learned my addr 172.21.15.3:0/972137087 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:43.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a93fff700 1 --2- 172.21.15.3:0/972137087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 0x7f6a940b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a937fe700 1 -- 172.21.15.3:0/972137087 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672155866 0 0) 0x7f6a940b04e0 con 0x7f6a940a7250 2022-07-21T01:40:43.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a937fe700 1 -- 172.21.15.3:0/972137087 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a80003260 con 0x7f6a940a7250 2022-07-21T01:40:43.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a937fe700 1 -- 172.21.15.3:0/972137087 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614935868 0 0) 0x7f6a940b0780 con 0x7f6a940b1f40 2022-07-21T01:40:43.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a937fe700 1 -- 172.21.15.3:0/972137087 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a940b04e0 con 0x7f6a940b1f40 2022-07-21T01:40:43.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a93fff700 1 -- 172.21.15.3:0/972137087 >> v1:172.21.15.143:6789/0 conn(0x7f6a940b1f40 legacy=0x7f6a940b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a93fff700 1 -- 172.21.15.3:0/972137087 >> v1:172.21.15.3:6789/0 conn(0x7f6a940a7250 legacy=0x7f6a941448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a93fff700 1 -- 172.21.15.3:0/972137087 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a940b0a20 con 0x7f6a94148530 2022-07-21T01:40:43.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a93fff700 1 --2- 172.21.15.3:0/972137087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 0x7f6a940b1160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f6a840032f0 tx=0x7f6a840091d0).ready entity=mon.2 client_cookie=d566b3c6bdb237e9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:43.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a937fe700 1 -- 172.21.15.3:0/972137087 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8400b640 con 0x7f6a94148530 2022-07-21T01:40:43.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a937fe700 1 -- 172.21.15.3:0/972137087 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8400b7a0 con 0x7f6a94148530 2022-07-21T01:40:43.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a937fe700 1 -- 172.21.15.3:0/972137087 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8400b910 con 0x7f6a94148530 2022-07-21T01:40:43.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 -- 172.21.15.3:0/972137087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 msgr2=0x7f6a940b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 --2- 172.21.15.3:0/972137087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 0x7f6a940b1160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f6a840032f0 tx=0x7f6a840091d0).stop 2022-07-21T01:40:43.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 -- 172.21.15.3:0/972137087 shutdown_connections 2022-07-21T01:40:43.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 --2- 172.21.15.3:0/972137087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 0x7f6a940b1160 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 -- 172.21.15.3:0/972137087 >> 172.21.15.3:0/972137087 conn(0x7f6a9405f840 msgr2=0x7f6a9405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:43.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 -- 172.21.15.3:0/972137087 shutdown_connections 2022-07-21T01:40:43.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.157+0000 7f6a9ac17700 1 -- 172.21.15.3:0/972137087 wait complete. 2022-07-21T01:40:43.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 Processor -- start 2022-07-21T01:40:43.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 -- start start 2022-07-21T01:40:43.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940a7250 0x7f6a9414b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a940b1f40 0x7f6a940a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 0x7f6a940ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940ab580 con 0x7f6a940a7250 2022-07-21T01:40:43.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9414a030 con 0x7f6a940b1f40 2022-07-21T01:40:43.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9414a2d0 con 0x7f6a94148530 2022-07-21T01:40:43.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a989b3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940a7250 0x7f6a9414b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a989b3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940a7250 0x7f6a9414b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48382/0 (socket says 172.21.15.3:48382) 2022-07-21T01:40:43.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a989b3700 1 -- 172.21.15.3:0/1183561550 learned_addr learned my addr 172.21.15.3:0/1183561550 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:43.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a93fff700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a940b1f40 0x7f6a940a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a991b4700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 0x7f6a940ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a989b3700 1 -- 172.21.15.3:0/1183561550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 msgr2=0x7f6a940ab9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a989b3700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 0x7f6a940ab9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a989b3700 1 -- 172.21.15.3:0/1183561550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a940b1f40 msgr2=0x7f6a940a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a989b3700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a940b1f40 0x7f6a940a95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a989b3700 1 -- 172.21.15.3:0/1183561550 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a84009470 con 0x7f6a940a7250 2022-07-21T01:40:43.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a989b3700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940a7250 0x7f6a9414b690 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f6a88001110 tx=0x7f6a88007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:43.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a91ffb700 1 -- 172.21.15.3:0/1183561550 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a88012400 con 0x7f6a940a7250 2022-07-21T01:40:43.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a91ffb700 1 -- 172.21.15.3:0/1183561550 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a88012560 con 0x7f6a940a7250 2022-07-21T01:40:43.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9414a570 con 0x7f6a940a7250 2022-07-21T01:40:43.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a91ffb700 1 -- 172.21.15.3:0/1183561550 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a880128c0 con 0x7f6a940a7250 2022-07-21T01:40:43.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.161+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a940abea0 con 0x7f6a940a7250 2022-07-21T01:40:43.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.165+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a940574e0 con 0x7f6a940a7250 2022-07-21T01:40:43.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.165+0000 7f6a91ffb700 1 -- 172.21.15.3:0/1183561550 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6a88019030 con 0x7f6a940a7250 2022-07-21T01:40:43.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.165+0000 7f6a91ffb700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053a70 0x7f6a74055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.165+0000 7f6a91ffb700 1 -- 172.21.15.3:0/1183561550 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6a8806b3f0 con 0x7f6a940a7250 2022-07-21T01:40:43.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.165+0000 7f6a93fff700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053a70 0x7f6a74055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.165+0000 7f6a93fff700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053a70 0x7f6a74055f30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f6a8401a000 tx=0x7f6a84016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:43.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.169+0000 7f6a91ffb700 1 -- 172.21.15.3:0/1183561550 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6a880421b0 con 0x7f6a940a7250 2022-07-21T01:40:43.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.297+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a940b27d0 con 0x7f6a940a7250 2022-07-21T01:40:43.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.297+0000 7f6a91ffb700 1 -- 172.21.15.3:0/1183561550 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16277 (secure 0 0 0) 0x7f6a880457d0 con 0x7f6a940a7250 2022-07-21T01:40:43.299 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:43.300 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":101,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:42.037286+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:39.112065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:40.040729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:43.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053a70 msgr2=0x7f6a74055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053a70 0x7f6a74055f30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f6a8401a000 tx=0x7f6a84016000).stop 2022-07-21T01:40:43.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940a7250 msgr2=0x7f6a9414b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940a7250 0x7f6a9414b690 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f6a88001110 tx=0x7f6a88007a90).stop 2022-07-21T01:40:43.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 shutdown_connections 2022-07-21T01:40:43.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a94148530 0x7f6a940ab9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053a70 0x7f6a74055f30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a940b1f40 0x7f6a940a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 --2- 172.21.15.3:0/1183561550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940a7250 0x7f6a9414b690 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 >> 172.21.15.3:0/1183561550 conn(0x7f6a9405f840 msgr2=0x7f6a940a76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:43.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 shutdown_connections 2022-07-21T01:40:43.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.301+0000 7f6a9ac17700 1 -- 172.21.15.3:0/1183561550 wait complete. 2022-07-21T01:40:43.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:40:43.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8b05a9c700 1 Processor -- start 2022-07-21T01:40:43.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8b05a9c700 1 -- start start 2022-07-21T01:40:43.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8b05a9c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8b000ad750 0x7f8b00061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8b05a9c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b001405d0 0x7f8b00140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8b05a9c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b000aee80 con 0x7f8b000614b0 2022-07-21T01:40:43.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8b05a9c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b000b70b0 con 0x7f8b000ad750 2022-07-21T01:40:43.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8b05a9c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b000af0b0 con 0x7f8b001405d0 2022-07-21T01:40:43.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8afffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8b000ad750 0x7f8b00061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8b04a9a700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8b000614b0 0x7f8b000b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56670/0 (socket says 172.21.15.3:56670) 2022-07-21T01:40:43.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.425+0000 7f8afffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8b000ad750 0x7f8b00061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48392/0 (socket says 172.21.15.3:48392) 2022-07-21T01:40:43.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b04a9a700 1 -- 172.21.15.3:0/56273201 learned_addr learned my addr 172.21.15.3:0/56273201 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:43.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b0529b700 1 --2- 172.21.15.3:0/56273201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b001405d0 0x7f8b00140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8afffff700 1 -- 172.21.15.3:0/56273201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b001405d0 msgr2=0x7f8b00140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:43.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8afffff700 1 --2- 172.21.15.3:0/56273201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b001405d0 0x7f8b00140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8afffff700 1 -- 172.21.15.3:0/56273201 >> v1:172.21.15.3:6789/0 conn(0x7f8b000614b0 legacy=0x7f8b000b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8afffff700 1 -- 172.21.15.3:0/56273201 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b000af350 con 0x7f8b000ad750 2022-07-21T01:40:43.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8afffff700 1 --2- 172.21.15.3:0/56273201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b000ad750 0x7f8b00061890 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f8aec002f70 tx=0x7f8aec006df0).ready entity=mon.0 client_cookie=d0fcecf1565f792b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:43.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8aff7fe700 1 -- 172.21.15.3:0/56273201 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aec00d570 con 0x7f8b000ad750 2022-07-21T01:40:43.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8aff7fe700 1 -- 172.21.15.3:0/56273201 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec000e90 con 0x7f8b000ad750 2022-07-21T01:40:43.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8aff7fe700 1 -- 172.21.15.3:0/56273201 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aec00d8c0 con 0x7f8b000ad750 2022-07-21T01:40:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 -- 172.21.15.3:0/56273201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b000ad750 msgr2=0x7f8b00061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 --2- 172.21.15.3:0/56273201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b000ad750 0x7f8b00061890 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f8aec002f70 tx=0x7f8aec006df0).stop 2022-07-21T01:40:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 -- 172.21.15.3:0/56273201 shutdown_connections 2022-07-21T01:40:43.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 --2- 172.21.15.3:0/56273201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b001405d0 0x7f8b00140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 --2- 172.21.15.3:0/56273201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b000ad750 0x7f8b00061890 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 -- 172.21.15.3:0/56273201 >> 172.21.15.3:0/56273201 conn(0x7f8b00066f70 msgr2=0x7f8b00067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:43.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 -- 172.21.15.3:0/56273201 shutdown_connections 2022-07-21T01:40:43.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 -- 172.21.15.3:0/56273201 wait complete. 2022-07-21T01:40:43.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 Processor -- start 2022-07-21T01:40:43.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.429+0000 7f8b05a9c700 1 -- start start 2022-07-21T01:40:43.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b05a9c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b000614b0 0x7f8b000b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b05a9c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b000ad750 0x7f8b0013e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b05a9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b001405d0 0x7f8b0013ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b05a9c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b000aca50 con 0x7f8b001405d0 2022-07-21T01:40:43.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b05a9c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b000b0300 con 0x7f8b000614b0 2022-07-21T01:40:43.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b05a9c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b000acc80 con 0x7f8b000ad750 2022-07-21T01:40:43.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b0529b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b001405d0 0x7f8b0013ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b0529b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b001405d0 0x7f8b0013ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48400/0 (socket says 172.21.15.3:48400) 2022-07-21T01:40:43.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b0529b700 1 -- 172.21.15.3:0/1420217322 learned_addr learned my addr 172.21.15.3:0/1420217322 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:43.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b04a9a700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b000614b0 0x7f8b000b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8afffff700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b000ad750 0x7f8b0013e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b0529b700 1 -- 172.21.15.3:0/1420217322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b000ad750 msgr2=0x7f8b0013e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b0529b700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b000ad750 0x7f8b0013e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b0529b700 1 -- 172.21.15.3:0/1420217322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b000614b0 msgr2=0x7f8b000b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b0529b700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b000614b0 0x7f8b000b7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b0529b700 1 -- 172.21.15.3:0/1420217322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec00b180 con 0x7f8b001405d0 2022-07-21T01:40:43.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8afffff700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b000ad750 0x7f8b0013e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:43.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b0529b700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b001405d0 0x7f8b0013ee90 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f8af8006e00 tx=0x7f8af80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:43.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8afdffb700 1 -- 172.21.15.3:0/1420217322 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8af800c670 con 0x7f8b001405d0 2022-07-21T01:40:43.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b000ad2c0 con 0x7f8b001405d0 2022-07-21T01:40:43.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8afdffb700 1 -- 172.21.15.3:0/1420217322 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af800c7d0 con 0x7f8b001405d0 2022-07-21T01:40:43.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b000b35f0 con 0x7f8b001405d0 2022-07-21T01:40:43.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.433+0000 7f8afdffb700 1 -- 172.21.15.3:0/1420217322 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8af800cb30 con 0x7f8b001405d0 2022-07-21T01:40:43.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.437+0000 7f8afdffb700 1 -- 172.21.15.3:0/1420217322 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8af800cc90 con 0x7f8b001405d0 2022-07-21T01:40:43.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.437+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b000ace40 con 0x7f8b001405d0 2022-07-21T01:40:43.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.437+0000 7f8afdffb700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ae0053950 0x7f8ae0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.437+0000 7f8afdffb700 1 -- 172.21.15.3:0/1420217322 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8af806c370 con 0x7f8b001405d0 2022-07-21T01:40:43.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.437+0000 7f8b04a9a700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ae0053950 0x7f8ae0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.437+0000 7f8b04a9a700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ae0053950 0x7f8ae0055e10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8af400d000 tx=0x7f8af4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:43.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.445+0000 7f8afdffb700 1 -- 172.21.15.3:0/1420217322 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8af8043090 con 0x7f8b001405d0 2022-07-21T01:40:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.605+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8b000af5f0 con 0x7f8b001405d0 2022-07-21T01:40:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.613+0000 7f8afdffb700 1 -- 172.21.15.3:0/1420217322 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v27) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8af80160e0 con 0x7f8b001405d0 2022-07-21T01:40:43.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ae0053950 msgr2=0x7f8ae0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ae0053950 0x7f8ae0055e10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8af400d000 tx=0x7f8af4009040).stop 2022-07-21T01:40:43.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b001405d0 msgr2=0x7f8b0013ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b001405d0 0x7f8b0013ee90 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f8af8006e00 tx=0x7f8af80091d0).stop 2022-07-21T01:40:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 shutdown_connections 2022-07-21T01:40:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b001405d0 0x7f8b0013ee90 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b000ad750 0x7f8b0013e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8ae0053950 0x7f8ae0055e10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 --2- 172.21.15.3:0/1420217322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b000614b0 0x7f8b000b7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 >> 172.21.15.3:0/1420217322 conn(0x7f8b00066f70 msgr2=0x7f8b0013fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:43.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 shutdown_connections 2022-07-21T01:40:43.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.617+0000 7f8b05a9c700 1 -- 172.21.15.3:0/1420217322 wait complete. 2022-07-21T01:40:43.621 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:40:43.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:40:43.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.737+0000 7f291a44d700 1 Processor -- start 2022-07-21T01:40:43.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.737+0000 7f291a44d700 1 -- start start 2022-07-21T01:40:43.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.737+0000 7f291a44d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f29140b6a90 0x7f29140acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.737+0000 7f291a44d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29140b6e70 con 0x7f29140aa760 2022-07-21T01:40:43.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.737+0000 7f291a44d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2914061370 con 0x7f2914060aa0 2022-07-21T01:40:43.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.737+0000 7f291a44d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2914061610 con 0x7f29140b6a90 2022-07-21T01:40:43.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.737+0000 7f29189ea700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f29140aa760 0x7f291413c600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56688/0 (socket says 172.21.15.3:56688) 2022-07-21T01:40:43.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.737+0000 7f29189ea700 1 -- 172.21.15.3:0/4211802438 learned_addr learned my addr 172.21.15.3:0/4211802438 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:43.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.737+0000 7f2913fff700 1 --2- 172.21.15.3:0/4211802438 >> v2:172.21.15.143:3300/0 conn(0x7f29140b6a90 0x7f29140acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f29137fe700 1 -- 172.21.15.3:0/4211802438 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145673776 0 0) 0x7f29140b6e70 con 0x7f29140aa760 2022-07-21T01:40:43.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f29137fe700 1 -- 172.21.15.3:0/4211802438 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f4003260 con 0x7f29140aa760 2022-07-21T01:40:43.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f2913fff700 1 -- 172.21.15.3:0/4211802438 >> v1:172.21.15.143:6789/0 conn(0x7f2914060aa0 legacy=0x7f29140aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f2913fff700 1 -- 172.21.15.3:0/4211802438 >> v1:172.21.15.3:6789/0 conn(0x7f29140aa760 legacy=0x7f291413c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f2913fff700 1 -- 172.21.15.3:0/4211802438 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29140618b0 con 0x7f29140b6a90 2022-07-21T01:40:43.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f2913fff700 1 --2- 172.21.15.3:0/4211802438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29140b6a90 0x7f29140acee0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f2904002f70 tx=0x7f2904006e80).ready entity=mon.1 client_cookie=9c3d90adaad7481b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:43.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f29137fe700 1 -- 172.21.15.3:0/4211802438 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290400f730 con 0x7f29140b6a90 2022-07-21T01:40:43.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f29137fe700 1 -- 172.21.15.3:0/4211802438 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2904000e90 con 0x7f29140b6a90 2022-07-21T01:40:43.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f29137fe700 1 -- 172.21.15.3:0/4211802438 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290400fa80 con 0x7f29140b6a90 2022-07-21T01:40:43.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f291a44d700 1 -- 172.21.15.3:0/4211802438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29140b6a90 msgr2=0x7f29140acee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f291a44d700 1 --2- 172.21.15.3:0/4211802438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29140b6a90 0x7f29140acee0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f2904002f70 tx=0x7f2904006e80).stop 2022-07-21T01:40:43.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f291a44d700 1 -- 172.21.15.3:0/4211802438 shutdown_connections 2022-07-21T01:40:43.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f291a44d700 1 --2- 172.21.15.3:0/4211802438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29140b6a90 0x7f29140acee0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f291a44d700 1 -- 172.21.15.3:0/4211802438 >> 172.21.15.3:0/4211802438 conn(0x7f29140b3610 msgr2=0x7f29140b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:43.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f291a44d700 1 -- 172.21.15.3:0/4211802438 shutdown_connections 2022-07-21T01:40:43.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.741+0000 7f291a44d700 1 -- 172.21.15.3:0/4211802438 wait complete. 2022-07-21T01:40:43.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 Processor -- start 2022-07-21T01:40:43.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 -- start start 2022-07-21T01:40:43.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2914060aa0 0x7f29141367d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29140aa760 0x7f29140ad420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29140b6a90 0x7f29140ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140ade00 con 0x7f29140b6a90 2022-07-21T01:40:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140afa50 con 0x7f2914060aa0 2022-07-21T01:40:43.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291414f040 con 0x7f29140aa760 2022-07-21T01:40:43.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f29189ea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29140b6a90 0x7f29140ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f29189ea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29140b6a90 0x7f29140ad910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48414/0 (socket says 172.21.15.3:48414) 2022-07-21T01:40:43.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f29189ea700 1 -- 172.21.15.3:0/3978721619 learned_addr learned my addr 172.21.15.3:0/3978721619 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:43.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f2913fff700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2914060aa0 0x7f29141367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f290bfff700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29140aa760 0x7f29140ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f29189ea700 1 -- 172.21.15.3:0/3978721619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29140aa760 msgr2=0x7f29140ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f29189ea700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29140aa760 0x7f29140ad420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f29189ea700 1 -- 172.21.15.3:0/3978721619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2914060aa0 msgr2=0x7f29141367d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f29189ea700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2914060aa0 0x7f29141367d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f29189ea700 1 -- 172.21.15.3:0/3978721619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290400c310 con 0x7f29140b6a90 2022-07-21T01:40:43.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f29189ea700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29140b6a90 0x7f29140ad910 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f290c001490 tx=0x7f290c007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:43.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f2911ffb700 1 -- 172.21.15.3:0/3978721619 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290c005ba0 con 0x7f29140b6a90 2022-07-21T01:40:43.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f2911ffb700 1 -- 172.21.15.3:0/3978721619 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c005d00 con 0x7f29140b6a90 2022-07-21T01:40:43.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291414f2a0 con 0x7f29140b6a90 2022-07-21T01:40:43.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f2911ffb700 1 -- 172.21.15.3:0/3978721619 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290c0072c0 con 0x7f29140b6a90 2022-07-21T01:40:43.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.745+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2914136cc0 con 0x7f29140b6a90 2022-07-21T01:40:43.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.749+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29140afe80 con 0x7f29140b6a90 2022-07-21T01:40:43.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.749+0000 7f2911ffb700 1 -- 172.21.15.3:0/3978721619 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f290c010070 con 0x7f29140b6a90 2022-07-21T01:40:43.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.749+0000 7f2911ffb700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28f8053a70 0x7f28f8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:43.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.749+0000 7f2911ffb700 1 -- 172.21.15.3:0/3978721619 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f290c06b970 con 0x7f29140b6a90 2022-07-21T01:40:43.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.749+0000 7f2913fff700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28f8053a70 0x7f28f8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:43.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.749+0000 7f2913fff700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28f8053a70 0x7f28f8055f30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f290401c000 tx=0x7f2904017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:43.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.753+0000 7f2911ffb700 1 -- 172.21.15.3:0/3978721619 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29140afe80 con 0x7f29140b6a90 2022-07-21T01:40:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.917+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f29140afe80 con 0x7f29140b6a90 2022-07-21T01:40:43.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.925+0000 7f2911ffb700 1 -- 172.21.15.3:0/3978721619 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v28) v1 ==== 184+0+0 (secure 0 0 0) 0x7f29140afe80 con 0x7f29140b6a90 2022-07-21T01:40:43.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28f8053a70 msgr2=0x7f28f8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28f8053a70 0x7f28f8055f30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f290401c000 tx=0x7f2904017000).stop 2022-07-21T01:40:43.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29140b6a90 msgr2=0x7f29140ad910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:43.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29140b6a90 0x7f29140ad910 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f290c001490 tx=0x7f290c007dc0).stop 2022-07-21T01:40:43.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 shutdown_connections 2022-07-21T01:40:43.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29140b6a90 0x7f29140ad910 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29140aa760 0x7f29140ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28f8053a70 0x7f28f8055f30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 --2- 172.21.15.3:0/3978721619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2914060aa0 0x7f29141367d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:43.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 >> 172.21.15.3:0/3978721619 conn(0x7f29140b3610 msgr2=0x7f29140b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:43.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 shutdown_connections 2022-07-21T01:40:43.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:43.929+0000 7f291a44d700 1 -- 172.21.15.3:0/3978721619 wait complete. 2022-07-21T01:40:43.933 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:40:43.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:44.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 Processor -- start 2022-07-21T01:40:44.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 -- start start 2022-07-21T01:40:44.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcd800b0630 0x7fcd800adec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 0x7fcd80140730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd800b2c80 con 0x7fcd800aa2b0 2022-07-21T01:40:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd800b2900 con 0x7fcd800b0630 2022-07-21T01:40:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800b2eb0 con 0x7fcd800b0a10 2022-07-21T01:40:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd8527f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcd800b0630 0x7fcd800adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:44.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd8527f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcd800b0630 0x7fcd800adec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48420/0 (socket says 172.21.15.3:48420) 2022-07-21T01:40:44.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd8527f700 1 -- 172.21.15.3:0/1219868914 learned_addr learned my addr 172.21.15.3:0/1219868914 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:44.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd85a80700 1 --2- 172.21.15.3:0/1219868914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 0x7fcd80140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:44.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd8527f700 1 -- 172.21.15.3:0/1219868914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 msgr2=0x7fcd80140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:44.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd8527f700 1 --2- 172.21.15.3:0/1219868914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 0x7fcd80140730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd8527f700 1 -- 172.21.15.3:0/1219868914 >> v1:172.21.15.143:6789/0 conn(0x7fcd800aa2b0 legacy=0x7fcd800ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd8527f700 1 -- 172.21.15.3:0/1219868914 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd800a9d00 con 0x7fcd800b0630 2022-07-21T01:40:44.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd85a80700 1 --2- 172.21.15.3:0/1219868914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 0x7fcd80140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:44.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd8527f700 1 --2- 172.21.15.3:0/1219868914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800b0630 0x7fcd800adec0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7fcd7c002f70 tx=0x7fcd7c006eb0).ready entity=mon.0 client_cookie=b58251ad2779ea61 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:44.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd77fff700 1 -- 172.21.15.3:0/1219868914 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd7c00d5c0 con 0x7fcd800b0630 2022-07-21T01:40:44.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd77fff700 1 -- 172.21.15.3:0/1219868914 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c000e90 con 0x7fcd800b0630 2022-07-21T01:40:44.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd77fff700 1 -- 172.21.15.3:0/1219868914 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd7c00d950 con 0x7fcd800b0630 2022-07-21T01:40:44.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 -- 172.21.15.3:0/1219868914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800b0630 msgr2=0x7fcd800adec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 --2- 172.21.15.3:0/1219868914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800b0630 0x7fcd800adec0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7fcd7c002f70 tx=0x7fcd7c006eb0).stop 2022-07-21T01:40:44.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 -- 172.21.15.3:0/1219868914 shutdown_connections 2022-07-21T01:40:44.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 --2- 172.21.15.3:0/1219868914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 0x7fcd80140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 --2- 172.21.15.3:0/1219868914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800b0630 0x7fcd800adec0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 -- 172.21.15.3:0/1219868914 >> 172.21.15.3:0/1219868914 conn(0x7fcd80060950 msgr2=0x7fcd80060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:44.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 -- 172.21.15.3:0/1219868914 shutdown_connections 2022-07-21T01:40:44.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.045+0000 7fcd874e3700 1 -- 172.21.15.3:0/1219868914 wait complete. 2022-07-21T01:40:44.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.049+0000 7fcd874e3700 1 Processor -- start 2022-07-21T01:40:44.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.049+0000 7fcd874e3700 1 -- start start 2022-07-21T01:40:44.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.049+0000 7fcd874e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800aa2b0 0x7fcd800abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.049+0000 7fcd874e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd800b0630 0x7fcd80138d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd874e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 0x7fcd80139230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd874e3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd80140400 con 0x7fcd800aa2b0 2022-07-21T01:40:44.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd874e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8013c6b0 con 0x7fcd800b0630 2022-07-21T01:40:44.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd874e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd801455d0 con 0x7fcd800b0a10 2022-07-21T01:40:44.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd8527f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800aa2b0 0x7fcd800abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:44.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd8527f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800aa2b0 0x7fcd800abbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48426/0 (socket says 172.21.15.3:48426) 2022-07-21T01:40:44.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd8527f700 1 -- 172.21.15.3:0/3454966807 learned_addr learned my addr 172.21.15.3:0/3454966807 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:44.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd8527f700 1 -- 172.21.15.3:0/3454966807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 msgr2=0x7fcd80139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd8527f700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 0x7fcd80139230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:40:44.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd8527f700 1 -- 172.21.15.3:0/3454966807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd800b0630 msgr2=0x7fcd80138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd8527f700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd800b0630 0x7fcd80138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:40:44.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd8527f700 1 -- 172.21.15.3:0/3454966807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c00b1e0 con 0x7fcd800aa2b0 2022-07-21T01:40:44.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd8527f700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800aa2b0 0x7fcd800abbd0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7fcd7c002f70 tx=0x7fcd7c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:44.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd767fc700 1 -- 172.21.15.3:0/3454966807 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd7c0117e0 con 0x7fcd800aa2b0 2022-07-21T01:40:44.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd767fc700 1 -- 172.21.15.3:0/3454966807 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c011940 con 0x7fcd800aa2b0 2022-07-21T01:40:44.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd80139720 con 0x7fcd800aa2b0 2022-07-21T01:40:44.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd767fc700 1 -- 172.21.15.3:0/3454966807 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd7c011ca0 con 0x7fcd800aa2b0 2022-07-21T01:40:44.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd801405f0 con 0x7fcd800aa2b0 2022-07-21T01:40:44.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.053+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd80140ab0 con 0x7fcd800aa2b0 2022-07-21T01:40:44.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.057+0000 7fcd767fc700 1 -- 172.21.15.3:0/3454966807 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd7c025030 con 0x7fcd800aa2b0 2022-07-21T01:40:44.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.057+0000 7fcd767fc700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd68053b00 0x7fcd68055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.057+0000 7fcd767fc700 1 -- 172.21.15.3:0/3454966807 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcd7c06d2e0 con 0x7fcd800aa2b0 2022-07-21T01:40:44.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.057+0000 7fcd84a7e700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd68053b00 0x7fcd68055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:44.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.057+0000 7fcd84a7e700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd68053b00 0x7fcd68055fc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fcd800b7900 tx=0x7fcd70007810).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:44.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.065+0000 7fcd767fc700 1 -- 172.21.15.3:0/3454966807 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd80140ab0 con 0x7fcd800aa2b0 2022-07-21T01:40:44.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.225+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd80140ab0 con 0x7fcd800aa2b0 2022-07-21T01:40:44.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.225+0000 7fcd767fc700 1 -- 172.21.15.3:0/3454966807 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2484 (secure 0 0 0) 0x7fcd80140ab0 con 0x7fcd800aa2b0 2022-07-21T01:40:44.229 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:44.229 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/485774043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":485774043},{"type":"v1","addr":"172.21.15.143:6833","nonce":485774043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":6936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/756498280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":756498280},{"type":"v1","addr":"172.21.15.3:6835","nonce":756498280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:41.034617+0000","modified":"2022-07-21T01:40:42.719494+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6942},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6942":{"gid":6942,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1935398064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1935398064},{"type":"v1","addr":"172.21.15.3:6837","nonce":1935398064}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-21T01:40:44.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd68053b00 msgr2=0x7fcd68055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd68053b00 0x7fcd68055fc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fcd800b7900 tx=0x7fcd70007810).stop 2022-07-21T01:40:44.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800aa2b0 msgr2=0x7fcd800abbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800aa2b0 0x7fcd800abbd0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7fcd7c002f70 tx=0x7fcd7c015000).stop 2022-07-21T01:40:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 shutdown_connections 2022-07-21T01:40:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd800b0a10 0x7fcd80139230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:40:44.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcd68053b00 0x7fcd68055fc0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd800b0630 0x7fcd80138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:40:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 --2- 172.21.15.3:0/3454966807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd800aa2b0 0x7fcd800abbd0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 >> 172.21.15.3:0/3454966807 conn(0x7fcd80060950 msgr2=0x7fcd800b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:44.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 shutdown_connections 2022-07-21T01:40:44.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.229+0000 7fcd874e3700 1 -- 172.21.15.3:0/3454966807 wait complete. 2022-07-21T01:40:44.235 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 48 2022-07-21T01:40:44.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:40:41.034617+0000', 'modified': '2022-07-21T01:40:42.719494+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6942}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6942': {'gid': 6942, 'name': 'c', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/1935398064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 1935398064}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 1935398064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:40:44.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:40:44.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:40:44.664 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6942,"want_state":"up:active","state":"up:active","rank_uptime":3.6183743100000001,"mdsmap_epoch":48,"osdmap_epoch":101,"osdmap_epoch_barrier":101,"uptime":8.3038115749999992} 2022-07-21T01:40:44.665 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6942, "mdsmap_epoch": 48, "osdmap_epoch": 101, "osdmap_epoch_barrier": 101, "rank_uptime": 3.61837431, "state": "up:active", "uptime": 8.303811575, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:40:44.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:44.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.769+0000 7f3399172700 1 Processor -- start 2022-07-21T01:40:44.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.769+0000 7f3399172700 1 -- start start 2022-07-21T01:40:44.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f33940a8c50 0x7f3394152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33940a7250 con 0x7f33940b01a0 2022-07-21T01:40:44.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33940a7480 con 0x7f33940b2710 2022-07-21T01:40:44.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33940a7720 con 0x7f33940a8c50 2022-07-21T01:40:44.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f33937fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f33940b01a0 0x7f33940a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56714/0 (socket says 172.21.15.3:56714) 2022-07-21T01:40:44.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f33937fe700 1 -- 172.21.15.3:0/3943511712 learned_addr learned my addr 172.21.15.3:0/3943511712 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:44.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3398971700 1 --2- 172.21.15.3:0/3943511712 >> v2:172.21.15.143:3300/0 conn(0x7f33940a8c50 0x7f3394152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:44.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3392ffd700 1 -- 172.21.15.3:0/3943511712 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002645291 0 0) 0x7f33940a7250 con 0x7f33940b01a0 2022-07-21T01:40:44.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3392ffd700 1 -- 172.21.15.3:0/3943511712 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3374003260 con 0x7f33940b01a0 2022-07-21T01:40:44.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3392ffd700 1 -- 172.21.15.3:0/3943511712 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424251820 0 0) 0x7f33940a7480 con 0x7f33940b2710 2022-07-21T01:40:44.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3392ffd700 1 -- 172.21.15.3:0/3943511712 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33940a7250 con 0x7f33940b2710 2022-07-21T01:40:44.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3398971700 1 -- 172.21.15.3:0/3943511712 >> v1:172.21.15.143:6789/0 conn(0x7f33940b2710 legacy=0x7f339414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3398971700 1 -- 172.21.15.3:0/3943511712 >> v1:172.21.15.3:6789/0 conn(0x7f33940b01a0 legacy=0x7f33940a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3398971700 1 -- 172.21.15.3:0/3943511712 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33940a79c0 con 0x7f33940a8c50 2022-07-21T01:40:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3398971700 1 --2- 172.21.15.3:0/3943511712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940a8c50 0x7f3394152d70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f338c002f70 tx=0x7f338c006df0).ready entity=mon.1 client_cookie=f21aa3c41d846d21 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3392ffd700 1 -- 172.21.15.3:0/3943511712 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f338c00d540 con 0x7f33940a8c50 2022-07-21T01:40:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3392ffd700 1 -- 172.21.15.3:0/3943511712 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c000e90 con 0x7f33940a8c50 2022-07-21T01:40:44.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3392ffd700 1 -- 172.21.15.3:0/3943511712 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f338c00d890 con 0x7f33940a8c50 2022-07-21T01:40:44.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 -- 172.21.15.3:0/3943511712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940a8c50 msgr2=0x7f3394152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 --2- 172.21.15.3:0/3943511712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940a8c50 0x7f3394152d70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f338c002f70 tx=0x7f338c006df0).stop 2022-07-21T01:40:44.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 -- 172.21.15.3:0/3943511712 shutdown_connections 2022-07-21T01:40:44.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 --2- 172.21.15.3:0/3943511712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940a8c50 0x7f3394152d70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 -- 172.21.15.3:0/3943511712 >> 172.21.15.3:0/3943511712 conn(0x7f3394060310 msgr2=0x7f33940b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:44.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 -- 172.21.15.3:0/3943511712 shutdown_connections 2022-07-21T01:40:44.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.773+0000 7f3399172700 1 -- 172.21.15.3:0/3943511712 wait complete. 2022-07-21T01:40:44.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 Processor -- start 2022-07-21T01:40:44.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 -- start start 2022-07-21T01:40:44.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33940a8c50 0x7f33940af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33940b01a0 0x7f33940b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b2710 0x7f33940b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33940b2f60 con 0x7f33940a8c50 2022-07-21T01:40:44.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33940b3840 con 0x7f33940b2710 2022-07-21T01:40:44.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33940b3190 con 0x7f33940b01a0 2022-07-21T01:40:44.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3393fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33940a8c50 0x7f33940af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:44.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3393fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33940a8c50 0x7f33940af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48438/0 (socket says 172.21.15.3:48438) 2022-07-21T01:40:44.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3393fff700 1 -- 172.21.15.3:0/1462016542 learned_addr learned my addr 172.21.15.3:0/1462016542 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:44.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3398971700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b2710 0x7f33940b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:44.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f33937fe700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33940b01a0 0x7f33940b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:44.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3393fff700 1 -- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33940b01a0 msgr2=0x7f33940b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3393fff700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33940b01a0 0x7f33940b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3393fff700 1 -- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b2710 msgr2=0x7f33940b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3393fff700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b2710 0x7f33940b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3393fff700 1 -- 172.21.15.3:0/1462016542 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c00b180 con 0x7f33940a8c50 2022-07-21T01:40:44.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3398971700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b2710 0x7f33940b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:40:44.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f33937fe700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33940b01a0 0x7f33940b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:40:44.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3393fff700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33940a8c50 0x7f33940af180 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f3384001490 tx=0x7f3384007e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:44.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f33917fa700 1 -- 172.21.15.3:0/1462016542 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3384002030 con 0x7f33940a8c50 2022-07-21T01:40:44.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f33917fa700 1 -- 172.21.15.3:0/1462016542 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3384002190 con 0x7f33940a8c50 2022-07-21T01:40:44.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f33917fa700 1 -- 172.21.15.3:0/1462016542 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33840024f0 con 0x7f33940a8c50 2022-07-21T01:40:44.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33940b3aa0 con 0x7f33940a8c50 2022-07-21T01:40:44.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.777+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33940aa4f0 con 0x7f33940a8c50 2022-07-21T01:40:44.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.781+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33940574e0 con 0x7f33940a8c50 2022-07-21T01:40:44.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.781+0000 7f33917fa700 1 -- 172.21.15.3:0/1462016542 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3384010070 con 0x7f33940a8c50 2022-07-21T01:40:44.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.781+0000 7f33917fa700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3378053a10 0x7f3378055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:44.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.781+0000 7f33917fa700 1 -- 172.21.15.3:0/1462016542 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f338406b710 con 0x7f33940a8c50 2022-07-21T01:40:44.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.781+0000 7f33937fe700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3378053a10 0x7f3378055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:44.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.781+0000 7f33937fe700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3378053a10 0x7f3378055ed0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f337c008000 tx=0x7f337c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:44.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.789+0000 7f33917fa700 1 -- 172.21.15.3:0/1462016542 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f33840424d0 con 0x7f33940a8c50 2022-07-21T01:40:44.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.917+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33940b54b0 con 0x7f33940a8c50 2022-07-21T01:40:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.917+0000 7f33917fa700 1 -- 172.21.15.3:0/1462016542 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16277 (secure 0 0 0) 0x7f3384045af0 con 0x7f33940a8c50 2022-07-21T01:40:44.919 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:44.919 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":101,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:42.037286+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:39.112065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:40.040729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:44.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.917+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3378053a10 msgr2=0x7f3378055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.917+0000 7f3399172700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3378053a10 0x7f3378055ed0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f337c008000 tx=0x7f337c009040).stop 2022-07-21T01:40:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.917+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33940a8c50 msgr2=0x7f33940af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.917+0000 7f3399172700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33940a8c50 0x7f33940af180 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f3384001490 tx=0x7f3384007e60).stop 2022-07-21T01:40:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.921+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 shutdown_connections 2022-07-21T01:40:44.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.921+0000 7f3399172700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3378053a10 0x7f3378055ed0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.921+0000 7f3399172700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b2710 0x7f33940b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.921+0000 7f3399172700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f33940b01a0 0x7f33940b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.921+0000 7f3399172700 1 --2- 172.21.15.3:0/1462016542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33940a8c50 0x7f33940af180 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.921+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 >> 172.21.15.3:0/1462016542 conn(0x7f3394060310 msgr2=0x7f33941499b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.921+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 shutdown_connections 2022-07-21T01:40:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:44.921+0000 7f3399172700 1 -- 172.21.15.3:0/1462016542 wait complete. 2022-07-21T01:40:44.933 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:40:39.112065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:40:40.040729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:40:44.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:40:45.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 Processor -- start 2022-07-21T01:40:45.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 -- start start 2022-07-21T01:40:45.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 0x7fb8c405f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb8c40b00a0 0x7fb8c414b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb8c40acb40 0x7fb8c40af350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c40614f0 con 0x7fb8c40b00a0 2022-07-21T01:40:45.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c414b770 con 0x7fb8c40acb40 2022-07-21T01:40:45.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c414ba10 con 0x7fb8c414ad10 2022-07-21T01:40:45.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c1d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb8c40b00a0 0x7fb8c414b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c1d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb8c40b00a0 0x7fb8c414b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48446/0 (socket says 172.21.15.3:48446) 2022-07-21T01:40:45.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c1d9b700 1 -- 172.21.15.3:0/4033393559 learned_addr learned my addr 172.21.15.3:0/4033393559 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:45.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c2d9d700 1 --2- 172.21.15.3:0/4033393559 >> v2:172.21.15.143:3300/0 conn(0x7fb8c40acb40 0x7fb8c40af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c259c700 1 --2- 172.21.15.3:0/4033393559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 0x7fb8c405f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c1d9b700 1 -- 172.21.15.3:0/4033393559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 msgr2=0x7fb8c405f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:45.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c1d9b700 1 --2- 172.21.15.3:0/4033393559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 0x7fb8c405f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c1d9b700 1 -- 172.21.15.3:0/4033393559 >> v2:172.21.15.143:3300/0 conn(0x7fb8c40acb40 msgr2=0x7fb8c40af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c1d9b700 1 --2- 172.21.15.3:0/4033393559 >> v2:172.21.15.143:3300/0 conn(0x7fb8c40acb40 0x7fb8c40af350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c1d9b700 1 -- 172.21.15.3:0/4033393559 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c40aeeb0 con 0x7fb8c40b00a0 2022-07-21T01:40:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c2d9d700 1 --2- 172.21.15.3:0/4033393559 >> v2:172.21.15.143:3300/0 conn(0x7fb8c40acb40 0x7fb8c40af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c1d9b700 1 --2- 172.21.15.3:0/4033393559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 0x7fb8c414b0f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7fb8b4002f70 tx=0x7fb8b4006eb0).ready entity=mon.0 client_cookie=a0de7508d633e028 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c159a700 1 -- 172.21.15.3:0/4033393559 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8b400d570 con 0x7fb8c40b00a0 2022-07-21T01:40:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c159a700 1 -- 172.21.15.3:0/4033393559 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b4000e90 con 0x7fb8c40b00a0 2022-07-21T01:40:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c159a700 1 -- 172.21.15.3:0/4033393559 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8b400d900 con 0x7fb8c40b00a0 2022-07-21T01:40:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 -- 172.21.15.3:0/4033393559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 msgr2=0x7fb8c414b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/4033393559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 0x7fb8c414b0f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7fb8b4002f70 tx=0x7fb8b4006eb0).stop 2022-07-21T01:40:45.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 -- 172.21.15.3:0/4033393559 shutdown_connections 2022-07-21T01:40:45.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/4033393559 >> v2:172.21.15.143:3300/0 conn(0x7fb8c40acb40 0x7fb8c40af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/4033393559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 0x7fb8c414b0f0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/4033393559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 0x7fb8c405f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 -- 172.21.15.3:0/4033393559 >> 172.21.15.3:0/4033393559 conn(0x7fb8c40b5700 msgr2=0x7fb8c40b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:45.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 -- 172.21.15.3:0/4033393559 shutdown_connections 2022-07-21T01:40:45.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.041+0000 7fb8c8f43700 1 -- 172.21.15.3:0/4033393559 wait complete. 2022-07-21T01:40:45.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 Processor -- start 2022-07-21T01:40:45.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 -- start start 2022-07-21T01:40:45.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c40acb40 0x7fb8c414bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 0x7fb8c41426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 0x7fb8c4142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4147870 con 0x7fb8c40b00a0 2022-07-21T01:40:45.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4143090 con 0x7fb8c40acb40 2022-07-21T01:40:45.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c41458e0 con 0x7fb8c414ad10 2022-07-21T01:40:45.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 0x7fb8c41426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 0x7fb8c41426b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48452/0 (socket says 172.21.15.3:48452) 2022-07-21T01:40:45.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c1d9b700 1 -- 172.21.15.3:0/3479573971 learned_addr learned my addr 172.21.15.3:0/3479573971 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:45.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c259c700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c40acb40 0x7fb8c414bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c2d9d700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 0x7fb8c4142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c1d9b700 1 -- 172.21.15.3:0/3479573971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 msgr2=0x7fb8c4142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c1d9b700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 0x7fb8c4142ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c1d9b700 1 -- 172.21.15.3:0/3479573971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c40acb40 msgr2=0x7fb8c414bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c1d9b700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c40acb40 0x7fb8c414bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c1d9b700 1 -- 172.21.15.3:0/3479573971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b400b1e0 con 0x7fb8c40b00a0 2022-07-21T01:40:45.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c259c700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c40acb40 0x7fb8c414bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c1d9b700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 0x7fb8c41426b0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7fb8c41431d0 tx=0x7fb8b4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8b37fe700 1 -- 172.21.15.3:0/3479573971 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8b4011860 con 0x7fb8c40b00a0 2022-07-21T01:40:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8b37fe700 1 -- 172.21.15.3:0/3479573971 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b40119c0 con 0x7fb8c40b00a0 2022-07-21T01:40:45.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c40b49b0 con 0x7fb8c40b00a0 2022-07-21T01:40:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8b37fe700 1 -- 172.21.15.3:0/3479573971 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8b4011d20 con 0x7fb8c40b00a0 2022-07-21T01:40:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.045+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c4145b10 con 0x7fb8c40b00a0 2022-07-21T01:40:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.049+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c40574e0 con 0x7fb8c40b00a0 2022-07-21T01:40:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.049+0000 7fb8b37fe700 1 -- 172.21.15.3:0/3479573971 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb8b4025030 con 0x7fb8c40b00a0 2022-07-21T01:40:45.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.049+0000 7fb8b37fe700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8a0053aa0 0x7fb8a0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.049+0000 7fb8b37fe700 1 -- 172.21.15.3:0/3479573971 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb8b406d250 con 0x7fb8c40b00a0 2022-07-21T01:40:45.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.049+0000 7fb8c259c700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8a0053aa0 0x7fb8a0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.049+0000 7fb8c259c700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8a0053aa0 0x7fb8a0055f60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fb8b800c000 tx=0x7fb8b8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.057+0000 7fb8b37fe700 1 -- 172.21.15.3:0/3479573971 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb8b4043f80 con 0x7fb8c40b00a0 2022-07-21T01:40:45.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.181+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb8c40447e0 con 0x7fb8c40b00a0 2022-07-21T01:40:45.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.181+0000 7fb8b37fe700 1 -- 172.21.15.3:0/3479573971 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v101) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb8b4023020 con 0x7fb8c40b00a0 2022-07-21T01:40:45.184 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:40:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8a0053aa0 msgr2=0x7fb8a0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8a0053aa0 0x7fb8a0055f60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fb8b800c000 tx=0x7fb8b8008040).stop 2022-07-21T01:40:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 msgr2=0x7fb8c41426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 0x7fb8c41426b0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7fb8c41431d0 tx=0x7fb8b4015000).stop 2022-07-21T01:40:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 shutdown_connections 2022-07-21T01:40:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c414ad10 0x7fb8c4142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8c40b00a0 0x7fb8c41426b0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8a0053aa0 0x7fb8a0055f60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 --2- 172.21.15.3:0/3479573971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c40acb40 0x7fb8c414bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 >> 172.21.15.3:0/3479573971 conn(0x7fb8c40b5700 msgr2=0x7fb8c40b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 shutdown_connections 2022-07-21T01:40:45.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.185+0000 7fb8c8f43700 1 -- 172.21.15.3:0/3479573971 wait complete. 2022-07-21T01:40:45.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:40:45.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.301+0000 7f91ef59e700 1 Processor -- start 2022-07-21T01:40:45.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.301+0000 7f91ef59e700 1 -- start start 2022-07-21T01:40:45.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 0x7f91f00b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f91f00ae940 0x7f91f00aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f00b0910 con 0x7f91f00b1240 2022-07-21T01:40:45.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00b1f70 con 0x7f91f00ae940 2022-07-21T01:40:45.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00af7c0 con 0x7f91f00b0530 2022-07-21T01:40:45.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91eed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f91f00ae940 0x7f91f00aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91eed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f91f00ae940 0x7f91f00aedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48462/0 (socket says 172.21.15.3:48462) 2022-07-21T01:40:45.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91eed9d700 1 -- 172.21.15.3:0/3689479937 learned_addr learned my addr 172.21.15.3:0/3689479937 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:45.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ee59c700 1 --2- 172.21.15.3:0/3689479937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 0x7f91f00b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91eed9d700 1 -- 172.21.15.3:0/3689479937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 msgr2=0x7f91f00b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:45.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91eed9d700 1 --2- 172.21.15.3:0/3689479937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 0x7f91f00b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91eed9d700 1 -- 172.21.15.3:0/3689479937 >> v1:172.21.15.3:6789/0 conn(0x7f91f00b1240 legacy=0x7f91f0142da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ed59a700 1 -- 172.21.15.3:0/3689479937 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220447515 0 0) 0x7f91f00b0910 con 0x7f91f00b1240 2022-07-21T01:40:45.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91eed9d700 1 -- 172.21.15.3:0/3689479937 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f00afa60 con 0x7f91f00ae940 2022-07-21T01:40:45.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91eed9d700 1 --2- 172.21.15.3:0/3689479937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 0x7f91f00aedf0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f91e8002f70 tx=0x7f91e8006f40).ready entity=mon.0 client_cookie=569e0130e449ca9c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ed59a700 1 -- 172.21.15.3:0/3689479937 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91e800da80 con 0x7f91f00ae940 2022-07-21T01:40:45.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ed59a700 1 -- 172.21.15.3:0/3689479937 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e8000e90 con 0x7f91f00ae940 2022-07-21T01:40:45.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ed59a700 1 -- 172.21.15.3:0/3689479937 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91e800ddd0 con 0x7f91f00ae940 2022-07-21T01:40:45.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 -- 172.21.15.3:0/3689479937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 msgr2=0x7f91f00aedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 --2- 172.21.15.3:0/3689479937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 0x7f91f00aedf0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f91e8002f70 tx=0x7f91e8006f40).stop 2022-07-21T01:40:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 -- 172.21.15.3:0/3689479937 shutdown_connections 2022-07-21T01:40:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 --2- 172.21.15.3:0/3689479937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 0x7f91f00aedf0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 --2- 172.21.15.3:0/3689479937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 0x7f91f00b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 -- 172.21.15.3:0/3689479937 >> 172.21.15.3:0/3689479937 conn(0x7f91f005c620 msgr2=0x7f91f005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:45.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 -- 172.21.15.3:0/3689479937 shutdown_connections 2022-07-21T01:40:45.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.305+0000 7f91ef59e700 1 -- 172.21.15.3:0/3689479937 wait complete. 2022-07-21T01:40:45.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 Processor -- start 2022-07-21T01:40:45.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 -- start start 2022-07-21T01:40:45.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 0x7f91f0141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 0x7f91f0156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f00b1240 0x7f91f0156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00670e0 con 0x7f91f00ae940 2022-07-21T01:40:45.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00ab870 con 0x7f91f00b1240 2022-07-21T01:40:45.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00abb10 con 0x7f91f00b0530 2022-07-21T01:40:45.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 0x7f91f0141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 0x7f91f0141c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48466/0 (socket says 172.21.15.3:48466) 2022-07-21T01:40:45.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ee59c700 1 -- 172.21.15.3:0/1986019578 learned_addr learned my addr 172.21.15.3:0/1986019578 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:45.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91eed9d700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f00b1240 0x7f91f0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91edd9b700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 0x7f91f0156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ee59c700 1 -- 172.21.15.3:0/1986019578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 msgr2=0x7f91f0156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ee59c700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 0x7f91f0156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ee59c700 1 -- 172.21.15.3:0/1986019578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f00b1240 msgr2=0x7f91f0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ee59c700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f00b1240 0x7f91f0156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ee59c700 1 -- 172.21.15.3:0/1986019578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e800b460 con 0x7f91f00ae940 2022-07-21T01:40:45.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ee59c700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 0x7f91f0141c60 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7f91e000c000 tx=0x7f91e00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91e77fe700 1 -- 172.21.15.3:0/1986019578 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91e000c850 con 0x7f91f00ae940 2022-07-21T01:40:45.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91e77fe700 1 -- 172.21.15.3:0/1986019578 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e000c9b0 con 0x7f91f00ae940 2022-07-21T01:40:45.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f0067310 con 0x7f91f00ae940 2022-07-21T01:40:45.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91e77fe700 1 -- 172.21.15.3:0/1986019578 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91e000cd10 con 0x7f91f00ae940 2022-07-21T01:40:45.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f01569f0 con 0x7f91f00ae940 2022-07-21T01:40:45.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.309+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f00b2190 con 0x7f91f00ae940 2022-07-21T01:40:45.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.313+0000 7f91e77fe700 1 -- 172.21.15.3:0/1986019578 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f91e0022070 con 0x7f91f00ae940 2022-07-21T01:40:45.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.313+0000 7f91e77fe700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f91d4053aa0 0x7f91d4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.313+0000 7f91e77fe700 1 -- 172.21.15.3:0/1986019578 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f91e006c890 con 0x7f91f00ae940 2022-07-21T01:40:45.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.313+0000 7f91edd9b700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f91d4053aa0 0x7f91d4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.313+0000 7f91edd9b700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f91d4053aa0 0x7f91d4055f60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f91d800b000 tx=0x7f91d8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.317+0000 7f91e77fe700 1 -- 172.21.15.3:0/1986019578 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f91e0043540 con 0x7f91f00ae940 2022-07-21T01:40:45.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.445+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f91f00674a0 con 0x7f91f00ae940 2022-07-21T01:40:45.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91e77fe700 1 -- 172.21.15.3:0/1986019578 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v101) v1 ==== 92+0+10 (secure 0 0 0) 0x7f91e006ae60 con 0x7f91f00ae940 2022-07-21T01:40:45.449 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:45.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f91d4053aa0 msgr2=0x7f91d4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f91d4053aa0 0x7f91d4055f60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f91d800b000 tx=0x7f91d8009040).stop 2022-07-21T01:40:45.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 msgr2=0x7f91f0141c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 0x7f91f0141c60 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7f91e000c000 tx=0x7f91e00091d0).stop 2022-07-21T01:40:45.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 shutdown_connections 2022-07-21T01:40:45.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f91d4053aa0 0x7f91d4055f60 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f00b1240 0x7f91f0156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f00b0530 0x7f91f0156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 --2- 172.21.15.3:0/1986019578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91f00ae940 0x7f91f0141c60 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 >> 172.21.15.3:0/1986019578 conn(0x7f91f005c620 msgr2=0x7f91f00aed20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:45.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 shutdown_connections 2022-07-21T01:40:45.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.449+0000 7f91ef59e700 1 -- 172.21.15.3:0/1986019578 wait complete. 2022-07-21T01:40:45.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:40:45.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd7b3c700 1 Processor -- start 2022-07-21T01:40:45.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd7b3c700 1 -- start start 2022-07-21T01:40:45.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd7b3c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6bd00b3de0 0x7f6bd0061f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd7b3c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0060340 con 0x7f6bd0061b70 2022-07-21T01:40:45.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd7b3c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd00b6bd0 con 0x7f6bd00b6760 2022-07-21T01:40:45.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd7b3c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd00aeb50 con 0x7f6bd00b3de0 2022-07-21T01:40:45.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd58d8700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6bd0061b70 0x7f6bd005f8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56754/0 (socket says 172.21.15.3:56754) 2022-07-21T01:40:45.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd58d8700 1 -- 172.21.15.3:0/2624825431 learned_addr learned my addr 172.21.15.3:0/2624825431 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:45.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd50d7700 1 --2- 172.21.15.3:0/2624825431 >> v2:172.21.15.143:3300/0 conn(0x7f6bd00b3de0 0x7f6bd0061f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.573+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847336310 0 0) 0x7f6bd0060340 con 0x7f6bd0061b70 2022-07-21T01:40:45.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bbc003260 con 0x7f6bd0061b70 2022-07-21T01:40:45.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960531578 0 0) 0x7f6bd00b6bd0 con 0x7f6bd00b6760 2022-07-21T01:40:45.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd0060340 con 0x7f6bd00b6760 2022-07-21T01:40:45.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 82802087 0 0) 0x7f6bbc003260 con 0x7f6bd0061b70 2022-07-21T01:40:45.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 >> v2:172.21.15.143:3300/0 conn(0x7f6bd00b3de0 msgr2=0x7f6bd0061f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:45.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 --2- 172.21.15.3:0/2624825431 >> v2:172.21.15.143:3300/0 conn(0x7f6bd00b3de0 0x7f6bd0061f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 >> v1:172.21.15.143:6789/0 conn(0x7f6bd00b6760 legacy=0x7f6bd0143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd00600a0 con 0x7f6bd0061b70 2022-07-21T01:40:45.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f6bc80019e0 con 0x7f6bd0061b70 2022-07-21T01:40:45.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 >> v1:172.21.15.3:6789/0 conn(0x7f6bd0061b70 legacy=0x7f6bd005f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bbc003930 0x7f6bbc003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bbc004500 0x7f6bbc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bbc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00aeb50 con 0x7f6bbc004e70 2022-07-21T01:40:45.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc00bd20 con 0x7f6bbc004500 2022-07-21T01:40:45.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc00bee0 con 0x7f6bbc003930 2022-07-21T01:40:45.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd60d9700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bbc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd58d8700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bbc003930 0x7f6bbc003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd50d7700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bbc004500 0x7f6bbc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd60d9700 1 -- 172.21.15.3:0/2624825431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bbc003930 msgr2=0x7f6bbc003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:45.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd60d9700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bbc003930 0x7f6bbc003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd60d9700 1 -- 172.21.15.3:0/2624825431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bbc004500 msgr2=0x7f6bbc004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:45.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd60d9700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bbc004500 0x7f6bbc004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd60d9700 1 -- 172.21.15.3:0/2624825431 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc003260 con 0x7f6bbc004e70 2022-07-21T01:40:45.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd50d7700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bbc004500 0x7f6bbc004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:45.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd60d9700 1 -- 172.21.15.3:0/2624825431 reap_dead start 2022-07-21T01:40:45.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd60d9700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bbc00b750 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f6bc0001110 tx=0x7f6bc0007a00).ready entity=mon.0 client_cookie=f96f344381e77801 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc0007d10 con 0x7f6bbc004e70 2022-07-21T01:40:45.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc0007030 con 0x7f6bbc004e70 2022-07-21T01:40:45.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd48d6700 1 -- 172.21.15.3:0/2624825431 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc0007390 con 0x7f6bbc004e70 2022-07-21T01:40:45.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/2624825431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 msgr2=0x7f6bbc00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd7b3c700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bbc00b750 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f6bc0001110 tx=0x7f6bc0007a00).stop 2022-07-21T01:40:45.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/2624825431 shutdown_connections 2022-07-21T01:40:45.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd7b3c700 1 --2- 172.21.15.3:0/2624825431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bbc00b750 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/2624825431 >> 172.21.15.3:0/2624825431 conn(0x7f6bd0063b40 msgr2=0x7f6bd0063f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:45.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/2624825431 shutdown_connections 2022-07-21T01:40:45.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.577+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/2624825431 wait complete. 2022-07-21T01:40:45.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 Processor -- start 2022-07-21T01:40:45.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 -- start start 2022-07-21T01:40:45.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bd0145050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd013deb0 0x7f6bd013b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd013b9b0 0x7f6bd01525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0144300 con 0x7f6bbc004e70 2022-07-21T01:40:45.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0150190 con 0x7f6bd013deb0 2022-07-21T01:40:45.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0156450 con 0x7f6bd013b9b0 2022-07-21T01:40:45.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd58d8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bd0145050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd58d8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bd0145050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48486/0 (socket says 172.21.15.3:48486) 2022-07-21T01:40:45.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd58d8700 1 -- 172.21.15.3:0/396883791 learned_addr learned my addr 172.21.15.3:0/396883791 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:45.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd50d7700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd013deb0 0x7f6bd013b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd60d9700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd013b9b0 0x7f6bd01525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd58d8700 1 -- 172.21.15.3:0/396883791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd013b9b0 msgr2=0x7f6bd01525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd58d8700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd013b9b0 0x7f6bd01525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd58d8700 1 -- 172.21.15.3:0/396883791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd013deb0 msgr2=0x7f6bd013b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd58d8700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd013deb0 0x7f6bd013b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd58d8700 1 -- 172.21.15.3:0/396883791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd01566f0 con 0x7f6bbc004e70 2022-07-21T01:40:45.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd50d7700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd013deb0 0x7f6bd013b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:40:45.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd58d8700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bd0145050 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f6bd0145540 tx=0x7f6bd0060230).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bceffd700 1 -- 172.21.15.3:0/396883791 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc800fa30 con 0x7f6bbc004e70 2022-07-21T01:40:45.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bceffd700 1 -- 172.21.15.3:0/396883791 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc800fb90 con 0x7f6bbc004e70 2022-07-21T01:40:45.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd0145910 con 0x7f6bbc004e70 2022-07-21T01:40:45.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bceffd700 1 -- 172.21.15.3:0/396883791 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc8012350 con 0x7f6bbc004e70 2022-07-21T01:40:45.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.581+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd013be30 con 0x7f6bbc004e70 2022-07-21T01:40:45.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.585+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd013e290 con 0x7f6bbc004e70 2022-07-21T01:40:45.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.585+0000 7f6bceffd700 1 -- 172.21.15.3:0/396883791 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6bc801b030 con 0x7f6bbc004e70 2022-07-21T01:40:45.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.585+0000 7f6bceffd700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053a70 0x7f6bb0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.585+0000 7f6bceffd700 1 -- 172.21.15.3:0/396883791 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6bc806bdd0 con 0x7f6bbc004e70 2022-07-21T01:40:45.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.585+0000 7f6bd50d7700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053a70 0x7f6bb0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.585+0000 7f6bd50d7700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053a70 0x7f6bb0055f30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f6bc400b000 tx=0x7f6bc4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.597+0000 7f6bceffd700 1 -- 172.21.15.3:0/396883791 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6bc8042a80 con 0x7f6bbc004e70 2022-07-21T01:40:45.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.725+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6bd01434b0 con 0x7f6bbc004e70 2022-07-21T01:40:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.725+0000 7f6bceffd700 1 -- 172.21.15.3:0/396883791 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+10 (secure 0 0 0) 0x7f6bc8019070 con 0x7f6bbc004e70 2022-07-21T01:40:45.727 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:45.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053a70 msgr2=0x7f6bb0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053a70 0x7f6bb0055f30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f6bc400b000 tx=0x7f6bc4008040).stop 2022-07-21T01:40:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 msgr2=0x7f6bd0145050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bd0145050 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f6bd0145540 tx=0x7f6bd0060230).stop 2022-07-21T01:40:45.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 shutdown_connections 2022-07-21T01:40:45.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd013b9b0 0x7f6bd01525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6bb0053a70 0x7f6bb0055f30 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd013deb0 0x7f6bd013b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 --2- 172.21.15.3:0/396883791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bbc004e70 0x7f6bd0145050 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 >> 172.21.15.3:0/396883791 conn(0x7f6bd0063b40 msgr2=0x7f6bd01407d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:45.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 shutdown_connections 2022-07-21T01:40:45.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.729+0000 7f6bd7b3c700 1 -- 172.21.15.3:0/396883791 wait complete. 2022-07-21T01:40:45.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:40:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc41d54700 1 Processor -- start 2022-07-21T01:40:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc41d54700 1 -- start start 2022-07-21T01:40:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc41d54700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c0614b0 0x7efc3c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc41d54700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efc3c0ad750 0x7efc3c0b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc41d54700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3c0aee80 con 0x7efc3c1405d0 2022-07-21T01:40:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc41d54700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3c0b70b0 con 0x7efc3c0ad750 2022-07-21T01:40:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc41d54700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0af0b0 con 0x7efc3c0614b0 2022-07-21T01:40:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc40d52700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c0614b0 0x7efc3c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc40d52700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c0614b0 0x7efc3c061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:55410/0 (socket says 172.21.15.3:55410) 2022-07-21T01:40:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc40d52700 1 -- 172.21.15.3:0/1552066239 learned_addr learned my addr 172.21.15.3:0/1552066239 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:45.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc41553700 1 --1- 172.21.15.3:0/1552066239 >> v1:172.21.15.143:6789/0 conn(0x7efc3c1405d0 0x7efc3c1466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:58890/0 (socket says 172.21.15.3:58890) 2022-07-21T01:40:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.853+0000 7efc3bfff700 1 --2- 172.21.15.3:0/1552066239 >> v2:172.21.15.143:3300/0 conn(0x7efc3c0ad750 0x7efc3c0b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc40d52700 1 -- 172.21.15.3:0/1552066239 >> v2:172.21.15.143:3300/0 conn(0x7efc3c0ad750 msgr2=0x7efc3c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc40d52700 1 --2- 172.21.15.3:0/1552066239 >> v2:172.21.15.143:3300/0 conn(0x7efc3c0ad750 0x7efc3c0b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc3b7fe700 1 -- 172.21.15.3:0/1552066239 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396424682 0 0) 0x7efc3c0aee80 con 0x7efc3c1405d0 2022-07-21T01:40:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc40d52700 1 -- 172.21.15.3:0/1552066239 >> v1:172.21.15.143:6789/0 conn(0x7efc3c1405d0 legacy=0x7efc3c1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc40d52700 1 -- 172.21.15.3:0/1552066239 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c0af350 con 0x7efc3c0614b0 2022-07-21T01:40:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc40d52700 1 --2- 172.21.15.3:0/1552066239 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c0614b0 0x7efc3c061890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7efc2c002f70 tx=0x7efc2c006eb0).ready entity=mon.2 client_cookie=9c58a0db0bf0fc38 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc3b7fe700 1 -- 172.21.15.3:0/1552066239 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc2c00d570 con 0x7efc3c0614b0 2022-07-21T01:40:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc3b7fe700 1 -- 172.21.15.3:0/1552066239 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2c000e90 con 0x7efc3c0614b0 2022-07-21T01:40:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc3b7fe700 1 -- 172.21.15.3:0/1552066239 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc2c00d8c0 con 0x7efc3c0614b0 2022-07-21T01:40:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc41d54700 1 -- 172.21.15.3:0/1552066239 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c0614b0 msgr2=0x7efc3c061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc41d54700 1 --2- 172.21.15.3:0/1552066239 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c0614b0 0x7efc3c061890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7efc2c002f70 tx=0x7efc2c006eb0).stop 2022-07-21T01:40:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc41d54700 1 -- 172.21.15.3:0/1552066239 shutdown_connections 2022-07-21T01:40:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc41d54700 1 --2- 172.21.15.3:0/1552066239 >> v2:172.21.15.143:3300/0 conn(0x7efc3c0ad750 0x7efc3c0b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc41d54700 1 --2- 172.21.15.3:0/1552066239 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c0614b0 0x7efc3c061890 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc41d54700 1 -- 172.21.15.3:0/1552066239 >> 172.21.15.3:0/1552066239 conn(0x7efc3c066f70 msgr2=0x7efc3c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc41d54700 1 -- 172.21.15.3:0/1552066239 shutdown_connections 2022-07-21T01:40:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.857+0000 7efc41d54700 1 -- 172.21.15.3:0/1552066239 wait complete. 2022-07-21T01:40:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc41d54700 1 Processor -- start 2022-07-21T01:40:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc41d54700 1 -- start start 2022-07-21T01:40:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc41d54700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0614b0 0x7efc3c05cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc41d54700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc3c0ad750 0x7efc3c0b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc41d54700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c1405d0 0x7efc3c0b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc41d54700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c155d20 con 0x7efc3c0614b0 2022-07-21T01:40:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc41d54700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c1336c0 con 0x7efc3c0ad750 2022-07-21T01:40:45.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc41d54700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0ab240 con 0x7efc3c1405d0 2022-07-21T01:40:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc40d52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0614b0 0x7efc3c05cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc40d52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0614b0 0x7efc3c05cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48500/0 (socket says 172.21.15.3:48500) 2022-07-21T01:40:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc40d52700 1 -- 172.21.15.3:0/67807286 learned_addr learned my addr 172.21.15.3:0/67807286 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc3bfff700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc3c0ad750 0x7efc3c0b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc41553700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c1405d0 0x7efc3c0b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc40d52700 1 -- 172.21.15.3:0/67807286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c1405d0 msgr2=0x7efc3c0b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc40d52700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c1405d0 0x7efc3c0b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc40d52700 1 -- 172.21.15.3:0/67807286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc3c0ad750 msgr2=0x7efc3c0b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc40d52700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc3c0ad750 0x7efc3c0b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc40d52700 1 -- 172.21.15.3:0/67807286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c00b1e0 con 0x7efc3c0614b0 2022-07-21T01:40:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc40d52700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0614b0 0x7efc3c05cb40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7efc2c002f70 tx=0x7efc2c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc39ffb700 1 -- 172.21.15.3:0/67807286 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc2c011860 con 0x7efc3c0614b0 2022-07-21T01:40:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.861+0000 7efc39ffb700 1 -- 172.21.15.3:0/67807286 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2c0119c0 con 0x7efc3c0614b0 2022-07-21T01:40:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.865+0000 7efc39ffb700 1 -- 172.21.15.3:0/67807286 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc2c011d20 con 0x7efc3c0614b0 2022-07-21T01:40:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.865+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c155a80 con 0x7efc3c0614b0 2022-07-21T01:40:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.865+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c0acab0 con 0x7efc3c0614b0 2022-07-21T01:40:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.865+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3c0ab400 con 0x7efc3c0614b0 2022-07-21T01:40:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.865+0000 7efc39ffb700 1 -- 172.21.15.3:0/67807286 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efc2c01a040 con 0x7efc3c0614b0 2022-07-21T01:40:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.865+0000 7efc39ffb700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc1c053950 0x7efc1c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:45.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.865+0000 7efc39ffb700 1 -- 172.21.15.3:0/67807286 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7efc2c06d060 con 0x7efc3c0614b0 2022-07-21T01:40:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.865+0000 7efc3bfff700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc1c053950 0x7efc1c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.865+0000 7efc3bfff700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc1c053950 0x7efc1c055e10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7efc3000b000 tx=0x7efc30006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:45.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.873+0000 7efc39ffb700 1 -- 172.21.15.3:0/67807286 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc3c0ab400 con 0x7efc3c0614b0 2022-07-21T01:40:45.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.997+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efc3c0ab400 con 0x7efc3c0614b0 2022-07-21T01:40:46.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:45.997+0000 7efc39ffb700 1 -- 172.21.15.3:0/67807286 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+10 (secure 0 0 0) 0x7efc3c0ab400 con 0x7efc3c0614b0 2022-07-21T01:40:46.000 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:46.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc1c053950 msgr2=0x7efc1c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:46.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc1c053950 0x7efc1c055e10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7efc3000b000 tx=0x7efc30006df0).stop 2022-07-21T01:40:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0614b0 msgr2=0x7efc3c05cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0614b0 0x7efc3c05cb40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7efc2c002f70 tx=0x7efc2c015000).stop 2022-07-21T01:40:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 shutdown_connections 2022-07-21T01:40:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc3c1405d0 0x7efc3c0b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc1c053950 0x7efc1c055e10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc3c0ad750 0x7efc3c0b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 --2- 172.21.15.3:0/67807286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0614b0 0x7efc3c05cb40 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 >> 172.21.15.3:0/67807286 conn(0x7efc3c066f70 msgr2=0x7efc3c0b7680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:46.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 shutdown_connections 2022-07-21T01:40:46.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.001+0000 7efc41d54700 1 -- 172.21.15.3:0/67807286 wait complete. 2022-07-21T01:40:46.025 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:40:46.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:46.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 Processor -- start 2022-07-21T01:40:46.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 -- start start 2022-07-21T01:40:46.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b01a0 0x7f93240a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f93240afd60 0x7f93240a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93240a7250 con 0x7f93240b2710 2022-07-21T01:40:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93240a7480 con 0x7f93240afd60 2022-07-21T01:40:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93240a7720 con 0x7f93240b01a0 2022-07-21T01:40:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f9329b7b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f93240afd60 0x7f93240a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f9329b7b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f93240afd60 0x7f93240a8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48512/0 (socket says 172.21.15.3:48512) 2022-07-21T01:40:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f9329b7b700 1 -- 172.21.15.3:0/3500013170 learned_addr learned my addr 172.21.15.3:0/3500013170 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f9328b79700 1 --2- 172.21.15.3:0/3500013170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b01a0 0x7f93240a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f9329b7b700 1 -- 172.21.15.3:0/3500013170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b01a0 msgr2=0x7f93240a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:46.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f9329b7b700 1 --2- 172.21.15.3:0/3500013170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b01a0 0x7f93240a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f9329b7b700 1 -- 172.21.15.3:0/3500013170 >> v1:172.21.15.143:6789/0 conn(0x7f93240b2710 legacy=0x7f932414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:46.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f9329b7b700 1 -- 172.21.15.3:0/3500013170 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93240a79c0 con 0x7f93240afd60 2022-07-21T01:40:46.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f9329b7b700 1 --2- 172.21.15.3:0/3500013170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240afd60 0x7f93240a8a30 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f931c002f70 tx=0x7f931c006eb0).ready entity=mon.0 client_cookie=9137a4b0f35e8eaa server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:46.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f931bfff700 1 -- 172.21.15.3:0/3500013170 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f931c00d570 con 0x7f93240afd60 2022-07-21T01:40:46.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f931bfff700 1 -- 172.21.15.3:0/3500013170 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931c000e90 con 0x7f93240afd60 2022-07-21T01:40:46.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f931bfff700 1 -- 172.21.15.3:0/3500013170 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f931c00d900 con 0x7f93240afd60 2022-07-21T01:40:46.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 -- 172.21.15.3:0/3500013170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240afd60 msgr2=0x7f93240a8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:46.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 --2- 172.21.15.3:0/3500013170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240afd60 0x7f93240a8a30 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f931c002f70 tx=0x7f931c006eb0).stop 2022-07-21T01:40:46.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 -- 172.21.15.3:0/3500013170 shutdown_connections 2022-07-21T01:40:46.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.133+0000 7f932a37c700 1 --2- 172.21.15.3:0/3500013170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240afd60 0x7f93240a8a30 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 --2- 172.21.15.3:0/3500013170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b01a0 0x7f93240a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 -- 172.21.15.3:0/3500013170 >> 172.21.15.3:0/3500013170 conn(0x7f9324060310 msgr2=0x7f93240b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:46.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 -- 172.21.15.3:0/3500013170 shutdown_connections 2022-07-21T01:40:46.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 -- 172.21.15.3:0/3500013170 wait complete. 2022-07-21T01:40:46.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 Processor -- start 2022-07-21T01:40:46.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 -- start start 2022-07-21T01:40:46.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93240afd60 0x7f9324143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:46.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240b01a0 0x7f932414c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:46.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b2710 0x7f932414c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:46.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9324156650 con 0x7f93240b01a0 2022-07-21T01:40:46.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932414caa0 con 0x7f93240afd60 2022-07-21T01:40:46.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93240b34b0 con 0x7f93240b2710 2022-07-21T01:40:46.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9328b79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240b01a0 0x7f932414c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:46.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9328b79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240b01a0 0x7f932414c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48516/0 (socket says 172.21.15.3:48516) 2022-07-21T01:40:46.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9328b79700 1 -- 172.21.15.3:0/3211777467 learned_addr learned my addr 172.21.15.3:0/3211777467 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:46.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9329b7b700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b2710 0x7f932414c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:46.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932937a700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93240afd60 0x7f9324143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:46.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9328b79700 1 -- 172.21.15.3:0/3211777467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b2710 msgr2=0x7f932414c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:46.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9328b79700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b2710 0x7f932414c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9328b79700 1 -- 172.21.15.3:0/3211777467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93240afd60 msgr2=0x7f9324143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:46.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9328b79700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93240afd60 0x7f9324143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9328b79700 1 -- 172.21.15.3:0/3211777467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c00b1e0 con 0x7f93240b01a0 2022-07-21T01:40:46.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f9328b79700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240b01a0 0x7f932414c0c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f932414cbe0 tx=0x7f93200091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:46.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f931a7fc700 1 -- 172.21.15.3:0/3211777467 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f932000c850 con 0x7f93240b01a0 2022-07-21T01:40:46.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f931a7fc700 1 -- 172.21.15.3:0/3211777467 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932000c9b0 con 0x7f93240b01a0 2022-07-21T01:40:46.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9324156840 con 0x7f93240b01a0 2022-07-21T01:40:46.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f931a7fc700 1 -- 172.21.15.3:0/3211777467 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f932000cd10 con 0x7f93240b01a0 2022-07-21T01:40:46.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.137+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9324144510 con 0x7f93240b01a0 2022-07-21T01:40:46.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.141+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93240574e0 con 0x7f93240b01a0 2022-07-21T01:40:46.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.141+0000 7f931a7fc700 1 -- 172.21.15.3:0/3211777467 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9320022070 con 0x7f93240b01a0 2022-07-21T01:40:46.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.141+0000 7f931a7fc700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f930c053b30 0x7f930c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:46.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.141+0000 7f932937a700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f930c053b30 0x7f930c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:46.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.141+0000 7f931a7fc700 1 -- 172.21.15.3:0/3211777467 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f932006c890 con 0x7f93240b01a0 2022-07-21T01:40:46.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.141+0000 7f932937a700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f930c053b30 0x7f930c055ff0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f931400b000 tx=0x7f9314009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:46.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.149+0000 7f931a7fc700 1 -- 172.21.15.3:0/3211777467 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9320043540 con 0x7f93240b01a0 2022-07-21T01:40:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.309+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9324069a70 con 0x7f93240b01a0 2022-07-21T01:40:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.309+0000 7f931a7fc700 1 -- 172.21.15.3:0/3211777467 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2484 (secure 0 0 0) 0x7f9320046b60 con 0x7f93240b01a0 2022-07-21T01:40:46.311 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:46.311 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/485774043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":485774043},{"type":"v1","addr":"172.21.15.143:6833","nonce":485774043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":6936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/756498280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":756498280},{"type":"v1","addr":"172.21.15.3:6835","nonce":756498280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:41.034617+0000","modified":"2022-07-21T01:40:42.719494+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6942},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6942":{"gid":6942,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1935398064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1935398064},{"type":"v1","addr":"172.21.15.3:6837","nonce":1935398064}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-21T01:40:46.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f930c053b30 msgr2=0x7f930c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:46.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f930c053b30 0x7f930c055ff0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f931400b000 tx=0x7f9314009040).stop 2022-07-21T01:40:46.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240b01a0 msgr2=0x7f932414c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:46.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240b01a0 0x7f932414c0c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f932414cbe0 tx=0x7f93200091d0).stop 2022-07-21T01:40:46.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 shutdown_connections 2022-07-21T01:40:46.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93240b2710 0x7f932414c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f93240b01a0 0x7f932414c0c0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f930c053b30 0x7f930c055ff0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 --2- 172.21.15.3:0/3211777467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93240afd60 0x7f9324143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:46.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 >> 172.21.15.3:0/3211777467 conn(0x7f9324060310 msgr2=0x7f93240af480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:46.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 shutdown_connections 2022-07-21T01:40:46.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:46.313+0000 7f932a37c700 1 -- 172.21.15.3:0/3211777467 wait complete. 2022-07-21T01:40:46.317 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 48 2022-07-21T01:40:46.628 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:40:41.034617+0000', 'modified': '2022-07-21T01:40:42.719494+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6942}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6942': {'gid': 6942, 'name': 'c', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/1935398064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 1935398064}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 1935398064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:40:46.628 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:40:46.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:40:46.750 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6942,"want_state":"up:active","state":"up:active","rank_uptime":5.7033238019999999,"mdsmap_epoch":48,"osdmap_epoch":101,"osdmap_epoch_barrier":101,"uptime":10.388759944} 2022-07-21T01:40:46.751 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6942, "mdsmap_epoch": 48, "osdmap_epoch": 101, "osdmap_epoch_barrier": 101, "rank_uptime": 5.703323802, "state": "up:active", "uptime": 10.388759944, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:40:46.752 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:40:46.752 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:40:46.752 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:40:46.753 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:46.753 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:46.762 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:40:46.778 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:46.779 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:46.839 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:46.840 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:46.841 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:46.841 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:40:46.846 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:40:46.847 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:46.849 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:40:46.866 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:46.866 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:46.923 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:46.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:46.925 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:40:46.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:40:46.921+0000 7fa21da00080 -1 init, newargv = 0x55fdbb94ff20 newargc=9 2022-07-21T01:40:46.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23034]: starting ceph client 2022-07-21T01:40:46.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23034]: starting fuse 2022-07-21T01:40:48.044 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:40:48.064 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:48.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:48.123 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:48.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:48.126 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:40:48.132 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:40:48.132 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:40:48.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:48.133 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:40:48.133 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:40:48.134 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:40:48.134 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:40:48.135 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:40:48.135 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:48.135 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:40:48.136 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:40:48.136 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:40:48.137 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:48.137 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:40:48.138 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:40:48.138 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:40:48.138 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:40:48.139 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:48.139 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:40:48.140 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:40:48.140 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:40:48.140 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:40:48.141 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:40:48.141 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:48.142 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:40:48.142 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:40:48.188 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.23034.asok 2022-07-21T01:40:48.191 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23034.asok 2022-07-21T01:40:48.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:48.193 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23034.asok status 2022-07-21T01:40:48.325 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:40:48.325 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:40:48.326 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:40:48.326 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:40:48.326 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:40:48.326 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:40:48.327 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:40:48.327 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "23034", 2022-07-21T01:40:48.327 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:40:48.327 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:48.328 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:40:48.328 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:40:48.328 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 7087, 2022-07-21T01:40:48.328 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:40:48.329 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:40:48.329 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:40:48.329 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 7087 2022-07-21T01:40:48.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:48.330 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:40:48.330 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:40:48.330 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:40:48.331 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 4252297510 2022-07-21T01:40:48.331 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:40:48.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:48.331 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:40:48.332 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:40:48.332 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:40:48.332 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 4252297510 2022-07-21T01:40:48.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:40:48.333 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.7087 172.21.15.171:0/4252297510", 2022-07-21T01:40:48.333 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/4252297510", 2022-07-21T01:40:48.333 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:40:48.333 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 48, 2022-07-21T01:40:48.334 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 101, 2022-07-21T01:40:48.334 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:40:48.334 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:40:48.334 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:40:48.335 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:40:48.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:48.337 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:48.337 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:48.344 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:40:48.345 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:48.346 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:48.346 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:48.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:40:48.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2e429700 1 Processor -- start 2022-07-21T01:40:48.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2e429700 1 -- start start 2022-07-21T01:40:48.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2e429700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a280b38a0 0x7f9a28064bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2e429700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a281470d0 0x7f9a281403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2e429700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a280b4f60 con 0x7f9a280647d0 2022-07-21T01:40:48.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2e429700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a280b4a60 con 0x7f9a281470d0 2022-07-21T01:40:48.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2e429700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a280b4cc0 con 0x7f9a280b38a0 2022-07-21T01:40:48.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2cc26700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a280b38a0 0x7f9a28064bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2d427700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f9a280647d0 0x7f9a281436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:58918/0 (socket says 172.21.15.3:58918) 2022-07-21T01:40:48.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2cc26700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a280b38a0 0x7f9a28064bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:55440/0 (socket says 172.21.15.3:55440) 2022-07-21T01:40:48.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2d427700 1 -- 172.21.15.3:0/3262677030 learned_addr learned my addr 172.21.15.3:0/3262677030 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:48.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2dc28700 1 --2- 172.21.15.3:0/3262677030 >> v2:172.21.15.143:3300/0 conn(0x7f9a281470d0 0x7f9a281403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a1ffff700 1 -- 172.21.15.3:0/3262677030 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298970707 0 0) 0x7f9a280b4f60 con 0x7f9a280647d0 2022-07-21T01:40:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2dc28700 1 -- 172.21.15.3:0/3262677030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a280b38a0 msgr2=0x7f9a28064bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2dc28700 1 --2- 172.21.15.3:0/3262677030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a280b38a0 0x7f9a28064bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2dc28700 1 -- 172.21.15.3:0/3262677030 >> v1:172.21.15.143:6789/0 conn(0x7f9a280647d0 legacy=0x7f9a281436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2dc28700 1 -- 172.21.15.3:0/3262677030 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a28143250 con 0x7f9a281470d0 2022-07-21T01:40:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.517+0000 7f9a2cc26700 1 --2- 172.21.15.3:0/3262677030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a280b38a0 0x7f9a28064bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:48.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2dc28700 1 --2- 172.21.15.3:0/3262677030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a281470d0 0x7f9a281403f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f9a200032f0 tx=0x7f9a200091d0).ready entity=mon.1 client_cookie=609a0f234248f473 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:48.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a1ffff700 1 -- 172.21.15.3:0/3262677030 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a2000b6c0 con 0x7f9a281470d0 2022-07-21T01:40:48.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a1ffff700 1 -- 172.21.15.3:0/3262677030 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2000b820 con 0x7f9a281470d0 2022-07-21T01:40:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a1ffff700 1 -- 172.21.15.3:0/3262677030 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a2000b990 con 0x7f9a281470d0 2022-07-21T01:40:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 -- 172.21.15.3:0/3262677030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a281470d0 msgr2=0x7f9a281403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 --2- 172.21.15.3:0/3262677030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a281470d0 0x7f9a281403f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f9a200032f0 tx=0x7f9a200091d0).stop 2022-07-21T01:40:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 -- 172.21.15.3:0/3262677030 shutdown_connections 2022-07-21T01:40:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 --2- 172.21.15.3:0/3262677030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a281470d0 0x7f9a281403f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 --2- 172.21.15.3:0/3262677030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a280b38a0 0x7f9a28064bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 -- 172.21.15.3:0/3262677030 >> 172.21.15.3:0/3262677030 conn(0x7f9a2805f840 msgr2=0x7f9a2805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 -- 172.21.15.3:0/3262677030 shutdown_connections 2022-07-21T01:40:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 -- 172.21.15.3:0/3262677030 wait complete. 2022-07-21T01:40:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 Processor -- start 2022-07-21T01:40:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.521+0000 7f9a2e429700 1 -- start start 2022-07-21T01:40:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2e429700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a280647d0 0x7f9a28155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2e429700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a280b38a0 0x7f9a280b5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2e429700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a281470d0 0x7f9a281456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2e429700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a280b64f0 con 0x7f9a280647d0 2022-07-21T01:40:48.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2e429700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a28144f40 con 0x7f9a280b38a0 2022-07-21T01:40:48.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2e429700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a28144a00 con 0x7f9a281470d0 2022-07-21T01:40:48.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2d427700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a280647d0 0x7f9a28155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2cc26700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a280b38a0 0x7f9a280b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2d427700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a280647d0 0x7f9a28155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48530/0 (socket says 172.21.15.3:48530) 2022-07-21T01:40:48.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2d427700 1 -- 172.21.15.3:0/4161505834 learned_addr learned my addr 172.21.15.3:0/4161505834 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:48.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2dc28700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a281470d0 0x7f9a281456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2d427700 1 -- 172.21.15.3:0/4161505834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a281470d0 msgr2=0x7f9a281456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2d427700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a281470d0 0x7f9a281456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2d427700 1 -- 172.21.15.3:0/4161505834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a280b38a0 msgr2=0x7f9a280b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2d427700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a280b38a0 0x7f9a280b5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2d427700 1 -- 172.21.15.3:0/4161505834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2400b180 con 0x7f9a280647d0 2022-07-21T01:40:48.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2d427700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a280647d0 0x7f9a28155910 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f9a18000ed0 tx=0x7f9a18007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:48.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a1e7fc700 1 -- 172.21.15.3:0/4161505834 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a180070d0 con 0x7f9a280647d0 2022-07-21T01:40:48.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a1e7fc700 1 -- 172.21.15.3:0/4161505834 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a18007230 con 0x7f9a280647d0 2022-07-21T01:40:48.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a1e7fc700 1 -- 172.21.15.3:0/4161505834 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a18005610 con 0x7f9a280647d0 2022-07-21T01:40:48.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a20009470 con 0x7f9a280647d0 2022-07-21T01:40:48.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.525+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a280b7a30 con 0x7f9a280647d0 2022-07-21T01:40:48.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.529+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a28144bc0 con 0x7f9a280647d0 2022-07-21T01:40:48.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.529+0000 7f9a1e7fc700 1 -- 172.21.15.3:0/4161505834 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9a18010070 con 0x7f9a280647d0 2022-07-21T01:40:48.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.529+0000 7f9a1e7fc700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9a10053950 0x7f9a10055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.529+0000 7f9a1e7fc700 1 -- 172.21.15.3:0/4161505834 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9a1806b6e0 con 0x7f9a280647d0 2022-07-21T01:40:48.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.529+0000 7f9a2cc26700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9a10053950 0x7f9a10055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.529+0000 7f9a2cc26700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9a10053950 0x7f9a10055e10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f9a2400b000 tx=0x7f9a24006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:48.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.533+0000 7f9a1e7fc700 1 -- 172.21.15.3:0/4161505834 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a28144bc0 con 0x7f9a280647d0 2022-07-21T01:40:48.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.665+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a28144bc0 con 0x7f9a280647d0 2022-07-21T01:40:48.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a1e7fc700 1 -- 172.21.15.3:0/4161505834 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16277 (secure 0 0 0) 0x7f9a28144bc0 con 0x7f9a280647d0 2022-07-21T01:40:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:48.670 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":101,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:40:42.037286+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:40:39.112065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-21T01:40:40.040729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:40:48.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9a10053950 msgr2=0x7f9a10055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9a10053950 0x7f9a10055e10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f9a2400b000 tx=0x7f9a24006df0).stop 2022-07-21T01:40:48.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a280647d0 msgr2=0x7f9a28155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a280647d0 0x7f9a28155910 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f9a18000ed0 tx=0x7f9a18007ac0).stop 2022-07-21T01:40:48.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 shutdown_connections 2022-07-21T01:40:48.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a281470d0 0x7f9a281456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9a10053950 0x7f9a10055e10 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a280b38a0 0x7f9a280b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 --2- 172.21.15.3:0/4161505834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a280647d0 0x7f9a28155910 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 >> 172.21.15.3:0/4161505834 conn(0x7f9a2805f840 msgr2=0x7f9a28142190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:48.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 shutdown_connections 2022-07-21T01:40:48.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.669+0000 7f9a2e429700 1 -- 172.21.15.3:0/4161505834 wait complete. 2022-07-21T01:40:48.683 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:40:39.112065+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:40:40.040729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:40:48.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:40:48.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4ac5d700 1 Processor -- start 2022-07-21T01:40:48.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4ac5d700 1 -- start start 2022-07-21T01:40:48.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4ac5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440b5ed0 0x7efc440b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4ac5d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc44061d90 con 0x7efc440b6ee0 2022-07-21T01:40:48.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4ac5d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc440ba5a0 con 0x7efc440619b0 2022-07-21T01:40:48.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4ac5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc44143fa0 con 0x7efc440b5ed0 2022-07-21T01:40:48.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4945a700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efc440b6ee0 0x7efc440b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56818/0 (socket says 172.21.15.3:56818) 2022-07-21T01:40:48.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4945a700 1 -- 172.21.15.3:0/957379217 learned_addr learned my addr 172.21.15.3:0/957379217 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:48.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4a45c700 1 --2- 172.21.15.3:0/957379217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440b5ed0 0x7efc440b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc48c59700 1 -- 172.21.15.3:0/957379217 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232213037 0 0) 0x7efc44061d90 con 0x7efc440b6ee0 2022-07-21T01:40:48.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc48c59700 1 -- 172.21.15.3:0/957379217 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c003260 con 0x7efc440b6ee0 2022-07-21T01:40:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc48c59700 1 -- 172.21.15.3:0/957379217 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423655743 0 0) 0x7efc440ba5a0 con 0x7efc440619b0 2022-07-21T01:40:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4a45c700 1 -- 172.21.15.3:0/957379217 >> v1:172.21.15.143:6789/0 conn(0x7efc440619b0 legacy=0x7efc44143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4a45c700 1 -- 172.21.15.3:0/957379217 >> v1:172.21.15.3:6789/0 conn(0x7efc440b6ee0 legacy=0x7efc440b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4a45c700 1 -- 172.21.15.3:0/957379217 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc44143150 con 0x7efc440b5ed0 2022-07-21T01:40:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc4a45c700 1 --2- 172.21.15.3:0/957379217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440b5ed0 0x7efc440b58a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7efc3c0032f0 tx=0x7efc3c0091d0).ready entity=mon.2 client_cookie=eb8be2542b442c07 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc48c59700 1 -- 172.21.15.3:0/957379217 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc3c00b6c0 con 0x7efc440b5ed0 2022-07-21T01:40:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc48c59700 1 -- 172.21.15.3:0/957379217 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c00b820 con 0x7efc440b5ed0 2022-07-21T01:40:48.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.797+0000 7efc48c59700 1 -- 172.21.15.3:0/957379217 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc3c00b990 con 0x7efc440b5ed0 2022-07-21T01:40:48.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.801+0000 7efc4ac5d700 1 -- 172.21.15.3:0/957379217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440b5ed0 msgr2=0x7efc440b58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.801+0000 7efc4ac5d700 1 --2- 172.21.15.3:0/957379217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440b5ed0 0x7efc440b58a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7efc3c0032f0 tx=0x7efc3c0091d0).stop 2022-07-21T01:40:48.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.801+0000 7efc4ac5d700 1 -- 172.21.15.3:0/957379217 shutdown_connections 2022-07-21T01:40:48.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.801+0000 7efc4ac5d700 1 --2- 172.21.15.3:0/957379217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440b5ed0 0x7efc440b58a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.801+0000 7efc4ac5d700 1 -- 172.21.15.3:0/957379217 >> 172.21.15.3:0/957379217 conn(0x7efc4406af80 msgr2=0x7efc4405fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:48.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.801+0000 7efc4ac5d700 1 -- 172.21.15.3:0/957379217 shutdown_connections 2022-07-21T01:40:48.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.801+0000 7efc4ac5d700 1 -- 172.21.15.3:0/957379217 wait complete. 2022-07-21T01:40:48.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.801+0000 7efc4ac5d700 1 Processor -- start 2022-07-21T01:40:48.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.801+0000 7efc4ac5d700 1 -- start start 2022-07-21T01:40:48.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4ac5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440619b0 0x7efc44113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4ac5d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc440b5ed0 0x7efc44113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4ac5d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc440b6ee0 0x7efc441457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4ac5d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc44113d50 con 0x7efc440b5ed0 2022-07-21T01:40:48.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4ac5d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc441533c0 con 0x7efc440b6ee0 2022-07-21T01:40:48.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4ac5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc44152e80 con 0x7efc440619b0 2022-07-21T01:40:48.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4945a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc440b5ed0 0x7efc44113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4945a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc440b5ed0 0x7efc44113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48544/0 (socket says 172.21.15.3:48544) 2022-07-21T01:40:48.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4945a700 1 -- 172.21.15.3:0/2508390632 learned_addr learned my addr 172.21.15.3:0/2508390632 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:48.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4a45c700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc440b6ee0 0x7efc441457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc49c5b700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440619b0 0x7efc44113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4945a700 1 -- 172.21.15.3:0/2508390632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440619b0 msgr2=0x7efc44113370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4945a700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440619b0 0x7efc44113370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4945a700 1 -- 172.21.15.3:0/2508390632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc440b6ee0 msgr2=0x7efc441457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4945a700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc440b6ee0 0x7efc441457a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4945a700 1 -- 172.21.15.3:0/2508390632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c009470 con 0x7efc440b5ed0 2022-07-21T01:40:48.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4945a700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc440b5ed0 0x7efc44113860 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7efc34001110 tx=0x7efc34007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:48.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc32ffd700 1 -- 172.21.15.3:0/2508390632 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc3400cdd0 con 0x7efc440b5ed0 2022-07-21T01:40:48.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc32ffd700 1 -- 172.21.15.3:0/2508390632 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc340070d0 con 0x7efc440b5ed0 2022-07-21T01:40:48.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc44153120 con 0x7efc440b5ed0 2022-07-21T01:40:48.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc32ffd700 1 -- 172.21.15.3:0/2508390632 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc34007430 con 0x7efc440b5ed0 2022-07-21T01:40:48.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.805+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc441451a0 con 0x7efc440b5ed0 2022-07-21T01:40:48.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.809+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc440574e0 con 0x7efc440b5ed0 2022-07-21T01:40:48.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.809+0000 7efc32ffd700 1 -- 172.21.15.3:0/2508390632 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efc3401a060 con 0x7efc440b5ed0 2022-07-21T01:40:48.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.809+0000 7efc32ffd700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc24053aa0 0x7efc24055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:48.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.809+0000 7efc32ffd700 1 -- 172.21.15.3:0/2508390632 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7efc3406b730 con 0x7efc440b5ed0 2022-07-21T01:40:48.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.809+0000 7efc49c5b700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc24053aa0 0x7efc24055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:48.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.809+0000 7efc49c5b700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc24053aa0 0x7efc24055f60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7efc4000d000 tx=0x7efc400079b0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:48.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.813+0000 7efc32ffd700 1 -- 172.21.15.3:0/2508390632 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc340423e0 con 0x7efc440b5ed0 2022-07-21T01:40:48.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.941+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7efc440ba580 con 0x7efc440b5ed0 2022-07-21T01:40:48.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.941+0000 7efc32ffd700 1 -- 172.21.15.3:0/2508390632 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v101) v1 ==== 110+0+10 (secure 0 0 0) 0x7efc34018070 con 0x7efc440b5ed0 2022-07-21T01:40:48.943 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:40:48.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc24053aa0 msgr2=0x7efc24055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc24053aa0 0x7efc24055f60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7efc4000d000 tx=0x7efc400079b0).stop 2022-07-21T01:40:48.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc440b5ed0 msgr2=0x7efc44113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:48.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc440b5ed0 0x7efc44113860 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7efc34001110 tx=0x7efc34007a80).stop 2022-07-21T01:40:48.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 shutdown_connections 2022-07-21T01:40:48.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efc24053aa0 0x7efc24055f60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc440b6ee0 0x7efc441457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc440b5ed0 0x7efc44113860 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 --2- 172.21.15.3:0/2508390632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc440619b0 0x7efc44113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:48.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 >> 172.21.15.3:0/2508390632 conn(0x7efc4406af80 msgr2=0x7efc440b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:48.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 shutdown_connections 2022-07-21T01:40:48.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:48.945+0000 7efc4ac5d700 1 -- 172.21.15.3:0/2508390632 wait complete. 2022-07-21T01:40:48.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:40:49.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.053+0000 7f50409cd700 1 Processor -- start 2022-07-21T01:40:49.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.053+0000 7f50409cd700 1 -- start start 2022-07-21T01:40:49.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f50409cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f50409cd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c143ab0 con 0x7f503c0b3260 2022-07-21T01:40:49.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f50409cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0b0c80 con 0x7f503c0ad000 2022-07-21T01:40:49.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f50409cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0ae810 con 0x7f503c0b08a0 2022-07-21T01:40:49.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503affd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f503c0b3260 0x7f503c1433a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56832/0 (socket says 172.21.15.3:56832) 2022-07-21T01:40:49.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503affd700 1 -- 172.21.15.3:0/2143718085 learned_addr learned my addr 172.21.15.3:0/2143718085 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:49.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503b7fe700 1 --2- 172.21.15.3:0/2143718085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503a7fc700 1 -- 172.21.15.3:0/2143718085 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190244516 0 0) 0x7f503c143ab0 con 0x7f503c0b3260 2022-07-21T01:40:49.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503a7fc700 1 -- 172.21.15.3:0/2143718085 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5028003260 con 0x7f503c0b3260 2022-07-21T01:40:49.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503a7fc700 1 -- 172.21.15.3:0/2143718085 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833343780 0 0) 0x7f503c0b0c80 con 0x7f503c0ad000 2022-07-21T01:40:49.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503a7fc700 1 -- 172.21.15.3:0/2143718085 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c143ab0 con 0x7f503c0ad000 2022-07-21T01:40:49.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503b7fe700 1 -- 172.21.15.3:0/2143718085 >> v1:172.21.15.143:6789/0 conn(0x7f503c0ad000 legacy=0x7f503c13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503b7fe700 1 -- 172.21.15.3:0/2143718085 >> v1:172.21.15.3:6789/0 conn(0x7f503c0b3260 legacy=0x7f503c1433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503b7fe700 1 -- 172.21.15.3:0/2143718085 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c0aeab0 con 0x7f503c0b08a0 2022-07-21T01:40:49.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503b7fe700 1 --2- 172.21.15.3:0/2143718085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c142eb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f502c002f70 tx=0x7f502c006e80).ready entity=mon.2 client_cookie=c32cce3cd87febad server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503a7fc700 1 -- 172.21.15.3:0/2143718085 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f502c010400 con 0x7f503c0b08a0 2022-07-21T01:40:49.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503a7fc700 1 -- 172.21.15.3:0/2143718085 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502c000e90 con 0x7f503c0b08a0 2022-07-21T01:40:49.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.057+0000 7f503a7fc700 1 -- 172.21.15.3:0/2143718085 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f502c010750 con 0x7f503c0b08a0 2022-07-21T01:40:49.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 -- 172.21.15.3:0/2143718085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 msgr2=0x7f503c142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 --2- 172.21.15.3:0/2143718085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c142eb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f502c002f70 tx=0x7f502c006e80).stop 2022-07-21T01:40:49.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 -- 172.21.15.3:0/2143718085 shutdown_connections 2022-07-21T01:40:49.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 --2- 172.21.15.3:0/2143718085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c142eb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 -- 172.21.15.3:0/2143718085 >> 172.21.15.3:0/2143718085 conn(0x7f503c0612d0 msgr2=0x7f503c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:49.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 -- 172.21.15.3:0/2143718085 shutdown_connections 2022-07-21T01:40:49.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 -- 172.21.15.3:0/2143718085 wait complete. 2022-07-21T01:40:49.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 Processor -- start 2022-07-21T01:40:49.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 -- start start 2022-07-21T01:40:49.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f503c0ad000 0x7f503c09ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c064b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f503c0b3260 0x7f503c14ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c14e360 con 0x7f503c0ad000 2022-07-21T01:40:49.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c141170 con 0x7f503c0b3260 2022-07-21T01:40:49.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.061+0000 7f50409cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c1517a0 con 0x7f503c0b08a0 2022-07-21T01:40:49.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f503c0ad000 0x7f503c09ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f503c0ad000 0x7f503c09ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48556/0 (socket says 172.21.15.3:48556) 2022-07-21T01:40:49.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503b7fe700 1 -- 172.21.15.3:0/2872590197 learned_addr learned my addr 172.21.15.3:0/2872590197 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:49.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503bfff700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f503c0b3260 0x7f503c14ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503affd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c064b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503b7fe700 1 -- 172.21.15.3:0/2872590197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 msgr2=0x7f503c064b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503b7fe700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c064b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503b7fe700 1 -- 172.21.15.3:0/2872590197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f503c0b3260 msgr2=0x7f503c14ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503b7fe700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f503c0b3260 0x7f503c14ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503b7fe700 1 -- 172.21.15.3:0/2872590197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502c00c310 con 0x7f503c0ad000 2022-07-21T01:40:49.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503affd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c064b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:40:49.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503b7fe700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f503c0ad000 0x7f503c09ef50 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f502c002f70 tx=0x7f502c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f5038ff9700 1 -- 172.21.15.3:0/2872590197 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f502c00f920 con 0x7f503c0ad000 2022-07-21T01:40:49.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f5038ff9700 1 -- 172.21.15.3:0/2872590197 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502c00fa80 con 0x7f503c0ad000 2022-07-21T01:40:49.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c14e590 con 0x7f503c0ad000 2022-07-21T01:40:49.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f5038ff9700 1 -- 172.21.15.3:0/2872590197 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f502c00fde0 con 0x7f503c0ad000 2022-07-21T01:40:49.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c065990 con 0x7f503c0ad000 2022-07-21T01:40:49.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c14e720 con 0x7f503c0ad000 2022-07-21T01:40:49.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f5038ff9700 1 -- 172.21.15.3:0/2872590197 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f502c026030 con 0x7f503c0ad000 2022-07-21T01:40:49.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f5038ff9700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f501c053aa0 0x7f501c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f5038ff9700 1 -- 172.21.15.3:0/2872590197 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f502c06c8a0 con 0x7f503c0ad000 2022-07-21T01:40:49.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.065+0000 7f503affd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f501c053aa0 0x7f501c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.069+0000 7f503affd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f501c053aa0 0x7f501c055f60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f5030008000 tx=0x7f5030009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.073+0000 7f5038ff9700 1 -- 172.21.15.3:0/2872590197 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f502c043660 con 0x7f503c0ad000 2022-07-21T01:40:49.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.189+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f503c0574e0 con 0x7f503c0ad000 2022-07-21T01:40:49.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.189+0000 7f5038ff9700 1 -- 172.21.15.3:0/2872590197 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v101) v1 ==== 92+0+10 (secure 0 0 0) 0x7f502c0240a0 con 0x7f503c0ad000 2022-07-21T01:40:49.194 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:49.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f501c053aa0 msgr2=0x7f501c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f501c053aa0 0x7f501c055f60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f5030008000 tx=0x7f5030009040).stop 2022-07-21T01:40:49.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f503c0ad000 msgr2=0x7f503c09ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f503c0ad000 0x7f503c09ef50 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f502c002f70 tx=0x7f502c015000).stop 2022-07-21T01:40:49.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 shutdown_connections 2022-07-21T01:40:49.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f501c053aa0 0x7f501c055f60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f503c0b3260 0x7f503c14ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f503c0b08a0 0x7f503c064b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 --2- 172.21.15.3:0/2872590197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f503c0ad000 0x7f503c09ef50 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 >> 172.21.15.3:0/2872590197 conn(0x7f503c0612d0 msgr2=0x7f503c141520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:49.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 shutdown_connections 2022-07-21T01:40:49.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.193+0000 7f50409cd700 1 -- 172.21.15.3:0/2872590197 wait complete. 2022-07-21T01:40:49.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:40:49.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 Processor -- start 2022-07-21T01:40:49.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 -- start start 2022-07-21T01:40:49.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2e100b00a0 0x7f2e1005f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e100acb40 0x7f2e1014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e100614f0 con 0x7f2e1014ad10 2022-07-21T01:40:49.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1014b770 con 0x7f2e100b00a0 2022-07-21T01:40:49.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1014ba10 con 0x7f2e100acb40 2022-07-21T01:40:49.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e14880700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2e100b00a0 0x7f2e1005f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e15081700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2e1014ad10 0x7f2e100af350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56844/0 (socket says 172.21.15.3:56844) 2022-07-21T01:40:49.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e15081700 1 -- 172.21.15.3:0/621460142 learned_addr learned my addr 172.21.15.3:0/621460142 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:49.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e15882700 1 --2- 172.21.15.3:0/621460142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e100acb40 0x7f2e1014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e14880700 1 -- 172.21.15.3:0/621460142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e100acb40 msgr2=0x7f2e1014b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:49.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e14880700 1 --2- 172.21.15.3:0/621460142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e100acb40 0x7f2e1014b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e14880700 1 -- 172.21.15.3:0/621460142 >> v1:172.21.15.3:6789/0 conn(0x7f2e1014ad10 legacy=0x7f2e100af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e14880700 1 -- 172.21.15.3:0/621460142 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e100aeeb0 con 0x7f2e100b00a0 2022-07-21T01:40:49.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e07fff700 1 -- 172.21.15.3:0/621460142 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748251395 0 0) 0x7f2e100614f0 con 0x7f2e1014ad10 2022-07-21T01:40:49.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e14880700 1 --2- 172.21.15.3:0/621460142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 0x7f2e1005f840 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f2e0c002f70 tx=0x7f2e0c006df0).ready entity=mon.0 client_cookie=ce0ad66d9adf67fd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e07fff700 1 -- 172.21.15.3:0/621460142 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e0c00d570 con 0x7f2e100b00a0 2022-07-21T01:40:49.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e07fff700 1 -- 172.21.15.3:0/621460142 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c000e90 con 0x7f2e100b00a0 2022-07-21T01:40:49.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e07fff700 1 -- 172.21.15.3:0/621460142 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e0c00d8c0 con 0x7f2e100b00a0 2022-07-21T01:40:49.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 -- 172.21.15.3:0/621460142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 msgr2=0x7f2e1005f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 --2- 172.21.15.3:0/621460142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 0x7f2e1005f840 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f2e0c002f70 tx=0x7f2e0c006df0).stop 2022-07-21T01:40:49.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 -- 172.21.15.3:0/621460142 shutdown_connections 2022-07-21T01:40:49.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 --2- 172.21.15.3:0/621460142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e100acb40 0x7f2e1014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 --2- 172.21.15.3:0/621460142 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 0x7f2e1005f840 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.325+0000 7f2e172e5700 1 -- 172.21.15.3:0/621460142 >> 172.21.15.3:0/621460142 conn(0x7f2e100b5700 msgr2=0x7f2e100b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:49.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 -- 172.21.15.3:0/621460142 shutdown_connections 2022-07-21T01:40:49.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 -- 172.21.15.3:0/621460142 wait complete. 2022-07-21T01:40:49.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 Processor -- start 2022-07-21T01:40:49.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 -- start start 2022-07-21T01:40:49.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e100acb40 0x7f2e10146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 0x7f2e10146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e1014ad10 0x7f2e1013b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e10146a10 con 0x7f2e100b00a0 2022-07-21T01:40:49.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e10143df0 con 0x7f2e100acb40 2022-07-21T01:40:49.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1013f9d0 con 0x7f2e1014ad10 2022-07-21T01:40:49.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e14880700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 0x7f2e10146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e14880700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 0x7f2e10146370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48572/0 (socket says 172.21.15.3:48572) 2022-07-21T01:40:49.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e14880700 1 -- 172.21.15.3:0/2205107519 learned_addr learned my addr 172.21.15.3:0/2205107519 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:49.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e15882700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e1014ad10 0x7f2e1013b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e15081700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e100acb40 0x7f2e10146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e14880700 1 -- 172.21.15.3:0/2205107519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e1014ad10 msgr2=0x7f2e1013b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e14880700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e1014ad10 0x7f2e1013b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e14880700 1 -- 172.21.15.3:0/2205107519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e100acb40 msgr2=0x7f2e10146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e14880700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e100acb40 0x7f2e10146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e14880700 1 -- 172.21.15.3:0/2205107519 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c00b180 con 0x7f2e100b00a0 2022-07-21T01:40:49.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e14880700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 0x7f2e10146370 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f2e0c002f70 tx=0x7f2e0c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e067fc700 1 -- 172.21.15.3:0/2205107519 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e0c00d8f0 con 0x7f2e100b00a0 2022-07-21T01:40:49.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e067fc700 1 -- 172.21.15.3:0/2205107519 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c011b80 con 0x7f2e100b00a0 2022-07-21T01:40:49.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.329+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1013fc70 con 0x7f2e100b00a0 2022-07-21T01:40:49.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.333+0000 7f2e067fc700 1 -- 172.21.15.3:0/2205107519 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e0c01e460 con 0x7f2e100b00a0 2022-07-21T01:40:49.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.333+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e10141ca0 con 0x7f2e100b00a0 2022-07-21T01:40:49.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.333+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e10140110 con 0x7f2e100b00a0 2022-07-21T01:40:49.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.333+0000 7f2e067fc700 1 -- 172.21.15.3:0/2205107519 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2e0c025030 con 0x7f2e100b00a0 2022-07-21T01:40:49.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.333+0000 7f2e067fc700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df80539e0 0x7f2df8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.333+0000 7f2e067fc700 1 -- 172.21.15.3:0/2205107519 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2e0c06d320 con 0x7f2e100b00a0 2022-07-21T01:40:49.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.333+0000 7f2e15081700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df80539e0 0x7f2df8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.333+0000 7f2e15081700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df80539e0 0x7f2df8055ea0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f2e0000d000 tx=0x7f2e00007c20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.341+0000 7f2e067fc700 1 -- 172.21.15.3:0/2205107519 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2e0c044010 con 0x7f2e100b00a0 2022-07-21T01:40:49.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.469+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2e100afa60 con 0x7f2e100b00a0 2022-07-21T01:40:49.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.469+0000 7f2e067fc700 1 -- 172.21.15.3:0/2205107519 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2e0c0230a0 con 0x7f2e100b00a0 2022-07-21T01:40:49.471 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:49.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df80539e0 msgr2=0x7f2df8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df80539e0 0x7f2df8055ea0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f2e0000d000 tx=0x7f2e00007c20).stop 2022-07-21T01:40:49.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 msgr2=0x7f2e10146370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 0x7f2e10146370 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f2e0c002f70 tx=0x7f2e0c015000).stop 2022-07-21T01:40:49.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 shutdown_connections 2022-07-21T01:40:49.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e1014ad10 0x7f2e1013b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e100b00a0 0x7f2e10146370 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2df80539e0 0x7f2df8055ea0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 --2- 172.21.15.3:0/2205107519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e100acb40 0x7f2e10146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 >> 172.21.15.3:0/2205107519 conn(0x7f2e100b5700 msgr2=0x7f2e10148eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:49.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 shutdown_connections 2022-07-21T01:40:49.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.473+0000 7f2e172e5700 1 -- 172.21.15.3:0/2205107519 wait complete. 2022-07-21T01:40:49.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:40:49.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55d24f2700 1 Processor -- start 2022-07-21T01:40:49.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55d24f2700 1 -- start start 2022-07-21T01:40:49.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55d24f2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55cc0ad4b0 0x7f55cc0af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55d24f2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc0aff30 con 0x7f55cc0af450 2022-07-21T01:40:49.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55d24f2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc05c7a0 con 0x7f55cc0ad9e0 2022-07-21T01:40:49.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55d24f2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc140990 con 0x7f55cc0ad4b0 2022-07-21T01:40:49.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55d14f0700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f55cc0af450 0x7f55cc1432e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56858/0 (socket says 172.21.15.3:56858) 2022-07-21T01:40:49.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55d14f0700 1 -- 172.21.15.3:0/3858964953 learned_addr learned my addr 172.21.15.3:0/3858964953 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:49.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55d0cef700 1 --2- 172.21.15.3:0/3858964953 >> v2:172.21.15.143:3300/0 conn(0x7f55cc0ad4b0 0x7f55cc0af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319103007 0 0) 0x7f55cc0aff30 con 0x7f55cc0af450 2022-07-21T01:40:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b4003260 con 0x7f55cc0af450 2022-07-21T01:40:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3841223503 0 0) 0x7f55b4003260 con 0x7f55cc0af450 2022-07-21T01:40:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 >> v2:172.21.15.143:3300/0 conn(0x7f55cc0ad4b0 msgr2=0x7f55cc0af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 --2- 172.21.15.3:0/3858964953 >> v2:172.21.15.143:3300/0 conn(0x7f55cc0ad4b0 0x7f55cc0af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 >> v1:172.21.15.143:6789/0 conn(0x7f55cc0ad9e0 legacy=0x7f55cc0a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc1439f0 con 0x7f55cc0af450 2022-07-21T01:40:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f55c80019e0 con 0x7f55cc0af450 2022-07-21T01:40:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 >> v1:172.21.15.3:6789/0 conn(0x7f55cc0af450 legacy=0x7f55cc1432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55b4003780 0x7f55b4003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55b4004500 0x7f55b4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b4004e70 0x7f55b400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc140990 con 0x7f55b4004500 2022-07-21T01:40:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b400bd20 con 0x7f55b4004e70 2022-07-21T01:40:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.601+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b400bee0 con 0x7f55b4003780 2022-07-21T01:40:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d0cef700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55b4004500 0x7f55b4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d14f0700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55b4003780 0x7f55b4003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d1cf1700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b4004e70 0x7f55b400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d0cef700 1 -- 172.21.15.3:0/3858964953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55b4003780 msgr2=0x7f55b4003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:49.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d0cef700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55b4003780 0x7f55b4003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d0cef700 1 -- 172.21.15.3:0/3858964953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b4004e70 msgr2=0x7f55b400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:49.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d0cef700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b4004e70 0x7f55b400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d1cf1700 1 -- 172.21.15.3:0/3858964953 reap_dead start 2022-07-21T01:40:49.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d0cef700 1 -- 172.21.15.3:0/3858964953 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b4003260 con 0x7f55b4004500 2022-07-21T01:40:49.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d14f0700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55b4003780 0x7f55b4003ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:49.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d0cef700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55b4004500 0x7f55b4004980 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f55c000c000 tx=0x7f55c0006df0).ready entity=mon.0 client_cookie=48e160b3e5a36163 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55c000e830 con 0x7f55b4004500 2022-07-21T01:40:49.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c000e990 con 0x7f55b4004500 2022-07-21T01:40:49.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55bffff700 1 -- 172.21.15.3:0/3858964953 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55c000ecf0 con 0x7f55b4004500 2022-07-21T01:40:49.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d24f2700 1 -- 172.21.15.3:0/3858964953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55b4004500 msgr2=0x7f55b4004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d24f2700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55b4004500 0x7f55b4004980 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f55c000c000 tx=0x7f55c0006df0).stop 2022-07-21T01:40:49.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d24f2700 1 -- 172.21.15.3:0/3858964953 shutdown_connections 2022-07-21T01:40:49.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d24f2700 1 --2- 172.21.15.3:0/3858964953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55b4004500 0x7f55b4004980 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d24f2700 1 -- 172.21.15.3:0/3858964953 >> 172.21.15.3:0/3858964953 conn(0x7f55cc0612d0 msgr2=0x7f55cc062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:49.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d24f2700 1 -- 172.21.15.3:0/3858964953 shutdown_connections 2022-07-21T01:40:49.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.605+0000 7f55d24f2700 1 -- 172.21.15.3:0/3858964953 wait complete. 2022-07-21T01:40:49.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 Processor -- start 2022-07-21T01:40:49.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 -- start start 2022-07-21T01:40:49.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b4004500 0x7f55cc142b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55cc137e70 0x7f55cc1382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc14f4a0 0x7f55cc14f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0abdc0 con 0x7f55cc14f4a0 2022-07-21T01:40:49.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0ac060 con 0x7f55b4004500 2022-07-21T01:40:49.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0ac300 con 0x7f55cc137e70 2022-07-21T01:40:49.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d1cf1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc14f4a0 0x7f55cc14f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d1cf1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc14f4a0 0x7f55cc14f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48592/0 (socket says 172.21.15.3:48592) 2022-07-21T01:40:49.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d1cf1700 1 -- 172.21.15.3:0/3048893842 learned_addr learned my addr 172.21.15.3:0/3048893842 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:49.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d14f0700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b4004500 0x7f55cc142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d0cef700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55cc137e70 0x7f55cc1382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d1cf1700 1 -- 172.21.15.3:0/3048893842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55cc137e70 msgr2=0x7f55cc1382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d1cf1700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55cc137e70 0x7f55cc1382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d1cf1700 1 -- 172.21.15.3:0/3048893842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b4004500 msgr2=0x7f55cc142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d1cf1700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b4004500 0x7f55cc142b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d1cf1700 1 -- 172.21.15.3:0/3048893842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc0ac5a0 con 0x7f55cc14f4a0 2022-07-21T01:40:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d1cf1700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc14f4a0 0x7f55cc14f950 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f55c8001c40 tx=0x7f55c4007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55be7fc700 1 -- 172.21.15.3:0/3048893842 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55c40133f0 con 0x7f55cc14f4a0 2022-07-21T01:40:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55be7fc700 1 -- 172.21.15.3:0/3048893842 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c4013550 con 0x7f55cc14f4a0 2022-07-21T01:40:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55cc1387e0 con 0x7f55cc14f4a0 2022-07-21T01:40:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55be7fc700 1 -- 172.21.15.3:0/3048893842 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55c40138b0 con 0x7f55cc14f4a0 2022-07-21T01:40:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.609+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc0b6130 con 0x7f55cc14f4a0 2022-07-21T01:40:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.613+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc0574e0 con 0x7f55cc14f4a0 2022-07-21T01:40:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.613+0000 7f55be7fc700 1 -- 172.21.15.3:0/3048893842 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f55c401b360 con 0x7f55cc14f4a0 2022-07-21T01:40:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.613+0000 7f55be7fc700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55ac053a70 0x7f55ac055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.613+0000 7f55be7fc700 1 -- 172.21.15.3:0/3048893842 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f55c406b740 con 0x7f55cc14f4a0 2022-07-21T01:40:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.613+0000 7f55d14f0700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55ac053a70 0x7f55ac055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.613+0000 7f55d14f0700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55ac053a70 0x7f55ac055f30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f55c8005000 tx=0x7f55cc061990).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.621+0000 7f55be7fc700 1 -- 172.21.15.3:0/3048893842 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f55c4042390 con 0x7f55cc14f4a0 2022-07-21T01:40:49.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.749+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55cc061b20 con 0x7f55cc14f4a0 2022-07-21T01:40:49.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55be7fc700 1 -- 172.21.15.3:0/3048893842 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+10 (secure 0 0 0) 0x7f55c40459b0 con 0x7f55cc14f4a0 2022-07-21T01:40:49.753 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:40:49.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55ac053a70 msgr2=0x7f55ac055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55ac053a70 0x7f55ac055f30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f55c8005000 tx=0x7f55cc061990).stop 2022-07-21T01:40:49.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc14f4a0 msgr2=0x7f55cc14f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc14f4a0 0x7f55cc14f950 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f55c8001c40 tx=0x7f55c4007a00).stop 2022-07-21T01:40:49.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 shutdown_connections 2022-07-21T01:40:49.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc14f4a0 0x7f55cc14f950 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55cc137e70 0x7f55cc1382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55ac053a70 0x7f55ac055f30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 --2- 172.21.15.3:0/3048893842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b4004500 0x7f55cc142b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 >> 172.21.15.3:0/3048893842 conn(0x7f55cc0612d0 msgr2=0x7f55cc13f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:49.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 shutdown_connections 2022-07-21T01:40:49.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.753+0000 7f55d24f2700 1 -- 172.21.15.3:0/3048893842 wait complete. 2022-07-21T01:40:49.774 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:40:49.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:49.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.877+0000 7f2b1df21700 1 Processor -- start 2022-07-21T01:40:49.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1df21700 1 -- start start 2022-07-21T01:40:49.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1df21700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2b180b13f0 0x7f2b180afd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1df21700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b180adf10 con 0x7f2b180b1920 2022-07-21T01:40:49.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1df21700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b180ae1b0 con 0x7f2b180b2710 2022-07-21T01:40:49.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1df21700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b180a9520 con 0x7f2b180b13f0 2022-07-21T01:40:49.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1d720700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2b180b1920 0x7f2b1814a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56882/0 (socket says 172.21.15.3:56882) 2022-07-21T01:40:49.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1d720700 1 -- 172.21.15.3:0/2126738579 learned_addr learned my addr 172.21.15.3:0/2126738579 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:49.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b17fff700 1 --2- 172.21.15.3:0/2126738579 >> v2:172.21.15.143:3300/0 conn(0x7f2b180b13f0 0x7f2b180afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229072616 0 0) 0x7f2b180adf10 con 0x7f2b180b1920 2022-07-21T01:40:49.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2afc003260 con 0x7f2b180b1920 2022-07-21T01:40:49.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4041061729 0 0) 0x7f2afc003260 con 0x7f2b180b1920 2022-07-21T01:40:49.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 >> v2:172.21.15.143:3300/0 conn(0x7f2b180b13f0 msgr2=0x7f2b180afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:49.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 --2- 172.21.15.3:0/2126738579 >> v2:172.21.15.143:3300/0 conn(0x7f2b180b13f0 0x7f2b180afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 >> v1:172.21.15.143:6789/0 conn(0x7f2b180b2710 legacy=0x7f2b180abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b180a99c0 con 0x7f2b180b1920 2022-07-21T01:40:49.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f2b100019e0 con 0x7f2b180b1920 2022-07-21T01:40:49.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 >> v1:172.21.15.3:6789/0 conn(0x7f2b180b1920 legacy=0x7f2b1814a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2afc003750 0x7f2afc003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2afc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2afc004e70 0x7f2afc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180a9520 con 0x7f2afc004500 2022-07-21T01:40:49.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc00bc70 con 0x7f2afc004e70 2022-07-21T01:40:49.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc00be30 con 0x7f2afc003750 2022-07-21T01:40:49.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b17fff700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2afc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1cf1f700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2afc003750 0x7f2afc003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1d720700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2afc004e70 0x7f2afc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b17fff700 1 -- 172.21.15.3:0/2126738579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2afc003750 msgr2=0x7f2afc003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:49.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b17fff700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2afc003750 0x7f2afc003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b17fff700 1 -- 172.21.15.3:0/2126738579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2afc004e70 msgr2=0x7f2afc00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:49.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b17fff700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2afc004e70 0x7f2afc00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b1d720700 1 -- 172.21.15.3:0/2126738579 reap_dead start 2022-07-21T01:40:49.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b17fff700 1 -- 172.21.15.3:0/2126738579 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc003260 con 0x7f2afc004500 2022-07-21T01:40:49.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.881+0000 7f2b17fff700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2afc004980 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7f2b0400c000 tx=0x7f2b04006df0).ready entity=mon.0 client_cookie=aab44fc99cf5e909 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0400e5f0 con 0x7f2afc004500 2022-07-21T01:40:49.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0400e750 con 0x7f2afc004500 2022-07-21T01:40:49.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b177fe700 1 -- 172.21.15.3:0/2126738579 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0400eab0 con 0x7f2afc004500 2022-07-21T01:40:49.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b1df21700 1 -- 172.21.15.3:0/2126738579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 msgr2=0x7f2afc004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b1df21700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2afc004980 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7f2b0400c000 tx=0x7f2b04006df0).stop 2022-07-21T01:40:49.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b1df21700 1 -- 172.21.15.3:0/2126738579 shutdown_connections 2022-07-21T01:40:49.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b1df21700 1 --2- 172.21.15.3:0/2126738579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2afc004980 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b1df21700 1 -- 172.21.15.3:0/2126738579 >> 172.21.15.3:0/2126738579 conn(0x7f2b18060310 msgr2=0x7f2b180b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:49.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b1df21700 1 -- 172.21.15.3:0/2126738579 shutdown_connections 2022-07-21T01:40:49.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b1df21700 1 -- 172.21.15.3:0/2126738579 wait complete. 2022-07-21T01:40:49.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b1df21700 1 Processor -- start 2022-07-21T01:40:49.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.885+0000 7f2b1df21700 1 -- start start 2022-07-21T01:40:49.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1df21700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2b180ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1df21700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1805c620 0x7f2b1813e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1df21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1805ca80 0x7f2b1809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1df21700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180a03f0 con 0x7f2afc004500 2022-07-21T01:40:49.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1df21700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b181524e0 con 0x7f2b1805c620 2022-07-21T01:40:49.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1df21700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b18152780 con 0x7f2b1805ca80 2022-07-21T01:40:49.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1cf1f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2b180ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1cf1f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2b180ac130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48610/0 (socket says 172.21.15.3:48610) 2022-07-21T01:40:49.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1cf1f700 1 -- 172.21.15.3:0/2370907920 learned_addr learned my addr 172.21.15.3:0/2370907920 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:49.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b17fff700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1805c620 0x7f2b1813e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1d720700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1805ca80 0x7f2b1809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1cf1f700 1 -- 172.21.15.3:0/2370907920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1805ca80 msgr2=0x7f2b1809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1cf1f700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1805ca80 0x7f2b1809ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1cf1f700 1 -- 172.21.15.3:0/2370907920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1805c620 msgr2=0x7f2b1813e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:49.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1cf1f700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1805c620 0x7f2b1813e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:49.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1cf1f700 1 -- 172.21.15.3:0/2370907920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18152a20 con 0x7f2afc004500 2022-07-21T01:40:49.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1cf1f700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2b180ac130 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f2b0c00d000 tx=0x7f2b0c007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b15ffb700 1 -- 172.21.15.3:0/2370907920 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0c002e10 con 0x7f2afc004500 2022-07-21T01:40:49.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b15ffb700 1 -- 172.21.15.3:0/2370907920 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0c0114c0 con 0x7f2afc004500 2022-07-21T01:40:49.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b15ffb700 1 -- 172.21.15.3:0/2370907920 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0c011820 con 0x7f2afc004500 2022-07-21T01:40:49.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b18152cc0 con 0x7f2afc004500 2022-07-21T01:40:49.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1805d3b0 con 0x7f2afc004500 2022-07-21T01:40:49.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.889+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b18152e50 con 0x7f2afc004500 2022-07-21T01:40:49.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.893+0000 7f2b15ffb700 1 -- 172.21.15.3:0/2370907920 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2b0c01a030 con 0x7f2afc004500 2022-07-21T01:40:49.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.893+0000 7f2b15ffb700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2af80539e0 0x7f2af8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:49.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.893+0000 7f2b15ffb700 1 -- 172.21.15.3:0/2370907920 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2b0c06b9b0 con 0x7f2afc004500 2022-07-21T01:40:49.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.893+0000 7f2b17fff700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2af80539e0 0x7f2af8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:49.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.893+0000 7f2b17fff700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2af80539e0 0x7f2af8055ea0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f2b0401d000 tx=0x7f2b04017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:49.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:49.901+0000 7f2b15ffb700 1 -- 172.21.15.3:0/2370907920 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b0c0426b0 con 0x7f2afc004500 2022-07-21T01:40:50.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.065+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b180574e0 con 0x7f2afc004500 2022-07-21T01:40:50.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.065+0000 7f2b15ffb700 1 -- 172.21.15.3:0/2370907920 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2484 (secure 0 0 0) 0x7f2b0c045cd0 con 0x7f2afc004500 2022-07-21T01:40:50.068 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:50.068 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/485774043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":485774043},{"type":"v1","addr":"172.21.15.143:6833","nonce":485774043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":6936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/756498280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":756498280},{"type":"v1","addr":"172.21.15.3:6835","nonce":756498280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:41.034617+0000","modified":"2022-07-21T01:40:42.719494+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6942},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6942":{"gid":6942,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1935398064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1935398064},{"type":"v1","addr":"172.21.15.3:6837","nonce":1935398064}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-21T01:40:50.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2af80539e0 msgr2=0x7f2af8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:50.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2af80539e0 0x7f2af8055ea0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f2b0401d000 tx=0x7f2b04017000).stop 2022-07-21T01:40:50.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 msgr2=0x7f2b180ac130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:50.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2b180ac130 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f2b0c00d000 tx=0x7f2b0c007a00).stop 2022-07-21T01:40:50.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 shutdown_connections 2022-07-21T01:40:50.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1805ca80 0x7f2b1809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:50.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2af80539e0 0x7f2af8055ea0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:50.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1805c620 0x7f2b1813e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:50.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 --2- 172.21.15.3:0/2370907920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2afc004500 0x7f2b180ac130 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:50.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 >> 172.21.15.3:0/2370907920 conn(0x7f2b18060310 msgr2=0x7f2b18144b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:50.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 shutdown_connections 2022-07-21T01:40:50.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:50.069+0000 7f2b1df21700 1 -- 172.21.15.3:0/2370907920 wait complete. 2022-07-21T01:40:50.074 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 48 2022-07-21T01:40:50.393 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:40:41.034617+0000', 'modified': '2022-07-21T01:40:42.719494+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6942}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6942': {'gid': 6942, 'name': 'c', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/1935398064', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 1935398064}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 1935398064}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:40:50.393 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:40:50.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:40:50.511 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":6942,"want_state":"up:active","state":"up:active","rank_uptime":9.4652996839999997,"mdsmap_epoch":48,"osdmap_epoch":101,"osdmap_epoch_barrier":101,"uptime":14.150744759} 2022-07-21T01:40:50.512 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 6942, "mdsmap_epoch": 48, "osdmap_epoch": 101, "osdmap_epoch_barrier": 101, "rank_uptime": 9.465299684, "state": "up:active", "uptime": 14.150744759, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:40:50.513 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:40:50.513 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:40:50.514 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:40:50.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:50.515 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:50.524 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:40:50.542 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:50.543 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:50.558 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:50.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:50.560 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:40:50.610 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:40:50.612 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:50.616 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:40:50.633 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:40:50.634 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:40:50.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:40:50.681+0000 7effc665c080 -1 init, newargv = 0x557823e49a60 newargc=9 2022-07-21T01:40:50.687 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22626]: starting ceph client 2022-07-21T01:40:50.694 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:40:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:40:50.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:50.698 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:40:50.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22626]: starting fuse 2022-07-21T01:40:50.705 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:40:50.706 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:40:50.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:50.707 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:40:50.708 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:40:50.709 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:40:50.710 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:40:50.711 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:40:50.712 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:40:50.712 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:40:50.713 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:40:50.714 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:40:50.715 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:40:50.715 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:40:50.716 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:40:50.717 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:40:50.717 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:40:50.717 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:40:50.718 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:40:50.718 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:40:50.719 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:40:50.720 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:40:50.720 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:40:50.720 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:40:50.721 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:40:50.721 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:40:50.766 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.22626.asok 2022-07-21T01:40:50.771 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22626.asok 2022-07-21T01:40:50.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:50.773 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22626.asok status 2022-07-21T01:40:50.907 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:40:50.908 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:40:50.908 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:40:50.909 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:40:50.909 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:40:50.910 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:40:50.910 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:40:50.911 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "22626", 2022-07-21T01:40:50.911 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:40:50.912 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:40:50.913 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:40:50.913 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:40:50.914 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 7140, 2022-07-21T01:40:50.914 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:40:50.915 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:40:50.915 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:40:50.916 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 7140 2022-07-21T01:40:50.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:40:50.917 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:40:50.918 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:40:50.918 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:40:50.919 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 750186402 2022-07-21T01:40:50.919 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:40:50.920 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:40:50.920 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:40:50.921 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:40:50.921 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:40:50.922 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 750186402 2022-07-21T01:40:50.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:40:50.923 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.7140 172.21.15.174:0/750186402", 2022-07-21T01:40:50.924 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/750186402", 2022-07-21T01:40:50.925 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:40:50.925 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 48, 2022-07-21T01:40:50.926 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 101, 2022-07-21T01:40:50.926 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:40:50.926 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:40:50.926 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:40:50.927 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:40:50.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:50.929 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:50.929 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:50.938 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:40:50.939 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:40:50.939 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:50.940 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:50.958 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-21T01:40:50.959 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest/mnt.0 && sudo mkdir -p foo 2022-07-21T01:40:50.991 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-21T01:40:50.992 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:40:50.993 DEBUG:teuthology.orchestra.run.smithi171:> import stat 2022-07-21T01:40:50.993 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:40:50.994 DEBUG:teuthology.orchestra.run.smithi171:> print(os.stat("/home/ubuntu/cephtest/mnt.0/foo").st_ino) 2022-07-21T01:40:50.995 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:40:51.073 INFO:teuthology.orchestra.run.smithi171.stdout:1099511627776 2022-07-21T01:40:51.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.372+0000 7fadd3fff700 1 Processor -- start 2022-07-21T01:40:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.372+0000 7fadd3fff700 1 -- start start 2022-07-21T01:40:51.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fadd40b2710 0x7fadd414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:51.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fadd40b01a0 0x7fadd414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:51.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40afd60 0x7fadd40a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:51.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd40a7250 con 0x7fadd40b2710 2022-07-21T01:40:51.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd40a7480 con 0x7fadd40b01a0 2022-07-21T01:40:51.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40a7720 con 0x7fadd40afd60 2022-07-21T01:40:51.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd2ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fadd40b2710 0x7fadd414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:51.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd2ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fadd40b2710 0x7fadd414afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48618/0 (socket says 172.21.15.3:48618) 2022-07-21T01:40:51.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd2ffd700 1 -- 172.21.15.3:0/2087393769 learned_addr learned my addr 172.21.15.3:0/2087393769 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:51.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd37fe700 1 --2- 172.21.15.3:0/2087393769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40afd60 0x7fadd40a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:51.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd27fc700 1 --2- 172.21.15.3:0/2087393769 >> v2:172.21.15.143:3300/0 conn(0x7fadd40b01a0 0x7fadd414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:51.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd2ffd700 1 -- 172.21.15.3:0/2087393769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40afd60 msgr2=0x7fadd40a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:51.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd2ffd700 1 --2- 172.21.15.3:0/2087393769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40afd60 0x7fadd40a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd2ffd700 1 -- 172.21.15.3:0/2087393769 >> v2:172.21.15.143:3300/0 conn(0x7fadd40b01a0 msgr2=0x7fadd414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:51.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd2ffd700 1 --2- 172.21.15.3:0/2087393769 >> v2:172.21.15.143:3300/0 conn(0x7fadd40b01a0 0x7fadd414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd2ffd700 1 -- 172.21.15.3:0/2087393769 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd40a79c0 con 0x7fadd40b2710 2022-07-21T01:40:51.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd27fc700 1 --2- 172.21.15.3:0/2087393769 >> v2:172.21.15.143:3300/0 conn(0x7fadd40b01a0 0x7fadd414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:40:51.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd2ffd700 1 --2- 172.21.15.3:0/2087393769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b2710 0x7fadd414afd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7fadcc0032f0 tx=0x7fadcc009290).ready entity=mon.0 client_cookie=5d89a66f79f69146 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:51.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd1ffb700 1 -- 172.21.15.3:0/2087393769 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadcc00b870 con 0x7fadd40b2710 2022-07-21T01:40:51.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd1ffb700 1 -- 172.21.15.3:0/2087393769 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc00b9d0 con 0x7fadd40b2710 2022-07-21T01:40:51.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd1ffb700 1 -- 172.21.15.3:0/2087393769 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadcc00bb40 con 0x7fadd40b2710 2022-07-21T01:40:51.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 -- 172.21.15.3:0/2087393769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b2710 msgr2=0x7fadd414afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:51.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 --2- 172.21.15.3:0/2087393769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b2710 0x7fadd414afd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7fadcc0032f0 tx=0x7fadcc009290).stop 2022-07-21T01:40:51.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 -- 172.21.15.3:0/2087393769 shutdown_connections 2022-07-21T01:40:51.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 --2- 172.21.15.3:0/2087393769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40afd60 0x7fadd40a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 --2- 172.21.15.3:0/2087393769 >> v2:172.21.15.143:3300/0 conn(0x7fadd40b01a0 0x7fadd414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 --2- 172.21.15.3:0/2087393769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b2710 0x7fadd414afd0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 -- 172.21.15.3:0/2087393769 >> 172.21.15.3:0/2087393769 conn(0x7fadd4060310 msgr2=0x7fadd40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:51.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 -- 172.21.15.3:0/2087393769 shutdown_connections 2022-07-21T01:40:51.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 -- 172.21.15.3:0/2087393769 wait complete. 2022-07-21T01:40:51.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 Processor -- start 2022-07-21T01:40:51.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.376+0000 7fadd3fff700 1 -- start start 2022-07-21T01:40:51.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd40afd60 0x7fadd40679b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:51.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b01a0 0x7fadd40aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:51.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40b2710 0x7fadd414baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:51.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd3fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd414d8e0 con 0x7fadd40b01a0 2022-07-21T01:40:51.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd3fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40b35c0 con 0x7fadd40afd60 2022-07-21T01:40:51.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd3fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40ab290 con 0x7fadd40b2710 2022-07-21T01:40:51.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd27fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b01a0 0x7fadd40aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:51.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd27fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b01a0 0x7fadd40aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48628/0 (socket says 172.21.15.3:48628) 2022-07-21T01:40:51.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd27fc700 1 -- 172.21.15.3:0/3953659366 learned_addr learned my addr 172.21.15.3:0/3953659366 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:51.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd37fe700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40b2710 0x7fadd414baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:51.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd2ffd700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd40afd60 0x7fadd40679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:51.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd27fc700 1 -- 172.21.15.3:0/3953659366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40b2710 msgr2=0x7fadd414baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:51.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd27fc700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40b2710 0x7fadd414baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd27fc700 1 -- 172.21.15.3:0/3953659366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd40afd60 msgr2=0x7fadd40679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:51.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd27fc700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd40afd60 0x7fadd40679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd27fc700 1 -- 172.21.15.3:0/3953659366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc009530 con 0x7fadd40b01a0 2022-07-21T01:40:51.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd27fc700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b01a0 0x7fadd40aedc0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7fadbc007000 tx=0x7fadbc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:51.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadcbfff700 1 -- 172.21.15.3:0/3953659366 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadbc010580 con 0x7fadd40b01a0 2022-07-21T01:40:51.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadcbfff700 1 -- 172.21.15.3:0/3953659366 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc0106e0 con 0x7fadd40b01a0 2022-07-21T01:40:51.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd40a9b40 con 0x7fadd40b01a0 2022-07-21T01:40:51.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadcbfff700 1 -- 172.21.15.3:0/3953659366 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadbc010a40 con 0x7fadd40b01a0 2022-07-21T01:40:51.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.380+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd414cbd0 con 0x7fadd40b01a0 2022-07-21T01:40:51.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.384+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd40a9ed0 con 0x7fadd40b01a0 2022-07-21T01:40:51.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.384+0000 7fadcbfff700 1 -- 172.21.15.3:0/3953659366 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fadbc010ba0 con 0x7fadd40b01a0 2022-07-21T01:40:51.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.384+0000 7fadcbfff700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fadb8053a70 0x7fadb8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:51.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.384+0000 7fadcbfff700 1 -- 172.21.15.3:0/3953659366 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fadbc06bf10 con 0x7fadd40b01a0 2022-07-21T01:40:51.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.384+0000 7fadd2ffd700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fadb8053a70 0x7fadb8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:51.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.384+0000 7fadd2ffd700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fadb8053a70 0x7fadb8055f30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fadcc01b000 tx=0x7fadcc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:51.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.392+0000 7fadcbfff700 1 -- 172.21.15.3:0/3953659366 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fadbc042bc0 con 0x7fadd40b01a0 2022-07-21T01:40:51.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.556+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadd40574e0 con 0x7fadd40b01a0 2022-07-21T01:40:51.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.556+0000 7fadcbfff700 1 -- 172.21.15.3:0/3953659366 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2484 (secure 0 0 0) 0x7fadbc0461e0 con 0x7fadd40b01a0 2022-07-21T01:40:51.559 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:51.559 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/485774043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":485774043},{"type":"v1","addr":"172.21.15.143:6833","nonce":485774043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":6936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/756498280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":756498280},{"type":"v1","addr":"172.21.15.3:6835","nonce":756498280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:41.034617+0000","modified":"2022-07-21T01:40:42.719494+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6942},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6942":{"gid":6942,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1935398064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1935398064},{"type":"v1","addr":"172.21.15.3:6837","nonce":1935398064}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-21T01:40:51.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fadb8053a70 msgr2=0x7fadb8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:51.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fadb8053a70 0x7fadb8055f30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fadcc01b000 tx=0x7fadcc016000).stop 2022-07-21T01:40:51.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b01a0 msgr2=0x7fadd40aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:51.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b01a0 0x7fadd40aedc0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7fadbc007000 tx=0x7fadbc008040).stop 2022-07-21T01:40:51.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 shutdown_connections 2022-07-21T01:40:51.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd40b2710 0x7fadd414baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd40b01a0 0x7fadd40aedc0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fadb8053a70 0x7fadb8055f30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 --2- 172.21.15.3:0/3953659366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd40afd60 0x7fadd40679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:51.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 >> 172.21.15.3:0/3953659366 conn(0x7fadd4060310 msgr2=0x7fadd4148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:51.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 shutdown_connections 2022-07-21T01:40:51.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:51.560+0000 7fadd3fff700 1 -- 172.21.15.3:0/3953659366 wait complete. 2022-07-21T01:40:51.564 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 48 2022-07-21T01:40:51.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump inode 0x10000000000 2022-07-21T01:40:51.675 INFO:teuthology.orchestra.run.smithi003.stdout:{"path":"/foo","ino":1099511627776,"rdev":0,"ctime":"2022-07-21T01:40:50.980126+0000","btime":"2022-07-21T01:40:50.980126+0000","mode":16877,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-07-21T01:40:50.980126+0000","atime":"2022-07-21T01:40:50.980126+0000","time_warp_seq":0,"change_attr":0,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"version":2,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1},"nref":4,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"xattrlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":7087,"pending":"pAsxLsXsxFsx","issued":"pAsxLsXsxFsx","wanted":"-","last_sent":1}],"loner":7087,"want_loner":7087,"mds_caps_wanted":[]} 2022-07-21T01:40:51.676 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2022-07-21T01:40:50.980126+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "backtrace_version": 2, "btime": "2022-07-21T01:40:50.980126+0000", "change_attr": 0, "client_caps": [ { "client_id": 7087, "issued": "pAsxLsXsxFsx", "last_sent": 1, "pending": "pAsxLsXsxFsx", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-07-21T01:40:50.980126+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "flocklock": {}, "gid": 0, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": 7087, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2022-07-21T01:40:50.980126+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "nlink": 1, "nref": 4, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/foo", "pins": { "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "request": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 2, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": 7087, "xattr_version": 1, "xattrlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "xattrs": [] } 2022-07-21T01:40:51.677 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:40:51.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:51.692 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:51.712 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:51.734 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:51.755 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:40:51.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:51.756 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:51.757 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:51.765 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:40:51.766 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:51.767 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:40:51.767 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:40:51.767 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:51.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23034]: fuse finished with error 0 and tester_r 0 2022-07-21T01:40:51.824 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:51.825 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:51.825 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:51.832 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:40:51.833 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:40:51.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:40:51.927 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:51.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:51.937 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:51.938 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:51.947 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:40:51.949 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:40:51.949 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:40:51.950 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:40:51.951 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:51.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22626]: fuse finished with error 0 and tester_r 0 2022-07-21T01:40:51.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:51.970 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:51.970 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:52.018 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:40:52.019 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:40:52.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:40:52.084 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:52.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:40:52.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.212+0000 7f0beffff700 1 Processor -- start 2022-07-21T01:40:52.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.212+0000 7f0beffff700 1 -- start start 2022-07-21T01:40:52.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0bf00614b0 0x7f0bf0061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0bf00ad750 0x7f0bf00b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf01405d0 0x7f0bf0140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf00aee80 con 0x7f0bf00614b0 2022-07-21T01:40:52.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf00b70b0 con 0x7f0bf00ad750 2022-07-21T01:40:52.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00af0b0 con 0x7f0bf01405d0 2022-07-21T01:40:52.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beeffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0bf00614b0 0x7f0bf0061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beeffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0bf00614b0 0x7f0bf0061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48634/0 (socket says 172.21.15.3:48634) 2022-07-21T01:40:52.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beeffd700 1 -- 172.21.15.3:0/1634865442 learned_addr learned my addr 172.21.15.3:0/1634865442 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:52.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0be7fff700 1 --2- 172.21.15.3:0/1634865442 >> v2:172.21.15.143:3300/0 conn(0x7f0bf00ad750 0x7f0bf00b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0bef7fe700 1 --2- 172.21.15.3:0/1634865442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf01405d0 0x7f0bf0140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beeffd700 1 -- 172.21.15.3:0/1634865442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf01405d0 msgr2=0x7f0bf0140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:52.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beeffd700 1 --2- 172.21.15.3:0/1634865442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf01405d0 0x7f0bf0140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beeffd700 1 -- 172.21.15.3:0/1634865442 >> v2:172.21.15.143:3300/0 conn(0x7f0bf00ad750 msgr2=0x7f0bf00b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:52.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beeffd700 1 --2- 172.21.15.3:0/1634865442 >> v2:172.21.15.143:3300/0 conn(0x7f0bf00ad750 0x7f0bf00b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beeffd700 1 -- 172.21.15.3:0/1634865442 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf00af350 con 0x7f0bf00614b0 2022-07-21T01:40:52.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beeffd700 1 --2- 172.21.15.3:0/1634865442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 0x7f0bf0061890 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f0bf00aed30 tx=0x7f0be0006eb0).ready entity=mon.0 client_cookie=28c478f952b31017 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:52.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0bee7fc700 1 -- 172.21.15.3:0/1634865442 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0be000e610 con 0x7f0bf00614b0 2022-07-21T01:40:52.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0bee7fc700 1 -- 172.21.15.3:0/1634865442 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be0000e90 con 0x7f0bf00614b0 2022-07-21T01:40:52.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0bee7fc700 1 -- 172.21.15.3:0/1634865442 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0be000e9a0 con 0x7f0bf00614b0 2022-07-21T01:40:52.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 -- 172.21.15.3:0/1634865442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 msgr2=0x7f0bf0061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 --2- 172.21.15.3:0/1634865442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 0x7f0bf0061890 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f0bf00aed30 tx=0x7f0be0006eb0).stop 2022-07-21T01:40:52.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 -- 172.21.15.3:0/1634865442 shutdown_connections 2022-07-21T01:40:52.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 --2- 172.21.15.3:0/1634865442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf01405d0 0x7f0bf0140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 --2- 172.21.15.3:0/1634865442 >> v2:172.21.15.143:3300/0 conn(0x7f0bf00ad750 0x7f0bf00b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 --2- 172.21.15.3:0/1634865442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 0x7f0bf0061890 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 -- 172.21.15.3:0/1634865442 >> 172.21.15.3:0/1634865442 conn(0x7f0bf0066f70 msgr2=0x7f0bf0067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:52.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 -- 172.21.15.3:0/1634865442 shutdown_connections 2022-07-21T01:40:52.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.216+0000 7f0beffff700 1 -- 172.21.15.3:0/1634865442 wait complete. 2022-07-21T01:40:52.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beffff700 1 Processor -- start 2022-07-21T01:40:52.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beffff700 1 -- start start 2022-07-21T01:40:52.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 0x7f0bf00b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf00ad750 0x7f0bf01524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf01405d0 0x7f0bf01529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00b0b10 con 0x7f0bf00614b0 2022-07-21T01:40:52.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00b08b0 con 0x7f0bf01405d0 2022-07-21T01:40:52.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00b0150 con 0x7f0bf00ad750 2022-07-21T01:40:52.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 0x7f0bf00b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 0x7f0bf00b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48638/0 (socket says 172.21.15.3:48638) 2022-07-21T01:40:52.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beeffd700 1 -- 172.21.15.3:0/3203792606 learned_addr learned my addr 172.21.15.3:0/3203792606 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:52.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0bef7fe700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf01405d0 0x7f0bf01529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0be7fff700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf00ad750 0x7f0bf01524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beeffd700 1 -- 172.21.15.3:0/3203792606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf00ad750 msgr2=0x7f0bf01524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beeffd700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf00ad750 0x7f0bf01524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beeffd700 1 -- 172.21.15.3:0/3203792606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf01405d0 msgr2=0x7f0bf01529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beeffd700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf01405d0 0x7f0bf01529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beeffd700 1 -- 172.21.15.3:0/3203792606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be000b1e0 con 0x7f0bf00614b0 2022-07-21T01:40:52.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0beeffd700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 0x7f0bf00b34c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f0bf00aed30 tx=0x7f0be0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:52.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.220+0000 7f0becff9700 1 -- 172.21.15.3:0/3203792606 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0be0011760 con 0x7f0bf00614b0 2022-07-21T01:40:52.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.224+0000 7f0becff9700 1 -- 172.21.15.3:0/3203792606 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be00118c0 con 0x7f0bf00614b0 2022-07-21T01:40:52.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.224+0000 7f0becff9700 1 -- 172.21.15.3:0/3203792606 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0be0011c20 con 0x7f0bf00614b0 2022-07-21T01:40:52.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.224+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf0152ed0 con 0x7f0bf00614b0 2022-07-21T01:40:52.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.224+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf01567d0 con 0x7f0bf00614b0 2022-07-21T01:40:52.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.224+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf00b0310 con 0x7f0bf00614b0 2022-07-21T01:40:52.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.224+0000 7f0becff9700 1 -- 172.21.15.3:0/3203792606 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0be001a040 con 0x7f0bf00614b0 2022-07-21T01:40:52.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.224+0000 7f0becff9700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bd4053950 0x7f0bd4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.224+0000 7f0becff9700 1 -- 172.21.15.3:0/3203792606 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0be006d070 con 0x7f0bf00614b0 2022-07-21T01:40:52.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.224+0000 7f0be7fff700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bd4053950 0x7f0bd4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.228+0000 7f0be7fff700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bd4053950 0x7f0bd4055e10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f0bd800b000 tx=0x7f0bd8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:52.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.232+0000 7f0becff9700 1 -- 172.21.15.3:0/3203792606 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0be0043d20 con 0x7f0bf00614b0 2022-07-21T01:40:52.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.356+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f0bf00af5f0 con 0x7f0bf00614b0 2022-07-21T01:40:52.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.360+0000 7f0becff9700 1 -- 172.21.15.3:0/3203792606 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v101) v1 ==== 81+0+34043 (secure 0 0 0) 0x7f0be0023020 con 0x7f0bf00614b0 2022-07-21T01:40:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:52.364 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:40:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 101, 2022-07-21T01:40:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:40:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:40:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:40:42.037286+0000", 2022-07-21T01:40:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:40:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:40:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:40:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:40:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:40:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:40:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:40:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:40:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:40:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:40:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:40:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:40:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 18, 2022-07-21T01:40:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:40:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:40:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:40:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:40:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:40:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:40:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:40:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:40:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:40:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:40:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:40:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:40:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:40:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:40:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:40:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:40:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:40:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:40:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:40:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:40:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:40:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:40:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:40:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.410 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 17, 2022-07-21T01:40:52.410 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:40:52.411 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:40:39.112065+0000", 2022-07-21T01:40:52.411 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:52.411 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:52.412 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:52.412 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:52.412 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:52.412 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:52.412 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:52.413 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:52.413 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:52.413 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:52.413 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:52.414 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:52.414 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:52.414 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:52.414 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:52.415 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:52.415 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:52.415 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:52.415 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:52.416 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:52.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.416 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "100", 2022-07-21T01:40:52.416 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:52.417 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:52.417 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:52.417 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:52.417 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:52.418 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:52.418 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:52.418 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:52.418 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:52.419 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:52.419 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:52.419 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:52.419 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:52.420 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:52.420 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:52.420 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:52.420 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:52.421 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:52.421 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:52.421 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:52.421 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:52.422 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:52.422 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:52.422 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:52.422 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:52.423 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:52.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.423 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:52.423 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:52.423 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:52.424 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:52.424 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:52.424 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:52.425 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:52.425 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:52.425 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:52.426 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:52.426 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:52.426 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:40:52.426 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:40:52.427 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:40:52.427 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:40:52.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.427 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:52.428 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:40:52.428 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:40:52.428 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.429 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.430 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 18, 2022-07-21T01:40:52.430 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:40:52.431 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:40:40.040729+0000", 2022-07-21T01:40:52.431 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:52.431 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:52.431 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:52.432 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:52.432 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:52.432 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:52.432 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:52.432 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:52.433 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:52.433 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:52.433 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:52.433 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:52.434 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:52.434 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:52.434 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:52.435 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:52.435 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:52.435 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:52.435 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:52.436 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:52.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.436 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "101", 2022-07-21T01:40:52.437 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:52.437 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:52.437 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:52.437 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:52.438 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:52.438 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:52.438 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:52.439 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:52.439 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:52.440 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:52.440 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:52.440 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:52.441 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:52.441 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:52.441 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:52.441 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:52.441 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:52.442 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:52.442 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:52.442 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:52.442 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:52.443 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:52.443 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:52.443 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:52.444 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:52.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:52.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.444 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:52.445 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:52.445 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:52.445 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:52.445 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:52.446 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:52.446 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:52.446 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:52.446 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:52.447 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:52.447 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:52.447 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:40:52.447 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:52.447 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:40:52.448 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:40:52.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.449 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:52.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:40:52.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.449 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:40:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.454 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:40:52.454 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:52.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.455 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.455 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:40:52.455 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:52.456 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.456 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.456 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:52.457 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.458 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:40:52.458 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:52.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:40:52.459 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:52.459 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.459 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.460 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:52.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.461 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.461 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:40:52.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:52.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.462 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:40:52.462 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:52.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.463 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:52.463 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.464 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:40:52.464 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:52.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.465 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.465 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:40:52.465 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:52.466 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.466 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.466 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:40:52.466 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:40:52.467 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:40:52.467 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:40:52.467 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:52.467 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:52.468 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:52.468 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:52.469 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:40:52.469 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:52.469 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:52.469 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:52.470 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:52.470 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:52.470 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:52.470 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:52.471 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:52.471 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:52.471 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:52.471 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:52.471 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:40:52.472 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:52.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.473 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:40:52.473 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:52.474 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.474 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.474 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:52.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.475 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:40:52.475 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:52.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:40:52.477 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:52.477 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.477 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.477 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:52.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:40:52.479 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:52.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.479 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.479 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:40:52.480 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:52.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.480 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.481 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:52.481 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.481 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.482 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:40:52.482 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:52.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.483 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.483 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:40:52.483 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:52.483 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.483 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.484 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:40:52.484 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:40:52.484 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:40:52.485 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:40:52.485 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:52.485 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:52.485 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:52.485 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.486 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:52.486 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:40:52.487 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:52.487 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:52.487 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:52.487 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:52.488 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:52.488 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:52.488 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:52.488 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:52.488 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:52.489 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:52.489 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:52.489 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.490 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:40:52.490 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:52.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.491 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:40:52.491 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:52.491 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:40:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:40:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:40:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:40:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:40:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:40:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:40:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:40:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:40:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:40:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:40:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:40:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:40:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:40:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:40:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:40:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:40:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:40:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:40:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:40:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:40:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:40:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:40:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:40:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 98, 2022-07-21T01:40:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:40:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:40:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:40:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:40:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:40:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:40:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:40:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:40:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:40:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:40:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:40:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:40:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:40:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:40:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:40:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:40:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:40:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:40:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:40:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:40:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:40:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:40:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:40:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:40:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:40:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:40:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 98, 2022-07-21T01:40:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:40:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:40:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:40:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:40:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:40:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:40:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:40:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:40:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:40:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:40:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:40:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:40:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:40:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:40:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:40:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:40:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:40:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:40:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:40:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:40:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:40:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:40:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:40:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:40:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:40:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:40:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:40:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:40:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:40:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:40:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:40:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:40:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:40:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:40:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:40:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:40:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:40:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:40:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:40:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:40:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:40:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:40:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:40:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:40:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:40:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:40:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:40:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:40:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:40:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:40:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:40:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:40:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:40:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:40:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:40:52.618 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:40:52.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bd4053950 msgr2=0x7f0bd4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bd4053950 0x7f0bd4055e10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f0bd800b000 tx=0x7f0bd8006df0).stop 2022-07-21T01:40:52.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 msgr2=0x7f0bf00b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 0x7f0bf00b34c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f0bf00aed30 tx=0x7f0be0015000).stop 2022-07-21T01:40:52.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 shutdown_connections 2022-07-21T01:40:52.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bd4053950 0x7f0bd4055e10 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf01405d0 0x7f0bf01529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf00ad750 0x7f0bf01524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 --2- 172.21.15.3:0/3203792606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bf00614b0 0x7f0bf00b34c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 >> 172.21.15.3:0/3203792606 conn(0x7f0bf0066f70 msgr2=0x7f0bf0144920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:52.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 shutdown_connections 2022-07-21T01:40:52.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.364+0000 7f0beffff700 1 -- 172.21.15.3:0/3203792606 wait complete. 2022-07-21T01:40:52.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:52.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b02789700 1 Processor -- start 2022-07-21T01:40:52.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b02789700 1 -- start start 2022-07-21T01:40:52.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b02789700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 0x7f0afc0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b02789700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0afc0a7250 0x7f0afc1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b02789700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0afc0b04e0 con 0x7f0afc148530 2022-07-21T01:40:52.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b02789700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0afc0a9250 con 0x7f0afc0a7250 2022-07-21T01:40:52.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b02789700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc0b0780 con 0x7f0afc0b1f40 2022-07-21T01:40:52.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b00d26700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0afc0a7250 0x7f0afc1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b00d26700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0afc0a7250 0x7f0afc1448b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48650/0 (socket says 172.21.15.3:48650) 2022-07-21T01:40:52.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0afbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 0x7f0afc0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b00d26700 1 -- 172.21.15.3:0/234844494 learned_addr learned my addr 172.21.15.3:0/234844494 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:52.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b00d26700 1 -- 172.21.15.3:0/234844494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 msgr2=0x7f0afc0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:52.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b00d26700 1 --2- 172.21.15.3:0/234844494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 0x7f0afc0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b00d26700 1 -- 172.21.15.3:0/234844494 >> v1:172.21.15.143:6789/0 conn(0x7f0afc148530 legacy=0x7f0afc0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b00d26700 1 -- 172.21.15.3:0/234844494 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc0b0a20 con 0x7f0afc0a7250 2022-07-21T01:40:52.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0b00d26700 1 --2- 172.21.15.3:0/234844494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 0x7f0afc1448b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7f0af4002f70 tx=0x7f0af4006eb0).ready entity=mon.0 client_cookie=4c812adb3645c960 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:52.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0afaffd700 1 -- 172.21.15.3:0/234844494 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0af400d570 con 0x7f0afc0a7250 2022-07-21T01:40:52.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0afaffd700 1 -- 172.21.15.3:0/234844494 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af4000e90 con 0x7f0afc0a7250 2022-07-21T01:40:52.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.736+0000 7f0afaffd700 1 -- 172.21.15.3:0/234844494 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0af400d900 con 0x7f0afc0a7250 2022-07-21T01:40:52.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 -- 172.21.15.3:0/234844494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 msgr2=0x7f0afc1448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 --2- 172.21.15.3:0/234844494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 0x7f0afc1448b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7f0af4002f70 tx=0x7f0af4006eb0).stop 2022-07-21T01:40:52.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 -- 172.21.15.3:0/234844494 shutdown_connections 2022-07-21T01:40:52.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 --2- 172.21.15.3:0/234844494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 0x7f0afc1448b0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 --2- 172.21.15.3:0/234844494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 0x7f0afc0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 -- 172.21.15.3:0/234844494 >> 172.21.15.3:0/234844494 conn(0x7f0afc05f840 msgr2=0x7f0afc05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:52.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 -- 172.21.15.3:0/234844494 shutdown_connections 2022-07-21T01:40:52.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 -- 172.21.15.3:0/234844494 wait complete. 2022-07-21T01:40:52.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 Processor -- start 2022-07-21T01:40:52.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.740+0000 7f0b02789700 1 -- start start 2022-07-21T01:40:52.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0b02789700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 0x7f0afc14d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0b02789700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 0x7f0afc13e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0b02789700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0afc148530 0x7f0afc14a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0b02789700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc148fa0 con 0x7f0afc0a7250 2022-07-21T01:40:52.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0b02789700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc13e7f0 con 0x7f0afc148530 2022-07-21T01:40:52.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0b02789700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc143320 con 0x7f0afc0b1f40 2022-07-21T01:40:52.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 0x7f0afc14d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 0x7f0afc14d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48652/0 (socket says 172.21.15.3:48652) 2022-07-21T01:40:52.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afbfff700 1 -- 172.21.15.3:0/4243696213 learned_addr learned my addr 172.21.15.3:0/4243696213 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:52.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afb7fe700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 0x7f0afc13e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0b00d26700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0afc148530 0x7f0afc14a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afbfff700 1 -- 172.21.15.3:0/4243696213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 msgr2=0x7f0afc13e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afbfff700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 0x7f0afc13e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afbfff700 1 -- 172.21.15.3:0/4243696213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0afc148530 msgr2=0x7f0afc14a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afbfff700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0afc148530 0x7f0afc14a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afbfff700 1 -- 172.21.15.3:0/4243696213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af400b1e0 con 0x7f0afc0a7250 2022-07-21T01:40:52.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0afbfff700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 0x7f0afc14d360 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f0afc13e980 tx=0x7f0aec0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:52.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0af97fa700 1 -- 172.21.15.3:0/4243696213 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0aec00ee60 con 0x7f0afc0a7250 2022-07-21T01:40:52.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0af97fa700 1 -- 172.21.15.3:0/4243696213 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aec00c850 con 0x7f0afc0a7250 2022-07-21T01:40:52.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0afc1435c0 con 0x7f0afc0a7250 2022-07-21T01:40:52.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0af97fa700 1 -- 172.21.15.3:0/4243696213 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0aec00cbb0 con 0x7f0afc0a7250 2022-07-21T01:40:52.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.744+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0afc0adc90 con 0x7f0afc0a7250 2022-07-21T01:40:52.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.748+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0afc0574e0 con 0x7f0afc0a7250 2022-07-21T01:40:52.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.748+0000 7f0af97fa700 1 -- 172.21.15.3:0/4243696213 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0aec022070 con 0x7f0afc0a7250 2022-07-21T01:40:52.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.748+0000 7f0af97fa700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ad4053950 0x7f0ad4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:52.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.748+0000 7f0af97fa700 1 -- 172.21.15.3:0/4243696213 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0aec06c7d0 con 0x7f0afc0a7250 2022-07-21T01:40:52.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.748+0000 7f0afb7fe700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ad4053950 0x7f0ad4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:52.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.748+0000 7f0afb7fe700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ad4053950 0x7f0ad4055e10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f0ae400d000 tx=0x7f0ae4007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:52.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.752+0000 7f0af97fa700 1 -- 172.21.15.3:0/4243696213 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0aec043480 con 0x7f0afc0a7250 2022-07-21T01:40:52.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0afc0447e0 con 0x7f0afc0a7250 2022-07-21T01:40:52.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0af97fa700 1 -- 172.21.15.3:0/4243696213 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2484 (secure 0 0 0) 0x7f0aec046aa0 con 0x7f0afc0a7250 2022-07-21T01:40:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:52.936 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":48,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/485774043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":485774043},{"type":"v1","addr":"172.21.15.143:6833","nonce":485774043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":6936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/756498280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":756498280},{"type":"v1","addr":"172.21.15.3:6835","nonce":756498280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:41.034617+0000","modified":"2022-07-21T01:40:42.719494+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6942},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6942":{"gid":6942,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1935398064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1935398064},{"type":"v1","addr":"172.21.15.3:6837","nonce":1935398064}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-21T01:40:52.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ad4053950 msgr2=0x7f0ad4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ad4053950 0x7f0ad4055e10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f0ae400d000 tx=0x7f0ae4007b90).stop 2022-07-21T01:40:52.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 msgr2=0x7f0afc14d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:52.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 0x7f0afc14d360 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f0afc13e980 tx=0x7f0aec0091d0).stop 2022-07-21T01:40:52.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 shutdown_connections 2022-07-21T01:40:52.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0ad4053950 0x7f0ad4055e10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0afc148530 0x7f0afc14a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc0b1f40 0x7f0afc13e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 --2- 172.21.15.3:0/4243696213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0afc0a7250 0x7f0afc14d360 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:52.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 >> 172.21.15.3:0/4243696213 conn(0x7f0afc05f840 msgr2=0x7f0afc0ae980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:52.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 shutdown_connections 2022-07-21T01:40:52.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:52.916+0000 7f0b02789700 1 -- 172.21.15.3:0/4243696213 wait complete. 2022-07-21T01:40:52.947 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 48 2022-07-21T01:40:52.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:40:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.056+0000 7fd151ab0700 1 Processor -- start 2022-07-21T01:40:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.056+0000 7fd151ab0700 1 -- start start 2022-07-21T01:40:53.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd14c060e90 0x7fd14c0ab0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd14c0aff10 0x7fd14c1435f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd14c067370 con 0x7fd14c066f90 2022-07-21T01:40:53.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd14c0b0320 con 0x7fd14c0aff10 2022-07-21T01:40:53.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd14c0a95e0 con 0x7fd14c060e90 2022-07-21T01:40:53.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd14c0aff10 0x7fd14c1435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd14c0aff10 0x7fd14c1435f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48662/0 (socket says 172.21.15.3:48662) 2022-07-21T01:40:53.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14bfff700 1 -- 172.21.15.3:0/2396660418 learned_addr learned my addr 172.21.15.3:0/2396660418 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:53.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd143fff700 1 --2- 172.21.15.3:0/2396660418 >> v2:172.21.15.143:3300/0 conn(0x7fd14c060e90 0x7fd14c0ab0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14bfff700 1 -- 172.21.15.3:0/2396660418 >> v2:172.21.15.143:3300/0 conn(0x7fd14c060e90 msgr2=0x7fd14c0ab0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:53.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14bfff700 1 --2- 172.21.15.3:0/2396660418 >> v2:172.21.15.143:3300/0 conn(0x7fd14c060e90 0x7fd14c0ab0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14bfff700 1 -- 172.21.15.3:0/2396660418 >> v1:172.21.15.143:6789/0 conn(0x7fd14c066f90 legacy=0x7fd14c0aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14bfff700 1 -- 172.21.15.3:0/2396660418 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c0b0540 con 0x7fd14c0aff10 2022-07-21T01:40:53.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14bfff700 1 --2- 172.21.15.3:0/2396660418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c0aff10 0x7fd14c1435f0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7fd1440032f0 tx=0x7fd144009290).ready entity=mon.0 client_cookie=bf5e8dce33e87aae server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:53.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14affd700 1 -- 172.21.15.3:0/2396660418 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd14400b870 con 0x7fd14c0aff10 2022-07-21T01:40:53.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14affd700 1 -- 172.21.15.3:0/2396660418 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14400b9d0 con 0x7fd14c0aff10 2022-07-21T01:40:53.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd14affd700 1 -- 172.21.15.3:0/2396660418 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd14400bb40 con 0x7fd14c0aff10 2022-07-21T01:40:53.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 -- 172.21.15.3:0/2396660418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c0aff10 msgr2=0x7fd14c1435f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 --2- 172.21.15.3:0/2396660418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c0aff10 0x7fd14c1435f0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7fd1440032f0 tx=0x7fd144009290).stop 2022-07-21T01:40:53.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 -- 172.21.15.3:0/2396660418 shutdown_connections 2022-07-21T01:40:53.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 --2- 172.21.15.3:0/2396660418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c0aff10 0x7fd14c1435f0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 --2- 172.21.15.3:0/2396660418 >> v2:172.21.15.143:3300/0 conn(0x7fd14c060e90 0x7fd14c0ab0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 -- 172.21.15.3:0/2396660418 >> 172.21.15.3:0/2396660418 conn(0x7fd14c0b5570 msgr2=0x7fd14c0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:53.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 -- 172.21.15.3:0/2396660418 shutdown_connections 2022-07-21T01:40:53.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.060+0000 7fd151ab0700 1 -- 172.21.15.3:0/2396660418 wait complete. 2022-07-21T01:40:53.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 Processor -- start 2022-07-21T01:40:53.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 -- start start 2022-07-21T01:40:53.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c060e90 0x7fd14c141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c066f90 0x7fd14c136ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd14c0aff10 0x7fd14c136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c13fa20 con 0x7fd14c066f90 2022-07-21T01:40:53.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c13ef10 con 0x7fd14c060e90 2022-07-21T01:40:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c138bd0 con 0x7fd14c0aff10 2022-07-21T01:40:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd143fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c066f90 0x7fd14c136ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd14b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c060e90 0x7fd14c141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd143fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c066f90 0x7fd14c136ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48668/0 (socket says 172.21.15.3:48668) 2022-07-21T01:40:53.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd143fff700 1 -- 172.21.15.3:0/1672717522 learned_addr learned my addr 172.21.15.3:0/1672717522 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:53.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd14bfff700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd14c0aff10 0x7fd14c136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd143fff700 1 -- 172.21.15.3:0/1672717522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd14c0aff10 msgr2=0x7fd14c136fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd143fff700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd14c0aff10 0x7fd14c136fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd143fff700 1 -- 172.21.15.3:0/1672717522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c060e90 msgr2=0x7fd14c141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd143fff700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c060e90 0x7fd14c141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd143fff700 1 -- 172.21.15.3:0/1672717522 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144009530 con 0x7fd14c066f90 2022-07-21T01:40:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd14b7fe700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c060e90 0x7fd14c141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd143fff700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c066f90 0x7fd14c136ae0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7fd14c137af0 tx=0x7fd134006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:53.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd1497fa700 1 -- 172.21.15.3:0/1672717522 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd134010780 con 0x7fd14c066f90 2022-07-21T01:40:53.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd1497fa700 1 -- 172.21.15.3:0/1672717522 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1340108e0 con 0x7fd14c066f90 2022-07-21T01:40:53.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14c13fc10 con 0x7fd14c066f90 2022-07-21T01:40:53.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd1497fa700 1 -- 172.21.15.3:0/1672717522 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd134010c40 con 0x7fd14c066f90 2022-07-21T01:40:53.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.064+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14c13f3e0 con 0x7fd14c066f90 2022-07-21T01:40:53.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.068+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14c13f5a0 con 0x7fd14c066f90 2022-07-21T01:40:53.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.068+0000 7fd1497fa700 1 -- 172.21.15.3:0/1672717522 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd13401c460 con 0x7fd14c066f90 2022-07-21T01:40:53.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.068+0000 7fd1497fa700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd130053aa0 0x7fd130055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.068+0000 7fd1497fa700 1 -- 172.21.15.3:0/1672717522 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd13406c4d0 con 0x7fd14c066f90 2022-07-21T01:40:53.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.068+0000 7fd14b7fe700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd130053aa0 0x7fd130055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.068+0000 7fd14b7fe700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd130053aa0 0x7fd130055f60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fd13c00d000 tx=0x7fd13c000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:53.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.072+0000 7fd1497fa700 1 -- 172.21.15.3:0/1672717522 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd134043190 con 0x7fd14c066f90 2022-07-21T01:40:53.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.236+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd14c0574e0 con 0x7fd14c066f90 2022-07-21T01:40:53.549 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:40:53.548+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:40:53.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.560+0000 7fd1497fa700 1 -- 172.21.15.3:0/1672717522 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd1340467b0 con 0x7fd14c066f90 2022-07-21T01:40:53.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd130053aa0 msgr2=0x7fd130055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd130053aa0 0x7fd130055f60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fd13c00d000 tx=0x7fd13c000bc0).stop 2022-07-21T01:40:53.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c066f90 msgr2=0x7fd14c136ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c066f90 0x7fd14c136ae0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7fd14c137af0 tx=0x7fd134006df0).stop 2022-07-21T01:40:53.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 shutdown_connections 2022-07-21T01:40:53.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd14c0aff10 0x7fd14c136fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd14c066f90 0x7fd14c136ae0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd130053aa0 0x7fd130055f60 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 --2- 172.21.15.3:0/1672717522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c060e90 0x7fd14c141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 >> 172.21.15.3:0/1672717522 conn(0x7fd14c0b5570 msgr2=0x7fd14c060380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:53.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 shutdown_connections 2022-07-21T01:40:53.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.564+0000 7fd151ab0700 1 -- 172.21.15.3:0/1672717522 wait complete. 2022-07-21T01:40:53.571 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:40:53.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:53.593 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:40:53.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.688+0000 7f08fe7ac700 1 Processor -- start 2022-07-21T01:40:53.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.688+0000 7f08fe7ac700 1 -- start start 2022-07-21T01:40:53.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.688+0000 7f08fe7ac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f08f80b08a0 0x7f08f8142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.688+0000 7f08fe7ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b3260 0x7f08f81433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.688+0000 7f08fe7ac700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f80b0c80 con 0x7f08f80ad000 2022-07-21T01:40:53.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.688+0000 7f08fe7ac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f80ae810 con 0x7f08f80b08a0 2022-07-21T01:40:53.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.688+0000 7f08fe7ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f80aeab0 con 0x7f08f80b3260 2022-07-21T01:40:53.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fdfab700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f08f80ad000 0x7f08f805f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56970/0 (socket says 172.21.15.3:56970) 2022-07-21T01:40:53.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fdfab700 1 -- 172.21.15.3:0/2641604205 learned_addr learned my addr 172.21.15.3:0/2641604205 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:53.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fd7aa700 1 --2- 172.21.15.3:0/2641604205 >> v2:172.21.15.143:3300/0 conn(0x7f08f80b08a0 0x7f08f8142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fcfa9700 1 --2- 172.21.15.3:0/2641604205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b3260 0x7f08f81433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08e7fff700 1 -- 172.21.15.3:0/2641604205 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684461952 0 0) 0x7f08f80b0c80 con 0x7f08f80ad000 2022-07-21T01:40:53.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08e7fff700 1 -- 172.21.15.3:0/2641604205 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e0003260 con 0x7f08f80ad000 2022-07-21T01:40:53.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fd7aa700 1 -- 172.21.15.3:0/2641604205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b3260 msgr2=0x7f08f81433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:53.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fd7aa700 1 --2- 172.21.15.3:0/2641604205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b3260 0x7f08f81433a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fd7aa700 1 -- 172.21.15.3:0/2641604205 >> v1:172.21.15.3:6789/0 conn(0x7f08f80ad000 legacy=0x7f08f805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fd7aa700 1 -- 172.21.15.3:0/2641604205 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f80aed50 con 0x7f08f80b08a0 2022-07-21T01:40:53.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fd7aa700 1 --2- 172.21.15.3:0/2641604205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80b08a0 0x7f08f8142eb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f08f4002f70 tx=0x7f08f4006df0).ready entity=mon.1 client_cookie=213eb86cb482c29 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:53.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08e7fff700 1 -- 172.21.15.3:0/2641604205 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08f400d4a0 con 0x7f08f80b08a0 2022-07-21T01:40:53.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08e7fff700 1 -- 172.21.15.3:0/2641604205 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f4000e90 con 0x7f08f80b08a0 2022-07-21T01:40:53.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08e7fff700 1 -- 172.21.15.3:0/2641604205 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08f400d7f0 con 0x7f08f80b08a0 2022-07-21T01:40:53.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fe7ac700 1 -- 172.21.15.3:0/2641604205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80b08a0 msgr2=0x7f08f8142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fe7ac700 1 --2- 172.21.15.3:0/2641604205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80b08a0 0x7f08f8142eb0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f08f4002f70 tx=0x7f08f4006df0).stop 2022-07-21T01:40:53.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fe7ac700 1 -- 172.21.15.3:0/2641604205 shutdown_connections 2022-07-21T01:40:53.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fe7ac700 1 --2- 172.21.15.3:0/2641604205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b3260 0x7f08f81433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fe7ac700 1 --2- 172.21.15.3:0/2641604205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80b08a0 0x7f08f8142eb0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fe7ac700 1 -- 172.21.15.3:0/2641604205 >> 172.21.15.3:0/2641604205 conn(0x7f08f80612d0 msgr2=0x7f08f8062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:53.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fe7ac700 1 -- 172.21.15.3:0/2641604205 shutdown_connections 2022-07-21T01:40:53.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fe7ac700 1 -- 172.21.15.3:0/2641604205 wait complete. 2022-07-21T01:40:53.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.692+0000 7f08fe7ac700 1 Processor -- start 2022-07-21T01:40:53.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fe7ac700 1 -- start start 2022-07-21T01:40:53.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fe7ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80ad000 0x7f08f8155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fe7ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b08a0 0x7f08f80aa780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fe7ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08f80b3260 0x7f08f80ab580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fe7ac700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f813dff0 con 0x7f08f80b3260 2022-07-21T01:40:53.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fe7ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f813e290 con 0x7f08f80ad000 2022-07-21T01:40:53.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fe7ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f80aae40 con 0x7f08f80b08a0 2022-07-21T01:40:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fdfab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08f80b3260 0x7f08f80ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fdfab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08f80b3260 0x7f08f80ab580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48696/0 (socket says 172.21.15.3:48696) 2022-07-21T01:40:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fdfab700 1 -- 172.21.15.3:0/3977794911 learned_addr learned my addr 172.21.15.3:0/3977794911 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:53.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fcfa9700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b08a0 0x7f08f80aa780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fd7aa700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80ad000 0x7f08f8155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fdfab700 1 -- 172.21.15.3:0/3977794911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b08a0 msgr2=0x7f08f80aa780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fdfab700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b08a0 0x7f08f80aa780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fdfab700 1 -- 172.21.15.3:0/3977794911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80ad000 msgr2=0x7f08f8155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fdfab700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80ad000 0x7f08f8155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fdfab700 1 -- 172.21.15.3:0/3977794911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f400b180 con 0x7f08f80b3260 2022-07-21T01:40:53.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fdfab700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08f80b3260 0x7f08f80ab580 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f08f0001490 tx=0x7f08f0007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:53.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08e67fc700 1 -- 172.21.15.3:0/3977794911 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08f0005c70 con 0x7f08f80b3260 2022-07-21T01:40:53.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08e67fc700 1 -- 172.21.15.3:0/3977794911 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f0005dd0 con 0x7f08f80b3260 2022-07-21T01:40:53.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f80ab0e0 con 0x7f08f80b3260 2022-07-21T01:40:53.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08e67fc700 1 -- 172.21.15.3:0/3977794911 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08f00072c0 con 0x7f08f80b3260 2022-07-21T01:40:53.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.696+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f80aba70 con 0x7f08f80b3260 2022-07-21T01:40:53.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.700+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f80574e0 con 0x7f08f80b3260 2022-07-21T01:40:53.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.700+0000 7f08e67fc700 1 -- 172.21.15.3:0/3977794911 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f08f001a060 con 0x7f08f80b3260 2022-07-21T01:40:53.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.700+0000 7f08e67fc700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08d80539e0 0x7f08d8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.700+0000 7f08fd7aa700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08d80539e0 0x7f08d8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:53.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.700+0000 7f08e67fc700 1 -- 172.21.15.3:0/3977794911 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f08f006ba90 con 0x7f08f80b3260 2022-07-21T01:40:53.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.700+0000 7f08fd7aa700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08d80539e0 0x7f08d8055ea0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f08f401a000 tx=0x7f08f4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:53.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.708+0000 7f08e67fc700 1 -- 172.21.15.3:0/3977794911 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08f0042800 con 0x7f08f80b3260 2022-07-21T01:40:53.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.868+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08f80b2e00 con 0x7f08f80b3260 2022-07-21T01:40:53.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08e67fc700 1 -- 172.21.15.3:0/3977794911 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2127 (secure 0 0 0) 0x7f08f0018020 con 0x7f08f80b3260 2022-07-21T01:40:53.874 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:53.874 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":49,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/485774043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":485774043},{"type":"v1","addr":"172.21.15.143:6833","nonce":485774043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":6936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/756498280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":756498280},{"type":"v1","addr":"172.21.15.3:6835","nonce":756498280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:40:41.034617+0000","modified":"2022-07-21T01:40:53.549760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-21T01:40:53.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08d80539e0 msgr2=0x7f08d8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08d80539e0 0x7f08d8055ea0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f08f401a000 tx=0x7f08f4014000).stop 2022-07-21T01:40:53.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08f80b3260 msgr2=0x7f08f80ab580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:53.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08f80b3260 0x7f08f80ab580 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f08f0001490 tx=0x7f08f0007dc0).stop 2022-07-21T01:40:53.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 shutdown_connections 2022-07-21T01:40:53.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08f80b3260 0x7f08f80ab580 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80b08a0 0x7f08f80aa780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08d80539e0 0x7f08d8055ea0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 --2- 172.21.15.3:0/3977794911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80ad000 0x7f08f8155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:53.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 >> 172.21.15.3:0/3977794911 conn(0x7f08f80612d0 msgr2=0x7f08f813efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:53.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 shutdown_connections 2022-07-21T01:40:53.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.872+0000 7f08fe7ac700 1 -- 172.21.15.3:0/3977794911 wait complete. 2022-07-21T01:40:53.879 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 49 2022-07-21T01:40:53.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:40:53.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.988+0000 7f7879240700 1 Processor -- start 2022-07-21T01:40:53.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.988+0000 7f7879240700 1 -- start start 2022-07-21T01:40:53.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 0x7f78741401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:53.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f78740ac880 0x7f78741406d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7874067a10 con 0x7f78740b0270 2022-07-21T01:40:54.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78740ad740 con 0x7f78740ac880 2022-07-21T01:40:54.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7874067cb0 con 0x7f78740b2710 2022-07-21T01:40:54.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7873fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 0x7f78741401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7878a3f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f78740ac880 0x7f78741406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f786bfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f78740b0270 0x7f7874143000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:59096/0 (socket says 172.21.15.3:59096) 2022-07-21T01:40:54.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f786bfff700 1 -- 172.21.15.3:0/3926913984 learned_addr learned my addr 172.21.15.3:0/3926913984 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:54.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7878a3f700 1 -- 172.21.15.3:0/3926913984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 msgr2=0x7f78741401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:54.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7878a3f700 1 --2- 172.21.15.3:0/3926913984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 0x7f78741401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7878a3f700 1 -- 172.21.15.3:0/3926913984 >> v1:172.21.15.143:6789/0 conn(0x7f78740b0270 legacy=0x7f7874143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:54.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7878a3f700 1 -- 172.21.15.3:0/3926913984 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7874147160 con 0x7f78740ac880 2022-07-21T01:40:54.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f78737fe700 1 -- 172.21.15.3:0/3926913984 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910555079 0 0) 0x7f7874067a10 con 0x7f78740b0270 2022-07-21T01:40:54.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7878a3f700 1 --2- 172.21.15.3:0/3926913984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740ac880 0x7f78741406d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f786c0032f0 tx=0x7f786c0091d0).ready entity=mon.1 client_cookie=2ea8c7a5daf4de8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:54.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f78737fe700 1 -- 172.21.15.3:0/3926913984 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f786c00b6c0 con 0x7f78740ac880 2022-07-21T01:40:54.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f78737fe700 1 -- 172.21.15.3:0/3926913984 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c00b820 con 0x7f78740ac880 2022-07-21T01:40:54.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f78737fe700 1 -- 172.21.15.3:0/3926913984 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f786c00b990 con 0x7f78740ac880 2022-07-21T01:40:54.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 -- 172.21.15.3:0/3926913984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740ac880 msgr2=0x7f78741406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:54.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 --2- 172.21.15.3:0/3926913984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740ac880 0x7f78741406d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f786c0032f0 tx=0x7f786c0091d0).stop 2022-07-21T01:40:54.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 -- 172.21.15.3:0/3926913984 shutdown_connections 2022-07-21T01:40:54.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 --2- 172.21.15.3:0/3926913984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740ac880 0x7f78741406d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 --2- 172.21.15.3:0/3926913984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 0x7f78741401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 -- 172.21.15.3:0/3926913984 >> 172.21.15.3:0/3926913984 conn(0x7f7874060310 msgr2=0x7f78740b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:54.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 -- 172.21.15.3:0/3926913984 shutdown_connections 2022-07-21T01:40:54.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.992+0000 7f7879240700 1 -- 172.21.15.3:0/3926913984 wait complete. 2022-07-21T01:40:54.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7879240700 1 Processor -- start 2022-07-21T01:40:54.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7879240700 1 -- start start 2022-07-21T01:40:54.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7879240700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78740ac880 0x7f787413fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7879240700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740b0270 0x7f78741559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7879240700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 0x7f7874155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7879240700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7874144f50 con 0x7f78740ac880 2022-07-21T01:40:54.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7879240700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78741451f0 con 0x7f78740b0270 2022-07-21T01:40:54.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7879240700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7874141040 con 0x7f78740b2710 2022-07-21T01:40:54.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7873fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78740ac880 0x7f787413fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7873fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78740ac880 0x7f787413fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48704/0 (socket says 172.21.15.3:48704) 2022-07-21T01:40:54.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7873fff700 1 -- 172.21.15.3:0/896979091 learned_addr learned my addr 172.21.15.3:0/896979091 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:54.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f7878a3f700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 0x7f7874155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:53.996+0000 7f786bfff700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740b0270 0x7f78741559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f786bfff700 1 -- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 msgr2=0x7f7874155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:54.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f786bfff700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 0x7f7874155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f786bfff700 1 -- 172.21.15.3:0/896979091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78740ac880 msgr2=0x7f787413fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:54.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f786bfff700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78740ac880 0x7f787413fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f786bfff700 1 -- 172.21.15.3:0/896979091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c009470 con 0x7f78740b0270 2022-07-21T01:40:54.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f7878a3f700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 0x7f7874155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:54.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f786bfff700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740b0270 0x7f78741559c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f785c000ed0 tx=0x7f785c007a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:54.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f7871ffb700 1 -- 172.21.15.3:0/896979091 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f785c0070d0 con 0x7f78740b0270 2022-07-21T01:40:54.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f7871ffb700 1 -- 172.21.15.3:0/896979091 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c007230 con 0x7f78740b0270 2022-07-21T01:40:54.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7874143ce0 con 0x7f78740b0270 2022-07-21T01:40:54.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f7871ffb700 1 -- 172.21.15.3:0/896979091 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f785c005610 con 0x7f78740b0270 2022-07-21T01:40:54.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7874145ee0 con 0x7f78740b0270 2022-07-21T01:40:54.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78741453b0 con 0x7f78740b0270 2022-07-21T01:40:54.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f7871ffb700 1 -- 172.21.15.3:0/896979091 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f785c01a060 con 0x7f78740b0270 2022-07-21T01:40:54.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.000+0000 7f7871ffb700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7858053950 0x7f7858055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.004+0000 7f7871ffb700 1 -- 172.21.15.3:0/896979091 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f785c06b750 con 0x7f78740b0270 2022-07-21T01:40:54.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.004+0000 7f7873fff700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7858053950 0x7f7858055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.004+0000 7f7873fff700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7858053950 0x7f7858055e10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f786400b000 tx=0x7f7864006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:54.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.012+0000 7f7871ffb700 1 -- 172.21.15.3:0/896979091 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f785c0424c0 con 0x7f78740b0270 2022-07-21T01:40:54.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.172+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f78740574e0 con 0x7f78740b0270 2022-07-21T01:40:54.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.572+0000 7f7871ffb700 1 -- 172.21.15.3:0/896979091 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f785c018020 con 0x7f78740b0270 2022-07-21T01:40:54.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7858053950 msgr2=0x7f7858055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:54.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7858053950 0x7f7858055e10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f786400b000 tx=0x7f7864006df0).stop 2022-07-21T01:40:54.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740b0270 msgr2=0x7f78741559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:54.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740b0270 0x7f78741559c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f785c000ed0 tx=0x7f785c007a00).stop 2022-07-21T01:40:54.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 shutdown_connections 2022-07-21T01:40:54.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78740b2710 0x7f7874155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7858053950 0x7f7858055e10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78740b0270 0x7f78741559c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 --2- 172.21.15.3:0/896979091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78740ac880 0x7f787413fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 >> 172.21.15.3:0/896979091 conn(0x7f7874060310 msgr2=0x7f787405be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:54.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 shutdown_connections 2022-07-21T01:40:54.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.576+0000 7f7879240700 1 -- 172.21.15.3:0/896979091 wait complete. 2022-07-21T01:40:54.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:40:54.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.708+0000 7f2194ca2700 1 Processor -- start 2022-07-21T01:40:54.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.708+0000 7f2194ca2700 1 -- start start 2022-07-21T01:40:54.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f219005c960 0x7f21900b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219005d240 0x7f219013f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21900acfc0 0x7f21900aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2190067f00 con 0x7f21900acfc0 2022-07-21T01:40:54.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219013fa10 con 0x7f219005c960 2022-07-21T01:40:54.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219005cd40 con 0x7f219005d240 2022-07-21T01:40:54.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21900acfc0 0x7f21900aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21900acfc0 0x7f21900aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48712/0 (socket says 172.21.15.3:48712) 2022-07-21T01:40:54.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218ffff700 1 -- 172.21.15.3:0/3652466304 learned_addr learned my addr 172.21.15.3:0/3652466304 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:54.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218f7fe700 1 --2- 172.21.15.3:0/3652466304 >> v2:172.21.15.143:3300/0 conn(0x7f219005c960 0x7f21900b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218effd700 1 --2- 172.21.15.3:0/3652466304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219005d240 0x7f219013f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218ffff700 1 -- 172.21.15.3:0/3652466304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219005d240 msgr2=0x7f219013f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:54.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218ffff700 1 --2- 172.21.15.3:0/3652466304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219005d240 0x7f219013f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218ffff700 1 -- 172.21.15.3:0/3652466304 >> v2:172.21.15.143:3300/0 conn(0x7f219005c960 msgr2=0x7f21900b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:54.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218ffff700 1 --2- 172.21.15.3:0/3652466304 >> v2:172.21.15.143:3300/0 conn(0x7f219005c960 0x7f21900b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218ffff700 1 -- 172.21.15.3:0/3652466304 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21900b1920 con 0x7f21900acfc0 2022-07-21T01:40:54.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218ffff700 1 --2- 172.21.15.3:0/3652466304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21900acfc0 0x7f21900aee70 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f2180002f70 tx=0x7f2180006eb0).ready entity=mon.0 client_cookie=857d2b812373eac2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:54.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218e7fc700 1 -- 172.21.15.3:0/3652466304 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218000d570 con 0x7f21900acfc0 2022-07-21T01:40:54.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218e7fc700 1 -- 172.21.15.3:0/3652466304 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2180000e90 con 0x7f21900acfc0 2022-07-21T01:40:54.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f218e7fc700 1 -- 172.21.15.3:0/3652466304 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218000d900 con 0x7f21900acfc0 2022-07-21T01:40:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 -- 172.21.15.3:0/3652466304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21900acfc0 msgr2=0x7f21900aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 --2- 172.21.15.3:0/3652466304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21900acfc0 0x7f21900aee70 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f2180002f70 tx=0x7f2180006eb0).stop 2022-07-21T01:40:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 -- 172.21.15.3:0/3652466304 shutdown_connections 2022-07-21T01:40:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 --2- 172.21.15.3:0/3652466304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21900acfc0 0x7f21900aee70 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 --2- 172.21.15.3:0/3652466304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219005d240 0x7f219013f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 --2- 172.21.15.3:0/3652466304 >> v2:172.21.15.143:3300/0 conn(0x7f219005c960 0x7f21900b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 -- 172.21.15.3:0/3652466304 >> 172.21.15.3:0/3652466304 conn(0x7f2190060310 msgr2=0x7f21900b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:54.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 -- 172.21.15.3:0/3652466304 shutdown_connections 2022-07-21T01:40:54.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.712+0000 7f2194ca2700 1 -- 172.21.15.3:0/3652466304 wait complete. 2022-07-21T01:40:54.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 Processor -- start 2022-07-21T01:40:54.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 -- start start 2022-07-21T01:40:54.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219005c960 0x7f21900b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f219005d240 0x7f21900aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21900acfc0 0x7f2190132440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21900abf80 con 0x7f219005d240 2022-07-21T01:40:54.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21901412b0 con 0x7f219005c960 2022-07-21T01:40:54.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2190141830 con 0x7f21900acfc0 2022-07-21T01:40:54.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f219005d240 0x7f21900aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219005c960 0x7f21900b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f219005d240 0x7f21900aa990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48720/0 (socket says 172.21.15.3:48720) 2022-07-21T01:40:54.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218effd700 1 -- 172.21.15.3:0/801351387 learned_addr learned my addr 172.21.15.3:0/801351387 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:54.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218ffff700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21900acfc0 0x7f2190132440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218f7fe700 1 -- 172.21.15.3:0/801351387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21900acfc0 msgr2=0x7f2190132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:54.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218f7fe700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21900acfc0 0x7f2190132440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218f7fe700 1 -- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f219005d240 msgr2=0x7f21900aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:54.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218f7fe700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f219005d240 0x7f21900aa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:54.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218f7fe700 1 -- 172.21.15.3:0/801351387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218000b1e0 con 0x7f219005c960 2022-07-21T01:40:54.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218effd700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f219005d240 0x7f21900aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:54.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218f7fe700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219005c960 0x7f21900b0140 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f2190142b90 tx=0x7f21880091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:54.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218cff9700 1 -- 172.21.15.3:0/801351387 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21880126d0 con 0x7f219005c960 2022-07-21T01:40:54.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218cff9700 1 -- 172.21.15.3:0/801351387 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2188012830 con 0x7f219005c960 2022-07-21T01:40:54.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2190140e50 con 0x7f219005c960 2022-07-21T01:40:54.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f218cff9700 1 -- 172.21.15.3:0/801351387 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2188012b90 con 0x7f219005c960 2022-07-21T01:40:54.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.716+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2190141520 con 0x7f219005c960 2022-07-21T01:40:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.720+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2190140cb0 con 0x7f219005c960 2022-07-21T01:40:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.720+0000 7f218cff9700 1 -- 172.21.15.3:0/801351387 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2188022070 con 0x7f219005c960 2022-07-21T01:40:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.720+0000 7f218cff9700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2174053a70 0x7f2174055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.720+0000 7f218cff9700 1 -- 172.21.15.3:0/801351387 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f218806c660 con 0x7f219005c960 2022-07-21T01:40:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.720+0000 7f218effd700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2174053a70 0x7f2174055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.720+0000 7f218effd700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2174053a70 0x7f2174055f30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f217800b000 tx=0x7f2178006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.732+0000 7f218cff9700 1 -- 172.21.15.3:0/801351387 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2190140cb0 con 0x7f219005c960 2022-07-21T01:40:54.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:54.860+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2190140cb0 con 0x7f219005c960 2022-07-21T01:40:55.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.576+0000 7f218cff9700 1 -- 172.21.15.3:0/801351387 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) v1 ==== 185+0+0 (secure 0 0 0) 0x7f2190140cb0 con 0x7f219005c960 2022-07-21T01:40:55.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.576+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2174053a70 msgr2=0x7f2174055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:55.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.576+0000 7f2194ca2700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2174053a70 0x7f2174055f30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f217800b000 tx=0x7f2178006df0).stop 2022-07-21T01:40:55.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.576+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219005c960 msgr2=0x7f21900b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:55.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.576+0000 7f2194ca2700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219005c960 0x7f21900b0140 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f2190142b90 tx=0x7f21880091d0).stop 2022-07-21T01:40:55.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.580+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 shutdown_connections 2022-07-21T01:40:55.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.580+0000 7f2194ca2700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21900acfc0 0x7f2190132440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:55.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.580+0000 7f2194ca2700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f219005d240 0x7f21900aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:55.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.580+0000 7f2194ca2700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2174053a70 0x7f2174055f30 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:55.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.580+0000 7f2194ca2700 1 --2- 172.21.15.3:0/801351387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219005c960 0x7f21900b0140 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:55.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.580+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 >> 172.21.15.3:0/801351387 conn(0x7f2190060310 msgr2=0x7f21900b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:55.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.580+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 shutdown_connections 2022-07-21T01:40:55.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.580+0000 7f2194ca2700 1 -- 172.21.15.3:0/801351387 wait complete. 2022-07-21T01:40:55.585 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:40:55.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:40:55.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.700+0000 7f0fff59e700 1 Processor -- start 2022-07-21T01:40:55.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.700+0000 7f0fff59e700 1 -- start start 2022-07-21T01:40:55.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f10000ad750 0x7f1000061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:55.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f10001405d0 0x7f1000140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:55.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10000aee80 con 0x7f10000614b0 2022-07-21T01:40:55.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10000b70b0 con 0x7f10000ad750 2022-07-21T01:40:55.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10000af0b0 con 0x7f10001405d0 2022-07-21T01:40:55.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffdd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f10000ad750 0x7f1000061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffe59c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f10000614b0 0x7f10000b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57004/0 (socket says 172.21.15.3:57004) 2022-07-21T01:40:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffdd9b700 1 --2- 172.21.15.3:0/3696904229 >> v2:172.21.15.3:3300/0 conn(0x7f10000ad750 0x7f1000061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48726/0 (socket says 172.21.15.3:48726) 2022-07-21T01:40:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffe59c700 1 -- 172.21.15.3:0/3696904229 learned_addr learned my addr 172.21.15.3:0/3696904229 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffed9d700 1 --2- 172.21.15.3:0/3696904229 >> v2:172.21.15.143:3300/0 conn(0x7f10001405d0 0x7f1000140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffdd9b700 1 -- 172.21.15.3:0/3696904229 >> v2:172.21.15.143:3300/0 conn(0x7f10001405d0 msgr2=0x7f1000140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:55.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffdd9b700 1 --2- 172.21.15.3:0/3696904229 >> v2:172.21.15.143:3300/0 conn(0x7f10001405d0 0x7f1000140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:55.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffdd9b700 1 -- 172.21.15.3:0/3696904229 >> v1:172.21.15.3:6789/0 conn(0x7f10000614b0 legacy=0x7f10000b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:55.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffd59a700 1 -- 172.21.15.3:0/3696904229 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425431270 0 0) 0x7f10000aee80 con 0x7f10000614b0 2022-07-21T01:40:55.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffdd9b700 1 -- 172.21.15.3:0/3696904229 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10000af350 con 0x7f10000ad750 2022-07-21T01:40:55.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffdd9b700 1 --2- 172.21.15.3:0/3696904229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000ad750 0x7f1000061890 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f0ff0002f70 tx=0x7f0ff0006df0).ready entity=mon.0 client_cookie=303d96ef760ae483 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:55.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffd59a700 1 -- 172.21.15.3:0/3696904229 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ff000d570 con 0x7f10000ad750 2022-07-21T01:40:55.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffd59a700 1 -- 172.21.15.3:0/3696904229 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff0000e90 con 0x7f10000ad750 2022-07-21T01:40:55.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0ffd59a700 1 -- 172.21.15.3:0/3696904229 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ff000d8c0 con 0x7f10000ad750 2022-07-21T01:40:55.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 -- 172.21.15.3:0/3696904229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000ad750 msgr2=0x7f1000061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:55.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 --2- 172.21.15.3:0/3696904229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000ad750 0x7f1000061890 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f0ff0002f70 tx=0x7f0ff0006df0).stop 2022-07-21T01:40:55.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 -- 172.21.15.3:0/3696904229 shutdown_connections 2022-07-21T01:40:55.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 --2- 172.21.15.3:0/3696904229 >> v2:172.21.15.143:3300/0 conn(0x7f10001405d0 0x7f1000140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:55.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 --2- 172.21.15.3:0/3696904229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000ad750 0x7f1000061890 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:55.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 -- 172.21.15.3:0/3696904229 >> 172.21.15.3:0/3696904229 conn(0x7f1000066f70 msgr2=0x7f1000067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:55.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 -- 172.21.15.3:0/3696904229 shutdown_connections 2022-07-21T01:40:55.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.704+0000 7f0fff59e700 1 -- 172.21.15.3:0/3696904229 wait complete. 2022-07-21T01:40:55.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0fff59e700 1 Processor -- start 2022-07-21T01:40:55.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0fff59e700 1 -- start start 2022-07-21T01:40:55.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0fff59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000614b0 0x7f1000142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:55.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0fff59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10000ad750 0x7f10000b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:55.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0fff59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10001405d0 0x7f10000b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:55.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0fff59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10001559f0 con 0x7f10000614b0 2022-07-21T01:40:55.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0fff59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10001568a0 con 0x7f10000ad750 2022-07-21T01:40:55.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0fff59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1000155c20 con 0x7f10001405d0 2022-07-21T01:40:55.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0ffe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000614b0 0x7f1000142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:55.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0ffe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000614b0 0x7f1000142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48730/0 (socket says 172.21.15.3:48730) 2022-07-21T01:40:55.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0ffe59c700 1 -- 172.21.15.3:0/3840750122 learned_addr learned my addr 172.21.15.3:0/3840750122 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:55.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0ffdd9b700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10000ad750 0x7f10000b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:55.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.708+0000 7f0ffed9d700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10001405d0 0x7f10000b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:55.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0ffe59c700 1 -- 172.21.15.3:0/3840750122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10001405d0 msgr2=0x7f10000b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:55.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0ffe59c700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10001405d0 0x7f10000b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:55.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0ffe59c700 1 -- 172.21.15.3:0/3840750122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10000ad750 msgr2=0x7f10000b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:55.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0ffe59c700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10000ad750 0x7f10000b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:55.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0ffe59c700 1 -- 172.21.15.3:0/3840750122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff000b180 con 0x7f10000614b0 2022-07-21T01:40:55.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0ffe59c700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000614b0 0x7f1000142970 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f0ff4000ed0 tx=0x7f0ff4007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:55.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0fef7fe700 1 -- 172.21.15.3:0/3840750122 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ff40070d0 con 0x7f10000614b0 2022-07-21T01:40:55.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1000155ec0 con 0x7f10000614b0 2022-07-21T01:40:55.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0fef7fe700 1 -- 172.21.15.3:0/3840750122 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff4007230 con 0x7f10000614b0 2022-07-21T01:40:55.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0fef7fe700 1 -- 172.21.15.3:0/3840750122 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ff4005610 con 0x7f10000614b0 2022-07-21T01:40:55.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1000156400 con 0x7f10000614b0 2022-07-21T01:40:55.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10000af5f0 con 0x7f10000614b0 2022-07-21T01:40:55.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0fef7fe700 1 -- 172.21.15.3:0/3840750122 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0ff4010070 con 0x7f10000614b0 2022-07-21T01:40:55.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0fef7fe700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0fdc053a70 0x7f0fdc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:55.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0fef7fe700 1 -- 172.21.15.3:0/3840750122 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f0ff406b740 con 0x7f10000614b0 2022-07-21T01:40:55.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.712+0000 7f0ffdd9b700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0fdc053a70 0x7f0fdc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:55.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.716+0000 7f0ffdd9b700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0fdc053a70 0x7f0fdc055f30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f0ff001a000 tx=0x7f0ff0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:55.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.716+0000 7f0fef7fe700 1 -- 172.21.15.3:0/3840750122 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0ff40424d0 con 0x7f10000614b0 2022-07-21T01:40:55.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:55.840+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10000574e0 con 0x7f10000614b0 2022-07-21T01:40:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.588+0000 7f0fef7fe700 1 -- 172.21.15.3:0/3840750122 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0ff4045af0 con 0x7f10000614b0 2022-07-21T01:40:56.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0fdc053a70 msgr2=0x7f0fdc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:56.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0fdc053a70 0x7f0fdc055f30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f0ff001a000 tx=0x7f0ff0014000).stop 2022-07-21T01:40:56.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000614b0 msgr2=0x7f1000142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:56.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000614b0 0x7f1000142970 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f0ff4000ed0 tx=0x7f0ff4007ac0).stop 2022-07-21T01:40:56.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 shutdown_connections 2022-07-21T01:40:56.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10001405d0 0x7f10000b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0fdc053a70 0x7f0fdc055f30 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10000ad750 0x7f10000b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 --2- 172.21.15.3:0/3840750122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10000614b0 0x7f1000142970 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 >> 172.21.15.3:0/3840750122 conn(0x7f1000066f70 msgr2=0x7f1000145540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:56.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 shutdown_connections 2022-07-21T01:40:56.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.592+0000 7f0fff59e700 1 -- 172.21.15.3:0/3840750122 wait complete. 2022-07-21T01:40:56.598 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:40:56.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:40:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941ef41700 1 Processor -- start 2022-07-21T01:40:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941ef41700 1 -- start start 2022-07-21T01:40:56.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941ef41700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180b29e0 0x7f94180672e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:56.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941ef41700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f94180af2c0 0x7f94180677d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:56.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941ef41700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94181445e0 con 0x7f94180afef0 2022-07-21T01:40:56.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941ef41700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180b2dc0 con 0x7f94180af2c0 2022-07-21T01:40:56.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941ef41700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9418145490 con 0x7f94180b29e0 2022-07-21T01:40:56.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941d73e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f94180af2c0 0x7f94180677d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:56.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941e740700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f94180afef0 0x7f9418141010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57022/0 (socket says 172.21.15.3:57022) 2022-07-21T01:40:56.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941d73e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f94180af2c0 0x7f94180677d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48740/0 (socket says 172.21.15.3:48740) 2022-07-21T01:40:56.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941e740700 1 -- 172.21.15.3:0/3551790171 learned_addr learned my addr 172.21.15.3:0/3551790171 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:56.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.716+0000 7f941df3f700 1 --2- 172.21.15.3:0/3551790171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180b29e0 0x7f94180672e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:56.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941d73e700 1 -- 172.21.15.3:0/3551790171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180b29e0 msgr2=0x7f94180672e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:56.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941d73e700 1 --2- 172.21.15.3:0/3551790171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180b29e0 0x7f94180672e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941d73e700 1 -- 172.21.15.3:0/3551790171 >> v1:172.21.15.3:6789/0 conn(0x7f94180afef0 legacy=0x7f9418141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:56.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941d73e700 1 -- 172.21.15.3:0/3551790171 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9418144810 con 0x7f94180af2c0 2022-07-21T01:40:56.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941cf3d700 1 -- 172.21.15.3:0/3551790171 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700748953 0 0) 0x7f94181445e0 con 0x7f94180afef0 2022-07-21T01:40:56.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941d73e700 1 --2- 172.21.15.3:0/3551790171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 0x7f94180677d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f9408002f70 tx=0x7f9408006df0).ready entity=mon.0 client_cookie=7fdee861875b1dbd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:56.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941cf3d700 1 -- 172.21.15.3:0/3551790171 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940800d570 con 0x7f94180af2c0 2022-07-21T01:40:56.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941cf3d700 1 -- 172.21.15.3:0/3551790171 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408000e90 con 0x7f94180af2c0 2022-07-21T01:40:56.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941cf3d700 1 -- 172.21.15.3:0/3551790171 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940800d8c0 con 0x7f94180af2c0 2022-07-21T01:40:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 -- 172.21.15.3:0/3551790171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 msgr2=0x7f94180677d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 --2- 172.21.15.3:0/3551790171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 0x7f94180677d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f9408002f70 tx=0x7f9408006df0).stop 2022-07-21T01:40:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 -- 172.21.15.3:0/3551790171 shutdown_connections 2022-07-21T01:40:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 --2- 172.21.15.3:0/3551790171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 0x7f94180677d0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 --2- 172.21.15.3:0/3551790171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180b29e0 0x7f94180672e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 -- 172.21.15.3:0/3551790171 >> 172.21.15.3:0/3551790171 conn(0x7f94180613c0 msgr2=0x7f9418062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 -- 172.21.15.3:0/3551790171 shutdown_connections 2022-07-21T01:40:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 -- 172.21.15.3:0/3551790171 wait complete. 2022-07-21T01:40:56.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 Processor -- start 2022-07-21T01:40:56.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.720+0000 7f941ef41700 1 -- start start 2022-07-21T01:40:56.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941ef41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 0x7f94181410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:56.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941ef41700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180afef0 0x7f9418150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941ef41700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94180b29e0 0x7f94181425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941ef41700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181416f0 con 0x7f94180af2c0 2022-07-21T01:40:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941ef41700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9418150480 con 0x7f94180b29e0 2022-07-21T01:40:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941ef41700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9418150720 con 0x7f94180afef0 2022-07-21T01:40:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941df3f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 0x7f94181410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941df3f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 0x7f94181410c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48744/0 (socket says 172.21.15.3:48744) 2022-07-21T01:40:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941df3f700 1 -- 172.21.15.3:0/1036729050 learned_addr learned my addr 172.21.15.3:0/1036729050 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941e740700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94180b29e0 0x7f94181425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941d73e700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180afef0 0x7f9418150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941df3f700 1 -- 172.21.15.3:0/1036729050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180afef0 msgr2=0x7f9418150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941df3f700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180afef0 0x7f9418150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941df3f700 1 -- 172.21.15.3:0/1036729050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94180b29e0 msgr2=0x7f94181425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941df3f700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94180b29e0 0x7f94181425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941df3f700 1 -- 172.21.15.3:0/1036729050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940800b180 con 0x7f94180af2c0 2022-07-21T01:40:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941df3f700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 0x7f94181410c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f941400c000 tx=0x7f9414006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:56.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f9406ffd700 1 -- 172.21.15.3:0/1036729050 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9414011750 con 0x7f94180af2c0 2022-07-21T01:40:56.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f9406ffd700 1 -- 172.21.15.3:0/1036729050 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94140118b0 con 0x7f94180af2c0 2022-07-21T01:40:56.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94181509c0 con 0x7f94180af2c0 2022-07-21T01:40:56.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f9406ffd700 1 -- 172.21.15.3:0/1036729050 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9414011c10 con 0x7f94180af2c0 2022-07-21T01:40:56.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.724+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94180a2180 con 0x7f94180af2c0 2022-07-21T01:40:56.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.728+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9418150b50 con 0x7f94180af2c0 2022-07-21T01:40:56.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.728+0000 7f9406ffd700 1 -- 172.21.15.3:0/1036729050 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9414011d70 con 0x7f94180af2c0 2022-07-21T01:40:56.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.728+0000 7f9406ffd700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93f8053b00 0x7f93f8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:56.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.728+0000 7f9406ffd700 1 -- 172.21.15.3:0/1036729050 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f941406c170 con 0x7f94180af2c0 2022-07-21T01:40:56.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.728+0000 7f941d73e700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93f8053b00 0x7f93f8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:56.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.728+0000 7f941d73e700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93f8053b00 0x7f93f8055fc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f940801a000 tx=0x7f9408014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:56.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.736+0000 7f9406ffd700 1 -- 172.21.15.3:0/1036729050 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f94140431b0 con 0x7f94180af2c0 2022-07-21T01:40:56.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.868+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f941805f3d0 con 0x7f94180af2c0 2022-07-21T01:40:56.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f9406ffd700 1 -- 172.21.15.3:0/1036729050 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f94140467d0 con 0x7f94180af2c0 2022-07-21T01:40:56.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93f8053b00 msgr2=0x7f93f8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:56.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93f8053b00 0x7f93f8055fc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f940801a000 tx=0x7f9408014000).stop 2022-07-21T01:40:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 msgr2=0x7f94181410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 0x7f94181410c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f941400c000 tx=0x7f9414006df0).stop 2022-07-21T01:40:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 shutdown_connections 2022-07-21T01:40:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93f8053b00 0x7f93f8055fc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94180b29e0 0x7f94181425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94180afef0 0x7f9418150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 --2- 172.21.15.3:0/1036729050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94180af2c0 0x7f94181410c0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 >> 172.21.15.3:0/1036729050 conn(0x7f94180613c0 msgr2=0x7f9418142f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:56.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 shutdown_connections 2022-07-21T01:40:56.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.876+0000 7f941ef41700 1 -- 172.21.15.3:0/1036729050 wait complete. 2022-07-21T01:40:56.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-07-21T01:40:56.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b8091a700 1 Processor -- start 2022-07-21T01:40:56.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b8091a700 1 -- start start 2022-07-21T01:40:57.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b8091a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5b7c0b07a0 0x7f5b7c0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b8091a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5b7c147730 0x7f5b7c0acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b8091a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c147b80 con 0x7f5b7c0b5630 2022-07-21T01:40:57.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b8091a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7c145470 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b8091a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7c0ae180 con 0x7f5b7c147730 2022-07-21T01:40:57.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b7a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5b7c0b07a0 0x7f5b7c0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b7ad9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5b7c0b5630 0x7f5b7c067730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57032/0 (socket says 172.21.15.3:57032) 2022-07-21T01:40:57.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b7ad9d700 1 -- 172.21.15.3:0/3119363566 learned_addr learned my addr 172.21.15.3:0/3119363566 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:57.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b7a59c700 1 -- 172.21.15.3:0/3119363566 >> v2:172.21.15.143:3300/0 conn(0x7f5b7c147730 msgr2=0x7f5b7c0acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:57.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b7a59c700 1 --2- 172.21.15.3:0/3119363566 >> v2:172.21.15.143:3300/0 conn(0x7f5b7c147730 0x7f5b7c0acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:40:57.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b7a59c700 1 -- 172.21.15.3:0/3119363566 >> v1:172.21.15.3:6789/0 conn(0x7f5b7c0b5630 legacy=0x7f5b7c067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b79d9b700 1 -- 172.21.15.3:0/3119363566 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756586526 0 0) 0x7f5b7c147b80 con 0x7f5b7c0b5630 2022-07-21T01:40:57.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b7a59c700 1 -- 172.21.15.3:0/3119363566 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0b5a10 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b7a59c700 1 --2- 172.21.15.3:0/3119363566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 0x7f5b7c0af6e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f5b6c002f70 tx=0x7f5b6c006df0).ready entity=mon.0 client_cookie=e5cf9741a86bfb4c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:57.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b79d9b700 1 -- 172.21.15.3:0/3119363566 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b6c00d5c0 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b79d9b700 1 -- 172.21.15.3:0/3119363566 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c000e90 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:56.996+0000 7f5b79d9b700 1 -- 172.21.15.3:0/3119363566 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b6c00d910 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 -- 172.21.15.3:0/3119363566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 msgr2=0x7f5b7c0af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 --2- 172.21.15.3:0/3119363566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 0x7f5b7c0af6e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f5b6c002f70 tx=0x7f5b6c006df0).stop 2022-07-21T01:40:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 -- 172.21.15.3:0/3119363566 shutdown_connections 2022-07-21T01:40:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 --2- 172.21.15.3:0/3119363566 >> v2:172.21.15.143:3300/0 conn(0x7f5b7c147730 0x7f5b7c0acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:40:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 --2- 172.21.15.3:0/3119363566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 0x7f5b7c0af6e0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 -- 172.21.15.3:0/3119363566 >> 172.21.15.3:0/3119363566 conn(0x7f5b7c0626f0 msgr2=0x7f5b7c062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 -- 172.21.15.3:0/3119363566 shutdown_connections 2022-07-21T01:40:57.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 -- 172.21.15.3:0/3119363566 wait complete. 2022-07-21T01:40:57.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 Processor -- start 2022-07-21T01:40:57.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 -- start start 2022-07-21T01:40:57.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 0x7f5b7c140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b7c0b5630 0x7f5b7c15a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b7c147730 0x7f5b7c140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c144cc0 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c145f80 con 0x7f5b7c0b5630 2022-07-21T01:40:57.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b8091a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c146220 con 0x7f5b7c147730 2022-07-21T01:40:57.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 0x7f5b7c140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b7c0b5630 0x7f5b7c15a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 0x7f5b7c140cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48760/0 (socket says 172.21.15.3:48760) 2022-07-21T01:40:57.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7ad9d700 1 -- 172.21.15.3:0/2140607334 learned_addr learned my addr 172.21.15.3:0/2140607334 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:57.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7b59e700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b7c147730 0x7f5b7c140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7ad9d700 1 -- 172.21.15.3:0/2140607334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b7c147730 msgr2=0x7f5b7c140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7ad9d700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b7c147730 0x7f5b7c140130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7ad9d700 1 -- 172.21.15.3:0/2140607334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b7c0b5630 msgr2=0x7f5b7c15a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7ad9d700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b7c0b5630 0x7f5b7c15a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.000+0000 7f5b7ad9d700 1 -- 172.21.15.3:0/2140607334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c00b180 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b7ad9d700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 0x7f5b7c140cf0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f5b70000ed0 tx=0x7f5b70007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:57.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b6bfff700 1 -- 172.21.15.3:0/2140607334 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b700070d0 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b6bfff700 1 -- 172.21.15.3:0/2140607334 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b70007230 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b6bfff700 1 -- 172.21.15.3:0/2140607334 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b70005610 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c1464c0 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c15ae80 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7c0574e0 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b6bfff700 1 -- 172.21.15.3:0/2140607334 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5b70010070 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b6bfff700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b58053b00 0x7f5b58055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b6bfff700 1 -- 172.21.15.3:0/2140607334 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f5b7006b7e0 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.004+0000 7f5b7a59c700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b58053b00 0x7f5b58055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.008+0000 7f5b7a59c700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b58053b00 0x7f5b58055fc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f5b6c01a000 tx=0x7f5b6c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:57.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.012+0000 7f5b6bfff700 1 -- 172.21.15.3:0/2140607334 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5b70042880 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.180+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7f5b7c061020 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.588+0000 7f5b6bfff700 1 -- 172.21.15.3:0/2140607334 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v335) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5b70045ea0 con 0x7f5b7c0b07a0 2022-07-21T01:40:57.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b58053b00 msgr2=0x7f5b58055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b58053b00 0x7f5b58055fc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f5b6c01a000 tx=0x7f5b6c014000).stop 2022-07-21T01:40:57.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 msgr2=0x7f5b7c140cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 0x7f5b7c140cf0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f5b70000ed0 tx=0x7f5b70007ac0).stop 2022-07-21T01:40:57.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 shutdown_connections 2022-07-21T01:40:57.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b7c147730 0x7f5b7c140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5b58053b00 0x7f5b58055fc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b7c0b5630 0x7f5b7c15a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 --2- 172.21.15.3:0/2140607334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b7c0b07a0 0x7f5b7c140cf0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 >> 172.21.15.3:0/2140607334 conn(0x7f5b7c0626f0 msgr2=0x7f5b7c147ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:57.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 shutdown_connections 2022-07-21T01:40:57.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.592+0000 7f5b8091a700 1 -- 172.21.15.3:0/2140607334 wait complete. 2022-07-21T01:40:57.610 INFO:tasks.cephfs_test_runner:test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-21T01:40:57.610 INFO:tasks.cephfs_test_runner:Starting test: test_filtered_df (tasks.cephfs.test_misc.TestMisc) 2022-07-21T01:40:57.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-07-21T01:40:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41e88b700 1 Processor -- start 2022-07-21T01:40:57.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41e88b700 1 -- start start 2022-07-21T01:40:57.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41e88b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb4180b01a0 0x7fb4180a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41e88b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180afd60 0x7fb4180a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41e88b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4180a7250 con 0x7fb4180b2710 2022-07-21T01:40:57.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41e88b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4180a7480 con 0x7fb4180b01a0 2022-07-21T01:40:57.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41e88b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180a7720 con 0x7fb4180afd60 2022-07-21T01:40:57.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41d889700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb4180b2710 0x7fb41814afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57044/0 (socket says 172.21.15.3:57044) 2022-07-21T01:40:57.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41d889700 1 -- 172.21.15.3:0/1497893177 learned_addr learned my addr 172.21.15.3:0/1497893177 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:57.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.720+0000 7fb41d088700 1 --2- 172.21.15.3:0/1497893177 >> v2:172.21.15.143:3300/0 conn(0x7fb4180b01a0 0x7fb4180a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e08a700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180afd60 0x7fb4180a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974554098 0 0) 0x7fb4180a7250 con 0x7fb4180b2710 2022-07-21T01:40:57.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3fc003260 con 0x7fb4180b2710 2022-07-21T01:40:57.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170710387 0 0) 0x7fb3fc003260 con 0x7fb4180b2710 2022-07-21T01:40:57.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180afd60 msgr2=0x7fb4180a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:57.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180afd60 0x7fb4180a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 >> v2:172.21.15.143:3300/0 conn(0x7fb4180b01a0 msgr2=0x7fb4180a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:57.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 --2- 172.21.15.3:0/1497893177 >> v2:172.21.15.143:3300/0 conn(0x7fb4180b01a0 0x7fb4180a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4180a79c0 con 0x7fb4180b2710 2022-07-21T01:40:57.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fb40c001660 con 0x7fb4180b2710 2022-07-21T01:40:57.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 >> v1:172.21.15.3:6789/0 conn(0x7fb4180b2710 legacy=0x7fb41814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180afd60 0x7fb3fc003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc003c80 0x7fb3fc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3fc0049f0 0x7fb3fc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180a7480 con 0x7fb3fc0049f0 2022-07-21T01:40:57.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180a7720 con 0x7fb3fc003c80 2022-07-21T01:40:57.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc00b750 con 0x7fb4180afd60 2022-07-21T01:40:57.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e08a700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3fc0049f0 0x7fb3fc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41d088700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc003c80 0x7fb3fc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41d889700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180afd60 0x7fb3fc003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e08a700 1 -- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180afd60 msgr2=0x7fb3fc003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:57.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e08a700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180afd60 0x7fb3fc003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e08a700 1 -- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc003c80 msgr2=0x7fb3fc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:57.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e08a700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc003c80 0x7fb3fc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e08a700 1 -- 172.21.15.3:0/1497893177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc003260 con 0x7fb3fc0049f0 2022-07-21T01:40:57.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41d088700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc003c80 0x7fb3fc004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:40:57.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e08a700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3fc0049f0 0x7fb3fc004e70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7fb3fc0054d0 tx=0x7fb4100091d0).ready entity=mon.0 client_cookie=9a95426690220f3c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:57.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb41000e7e0 con 0x7fb3fc0049f0 2022-07-21T01:40:57.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41000e940 con 0x7fb3fc0049f0 2022-07-21T01:40:57.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41c887700 1 -- 172.21.15.3:0/1497893177 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb41000eca0 con 0x7fb3fc0049f0 2022-07-21T01:40:57.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e88b700 1 -- 172.21.15.3:0/1497893177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3fc0049f0 msgr2=0x7fb3fc004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e88b700 1 --2- 172.21.15.3:0/1497893177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3fc0049f0 0x7fb3fc004e70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7fb3fc0054d0 tx=0x7fb4100091d0).stop 2022-07-21T01:40:57.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e08a700 1 -- 172.21.15.3:0/1497893177 reap_dead start 2022-07-21T01:40:57.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e88b700 1 -- 172.21.15.3:0/1497893177 shutdown_connections 2022-07-21T01:40:57.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e88b700 1 -- 172.21.15.3:0/1497893177 >> 172.21.15.3:0/1497893177 conn(0x7fb418060310 msgr2=0x7fb4180b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:57.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e88b700 1 -- 172.21.15.3:0/1497893177 shutdown_connections 2022-07-21T01:40:57.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.724+0000 7fb41e88b700 1 -- 172.21.15.3:0/1497893177 wait complete. 2022-07-21T01:40:57.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 Processor -- start 2022-07-21T01:40:57.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 -- start start 2022-07-21T01:40:57.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb41814a9f0 0x7fb418067120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb418063c50 0x7fb418145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb418145af0 0x7fb418156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41814add0 con 0x7fb418145af0 2022-07-21T01:40:57.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418066740 con 0x7fb418063c50 2022-07-21T01:40:57.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180669e0 con 0x7fb41814a9f0 2022-07-21T01:40:57.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e08a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb418145af0 0x7fb418156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41d088700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb418063c50 0x7fb418145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e08a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb418145af0 0x7fb418156610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48778/0 (socket says 172.21.15.3:48778) 2022-07-21T01:40:57.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e08a700 1 -- 172.21.15.3:0/615784718 learned_addr learned my addr 172.21.15.3:0/615784718 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:57.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41d889700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb41814a9f0 0x7fb418067120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e08a700 1 -- 172.21.15.3:0/615784718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb41814a9f0 msgr2=0x7fb418067120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e08a700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb41814a9f0 0x7fb418067120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e08a700 1 -- 172.21.15.3:0/615784718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb418063c50 msgr2=0x7fb418145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:57.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e08a700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb418063c50 0x7fb418145600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:57.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e08a700 1 -- 172.21.15.3:0/615784718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb418066c80 con 0x7fb418145af0 2022-07-21T01:40:57.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e08a700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb418145af0 0x7fb418156610 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7fb418061eb0 tx=0x7fb41001a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:57.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb40affd700 1 -- 172.21.15.3:0/615784718 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb41000e7e0 con 0x7fb418145af0 2022-07-21T01:40:57.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb40affd700 1 -- 172.21.15.3:0/615784718 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41001b850 con 0x7fb418145af0 2022-07-21T01:40:57.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4180678b0 con 0x7fb418145af0 2022-07-21T01:40:57.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb40affd700 1 -- 172.21.15.3:0/615784718 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb410009600 con 0x7fb418145af0 2022-07-21T01:40:57.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.728+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb418146120 con 0x7fb418145af0 2022-07-21T01:40:57.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.732+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb418066e10 con 0x7fb418145af0 2022-07-21T01:40:57.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.732+0000 7fb40affd700 1 -- 172.21.15.3:0/615784718 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb41001bbb0 con 0x7fb418145af0 2022-07-21T01:40:57.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.732+0000 7fb40affd700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb400053b00 0x7fb400055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:57.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.732+0000 7fb40affd700 1 -- 172.21.15.3:0/615784718 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fb41006da70 con 0x7fb418145af0 2022-07-21T01:40:57.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.732+0000 7fb41d889700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb400053b00 0x7fb400055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:57.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.732+0000 7fb41d889700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb400053b00 0x7fb400055fc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7fb40c005000 tx=0x7fb41806a5f0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:57.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.736+0000 7fb40affd700 1 -- 172.21.15.3:0/615784718 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb410044b10 con 0x7fb418145af0 2022-07-21T01:40:57.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:57.904+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7fb4180574e0 con 0x7fb418145af0 2022-07-21T01:40:58.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.600+0000 7fb40affd700 1 -- 172.21.15.3:0/615784718 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v336) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb41006de60 con 0x7fb418145af0 2022-07-21T01:40:58.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb400053b00 msgr2=0x7fb400055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:58.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb400053b00 0x7fb400055fc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7fb40c005000 tx=0x7fb41806a5f0).stop 2022-07-21T01:40:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb418145af0 msgr2=0x7fb418156610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb418145af0 0x7fb418156610 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7fb418061eb0 tx=0x7fb41001a000).stop 2022-07-21T01:40:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 shutdown_connections 2022-07-21T01:40:58.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb418145af0 0x7fb418156610 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb400053b00 0x7fb400055fc0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb418063c50 0x7fb418145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 --2- 172.21.15.3:0/615784718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb41814a9f0 0x7fb418067120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 >> 172.21.15.3:0/615784718 conn(0x7fb418060310 msgr2=0x7fb4180adef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 shutdown_connections 2022-07-21T01:40:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.604+0000 7fb41e88b700 1 -- 172.21.15.3:0/615784718 wait complete. 2022-07-21T01:40:58.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:40:58.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a7271700 1 Processor -- start 2022-07-21T01:40:58.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a7271700 1 -- start start 2022-07-21T01:40:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a7271700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd3a0141af0 0x7fd3a00b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a7271700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a00a7960 con 0x7fd3a00b2050 2022-07-21T01:40:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a7271700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a00b07e0 con 0x7fd3a00b1460 2022-07-21T01:40:58.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a7271700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a00b18d0 con 0x7fd3a0141af0 2022-07-21T01:40:58.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a500d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd3a00b2050 0x7fd3a00a7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57064/0 (socket says 172.21.15.3:57064) 2022-07-21T01:40:58.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a500d700 1 -- 172.21.15.3:0/2739956312 learned_addr learned my addr 172.21.15.3:0/2739956312 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:58.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a580e700 1 --2- 172.21.15.3:0/2739956312 >> v2:172.21.15.143:3300/0 conn(0x7fd3a0141af0 0x7fd3a00b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:58.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd397fff700 1 -- 172.21.15.3:0/2739956312 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285136555 0 0) 0x7fd3a00a7960 con 0x7fd3a00b2050 2022-07-21T01:40:58.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd397fff700 1 -- 172.21.15.3:0/2739956312 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd384003260 con 0x7fd3a00b2050 2022-07-21T01:40:58.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a580e700 1 -- 172.21.15.3:0/2739956312 >> v1:172.21.15.143:6789/0 conn(0x7fd3a00b1460 legacy=0x7fd3a01413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:58.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a580e700 1 -- 172.21.15.3:0/2739956312 >> v1:172.21.15.3:6789/0 conn(0x7fd3a00b2050 legacy=0x7fd3a00a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:58.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a580e700 1 -- 172.21.15.3:0/2739956312 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a00b0a80 con 0x7fd3a0141af0 2022-07-21T01:40:58.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd3a580e700 1 --2- 172.21.15.3:0/2739956312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a0141af0 0x7fd3a00b0140 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd3980032f0 tx=0x7fd3980091d0).ready entity=mon.1 client_cookie=f3c37709fc6e5311 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:58.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd397fff700 1 -- 172.21.15.3:0/2739956312 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd39800b6c0 con 0x7fd3a0141af0 2022-07-21T01:40:58.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd397fff700 1 -- 172.21.15.3:0/2739956312 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39800b820 con 0x7fd3a0141af0 2022-07-21T01:40:58.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.732+0000 7fd397fff700 1 -- 172.21.15.3:0/2739956312 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd39800b990 con 0x7fd3a0141af0 2022-07-21T01:40:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 -- 172.21.15.3:0/2739956312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a0141af0 msgr2=0x7fd3a00b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 --2- 172.21.15.3:0/2739956312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a0141af0 0x7fd3a00b0140 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd3980032f0 tx=0x7fd3980091d0).stop 2022-07-21T01:40:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 -- 172.21.15.3:0/2739956312 shutdown_connections 2022-07-21T01:40:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 --2- 172.21.15.3:0/2739956312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a0141af0 0x7fd3a00b0140 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 -- 172.21.15.3:0/2739956312 >> 172.21.15.3:0/2739956312 conn(0x7fd3a005f840 msgr2=0x7fd3a005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:58.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 -- 172.21.15.3:0/2739956312 shutdown_connections 2022-07-21T01:40:58.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 -- 172.21.15.3:0/2739956312 wait complete. 2022-07-21T01:40:58.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 Processor -- start 2022-07-21T01:40:58.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 -- start start 2022-07-21T01:40:58.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a00b1460 0x7fd3a00af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:58.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a00b2050 0x7fd3a0149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:58.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a0141af0 0x7fd3a0149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:58.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a00abbe0 con 0x7fd3a0141af0 2022-07-21T01:40:58.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a0149c50 con 0x7fd3a00b2050 2022-07-21T01:40:58.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a7271700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a014d530 con 0x7fd3a00b1460 2022-07-21T01:40:58.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a580e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a0141af0 0x7fd3a0149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:58.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a580e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a0141af0 0x7fd3a0149760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48794/0 (socket says 172.21.15.3:48794) 2022-07-21T01:40:58.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a580e700 1 -- 172.21.15.3:0/3044840494 learned_addr learned my addr 172.21.15.3:0/3044840494 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:58.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a500d700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a00b1460 0x7fd3a00af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:58.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.736+0000 7fd3a480c700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a00b2050 0x7fd3a0149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:58.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a580e700 1 -- 172.21.15.3:0/3044840494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a00b1460 msgr2=0x7fd3a00af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:58.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a580e700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a00b1460 0x7fd3a00af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a580e700 1 -- 172.21.15.3:0/3044840494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a00b2050 msgr2=0x7fd3a0149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:58.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a580e700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a00b2050 0x7fd3a0149270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a580e700 1 -- 172.21.15.3:0/3044840494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd398009470 con 0x7fd3a0141af0 2022-07-21T01:40:58.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a580e700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a0141af0 0x7fd3a0149760 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7fd3a014b650 tx=0x7fd398016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:58.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3967fc700 1 -- 172.21.15.3:0/3044840494 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd39800b6c0 con 0x7fd3a0141af0 2022-07-21T01:40:58.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3967fc700 1 -- 172.21.15.3:0/3044840494 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3980105b0 con 0x7fd3a0141af0 2022-07-21T01:40:58.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a00a96c0 con 0x7fd3a0141af0 2022-07-21T01:40:58.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3967fc700 1 -- 172.21.15.3:0/3044840494 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd398010910 con 0x7fd3a0141af0 2022-07-21T01:40:58.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a00afff0 con 0x7fd3a0141af0 2022-07-21T01:40:58.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3967fc700 1 -- 172.21.15.3:0/3044840494 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd39800f070 con 0x7fd3a0141af0 2022-07-21T01:40:58.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a00a9850 con 0x7fd3a0141af0 2022-07-21T01:40:58.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3967fc700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd388053a10 0x7fd388055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:58.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3967fc700 1 -- 172.21.15.3:0/3044840494 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fd39806d260 con 0x7fd3a0141af0 2022-07-21T01:40:58.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a500d700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd388053a10 0x7fd388055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:58.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.740+0000 7fd3a500d700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd388053a10 0x7fd388055ed0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fd39c00b000 tx=0x7fd39c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:58.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.748+0000 7fd3967fc700 1 -- 172.21.15.3:0/3044840494 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd3980441f0 con 0x7fd3a0141af0 2022-07-21T01:40:58.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.880+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd3a00574e0 con 0x7fd3a0141af0 2022-07-21T01:40:58.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.888+0000 7fd3967fc700 1 -- 172.21.15.3:0/3044840494 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd3980150c0 con 0x7fd3a0141af0 2022-07-21T01:40:58.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd388053a10 msgr2=0x7fd388055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:58.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd388053a10 0x7fd388055ed0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fd39c00b000 tx=0x7fd39c009040).stop 2022-07-21T01:40:58.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a0141af0 msgr2=0x7fd3a0149760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:58.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a0141af0 0x7fd3a0149760 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7fd3a014b650 tx=0x7fd398016000).stop 2022-07-21T01:40:58.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 shutdown_connections 2022-07-21T01:40:58.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3a0141af0 0x7fd3a0149760 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd388053a10 0x7fd388055ed0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a00b2050 0x7fd3a0149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 --2- 172.21.15.3:0/3044840494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a00b1460 0x7fd3a00af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:58.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 >> 172.21.15.3:0/3044840494 conn(0x7fd3a005f840 msgr2=0x7fd3a00ad5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:58.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 shutdown_connections 2022-07-21T01:40:58.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:58.892+0000 7fd3a7271700 1 -- 172.21.15.3:0/3044840494 wait complete. 2022-07-21T01:40:58.905 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:40:58.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:58.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:58.944 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:58.966 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:40:58.989 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:40:58.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:58.991 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:58.991 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:40:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:40:59.000 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:40:59.002 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:40:59.003 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:40:59.003 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:40:59.004 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:40:59.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:40:59.015 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:40:59.015 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:40:59.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:40:59.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b9816700 1 Processor -- start 2022-07-21T01:40:59.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b9816700 1 -- start start 2022-07-21T01:40:59.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b9816700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f45b40614b0 0x7f45b4061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b9816700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f45b41405d0 0x7f45b4140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b9816700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b40aee80 con 0x7f45b40ad750 2022-07-21T01:40:59.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b9816700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b40b70b0 con 0x7f45b40614b0 2022-07-21T01:40:59.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b9816700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b40af0b0 con 0x7f45b41405d0 2022-07-21T01:40:59.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b8814700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f45b40614b0 0x7f45b4061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b8814700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f45b40614b0 0x7f45b4061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48798/0 (socket says 172.21.15.3:48798) 2022-07-21T01:40:59.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b8814700 1 -- 172.21.15.3:0/2600871789 learned_addr learned my addr 172.21.15.3:0/2600871789 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:59.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b9015700 1 --2- 172.21.15.3:0/2600871789 >> v2:172.21.15.143:3300/0 conn(0x7f45b41405d0 0x7f45b4140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b8814700 1 -- 172.21.15.3:0/2600871789 >> v2:172.21.15.143:3300/0 conn(0x7f45b41405d0 msgr2=0x7f45b4140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b8814700 1 --2- 172.21.15.3:0/2600871789 >> v2:172.21.15.143:3300/0 conn(0x7f45b41405d0 0x7f45b4140a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b8814700 1 -- 172.21.15.3:0/2600871789 >> v1:172.21.15.143:6789/0 conn(0x7f45b40ad750 legacy=0x7f45b40b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b8814700 1 -- 172.21.15.3:0/2600871789 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b40af350 con 0x7f45b40614b0 2022-07-21T01:40:59.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b8814700 1 --2- 172.21.15.3:0/2600871789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b40614b0 0x7f45b4061890 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f45b40aed30 tx=0x7f45a4006eb0).ready entity=mon.0 client_cookie=f496bc0a55cf878f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b37fe700 1 -- 172.21.15.3:0/2600871789 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45a400d610 con 0x7f45b40614b0 2022-07-21T01:40:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b37fe700 1 -- 172.21.15.3:0/2600871789 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a4000e90 con 0x7f45b40614b0 2022-07-21T01:40:59.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.124+0000 7f45b37fe700 1 -- 172.21.15.3:0/2600871789 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45a400d960 con 0x7f45b40614b0 2022-07-21T01:40:59.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 -- 172.21.15.3:0/2600871789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b40614b0 msgr2=0x7f45b4061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 --2- 172.21.15.3:0/2600871789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b40614b0 0x7f45b4061890 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f45b40aed30 tx=0x7f45a4006eb0).stop 2022-07-21T01:40:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 -- 172.21.15.3:0/2600871789 shutdown_connections 2022-07-21T01:40:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 --2- 172.21.15.3:0/2600871789 >> v2:172.21.15.143:3300/0 conn(0x7f45b41405d0 0x7f45b4140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 --2- 172.21.15.3:0/2600871789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b40614b0 0x7f45b4061890 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 -- 172.21.15.3:0/2600871789 >> 172.21.15.3:0/2600871789 conn(0x7f45b4066f70 msgr2=0x7f45b4067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:59.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 -- 172.21.15.3:0/2600871789 shutdown_connections 2022-07-21T01:40:59.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 -- 172.21.15.3:0/2600871789 wait complete. 2022-07-21T01:40:59.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 Processor -- start 2022-07-21T01:40:59.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 -- start start 2022-07-21T01:40:59.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45b40614b0 0x7f45b4152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45b40ad750 0x7f45b4143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b41405d0 0x7f45b4143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4152df0 con 0x7f45b41405d0 2022-07-21T01:40:59.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.128+0000 7f45b9816700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4153020 con 0x7f45b40ad750 2022-07-21T01:40:59.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9816700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4142030 con 0x7f45b40614b0 2022-07-21T01:40:59.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9015700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b41405d0 0x7f45b4143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9015700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b41405d0 0x7f45b4143960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48808/0 (socket says 172.21.15.3:48808) 2022-07-21T01:40:59.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9015700 1 -- 172.21.15.3:0/3969626779 learned_addr learned my addr 172.21.15.3:0/3969626779 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:59.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b8814700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45b40614b0 0x7f45b4152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b3fff700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45b40ad750 0x7f45b4143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9015700 1 -- 172.21.15.3:0/3969626779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45b40614b0 msgr2=0x7f45b4152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9015700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45b40614b0 0x7f45b4152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9015700 1 -- 172.21.15.3:0/3969626779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45b40ad750 msgr2=0x7f45b4143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9015700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45b40ad750 0x7f45b4143470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9015700 1 -- 172.21.15.3:0/3969626779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a400b1e0 con 0x7f45b41405d0 2022-07-21T01:40:59.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9015700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b41405d0 0x7f45b4143960 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f45ac00a000 tx=0x7f45ac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:59.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b1ffb700 1 -- 172.21.15.3:0/3969626779 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45ac011780 con 0x7f45b41405d0 2022-07-21T01:40:59.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b1ffb700 1 -- 172.21.15.3:0/3969626779 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac0118e0 con 0x7f45b41405d0 2022-07-21T01:40:59.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b41422d0 con 0x7f45b41405d0 2022-07-21T01:40:59.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b1ffb700 1 -- 172.21.15.3:0/3969626779 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45ac011c40 con 0x7f45b41405d0 2022-07-21T01:40:59.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b4143e50 con 0x7f45b41405d0 2022-07-21T01:40:59.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b4141740 con 0x7f45b41405d0 2022-07-21T01:40:59.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b1ffb700 1 -- 172.21.15.3:0/3969626779 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f45ac01c460 con 0x7f45b41405d0 2022-07-21T01:40:59.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b1ffb700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f45940539e0 0x7f4594055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b1ffb700 1 -- 172.21.15.3:0/3969626779 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f45ac06c0f0 con 0x7f45b41405d0 2022-07-21T01:40:59.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.132+0000 7f45b8814700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f45940539e0 0x7f4594055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.136+0000 7f45b8814700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f45940539e0 0x7f4594055ea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f45a401c000 tx=0x7f45a4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:59.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.140+0000 7f45b1ffb700 1 -- 172.21.15.3:0/3969626779 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45ac043090 con 0x7f45b41405d0 2022-07-21T01:40:59.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.268+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f45b4142460 con 0x7f45b41405d0 2022-07-21T01:40:59.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b1ffb700 1 -- 172.21.15.3:0/3969626779 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v104) v1 ==== 81+0+29219 (secure 0 0 0) 0x7f45ac0466b0 con 0x7f45b41405d0 2022-07-21T01:40:59.273 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:59.274 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:40:59.274 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 104, 2022-07-21T01:40:59.274 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:40:59.274 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:40:59.275 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:40:56.573329+0000", 2022-07-21T01:40:59.275 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:59.275 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:40:59.276 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:40:59.276 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:40:59.276 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:40:59.276 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:40:59.277 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:40:59.277 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:40:59.277 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:40:59.277 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:59.277 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:40:59.278 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:40:59.278 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:40:59.278 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:40:59.279 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:40:59.279 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 18, 2022-07-21T01:40:59.279 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:40:59.279 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:40:59.279 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:40:59.280 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:40:59.280 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:40:59.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.280 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:40:59.281 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:40:59.281 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:40:59.281 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:59.281 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:59.281 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:59.282 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:59.282 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:59.282 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:59.282 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:59.283 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:59.283 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:40:59.283 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:40:59.283 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:40:59.283 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:40:59.284 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:40:59.284 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:59.284 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:59.284 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:59.285 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:59.285 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:59.285 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:59.285 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:59.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.286 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:40:59.286 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:59.286 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:59.286 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:59.287 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:59.287 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:59.287 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:59.287 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:59.288 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:59.288 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:59.288 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:59.288 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:59.288 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:59.289 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:59.289 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:59.289 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:59.289 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:59.289 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:59.290 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:59.290 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:59.290 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:59.290 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:59.291 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:59.291 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:59.291 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:59.291 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:59.292 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:59.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.292 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:59.292 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:59.292 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:59.293 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:59.293 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:59.293 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:59.293 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:59.293 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:59.294 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:59.294 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:59.294 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:59.294 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:40:59.295 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:40:59.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.295 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:59.296 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:40:59.296 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.297 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:40:59.297 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:40:59.298 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:40:59.298 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:40:59.298 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:40:59.298 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:40:59.299 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:40:59.299 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:40:59.299 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:40:59.299 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:40:59.299 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:40:59.300 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:40:59.300 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:40:59.300 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:40:59.300 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:40:59.301 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:40:59.301 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:40:59.301 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:40:59.301 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:40:59.301 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:40:59.302 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:40:59.302 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:40:59.302 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:40:59.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.303 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:40:59.303 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:40:59.303 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:40:59.303 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:40:59.303 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:40:59.304 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:40:59.304 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:40:59.304 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:40:59.304 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:40:59.305 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:40:59.305 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:40:59.305 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:40:59.305 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:40:59.306 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:40:59.306 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:40:59.306 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:40:59.306 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:40:59.307 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:40:59.307 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:40:59.307 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:40:59.307 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:40:59.308 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:40:59.308 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:40:59.308 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:40:59.308 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:40:59.309 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:40:59.309 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:40:59.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.309 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:40:59.309 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:40:59.310 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:40:59.310 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:40:59.311 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:40:59.311 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:40:59.311 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:40:59.311 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:40:59.311 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:40:59.312 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:40:59.312 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:40:59.312 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:40:59.312 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:40:59.313 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:40:59.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.313 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:59.313 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:40:59.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.314 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:59.314 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:40:59.314 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:59.315 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:59.315 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:59.315 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:59.315 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:59.315 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:59.316 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:59.316 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:59.316 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:59.316 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:59.317 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:59.317 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.317 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.318 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:40:59.318 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:59.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.318 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.319 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:40:59.319 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:59.319 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.319 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.320 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:59.320 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.320 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.321 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:40:59.321 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:59.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.322 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.322 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:40:59.322 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:59.322 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.322 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.323 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:59.323 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.324 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.324 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:40:59.324 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:59.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.325 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.325 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:40:59.325 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:59.325 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.326 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.326 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:59.326 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.327 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.327 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:40:59.327 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:59.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.328 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.328 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:40:59.328 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:40:59.329 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.329 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.329 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:40:59.330 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:40:59.330 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:40:59.330 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:40:59.330 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:59.331 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:59.331 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:59.331 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.332 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:59.332 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:40:59.332 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:59.332 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:59.333 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:59.333 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:59.333 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:59.333 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:59.333 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:59.334 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:59.334 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:59.334 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:59.334 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:59.335 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.335 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.335 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:40:59.336 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:59.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.336 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.336 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:40:59.337 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:59.337 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.337 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.338 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:59.338 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.338 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.339 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:40:59.339 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:59.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.339 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.340 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:40:59.340 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:59.340 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.340 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.341 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:59.341 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.341 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.342 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:40:59.342 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:59.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.342 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.343 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:40:59.343 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:59.343 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.343 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.344 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:59.344 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.344 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.345 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:40:59.345 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:59.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.345 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.346 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:40:59.346 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:40:59.346 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.346 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.347 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:40:59.347 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:40:59.347 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:40:59.348 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:40:59.348 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:59.348 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:59.348 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:59.349 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.349 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:59.350 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:40:59.350 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:59.350 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:59.350 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:59.351 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:59.351 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:59.351 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:59.351 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:59.352 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:59.352 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:59.352 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:59.352 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:59.353 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.353 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.353 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:40:59.354 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:59.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.354 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:40:59.355 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:59.355 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.355 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.356 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:59.356 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.356 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.357 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:40:59.357 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:59.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.358 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.358 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:40:59.358 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:59.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.358 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.359 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:59.359 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.359 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.360 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:40:59.360 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:59.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.361 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.361 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:40:59.361 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:59.361 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.361 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.362 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:40:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:59.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.364 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:40:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:40:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:40:59.366 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:40:59.366 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:40:59.366 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:40:59.366 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:59.367 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:59.367 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:59.368 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.369 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:59.370 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:40:59.370 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:59.371 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:59.371 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:59.372 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:59.372 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:59.373 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:59.373 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:40:59.374 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:59.374 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:59.375 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:59.375 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:59.376 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.377 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.377 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:40:59.378 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:59.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.379 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.380 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:40:59.380 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:59.381 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.381 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.382 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:59.383 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.384 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:40:59.385 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:59.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.386 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.386 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:40:59.387 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:59.387 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.388 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.389 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:59.389 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.390 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.391 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:40:59.391 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:59.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.393 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.393 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:40:59.394 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:59.394 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.395 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.396 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:59.396 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.398 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:40:59.398 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:59.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.400 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.400 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:40:59.400 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:40:59.400 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.401 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.401 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:40:59.401 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:40:59.402 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:40:59.402 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:40:59.402 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:59.403 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:59.403 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:59.403 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.404 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:59.404 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:40:59.404 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:59.405 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:59.406 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:59.406 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:59.407 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:59.407 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:59.408 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:59.408 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 98, 2022-07-21T01:40:59.409 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:59.409 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:59.409 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:59.410 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.411 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.411 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:40:59.411 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:59.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.412 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.412 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:40:59.413 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:59.413 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.413 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.414 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:59.414 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.415 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:40:59.415 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:59.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.415 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.416 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:40:59.416 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:59.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.416 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.417 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:59.417 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.418 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.418 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:40:59.418 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:59.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.419 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.419 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:40:59.419 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:59.419 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.420 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.420 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:59.420 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.421 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:40:59.421 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:59.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.422 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:40:59.422 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:40:59.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.423 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.423 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:40:59.424 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:40:59.424 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:40:59.424 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:40:59.424 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:59.425 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:59.425 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:59.425 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.426 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:59.426 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:40:59.426 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:59.427 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:59.427 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:59.427 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:59.427 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:59.427 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:59.428 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:59.428 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 88, 2022-07-21T01:40:59.428 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:59.429 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:59.429 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:59.429 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.430 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.430 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:40:59.431 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:59.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.431 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.431 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:40:59.432 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:59.432 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.432 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.433 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:59.433 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.434 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:40:59.434 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:59.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.434 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.435 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:40:59.435 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:59.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.435 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.436 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:59.436 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.437 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:40:59.437 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:59.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.438 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.438 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:40:59.438 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:59.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.439 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.439 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:59.439 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.440 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:40:59.440 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:59.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.441 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.441 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:40:59.441 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:40:59.442 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.442 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.442 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:40:59.442 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:40:59.443 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:40:59.443 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:40:59.443 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:59.443 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:59.444 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:59.444 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:59.445 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:40:59.445 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:59.446 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:59.446 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:59.446 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:59.446 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:59.447 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:59.447 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:59.447 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 98, 2022-07-21T01:40:59.448 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:59.448 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:59.448 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:59.449 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.449 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.449 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:40:59.450 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:59.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.451 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:40:59.451 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:59.451 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.451 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.452 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:59.452 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.452 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.453 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:40:59.453 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:59.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.454 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:40:59.454 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:59.454 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.455 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.455 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:59.455 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.456 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:40:59.456 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:59.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.457 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:40:59.458 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:59.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.459 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:59.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:40:59.460 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:59.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.461 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:40:59.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:40:59.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.461 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.462 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:40:59.462 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:40:59.462 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:40:59.463 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:40:59.463 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:59.463 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:59.463 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:59.464 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:59.465 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:40:59.465 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:40:59.465 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:40:59.465 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:40:59.466 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:40:59.466 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:40:59.466 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:40:59.466 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:40:59.467 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 99, 2022-07-21T01:40:59.467 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:40:59.467 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:40:59.467 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:40:59.468 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.468 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.468 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:40:59.468 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:59.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.469 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.469 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:40:59.470 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:59.470 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.470 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.471 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:40:59.471 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:40:59.472 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:59.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.473 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:40:59.474 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:59.474 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.474 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.475 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:40:59.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.475 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:40:59.476 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:59.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.477 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:40:59.477 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:59.477 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.478 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.478 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:40:59.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:40:59.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.479 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:40:59.479 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:40:59.479 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:59.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.480 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:40:59.480 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:40:59.480 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:40:59.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.481 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.481 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:40:59.481 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:40:59.482 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:40:59.482 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:40:59.482 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:40:59.482 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:40:59.483 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:40:59.483 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:40:59.483 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.483 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:59.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:40:59.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:40:59.484 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:59.485 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:59.485 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:59.485 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:59.485 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:59.485 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:40:59.486 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:59.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.486 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:40:59.487 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:59.487 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:59.487 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:59.487 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:59.488 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:59.488 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:40:59.488 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:59.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.489 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:40:59.489 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:59.489 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:59.489 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:59.490 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:59.490 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:59.490 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:40:59.490 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:59.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.491 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:40:59.491 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:59.491 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:59.492 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:59.492 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:59.492 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:59.492 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:40:59.493 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:59.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.493 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:40:59.493 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:59.494 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:59.494 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:59.494 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:59.494 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:59.495 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:40:59.495 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:59.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.496 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:40:59.496 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:59.496 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:59.496 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:59.496 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:59.497 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:59.497 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:40:59.497 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:59.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.498 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:40:59.498 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:59.498 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:59.499 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:59.499 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:59.499 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:59.499 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:40:59.499 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:59.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:40:59.500 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:40:59.500 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:40:59.501 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:40:59.501 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:40:59.501 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:40:59.501 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:40:59.502 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:40:59.502 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:40:59.502 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.502 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:40:59.503 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:40:59.503 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:40:59.503 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:40:59.503 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:40:59.503 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:40:59.504 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6837/1935398064": "2022-07-22T01:40:53.240884+0000", 2022-07-21T01:40:59.504 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6836/1935398064": "2022-07-22T01:40:53.240884+0000" 2022-07-21T01:40:59.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.504 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:40:59.505 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:40:59.505 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:40:59.505 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:59.505 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:40:59.506 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:40:59.506 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:59.506 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:40:59.506 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:59.507 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:59.507 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:40:59.507 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:40:59.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.508 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:40:59.508 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:40:59.508 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:40:59.508 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:40:59.509 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:40:59.509 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:40:59.509 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:40:59.509 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:40:59.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:40:59.510 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:40:59.510 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:40:59.510 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:40:59.510 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:40:59.511 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:40:59.511 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:40:59.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f45940539e0 msgr2=0x7f4594055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f45940539e0 0x7f4594055ea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f45a401c000 tx=0x7f45a4015000).stop 2022-07-21T01:40:59.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b41405d0 msgr2=0x7f45b4143960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b41405d0 0x7f45b4143960 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f45ac00a000 tx=0x7f45ac008040).stop 2022-07-21T01:40:59.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 shutdown_connections 2022-07-21T01:40:59.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45b41405d0 0x7f45b4143960 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f45940539e0 0x7f4594055ea0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45b40ad750 0x7f45b4143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 --2- 172.21.15.3:0/3969626779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45b40614b0 0x7f45b4152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 >> 172.21.15.3:0/3969626779 conn(0x7f45b4066f70 msgr2=0x7f45b405cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:59.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 shutdown_connections 2022-07-21T01:40:59.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.272+0000 7f45b9816700 1 -- 172.21.15.3:0/3969626779 wait complete. 2022-07-21T01:40:59.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:59.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.628+0000 7fef7759e700 1 Processor -- start 2022-07-21T01:40:59.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.628+0000 7fef7759e700 1 -- start start 2022-07-21T01:40:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.628+0000 7fef7759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef78061890 0x7fef781466b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.628+0000 7fef7759e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef780aee80 con 0x7fef780614b0 2022-07-21T01:40:59.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.628+0000 7fef7759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef780b70b0 con 0x7fef780ad750 2022-07-21T01:40:59.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.628+0000 7fef7759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef780af0b0 con 0x7fef78061890 2022-07-21T01:40:59.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fef780614b0 0x7fef780b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57094/0 (socket says 172.21.15.3:57094) 2022-07-21T01:40:59.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7659c700 1 -- 172.21.15.3:0/2200554728 learned_addr learned my addr 172.21.15.3:0/2200554728 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:59.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef76d9d700 1 --2- 172.21.15.3:0/2200554728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef78061890 0x7fef781466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7559a700 1 -- 172.21.15.3:0/2200554728 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124928424 0 0) 0x7fef780aee80 con 0x7fef780614b0 2022-07-21T01:40:59.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7559a700 1 -- 172.21.15.3:0/2200554728 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef58003260 con 0x7fef780614b0 2022-07-21T01:40:59.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7559a700 1 -- 172.21.15.3:0/2200554728 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754042019 0 0) 0x7fef780b70b0 con 0x7fef780ad750 2022-07-21T01:40:59.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef76d9d700 1 -- 172.21.15.3:0/2200554728 >> v1:172.21.15.143:6789/0 conn(0x7fef780ad750 legacy=0x7fef781405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef76d9d700 1 -- 172.21.15.3:0/2200554728 >> v1:172.21.15.3:6789/0 conn(0x7fef780614b0 legacy=0x7fef780b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef76d9d700 1 -- 172.21.15.3:0/2200554728 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef780af350 con 0x7fef78061890 2022-07-21T01:40:59.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef76d9d700 1 --2- 172.21.15.3:0/2200554728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef78061890 0x7fef781466b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fef700032f0 tx=0x7fef700091d0).ready entity=mon.2 client_cookie=1e6279acacbb13d4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:59.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7559a700 1 -- 172.21.15.3:0/2200554728 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef7000b6c0 con 0x7fef78061890 2022-07-21T01:40:59.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7559a700 1 -- 172.21.15.3:0/2200554728 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7000b820 con 0x7fef78061890 2022-07-21T01:40:59.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7559a700 1 -- 172.21.15.3:0/2200554728 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef7000b990 con 0x7fef78061890 2022-07-21T01:40:59.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7759e700 1 -- 172.21.15.3:0/2200554728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef78061890 msgr2=0x7fef781466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7759e700 1 --2- 172.21.15.3:0/2200554728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef78061890 0x7fef781466b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fef700032f0 tx=0x7fef700091d0).stop 2022-07-21T01:40:59.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7759e700 1 -- 172.21.15.3:0/2200554728 shutdown_connections 2022-07-21T01:40:59.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7759e700 1 --2- 172.21.15.3:0/2200554728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef78061890 0x7fef781466b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7759e700 1 -- 172.21.15.3:0/2200554728 >> 172.21.15.3:0/2200554728 conn(0x7fef78066f70 msgr2=0x7fef78067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:59.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7759e700 1 -- 172.21.15.3:0/2200554728 shutdown_connections 2022-07-21T01:40:59.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.632+0000 7fef7759e700 1 -- 172.21.15.3:0/2200554728 wait complete. 2022-07-21T01:40:59.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7759e700 1 Processor -- start 2022-07-21T01:40:59.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7759e700 1 -- start start 2022-07-21T01:40:59.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef780614b0 0x7fef78143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78061890 0x7fef780ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef780ad750 0x7fef780acae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef780b21c0 con 0x7fef780614b0 2022-07-21T01:40:59.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef780b2460 con 0x7fef78061890 2022-07-21T01:40:59.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef780abeb0 con 0x7fef780ad750 2022-07-21T01:40:59.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef780614b0 0x7fef78143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef75d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78061890 0x7fef780ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef75d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78061890 0x7fef780ac5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:54268/0 (socket says 172.21.15.3:54268) 2022-07-21T01:40:59.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef7659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef780614b0 0x7fef78143510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48818/0 (socket says 172.21.15.3:48818) 2022-07-21T01:40:59.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef75d9b700 1 -- 172.21.15.3:0/3181447211 learned_addr learned my addr 172.21.15.3:0/3181447211 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:59.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.636+0000 7fef76d9d700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef780ad750 0x7fef780acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef7659c700 1 -- 172.21.15.3:0/3181447211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef780ad750 msgr2=0x7fef780acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef7659c700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef780ad750 0x7fef780acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef7659c700 1 -- 172.21.15.3:0/3181447211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78061890 msgr2=0x7fef780ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef7659c700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78061890 0x7fef780ac5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef7659c700 1 -- 172.21.15.3:0/3181447211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef70009470 con 0x7fef780614b0 2022-07-21T01:40:59.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef7659c700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef780614b0 0x7fef78143510 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7fef68001110 tx=0x7fef680079e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:59.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef6f7fe700 1 -- 172.21.15.3:0/3181447211 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef6800cdb0 con 0x7fef780614b0 2022-07-21T01:40:59.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef6f7fe700 1 -- 172.21.15.3:0/3181447211 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef680070d0 con 0x7fef780614b0 2022-07-21T01:40:59.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef780ac150 con 0x7fef780614b0 2022-07-21T01:40:59.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef6f7fe700 1 -- 172.21.15.3:0/3181447211 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef68007430 con 0x7fef780614b0 2022-07-21T01:40:59.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef781530d0 con 0x7fef780614b0 2022-07-21T01:40:59.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef780b2620 con 0x7fef780614b0 2022-07-21T01:40:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef6f7fe700 1 -- 172.21.15.3:0/3181447211 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef68010070 con 0x7fef780614b0 2022-07-21T01:40:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef6f7fe700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef5c053a10 0x7fef5c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef75d9b700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef5c053a10 0x7fef5c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef6f7fe700 1 -- 172.21.15.3:0/3181447211 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fef6806b160 con 0x7fef780614b0 2022-07-21T01:40:59.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.640+0000 7fef75d9b700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef5c053a10 0x7fef5c055ed0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fef6000c000 tx=0x7fef60008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:59.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.648+0000 7fef6f7fe700 1 -- 172.21.15.3:0/3181447211 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef68042200 con 0x7fef780614b0 2022-07-21T01:40:59.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.808+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef780af5f0 con 0x7fef780614b0 2022-07-21T01:40:59.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.808+0000 7fef6f7fe700 1 -- 172.21.15.3:0/3181447211 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+1553 (secure 0 0 0) 0x7fef68045820 con 0x7fef780614b0 2022-07-21T01:40:59.812 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:40:59.813 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/485774043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":485774043},{"type":"v1","addr":"172.21.15.143:6833","nonce":485774043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":6936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/756498280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":756498280},{"type":"v1","addr":"172.21.15.3:6835","nonce":756498280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":7170,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/2697523537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2697523537},{"type":"v1","addr":"172.21.15.3:6837","nonce":2697523537}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51}],"filesystems":[]} 2022-07-21T01:40:59.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef5c053a10 msgr2=0x7fef5c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef5c053a10 0x7fef5c055ed0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fef6000c000 tx=0x7fef60008040).stop 2022-07-21T01:40:59.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef780614b0 msgr2=0x7fef78143510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef780614b0 0x7fef78143510 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7fef68001110 tx=0x7fef680079e0).stop 2022-07-21T01:40:59.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 shutdown_connections 2022-07-21T01:40:59.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef780ad750 0x7fef780acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef5c053a10 0x7fef5c055ed0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78061890 0x7fef780ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 --2- 172.21.15.3:0/3181447211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef780614b0 0x7fef78143510 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 >> 172.21.15.3:0/3181447211 conn(0x7fef78066f70 msgr2=0x7fef78141140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:59.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 shutdown_connections 2022-07-21T01:40:59.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.812+0000 7fef7759e700 1 -- 172.21.15.3:0/3181447211 wait complete. 2022-07-21T01:40:59.817 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 51 2022-07-21T01:40:59.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:40:59.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.920+0000 7f8869590700 1 Processor -- start 2022-07-21T01:40:59.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.920+0000 7f8869590700 1 -- start start 2022-07-21T01:40:59.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88640b6420 0x7f8864061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b41f0 0x7f8864061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88640b60e0 con 0x7f8864061fe0 2022-07-21T01:40:59.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88640b2100 con 0x7f88640b6420 2022-07-21T01:40:59.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88640b23a0 con 0x7f88640b41f0 2022-07-21T01:40:59.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8863fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88640b6420 0x7f8864061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f88637fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b41f0 0x7f8864061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8868d8f700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8864061fe0 0x7f88641465c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:59226/0 (socket says 172.21.15.3:59226) 2022-07-21T01:40:59.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8868d8f700 1 -- 172.21.15.3:0/713819420 learned_addr learned my addr 172.21.15.3:0/713819420 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:40:59.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f88637fe700 1 -- 172.21.15.3:0/713819420 >> v2:172.21.15.143:3300/0 conn(0x7f88640b6420 msgr2=0x7f8864061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:40:59.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f88637fe700 1 --2- 172.21.15.3:0/713819420 >> v2:172.21.15.143:3300/0 conn(0x7f88640b6420 0x7f8864061600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8862ffd700 1 -- 172.21.15.3:0/713819420 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812775660 0 0) 0x7f885c001490 con 0x7f8864061fe0 2022-07-21T01:40:59.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f88637fe700 1 -- 172.21.15.3:0/713819420 >> v1:172.21.15.143:6789/0 conn(0x7f8864061fe0 legacy=0x7f88641465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f88637fe700 1 -- 172.21.15.3:0/713819420 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88640b2640 con 0x7f88640b41f0 2022-07-21T01:40:59.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f88637fe700 1 --2- 172.21.15.3:0/713819420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b41f0 0x7f8864061af0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f884c002f70 tx=0x7f884c006df0).ready entity=mon.2 client_cookie=f8a47315867115e5 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:59.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8862ffd700 1 -- 172.21.15.3:0/713819420 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f884c00d570 con 0x7f88640b41f0 2022-07-21T01:40:59.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8862ffd700 1 -- 172.21.15.3:0/713819420 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884c000e90 con 0x7f88640b41f0 2022-07-21T01:40:59.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8862ffd700 1 -- 172.21.15.3:0/713819420 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f884c00d8c0 con 0x7f88640b41f0 2022-07-21T01:40:59.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 -- 172.21.15.3:0/713819420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b41f0 msgr2=0x7f8864061af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 --2- 172.21.15.3:0/713819420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b41f0 0x7f8864061af0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f884c002f70 tx=0x7f884c006df0).stop 2022-07-21T01:40:59.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 -- 172.21.15.3:0/713819420 shutdown_connections 2022-07-21T01:40:59.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 --2- 172.21.15.3:0/713819420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b41f0 0x7f8864061af0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 --2- 172.21.15.3:0/713819420 >> v2:172.21.15.143:3300/0 conn(0x7f88640b6420 0x7f8864061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 -- 172.21.15.3:0/713819420 >> 172.21.15.3:0/713819420 conn(0x7f886405c620 msgr2=0x7f886405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:40:59.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 -- 172.21.15.3:0/713819420 shutdown_connections 2022-07-21T01:40:59.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.924+0000 7f8869590700 1 -- 172.21.15.3:0/713819420 wait complete. 2022-07-21T01:40:59.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8869590700 1 Processor -- start 2022-07-21T01:40:59.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8869590700 1 -- start start 2022-07-21T01:40:59.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8869590700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8864061fe0 0x7f88640b7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8869590700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88640b41f0 0x7f8864156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8869590700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b6420 0x7f8864156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8869590700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8864143340 con 0x7f8864061fe0 2022-07-21T01:40:59.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8869590700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88641436e0 con 0x7f88640b41f0 2022-07-21T01:40:59.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8869590700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8864143980 con 0x7f88640b6420 2022-07-21T01:40:59.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8863fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8864061fe0 0x7f88640b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8863fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8864061fe0 0x7f88640b7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48832/0 (socket says 172.21.15.3:48832) 2022-07-21T01:40:59.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8863fff700 1 -- 172.21.15.3:0/2031123570 learned_addr learned my addr 172.21.15.3:0/2031123570 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:40:59.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f88637fe700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88640b41f0 0x7f8864156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8868d8f700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b6420 0x7f8864156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8863fff700 1 -- 172.21.15.3:0/2031123570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b6420 msgr2=0x7f8864156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8863fff700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b6420 0x7f8864156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8863fff700 1 -- 172.21.15.3:0/2031123570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88640b41f0 msgr2=0x7f8864156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:40:59.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8863fff700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88640b41f0 0x7f8864156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:40:59.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.928+0000 7f8863fff700 1 -- 172.21.15.3:0/2031123570 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c00b180 con 0x7f8864061fe0 2022-07-21T01:40:59.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f8863fff700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8864061fe0 0x7f88640b7730 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f885400c000 tx=0x7f8854006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:59.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f88617fa700 1 -- 172.21.15.3:0/2031123570 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8854011750 con 0x7f8864061fe0 2022-07-21T01:40:59.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f88617fa700 1 -- 172.21.15.3:0/2031123570 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88540118b0 con 0x7f8864061fe0 2022-07-21T01:40:59.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88640b07f0 con 0x7f8864061fe0 2022-07-21T01:40:59.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f88617fa700 1 -- 172.21.15.3:0/2031123570 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8854011c10 con 0x7f8864061fe0 2022-07-21T01:40:59.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8864143c20 con 0x7f8864061fe0 2022-07-21T01:40:59.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8864155ea0 con 0x7f8864061fe0 2022-07-21T01:40:59.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f88617fa700 1 -- 172.21.15.3:0/2031123570 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8854011d70 con 0x7f8864061fe0 2022-07-21T01:40:59.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f88617fa700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8848053a70 0x7f8848055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:40:59.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f88617fa700 1 -- 172.21.15.3:0/2031123570 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f885406c170 con 0x7f8864061fe0 2022-07-21T01:40:59.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f88637fe700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8848053a70 0x7f8848055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:40:59.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.932+0000 7f88637fe700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8848053a70 0x7f8848055f30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f884c01a000 tx=0x7f884c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:40:59.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:40:59.940+0000 7f88617fa700 1 -- 172.21.15.3:0/2031123570 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8864155ea0 con 0x7f8864061fe0 2022-07-21T01:41:00.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.104+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8864155ea0 con 0x7f8864061fe0 2022-07-21T01:41:00.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.104+0000 7f88617fa700 1 -- 172.21.15.3:0/2031123570 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+1553 (secure 0 0 0) 0x7f8864155ea0 con 0x7f8864061fe0 2022-07-21T01:41:00.107 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:00.108 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/485774043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":485774043},{"type":"v1","addr":"172.21.15.143:6833","nonce":485774043}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":6936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/756498280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":756498280},{"type":"v1","addr":"172.21.15.3:6835","nonce":756498280}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":45},{"gid":7170,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/2697523537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":2697523537},{"type":"v1","addr":"172.21.15.3:6837","nonce":2697523537}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":51}],"filesystems":[]} 2022-07-21T01:41:00.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8848053a70 msgr2=0x7f8848055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:00.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8848053a70 0x7f8848055f30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f884c01a000 tx=0x7f884c014000).stop 2022-07-21T01:41:00.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8864061fe0 msgr2=0x7f88640b7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:00.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8864061fe0 0x7f88640b7730 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f885400c000 tx=0x7f8854006df0).stop 2022-07-21T01:41:00.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 shutdown_connections 2022-07-21T01:41:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88640b6420 0x7f8864156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8848053a70 0x7f8848055f30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88640b41f0 0x7f8864156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 --2- 172.21.15.3:0/2031123570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8864061fe0 0x7f88640b7730 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:00.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 >> 172.21.15.3:0/2031123570 conn(0x7f886405c620 msgr2=0x7f8864146b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:00.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 shutdown_connections 2022-07-21T01:41:00.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.108+0000 7f8869590700 1 -- 172.21.15.3:0/2031123570 wait complete. 2022-07-21T01:41:00.113 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 51 2022-07-21T01:41:00.126 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:41:00.126 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:41:00.127 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:41:00.127 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:41:00.127 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:41:00.128 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:41:00.128 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:41:00.128 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:41:00.129 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:41:00.130 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:41:00.130 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:41:00.131 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:41:00.143 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:41:00.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:41:00.145 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:41:00.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:41:00.147 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:41:00.147 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:41:00.190 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:41:00.191 DEBUG:teuthology.parallel:result is None 2022-07-21T01:41:00.194 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:41:00.194 DEBUG:teuthology.parallel:result is None 2022-07-21T01:41:00.196 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:41:00.197 DEBUG:teuthology.parallel:result is None 2022-07-21T01:41:00.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:41:00.264 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:41:00.265 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:41:00.274 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:41:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a1282b700 1 Processor -- start 2022-07-21T01:41:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a1282b700 1 -- start start 2022-07-21T01:41:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a1282b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2a0c0aa5b0 0x7f2a0c0667a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a1282b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0aadf0 con 0x7f2a0c1437f0 2022-07-21T01:41:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a1282b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c143fd0 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a1282b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0c05f9c0 con 0x7f2a0c0aa5b0 2022-07-21T01:41:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a0bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2a0c1437f0 0x7f2a0c0aeaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57144/0 (socket says 172.21.15.3:57144) 2022-07-21T01:41:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a0bfff700 1 -- 172.21.15.3:0/3829236857 learned_addr learned my addr 172.21.15.3:0/3829236857 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a10dc8700 1 --2- 172.21.15.3:0/3829236857 >> v2:172.21.15.143:3300/0 conn(0x7f2a0c0aa5b0 0x7f2a0c0667a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a0affd700 1 -- 172.21.15.3:0/3829236857 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710523695 0 0) 0x7f2a0c0aadf0 con 0x7f2a0c1437f0 2022-07-21T01:41:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a0affd700 1 -- 172.21.15.3:0/3829236857 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f8003260 con 0x7f2a0c1437f0 2022-07-21T01:41:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a10dc8700 1 -- 172.21.15.3:0/3829236857 >> v1:172.21.15.143:6789/0 conn(0x7f2a0c0af1b0 legacy=0x7f2a0c0a9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a10dc8700 1 -- 172.21.15.3:0/3829236857 >> v1:172.21.15.3:6789/0 conn(0x7f2a0c1437f0 legacy=0x7f2a0c0aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a0affd700 1 -- 172.21.15.3:0/3829236857 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544505471 0 0) 0x7f2a0c143fd0 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a10dc8700 1 -- 172.21.15.3:0/3829236857 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c0b1bd0 con 0x7f2a0c0aa5b0 2022-07-21T01:41:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a10dc8700 1 --2- 172.21.15.3:0/3829236857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c0aa5b0 0x7f2a0c0667a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f2a00001780 tx=0x7f2a040091d0).ready entity=mon.1 client_cookie=55c5ab7cf1e66c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a0affd700 1 -- 172.21.15.3:0/3829236857 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a000015e0 con 0x7f2a0c0aa5b0 2022-07-21T01:41:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a0affd700 1 -- 172.21.15.3:0/3829236857 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0400b580 con 0x7f2a0c0aa5b0 2022-07-21T01:41:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.296+0000 7f2a0affd700 1 -- 172.21.15.3:0/3829236857 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a0400b6f0 con 0x7f2a0c0aa5b0 2022-07-21T01:41:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 -- 172.21.15.3:0/3829236857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c0aa5b0 msgr2=0x7f2a0c0667a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 --2- 172.21.15.3:0/3829236857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c0aa5b0 0x7f2a0c0667a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f2a00001780 tx=0x7f2a040091d0).stop 2022-07-21T01:41:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 -- 172.21.15.3:0/3829236857 shutdown_connections 2022-07-21T01:41:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 --2- 172.21.15.3:0/3829236857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c0aa5b0 0x7f2a0c0667a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:00.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 -- 172.21.15.3:0/3829236857 >> 172.21.15.3:0/3829236857 conn(0x7f2a0c0b5570 msgr2=0x7f2a0c0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 -- 172.21.15.3:0/3829236857 shutdown_connections 2022-07-21T01:41:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 -- 172.21.15.3:0/3829236857 wait complete. 2022-07-21T01:41:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 Processor -- start 2022-07-21T01:41:00.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 -- start start 2022-07-21T01:41:00.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a0c0aa5b0 0x7f2a0c13ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a0c0af1b0 0x7f2a0c0b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c1437f0 0x7f2a0c13df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c13bf10 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c13d750 con 0x7f2a0c1437f0 2022-07-21T01:41:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a1282b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c13c470 con 0x7f2a0c0aa5b0 2022-07-21T01:41:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a0c0af1b0 0x7f2a0c0b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:00.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a0c0aa5b0 0x7f2a0c13ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a0c0af1b0 0x7f2a0c0b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48872/0 (socket says 172.21.15.3:48872) 2022-07-21T01:41:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0b7fe700 1 -- 172.21.15.3:0/58663915 learned_addr learned my addr 172.21.15.3:0/58663915 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a10dc8700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c1437f0 0x7f2a0c13df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:00.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0b7fe700 1 -- 172.21.15.3:0/58663915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a0c0aa5b0 msgr2=0x7f2a0c13ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0b7fe700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a0c0aa5b0 0x7f2a0c13ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0b7fe700 1 -- 172.21.15.3:0/58663915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c1437f0 msgr2=0x7f2a0c13df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:00.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0b7fe700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c1437f0 0x7f2a0c13df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0b7fe700 1 -- 172.21.15.3:0/58663915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04009470 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0bfff700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a0c0aa5b0 0x7f2a0c13ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a10dc8700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c1437f0 0x7f2a0c13df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.300+0000 7f2a0b7fe700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a0c0af1b0 0x7f2a0c0b4440 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7f2a0c13c600 tx=0x7f29fc007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:00.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a097fa700 1 -- 172.21.15.3:0/58663915 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29fc012400 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a097fa700 1 -- 172.21.15.3:0/58663915 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29fc012560 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a097fa700 1 -- 172.21.15.3:0/58663915 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29fc0128c0 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c13d4b0 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c1409d0 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c0aaa00 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a097fa700 1 -- 172.21.15.3:0/58663915 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f29fc019030 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a097fa700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f29ec0539e0 0x7f29ec055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a097fa700 1 -- 172.21.15.3:0/58663915 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f29fc06b1f0 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a0bfff700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f29ec0539e0 0x7f29ec055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.304+0000 7f2a0bfff700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f29ec0539e0 0x7f29ec055ea0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f2a0000d000 tx=0x7f2a00001880).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:00.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.312+0000 7f2a097fa700 1 -- 172.21.15.3:0/58663915 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29fc042190 con 0x7f2a0c0af1b0 2022-07-21T01:41:00.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:00.452+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f2a0c0574e0 con 0x7f2a0c0af1b0 2022-07-21T01:41:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.296+0000 7f2a097fa700 1 -- 172.21.15.3:0/58663915 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v106) v1 ==== 97+0+0 (secure 0 0 0) 0x7f29fc0457b0 con 0x7f2a0c0af1b0 2022-07-21T01:41:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.296+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f29ec0539e0 msgr2=0x7f29ec055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.296+0000 7f2a1282b700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f29ec0539e0 0x7f29ec055ea0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f2a0000d000 tx=0x7f2a00001880).stop 2022-07-21T01:41:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.296+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a0c0af1b0 msgr2=0x7f2a0c0b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.296+0000 7f2a1282b700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a0c0af1b0 0x7f2a0c0b4440 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7f2a0c13c600 tx=0x7f29fc007b50).stop 2022-07-21T01:41:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.296+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 shutdown_connections 2022-07-21T01:41:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.296+0000 7f2a1282b700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f29ec0539e0 0x7f29ec055ea0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.300+0000 7f2a1282b700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c1437f0 0x7f2a0c13df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.300+0000 7f2a1282b700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a0c0af1b0 0x7f2a0c0b4440 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.300+0000 7f2a1282b700 1 --2- 172.21.15.3:0/58663915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a0c0aa5b0 0x7f2a0c13ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.300+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 >> 172.21.15.3:0/58663915 conn(0x7f2a0c0b5570 msgr2=0x7f2a0c0600d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:01.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.300+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 shutdown_connections 2022-07-21T01:41:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.300+0000 7f2a1282b700 1 -- 172.21.15.3:0/58663915 wait complete. 2022-07-21T01:41:01.304 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:41:01.316 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:41:01.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:41:01.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f511090d700 1 Processor -- start 2022-07-21T01:41:01.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f511090d700 1 -- start start 2022-07-21T01:41:01.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f511090d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f510c0b3260 0x7f510c1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f511090d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0ad000 0x7f510c13fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f511090d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c143ab0 con 0x7f510c0b08a0 2022-07-21T01:41:01.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f511090d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f510c0b0c80 con 0x7f510c0b3260 2022-07-21T01:41:01.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f511090d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510c0ae810 con 0x7f510c0ad000 2022-07-21T01:41:01.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f510c0b3260 0x7f510c1435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510ad9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f510c0b08a0 0x7f510c142eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57160/0 (socket says 172.21.15.3:57160) 2022-07-21T01:41:01.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510ad9d700 1 -- 172.21.15.3:0/3435252425 learned_addr learned my addr 172.21.15.3:0/3435252425 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:01.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510b59e700 1 --2- 172.21.15.3:0/3435252425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0ad000 0x7f510c13fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510a59c700 1 -- 172.21.15.3:0/3435252425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0ad000 msgr2=0x7f510c13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:01.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510a59c700 1 --2- 172.21.15.3:0/3435252425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0ad000 0x7f510c13fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510a59c700 1 -- 172.21.15.3:0/3435252425 >> v1:172.21.15.3:6789/0 conn(0x7f510c0b08a0 legacy=0x7f510c142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510a59c700 1 -- 172.21.15.3:0/3435252425 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c0aeab0 con 0x7f510c0b3260 2022-07-21T01:41:01.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510b59e700 1 --2- 172.21.15.3:0/3435252425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0ad000 0x7f510c13fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:01.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f510a59c700 1 --2- 172.21.15.3:0/3435252425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 0x7f510c1435c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f50f40032f0 tx=0x7f50f40091d0).ready entity=mon.0 client_cookie=b15adffac4e4f2ab server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:01.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f5109d9b700 1 -- 172.21.15.3:0/3435252425 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50f4009ea0 con 0x7f510c0b3260 2022-07-21T01:41:01.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f5109d9b700 1 -- 172.21.15.3:0/3435252425 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f400b8a0 con 0x7f510c0b3260 2022-07-21T01:41:01.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.424+0000 7f5109d9b700 1 -- 172.21.15.3:0/3435252425 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50f400ba10 con 0x7f510c0b3260 2022-07-21T01:41:01.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 -- 172.21.15.3:0/3435252425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 msgr2=0x7f510c1435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 --2- 172.21.15.3:0/3435252425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 0x7f510c1435c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f50f40032f0 tx=0x7f50f40091d0).stop 2022-07-21T01:41:01.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 -- 172.21.15.3:0/3435252425 shutdown_connections 2022-07-21T01:41:01.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 --2- 172.21.15.3:0/3435252425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0ad000 0x7f510c13fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 --2- 172.21.15.3:0/3435252425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 0x7f510c1435c0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 -- 172.21.15.3:0/3435252425 >> 172.21.15.3:0/3435252425 conn(0x7f510c0612d0 msgr2=0x7f510c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:01.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 -- 172.21.15.3:0/3435252425 shutdown_connections 2022-07-21T01:41:01.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 -- 172.21.15.3:0/3435252425 wait complete. 2022-07-21T01:41:01.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 Processor -- start 2022-07-21T01:41:01.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.428+0000 7f511090d700 1 -- start start 2022-07-21T01:41:01.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f511090d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f510c0ad000 0x7f510c13f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f511090d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0b08a0 0x7f510c1397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f511090d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 0x7f510c139ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f511090d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510c14e250 con 0x7f510c0b3260 2022-07-21T01:41:01.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f511090d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510c14e4f0 con 0x7f510c0ad000 2022-07-21T01:41:01.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f511090d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510c14e790 con 0x7f510c0b08a0 2022-07-21T01:41:01.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 0x7f510c139ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 0x7f510c139ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48890/0 (socket says 172.21.15.3:48890) 2022-07-21T01:41:01.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510b59e700 1 -- 172.21.15.3:0/4084908123 learned_addr learned my addr 172.21.15.3:0/4084908123 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:01.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510ad9d700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f510c0ad000 0x7f510c13f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510a59c700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0b08a0 0x7f510c1397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510b59e700 1 -- 172.21.15.3:0/4084908123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0b08a0 msgr2=0x7f510c1397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510b59e700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0b08a0 0x7f510c1397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510b59e700 1 -- 172.21.15.3:0/4084908123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f510c0ad000 msgr2=0x7f510c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510b59e700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f510c0ad000 0x7f510c13f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510b59e700 1 -- 172.21.15.3:0/4084908123 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f4009470 con 0x7f510c0b3260 2022-07-21T01:41:01.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f510b59e700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 0x7f510c139ce0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f5104007000 tx=0x7f5104008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:01.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f5103fff700 1 -- 172.21.15.3:0/4084908123 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5104015070 con 0x7f510c0b3260 2022-07-21T01:41:01.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f5103fff700 1 -- 172.21.15.3:0/4084908123 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510400ce80 con 0x7f510c0b3260 2022-07-21T01:41:01.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f5103fff700 1 -- 172.21.15.3:0/4084908123 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5104010770 con 0x7f510c0b3260 2022-07-21T01:41:01.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510c14ea30 con 0x7f510c0b3260 2022-07-21T01:41:01.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510c138110 con 0x7f510c0b3260 2022-07-21T01:41:01.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.432+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510c14ebc0 con 0x7f510c0b3260 2022-07-21T01:41:01.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.436+0000 7f5103fff700 1 -- 172.21.15.3:0/4084908123 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f51040108d0 con 0x7f510c0b3260 2022-07-21T01:41:01.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.436+0000 7f5103fff700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f50f00539e0 0x7f50f0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.436+0000 7f5103fff700 1 -- 172.21.15.3:0/4084908123 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f510406ba20 con 0x7f510c0b3260 2022-07-21T01:41:01.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.436+0000 7f510ad9d700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f50f00539e0 0x7f50f0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.436+0000 7f510ad9d700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f50f00539e0 0x7f50f0055ea0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f50fc00d000 tx=0x7f50fc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:01.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.444+0000 7f5103fff700 1 -- 172.21.15.3:0/4084908123 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5104042ac0 con 0x7f510c0b3260 2022-07-21T01:41:01.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.616+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f510c0574e0 con 0x7f510c0b3260 2022-07-21T01:41:01.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.616+0000 7f5103fff700 1 -- 172.21.15.3:0/4084908123 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v28) v1 ==== 80+0+4255 (secure 0 0 0) 0x7f51040460e0 con 0x7f510c0b3260 2022-07-21T01:41:01.620 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:01.620 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:41:01.620 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:41:01.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.621 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:41:01.621 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:41:01.621 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.622 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:01.622 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.622 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.623 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.623 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.624 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:41:01.624 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:41:01.624 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.624 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:01.625 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.625 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.625 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.626 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:41:01.626 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:41:01.627 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.627 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:01.627 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.627 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.628 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.628 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.629 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:41:01.629 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:41:01.629 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.629 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.630 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.630 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.631 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:41:01.631 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:41:01.631 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.632 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.632 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.632 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.632 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.633 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:41:01.633 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:41:01.634 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.634 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.634 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.634 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.635 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:41:01.636 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:41:01.636 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.636 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.636 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.637 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.637 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.638 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:41:01.638 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:41:01.638 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.638 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.639 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.639 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.640 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:41:01.640 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:41:01.640 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.641 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.641 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.641 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.642 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:41:01.642 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:41:01.643 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.643 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.643 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.643 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.644 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:41:01.645 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:41:01.645 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.645 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.646 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.646 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.647 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.647 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:41:01.648 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:41:01.648 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.648 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:01.648 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:41:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:41:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:41:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:41:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.650 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:41:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:41:01.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:41:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:41:01.653 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:41:01.654 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.655 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:41:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:41:01.657 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:41:01.658 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:41:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:01.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:01.660 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:41:01.661 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:41:01.661 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:01.661 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:41:01.662 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:41:01.662 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:01.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:01.663 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:01.664 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:41:01.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f50f00539e0 msgr2=0x7f50f0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f50f00539e0 0x7f50f0055ea0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f50fc00d000 tx=0x7f50fc009040).stop 2022-07-21T01:41:01.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 msgr2=0x7f510c139ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 0x7f510c139ce0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f5104007000 tx=0x7f5104008040).stop 2022-07-21T01:41:01.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 shutdown_connections 2022-07-21T01:41:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f510c0b3260 0x7f510c139ce0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f510c0b08a0 0x7f510c1397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f50f00539e0 0x7f50f0055ea0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 --2- 172.21.15.3:0/4084908123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f510c0ad000 0x7f510c13f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 >> 172.21.15.3:0/4084908123 conn(0x7f510c0612d0 msgr2=0x7f510c1413e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:01.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 shutdown_connections 2022-07-21T01:41:01.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.620+0000 7f511090d700 1 -- 172.21.15.3:0/4084908123 wait complete. 2022-07-21T01:41:01.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:01.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe465c3700 1 Processor -- start 2022-07-21T01:41:01.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe465c3700 1 -- start start 2022-07-21T01:41:01.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe465c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400b1f40 0x7fbe400b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe465c3700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe400b04e0 con 0x7fbe40148530 2022-07-21T01:41:01.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe465c3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe400b0780 con 0x7fbe400a7250 2022-07-21T01:41:01.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe465c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400a9250 con 0x7fbe400b1f40 2022-07-21T01:41:01.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbe40148530 0x7fbe400b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57174/0 (socket says 172.21.15.3:57174) 2022-07-21T01:41:01.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3f7fe700 1 -- 172.21.15.3:0/3193246271 learned_addr learned my addr 172.21.15.3:0/3193246271 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:01.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3ffff700 1 --2- 172.21.15.3:0/3193246271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400b1f40 0x7fbe400b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3effd700 1 -- 172.21.15.3:0/3193246271 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223013721 0 0) 0x7fbe400b04e0 con 0x7fbe40148530 2022-07-21T01:41:01.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3effd700 1 -- 172.21.15.3:0/3193246271 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe24003260 con 0x7fbe40148530 2022-07-21T01:41:01.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3effd700 1 -- 172.21.15.3:0/3193246271 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253695318 0 0) 0x7fbe400b0780 con 0x7fbe400a7250 2022-07-21T01:41:01.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3ffff700 1 -- 172.21.15.3:0/3193246271 >> v1:172.21.15.143:6789/0 conn(0x7fbe400a7250 legacy=0x7fbe401448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3ffff700 1 -- 172.21.15.3:0/3193246271 >> v1:172.21.15.3:6789/0 conn(0x7fbe40148530 legacy=0x7fbe400b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3ffff700 1 -- 172.21.15.3:0/3193246271 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe400b0a20 con 0x7fbe400b1f40 2022-07-21T01:41:01.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3ffff700 1 --2- 172.21.15.3:0/3193246271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400b1f40 0x7fbe400b1160 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fbe38001b80 tx=0x7fbe28006df0).ready entity=mon.2 client_cookie=b7a46f9c3aa8224d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:01.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3effd700 1 -- 172.21.15.3:0/3193246271 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe380019e0 con 0x7fbe400b1f40 2022-07-21T01:41:01.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3effd700 1 -- 172.21.15.3:0/3193246271 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe28000e90 con 0x7fbe400b1f40 2022-07-21T01:41:01.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.776+0000 7fbe3effd700 1 -- 172.21.15.3:0/3193246271 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe2800d4a0 con 0x7fbe400b1f40 2022-07-21T01:41:01.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 -- 172.21.15.3:0/3193246271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400b1f40 msgr2=0x7fbe400b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 --2- 172.21.15.3:0/3193246271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400b1f40 0x7fbe400b1160 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fbe38001b80 tx=0x7fbe28006df0).stop 2022-07-21T01:41:01.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 -- 172.21.15.3:0/3193246271 shutdown_connections 2022-07-21T01:41:01.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 --2- 172.21.15.3:0/3193246271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400b1f40 0x7fbe400b1160 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 -- 172.21.15.3:0/3193246271 >> 172.21.15.3:0/3193246271 conn(0x7fbe4005f840 msgr2=0x7fbe4005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:01.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 -- 172.21.15.3:0/3193246271 shutdown_connections 2022-07-21T01:41:01.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 -- 172.21.15.3:0/3193246271 wait complete. 2022-07-21T01:41:01.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 Processor -- start 2022-07-21T01:41:01.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 -- start start 2022-07-21T01:41:01.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400a7250 0x7fbe400aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe400b1f40 0x7fbe400ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe40148530 0x7fbe40062c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400ac1b0 con 0x7fbe400b1f40 2022-07-21T01:41:01.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400ac450 con 0x7fbe40148530 2022-07-21T01:41:01.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe465c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400ac6f0 con 0x7fbe400a7250 2022-07-21T01:41:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe3f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe400b1f40 0x7fbe400ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe3ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400a7250 0x7fbe400aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe3f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe400b1f40 0x7fbe400ab130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48900/0 (socket says 172.21.15.3:48900) 2022-07-21T01:41:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe3f7fe700 1 -- 172.21.15.3:0/3011203312 learned_addr learned my addr 172.21.15.3:0/3011203312 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.780+0000 7fbe44b60700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe40148530 0x7fbe40062c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3f7fe700 1 -- 172.21.15.3:0/3011203312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400a7250 msgr2=0x7fbe400aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3f7fe700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400a7250 0x7fbe400aea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3f7fe700 1 -- 172.21.15.3:0/3011203312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe40148530 msgr2=0x7fbe40062c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3f7fe700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe40148530 0x7fbe40062c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3f7fe700 1 -- 172.21.15.3:0/3011203312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2800b180 con 0x7fbe400b1f40 2022-07-21T01:41:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3f7fe700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe400b1f40 0x7fbe400ab130 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7fbe380016f0 tx=0x7fbe38001d00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3d7fa700 1 -- 172.21.15.3:0/3011203312 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe38004ab0 con 0x7fbe400b1f40 2022-07-21T01:41:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3d7fa700 1 -- 172.21.15.3:0/3011203312 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe38004c10 con 0x7fbe400b1f40 2022-07-21T01:41:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3d7fa700 1 -- 172.21.15.3:0/3011203312 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe38007170 con 0x7fbe400b1f40 2022-07-21T01:41:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe40065400 con 0x7fbe400b1f40 2022-07-21T01:41:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4015a220 con 0x7fbe400b1f40 2022-07-21T01:41:01.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe400629a0 con 0x7fbe400b1f40 2022-07-21T01:41:01.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.784+0000 7fbe3d7fa700 1 -- 172.21.15.3:0/3011203312 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbe38010070 con 0x7fbe400b1f40 2022-07-21T01:41:01.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.788+0000 7fbe3d7fa700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe18053b00 0x7fbe18055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:01.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.788+0000 7fbe3d7fa700 1 -- 172.21.15.3:0/3011203312 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fbe3806b6f0 con 0x7fbe400b1f40 2022-07-21T01:41:01.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.788+0000 7fbe3ffff700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe18053b00 0x7fbe18055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:01.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.788+0000 7fbe3ffff700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe18053b00 0x7fbe18055fc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7fbe28019000 tx=0x7fbe400b0670).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:01.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.792+0000 7fbe3d7fa700 1 -- 172.21.15.3:0/3011203312 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbe400629a0 con 0x7fbe400b1f40 2022-07-21T01:41:01.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.916+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe4015a220 con 0x7fbe400b1f40 2022-07-21T01:41:01.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.916+0000 7fbe3d7fa700 1 -- 172.21.15.3:0/3011203312 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+13416 (secure 0 0 0) 0x7fbe4015a220 con 0x7fbe400b1f40 2022-07-21T01:41:01.920 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:01.921 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":106,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:01.286626+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:01.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe18053b00 msgr2=0x7fbe18055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe18053b00 0x7fbe18055fc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7fbe28019000 tx=0x7fbe400b0670).stop 2022-07-21T01:41:01.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe400b1f40 msgr2=0x7fbe400ab130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:01.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe400b1f40 0x7fbe400ab130 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7fbe380016f0 tx=0x7fbe38001d00).stop 2022-07-21T01:41:01.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 shutdown_connections 2022-07-21T01:41:01.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe18053b00 0x7fbe18055fc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe40148530 0x7fbe40062c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe400b1f40 0x7fbe400ab130 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 --2- 172.21.15.3:0/3011203312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe400a7250 0x7fbe400aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:01.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 >> 172.21.15.3:0/3011203312 conn(0x7fbe4005f840 msgr2=0x7fbe400a76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:01.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 shutdown_connections 2022-07-21T01:41:01.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:01.920+0000 7fbe465c3700 1 -- 172.21.15.3:0/3011203312 wait complete. 2022-07-21T01:41:01.935 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:41:01.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:41:02.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.040+0000 7fa7863ff700 1 Processor -- start 2022-07-21T01:41:02.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.040+0000 7fa7863ff700 1 -- start start 2022-07-21T01:41:02.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800b0b20 0x7fa7800afd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7800ac520 con 0x7fa7800ae870 2022-07-21T01:41:02.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7800b17a0 con 0x7fa7800b13c0 2022-07-21T01:41:02.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800b0450 con 0x7fa7800b0b20 2022-07-21T01:41:02.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa785bfe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa7800ae870 0x7fa7800aece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57188/0 (socket says 172.21.15.3:57188) 2022-07-21T01:41:02.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa785bfe700 1 -- 172.21.15.3:0/4101534110 learned_addr learned my addr 172.21.15.3:0/4101534110 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:02.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa784bfc700 1 --2- 172.21.15.3:0/4101534110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800b0b20 0x7fa7800afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa777fff700 1 -- 172.21.15.3:0/4101534110 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501197501 0 0) 0x7fa7800ac520 con 0x7fa7800ae870 2022-07-21T01:41:02.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa777fff700 1 -- 172.21.15.3:0/4101534110 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa764003260 con 0x7fa7800ae870 2022-07-21T01:41:02.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa777fff700 1 -- 172.21.15.3:0/4101534110 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137525675 0 0) 0x7fa7800b17a0 con 0x7fa7800b13c0 2022-07-21T01:41:02.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa777fff700 1 -- 172.21.15.3:0/4101534110 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7800ac520 con 0x7fa7800b13c0 2022-07-21T01:41:02.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa784bfc700 1 -- 172.21.15.3:0/4101534110 >> v1:172.21.15.143:6789/0 conn(0x7fa7800b13c0 legacy=0x7fa780142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa777fff700 1 -- 172.21.15.3:0/4101534110 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3228036015 0 0) 0x7fa764003260 con 0x7fa7800ae870 2022-07-21T01:41:02.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa784bfc700 1 -- 172.21.15.3:0/4101534110 >> v1:172.21.15.3:6789/0 conn(0x7fa7800ae870 legacy=0x7fa7800aece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa784bfc700 1 -- 172.21.15.3:0/4101534110 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800b06f0 con 0x7fa7800b0b20 2022-07-21T01:41:02.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa784bfc700 1 --2- 172.21.15.3:0/4101534110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800b0b20 0x7fa7800afd80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa778001490 tx=0x7fa770006df0).ready entity=mon.2 client_cookie=9f57ebb48d0a3050 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa777fff700 1 -- 172.21.15.3:0/4101534110 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7780019e0 con 0x7fa7800b0b20 2022-07-21T01:41:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa777fff700 1 -- 172.21.15.3:0/4101534110 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa770000e90 con 0x7fa7800b0b20 2022-07-21T01:41:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa777fff700 1 -- 172.21.15.3:0/4101534110 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa770012460 con 0x7fa7800b0b20 2022-07-21T01:41:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 -- 172.21.15.3:0/4101534110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800b0b20 msgr2=0x7fa7800afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 --2- 172.21.15.3:0/4101534110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800b0b20 0x7fa7800afd80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa778001490 tx=0x7fa770006df0).stop 2022-07-21T01:41:02.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 -- 172.21.15.3:0/4101534110 shutdown_connections 2022-07-21T01:41:02.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 --2- 172.21.15.3:0/4101534110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800b0b20 0x7fa7800afd80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 -- 172.21.15.3:0/4101534110 >> 172.21.15.3:0/4101534110 conn(0x7fa7800626f0 msgr2=0x7fa780062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:02.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 -- 172.21.15.3:0/4101534110 shutdown_connections 2022-07-21T01:41:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.044+0000 7fa7863ff700 1 -- 172.21.15.3:0/4101534110 wait complete. 2022-07-21T01:41:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 Processor -- start 2022-07-21T01:41:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 -- start start 2022-07-21T01:41:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800ae870 0x7fa780152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7800b0b20 0x7fa780140890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7800b13c0 0x7fa7801470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa780153610 con 0x7fa7800b13c0 2022-07-21T01:41:02.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78013c990 con 0x7fa7800b0b20 2022-07-21T01:41:02.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa780152b50 con 0x7fa7800ae870 2022-07-21T01:41:02.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa785bfe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7800b13c0 0x7fa7801470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa784bfc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7800b0b20 0x7fa780140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa785bfe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7800b13c0 0x7fa7801470c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48916/0 (socket says 172.21.15.3:48916) 2022-07-21T01:41:02.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa785bfe700 1 -- 172.21.15.3:0/743864771 learned_addr learned my addr 172.21.15.3:0/743864771 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:02.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7853fd700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800ae870 0x7fa780152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa785bfe700 1 -- 172.21.15.3:0/743864771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800ae870 msgr2=0x7fa780152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa785bfe700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800ae870 0x7fa780152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa785bfe700 1 -- 172.21.15.3:0/743864771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7800b0b20 msgr2=0x7fa780140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa785bfe700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7800b0b20 0x7fa780140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa785bfe700 1 -- 172.21.15.3:0/743864771 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77000b180 con 0x7fa7800b13c0 2022-07-21T01:41:02.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7853fd700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800ae870 0x7fa780152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:02.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa785bfe700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7800b13c0 0x7fa7801470c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7fa778001bc0 tx=0x7fa778001a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7767fc700 1 -- 172.21.15.3:0/743864771 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa778001cf0 con 0x7fa7800b13c0 2022-07-21T01:41:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7767fc700 1 -- 172.21.15.3:0/743864771 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778004ab0 con 0x7fa7800b13c0 2022-07-21T01:41:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7801569c0 con 0x7fa7800b13c0 2022-07-21T01:41:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7767fc700 1 -- 172.21.15.3:0/743864771 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa778004e10 con 0x7fa7800b13c0 2022-07-21T01:41:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.048+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78013ce60 con 0x7fa7800b13c0 2022-07-21T01:41:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.052+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7800ae3d0 con 0x7fa7800b13c0 2022-07-21T01:41:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.052+0000 7fa7767fc700 1 -- 172.21.15.3:0/743864771 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa77801b070 con 0x7fa7800b13c0 2022-07-21T01:41:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.052+0000 7fa7767fc700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa768053b30 0x7fa768055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.052+0000 7fa7767fc700 1 -- 172.21.15.3:0/743864771 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa77806b620 con 0x7fa7800b13c0 2022-07-21T01:41:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.052+0000 7fa7853fd700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa768053b30 0x7fa768055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.052+0000 7fa7853fd700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa768053b30 0x7fa768055ff0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fa77c00b000 tx=0x7fa77c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.060+0000 7fa7767fc700 1 -- 172.21.15.3:0/743864771 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa778042680 con 0x7fa7800b13c0 2022-07-21T01:41:02.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.184+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa7800574e0 con 0x7fa7800b13c0 2022-07-21T01:41:02.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.184+0000 7fa7767fc700 1 -- 172.21.15.3:0/743864771 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v106) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa778045ca0 con 0x7fa7800b13c0 2022-07-21T01:41:02.189 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:41:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa768053b30 msgr2=0x7fa768055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa768053b30 0x7fa768055ff0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fa77c00b000 tx=0x7fa77c009040).stop 2022-07-21T01:41:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7800b13c0 msgr2=0x7fa7801470c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7800b13c0 0x7fa7801470c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7fa778001bc0 tx=0x7fa778001a40).stop 2022-07-21T01:41:02.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 shutdown_connections 2022-07-21T01:41:02.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7800b13c0 0x7fa7801470c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa768053b30 0x7fa768055ff0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7800b0b20 0x7fa780140890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 --2- 172.21.15.3:0/743864771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7800ae870 0x7fa780152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 >> 172.21.15.3:0/743864771 conn(0x7fa7800626f0 msgr2=0x7fa7800b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:02.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 shutdown_connections 2022-07-21T01:41:02.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.188+0000 7fa7863ff700 1 -- 172.21.15.3:0/743864771 wait complete. 2022-07-21T01:41:02.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:41:02.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:41:02.254 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:41:02.257 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:41:02.260 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:41:02.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.312+0000 7fd660806700 1 Processor -- start 2022-07-21T01:41:02.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.312+0000 7fd660806700 1 -- start start 2022-07-21T01:41:02.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.312+0000 7fd660806700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c0ad750 0x7fd65c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.312+0000 7fd660806700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65c0aee80 con 0x7fd65c1405d0 2022-07-21T01:41:02.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.312+0000 7fd660806700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65c0b70b0 con 0x7fd65c0614b0 2022-07-21T01:41:02.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.312+0000 7fd660806700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c0af0b0 con 0x7fd65c0ad750 2022-07-21T01:41:02.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd65c1405d0 0x7fd65c1466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57204/0 (socket says 172.21.15.3:57204) 2022-07-21T01:41:02.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65bfff700 1 -- 172.21.15.3:0/444219891 learned_addr learned my addr 172.21.15.3:0/444219891 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:02.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65affd700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c0ad750 0x7fd65c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015658974 0 0) 0x7fd65c0aee80 con 0x7fd65c1405d0 2022-07-21T01:41:02.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd648003260 con 0x7fd65c1405d0 2022-07-21T01:41:02.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371624488 0 0) 0x7fd65c0b70b0 con 0x7fd65c0614b0 2022-07-21T01:41:02.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd65c0aee80 con 0x7fd65c0614b0 2022-07-21T01:41:02.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2352275824 0 0) 0x7fd648003260 con 0x7fd65c1405d0 2022-07-21T01:41:02.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c0ad750 msgr2=0x7fd65c061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:02.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c0ad750 0x7fd65c061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 >> v1:172.21.15.143:6789/0 conn(0x7fd65c0614b0 legacy=0x7fd65c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c0af350 con 0x7fd65c1405d0 2022-07-21T01:41:02.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fd6540019e0 con 0x7fd65c1405d0 2022-07-21T01:41:02.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 >> v1:172.21.15.3:6789/0 conn(0x7fd65c1405d0 legacy=0x7fd65c1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648003910 0x7fd648003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd648004500 0x7fd648004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c0ad750 0x7fd648004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c0af0b0 con 0x7fd648004500 2022-07-21T01:41:02.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64800b800 con 0x7fd648003910 2022-07-21T01:41:02.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64800b990 con 0x7fd65c0ad750 2022-07-21T01:41:02.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65affd700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd648004500 0x7fd648004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65b7fe700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648003910 0x7fd648003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65bfff700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c0ad750 0x7fd648004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65affd700 1 -- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c0ad750 msgr2=0x7fd648004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:02.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65affd700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c0ad750 0x7fd648004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65affd700 1 -- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648003910 msgr2=0x7fd648003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:02.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65affd700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648003910 0x7fd648003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65affd700 1 -- 172.21.15.3:0/444219891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd648003260 con 0x7fd648004500 2022-07-21T01:41:02.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65b7fe700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648003910 0x7fd648003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:02.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65affd700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd648004500 0x7fd648004980 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7fd6480054d0 tx=0x7fd64c006df0).ready entity=mon.0 client_cookie=b6f229c7cd6434a9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd64c00e830 con 0x7fd648004500 2022-07-21T01:41:02.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64c00be90 con 0x7fd648004500 2022-07-21T01:41:02.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65a7fc700 1 -- 172.21.15.3:0/444219891 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd64c00eb80 con 0x7fd648004500 2022-07-21T01:41:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd660806700 1 -- 172.21.15.3:0/444219891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd648004500 msgr2=0x7fd648004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd660806700 1 --2- 172.21.15.3:0/444219891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd648004500 0x7fd648004980 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7fd6480054d0 tx=0x7fd64c006df0).stop 2022-07-21T01:41:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd65bfff700 1 -- 172.21.15.3:0/444219891 reap_dead start 2022-07-21T01:41:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd660806700 1 -- 172.21.15.3:0/444219891 shutdown_connections 2022-07-21T01:41:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd660806700 1 -- 172.21.15.3:0/444219891 >> 172.21.15.3:0/444219891 conn(0x7fd65c066f70 msgr2=0x7fd65c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd660806700 1 -- 172.21.15.3:0/444219891 shutdown_connections 2022-07-21T01:41:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.316+0000 7fd660806700 1 -- 172.21.15.3:0/444219891 wait complete. 2022-07-21T01:41:02.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 Processor -- start 2022-07-21T01:41:02.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 -- start start 2022-07-21T01:41:02.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648004500 0x7fd65c14a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd65c0b1010 0x7fd65c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c09ef30 0x7fd65c0ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c0afb70 con 0x7fd65c0b1010 2022-07-21T01:41:02.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c0b0240 con 0x7fd648004500 2022-07-21T01:41:02.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c156590 con 0x7fd65c09ef30 2022-07-21T01:41:02.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd65c0b1010 0x7fd65c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd65c0b1010 0x7fd65c09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48934/0 (socket says 172.21.15.3:48934) 2022-07-21T01:41:02.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65affd700 1 -- 172.21.15.3:0/2761713434 learned_addr learned my addr 172.21.15.3:0/2761713434 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:02.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65b7fe700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648004500 0x7fd65c14a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65bfff700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c09ef30 0x7fd65c0ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65affd700 1 -- 172.21.15.3:0/2761713434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c09ef30 msgr2=0x7fd65c0ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65affd700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c09ef30 0x7fd65c0ac6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65affd700 1 -- 172.21.15.3:0/2761713434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648004500 msgr2=0x7fd65c14a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65affd700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648004500 0x7fd65c14a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65affd700 1 -- 172.21.15.3:0/2761713434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c156830 con 0x7fd65c0b1010 2022-07-21T01:41:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65bfff700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c09ef30 0x7fd65c0ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd65affd700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd65c0b1010 0x7fd65c09ea40 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7fd6480054d0 tx=0x7fd64c019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd658ff9700 1 -- 172.21.15.3:0/2761713434 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd64c00eee0 con 0x7fd65c0b1010 2022-07-21T01:41:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd658ff9700 1 -- 172.21.15.3:0/2761713434 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64c000e90 con 0x7fd65c0b1010 2022-07-21T01:41:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65c0afda0 con 0x7fd65c0b1010 2022-07-21T01:41:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd658ff9700 1 -- 172.21.15.3:0/2761713434 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd64c00b3c0 con 0x7fd65c0b1010 2022-07-21T01:41:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.320+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65c0b38c0 con 0x7fd65c0b1010 2022-07-21T01:41:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.324+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65c0af5f0 con 0x7fd65c0b1010 2022-07-21T01:41:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.324+0000 7fd658ff9700 1 -- 172.21.15.3:0/2761713434 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd64c02d070 con 0x7fd65c0b1010 2022-07-21T01:41:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.324+0000 7fd658ff9700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd63c053b30 0x7fd63c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.324+0000 7fd658ff9700 1 -- 172.21.15.3:0/2761713434 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fd64c06d770 con 0x7fd65c0b1010 2022-07-21T01:41:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.324+0000 7fd65b7fe700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd63c053b30 0x7fd63c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.324+0000 7fd65b7fe700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd63c053b30 0x7fd63c055ff0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fd65000d000 tx=0x7fd650009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.328+0000 7fd658ff9700 1 -- 172.21.15.3:0/2761713434 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd64c044870 con 0x7fd65c0b1010 2022-07-21T01:41:02.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd65c0574e0 con 0x7fd65c0b1010 2022-07-21T01:41:02.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd658ff9700 1 -- 172.21.15.3:0/2761713434 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v106) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd64c047e90 con 0x7fd65c0b1010 2022-07-21T01:41:02.458 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd63c053b30 msgr2=0x7fd63c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd63c053b30 0x7fd63c055ff0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fd65000d000 tx=0x7fd650009040).stop 2022-07-21T01:41:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd65c0b1010 msgr2=0x7fd65c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd65c0b1010 0x7fd65c09ea40 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7fd6480054d0 tx=0x7fd64c019000).stop 2022-07-21T01:41:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 shutdown_connections 2022-07-21T01:41:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd65c09ef30 0x7fd65c0ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd65c0b1010 0x7fd65c09ea40 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd63c053b30 0x7fd63c055ff0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 --2- 172.21.15.3:0/2761713434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd648004500 0x7fd65c14a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 >> 172.21.15.3:0/2761713434 conn(0x7fd65c066f70 msgr2=0x7fd65c143920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 shutdown_connections 2022-07-21T01:41:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.456+0000 7fd660806700 1 -- 172.21.15.3:0/2761713434 wait complete. 2022-07-21T01:41:02.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:41:02.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.572+0000 7fe68218d700 1 Processor -- start 2022-07-21T01:41:02.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.572+0000 7fe68218d700 1 -- start start 2022-07-21T01:41:02.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.572+0000 7fe68218d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 0x7fe67c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.572+0000 7fe68218d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe67c148530 0x7fe67c0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.572+0000 7fe68218d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe67c0a7250 0x7fe67c1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.572+0000 7fe68218d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c0a9250 con 0x7fe67c148530 2022-07-21T01:41:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.572+0000 7fe68218d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c0b04e0 con 0x7fe67c0a7250 2022-07-21T01:41:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.572+0000 7fe68218d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c0b0780 con 0x7fe67c0b1f40 2022-07-21T01:41:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67affd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe67c148530 0x7fe67c0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67affd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe67c148530 0x7fe67c0b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48940/0 (socket says 172.21.15.3:48940) 2022-07-21T01:41:02.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67affd700 1 -- 172.21.15.3:0/2949489510 learned_addr learned my addr 172.21.15.3:0/2949489510 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:02.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67bfff700 1 --2- 172.21.15.3:0/2949489510 >> v2:172.21.15.143:3300/0 conn(0x7fe67c0a7250 0x7fe67c1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67b7fe700 1 --2- 172.21.15.3:0/2949489510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 0x7fe67c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67affd700 1 -- 172.21.15.3:0/2949489510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 msgr2=0x7fe67c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:02.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67affd700 1 --2- 172.21.15.3:0/2949489510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 0x7fe67c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67affd700 1 -- 172.21.15.3:0/2949489510 >> v2:172.21.15.143:3300/0 conn(0x7fe67c0a7250 msgr2=0x7fe67c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:02.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67affd700 1 --2- 172.21.15.3:0/2949489510 >> v2:172.21.15.143:3300/0 conn(0x7fe67c0a7250 0x7fe67c1448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67affd700 1 -- 172.21.15.3:0/2949489510 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0b0a20 con 0x7fe67c148530 2022-07-21T01:41:02.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67bfff700 1 --2- 172.21.15.3:0/2949489510 >> v2:172.21.15.143:3300/0 conn(0x7fe67c0a7250 0x7fe67c1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:02.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67affd700 1 --2- 172.21.15.3:0/2949489510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c148530 0x7fe67c0b2320 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7fe6740032f0 tx=0x7fe674009290).ready entity=mon.0 client_cookie=202b122756bdd8a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67a7fc700 1 -- 172.21.15.3:0/2949489510 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe67400b870 con 0x7fe67c148530 2022-07-21T01:41:02.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67a7fc700 1 -- 172.21.15.3:0/2949489510 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67400b9d0 con 0x7fe67c148530 2022-07-21T01:41:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67a7fc700 1 -- 172.21.15.3:0/2949489510 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe67400bb40 con 0x7fe67c148530 2022-07-21T01:41:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 -- 172.21.15.3:0/2949489510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c148530 msgr2=0x7fe67c0b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 --2- 172.21.15.3:0/2949489510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c148530 0x7fe67c0b2320 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7fe6740032f0 tx=0x7fe674009290).stop 2022-07-21T01:41:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 -- 172.21.15.3:0/2949489510 shutdown_connections 2022-07-21T01:41:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 --2- 172.21.15.3:0/2949489510 >> v2:172.21.15.143:3300/0 conn(0x7fe67c0a7250 0x7fe67c1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 --2- 172.21.15.3:0/2949489510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c148530 0x7fe67c0b2320 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 --2- 172.21.15.3:0/2949489510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 0x7fe67c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 -- 172.21.15.3:0/2949489510 >> 172.21.15.3:0/2949489510 conn(0x7fe67c05f840 msgr2=0x7fe67c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:02.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 -- 172.21.15.3:0/2949489510 shutdown_connections 2022-07-21T01:41:02.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 -- 172.21.15.3:0/2949489510 wait complete. 2022-07-21T01:41:02.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 Processor -- start 2022-07-21T01:41:02.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 -- start start 2022-07-21T01:41:02.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c0a7250 0x7fe67c142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 0x7fe67c156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe67c148530 0x7fe67c157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c149780 con 0x7fe67c0a7250 2022-07-21T01:41:02.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c149a20 con 0x7fe67c148530 2022-07-21T01:41:02.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe68218d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c149cc0 con 0x7fe67c0b1f40 2022-07-21T01:41:02.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c0a7250 0x7fe67c142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c0a7250 0x7fe67c142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48946/0 (socket says 172.21.15.3:48946) 2022-07-21T01:41:02.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.576+0000 7fe67b7fe700 1 -- 172.21.15.3:0/435428379 learned_addr learned my addr 172.21.15.3:0/435428379 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:02.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67bfff700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe67c148530 0x7fe67c157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67affd700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 0x7fe67c156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67b7fe700 1 -- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 msgr2=0x7fe67c156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67b7fe700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 0x7fe67c156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67b7fe700 1 -- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe67c148530 msgr2=0x7fe67c157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67b7fe700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe67c148530 0x7fe67c157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67b7fe700 1 -- 172.21.15.3:0/435428379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe674009530 con 0x7fe67c0a7250 2022-07-21T01:41:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67bfff700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe67c148530 0x7fe67c157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67affd700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 0x7fe67c156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67b7fe700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c0a7250 0x7fe67c142770 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7fe6640030b0 tx=0x7fe664008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe678ff9700 1 -- 172.21.15.3:0/435428379 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe664011780 con 0x7fe67c0a7250 2022-07-21T01:41:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe678ff9700 1 -- 172.21.15.3:0/435428379 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6640118e0 con 0x7fe67c0a7250 2022-07-21T01:41:02.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c149f60 con 0x7fe67c0a7250 2022-07-21T01:41:02.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe678ff9700 1 -- 172.21.15.3:0/435428379 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe664011c40 con 0x7fe67c0a7250 2022-07-21T01:41:02.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c156930 con 0x7fe67c0a7250 2022-07-21T01:41:02.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c13fb20 con 0x7fe67c0a7250 2022-07-21T01:41:02.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe678ff9700 1 -- 172.21.15.3:0/435428379 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe66401c460 con 0x7fe67c0a7250 2022-07-21T01:41:02.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe678ff9700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe660053a10 0x7fe660055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe678ff9700 1 -- 172.21.15.3:0/435428379 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fe66406c020 con 0x7fe67c0a7250 2022-07-21T01:41:02.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.580+0000 7fe67affd700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe660053a10 0x7fe660055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.584+0000 7fe67affd700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe660053a10 0x7fe660055ed0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fe674007000 tx=0x7fe674016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.584+0000 7fe678ff9700 1 -- 172.21.15.3:0/435428379 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe664043000 con 0x7fe67c0a7250 2022-07-21T01:41:02.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.716+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe67c14a0f0 con 0x7fe67c0a7250 2022-07-21T01:41:02.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.716+0000 7fe678ff9700 1 -- 172.21.15.3:0/435428379 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v106) v1 ==== 84+0+143 (secure 0 0 0) 0x7fe664046620 con 0x7fe67c0a7250 2022-07-21T01:41:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:02.721 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:41:02.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:41:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:41:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:41:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:41:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:02.725 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:41:02.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe660053a10 msgr2=0x7fe660055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe660053a10 0x7fe660055ed0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fe674007000 tx=0x7fe674016000).stop 2022-07-21T01:41:02.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c0a7250 msgr2=0x7fe67c142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c0a7250 0x7fe67c142770 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7fe6640030b0 tx=0x7fe664008040).stop 2022-07-21T01:41:02.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 shutdown_connections 2022-07-21T01:41:02.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe660053a10 0x7fe660055ed0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe67c148530 0x7fe67c157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe67c0b1f40 0x7fe67c156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 --2- 172.21.15.3:0/435428379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe67c0a7250 0x7fe67c142770 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 >> 172.21.15.3:0/435428379 conn(0x7fe67c05f840 msgr2=0x7fe67c1467c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:02.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 shutdown_connections 2022-07-21T01:41:02.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.720+0000 7fe68218d700 1 -- 172.21.15.3:0/435428379 wait complete. 2022-07-21T01:41:02.733 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:41:02.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:41:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.836+0000 7f67e94f9700 1 Processor -- start 2022-07-21T01:41:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.836+0000 7f67e94f9700 1 -- start start 2022-07-21T01:41:02.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.836+0000 7f67e94f9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f67e40af6e0 0x7f67e4147730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.836+0000 7f67e94f9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e40b5a10 con 0x7f67e40b07a0 2022-07-21T01:41:02.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.836+0000 7f67e94f9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e40ac990 con 0x7f67e40b5630 2022-07-21T01:41:02.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.836+0000 7f67e94f9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e4145470 con 0x7f67e40af6e0 2022-07-21T01:41:02.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.836+0000 7f67e8cf8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f67e40af6e0 0x7f67e4147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.836+0000 7f67e37fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f67e40b07a0 0x7f67e40acdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57238/0 (socket says 172.21.15.3:57238) 2022-07-21T01:41:02.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.836+0000 7f67e37fe700 1 -- 172.21.15.3:0/4065302633 learned_addr learned my addr 172.21.15.3:0/4065302633 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:02.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e2ffd700 1 -- 172.21.15.3:0/4065302633 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740668274 0 0) 0x7f67e40b5a10 con 0x7f67e40b07a0 2022-07-21T01:41:02.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e2ffd700 1 -- 172.21.15.3:0/4065302633 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d0003260 con 0x7f67e40b07a0 2022-07-21T01:41:02.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e8cf8700 1 -- 172.21.15.3:0/4065302633 >> v1:172.21.15.143:6789/0 conn(0x7f67e40b5630 legacy=0x7f67e4067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e8cf8700 1 -- 172.21.15.3:0/4065302633 >> v1:172.21.15.3:6789/0 conn(0x7f67e40b07a0 legacy=0x7f67e40acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e2ffd700 1 -- 172.21.15.3:0/4065302633 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 925983834 0 0) 0x7f67d0003260 con 0x7f67e40b07a0 2022-07-21T01:41:02.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e8cf8700 1 -- 172.21.15.3:0/4065302633 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e40b1bc0 con 0x7f67e40af6e0 2022-07-21T01:41:02.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e8cf8700 1 --2- 172.21.15.3:0/4065302633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40af6e0 0x7f67e4147730 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f67d8001800 tx=0x7f67d4006df0).ready entity=mon.0 client_cookie=92125f1b1ffcb02e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e2ffd700 1 -- 172.21.15.3:0/4065302633 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67d8001660 con 0x7f67e40af6e0 2022-07-21T01:41:02.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e2ffd700 1 -- 172.21.15.3:0/4065302633 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d4000e90 con 0x7f67e40af6e0 2022-07-21T01:41:02.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e2ffd700 1 -- 172.21.15.3:0/4065302633 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67d4011400 con 0x7f67e40af6e0 2022-07-21T01:41:02.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e94f9700 1 -- 172.21.15.3:0/4065302633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40af6e0 msgr2=0x7f67e4147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e94f9700 1 --2- 172.21.15.3:0/4065302633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40af6e0 0x7f67e4147730 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f67d8001800 tx=0x7f67d4006df0).stop 2022-07-21T01:41:02.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e94f9700 1 -- 172.21.15.3:0/4065302633 shutdown_connections 2022-07-21T01:41:02.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e94f9700 1 --2- 172.21.15.3:0/4065302633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40af6e0 0x7f67e4147730 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e94f9700 1 -- 172.21.15.3:0/4065302633 >> 172.21.15.3:0/4065302633 conn(0x7f67e40626f0 msgr2=0x7f67e4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:02.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e94f9700 1 -- 172.21.15.3:0/4065302633 shutdown_connections 2022-07-21T01:41:02.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e94f9700 1 -- 172.21.15.3:0/4065302633 wait complete. 2022-07-21T01:41:02.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e94f9700 1 Processor -- start 2022-07-21T01:41:02.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.840+0000 7f67e94f9700 1 -- start start 2022-07-21T01:41:02.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e94f9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e40af6e0 0x7f67e4065130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e94f9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40b07a0 0x7f67e414a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e94f9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67e40b5630 0x7f67e41407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e94f9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e40ae180 con 0x7f67e40b07a0 2022-07-21T01:41:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e94f9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e415abe0 con 0x7f67e40af6e0 2022-07-21T01:41:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e94f9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e415ae80 con 0x7f67e40b5630 2022-07-21T01:41:02.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40b07a0 0x7f67e414a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40b07a0 0x7f67e414a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48962/0 (socket says 172.21.15.3:48962) 2022-07-21T01:41:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e37fe700 1 -- 172.21.15.3:0/3510321421 learned_addr learned my addr 172.21.15.3:0/3510321421 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e3fff700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e40af6e0 0x7f67e4065130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e8cf8700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67e40b5630 0x7f67e41407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e37fe700 1 -- 172.21.15.3:0/3510321421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67e40b5630 msgr2=0x7f67e41407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e37fe700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67e40b5630 0x7f67e41407f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e37fe700 1 -- 172.21.15.3:0/3510321421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e40af6e0 msgr2=0x7f67e4065130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e37fe700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e40af6e0 0x7f67e4065130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e37fe700 1 -- 172.21.15.3:0/3510321421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d400b180 con 0x7f67e40b07a0 2022-07-21T01:41:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e8cf8700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67e40b5630 0x7f67e41407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e37fe700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40b07a0 0x7f67e414a240 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f67d8001840 tx=0x7f67d8001920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e17fa700 1 -- 172.21.15.3:0/3510321421 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67d800ce10 con 0x7f67e40b07a0 2022-07-21T01:41:02.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e17fa700 1 -- 172.21.15.3:0/3510321421 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d8007170 con 0x7f67e40b07a0 2022-07-21T01:41:02.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e415b120 con 0x7f67e40b07a0 2022-07-21T01:41:02.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e17fa700 1 -- 172.21.15.3:0/3510321421 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67d8004730 con 0x7f67e40b07a0 2022-07-21T01:41:02.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e4146930 con 0x7f67e40b07a0 2022-07-21T01:41:02.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.844+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e40b1680 con 0x7f67e40b07a0 2022-07-21T01:41:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.848+0000 7f67e17fa700 1 -- 172.21.15.3:0/3510321421 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f67d8004890 con 0x7f67e40b07a0 2022-07-21T01:41:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.848+0000 7f67e17fa700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f67c4053b30 0x7f67c4055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.848+0000 7f67e17fa700 1 -- 172.21.15.3:0/3510321421 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f67d806be10 con 0x7f67e40b07a0 2022-07-21T01:41:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.848+0000 7f67e3fff700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f67c4053b30 0x7f67c4055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.848+0000 7f67e3fff700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f67c4053b30 0x7f67c4055ff0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f67dc00d000 tx=0x7f67dc000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.856+0000 7f67e17fa700 1 -- 172.21.15.3:0/3510321421 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f67d8042e90 con 0x7f67e40b07a0 2022-07-21T01:41:02.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:02.976+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f67e40574e0 con 0x7f67e40b07a0 2022-07-21T01:41:03.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.640+0000 7f67e17fa700 1 -- 172.21.15.3:0/3510321421 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v107) v1 ==== 133+0+0 (secure 0 0 0) 0x7f67d80464b0 con 0x7f67e40b07a0 2022-07-21T01:41:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f67c4053b30 msgr2=0x7f67c4055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f67c4053b30 0x7f67c4055ff0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f67dc00d000 tx=0x7f67dc000f40).stop 2022-07-21T01:41:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40b07a0 msgr2=0x7f67e414a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40b07a0 0x7f67e414a240 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f67d8001840 tx=0x7f67d8001920).stop 2022-07-21T01:41:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 shutdown_connections 2022-07-21T01:41:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67e40b5630 0x7f67e41407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e40b07a0 0x7f67e414a240 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:03.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f67c4053b30 0x7f67c4055ff0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:03.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 --2- 172.21.15.3:0/3510321421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e40af6e0 0x7f67e4065130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 >> 172.21.15.3:0/3510321421 conn(0x7f67e40626f0 msgr2=0x7f67e414b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 shutdown_connections 2022-07-21T01:41:03.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.644+0000 7f67e94f9700 1 -- 172.21.15.3:0/3510321421 wait complete. 2022-07-21T01:41:03.653 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:41:03.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:41:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 Processor -- start 2022-07-21T01:41:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 -- start start 2022-07-21T01:41:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4e340a8c50 0x7f4e34152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e340a7250 con 0x7f4e340b2710 2022-07-21T01:41:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e340a7480 con 0x7f4e340b01a0 2022-07-21T01:41:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e340a7720 con 0x7f4e340a8c50 2022-07-21T01:41:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e3359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f4e340a8c50 0x7f4e34152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e32d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4e340b2710 0x7f4e3414afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57248/0 (socket says 172.21.15.3:57248) 2022-07-21T01:41:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e32d9d700 1 -- 172.21.15.3:0/3741099706 learned_addr learned my addr 172.21.15.3:0/3741099706 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:03.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e31d9b700 1 -- 172.21.15.3:0/3741099706 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748689402 0 0) 0x7f4e340a7250 con 0x7f4e340b2710 2022-07-21T01:41:03.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e31d9b700 1 -- 172.21.15.3:0/3741099706 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e14003260 con 0x7f4e340b2710 2022-07-21T01:41:03.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e3359e700 1 -- 172.21.15.3:0/3741099706 >> v1:172.21.15.143:6789/0 conn(0x7f4e340b01a0 legacy=0x7f4e340a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:03.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e3359e700 1 -- 172.21.15.3:0/3741099706 >> v1:172.21.15.3:6789/0 conn(0x7f4e340b2710 legacy=0x7f4e3414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:03.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e3359e700 1 -- 172.21.15.3:0/3741099706 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e340a79c0 con 0x7f4e340a8c50 2022-07-21T01:41:03.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e3359e700 1 --2- 172.21.15.3:0/3741099706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340a8c50 0x7f4e34152d70 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f4e2c002f70 tx=0x7f4e2c006df0).ready entity=mon.0 client_cookie=be984bd386d637fa server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:03.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e31d9b700 1 -- 172.21.15.3:0/3741099706 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e2c00d610 con 0x7f4e340a8c50 2022-07-21T01:41:03.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e31d9b700 1 -- 172.21.15.3:0/3741099706 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c000e90 con 0x7f4e340a8c50 2022-07-21T01:41:03.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e31d9b700 1 -- 172.21.15.3:0/3741099706 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e2c00d960 con 0x7f4e340a8c50 2022-07-21T01:41:03.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 -- 172.21.15.3:0/3741099706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340a8c50 msgr2=0x7f4e34152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:03.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 --2- 172.21.15.3:0/3741099706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340a8c50 0x7f4e34152d70 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f4e2c002f70 tx=0x7f4e2c006df0).stop 2022-07-21T01:41:03.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 -- 172.21.15.3:0/3741099706 shutdown_connections 2022-07-21T01:41:03.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 --2- 172.21.15.3:0/3741099706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340a8c50 0x7f4e34152d70 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:03.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.776+0000 7f4e389fb700 1 -- 172.21.15.3:0/3741099706 >> 172.21.15.3:0/3741099706 conn(0x7f4e34060310 msgr2=0x7f4e340b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:03.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 -- 172.21.15.3:0/3741099706 shutdown_connections 2022-07-21T01:41:03.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 -- 172.21.15.3:0/3741099706 wait complete. 2022-07-21T01:41:03.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 Processor -- start 2022-07-21T01:41:03.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 -- start start 2022-07-21T01:41:03.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e340a8c50 0x7f4e340b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:03.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e340b01a0 0x7f4e3414baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:03.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340b2710 0x7f4e3414bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:03.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3414d870 con 0x7f4e340b2710 2022-07-21T01:41:03.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340ac4f0 con 0x7f4e340b01a0 2022-07-21T01:41:03.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e389fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340ac790 con 0x7f4e340a8c50 2022-07-21T01:41:03.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340b2710 0x7f4e3414bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:03.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340b2710 0x7f4e3414bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48978/0 (socket says 172.21.15.3:48978) 2022-07-21T01:41:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3359e700 1 -- 172.21.15.3:0/3111828607 learned_addr learned my addr 172.21.15.3:0/3111828607 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3259c700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e340b01a0 0x7f4e3414baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e32d9d700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e340a8c50 0x7f4e340b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3359e700 1 -- 172.21.15.3:0/3111828607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e340a8c50 msgr2=0x7f4e340b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3359e700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e340a8c50 0x7f4e340b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3359e700 1 -- 172.21.15.3:0/3111828607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e340b01a0 msgr2=0x7f4e3414baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3359e700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e340b01a0 0x7f4e3414baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3359e700 1 -- 172.21.15.3:0/3111828607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c00b180 con 0x7f4e340b2710 2022-07-21T01:41:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e32d9d700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e340a8c50 0x7f4e340b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.780+0000 7f4e3359e700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340b2710 0x7f4e3414bf90 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f4e2c002f70 tx=0x7f4e2c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e2bfff700 1 -- 172.21.15.3:0/3111828607 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e2c016070 con 0x7f4e340b2710 2022-07-21T01:41:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e2bfff700 1 -- 172.21.15.3:0/3111828607 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c00d290 con 0x7f4e340b2710 2022-07-21T01:41:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e340b32b0 con 0x7f4e340b2710 2022-07-21T01:41:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e2bfff700 1 -- 172.21.15.3:0/3111828607 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e2c013ca0 con 0x7f4e340b2710 2022-07-21T01:41:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e340b42a0 con 0x7f4e340b2710 2022-07-21T01:41:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e2bfff700 1 -- 172.21.15.3:0/3111828607 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4e2c025030 con 0x7f4e340b2710 2022-07-21T01:41:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e340574e0 con 0x7f4e340b2710 2022-07-21T01:41:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e2bfff700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e18053b00 0x7f4e18055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e2bfff700 1 -- 172.21.15.3:0/3111828607 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f4e2c06cee0 con 0x7f4e340b2710 2022-07-21T01:41:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.784+0000 7f4e32d9d700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e18053b00 0x7f4e18055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.788+0000 7f4e32d9d700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e18053b00 0x7f4e18055fc0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f4e1c00b000 tx=0x7f4e1c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:03.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.792+0000 7f4e2bfff700 1 -- 172.21.15.3:0/3111828607 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4e2c043d60 con 0x7f4e340b2710 2022-07-21T01:41:03.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:03.920+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f4e34069a70 con 0x7f4e340b2710 2022-07-21T01:41:04.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.280+0000 7f4e2bfff700 1 -- 172.21.15.3:0/3111828607 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v108) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4e2c0230a0 con 0x7f4e340b2710 2022-07-21T01:41:04.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.284+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e18053b00 msgr2=0x7f4e18055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.284+0000 7f4e389fb700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e18053b00 0x7f4e18055fc0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f4e1c00b000 tx=0x7f4e1c009040).stop 2022-07-21T01:41:04.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.284+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340b2710 msgr2=0x7f4e3414bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.284+0000 7f4e389fb700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340b2710 0x7f4e3414bf90 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f4e2c002f70 tx=0x7f4e2c014000).stop 2022-07-21T01:41:04.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.288+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 shutdown_connections 2022-07-21T01:41:04.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.288+0000 7f4e389fb700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4e340b2710 0x7f4e3414bf90 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.288+0000 7f4e389fb700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4e18053b00 0x7f4e18055fc0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.288+0000 7f4e389fb700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e340b01a0 0x7f4e3414baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.288+0000 7f4e389fb700 1 --2- 172.21.15.3:0/3111828607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e340a8c50 0x7f4e340b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.288+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 >> 172.21.15.3:0/3111828607 conn(0x7f4e34060310 msgr2=0x7f4e34144bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:04.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.288+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 shutdown_connections 2022-07-21T01:41:04.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.288+0000 7f4e389fb700 1 -- 172.21.15.3:0/3111828607 wait complete. 2022-07-21T01:41:04.295 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:41:04.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:41:04.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.408+0000 7fc5db59e700 1 Processor -- start 2022-07-21T01:41:04.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.408+0000 7fc5db59e700 1 -- start start 2022-07-21T01:41:04.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc0614b0 0x7fc5dc061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc5dc0ad750 0x7fc5dc0b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc0aee80 con 0x7fc5dc1405d0 2022-07-21T01:41:04.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5dc0b70b0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc0af0b0 con 0x7fc5dc0614b0 2022-07-21T01:41:04.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5dad9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc5dc1405d0 0x7fc5dc1466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57264/0 (socket says 172.21.15.3:57264) 2022-07-21T01:41:04.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5dad9d700 1 -- 172.21.15.3:0/3675104837 learned_addr learned my addr 172.21.15.3:0/3675104837 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:04.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d9d9b700 1 --2- 172.21.15.3:0/3675104837 >> v2:172.21.15.3:3300/0 conn(0x7fc5dc0ad750 0x7fc5dc0b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5da59c700 1 --2- 172.21.15.3:0/3675104837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc0614b0 0x7fc5dc061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d959a700 1 -- 172.21.15.3:0/3675104837 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761409781 0 0) 0x7fc5dc0aee80 con 0x7fc5dc1405d0 2022-07-21T01:41:04.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d959a700 1 -- 172.21.15.3:0/3675104837 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c0003260 con 0x7fc5dc1405d0 2022-07-21T01:41:04.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d9d9b700 1 -- 172.21.15.3:0/3675104837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc0614b0 msgr2=0x7fc5dc061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:04.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d9d9b700 1 --2- 172.21.15.3:0/3675104837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc0614b0 0x7fc5dc061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d9d9b700 1 -- 172.21.15.3:0/3675104837 >> v1:172.21.15.3:6789/0 conn(0x7fc5dc1405d0 legacy=0x7fc5dc1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d9d9b700 1 -- 172.21.15.3:0/3675104837 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc0af350 con 0x7fc5dc0ad750 2022-07-21T01:41:04.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d9d9b700 1 --2- 172.21.15.3:0/3675104837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 0x7fc5dc0b9ff0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7fc5d0002f70 tx=0x7fc5d0006df0).ready entity=mon.0 client_cookie=748c06f2d80a1b09 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:04.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d959a700 1 -- 172.21.15.3:0/3675104837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d000d4a0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d959a700 1 -- 172.21.15.3:0/3675104837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d0000e90 con 0x7fc5dc0ad750 2022-07-21T01:41:04.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5d959a700 1 -- 172.21.15.3:0/3675104837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d000d7f0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 -- 172.21.15.3:0/3675104837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 msgr2=0x7fc5dc0b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 --2- 172.21.15.3:0/3675104837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 0x7fc5dc0b9ff0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7fc5d0002f70 tx=0x7fc5d0006df0).stop 2022-07-21T01:41:04.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 -- 172.21.15.3:0/3675104837 shutdown_connections 2022-07-21T01:41:04.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 --2- 172.21.15.3:0/3675104837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 0x7fc5dc0b9ff0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 --2- 172.21.15.3:0/3675104837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc0614b0 0x7fc5dc061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 -- 172.21.15.3:0/3675104837 >> 172.21.15.3:0/3675104837 conn(0x7fc5dc066f70 msgr2=0x7fc5dc067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:04.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 -- 172.21.15.3:0/3675104837 shutdown_connections 2022-07-21T01:41:04.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.412+0000 7fc5db59e700 1 -- 172.21.15.3:0/3675104837 wait complete. 2022-07-21T01:41:04.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 Processor -- start 2022-07-21T01:41:04.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 -- start start 2022-07-21T01:41:04.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5dc0614b0 0x7fc5dc145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 0x7fc5dc152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc1405d0 0x7fc5dc147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc1532b0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc144580 con 0x7fc5dc0614b0 2022-07-21T01:41:04.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc144820 con 0x7fc5dc1405d0 2022-07-21T01:41:04.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5d9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 0x7fc5dc152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5d9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 0x7fc5dc152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48990/0 (socket says 172.21.15.3:48990) 2022-07-21T01:41:04.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5d9d9b700 1 -- 172.21.15.3:0/2470146345 learned_addr learned my addr 172.21.15.3:0/2470146345 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:04.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5da59c700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5dc0614b0 0x7fc5dc145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5dad9d700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc1405d0 0x7fc5dc147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5d9d9b700 1 -- 172.21.15.3:0/2470146345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc1405d0 msgr2=0x7fc5dc147af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5d9d9b700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc1405d0 0x7fc5dc147af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5d9d9b700 1 -- 172.21.15.3:0/2470146345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5dc0614b0 msgr2=0x7fc5dc145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5d9d9b700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5dc0614b0 0x7fc5dc145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5d9d9b700 1 -- 172.21.15.3:0/2470146345 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d000b180 con 0x7fc5dc0ad750 2022-07-21T01:41:04.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5dad9d700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc1405d0 0x7fc5dc147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:04.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5d9d9b700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 0x7fc5dc152dc0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7fc5d0002f70 tx=0x7fc5d0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:04.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5cb7fe700 1 -- 172.21.15.3:0/2470146345 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d0013ac0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5cb7fe700 1 -- 172.21.15.3:0/2470146345 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d0013c20 con 0x7fc5dc0ad750 2022-07-21T01:41:04.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc144ac0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5cb7fe700 1 -- 172.21.15.3:0/2470146345 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d001e400 con 0x7fc5dc0ad750 2022-07-21T01:41:04.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.416+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5dc147fe0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.420+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5dc0af5f0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.420+0000 7fc5cb7fe700 1 -- 172.21.15.3:0/2470146345 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc5d0025030 con 0x7fc5dc0ad750 2022-07-21T01:41:04.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.420+0000 7fc5cb7fe700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5b80539e0 0x7fc5b8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.420+0000 7fc5cb7fe700 1 -- 172.21.15.3:0/2470146345 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6304+0+0 (secure 0 0 0) 0x7fc5d006ce30 con 0x7fc5dc0ad750 2022-07-21T01:41:04.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.420+0000 7fc5da59c700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5b80539e0 0x7fc5b8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.420+0000 7fc5da59c700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5b80539e0 0x7fc5b8055ea0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fc5cc00b000 tx=0x7fc5cc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:04.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.428+0000 7fc5cb7fe700 1 -- 172.21.15.3:0/2470146345 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc5d0043cf0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.588+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fc5dc0574e0 con 0x7fc5dc0ad750 2022-07-21T01:41:04.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5cb7fe700 1 -- 172.21.15.3:0/2470146345 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 19 and data pool 20 v53) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc5d0023070 con 0x7fc5dc0ad750 2022-07-21T01:41:04.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5b80539e0 msgr2=0x7fc5b8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5b80539e0 0x7fc5b8055ea0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fc5cc00b000 tx=0x7fc5cc006df0).stop 2022-07-21T01:41:04.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 msgr2=0x7fc5dc152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 0x7fc5dc152dc0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7fc5d0002f70 tx=0x7fc5d0014000).stop 2022-07-21T01:41:04.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 shutdown_connections 2022-07-21T01:41:04.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5dc1405d0 0x7fc5dc147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5dc0ad750 0x7fc5dc152dc0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5b80539e0 0x7fc5b8055ea0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 --2- 172.21.15.3:0/2470146345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5dc0614b0 0x7fc5dc145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 >> 172.21.15.3:0/2470146345 conn(0x7fc5dc066f70 msgr2=0x7fc5dc0b89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:04.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 shutdown_connections 2022-07-21T01:41:04.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.600+0000 7fc5db59e700 1 -- 172.21.15.3:0/2470146345 wait complete. 2022-07-21T01:41:04.612 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 19 and data pool 20 2022-07-21T01:41:04.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:04.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.712+0000 7ff953601700 1 Processor -- start 2022-07-21T01:41:04.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.712+0000 7ff953601700 1 -- start start 2022-07-21T01:41:04.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff94c0aa2b0 0x7ff94c0ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c0b2c80 con 0x7ff94c0b0a10 2022-07-21T01:41:04.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c0b2900 con 0x7ff94c0b0630 2022-07-21T01:41:04.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c0b2eb0 con 0x7ff94c0aa2b0 2022-07-21T01:41:04.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff951b9e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff94c0b0a10 0x7ff94c140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57294/0 (socket says 172.21.15.3:57294) 2022-07-21T01:41:04.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff951b9e700 1 -- 172.21.15.3:0/944448323 learned_addr learned my addr 172.21.15.3:0/944448323 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:04.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff950b9c700 1 --2- 172.21.15.3:0/944448323 >> v2:172.21.15.143:3300/0 conn(0x7ff94c0aa2b0 0x7ff94c0ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff943fff700 1 -- 172.21.15.3:0/944448323 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290302859 0 0) 0x7ff94c0b2900 con 0x7ff94c0b0630 2022-07-21T01:41:04.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff943fff700 1 -- 172.21.15.3:0/944448323 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff930003260 con 0x7ff94c0b0630 2022-07-21T01:41:04.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff943fff700 1 -- 172.21.15.3:0/944448323 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768772096 0 0) 0x7ff94c0b2c80 con 0x7ff94c0b0a10 2022-07-21T01:41:04.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff943fff700 1 -- 172.21.15.3:0/944448323 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff94c0b2900 con 0x7ff94c0b0a10 2022-07-21T01:41:04.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff950b9c700 1 -- 172.21.15.3:0/944448323 >> v1:172.21.15.143:6789/0 conn(0x7ff94c0b0630 legacy=0x7ff94c0adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff950b9c700 1 -- 172.21.15.3:0/944448323 >> v1:172.21.15.3:6789/0 conn(0x7ff94c0b0a10 legacy=0x7ff94c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff950b9c700 1 -- 172.21.15.3:0/944448323 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c0a9d00 con 0x7ff94c0aa2b0 2022-07-21T01:41:04.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff950b9c700 1 --2- 172.21.15.3:0/944448323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff94c0aa2b0 0x7ff94c0ae5d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7ff93c002f70 tx=0x7ff93c006df0).ready entity=mon.1 client_cookie=612e6580d55bc492 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:04.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff943fff700 1 -- 172.21.15.3:0/944448323 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff93c00d540 con 0x7ff94c0aa2b0 2022-07-21T01:41:04.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff943fff700 1 -- 172.21.15.3:0/944448323 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93c000e90 con 0x7ff94c0aa2b0 2022-07-21T01:41:04.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff943fff700 1 -- 172.21.15.3:0/944448323 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff93c00d890 con 0x7ff94c0aa2b0 2022-07-21T01:41:04.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 -- 172.21.15.3:0/944448323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff94c0aa2b0 msgr2=0x7ff94c0ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 --2- 172.21.15.3:0/944448323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff94c0aa2b0 0x7ff94c0ae5d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7ff93c002f70 tx=0x7ff93c006df0).stop 2022-07-21T01:41:04.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 -- 172.21.15.3:0/944448323 shutdown_connections 2022-07-21T01:41:04.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 --2- 172.21.15.3:0/944448323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff94c0aa2b0 0x7ff94c0ae5d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 -- 172.21.15.3:0/944448323 >> 172.21.15.3:0/944448323 conn(0x7ff94c060950 msgr2=0x7ff94c060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:04.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 -- 172.21.15.3:0/944448323 shutdown_connections 2022-07-21T01:41:04.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.716+0000 7ff953601700 1 -- 172.21.15.3:0/944448323 wait complete. 2022-07-21T01:41:04.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.720+0000 7ff953601700 1 Processor -- start 2022-07-21T01:41:04.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.720+0000 7ff953601700 1 -- start start 2022-07-21T01:41:04.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.720+0000 7ff953601700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff94c0aa2b0 0x7ff94c0ab390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.720+0000 7ff953601700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94c0b0630 0x7ff94c138d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.720+0000 7ff953601700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff94c0b0a10 0x7ff94c139230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff953601700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c13d7a0 con 0x7ff94c0b0a10 2022-07-21T01:41:04.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff953601700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0ac370 con 0x7ff94c0aa2b0 2022-07-21T01:41:04.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff953601700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0ac610 con 0x7ff94c0b0630 2022-07-21T01:41:04.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff951b9e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff94c0b0a10 0x7ff94c139230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff951b9e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff94c0b0a10 0x7ff94c139230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49020/0 (socket says 172.21.15.3:49020) 2022-07-21T01:41:04.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff951b9e700 1 -- 172.21.15.3:0/4001496225 learned_addr learned my addr 172.21.15.3:0/4001496225 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:04.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff95139d700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff94c0aa2b0 0x7ff94c0ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff950b9c700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94c0b0630 0x7ff94c138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff951b9e700 1 -- 172.21.15.3:0/4001496225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94c0b0630 msgr2=0x7ff94c138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff951b9e700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94c0b0630 0x7ff94c138d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff951b9e700 1 -- 172.21.15.3:0/4001496225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff94c0aa2b0 msgr2=0x7ff94c0ab390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff951b9e700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff94c0aa2b0 0x7ff94c0ab390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff951b9e700 1 -- 172.21.15.3:0/4001496225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93c00b180 con 0x7ff94c0b0a10 2022-07-21T01:41:04.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff951b9e700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff94c0b0a10 0x7ff94c139230 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7ff944001110 tx=0x7ff944007a20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:04.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff9427fc700 1 -- 172.21.15.3:0/4001496225 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff94400cd80 con 0x7ff94c0b0a10 2022-07-21T01:41:04.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff9427fc700 1 -- 172.21.15.3:0/4001496225 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9440070d0 con 0x7ff94c0b0a10 2022-07-21T01:41:04.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c13d060 con 0x7ff94c0b0a10 2022-07-21T01:41:04.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff9427fc700 1 -- 172.21.15.3:0/4001496225 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff944007430 con 0x7ff94c0b0a10 2022-07-21T01:41:04.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c139720 con 0x7ff94c0b0a10 2022-07-21T01:41:04.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c0574e0 con 0x7ff94c0b0a10 2022-07-21T01:41:04.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff9427fc700 1 -- 172.21.15.3:0/4001496225 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff944010070 con 0x7ff94c0b0a10 2022-07-21T01:41:04.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff9427fc700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff934053b00 0x7ff934055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff9427fc700 1 -- 172.21.15.3:0/4001496225 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff94406b580 con 0x7ff94c0b0a10 2022-07-21T01:41:04.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.724+0000 7ff95139d700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff934053b00 0x7ff934055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.728+0000 7ff95139d700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff934053b00 0x7ff934055fc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7ff94800d000 tx=0x7ff948007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:04.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.732+0000 7ff9427fc700 1 -- 172.21.15.3:0/4001496225 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff944042340 con 0x7ff94c0b0a10 2022-07-21T01:41:04.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.860+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff94c0447e0 con 0x7ff94c0b0a10 2022-07-21T01:41:04.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.860+0000 7ff9427fc700 1 -- 172.21.15.3:0/4001496225 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16298 (secure 0 0 0) 0x7ff944045960 con 0x7ff94c0b0a10 2022-07-21T01:41:04.865 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:04.865 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":110,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:04.591354+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:03.035970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:03.980079+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:04.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff934053b00 msgr2=0x7ff934055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff934053b00 0x7ff934055fc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7ff94800d000 tx=0x7ff948007da0).stop 2022-07-21T01:41:04.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff94c0b0a10 msgr2=0x7ff94c139230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff94c0b0a10 0x7ff94c139230 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7ff944001110 tx=0x7ff944007a20).stop 2022-07-21T01:41:04.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 shutdown_connections 2022-07-21T01:41:04.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff94c0b0a10 0x7ff94c139230 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94c0b0630 0x7ff94c138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff934053b00 0x7ff934055fc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 --2- 172.21.15.3:0/4001496225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff94c0aa2b0 0x7ff94c0ab390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 >> 172.21.15.3:0/4001496225 conn(0x7ff94c060950 msgr2=0x7ff94c0b47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:04.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 shutdown_connections 2022-07-21T01:41:04.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.864+0000 7ff953601700 1 -- 172.21.15.3:0/4001496225 wait complete. 2022-07-21T01:41:04.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:04.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.976+0000 7f659bfff700 1 Processor -- start 2022-07-21T01:41:04.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.976+0000 7f659bfff700 1 -- start start 2022-07-21T01:41:04.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.976+0000 7f659bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f659c140b00 0x7f659c064bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.976+0000 7f659bfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c0b4f60 con 0x7f659c0b38a0 2022-07-21T01:41:04.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.976+0000 7f659bfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c0b4a60 con 0x7f659c0647d0 2022-07-21T01:41:04.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.976+0000 7f659bfff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659c0b4cc0 con 0x7f659c140b00 2022-07-21T01:41:04.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.976+0000 7f659b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f659c140b00 0x7f659c064bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:04.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.976+0000 7f65927fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f659c0b38a0 0x7f659c1403f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57304/0 (socket says 172.21.15.3:57304) 2022-07-21T01:41:04.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.976+0000 7f65927fc700 1 -- 172.21.15.3:0/2419975853 learned_addr learned my addr 172.21.15.3:0/2419975853 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:04.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659a7fc700 1 -- 172.21.15.3:0/2419975853 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455630771 0 0) 0x7f659c0b4f60 con 0x7f659c0b38a0 2022-07-21T01:41:04.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659a7fc700 1 -- 172.21.15.3:0/2419975853 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f657c003260 con 0x7f659c0b38a0 2022-07-21T01:41:04.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659b7fe700 1 -- 172.21.15.3:0/2419975853 >> v1:172.21.15.143:6789/0 conn(0x7f659c0647d0 legacy=0x7f659c1436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659b7fe700 1 -- 172.21.15.3:0/2419975853 >> v1:172.21.15.3:6789/0 conn(0x7f659c0b38a0 legacy=0x7f659c1403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659b7fe700 1 -- 172.21.15.3:0/2419975853 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c143250 con 0x7f659c140b00 2022-07-21T01:41:04.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659b7fe700 1 --2- 172.21.15.3:0/2419975853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 0x7f659c064bb0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f65940032f0 tx=0x7f65940091d0).ready entity=mon.0 client_cookie=b190027d04e97275 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:04.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659a7fc700 1 -- 172.21.15.3:0/2419975853 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f659400b6c0 con 0x7f659c140b00 2022-07-21T01:41:04.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659a7fc700 1 -- 172.21.15.3:0/2419975853 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659400b820 con 0x7f659c140b00 2022-07-21T01:41:04.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659a7fc700 1 -- 172.21.15.3:0/2419975853 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f659400b990 con 0x7f659c140b00 2022-07-21T01:41:04.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 -- 172.21.15.3:0/2419975853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 msgr2=0x7f659c064bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:04.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 --2- 172.21.15.3:0/2419975853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 0x7f659c064bb0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f65940032f0 tx=0x7f65940091d0).stop 2022-07-21T01:41:04.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 -- 172.21.15.3:0/2419975853 shutdown_connections 2022-07-21T01:41:04.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 --2- 172.21.15.3:0/2419975853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 0x7f659c064bb0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:04.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 -- 172.21.15.3:0/2419975853 >> 172.21.15.3:0/2419975853 conn(0x7f659c05f840 msgr2=0x7f659c05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:04.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 -- 172.21.15.3:0/2419975853 shutdown_connections 2022-07-21T01:41:04.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 -- 172.21.15.3:0/2419975853 wait complete. 2022-07-21T01:41:04.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 Processor -- start 2022-07-21T01:41:04.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 -- start start 2022-07-21T01:41:04.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.980+0000 7f659bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f659c0647d0 0x7f659c0b7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:04.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f659c0b38a0 0x7f659c0b8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 0x7f659c155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c150510 con 0x7f659c140b00 2022-07-21T01:41:05.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c153160 con 0x7f659c0647d0 2022-07-21T01:41:05.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659c0b5f10 con 0x7f659c0b38a0 2022-07-21T01:41:05.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 0x7f659c155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 0x7f659c155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49032/0 (socket says 172.21.15.3:49032) 2022-07-21T01:41:05.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659b7fe700 1 -- 172.21.15.3:0/3433943878 learned_addr learned my addr 172.21.15.3:0/3433943878 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:05.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f65927fc700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f659c0b38a0 0x7f659c0b8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659affd700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f659c0647d0 0x7f659c0b7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659b7fe700 1 -- 172.21.15.3:0/3433943878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f659c0b38a0 msgr2=0x7f659c0b8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659b7fe700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f659c0b38a0 0x7f659c0b8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659b7fe700 1 -- 172.21.15.3:0/3433943878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f659c0647d0 msgr2=0x7f659c0b7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659b7fe700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f659c0647d0 0x7f659c0b7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659b7fe700 1 -- 172.21.15.3:0/3433943878 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6594009470 con 0x7f659c140b00 2022-07-21T01:41:05.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659b7fe700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 0x7f659c155dc0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f65940032f0 tx=0x7f6594016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:05.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f6598ff9700 1 -- 172.21.15.3:0/3433943878 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f659400b6c0 con 0x7f659c140b00 2022-07-21T01:41:05.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f6598ff9700 1 -- 172.21.15.3:0/3433943878 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6594021630 con 0x7f659c140b00 2022-07-21T01:41:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c0b6520 con 0x7f659c140b00 2022-07-21T01:41:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f6598ff9700 1 -- 172.21.15.3:0/3433943878 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6594021990 con 0x7f659c140b00 2022-07-21T01:41:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c1535f0 con 0x7f659c140b00 2022-07-21T01:41:05.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659c0574e0 con 0x7f659c140b00 2022-07-21T01:41:05.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.984+0000 7f6598ff9700 1 -- 172.21.15.3:0/3433943878 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6594027020 con 0x7f659c140b00 2022-07-21T01:41:05.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.988+0000 7f6598ff9700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6580053950 0x7f6580055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.988+0000 7f6598ff9700 1 -- 172.21.15.3:0/3433943878 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f659406d420 con 0x7f659c140b00 2022-07-21T01:41:05.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.988+0000 7f659affd700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6580053950 0x7f6580055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.988+0000 7f659affd700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6580053950 0x7f6580055e10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f658c00d000 tx=0x7f658c007b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:05.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:04.992+0000 7f6598ff9700 1 -- 172.21.15.3:0/3433943878 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f65940441e0 con 0x7f659c140b00 2022-07-21T01:41:05.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.120+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f659c069d90 con 0x7f659c140b00 2022-07-21T01:41:05.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.120+0000 7f6598ff9700 1 -- 172.21.15.3:0/3433943878 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16298 (secure 0 0 0) 0x7f6594015090 con 0x7f659c140b00 2022-07-21T01:41:05.125 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:05.126 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":110,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:04.591354+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:03.035970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:03.980079+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:05.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6580053950 msgr2=0x7f6580055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6580053950 0x7f6580055e10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f658c00d000 tx=0x7f658c007b50).stop 2022-07-21T01:41:05.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 msgr2=0x7f659c155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 0x7f659c155dc0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f65940032f0 tx=0x7f6594016000).stop 2022-07-21T01:41:05.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 shutdown_connections 2022-07-21T01:41:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f659c140b00 0x7f659c155dc0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f659c0b38a0 0x7f659c0b8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6580053950 0x7f6580055e10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 --2- 172.21.15.3:0/3433943878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f659c0647d0 0x7f659c0b7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 >> 172.21.15.3:0/3433943878 conn(0x7f659c05f840 msgr2=0x7f659c13ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 shutdown_connections 2022-07-21T01:41:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.124+0000 7f659bfff700 1 -- 172.21.15.3:0/3433943878 wait complete. 2022-07-21T01:41:05.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:41:05.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f253d52f700 1 Processor -- start 2022-07-21T01:41:05.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f253d52f700 1 -- start start 2022-07-21T01:41:05.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f253d52f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f253813e5f0 0x7f2538064a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f253d52f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f25380b3a30 0x7f25380b76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f253d52f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2538065530 con 0x7f25380b7bb0 2022-07-21T01:41:05.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f253d52f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2538066160 con 0x7f25380b3a30 2022-07-21T01:41:05.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f253d52f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25380b3260 con 0x7f253813e5f0 2022-07-21T01:41:05.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f252e7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f25380b3a30 0x7f25380b76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f252e7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f25380b3a30 0x7f25380b76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49042/0 (socket says 172.21.15.3:49042) 2022-07-21T01:41:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f25377fe700 1 --1- 172.21.15.3:0/1900739898 >> v1:172.21.15.143:6789/0 conn(0x7f25380b7bb0 0x7f2538066f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:59436/0 (socket says 172.21.15.3:59436) 2022-07-21T01:41:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f252e7fc700 1 -- 172.21.15.3:0/1900739898 learned_addr learned my addr 172.21.15.3:0/1900739898 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f2536ffd700 1 --2- 172.21.15.3:0/1900739898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f253813e5f0 0x7f2538064a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f252e7fc700 1 -- 172.21.15.3:0/1900739898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f253813e5f0 msgr2=0x7f2538064a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f252e7fc700 1 --2- 172.21.15.3:0/1900739898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f253813e5f0 0x7f2538064a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f252e7fc700 1 -- 172.21.15.3:0/1900739898 >> v1:172.21.15.143:6789/0 conn(0x7f25380b7bb0 legacy=0x7f2538066f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f252e7fc700 1 -- 172.21.15.3:0/1900739898 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2538064ff0 con 0x7f25380b3a30 2022-07-21T01:41:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f25367fc700 1 -- 172.21.15.3:0/1900739898 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489816417 0 0) 0x7f2538065530 con 0x7f25380b7bb0 2022-07-21T01:41:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f252e7fc700 1 --2- 172.21.15.3:0/1900739898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25380b3a30 0x7f25380b76c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f2520002f70 tx=0x7f2520006eb0).ready entity=mon.0 client_cookie=16415e2ae21591e9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f25367fc700 1 -- 172.21.15.3:0/1900739898 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f252000d570 con 0x7f25380b3a30 2022-07-21T01:41:05.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f25367fc700 1 -- 172.21.15.3:0/1900739898 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2520000e90 con 0x7f25380b3a30 2022-07-21T01:41:05.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.248+0000 7f25367fc700 1 -- 172.21.15.3:0/1900739898 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f252000d8c0 con 0x7f25380b3a30 2022-07-21T01:41:05.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 -- 172.21.15.3:0/1900739898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25380b3a30 msgr2=0x7f25380b76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 --2- 172.21.15.3:0/1900739898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25380b3a30 0x7f25380b76c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f2520002f70 tx=0x7f2520006eb0).stop 2022-07-21T01:41:05.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 -- 172.21.15.3:0/1900739898 shutdown_connections 2022-07-21T01:41:05.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 --2- 172.21.15.3:0/1900739898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25380b3a30 0x7f25380b76c0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 --2- 172.21.15.3:0/1900739898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f253813e5f0 0x7f2538064a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 -- 172.21.15.3:0/1900739898 >> 172.21.15.3:0/1900739898 conn(0x7f2538062b60 msgr2=0x7f2538062f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:05.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 -- 172.21.15.3:0/1900739898 shutdown_connections 2022-07-21T01:41:05.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 -- 172.21.15.3:0/1900739898 wait complete. 2022-07-21T01:41:05.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 Processor -- start 2022-07-21T01:41:05.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 -- start start 2022-07-21T01:41:05.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25380b3a30 0x7f25381424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f253813e5f0 0x7f25380afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25380b0640 0x7f25380aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2538062380 con 0x7f253813e5f0 2022-07-21T01:41:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25380af070 con 0x7f25380b0640 2022-07-21T01:41:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f253d52f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25380b0410 con 0x7f25380b3a30 2022-07-21T01:41:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f252e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f253813e5f0 0x7f25380afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f252e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f253813e5f0 0x7f25380afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49046/0 (socket says 172.21.15.3:49046) 2022-07-21T01:41:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f252e7fc700 1 -- 172.21.15.3:0/1081499964 learned_addr learned my addr 172.21.15.3:0/1081499964 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f25377fe700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25380b0640 0x7f25380aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.252+0000 7f2536ffd700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25380b3a30 0x7f25381424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f2536ffd700 1 -- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25380b0640 msgr2=0x7f25380aff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f2536ffd700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25380b0640 0x7f25380aff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f2536ffd700 1 -- 172.21.15.3:0/1081499964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f253813e5f0 msgr2=0x7f25380afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f2536ffd700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f253813e5f0 0x7f25380afa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f2536ffd700 1 -- 172.21.15.3:0/1081499964 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252000b1e0 con 0x7f25380b3a30 2022-07-21T01:41:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f25377fe700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25380b0640 0x7f25380aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f2536ffd700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25380b3a30 0x7f25381424e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f25380ab710 tx=0x7f2528006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f2534ff9700 1 -- 172.21.15.3:0/1081499964 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2528016070 con 0x7f25380b3a30 2022-07-21T01:41:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f2534ff9700 1 -- 172.21.15.3:0/1081499964 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528011750 con 0x7f25380b3a30 2022-07-21T01:41:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f2534ff9700 1 -- 172.21.15.3:0/1081499964 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2528011ab0 con 0x7f25380b3a30 2022-07-21T01:41:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25380625b0 con 0x7f25380b3a30 2022-07-21T01:41:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25380af570 con 0x7f25380b3a30 2022-07-21T01:41:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.256+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2538065290 con 0x7f25380b3a30 2022-07-21T01:41:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.260+0000 7f2534ff9700 1 -- 172.21.15.3:0/1081499964 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2528011c10 con 0x7f25380b3a30 2022-07-21T01:41:05.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.260+0000 7f2534ff9700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f251c053a70 0x7f251c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.260+0000 7f2534ff9700 1 -- 172.21.15.3:0/1081499964 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f252806c140 con 0x7f25380b3a30 2022-07-21T01:41:05.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.260+0000 7f252e7fc700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f251c053a70 0x7f251c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.260+0000 7f252e7fc700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f251c053a70 0x7f251c055f30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f252001c000 tx=0x7f2520015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:05.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.268+0000 7f2534ff9700 1 -- 172.21.15.3:0/1081499964 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2528042d70 con 0x7f25380b3a30 2022-07-21T01:41:05.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.432+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f25380574e0 con 0x7f25380b3a30 2022-07-21T01:41:05.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.600+0000 7f2534ff9700 1 -- 172.21.15.3:0/1081499964 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v54) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2528046390 con 0x7f25380b3a30 2022-07-21T01:41:05.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.604+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f251c053a70 msgr2=0x7f251c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.604+0000 7f253d52f700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f251c053a70 0x7f251c055f30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f252001c000 tx=0x7f2520015000).stop 2022-07-21T01:41:05.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.604+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25380b3a30 msgr2=0x7f25381424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.604+0000 7f253d52f700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25380b3a30 0x7f25381424e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f25380ab710 tx=0x7f2528006df0).stop 2022-07-21T01:41:05.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.608+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 shutdown_connections 2022-07-21T01:41:05.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.608+0000 7f253d52f700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f251c053a70 0x7f251c055f30 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.608+0000 7f253d52f700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25380b0640 0x7f25380aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.608+0000 7f253d52f700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f253813e5f0 0x7f25380afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.608+0000 7f253d52f700 1 --2- 172.21.15.3:0/1081499964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25380b3a30 0x7f25381424e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.608+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 >> 172.21.15.3:0/1081499964 conn(0x7f2538062b60 msgr2=0x7f2538141050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:05.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.608+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 shutdown_connections 2022-07-21T01:41:05.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.608+0000 7f253d52f700 1 -- 172.21.15.3:0/1081499964 wait complete. 2022-07-21T01:41:05.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:05.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.732+0000 7f1d58f05700 1 Processor -- start 2022-07-21T01:41:05.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.732+0000 7f1d58f05700 1 -- start start 2022-07-21T01:41:05.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d58f05700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1d540a9da0 0x7f1d54060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d58f05700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d54067030 con 0x7f1d540b31d0 2022-07-21T01:41:05.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d58f05700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d540b35b0 con 0x7f1d541402d0 2022-07-21T01:41:05.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d58f05700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d54064a70 con 0x7f1d540a9da0 2022-07-21T01:41:05.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d537fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1d540b31d0 0x7f1d54066020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57332/0 (socket says 172.21.15.3:57332) 2022-07-21T01:41:05.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d537fe700 1 -- 172.21.15.3:0/2816356389 learned_addr learned my addr 172.21.15.3:0/2816356389 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:05.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d4bfff700 1 --2- 172.21.15.3:0/2816356389 >> v2:172.21.15.143:3300/0 conn(0x7f1d540a9da0 0x7f1d54060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d52ffd700 1 -- 172.21.15.3:0/2816356389 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169649788 0 0) 0x7f1d54067030 con 0x7f1d540b31d0 2022-07-21T01:41:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d52ffd700 1 -- 172.21.15.3:0/2816356389 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d34003260 con 0x7f1d540b31d0 2022-07-21T01:41:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d52ffd700 1 -- 172.21.15.3:0/2816356389 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981994080 0 0) 0x7f1d540b35b0 con 0x7f1d541402d0 2022-07-21T01:41:05.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d52ffd700 1 -- 172.21.15.3:0/2816356389 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d54067030 con 0x7f1d541402d0 2022-07-21T01:41:05.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d4bfff700 1 -- 172.21.15.3:0/2816356389 >> v1:172.21.15.143:6789/0 conn(0x7f1d541402d0 legacy=0x7f1d54140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d4bfff700 1 -- 172.21.15.3:0/2816356389 >> v1:172.21.15.3:6789/0 conn(0x7f1d540b31d0 legacy=0x7f1d54066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d52ffd700 1 -- 172.21.15.3:0/2816356389 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446484138 0 0) 0x7f1d34003260 con 0x7f1d540b31d0 2022-07-21T01:41:05.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d4bfff700 1 -- 172.21.15.3:0/2816356389 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d54066af0 con 0x7f1d540a9da0 2022-07-21T01:41:05.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d4bfff700 1 --2- 172.21.15.3:0/2816356389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d540a9da0 0x7f1d54060950 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f1d44001110 tx=0x7f1d3c006df0).ready entity=mon.1 client_cookie=bdb414d3fcb9510f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:05.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d52ffd700 1 -- 172.21.15.3:0/2816356389 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d44001660 con 0x7f1d540a9da0 2022-07-21T01:41:05.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d52ffd700 1 -- 172.21.15.3:0/2816356389 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c000e90 con 0x7f1d540a9da0 2022-07-21T01:41:05.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d52ffd700 1 -- 172.21.15.3:0/2816356389 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d3c012460 con 0x7f1d540a9da0 2022-07-21T01:41:05.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d58f05700 1 -- 172.21.15.3:0/2816356389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d540a9da0 msgr2=0x7f1d54060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d58f05700 1 --2- 172.21.15.3:0/2816356389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d540a9da0 0x7f1d54060950 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f1d44001110 tx=0x7f1d3c006df0).stop 2022-07-21T01:41:05.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d58f05700 1 -- 172.21.15.3:0/2816356389 shutdown_connections 2022-07-21T01:41:05.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d58f05700 1 --2- 172.21.15.3:0/2816356389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d540a9da0 0x7f1d54060950 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.736+0000 7f1d58f05700 1 -- 172.21.15.3:0/2816356389 >> 172.21.15.3:0/2816356389 conn(0x7f1d54063870 msgr2=0x7f1d54063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:05.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.740+0000 7f1d58f05700 1 -- 172.21.15.3:0/2816356389 shutdown_connections 2022-07-21T01:41:05.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.740+0000 7f1d58f05700 1 -- 172.21.15.3:0/2816356389 wait complete. 2022-07-21T01:41:05.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.740+0000 7f1d58f05700 1 Processor -- start 2022-07-21T01:41:05.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.740+0000 7f1d58f05700 1 -- start start 2022-07-21T01:41:05.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d58f05700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d540a9da0 0x7f1d540ab950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d58f05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d540b31d0 0x7f1d540ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d58f05700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d541402d0 0x7f1d54146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d58f05700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5413f430 con 0x7f1d540a9da0 2022-07-21T01:41:05.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d58f05700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d54144f70 con 0x7f1d541402d0 2022-07-21T01:41:05.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d58f05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d540acb70 con 0x7f1d540b31d0 2022-07-21T01:41:05.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d537fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d540a9da0 0x7f1d540ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d537fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d540a9da0 0x7f1d540ab950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49058/0 (socket says 172.21.15.3:49058) 2022-07-21T01:41:05.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d537fe700 1 -- 172.21.15.3:0/2465119727 learned_addr learned my addr 172.21.15.3:0/2465119727 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:05.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d53fff700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d541402d0 0x7f1d54146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d4bfff700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d540b31d0 0x7f1d540ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d537fe700 1 -- 172.21.15.3:0/2465119727 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d540b31d0 msgr2=0x7f1d540ac680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d537fe700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d540b31d0 0x7f1d540ac680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d537fe700 1 -- 172.21.15.3:0/2465119727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d541402d0 msgr2=0x7f1d54146820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d537fe700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d541402d0 0x7f1d54146820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d537fe700 1 -- 172.21.15.3:0/2465119727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c00b180 con 0x7f1d540a9da0 2022-07-21T01:41:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d537fe700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d540a9da0 0x7f1d540ab950 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f1d44001840 tx=0x7f1d44001720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:05.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d517fa700 1 -- 172.21.15.3:0/2465119727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d44007170 con 0x7f1d540a9da0 2022-07-21T01:41:05.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d517fa700 1 -- 172.21.15.3:0/2465119727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d440072d0 con 0x7f1d540a9da0 2022-07-21T01:41:05.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5413f660 con 0x7f1d540a9da0 2022-07-21T01:41:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d517fa700 1 -- 172.21.15.3:0/2465119727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d44004710 con 0x7f1d540a9da0 2022-07-21T01:41:05.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.744+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5413fba0 con 0x7f1d540a9da0 2022-07-21T01:41:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.748+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d54066d90 con 0x7f1d540a9da0 2022-07-21T01:41:05.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.748+0000 7f1d517fa700 1 -- 172.21.15.3:0/2465119727 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1d4401b070 con 0x7f1d540a9da0 2022-07-21T01:41:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.748+0000 7f1d517fa700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1d38053a70 0x7f1d38055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.748+0000 7f1d517fa700 1 -- 172.21.15.3:0/2465119727 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1d4406b680 con 0x7f1d540a9da0 2022-07-21T01:41:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.748+0000 7f1d4bfff700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1d38053a70 0x7f1d38055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.748+0000 7f1d4bfff700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1d38053a70 0x7f1d38055f30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f1d3c01a000 tx=0x7f1d3c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:05.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.756+0000 7f1d517fa700 1 -- 172.21.15.3:0/2465119727 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1d44042440 con 0x7f1d540a9da0 2022-07-21T01:41:05.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d540574e0 con 0x7f1d540a9da0 2022-07-21T01:41:05.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d517fa700 1 -- 172.21.15.3:0/2465119727 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2484 (secure 0 0 0) 0x7f1d44045a60 con 0x7f1d540a9da0 2022-07-21T01:41:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:05.923 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":54,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1707710855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1707710855},{"type":"v1","addr":"172.21.15.143:6833","nonce":1707710855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/764850063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":764850063},{"type":"v1","addr":"172.21.15.3:6835","nonce":764850063}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:04.591520+0000","modified":"2022-07-21T01:41:05.594363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7254":{"gid":7254,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/151311757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":151311757},{"type":"v1","addr":"172.21.15.3:6837","nonce":151311757}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-21T01:41:05.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1d38053a70 msgr2=0x7f1d38055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1d38053a70 0x7f1d38055f30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f1d3c01a000 tx=0x7f1d3c014000).stop 2022-07-21T01:41:05.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d540a9da0 msgr2=0x7f1d540ab950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:05.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d540a9da0 0x7f1d540ab950 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f1d44001840 tx=0x7f1d44001720).stop 2022-07-21T01:41:05.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 shutdown_connections 2022-07-21T01:41:05.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1d38053a70 0x7f1d38055f30 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d541402d0 0x7f1d54146820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d540b31d0 0x7f1d540ac680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 --2- 172.21.15.3:0/2465119727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d540a9da0 0x7f1d540ab950 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:05.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 >> 172.21.15.3:0/2465119727 conn(0x7f1d54063870 msgr2=0x7f1d540b5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:05.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 shutdown_connections 2022-07-21T01:41:05.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:05.920+0000 7f1d58f05700 1 -- 172.21.15.3:0/2465119727 wait complete. 2022-07-21T01:41:05.933 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 54 2022-07-21T01:41:05.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.036+0000 7fa8794e5700 1 Processor -- start 2022-07-21T01:41:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.036+0000 7fa8794e5700 1 -- start start 2022-07-21T01:41:06.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.036+0000 7fa8794e5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b1de0 0x7fa8740ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.036+0000 7fa8794e5700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa874143650 con 0x7fa874139c70 2022-07-21T01:41:06.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.036+0000 7fa8794e5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8740ac600 con 0x7fa8740b2af0 2022-07-21T01:41:06.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.036+0000 7fa8794e5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740b31a0 con 0x7fa8740b1de0 2022-07-21T01:41:06.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.036+0000 7fa872ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa874139c70 0x7fa874142f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57346/0 (socket says 172.21.15.3:57346) 2022-07-21T01:41:06.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.036+0000 7fa872ffd700 1 -- 172.21.15.3:0/2522331084 learned_addr learned my addr 172.21.15.3:0/2522331084 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:06.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8727fc700 1 --2- 172.21.15.3:0/2522331084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b1de0 0x7fa8740ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa871ffb700 1 -- 172.21.15.3:0/2522331084 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420241258 0 0) 0x7fa874143650 con 0x7fa874139c70 2022-07-21T01:41:06.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa871ffb700 1 -- 172.21.15.3:0/2522331084 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa858003260 con 0x7fa874139c70 2022-07-21T01:41:06.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8727fc700 1 -- 172.21.15.3:0/2522331084 >> v1:172.21.15.143:6789/0 conn(0x7fa8740b2af0 legacy=0x7fa8740a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8727fc700 1 -- 172.21.15.3:0/2522331084 >> v1:172.21.15.3:6789/0 conn(0x7fa874139c70 legacy=0x7fa874142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8727fc700 1 -- 172.21.15.3:0/2522331084 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8741438c0 con 0x7fa8740b1de0 2022-07-21T01:41:06.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8727fc700 1 --2- 172.21.15.3:0/2522331084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b1de0 0x7fa8740ab6c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa868002f70 tx=0x7fa868006e80).ready entity=mon.2 client_cookie=19bb2263b2b178f4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa871ffb700 1 -- 172.21.15.3:0/2522331084 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa868014070 con 0x7fa8740b1de0 2022-07-21T01:41:06.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa871ffb700 1 -- 172.21.15.3:0/2522331084 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa868000e90 con 0x7fa8740b1de0 2022-07-21T01:41:06.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa871ffb700 1 -- 172.21.15.3:0/2522331084 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8680107f0 con 0x7fa8740b1de0 2022-07-21T01:41:06.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8794e5700 1 -- 172.21.15.3:0/2522331084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b1de0 msgr2=0x7fa8740ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8794e5700 1 --2- 172.21.15.3:0/2522331084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b1de0 0x7fa8740ab6c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa868002f70 tx=0x7fa868006e80).stop 2022-07-21T01:41:06.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8794e5700 1 -- 172.21.15.3:0/2522331084 shutdown_connections 2022-07-21T01:41:06.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8794e5700 1 --2- 172.21.15.3:0/2522331084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b1de0 0x7fa8740ab6c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8794e5700 1 -- 172.21.15.3:0/2522331084 >> 172.21.15.3:0/2522331084 conn(0x7fa87405f840 msgr2=0x7fa87405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:06.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8794e5700 1 -- 172.21.15.3:0/2522331084 shutdown_connections 2022-07-21T01:41:06.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8794e5700 1 -- 172.21.15.3:0/2522331084 wait complete. 2022-07-21T01:41:06.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8794e5700 1 Processor -- start 2022-07-21T01:41:06.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.040+0000 7fa8794e5700 1 -- start start 2022-07-21T01:41:06.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8794e5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8740b1de0 0x7fa87413fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8794e5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b2af0 0x7fa87413f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8794e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa87413eb30 0x7fa87413b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8794e5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87414ee20 con 0x7fa87413eb30 2022-07-21T01:41:06.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8794e5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87413db90 con 0x7fa8740b1de0 2022-07-21T01:41:06.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8794e5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87414f050 con 0x7fa8740b2af0 2022-07-21T01:41:06.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8737fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa87413eb30 0x7fa87413b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8740b1de0 0x7fa87413fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8740b1de0 0x7fa87413fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:54520/0 (socket says 172.21.15.3:54520) 2022-07-21T01:41:06.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa872ffd700 1 -- 172.21.15.3:0/709642838 learned_addr learned my addr 172.21.15.3:0/709642838 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:06.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8727fc700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b2af0 0x7fa87413f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8737fe700 1 -- 172.21.15.3:0/709642838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b2af0 msgr2=0x7fa87413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8737fe700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b2af0 0x7fa87413f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8737fe700 1 -- 172.21.15.3:0/709642838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8740b1de0 msgr2=0x7fa87413fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8737fe700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8740b1de0 0x7fa87413fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8737fe700 1 -- 172.21.15.3:0/709642838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86800c310 con 0x7fa87413eb30 2022-07-21T01:41:06.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8737fe700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa87413eb30 0x7fa87413b010 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7fa8740615d0 tx=0x7fa86c007eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa863fff700 1 -- 172.21.15.3:0/709642838 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa86c00f470 con 0x7fa87413eb30 2022-07-21T01:41:06.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa863fff700 1 -- 172.21.15.3:0/709642838 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86c00f5d0 con 0x7fa87413eb30 2022-07-21T01:41:06.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87414f2f0 con 0x7fa87413eb30 2022-07-21T01:41:06.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa863fff700 1 -- 172.21.15.3:0/709642838 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa86c00f930 con 0x7fa87413eb30 2022-07-21T01:41:06.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.044+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87413df60 con 0x7fa87413eb30 2022-07-21T01:41:06.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.048+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87413dd50 con 0x7fa87413eb30 2022-07-21T01:41:06.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.048+0000 7fa863fff700 1 -- 172.21.15.3:0/709642838 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa86c010070 con 0x7fa87413eb30 2022-07-21T01:41:06.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.048+0000 7fa863fff700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8500539e0 0x7fa850055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.048+0000 7fa863fff700 1 -- 172.21.15.3:0/709642838 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa86c06b680 con 0x7fa87413eb30 2022-07-21T01:41:06.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.048+0000 7fa872ffd700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8500539e0 0x7fa850055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.048+0000 7fa872ffd700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8500539e0 0x7fa850055ea0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7fa86400d000 tx=0x7fa864009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.056+0000 7fa863fff700 1 -- 172.21.15.3:0/709642838 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa86c042330 con 0x7fa87413eb30 2022-07-21T01:41:06.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.184+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa87413b840 con 0x7fa87413eb30 2022-07-21T01:41:06.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.184+0000 7fa863fff700 1 -- 172.21.15.3:0/709642838 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16285 (secure 0 0 0) 0x7fa86c045950 con 0x7fa87413eb30 2022-07-21T01:41:06.188 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:06.189 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":111,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:05.595017+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:03.035970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:03.980079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:06.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8500539e0 msgr2=0x7fa850055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8500539e0 0x7fa850055ea0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7fa86400d000 tx=0x7fa864009040).stop 2022-07-21T01:41:06.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa87413eb30 msgr2=0x7fa87413b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa87413eb30 0x7fa87413b010 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7fa8740615d0 tx=0x7fa86c007eb0).stop 2022-07-21T01:41:06.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 shutdown_connections 2022-07-21T01:41:06.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa87413eb30 0x7fa87413b010 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8740b2af0 0x7fa87413f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8500539e0 0x7fa850055ea0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 --2- 172.21.15.3:0/709642838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8740b1de0 0x7fa87413fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 >> 172.21.15.3:0/709642838 conn(0x7fa87405f840 msgr2=0x7fa8740b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:06.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 shutdown_connections 2022-07-21T01:41:06.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.188+0000 7fa8794e5700 1 -- 172.21.15.3:0/709642838 wait complete. 2022-07-21T01:41:06.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:41:06.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 Processor -- start 2022-07-21T01:41:06.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 -- start start 2022-07-21T01:41:06.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f40f80b3260 0x7f40f81435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40f80ad000 0x7f40f813fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f8143ab0 con 0x7f40f80b08a0 2022-07-21T01:41:06.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f80b0c80 con 0x7f40f80b3260 2022-07-21T01:41:06.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f80ae810 con 0x7f40f80ad000 2022-07-21T01:41:06.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f5d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f40f80b3260 0x7f40f81435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f40f80b08a0 0x7f40f8142eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57360/0 (socket says 172.21.15.3:57360) 2022-07-21T01:41:06.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f659c700 1 -- 172.21.15.3:0/3876897697 learned_addr learned my addr 172.21.15.3:0/3876897697 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:06.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f6d9d700 1 --2- 172.21.15.3:0/3876897697 >> v2:172.21.15.143:3300/0 conn(0x7f40f80ad000 0x7f40f813fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f5d9b700 1 -- 172.21.15.3:0/3876897697 >> v2:172.21.15.143:3300/0 conn(0x7f40f80ad000 msgr2=0x7f40f813fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:06.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f5d9b700 1 --2- 172.21.15.3:0/3876897697 >> v2:172.21.15.143:3300/0 conn(0x7f40f80ad000 0x7f40f813fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f5d9b700 1 -- 172.21.15.3:0/3876897697 >> v1:172.21.15.3:6789/0 conn(0x7f40f80b08a0 legacy=0x7f40f8142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f5d9b700 1 -- 172.21.15.3:0/3876897697 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f80aeab0 con 0x7f40f80b3260 2022-07-21T01:41:06.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f5d9b700 1 --2- 172.21.15.3:0/3876897697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 0x7f40f81435c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f40ec002f70 tx=0x7f40ec006df0).ready entity=mon.0 client_cookie=e10a6973e0fc4edb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f559a700 1 -- 172.21.15.3:0/3876897697 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40ec00d570 con 0x7f40f80b3260 2022-07-21T01:41:06.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f559a700 1 -- 172.21.15.3:0/3876897697 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec000e90 con 0x7f40f80b3260 2022-07-21T01:41:06.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f559a700 1 -- 172.21.15.3:0/3876897697 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40ec00d8c0 con 0x7f40f80b3260 2022-07-21T01:41:06.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 -- 172.21.15.3:0/3876897697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 msgr2=0x7f40f81435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 --2- 172.21.15.3:0/3876897697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 0x7f40f81435c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f40ec002f70 tx=0x7f40ec006df0).stop 2022-07-21T01:41:06.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 -- 172.21.15.3:0/3876897697 shutdown_connections 2022-07-21T01:41:06.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 --2- 172.21.15.3:0/3876897697 >> v2:172.21.15.143:3300/0 conn(0x7f40f80ad000 0x7f40f813fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 --2- 172.21.15.3:0/3876897697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 0x7f40f81435c0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.320+0000 7f40f759e700 1 -- 172.21.15.3:0/3876897697 >> 172.21.15.3:0/3876897697 conn(0x7f40f80612d0 msgr2=0x7f40f8062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:06.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 -- 172.21.15.3:0/3876897697 shutdown_connections 2022-07-21T01:41:06.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 -- 172.21.15.3:0/3876897697 wait complete. 2022-07-21T01:41:06.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 Processor -- start 2022-07-21T01:41:06.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 -- start start 2022-07-21T01:41:06.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40f80ad000 0x7f40f80b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f80b08a0 0x7f40f8063f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 0x7f40f8152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80b4fb0 con 0x7f40f80b3260 2022-07-21T01:41:06.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80b5250 con 0x7f40f80ad000 2022-07-21T01:41:06.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f81527b0 con 0x7f40f80b08a0 2022-07-21T01:41:06.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 0x7f40f8152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 0x7f40f8152ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49090/0 (socket says 172.21.15.3:49090) 2022-07-21T01:41:06.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f6d9d700 1 -- 172.21.15.3:0/937260681 learned_addr learned my addr 172.21.15.3:0/937260681 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:06.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f659c700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40f80ad000 0x7f40f80b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.324+0000 7f40f5d9b700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f80b08a0 0x7f40f8063f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f6d9d700 1 -- 172.21.15.3:0/937260681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f80b08a0 msgr2=0x7f40f8063f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f6d9d700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f80b08a0 0x7f40f8063f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f6d9d700 1 -- 172.21.15.3:0/937260681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40f80ad000 msgr2=0x7f40f80b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f6d9d700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40f80ad000 0x7f40f80b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f6d9d700 1 -- 172.21.15.3:0/937260681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec00b180 con 0x7f40f80b3260 2022-07-21T01:41:06.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f6d9d700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 0x7f40f8152ef0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f40f0006e00 tx=0x7f40f00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40e77fe700 1 -- 172.21.15.3:0/937260681 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40f000c640 con 0x7f40f80b3260 2022-07-21T01:41:06.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40e77fe700 1 -- 172.21.15.3:0/937260681 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f000c7a0 con 0x7f40f80b3260 2022-07-21T01:41:06.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f8152a50 con 0x7f40f80b3260 2022-07-21T01:41:06.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40e77fe700 1 -- 172.21.15.3:0/937260681 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40f000cb00 con 0x7f40f80b3260 2022-07-21T01:41:06.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f80ac7f0 con 0x7f40f80b3260 2022-07-21T01:41:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f8153720 con 0x7f40f80b3260 2022-07-21T01:41:06.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40e77fe700 1 -- 172.21.15.3:0/937260681 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f40f000cc60 con 0x7f40f80b3260 2022-07-21T01:41:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40e77fe700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f40d4053a70 0x7f40d4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40e77fe700 1 -- 172.21.15.3:0/937260681 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f40f006c270 con 0x7f40f80b3260 2022-07-21T01:41:06.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.328+0000 7f40f659c700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f40d4053a70 0x7f40d4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.332+0000 7f40f659c700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f40d4053a70 0x7f40d4055f30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f40e800d000 tx=0x7f40e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.336+0000 7f40e77fe700 1 -- 172.21.15.3:0/937260681 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f40f8153720 con 0x7f40f80b3260 2022-07-21T01:41:06.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.504+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f40f8153720 con 0x7f40f80b3260 2022-07-21T01:41:06.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40e77fe700 1 -- 172.21.15.3:0/937260681 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v29) v1 ==== 184+0+0 (secure 0 0 0) 0x7f40f8153720 con 0x7f40f80b3260 2022-07-21T01:41:06.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f40d4053a70 msgr2=0x7f40d4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f40d4053a70 0x7f40d4055f30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f40e800d000 tx=0x7f40e8009040).stop 2022-07-21T01:41:06.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 msgr2=0x7f40f8152ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 0x7f40f8152ef0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f40f0006e00 tx=0x7f40f00091d0).stop 2022-07-21T01:41:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 shutdown_connections 2022-07-21T01:41:06.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40f80b3260 0x7f40f8152ef0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f80b08a0 0x7f40f8063f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f40d4053a70 0x7f40d4055f30 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 --2- 172.21.15.3:0/937260681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40f80ad000 0x7f40f80b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 >> 172.21.15.3:0/937260681 conn(0x7f40f80612d0 msgr2=0x7f40f8142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 shutdown_connections 2022-07-21T01:41:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.512+0000 7f40f759e700 1 -- 172.21.15.3:0/937260681 wait complete. 2022-07-21T01:41:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:41:06.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:41:06.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.648+0000 7fc8894bb700 1 Processor -- start 2022-07-21T01:41:06.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.648+0000 7fc8894bb700 1 -- start start 2022-07-21T01:41:06.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc8840b4550 0x7fc8840b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc8840b6af0 0x7fc88412bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc884144240 con 0x7fc8840b6710 2022-07-21T01:41:06.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc884143d00 con 0x7fc8840b6af0 2022-07-21T01:41:06.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc884143fa0 con 0x7fc8840b4550 2022-07-21T01:41:06.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8837fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc8840b6af0 0x7fc88412bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8837fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc8840b6af0 0x7fc88412bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49098/0 (socket says 172.21.15.3:49098) 2022-07-21T01:41:06.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8837fe700 1 -- 172.21.15.3:0/2168330581 learned_addr learned my addr 172.21.15.3:0/2168330581 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:06.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8827fc700 1 --2- 172.21.15.3:0/2168330581 >> v2:172.21.15.143:3300/0 conn(0x7fc8840b4550 0x7fc8840b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8837fe700 1 -- 172.21.15.3:0/2168330581 >> v2:172.21.15.143:3300/0 conn(0x7fc8840b4550 msgr2=0x7fc8840b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:06.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8837fe700 1 --2- 172.21.15.3:0/2168330581 >> v2:172.21.15.143:3300/0 conn(0x7fc8840b4550 0x7fc8840b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8837fe700 1 -- 172.21.15.3:0/2168330581 >> v1:172.21.15.143:6789/0 conn(0x7fc8840b6710 legacy=0x7fc884061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8837fe700 1 -- 172.21.15.3:0/2168330581 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884061780 con 0x7fc8840b6af0 2022-07-21T01:41:06.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8837fe700 1 --2- 172.21.15.3:0/2168330581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6af0 0x7fc88412bb90 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7fc8841432b0 tx=0x7fc87c009290).ready entity=mon.0 client_cookie=3544941cfd9e6152 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc881ffb700 1 -- 172.21.15.3:0/2168330581 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc87c00b860 con 0x7fc8840b6af0 2022-07-21T01:41:06.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc881ffb700 1 -- 172.21.15.3:0/2168330581 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c00b9c0 con 0x7fc8840b6af0 2022-07-21T01:41:06.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc881ffb700 1 -- 172.21.15.3:0/2168330581 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc87c00bb30 con 0x7fc8840b6af0 2022-07-21T01:41:06.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 -- 172.21.15.3:0/2168330581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6af0 msgr2=0x7fc88412bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 --2- 172.21.15.3:0/2168330581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6af0 0x7fc88412bb90 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7fc8841432b0 tx=0x7fc87c009290).stop 2022-07-21T01:41:06.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 -- 172.21.15.3:0/2168330581 shutdown_connections 2022-07-21T01:41:06.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 --2- 172.21.15.3:0/2168330581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6af0 0x7fc88412bb90 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 --2- 172.21.15.3:0/2168330581 >> v2:172.21.15.143:3300/0 conn(0x7fc8840b4550 0x7fc8840b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 -- 172.21.15.3:0/2168330581 >> 172.21.15.3:0/2168330581 conn(0x7fc88405c620 msgr2=0x7fc88405ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:06.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 -- 172.21.15.3:0/2168330581 shutdown_connections 2022-07-21T01:41:06.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.652+0000 7fc8894bb700 1 -- 172.21.15.3:0/2168330581 wait complete. 2022-07-21T01:41:06.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8894bb700 1 Processor -- start 2022-07-21T01:41:06.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8894bb700 1 -- start start 2022-07-21T01:41:06.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8894bb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840b4550 0x7fc884146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8894bb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6710 0x7fc8840b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8894bb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8840b6af0 0x7fc8840b3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8894bb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc884149150 con 0x7fc8840b6710 2022-07-21T01:41:06.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8894bb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc884155d30 con 0x7fc8840b6af0 2022-07-21T01:41:06.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8894bb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc884155fd0 con 0x7fc8840b4550 2022-07-21T01:41:06.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8827fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6710 0x7fc8840b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8827fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6710 0x7fc8840b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49102/0 (socket says 172.21.15.3:49102) 2022-07-21T01:41:06.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8827fc700 1 -- 172.21.15.3:0/1654932402 learned_addr learned my addr 172.21.15.3:0/1654932402 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:06.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc882ffd700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840b4550 0x7fc884146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8837fe700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8840b6af0 0x7fc8840b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8827fc700 1 -- 172.21.15.3:0/1654932402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840b4550 msgr2=0x7fc884146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8827fc700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840b4550 0x7fc884146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8827fc700 1 -- 172.21.15.3:0/1654932402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8840b6af0 msgr2=0x7fc8840b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8827fc700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8840b6af0 0x7fc8840b3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8827fc700 1 -- 172.21.15.3:0/1654932402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c009530 con 0x7fc8840b6710 2022-07-21T01:41:06.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.656+0000 7fc8827fc700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6710 0x7fc8840b3230 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7fc87400c000 tx=0x7fc874006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc873fff700 1 -- 172.21.15.3:0/1654932402 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc874010590 con 0x7fc8840b6710 2022-07-21T01:41:06.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc884156270 con 0x7fc8840b6710 2022-07-21T01:41:06.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc873fff700 1 -- 172.21.15.3:0/1654932402 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8740106f0 con 0x7fc8840b6710 2022-07-21T01:41:06.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc873fff700 1 -- 172.21.15.3:0/1654932402 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc874010a50 con 0x7fc8840b6710 2022-07-21T01:41:06.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88412b7a0 con 0x7fc8840b6710 2022-07-21T01:41:06.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8840b3db0 con 0x7fc8840b6710 2022-07-21T01:41:06.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc873fff700 1 -- 172.21.15.3:0/1654932402 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc874010bb0 con 0x7fc8840b6710 2022-07-21T01:41:06.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc873fff700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc860053a70 0x7fc860055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc873fff700 1 -- 172.21.15.3:0/1654932402 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc87406c1b0 con 0x7fc8840b6710 2022-07-21T01:41:06.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc882ffd700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc860053a70 0x7fc860055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.660+0000 7fc882ffd700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc860053a70 0x7fc860055f30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fc87800d000 tx=0x7fc878000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.668+0000 7fc873fff700 1 -- 172.21.15.3:0/1654932402 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc874042de0 con 0x7fc8840b6710 2022-07-21T01:41:06.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.832+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc884061a20 con 0x7fc8840b6710 2022-07-21T01:41:06.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.840+0000 7fc873fff700 1 -- 172.21.15.3:0/1654932402 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v30) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc874046400 con 0x7fc8840b6710 2022-07-21T01:41:06.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc860053a70 msgr2=0x7fc860055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc860053a70 0x7fc860055f30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fc87800d000 tx=0x7fc878000bc0).stop 2022-07-21T01:41:06.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6710 msgr2=0x7fc8840b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6710 0x7fc8840b3230 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7fc87400c000 tx=0x7fc874006df0).stop 2022-07-21T01:41:06.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 shutdown_connections 2022-07-21T01:41:06.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc860053a70 0x7fc860055f30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8840b6af0 0x7fc8840b3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8840b6710 0x7fc8840b3230 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 --2- 172.21.15.3:0/1654932402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840b4550 0x7fc884146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 >> 172.21.15.3:0/1654932402 conn(0x7fc88405c620 msgr2=0x7fc8841410f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:06.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 shutdown_connections 2022-07-21T01:41:06.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.844+0000 7fc8894bb700 1 -- 172.21.15.3:0/1654932402 wait complete. 2022-07-21T01:41:06.849 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:41:06.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:06.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f703d6f4700 1 Processor -- start 2022-07-21T01:41:06.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f703d6f4700 1 -- start start 2022-07-21T01:41:06.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f703d6f4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7038148530 0x7f70380b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f703d6f4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380b04e0 con 0x7f70380a7250 2022-07-21T01:41:06.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f703d6f4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380b0780 con 0x7f70380b1f40 2022-07-21T01:41:06.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f703d6f4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70380a9250 con 0x7f7038148530 2022-07-21T01:41:06.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f70377fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f70380a7250 0x7f70381448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57392/0 (socket says 172.21.15.3:57392) 2022-07-21T01:41:06.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f70377fe700 1 -- 172.21.15.3:0/1311414436 learned_addr learned my addr 172.21.15.3:0/1311414436 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:06.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f70367fc700 1 --2- 172.21.15.3:0/1311414436 >> v2:172.21.15.143:3300/0 conn(0x7f7038148530 0x7f70380b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737864118 0 0) 0x7f70380b04e0 con 0x7f70380a7250 2022-07-21T01:41:06.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f701c003260 con 0x7f70380a7250 2022-07-21T01:41:06.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093132327 0 0) 0x7f70380b0780 con 0x7f70380b1f40 2022-07-21T01:41:06.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70380b04e0 con 0x7f70380b1f40 2022-07-21T01:41:06.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1238871643 0 0) 0x7f701c003260 con 0x7f70380a7250 2022-07-21T01:41:06.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 >> v2:172.21.15.143:3300/0 conn(0x7f7038148530 msgr2=0x7f70380b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:06.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 --2- 172.21.15.3:0/1311414436 >> v2:172.21.15.143:3300/0 conn(0x7f7038148530 0x7f70380b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 >> v1:172.21.15.143:6789/0 conn(0x7f70380b1f40 legacy=0x7f70380b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380b0a20 con 0x7f70380a7250 2022-07-21T01:41:06.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f70300019e0 con 0x7f70380a7250 2022-07-21T01:41:06.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 >> v1:172.21.15.3:6789/0 conn(0x7f70380a7250 legacy=0x7f70381448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f701c003910 0x7f701c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f701c004500 0x7f701c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f701c004e70 0x7f701c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c00bcf0 con 0x7f701c004e70 2022-07-21T01:41:06.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c00be60 con 0x7f701c004500 2022-07-21T01:41:06.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c00c020 con 0x7f701c003910 2022-07-21T01:41:06.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.960+0000 7f70377fe700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f701c004e70 0x7f701c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f7036ffd700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f701c003910 0x7f701c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f70367fc700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f701c004500 0x7f701c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f70377fe700 1 -- 172.21.15.3:0/1311414436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f701c003910 msgr2=0x7f701c003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:06.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f70377fe700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f701c003910 0x7f701c003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f70377fe700 1 -- 172.21.15.3:0/1311414436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f701c004500 msgr2=0x7f701c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:06.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f70377fe700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f701c004500 0x7f701c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f70377fe700 1 -- 172.21.15.3:0/1311414436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c003260 con 0x7f701c004e70 2022-07-21T01:41:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f70377fe700 1 -- 172.21.15.3:0/1311414436 reap_dead start 2022-07-21T01:41:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f70377fe700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f701c004e70 0x7f701c00b750 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7f70300017b0 tx=0x7f70380b0bb0).ready entity=mon.0 client_cookie=c12a73ab0f79470f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f703000cd10 con 0x7f701c004e70 2022-07-21T01:41:06.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703000ce70 con 0x7f701c004e70 2022-07-21T01:41:06.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f7035ffb700 1 -- 172.21.15.3:0/1311414436 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7030005b80 con 0x7f701c004e70 2022-07-21T01:41:06.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f703d6f4700 1 -- 172.21.15.3:0/1311414436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f701c004e70 msgr2=0x7f701c00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f703d6f4700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f701c004e70 0x7f701c00b750 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7f70300017b0 tx=0x7f70380b0bb0).stop 2022-07-21T01:41:06.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f703d6f4700 1 -- 172.21.15.3:0/1311414436 shutdown_connections 2022-07-21T01:41:06.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f703d6f4700 1 --2- 172.21.15.3:0/1311414436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f701c004e70 0x7f701c00b750 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f703d6f4700 1 -- 172.21.15.3:0/1311414436 >> 172.21.15.3:0/1311414436 conn(0x7f703805f840 msgr2=0x7f703805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:06.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f703d6f4700 1 -- 172.21.15.3:0/1311414436 shutdown_connections 2022-07-21T01:41:06.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.964+0000 7f703d6f4700 1 -- 172.21.15.3:0/1311414436 wait complete. 2022-07-21T01:41:06.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f703d6f4700 1 Processor -- start 2022-07-21T01:41:06.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f703d6f4700 1 -- start start 2022-07-21T01:41:06.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f703d6f4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f701c004e70 0x7f70380ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f703d6f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703814a180 0x7f7038156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f703d6f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7038157220 0x7f7038159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:06.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f703d6f4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703814bc20 con 0x7f7038157220 2022-07-21T01:41:06.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f703d6f4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380a9660 con 0x7f701c004e70 2022-07-21T01:41:06.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f703d6f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380a9900 con 0x7f703814a180 2022-07-21T01:41:06.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7038157220 0x7f7038159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7038157220 0x7f7038159e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49124/0 (socket says 172.21.15.3:49124) 2022-07-21T01:41:06.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70377fe700 1 -- 172.21.15.3:0/238194495 learned_addr learned my addr 172.21.15.3:0/238194495 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:06.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f7036ffd700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f701c004e70 0x7f70380ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70367fc700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703814a180 0x7f7038156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:06.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70377fe700 1 -- 172.21.15.3:0/238194495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703814a180 msgr2=0x7f7038156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70377fe700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703814a180 0x7f7038156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70377fe700 1 -- 172.21.15.3:0/238194495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f701c004e70 msgr2=0x7f70380ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:06.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70377fe700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f701c004e70 0x7f70380ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:06.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70377fe700 1 -- 172.21.15.3:0/238194495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380a9ba0 con 0x7f7038157220 2022-07-21T01:41:06.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f70377fe700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7038157220 0x7f7038159e30 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7f70300017b0 tx=0x7f7030013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:06.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.968+0000 7f7027fff700 1 -- 172.21.15.3:0/238194495 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7030001dd0 con 0x7f7038157220 2022-07-21T01:41:06.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f7027fff700 1 -- 172.21.15.3:0/238194495 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70300072e0 con 0x7f7038157220 2022-07-21T01:41:06.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7038148c20 con 0x7f7038157220 2022-07-21T01:41:06.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f7027fff700 1 -- 172.21.15.3:0/238194495 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7030007640 con 0x7f7038157220 2022-07-21T01:41:06.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703814a690 con 0x7f7038157220 2022-07-21T01:41:06.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f7027fff700 1 -- 172.21.15.3:0/238194495 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7030027070 con 0x7f7038157220 2022-07-21T01:41:06.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70380a9d30 con 0x7f7038157220 2022-07-21T01:41:06.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f7027fff700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7014053b00 0x7f7014055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f7036ffd700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7014053b00 0x7f7014055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f7027fff700 1 -- 172.21.15.3:0/238194495 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f703006c820 con 0x7f7038157220 2022-07-21T01:41:07.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.972+0000 7f7036ffd700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7014053b00 0x7f7014055fc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f702c003000 tx=0x7f702c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:07.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:06.976+0000 7f7027fff700 1 -- 172.21.15.3:0/238194495 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7030043560 con 0x7f7038157220 2022-07-21T01:41:07.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.140+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70380b0cc0 con 0x7f7038157220 2022-07-21T01:41:07.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.140+0000 7f7027fff700 1 -- 172.21.15.3:0/238194495 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2484 (secure 0 0 0) 0x7f7030046b80 con 0x7f7038157220 2022-07-21T01:41:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:07.146 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":54,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1707710855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1707710855},{"type":"v1","addr":"172.21.15.143:6833","nonce":1707710855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/764850063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":764850063},{"type":"v1","addr":"172.21.15.3:6835","nonce":764850063}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:04.591520+0000","modified":"2022-07-21T01:41:05.594363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7254":{"gid":7254,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/151311757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":151311757},{"type":"v1","addr":"172.21.15.3:6837","nonce":151311757}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-21T01:41:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7014053b00 msgr2=0x7f7014055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7014053b00 0x7f7014055fc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f702c003000 tx=0x7f702c009040).stop 2022-07-21T01:41:07.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7038157220 msgr2=0x7f7038159e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7038157220 0x7f7038159e30 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7f70300017b0 tx=0x7f7030013000).stop 2022-07-21T01:41:07.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 shutdown_connections 2022-07-21T01:41:07.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7038157220 0x7f7038159e30 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703814a180 0x7f7038156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7014053b00 0x7f7014055fc0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 --2- 172.21.15.3:0/238194495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f701c004e70 0x7f70380ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 >> 172.21.15.3:0/238194495 conn(0x7f703805f840 msgr2=0x7f70380b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:07.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 shutdown_connections 2022-07-21T01:41:07.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.144+0000 7f703d6f4700 1 -- 172.21.15.3:0/238194495 wait complete. 2022-07-21T01:41:07.154 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 54 2022-07-21T01:41:07.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:41:04.591520+0000', 'modified': '2022-07-21T01:41:05.594363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7254': {'gid': 7254, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/151311757', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 151311757}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 151311757}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:41:07.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:41:07.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:41:07.583 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":7254,"want_state":"up:active","state":"up:active","rank_uptime":2.979040629,"mdsmap_epoch":54,"osdmap_epoch":111,"osdmap_epoch_barrier":110,"uptime":7.3062602050000001} 2022-07-21T01:41:07.583 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 7254, "mdsmap_epoch": 54, "osdmap_epoch": 111, "osdmap_epoch_barrier": 110, "rank_uptime": 2.979040629, "state": "up:active", "uptime": 7.306260205, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:41:07.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:07.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.688+0000 7f1454920700 1 Processor -- start 2022-07-21T01:41:07.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.688+0000 7f1454920700 1 -- start start 2022-07-21T01:41:07.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f14500614b0 0x7f1450061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14500aee80 con 0x7f14501405d0 2022-07-21T01:41:07.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14500b70b0 con 0x7f14500ad750 2022-07-21T01:41:07.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14500af0b0 con 0x7f14500614b0 2022-07-21T01:41:07.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144f7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f14500614b0 0x7f1450061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f14501405d0 0x7f14501466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57412/0 (socket says 172.21.15.3:57412) 2022-07-21T01:41:07.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144ffff700 1 -- 172.21.15.3:0/2065112197 learned_addr learned my addr 172.21.15.3:0/2065112197 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:07.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144e7fc700 1 -- 172.21.15.3:0/2065112197 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730615211 0 0) 0x7f14500aee80 con 0x7f14501405d0 2022-07-21T01:41:07.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144e7fc700 1 -- 172.21.15.3:0/2065112197 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f143c003260 con 0x7f14501405d0 2022-07-21T01:41:07.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144f7fe700 1 -- 172.21.15.3:0/2065112197 >> v1:172.21.15.143:6789/0 conn(0x7f14500ad750 legacy=0x7f14500b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144f7fe700 1 -- 172.21.15.3:0/2065112197 >> v1:172.21.15.3:6789/0 conn(0x7f14501405d0 legacy=0x7f14501466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144f7fe700 1 -- 172.21.15.3:0/2065112197 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500af350 con 0x7f14500614b0 2022-07-21T01:41:07.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144f7fe700 1 --2- 172.21.15.3:0/2065112197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 0x7f1450061890 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f14500aed30 tx=0x7f1444006df0).ready entity=mon.0 client_cookie=3ca0381b0306ca4a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:07.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144e7fc700 1 -- 172.21.15.3:0/2065112197 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f144400d610 con 0x7f14500614b0 2022-07-21T01:41:07.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144e7fc700 1 -- 172.21.15.3:0/2065112197 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1444000e90 con 0x7f14500614b0 2022-07-21T01:41:07.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f144e7fc700 1 -- 172.21.15.3:0/2065112197 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f144400d960 con 0x7f14500614b0 2022-07-21T01:41:07.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 -- 172.21.15.3:0/2065112197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 msgr2=0x7f1450061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 --2- 172.21.15.3:0/2065112197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 0x7f1450061890 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f14500aed30 tx=0x7f1444006df0).stop 2022-07-21T01:41:07.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 -- 172.21.15.3:0/2065112197 shutdown_connections 2022-07-21T01:41:07.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 --2- 172.21.15.3:0/2065112197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 0x7f1450061890 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 -- 172.21.15.3:0/2065112197 >> 172.21.15.3:0/2065112197 conn(0x7f1450066f70 msgr2=0x7f1450067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:07.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 -- 172.21.15.3:0/2065112197 shutdown_connections 2022-07-21T01:41:07.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.692+0000 7f1454920700 1 -- 172.21.15.3:0/2065112197 wait complete. 2022-07-21T01:41:07.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 Processor -- start 2022-07-21T01:41:07.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 -- start start 2022-07-21T01:41:07.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 0x7f14500b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14500ad750 0x7f14500b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14501405d0 0x7f14500abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1450143a50 con 0x7f14500614b0 2022-07-21T01:41:07.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14500b16e0 con 0x7f14500ad750 2022-07-21T01:41:07.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14500aac50 con 0x7f14501405d0 2022-07-21T01:41:07.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 0x7f14500b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 0x7f14500b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49134/0 (socket says 172.21.15.3:49134) 2022-07-21T01:41:07.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144f7fe700 1 -- 172.21.15.3:0/3541196462 learned_addr learned my addr 172.21.15.3:0/3541196462 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:07.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144effd700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14500ad750 0x7f14500b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144ffff700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14501405d0 0x7f14500abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144f7fe700 1 -- 172.21.15.3:0/3541196462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14501405d0 msgr2=0x7f14500abfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144f7fe700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14501405d0 0x7f14500abfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144f7fe700 1 -- 172.21.15.3:0/3541196462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14500ad750 msgr2=0x7f14500b11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144f7fe700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14500ad750 0x7f14500b11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144f7fe700 1 -- 172.21.15.3:0/3541196462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144400b180 con 0x7f14500614b0 2022-07-21T01:41:07.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144effd700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14500ad750 0x7f14500b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:07.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144f7fe700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 0x7f14500b0d00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f14500b1820 tx=0x7f1444014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:07.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144cff9700 1 -- 172.21.15.3:0/3541196462 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1444016070 con 0x7f14500614b0 2022-07-21T01:41:07.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144cff9700 1 -- 172.21.15.3:0/3541196462 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144400de90 con 0x7f14500614b0 2022-07-21T01:41:07.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14500aaf30 con 0x7f14500614b0 2022-07-21T01:41:07.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f144cff9700 1 -- 172.21.15.3:0/3541196462 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1444002f70 con 0x7f14500614b0 2022-07-21T01:41:07.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.696+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14500ab470 con 0x7f14500614b0 2022-07-21T01:41:07.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.700+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14500ab8b0 con 0x7f14500614b0 2022-07-21T01:41:07.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.700+0000 7f144cff9700 1 -- 172.21.15.3:0/3541196462 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1444025030 con 0x7f14500614b0 2022-07-21T01:41:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.700+0000 7f144cff9700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1430053950 0x7f1430055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.700+0000 7f144cff9700 1 -- 172.21.15.3:0/3541196462 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f144406da50 con 0x7f14500614b0 2022-07-21T01:41:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.700+0000 7f144effd700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1430053950 0x7f1430055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.700+0000 7f144effd700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1430053950 0x7f1430055e10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f144000b000 tx=0x7f1440009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:07.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.708+0000 7f144cff9700 1 -- 172.21.15.3:0/3541196462 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1444043f50 con 0x7f14500614b0 2022-07-21T01:41:07.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.832+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14500574e0 con 0x7f14500614b0 2022-07-21T01:41:07.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f144cff9700 1 -- 172.21.15.3:0/3541196462 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f1444023030 con 0x7f14500614b0 2022-07-21T01:41:07.839 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:07.840 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":111,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:05.595017+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:03.035970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:03.980079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:07.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1430053950 msgr2=0x7f1430055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1430053950 0x7f1430055e10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f144000b000 tx=0x7f1440009040).stop 2022-07-21T01:41:07.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 msgr2=0x7f14500b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 0x7f14500b0d00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f14500b1820 tx=0x7f1444014000).stop 2022-07-21T01:41:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 shutdown_connections 2022-07-21T01:41:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14501405d0 0x7f14500abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1430053950 0x7f1430055e10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14500ad750 0x7f14500b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 --2- 172.21.15.3:0/3541196462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14500614b0 0x7f14500b0d00 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 >> 172.21.15.3:0/3541196462 conn(0x7f1450066f70 msgr2=0x7f1450067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 shutdown_connections 2022-07-21T01:41:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.836+0000 7f1454920700 1 -- 172.21.15.3:0/3541196462 wait complete. 2022-07-21T01:41:07.853 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:41:03.035970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:41:03.980079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:41:07.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:41:07.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca673e1700 1 Processor -- start 2022-07-21T01:41:07.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca673e1700 1 -- start start 2022-07-21T01:41:07.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca673e1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fca60064cd0 0x7fca60143a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca673e1700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca600b2400 con 0x7fca600b2e80 2022-07-21T01:41:07.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca673e1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca600650b0 con 0x7fca600b3ad0 2022-07-21T01:41:07.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca673e1700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca600623c0 con 0x7fca60064cd0 2022-07-21T01:41:07.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca663df700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fca60064cd0 0x7fca60143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca65bde700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fca600b2e80 0x7fca600b33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57422/0 (socket says 172.21.15.3:57422) 2022-07-21T01:41:07.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca65bde700 1 -- 172.21.15.3:0/1335409481 learned_addr learned my addr 172.21.15.3:0/1335409481 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:07.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca653dd700 1 -- 172.21.15.3:0/1335409481 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648767179 0 0) 0x7fca600b2400 con 0x7fca600b2e80 2022-07-21T01:41:07.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca653dd700 1 -- 172.21.15.3:0/1335409481 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca48003260 con 0x7fca600b2e80 2022-07-21T01:41:07.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca663df700 1 -- 172.21.15.3:0/1335409481 >> v1:172.21.15.143:6789/0 conn(0x7fca600b3ad0 legacy=0x7fca6014da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca663df700 1 -- 172.21.15.3:0/1335409481 >> v1:172.21.15.3:6789/0 conn(0x7fca600b2e80 legacy=0x7fca600b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca663df700 1 -- 172.21.15.3:0/1335409481 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca600b3ee0 con 0x7fca60064cd0 2022-07-21T01:41:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.968+0000 7fca663df700 1 --2- 172.21.15.3:0/1335409481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 0x7fca60143a30 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7fca50002f70 tx=0x7fca50006df0).ready entity=mon.0 client_cookie=2b81066199906c9b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca653dd700 1 -- 172.21.15.3:0/1335409481 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca5000d610 con 0x7fca60064cd0 2022-07-21T01:41:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca653dd700 1 -- 172.21.15.3:0/1335409481 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca50000e90 con 0x7fca60064cd0 2022-07-21T01:41:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca653dd700 1 -- 172.21.15.3:0/1335409481 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca5000d960 con 0x7fca60064cd0 2022-07-21T01:41:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca673e1700 1 -- 172.21.15.3:0/1335409481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 msgr2=0x7fca60143a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca673e1700 1 --2- 172.21.15.3:0/1335409481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 0x7fca60143a30 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7fca50002f70 tx=0x7fca50006df0).stop 2022-07-21T01:41:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca673e1700 1 -- 172.21.15.3:0/1335409481 shutdown_connections 2022-07-21T01:41:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca673e1700 1 --2- 172.21.15.3:0/1335409481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 0x7fca60143a30 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca673e1700 1 -- 172.21.15.3:0/1335409481 >> 172.21.15.3:0/1335409481 conn(0x7fca60062d70 msgr2=0x7fca6005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca673e1700 1 -- 172.21.15.3:0/1335409481 shutdown_connections 2022-07-21T01:41:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca673e1700 1 -- 172.21.15.3:0/1335409481 wait complete. 2022-07-21T01:41:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca673e1700 1 Processor -- start 2022-07-21T01:41:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.972+0000 7fca673e1700 1 -- start start 2022-07-21T01:41:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca673e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 0x7fca600ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca673e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca600b2e80 0x7fca600aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca673e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca600b3ad0 0x7fca600af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca673e1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca600b6d70 con 0x7fca60064cd0 2022-07-21T01:41:07.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca673e1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca600b7960 con 0x7fca600b2e80 2022-07-21T01:41:07.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca673e1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca600b7c00 con 0x7fca600b3ad0 2022-07-21T01:41:07.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca663df700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 0x7fca600ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca663df700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 0x7fca600ada60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49148/0 (socket says 172.21.15.3:49148) 2022-07-21T01:41:07.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca663df700 1 -- 172.21.15.3:0/1234443596 learned_addr learned my addr 172.21.15.3:0/1234443596 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:07.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca66be0700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca600b3ad0 0x7fca600af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca65bde700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca600b2e80 0x7fca600aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca663df700 1 -- 172.21.15.3:0/1234443596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca600b3ad0 msgr2=0x7fca600af0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca663df700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca600b3ad0 0x7fca600af0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca663df700 1 -- 172.21.15.3:0/1234443596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca600b2e80 msgr2=0x7fca600aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:07.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca663df700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca600b2e80 0x7fca600aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:07.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca663df700 1 -- 172.21.15.3:0/1234443596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5000b180 con 0x7fca60064cd0 2022-07-21T01:41:07.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca65bde700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca600b2e80 0x7fca600aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:07.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca663df700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 0x7fca600ada60 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7fca50002f70 tx=0x7fca50014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:07.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca4f7fe700 1 -- 172.21.15.3:0/1234443596 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca5000d290 con 0x7fca60064cd0 2022-07-21T01:41:07.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca4f7fe700 1 -- 172.21.15.3:0/1234443596 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca50013e90 con 0x7fca60064cd0 2022-07-21T01:41:07.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca4f7fe700 1 -- 172.21.15.3:0/1234443596 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca5001e5f0 con 0x7fca60064cd0 2022-07-21T01:41:07.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca600b7ea0 con 0x7fca60064cd0 2022-07-21T01:41:07.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca600af590 con 0x7fca60064cd0 2022-07-21T01:41:07.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.976+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca600af780 con 0x7fca60064cd0 2022-07-21T01:41:07.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.980+0000 7fca4f7fe700 1 -- 172.21.15.3:0/1234443596 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fca50025030 con 0x7fca60064cd0 2022-07-21T01:41:07.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.980+0000 7fca4f7fe700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca40053a70 0x7fca40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:07.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.980+0000 7fca4f7fe700 1 -- 172.21.15.3:0/1234443596 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fca5006ce20 con 0x7fca60064cd0 2022-07-21T01:41:07.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.980+0000 7fca65bde700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca40053a70 0x7fca40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:07.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.980+0000 7fca65bde700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca40053a70 0x7fca40055f30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fca5c00b000 tx=0x7fca5c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:07.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:07.988+0000 7fca4f7fe700 1 -- 172.21.15.3:0/1234443596 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fca50043a50 con 0x7fca60064cd0 2022-07-21T01:41:08.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fca600574e0 con 0x7fca60064cd0 2022-07-21T01:41:08.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca4f7fe700 1 -- 172.21.15.3:0/1234443596 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v111) v1 ==== 110+0+10 (secure 0 0 0) 0x7fca50023020 con 0x7fca60064cd0 2022-07-21T01:41:08.126 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:41:08.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca40053a70 msgr2=0x7fca40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca40053a70 0x7fca40055f30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fca5c00b000 tx=0x7fca5c009040).stop 2022-07-21T01:41:08.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 msgr2=0x7fca600ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 0x7fca600ada60 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7fca50002f70 tx=0x7fca50014000).stop 2022-07-21T01:41:08.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 shutdown_connections 2022-07-21T01:41:08.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca600b3ad0 0x7fca600af0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca40053a70 0x7fca40055f30 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca600b2e80 0x7fca600aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 --2- 172.21.15.3:0/1234443596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca60064cd0 0x7fca600ada60 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 >> 172.21.15.3:0/1234443596 conn(0x7fca60062d70 msgr2=0x7fca601426a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:08.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 shutdown_connections 2022-07-21T01:41:08.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.124+0000 7fca673e1700 1 -- 172.21.15.3:0/1234443596 wait complete. 2022-07-21T01:41:08.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:41:08.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.244+0000 7f045de03700 1 Processor -- start 2022-07-21T01:41:08.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.244+0000 7f045de03700 1 -- start start 2022-07-21T01:41:08.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.244+0000 7f045de03700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f04580614b0 0x7f0458061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.244+0000 7f045de03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04580ad750 0x7f04580b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.244+0000 7f045de03700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04580aee80 con 0x7f04581405d0 2022-07-21T01:41:08.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.244+0000 7f045de03700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04580b70b0 con 0x7f04580614b0 2022-07-21T01:41:08.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.244+0000 7f045de03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04580af0b0 con 0x7f04580ad750 2022-07-21T01:41:08.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.244+0000 7f045d602700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f04581405d0 0x7f04581466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57440/0 (socket says 172.21.15.3:57440) 2022-07-21T01:41:08.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.244+0000 7f045d602700 1 -- 172.21.15.3:0/3441307052 learned_addr learned my addr 172.21.15.3:0/3441307052 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:08.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f0457fff700 1 --2- 172.21.15.3:0/3441307052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04580ad750 0x7f04580b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f045ce01700 1 --2- 172.21.15.3:0/3441307052 >> v2:172.21.15.143:3300/0 conn(0x7f04580614b0 0x7f0458061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f04577fe700 1 -- 172.21.15.3:0/3441307052 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175280294 0 0) 0x7f04580aee80 con 0x7f04581405d0 2022-07-21T01:41:08.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f04577fe700 1 -- 172.21.15.3:0/3441307052 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0438003260 con 0x7f04581405d0 2022-07-21T01:41:08.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f0457fff700 1 -- 172.21.15.3:0/3441307052 >> v2:172.21.15.143:3300/0 conn(0x7f04580614b0 msgr2=0x7f0458061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:08.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f0457fff700 1 --2- 172.21.15.3:0/3441307052 >> v2:172.21.15.143:3300/0 conn(0x7f04580614b0 0x7f0458061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f0457fff700 1 -- 172.21.15.3:0/3441307052 >> v1:172.21.15.3:6789/0 conn(0x7f04581405d0 legacy=0x7f04581466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f0457fff700 1 -- 172.21.15.3:0/3441307052 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04580af350 con 0x7f04580ad750 2022-07-21T01:41:08.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f0457fff700 1 --2- 172.21.15.3:0/3441307052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04580ad750 0x7f04580b9ff0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0440002f70 tx=0x7f0440006df0).ready entity=mon.2 client_cookie=294246885740a8ee server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:08.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f04577fe700 1 -- 172.21.15.3:0/3441307052 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f044000d4a0 con 0x7f04580ad750 2022-07-21T01:41:08.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f04577fe700 1 -- 172.21.15.3:0/3441307052 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0440000e90 con 0x7f04580ad750 2022-07-21T01:41:08.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f04577fe700 1 -- 172.21.15.3:0/3441307052 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f044000d7f0 con 0x7f04580ad750 2022-07-21T01:41:08.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f045de03700 1 -- 172.21.15.3:0/3441307052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04580ad750 msgr2=0x7f04580b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f045de03700 1 --2- 172.21.15.3:0/3441307052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04580ad750 0x7f04580b9ff0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0440002f70 tx=0x7f0440006df0).stop 2022-07-21T01:41:08.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f045de03700 1 -- 172.21.15.3:0/3441307052 shutdown_connections 2022-07-21T01:41:08.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f045de03700 1 --2- 172.21.15.3:0/3441307052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04580ad750 0x7f04580b9ff0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f045de03700 1 --2- 172.21.15.3:0/3441307052 >> v2:172.21.15.143:3300/0 conn(0x7f04580614b0 0x7f0458061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f045de03700 1 -- 172.21.15.3:0/3441307052 >> 172.21.15.3:0/3441307052 conn(0x7f0458066f70 msgr2=0x7f0458067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f045de03700 1 -- 172.21.15.3:0/3441307052 shutdown_connections 2022-07-21T01:41:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.248+0000 7f045de03700 1 -- 172.21.15.3:0/3441307052 wait complete. 2022-07-21T01:41:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 Processor -- start 2022-07-21T01:41:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 -- start start 2022-07-21T01:41:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04580614b0 0x7f0458156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04580ad750 0x7f04580b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04581405d0 0x7f04580b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04581439f0 con 0x7f04580614b0 2022-07-21T01:41:08.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0458156950 con 0x7f04580ad750 2022-07-21T01:41:08.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0458143c20 con 0x7f04581405d0 2022-07-21T01:41:08.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045ce01700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04580614b0 0x7f0458156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045ce01700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04580614b0 0x7f0458156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49162/0 (socket says 172.21.15.3:49162) 2022-07-21T01:41:08.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045ce01700 1 -- 172.21.15.3:0/1666616271 learned_addr learned my addr 172.21.15.3:0/1666616271 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:08.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f0457fff700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04580ad750 0x7f04580b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045d602700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04581405d0 0x7f04580b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045ce01700 1 -- 172.21.15.3:0/1666616271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04581405d0 msgr2=0x7f04580b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045ce01700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04581405d0 0x7f04580b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045ce01700 1 -- 172.21.15.3:0/1666616271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04580ad750 msgr2=0x7f04580b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045ce01700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04580ad750 0x7f04580b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045ce01700 1 -- 172.21.15.3:0/1666616271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044000b180 con 0x7f04580614b0 2022-07-21T01:41:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045ce01700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04580614b0 0x7f0458156070 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f044800c000 tx=0x7f0448006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f0455ffb700 1 -- 172.21.15.3:0/1666616271 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0448010780 con 0x7f04580614b0 2022-07-21T01:41:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04581566b0 con 0x7f04580614b0 2022-07-21T01:41:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f0455ffb700 1 -- 172.21.15.3:0/1666616271 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04480108e0 con 0x7f04580614b0 2022-07-21T01:41:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f0455ffb700 1 -- 172.21.15.3:0/1666616271 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0448010c40 con 0x7f04580614b0 2022-07-21T01:41:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.252+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0458140f70 con 0x7f04580614b0 2022-07-21T01:41:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.256+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04580b1d80 con 0x7f04580614b0 2022-07-21T01:41:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.256+0000 7f0455ffb700 1 -- 172.21.15.3:0/1666616271 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0448026430 con 0x7f04580614b0 2022-07-21T01:41:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.256+0000 7f0455ffb700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f043c053950 0x7f043c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.256+0000 7f0455ffb700 1 -- 172.21.15.3:0/1666616271 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f044806cf50 con 0x7f04580614b0 2022-07-21T01:41:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.260+0000 7f0457fff700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f043c053950 0x7f043c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.260+0000 7f0457fff700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f043c053950 0x7f043c055e10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f044001a000 tx=0x7f0440014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:08.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.264+0000 7f0455ffb700 1 -- 172.21.15.3:0/1666616271 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0448043cb0 con 0x7f04580614b0 2022-07-21T01:41:08.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.388+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f04580af5f0 con 0x7f04580614b0 2022-07-21T01:41:08.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.388+0000 7f0455ffb700 1 -- 172.21.15.3:0/1666616271 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v111) v1 ==== 92+0+10 (secure 0 0 0) 0x7f04480472d0 con 0x7f04580614b0 2022-07-21T01:41:08.393 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:08.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f043c053950 msgr2=0x7f043c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f043c053950 0x7f043c055e10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f044001a000 tx=0x7f0440014000).stop 2022-07-21T01:41:08.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04580614b0 msgr2=0x7f0458156070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04580614b0 0x7f0458156070 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f044800c000 tx=0x7f0448006df0).stop 2022-07-21T01:41:08.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 shutdown_connections 2022-07-21T01:41:08.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04581405d0 0x7f04580b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f043c053950 0x7f043c055e10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04580ad750 0x7f04580b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 --2- 172.21.15.3:0/1666616271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04580614b0 0x7f0458156070 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 >> 172.21.15.3:0/1666616271 conn(0x7f0458066f70 msgr2=0x7f0458067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:08.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 shutdown_connections 2022-07-21T01:41:08.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.392+0000 7f045de03700 1 -- 172.21.15.3:0/1666616271 wait complete. 2022-07-21T01:41:08.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:41:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.516+0000 7ffb7b59e700 1 Processor -- start 2022-07-21T01:41:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.516+0000 7ffb7b59e700 1 -- start start 2022-07-21T01:41:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffb7c061ab0 0x7ffb7c143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7c063e40 con 0x7ffb7c140b10 2022-07-21T01:41:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7c062180 con 0x7ffb7c063a60 2022-07-21T01:41:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb7c13ff40 con 0x7ffb7c061ab0 2022-07-21T01:41:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb79d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffb7c061ab0 0x7ffb7c143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7ad9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ffb7c140b10 0x7ffb7c128e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57454/0 (socket says 172.21.15.3:57454) 2022-07-21T01:41:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7ad9d700 1 -- 172.21.15.3:0/3992573278 learned_addr learned my addr 172.21.15.3:0/3992573278 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb79d9b700 1 -- 172.21.15.3:0/3992573278 >> v1:172.21.15.143:6789/0 conn(0x7ffb7c063a60 legacy=0x7ffb7c0b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb79d9b700 1 -- 172.21.15.3:0/3992573278 >> v1:172.21.15.3:6789/0 conn(0x7ffb7c140b10 legacy=0x7ffb7c128e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb79d9b700 1 -- 172.21.15.3:0/3992573278 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7c140130 con 0x7ffb7c061ab0 2022-07-21T01:41:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb79d9b700 1 --2- 172.21.15.3:0/3992573278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 0x7ffb7c143ee0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7ffb64002f70 tx=0x7ffb64006e80).ready entity=mon.0 client_cookie=f880b731b2f75b7e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7959a700 1 -- 172.21.15.3:0/3992573278 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6400bac0 con 0x7ffb7c061ab0 2022-07-21T01:41:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7959a700 1 -- 172.21.15.3:0/3992573278 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64000e90 con 0x7ffb7c061ab0 2022-07-21T01:41:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7959a700 1 -- 172.21.15.3:0/3992573278 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6400be10 con 0x7ffb7c061ab0 2022-07-21T01:41:08.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 -- 172.21.15.3:0/3992573278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 msgr2=0x7ffb7c143ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 --2- 172.21.15.3:0/3992573278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 0x7ffb7c143ee0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7ffb64002f70 tx=0x7ffb64006e80).stop 2022-07-21T01:41:08.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 -- 172.21.15.3:0/3992573278 shutdown_connections 2022-07-21T01:41:08.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 --2- 172.21.15.3:0/3992573278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 0x7ffb7c143ee0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 -- 172.21.15.3:0/3992573278 >> 172.21.15.3:0/3992573278 conn(0x7ffb7c05c620 msgr2=0x7ffb7c05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:08.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 -- 172.21.15.3:0/3992573278 shutdown_connections 2022-07-21T01:41:08.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.520+0000 7ffb7b59e700 1 -- 172.21.15.3:0/3992573278 wait complete. 2022-07-21T01:41:08.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 Processor -- start 2022-07-21T01:41:08.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 -- start start 2022-07-21T01:41:08.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 0x7ffb7c1429c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb7c063a60 0x7ffb7c0644c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb7c140b10 0x7ffb7c0649b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c0af090 con 0x7ffb7c061ab0 2022-07-21T01:41:08.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c0669f0 con 0x7ffb7c140b10 2022-07-21T01:41:08.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c0afd60 con 0x7ffb7c063a60 2022-07-21T01:41:08.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 0x7ffb7c1429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 0x7ffb7c1429c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49176/0 (socket says 172.21.15.3:49176) 2022-07-21T01:41:08.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7a59c700 1 -- 172.21.15.3:0/92784404 learned_addr learned my addr 172.21.15.3:0/92784404 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:08.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7ad9d700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb7c140b10 0x7ffb7c0649b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb79d9b700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb7c063a60 0x7ffb7c0644c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7a59c700 1 -- 172.21.15.3:0/92784404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb7c063a60 msgr2=0x7ffb7c0644c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7a59c700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb7c063a60 0x7ffb7c0644c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7a59c700 1 -- 172.21.15.3:0/92784404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb7c140b10 msgr2=0x7ffb7c0649b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7a59c700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb7c140b10 0x7ffb7c0649b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7a59c700 1 -- 172.21.15.3:0/92784404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6400d410 con 0x7ffb7c061ab0 2022-07-21T01:41:08.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7a59c700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 0x7ffb7c1429c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7ffb7c069fd0 tx=0x7ffb6c007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:08.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb737fe700 1 -- 172.21.15.3:0/92784404 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6c007030 con 0x7ffb7c061ab0 2022-07-21T01:41:08.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb7c0afac0 con 0x7ffb7c061ab0 2022-07-21T01:41:08.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb737fe700 1 -- 172.21.15.3:0/92784404 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6c007190 con 0x7ffb7c061ab0 2022-07-21T01:41:08.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb737fe700 1 -- 172.21.15.3:0/92784404 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6c006510 con 0x7ffb7c061ab0 2022-07-21T01:41:08.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.524+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb7c0aead0 con 0x7ffb7c061ab0 2022-07-21T01:41:08.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.528+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb7c0574e0 con 0x7ffb7c061ab0 2022-07-21T01:41:08.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.528+0000 7ffb737fe700 1 -- 172.21.15.3:0/92784404 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ffb6c01b070 con 0x7ffb7c061ab0 2022-07-21T01:41:08.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.528+0000 7ffb737fe700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffb60053b30 0x7ffb60055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.528+0000 7ffb737fe700 1 -- 172.21.15.3:0/92784404 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ffb6c06b560 con 0x7ffb7c061ab0 2022-07-21T01:41:08.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.528+0000 7ffb79d9b700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffb60053b30 0x7ffb60055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.528+0000 7ffb79d9b700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffb60053b30 0x7ffb60055ff0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7ffb6401e000 tx=0x7ffb7c1402c0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:08.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.536+0000 7ffb737fe700 1 -- 172.21.15.3:0/92784404 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffb6c068e90 con 0x7ffb7c061ab0 2022-07-21T01:41:08.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.660+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffb7c0ad8b0 con 0x7ffb7c061ab0 2022-07-21T01:41:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.664+0000 7ffb737fe700 1 -- 172.21.15.3:0/92784404 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+10 (secure 0 0 0) 0x7ffb6c0470a0 con 0x7ffb7c061ab0 2022-07-21T01:41:08.667 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:08.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffb60053b30 msgr2=0x7ffb60055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffb60053b30 0x7ffb60055ff0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7ffb6401e000 tx=0x7ffb7c1402c0).stop 2022-07-21T01:41:08.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 msgr2=0x7ffb7c1429c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 0x7ffb7c1429c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7ffb7c069fd0 tx=0x7ffb6c007c00).stop 2022-07-21T01:41:08.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 shutdown_connections 2022-07-21T01:41:08.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffb60053b30 0x7ffb60055ff0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb7c140b10 0x7ffb7c0649b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb7c063a60 0x7ffb7c0644c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 --2- 172.21.15.3:0/92784404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb7c061ab0 0x7ffb7c1429c0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 >> 172.21.15.3:0/92784404 conn(0x7ffb7c05c620 msgr2=0x7ffb7c127f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:08.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 shutdown_connections 2022-07-21T01:41:08.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.668+0000 7ffb7b59e700 1 -- 172.21.15.3:0/92784404 wait complete. 2022-07-21T01:41:08.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:41:08.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe699dad700 1 Processor -- start 2022-07-21T01:41:08.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe699dad700 1 -- start start 2022-07-21T01:41:08.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe699dad700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe6940b0740 0x7fe6940b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe699dad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940a9da0 0x7fe6940b1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe699dad700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe694143ab0 con 0x7fe6940b38a0 2022-07-21T01:41:08.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe699dad700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe694143d50 con 0x7fe6940b0740 2022-07-21T01:41:08.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe699dad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe694143ff0 con 0x7fe6940a9da0 2022-07-21T01:41:08.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe692ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe6940b38a0 0x7fe69413d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57466/0 (socket says 172.21.15.3:57466) 2022-07-21T01:41:08.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe692ffd700 1 -- 172.21.15.3:0/1365261470 learned_addr learned my addr 172.21.15.3:0/1365261470 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:08.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe693fff700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940a9da0 0x7fe6940b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6937fe700 1 --2- 172.21.15.3:0/1365261470 >> v2:172.21.15.143:3300/0 conn(0x7fe6940b0740 0x7fe6940b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583954653 0 0) 0x7fe694143ab0 con 0x7fe6940b38a0 2022-07-21T01:41:08.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe680003260 con 0x7fe6940b38a0 2022-07-21T01:41:08.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2353146444 0 0) 0x7fe680003260 con 0x7fe6940b38a0 2022-07-21T01:41:08.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940a9da0 msgr2=0x7fe6940b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:08.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940a9da0 0x7fe6940b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 >> v2:172.21.15.143:3300/0 conn(0x7fe6940b0740 msgr2=0x7fe6940b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:08.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 --2- 172.21.15.3:0/1365261470 >> v2:172.21.15.143:3300/0 conn(0x7fe6940b0740 0x7fe6940b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe694144290 con 0x7fe6940b38a0 2022-07-21T01:41:08.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fe688001660 con 0x7fe6940b38a0 2022-07-21T01:41:08.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 >> v1:172.21.15.3:6789/0 conn(0x7fe6940b38a0 legacy=0x7fe69413d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe680003750 0x7fe680003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940a9da0 0x7fe680004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe680004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe694143d50 con 0x7fe6800049f0 2022-07-21T01:41:08.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe694143ff0 con 0x7fe680003750 2022-07-21T01:41:08.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68000fb60 con 0x7fe6940a9da0 2022-07-21T01:41:08.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe693fff700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe680004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6937fe700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe680003750 0x7fe680003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe692ffd700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940a9da0 0x7fe680004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe693fff700 1 -- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940a9da0 msgr2=0x7fe680004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:08.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe693fff700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940a9da0 0x7fe680004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe693fff700 1 -- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe680003750 msgr2=0x7fe680003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:08.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe693fff700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe680003750 0x7fe680003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe693fff700 1 -- 172.21.15.3:0/1365261470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe680003260 con 0x7fe6800049f0 2022-07-21T01:41:08.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe693fff700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe680004e70 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fe68400c000 tx=0x7fe684006df0).ready entity=mon.0 client_cookie=dff6a45ef36b96ae server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:08.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe68400e830 con 0x7fe6800049f0 2022-07-21T01:41:08.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68400e990 con 0x7fe6800049f0 2022-07-21T01:41:08.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.816+0000 7fe6927fc700 1 -- 172.21.15.3:0/1365261470 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe68400ecf0 con 0x7fe6800049f0 2022-07-21T01:41:08.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 -- 172.21.15.3:0/1365261470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 msgr2=0x7fe680004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 --2- 172.21.15.3:0/1365261470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe680004e70 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fe68400c000 tx=0x7fe684006df0).stop 2022-07-21T01:41:08.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe693fff700 1 -- 172.21.15.3:0/1365261470 reap_dead start 2022-07-21T01:41:08.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 -- 172.21.15.3:0/1365261470 shutdown_connections 2022-07-21T01:41:08.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 -- 172.21.15.3:0/1365261470 >> 172.21.15.3:0/1365261470 conn(0x7fe694060950 msgr2=0x7fe694060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:08.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 -- 172.21.15.3:0/1365261470 shutdown_connections 2022-07-21T01:41:08.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 -- 172.21.15.3:0/1365261470 wait complete. 2022-07-21T01:41:08.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 Processor -- start 2022-07-21T01:41:08.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 -- start start 2022-07-21T01:41:08.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe694140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6941522f0 0x7fe69413aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69413e040 0x7fe69414e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941516e0 con 0x7fe6800049f0 2022-07-21T01:41:08.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe694151910 con 0x7fe6941522f0 2022-07-21T01:41:08.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe699dad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe694151bb0 con 0x7fe69413e040 2022-07-21T01:41:08.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe6937fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe694140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe6937fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe694140510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49196/0 (socket says 172.21.15.3:49196) 2022-07-21T01:41:08.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe6937fe700 1 -- 172.21.15.3:0/829321163 learned_addr learned my addr 172.21.15.3:0/829321163 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:08.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe693fff700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69413e040 0x7fe69414e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe692ffd700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6941522f0 0x7fe69413aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe6937fe700 1 -- 172.21.15.3:0/829321163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69413e040 msgr2=0x7fe69414e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe6937fe700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69413e040 0x7fe69414e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe6937fe700 1 -- 172.21.15.3:0/829321163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6941522f0 msgr2=0x7fe69413aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe6937fe700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6941522f0 0x7fe69413aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe6937fe700 1 -- 172.21.15.3:0/829321163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe694151e50 con 0x7fe6800049f0 2022-07-21T01:41:08.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe6937fe700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe694140510 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7fe68c00b000 tx=0x7fe68c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:08.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.820+0000 7fe690ff9700 1 -- 172.21.15.3:0/829321163 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe68c00fa10 con 0x7fe6800049f0 2022-07-21T01:41:08.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe690ff9700 1 -- 172.21.15.3:0/829321163 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68c00fb70 con 0x7fe6800049f0 2022-07-21T01:41:08.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe690ff9700 1 -- 172.21.15.3:0/829321163 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe68c00fce0 con 0x7fe6800049f0 2022-07-21T01:41:08.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69413e4c0 con 0x7fe6800049f0 2022-07-21T01:41:08.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69414e820 con 0x7fe6800049f0 2022-07-21T01:41:08.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe694151fe0 con 0x7fe6800049f0 2022-07-21T01:41:08.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe690ff9700 1 -- 172.21.15.3:0/829321163 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe68c024070 con 0x7fe6800049f0 2022-07-21T01:41:08.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe690ff9700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe674053a10 0x7fe674055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:08.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe690ff9700 1 -- 172.21.15.3:0/829321163 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe68c06cde0 con 0x7fe6800049f0 2022-07-21T01:41:08.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe692ffd700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe674053a10 0x7fe674055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:08.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.824+0000 7fe692ffd700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe674053a10 0x7fe674055ed0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7fe68800d000 tx=0x7fe688009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:08.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.828+0000 7fe690ff9700 1 -- 172.21.15.3:0/829321163 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe68c043ba0 con 0x7fe6800049f0 2022-07-21T01:41:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.952+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe69413ddf0 con 0x7fe6800049f0 2022-07-21T01:41:08.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.952+0000 7fe690ff9700 1 -- 172.21.15.3:0/829321163 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe68c0471c0 con 0x7fe6800049f0 2022-07-21T01:41:08.957 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe674053a10 msgr2=0x7fe674055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe674053a10 0x7fe674055ed0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7fe68800d000 tx=0x7fe688009040).stop 2022-07-21T01:41:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 msgr2=0x7fe694140510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe694140510 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7fe68c00b000 tx=0x7fe68c0091d0).stop 2022-07-21T01:41:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 shutdown_connections 2022-07-21T01:41:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69413e040 0x7fe69414e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe674053a10 0x7fe674055ed0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6941522f0 0x7fe69413aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 --2- 172.21.15.3:0/829321163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6800049f0 0x7fe694140510 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:08.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 >> 172.21.15.3:0/829321163 conn(0x7fe694060950 msgr2=0x7fe6940b5210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:08.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 shutdown_connections 2022-07-21T01:41:08.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:08.956+0000 7fe699dad700 1 -- 172.21.15.3:0/829321163 wait complete. 2022-07-21T01:41:08.968 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:41:08.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:09.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.068+0000 7feff6394700 1 Processor -- start 2022-07-21T01:41:09.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.068+0000 7feff6394700 1 -- start start 2022-07-21T01:41:09.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feff00b13f0 0x7feff00afd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:09.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7feff00b1920 0x7feff00b0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff00b0740 con 0x7feff00b2710 2022-07-21T01:41:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff00adf10 con 0x7feff00b13f0 2022-07-21T01:41:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff00ae1b0 con 0x7feff00b1920 2022-07-21T01:41:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff4b91700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feff00b13f0 0x7feff00afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff5392700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7feff00b2710 0x7feff00abca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57486/0 (socket says 172.21.15.3:57486) 2022-07-21T01:41:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff5392700 1 -- 172.21.15.3:0/3390620252 learned_addr learned my addr 172.21.15.3:0/3390620252 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff5b93700 1 --2- 172.21.15.3:0/3390620252 >> v2:172.21.15.143:3300/0 conn(0x7feff00b1920 0x7feff00b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff4b91700 1 -- 172.21.15.3:0/3390620252 >> v2:172.21.15.143:3300/0 conn(0x7feff00b1920 msgr2=0x7feff00b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff4b91700 1 --2- 172.21.15.3:0/3390620252 >> v2:172.21.15.143:3300/0 conn(0x7feff00b1920 0x7feff00b0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff4b91700 1 -- 172.21.15.3:0/3390620252 >> v1:172.21.15.3:6789/0 conn(0x7feff00b2710 legacy=0x7feff00abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7fefdffff700 1 -- 172.21.15.3:0/3390620252 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008661470 0 0) 0x7feff00b0740 con 0x7feff00b2710 2022-07-21T01:41:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff4b91700 1 -- 172.21.15.3:0/3390620252 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff00a9520 con 0x7feff00b13f0 2022-07-21T01:41:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff4b91700 1 --2- 172.21.15.3:0/3390620252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b13f0 0x7feff00afd60 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7fefe0002f70 tx=0x7fefe0006df0).ready entity=mon.0 client_cookie=6bad920071e42c57 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7fefdffff700 1 -- 172.21.15.3:0/3390620252 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe000d570 con 0x7feff00b13f0 2022-07-21T01:41:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7fefdffff700 1 -- 172.21.15.3:0/3390620252 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe0000e90 con 0x7feff00b13f0 2022-07-21T01:41:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7fefdffff700 1 -- 172.21.15.3:0/3390620252 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe000d8c0 con 0x7feff00b13f0 2022-07-21T01:41:09.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 -- 172.21.15.3:0/3390620252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b13f0 msgr2=0x7feff00afd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:09.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 --2- 172.21.15.3:0/3390620252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b13f0 0x7feff00afd60 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7fefe0002f70 tx=0x7fefe0006df0).stop 2022-07-21T01:41:09.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 -- 172.21.15.3:0/3390620252 shutdown_connections 2022-07-21T01:41:09.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 --2- 172.21.15.3:0/3390620252 >> v2:172.21.15.143:3300/0 conn(0x7feff00b1920 0x7feff00b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:09.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 --2- 172.21.15.3:0/3390620252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b13f0 0x7feff00afd60 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:09.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 -- 172.21.15.3:0/3390620252 >> 172.21.15.3:0/3390620252 conn(0x7feff0060310 msgr2=0x7feff00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:09.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 -- 172.21.15.3:0/3390620252 shutdown_connections 2022-07-21T01:41:09.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.072+0000 7feff6394700 1 -- 172.21.15.3:0/3390620252 wait complete. 2022-07-21T01:41:09.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 Processor -- start 2022-07-21T01:41:09.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 -- start start 2022-07-21T01:41:09.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff00b13f0 0x7feff01429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:09.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b1920 0x7feff01416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:09.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff00b2710 0x7feff013ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff0065ce0 con 0x7feff00b1920 2022-07-21T01:41:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff00679c0 con 0x7feff00b2710 2022-07-21T01:41:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff0063e10 con 0x7feff00b13f0 2022-07-21T01:41:09.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff4b91700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b1920 0x7feff01416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:09.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff4b91700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b1920 0x7feff01416f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49212/0 (socket says 172.21.15.3:49212) 2022-07-21T01:41:09.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff4b91700 1 -- 172.21.15.3:0/4146884275 learned_addr learned my addr 172.21.15.3:0/4146884275 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:09.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff5392700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff00b13f0 0x7feff01429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:09.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff5b93700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff00b2710 0x7feff013ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:09.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff4b91700 1 -- 172.21.15.3:0/4146884275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff00b13f0 msgr2=0x7feff01429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:09.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff4b91700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff00b13f0 0x7feff01429a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:09.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff4b91700 1 -- 172.21.15.3:0/4146884275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff00b2710 msgr2=0x7feff013ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:09.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff4b91700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff00b2710 0x7feff013ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:09.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff4b91700 1 -- 172.21.15.3:0/4146884275 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe000b180 con 0x7feff00b1920 2022-07-21T01:41:09.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff4b91700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b1920 0x7feff01416f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7feff00644c0 tx=0x7fefe0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:09.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7fefde7fc700 1 -- 172.21.15.3:0/4146884275 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe0016070 con 0x7feff00b1920 2022-07-21T01:41:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7fefde7fc700 1 -- 172.21.15.3:0/4146884275 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe00138d0 con 0x7feff00b1920 2022-07-21T01:41:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff0063190 con 0x7feff00b1920 2022-07-21T01:41:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7fefde7fc700 1 -- 172.21.15.3:0/4146884275 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe0013c30 con 0x7feff00b1920 2022-07-21T01:41:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.076+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff0063970 con 0x7feff00b1920 2022-07-21T01:41:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.080+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff0063540 con 0x7feff00b1920 2022-07-21T01:41:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.080+0000 7fefde7fc700 1 -- 172.21.15.3:0/4146884275 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fefe0025030 con 0x7feff00b1920 2022-07-21T01:41:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.080+0000 7fefde7fc700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fefd0053b00 0x7fefd0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.080+0000 7fefde7fc700 1 -- 172.21.15.3:0/4146884275 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fefe006ce70 con 0x7feff00b1920 2022-07-21T01:41:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.080+0000 7feff5392700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fefd0053b00 0x7fefd0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.080+0000 7feff5392700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fefd0053b00 0x7fefd0055fc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fefec00d000 tx=0x7fefec007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.088+0000 7fefde7fc700 1 -- 172.21.15.3:0/4146884275 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fefe0043c30 con 0x7feff00b1920 2022-07-21T01:41:09.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.248+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feff0063b30 con 0x7feff00b1920 2022-07-21T01:41:09.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.252+0000 7fefde7fc700 1 -- 172.21.15.3:0/4146884275 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2484 (secure 0 0 0) 0x7fefe0023070 con 0x7feff00b1920 2022-07-21T01:41:09.256 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:09.256 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":54,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1707710855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1707710855},{"type":"v1","addr":"172.21.15.143:6833","nonce":1707710855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/764850063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":764850063},{"type":"v1","addr":"172.21.15.3:6835","nonce":764850063}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:04.591520+0000","modified":"2022-07-21T01:41:05.594363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7254":{"gid":7254,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/151311757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":151311757},{"type":"v1","addr":"172.21.15.3:6837","nonce":151311757}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-21T01:41:09.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fefd0053b00 msgr2=0x7fefd0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:09.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fefd0053b00 0x7fefd0055fc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fefec00d000 tx=0x7fefec007a00).stop 2022-07-21T01:41:09.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b1920 msgr2=0x7feff01416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:09.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b1920 0x7feff01416f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7feff00644c0 tx=0x7fefe0014000).stop 2022-07-21T01:41:09.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 shutdown_connections 2022-07-21T01:41:09.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fefd0053b00 0x7fefd0055fc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:09.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff00b2710 0x7feff013ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:09.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff00b1920 0x7feff01416f0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:09.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 --2- 172.21.15.3:0/4146884275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff00b13f0 0x7feff01429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:09.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 >> 172.21.15.3:0/4146884275 conn(0x7feff0060310 msgr2=0x7feff00abcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:09.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 shutdown_connections 2022-07-21T01:41:09.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:09.256+0000 7feff6394700 1 -- 172.21.15.3:0/4146884275 wait complete. 2022-07-21T01:41:09.261 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 54 2022-07-21T01:41:09.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:41:04.591520+0000', 'modified': '2022-07-21T01:41:05.594363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7254': {'gid': 7254, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/151311757', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 151311757}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 151311757}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:41:09.577 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:41:09.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:41:09.702 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":7254,"want_state":"up:active","state":"up:active","rank_uptime":5.0978778,"mdsmap_epoch":54,"osdmap_epoch":111,"osdmap_epoch_barrier":110,"uptime":9.4250970190000007} 2022-07-21T01:41:09.702 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 7254, "mdsmap_epoch": 54, "osdmap_epoch": 111, "osdmap_epoch_barrier": 110, "rank_uptime": 5.0978778, "state": "up:active", "uptime": 9.425097019, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:41:09.703 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:41:09.703 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:41:09.704 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:41:09.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:09.705 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:09.714 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:41:09.773 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:09.774 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:09.785 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:09.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:09.789 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:41:09.796 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:41:09.797 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:09.843 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:41:09.859 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:09.860 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:09.869 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:09.871 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:09.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:09.873 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:41:09.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:41:09.912+0000 7ff6b207d080 -1 init, newargv = 0x561f28fa9a90 newargc=9ceph-fuse[23128]: starting ceph client 2022-07-21T01:41:09.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr: 2022-07-21T01:41:09.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23128]: starting fuse 2022-07-21T01:41:11.021 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:41:11.040 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:11.041 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:11.100 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:11.101 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:11.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:11.104 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:41:11.110 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:41:11.113 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:41:11.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:11.114 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:41:11.115 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:41:11.116 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:41:11.117 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:41:11.118 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:41:11.119 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:41:11.120 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:41:11.120 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:41:11.121 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:41:11.122 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:41:11.123 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:41:11.123 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:41:11.124 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:41:11.124 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:41:11.124 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:41:11.125 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:41:11.125 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:41:11.126 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:41:11.126 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:41:11.127 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:41:11.127 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:41:11.128 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:41:11.128 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:41:11.207 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.23128.asok 2022-07-21T01:41:11.213 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23128.asok 2022-07-21T01:41:11.214 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:11.215 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23128.asok status 2022-07-21T01:41:11.304 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:41:11.305 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:41:11.305 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:41:11.305 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:41:11.306 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:41:11.306 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:41:11.306 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:41:11.307 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "23128", 2022-07-21T01:41:11.307 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:41:11.307 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:41:11.307 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:41:11.308 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:41:11.308 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 6509, 2022-07-21T01:41:11.308 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:41:11.308 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:41:11.309 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:41:11.309 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 6509 2022-07-21T01:41:11.309 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:41:11.309 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:41:11.310 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:41:11.310 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:41:11.310 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2669689771 2022-07-21T01:41:11.310 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:41:11.311 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:41:11.311 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:41:11.311 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:41:11.311 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:41:11.312 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2669689771 2022-07-21T01:41:11.312 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:41:11.312 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.6509 172.21.15.171:0/2669689771", 2022-07-21T01:41:11.312 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/2669689771", 2022-07-21T01:41:11.312 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:41:11.313 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 54, 2022-07-21T01:41:11.313 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 111, 2022-07-21T01:41:11.313 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:41:11.314 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:41:11.315 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:41:11.315 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:41:11.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:11.320 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:11.321 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:11.330 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:41:11.332 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:41:11.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:11.334 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:11.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:11.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.512+0000 7fecec3d4700 1 Processor -- start 2022-07-21T01:41:11.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.512+0000 7fecec3d4700 1 -- start start 2022-07-21T01:41:11.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fece40b1f40 0x7fece40b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40a7250 0x7fece41448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece40b04e0 con 0x7fece4148530 2022-07-21T01:41:11.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece40a9250 con 0x7fece40b1f40 2022-07-21T01:41:11.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece40b0780 con 0x7fece40a7250 2022-07-21T01:41:11.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecea170700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fece40b1f40 0x7fece40b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecea170700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fece40b1f40 0x7fece40b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49218/0 (socket says 172.21.15.3:49218) 2022-07-21T01:41:11.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecea170700 1 -- 172.21.15.3:0/217662442 learned_addr learned my addr 172.21.15.3:0/217662442 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:11.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecea971700 1 --2- 172.21.15.3:0/217662442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40a7250 0x7fece41448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecea170700 1 -- 172.21.15.3:0/217662442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40a7250 msgr2=0x7fece41448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:11.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecea170700 1 --2- 172.21.15.3:0/217662442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40a7250 0x7fece41448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecea170700 1 -- 172.21.15.3:0/217662442 >> v1:172.21.15.143:6789/0 conn(0x7fece4148530 legacy=0x7fece40b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:11.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecea170700 1 -- 172.21.15.3:0/217662442 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece40b0a20 con 0x7fece40b1f40 2022-07-21T01:41:11.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecea170700 1 --2- 172.21.15.3:0/217662442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40b1f40 0x7fece40b1160 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7fece41478c0 tx=0x7fecdc009290).ready entity=mon.0 client_cookie=c959b8d49d8789f3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:11.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fece916e700 1 -- 172.21.15.3:0/217662442 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecdc00b820 con 0x7fece40b1f40 2022-07-21T01:41:11.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fece916e700 1 -- 172.21.15.3:0/217662442 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc00b980 con 0x7fece40b1f40 2022-07-21T01:41:11.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fece916e700 1 -- 172.21.15.3:0/217662442 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecdc00bb30 con 0x7fece40b1f40 2022-07-21T01:41:11.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 -- 172.21.15.3:0/217662442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40b1f40 msgr2=0x7fece40b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 --2- 172.21.15.3:0/217662442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40b1f40 0x7fece40b1160 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7fece41478c0 tx=0x7fecdc009290).stop 2022-07-21T01:41:11.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 -- 172.21.15.3:0/217662442 shutdown_connections 2022-07-21T01:41:11.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 --2- 172.21.15.3:0/217662442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40a7250 0x7fece41448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 --2- 172.21.15.3:0/217662442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40b1f40 0x7fece40b1160 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 -- 172.21.15.3:0/217662442 >> 172.21.15.3:0/217662442 conn(0x7fece405f840 msgr2=0x7fece405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:11.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 -- 172.21.15.3:0/217662442 shutdown_connections 2022-07-21T01:41:11.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.516+0000 7fecec3d4700 1 -- 172.21.15.3:0/217662442 wait complete. 2022-07-21T01:41:11.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.520+0000 7fecec3d4700 1 Processor -- start 2022-07-21T01:41:11.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.520+0000 7fecec3d4700 1 -- start start 2022-07-21T01:41:11.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.520+0000 7fecec3d4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40a7250 0x7fece4143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecec3d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40b1f40 0x7fece4156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecec3d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fece4148530 0x7fece4156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecec3d4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece4140800 con 0x7fece40a7250 2022-07-21T01:41:11.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecec3d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece40ad710 con 0x7fece4148530 2022-07-21T01:41:11.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecec3d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece4140990 con 0x7fece40b1f40 2022-07-21T01:41:11.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea170700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40a7250 0x7fece4143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea170700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40a7250 0x7fece4143720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49224/0 (socket says 172.21.15.3:49224) 2022-07-21T01:41:11.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea170700 1 -- 172.21.15.3:0/227541127 learned_addr learned my addr 172.21.15.3:0/227541127 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:11.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fece996f700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40b1f40 0x7fece4156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea971700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fece4148530 0x7fece4156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea170700 1 -- 172.21.15.3:0/227541127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40b1f40 msgr2=0x7fece4156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea170700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40b1f40 0x7fece4156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea170700 1 -- 172.21.15.3:0/227541127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fece4148530 msgr2=0x7fece4156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea170700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fece4148530 0x7fece4156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea170700 1 -- 172.21.15.3:0/227541127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc009530 con 0x7fece40a7250 2022-07-21T01:41:11.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea971700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fece4148530 0x7fece4156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:11.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecea170700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40a7250 0x7fece4143720 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7fece41478c0 tx=0x7fecdc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:11.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fece37fe700 1 -- 172.21.15.3:0/227541127 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecdc021630 con 0x7fece40a7250 2022-07-21T01:41:11.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fece37fe700 1 -- 172.21.15.3:0/227541127 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc021790 con 0x7fece40a7250 2022-07-21T01:41:11.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece41415b0 con 0x7fece40a7250 2022-07-21T01:41:11.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fece37fe700 1 -- 172.21.15.3:0/227541127 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecdc021af0 con 0x7fece40a7250 2022-07-21T01:41:11.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece41412a0 con 0x7fece40a7250 2022-07-21T01:41:11.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece415acf0 con 0x7fece40a7250 2022-07-21T01:41:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.524+0000 7fece37fe700 1 -- 172.21.15.3:0/227541127 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fecdc027020 con 0x7fece40a7250 2022-07-21T01:41:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.528+0000 7fece37fe700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fecc4053980 0x7fecc4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.528+0000 7fece37fe700 1 -- 172.21.15.3:0/227541127 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fecdc06daa0 con 0x7fece40a7250 2022-07-21T01:41:11.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.528+0000 7fece996f700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fecc4053980 0x7fecc4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.528+0000 7fece996f700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fecc4053980 0x7fecc4055e40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fecd800d000 tx=0x7fecd8007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:11.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.532+0000 7fece37fe700 1 -- 172.21.15.3:0/227541127 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fecdc044860 con 0x7fece40a7250 2022-07-21T01:41:11.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.660+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fece4140ad0 con 0x7fece40a7250 2022-07-21T01:41:11.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.660+0000 7fece37fe700 1 -- 172.21.15.3:0/227541127 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16285 (secure 0 0 0) 0x7fecdc015070 con 0x7fece40a7250 2022-07-21T01:41:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:11.665 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":111,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:05.595017+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:03.035970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:03.980079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:11.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fecc4053980 msgr2=0x7fecc4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fecc4053980 0x7fecc4055e40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fecd800d000 tx=0x7fecd8007b70).stop 2022-07-21T01:41:11.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40a7250 msgr2=0x7fece4143720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40a7250 0x7fece4143720 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7fece41478c0 tx=0x7fecdc016000).stop 2022-07-21T01:41:11.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 shutdown_connections 2022-07-21T01:41:11.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fecc4053980 0x7fecc4055e40 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fece4148530 0x7fece4156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fece40b1f40 0x7fece4156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 --2- 172.21.15.3:0/227541127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fece40a7250 0x7fece4143720 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 >> 172.21.15.3:0/227541127 conn(0x7fece405f840 msgr2=0x7fece40af940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:11.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 shutdown_connections 2022-07-21T01:41:11.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.664+0000 7fecec3d4700 1 -- 172.21.15.3:0/227541127 wait complete. 2022-07-21T01:41:11.679 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:41:03.035970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:41:03.980079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:41:11.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:41:11.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.792+0000 7fae83fff700 1 Processor -- start 2022-07-21T01:41:11.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.792+0000 7fae83fff700 1 -- start start 2022-07-21T01:41:11.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.792+0000 7fae83fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840619b0 0x7fae840b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.792+0000 7fae83fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae840b6ee0 0x7fae84143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.792+0000 7fae83fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae84061d90 con 0x7fae840b5ed0 2022-07-21T01:41:11.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.792+0000 7fae83fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae84142f60 con 0x7fae840b6ee0 2022-07-21T01:41:11.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.792+0000 7fae83fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae840ba5a0 con 0x7fae840619b0 2022-07-21T01:41:11.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae837fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fae840b5ed0 0x7fae840b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57516/0 (socket says 172.21.15.3:57516) 2022-07-21T01:41:11.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae837fe700 1 -- 172.21.15.3:0/3839418259 learned_addr learned my addr 172.21.15.3:0/3839418259 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:11.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae82ffd700 1 --2- 172.21.15.3:0/3839418259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840619b0 0x7fae840b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae827fc700 1 --2- 172.21.15.3:0/3839418259 >> v2:172.21.15.143:3300/0 conn(0x7fae840b6ee0 0x7fae84143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae81ffb700 1 -- 172.21.15.3:0/3839418259 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223327511 0 0) 0x7fae84061d90 con 0x7fae840b5ed0 2022-07-21T01:41:11.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae81ffb700 1 -- 172.21.15.3:0/3839418259 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae64003260 con 0x7fae840b5ed0 2022-07-21T01:41:11.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae82ffd700 1 -- 172.21.15.3:0/3839418259 >> v2:172.21.15.143:3300/0 conn(0x7fae840b6ee0 msgr2=0x7fae84143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:11.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae82ffd700 1 --2- 172.21.15.3:0/3839418259 >> v2:172.21.15.143:3300/0 conn(0x7fae840b6ee0 0x7fae84143890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae82ffd700 1 -- 172.21.15.3:0/3839418259 >> v1:172.21.15.3:6789/0 conn(0x7fae840b5ed0 legacy=0x7fae840b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae82ffd700 1 -- 172.21.15.3:0/3839418259 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae84143150 con 0x7fae840619b0 2022-07-21T01:41:11.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae82ffd700 1 --2- 172.21.15.3:0/3839418259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840619b0 0x7fae840b48a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fae74002f70 tx=0x7fae74006df0).ready entity=mon.2 client_cookie=a9495eb97ab108f5 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:11.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae81ffb700 1 -- 172.21.15.3:0/3839418259 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae7400d4a0 con 0x7fae840619b0 2022-07-21T01:41:11.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae81ffb700 1 -- 172.21.15.3:0/3839418259 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae74000e90 con 0x7fae840619b0 2022-07-21T01:41:11.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae81ffb700 1 -- 172.21.15.3:0/3839418259 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae7400d7f0 con 0x7fae840619b0 2022-07-21T01:41:11.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 -- 172.21.15.3:0/3839418259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840619b0 msgr2=0x7fae840b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 --2- 172.21.15.3:0/3839418259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840619b0 0x7fae840b48a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fae74002f70 tx=0x7fae74006df0).stop 2022-07-21T01:41:11.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 -- 172.21.15.3:0/3839418259 shutdown_connections 2022-07-21T01:41:11.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 --2- 172.21.15.3:0/3839418259 >> v2:172.21.15.143:3300/0 conn(0x7fae840b6ee0 0x7fae84143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 --2- 172.21.15.3:0/3839418259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840619b0 0x7fae840b48a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 -- 172.21.15.3:0/3839418259 >> 172.21.15.3:0/3839418259 conn(0x7fae8406af80 msgr2=0x7fae8405fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:11.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 -- 172.21.15.3:0/3839418259 shutdown_connections 2022-07-21T01:41:11.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 -- 172.21.15.3:0/3839418259 wait complete. 2022-07-21T01:41:11.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 Processor -- start 2022-07-21T01:41:11.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 -- start start 2022-07-21T01:41:11.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae840619b0 0x7fae840b7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae840b5ed0 0x7fae841562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840b6ee0 0x7fae841528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae84152480 con 0x7fae840619b0 2022-07-21T01:41:11.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae840ae270 con 0x7fae840b5ed0 2022-07-21T01:41:11.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.796+0000 7fae83fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae841567c0 con 0x7fae840b6ee0 2022-07-21T01:41:11.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae82ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae840619b0 0x7fae840b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae82ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae840619b0 0x7fae840b7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49238/0 (socket says 172.21.15.3:49238) 2022-07-21T01:41:11.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae82ffd700 1 -- 172.21.15.3:0/690480396 learned_addr learned my addr 172.21.15.3:0/690480396 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:11.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae837fe700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840b6ee0 0x7fae841528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae827fc700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae840b5ed0 0x7fae841562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae82ffd700 1 -- 172.21.15.3:0/690480396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840b6ee0 msgr2=0x7fae841528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae82ffd700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840b6ee0 0x7fae841528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae82ffd700 1 -- 172.21.15.3:0/690480396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae840b5ed0 msgr2=0x7fae841562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae82ffd700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae840b5ed0 0x7fae841562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae82ffd700 1 -- 172.21.15.3:0/690480396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7400b180 con 0x7fae840619b0 2022-07-21T01:41:11.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae827fc700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae840b5ed0 0x7fae841562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:11.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae82ffd700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae840619b0 0x7fae840b7800 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fae74002f70 tx=0x7fae74014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:11.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae7bfff700 1 -- 172.21.15.3:0/690480396 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae74016070 con 0x7fae840619b0 2022-07-21T01:41:11.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae7bfff700 1 -- 172.21.15.3:0/690480396 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae740138d0 con 0x7fae840619b0 2022-07-21T01:41:11.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae840ae4d0 con 0x7fae840619b0 2022-07-21T01:41:11.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae7bfff700 1 -- 172.21.15.3:0/690480396 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae74013c30 con 0x7fae840619b0 2022-07-21T01:41:11.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae840aece0 con 0x7fae840619b0 2022-07-21T01:41:11.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae840adb50 con 0x7fae840619b0 2022-07-21T01:41:11.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.800+0000 7fae7bfff700 1 -- 172.21.15.3:0/690480396 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fae74025030 con 0x7fae840619b0 2022-07-21T01:41:11.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.804+0000 7fae7bfff700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae680539e0 0x7fae68055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:11.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.804+0000 7fae7bfff700 1 -- 172.21.15.3:0/690480396 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fae7406d170 con 0x7fae840619b0 2022-07-21T01:41:11.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.804+0000 7fae827fc700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae680539e0 0x7fae68055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:11.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.804+0000 7fae827fc700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae680539e0 0x7fae68055ea0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fae6c00b000 tx=0x7fae6c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:11.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.808+0000 7fae7bfff700 1 -- 172.21.15.3:0/690480396 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fae840adb50 con 0x7fae840619b0 2022-07-21T01:41:11.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.932+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fae840adb50 con 0x7fae840619b0 2022-07-21T01:41:11.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae7bfff700 1 -- 172.21.15.3:0/690480396 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v111) v1 ==== 110+0+10 (secure 0 0 0) 0x7fae840adb50 con 0x7fae840619b0 2022-07-21T01:41:11.937 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:41:11.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae680539e0 msgr2=0x7fae68055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae680539e0 0x7fae68055ea0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fae6c00b000 tx=0x7fae6c008040).stop 2022-07-21T01:41:11.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae840619b0 msgr2=0x7fae840b7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:11.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae840619b0 0x7fae840b7800 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fae74002f70 tx=0x7fae74014000).stop 2022-07-21T01:41:11.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 shutdown_connections 2022-07-21T01:41:11.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae840b6ee0 0x7fae841528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fae680539e0 0x7fae68055ea0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae840b5ed0 0x7fae841562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 --2- 172.21.15.3:0/690480396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae840619b0 0x7fae840b7800 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:11.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 >> 172.21.15.3:0/690480396 conn(0x7fae8406af80 msgr2=0x7fae8405fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:11.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 shutdown_connections 2022-07-21T01:41:11.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:11.936+0000 7fae83fff700 1 -- 172.21.15.3:0/690480396 wait complete. 2022-07-21T01:41:11.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:41:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe34d619700 1 Processor -- start 2022-07-21T01:41:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe34d619700 1 -- start start 2022-07-21T01:41:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe34d619700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe3480b3190 0x7fe3480b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe34d619700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe3480a9da0 0x7fe348147e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe34d619700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3480b3570 con 0x7fe3480b1a80 2022-07-21T01:41:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe34d619700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3480b2980 con 0x7fe3480b3190 2022-07-21T01:41:12.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe34d619700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3480b0e00 con 0x7fe3480a9da0 2022-07-21T01:41:12.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe347fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe3480b3190 0x7fe3480b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe3477fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe3480b1a80 0x7fe3480656a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57528/0 (socket says 172.21.15.3:57528) 2022-07-21T01:41:12.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe3477fe700 1 -- 172.21.15.3:0/297080038 learned_addr learned my addr 172.21.15.3:0/297080038 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:12.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe34ce18700 1 --2- 172.21.15.3:0/297080038 >> v2:172.21.15.143:3300/0 conn(0x7fe3480a9da0 0x7fe348147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe347fff700 1 -- 172.21.15.3:0/297080038 >> v2:172.21.15.143:3300/0 conn(0x7fe3480a9da0 msgr2=0x7fe348147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:12.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe347fff700 1 --2- 172.21.15.3:0/297080038 >> v2:172.21.15.143:3300/0 conn(0x7fe3480a9da0 0x7fe348147e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe347fff700 1 -- 172.21.15.3:0/297080038 >> v1:172.21.15.3:6789/0 conn(0x7fe3480b1a80 legacy=0x7fe3480656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe346ffd700 1 -- 172.21.15.3:0/297080038 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964496864 0 0) 0x7fe3480b3570 con 0x7fe3480b1a80 2022-07-21T01:41:12.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe347fff700 1 -- 172.21.15.3:0/297080038 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3480b10a0 con 0x7fe3480b3190 2022-07-21T01:41:12.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe347fff700 1 --2- 172.21.15.3:0/297080038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 0x7fe3480b2490 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7fe338002f70 tx=0x7fe338006df0).ready entity=mon.0 client_cookie=dbd6b7a2ec0cf01b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe346ffd700 1 -- 172.21.15.3:0/297080038 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe33800d570 con 0x7fe3480b3190 2022-07-21T01:41:12.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe346ffd700 1 -- 172.21.15.3:0/297080038 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe338000e90 con 0x7fe3480b3190 2022-07-21T01:41:12.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.064+0000 7fe346ffd700 1 -- 172.21.15.3:0/297080038 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe33800d8c0 con 0x7fe3480b3190 2022-07-21T01:41:12.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 -- 172.21.15.3:0/297080038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 msgr2=0x7fe3480b2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 --2- 172.21.15.3:0/297080038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 0x7fe3480b2490 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7fe338002f70 tx=0x7fe338006df0).stop 2022-07-21T01:41:12.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 -- 172.21.15.3:0/297080038 shutdown_connections 2022-07-21T01:41:12.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 --2- 172.21.15.3:0/297080038 >> v2:172.21.15.143:3300/0 conn(0x7fe3480a9da0 0x7fe348147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 --2- 172.21.15.3:0/297080038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 0x7fe3480b2490 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 -- 172.21.15.3:0/297080038 >> 172.21.15.3:0/297080038 conn(0x7fe348063870 msgr2=0x7fe348063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:12.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 -- 172.21.15.3:0/297080038 shutdown_connections 2022-07-21T01:41:12.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 -- 172.21.15.3:0/297080038 wait complete. 2022-07-21T01:41:12.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 Processor -- start 2022-07-21T01:41:12.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.068+0000 7fe34d619700 1 -- start start 2022-07-21T01:41:12.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34d619700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3480a9da0 0x7fe348149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34d619700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3480b1a80 0x7fe3480abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34d619700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 0x7fe348146e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34d619700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3480add30 con 0x7fe3480b3190 2022-07-21T01:41:12.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34d619700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3480acbc0 con 0x7fe3480a9da0 2022-07-21T01:41:12.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34d619700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3480ab6b0 con 0x7fe3480b1a80 2022-07-21T01:41:12.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34ce18700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 0x7fe348146e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34ce18700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 0x7fe348146e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49256/0 (socket says 172.21.15.3:49256) 2022-07-21T01:41:12.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34ce18700 1 -- 172.21.15.3:0/637954651 learned_addr learned my addr 172.21.15.3:0/637954651 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:12.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe3477fe700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3480b1a80 0x7fe3480abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe347fff700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3480a9da0 0x7fe348149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34ce18700 1 -- 172.21.15.3:0/637954651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3480b1a80 msgr2=0x7fe3480abdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34ce18700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3480b1a80 0x7fe3480abdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34ce18700 1 -- 172.21.15.3:0/637954651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3480a9da0 msgr2=0x7fe348149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34ce18700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3480a9da0 0x7fe348149d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34ce18700 1 -- 172.21.15.3:0/637954651 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33800b180 con 0x7fe3480b3190 2022-07-21T01:41:12.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34ce18700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 0x7fe348146e20 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7fe3480aeac0 tx=0x7fe3400091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe3457fa700 1 -- 172.21.15.3:0/637954651 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe340012600 con 0x7fe3480b3190 2022-07-21T01:41:12.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe3457fa700 1 -- 172.21.15.3:0/637954651 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe340012760 con 0x7fe3480b3190 2022-07-21T01:41:12.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3480ad830 con 0x7fe3480b3190 2022-07-21T01:41:12.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe3457fa700 1 -- 172.21.15.3:0/637954651 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe340012ac0 con 0x7fe3480b3190 2022-07-21T01:41:12.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.072+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3480ada90 con 0x7fe3480b3190 2022-07-21T01:41:12.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.076+0000 7fe3457fa700 1 -- 172.21.15.3:0/637954651 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe340022070 con 0x7fe3480b3190 2022-07-21T01:41:12.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.076+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3480574e0 con 0x7fe3480b3190 2022-07-21T01:41:12.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.076+0000 7fe3457fa700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c0539e0 0x7fe32c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.076+0000 7fe3457fa700 1 -- 172.21.15.3:0/637954651 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe34006c660 con 0x7fe3480b3190 2022-07-21T01:41:12.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.076+0000 7fe347fff700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c0539e0 0x7fe32c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.076+0000 7fe347fff700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c0539e0 0x7fe32c055ea0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fe33801a000 tx=0x7fe338014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.084+0000 7fe3457fa700 1 -- 172.21.15.3:0/637954651 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe340043310 con 0x7fe3480b3190 2022-07-21T01:41:12.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.208+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe348069a70 con 0x7fe3480b3190 2022-07-21T01:41:12.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.208+0000 7fe3457fa700 1 -- 172.21.15.3:0/637954651 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v111) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe340016020 con 0x7fe3480b3190 2022-07-21T01:41:12.213 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:12.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c0539e0 msgr2=0x7fe32c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c0539e0 0x7fe32c055ea0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fe33801a000 tx=0x7fe338014000).stop 2022-07-21T01:41:12.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 msgr2=0x7fe348146e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 0x7fe348146e20 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7fe3480aeac0 tx=0x7fe3400091d0).stop 2022-07-21T01:41:12.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 shutdown_connections 2022-07-21T01:41:12.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480b3190 0x7fe348146e20 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3480b1a80 0x7fe3480abdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe32c0539e0 0x7fe32c055ea0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 --2- 172.21.15.3:0/637954651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3480a9da0 0x7fe348149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 >> 172.21.15.3:0/637954651 conn(0x7fe348063870 msgr2=0x7fe348061040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:12.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 shutdown_connections 2022-07-21T01:41:12.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.212+0000 7fe34d619700 1 -- 172.21.15.3:0/637954651 wait complete. 2022-07-21T01:41:12.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:41:12.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.336+0000 7f36eee0f700 1 Processor -- start 2022-07-21T01:41:12.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.336+0000 7f36eee0f700 1 -- start start 2022-07-21T01:41:12.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.336+0000 7f36eee0f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f36e80619b0 0x7f36e80b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.336+0000 7f36eee0f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e8061d90 con 0x7f36e80b5ed0 2022-07-21T01:41:12.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.336+0000 7f36eee0f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e80ba5a0 con 0x7f36e80b6ee0 2022-07-21T01:41:12.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.336+0000 7f36eee0f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e8143fa0 con 0x7f36e80619b0 2022-07-21T01:41:12.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ede0d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f36e80619b0 0x7f36e80b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ed60c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f36e80b6ee0 0x7f36e8143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:59658/0 (socket says 172.21.15.3:59658) 2022-07-21T01:41:12.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ed60c700 1 -- 172.21.15.3:0/940526820 learned_addr learned my addr 172.21.15.3:0/940526820 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:12.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ee60e700 1 --1- 172.21.15.3:0/940526820 >> v1:172.21.15.3:6789/0 conn(0x7f36e80b5ed0 0x7f36e80b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57544/0 (socket says 172.21.15.3:57544) 2022-07-21T01:41:12.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ece0b700 1 -- 172.21.15.3:0/940526820 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214057779 0 0) 0x7f36e8061d90 con 0x7f36e80b5ed0 2022-07-21T01:41:12.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ede0d700 1 -- 172.21.15.3:0/940526820 >> v1:172.21.15.143:6789/0 conn(0x7f36e80b6ee0 legacy=0x7f36e8143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ede0d700 1 -- 172.21.15.3:0/940526820 >> v1:172.21.15.3:6789/0 conn(0x7f36e80b5ed0 legacy=0x7f36e80b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ede0d700 1 -- 172.21.15.3:0/940526820 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e8143150 con 0x7f36e80619b0 2022-07-21T01:41:12.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ede0d700 1 --2- 172.21.15.3:0/940526820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 0x7f36e80b48a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f36e4001110 tx=0x7f36dc006df0).ready entity=mon.0 client_cookie=6bd75f56e20ccb03 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ece0b700 1 -- 172.21.15.3:0/940526820 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36dc00d5c0 con 0x7f36e80619b0 2022-07-21T01:41:12.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ece0b700 1 -- 172.21.15.3:0/940526820 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc000e90 con 0x7f36e80619b0 2022-07-21T01:41:12.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36ece0b700 1 -- 172.21.15.3:0/940526820 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36dc00d910 con 0x7f36e80619b0 2022-07-21T01:41:12.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36eee0f700 1 -- 172.21.15.3:0/940526820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 msgr2=0x7f36e80b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36eee0f700 1 --2- 172.21.15.3:0/940526820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 0x7f36e80b48a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f36e4001110 tx=0x7f36dc006df0).stop 2022-07-21T01:41:12.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36eee0f700 1 -- 172.21.15.3:0/940526820 shutdown_connections 2022-07-21T01:41:12.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36eee0f700 1 --2- 172.21.15.3:0/940526820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 0x7f36e80b48a0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36eee0f700 1 -- 172.21.15.3:0/940526820 >> 172.21.15.3:0/940526820 conn(0x7f36e806af80 msgr2=0x7f36e805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:12.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36eee0f700 1 -- 172.21.15.3:0/940526820 shutdown_connections 2022-07-21T01:41:12.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.340+0000 7f36eee0f700 1 -- 172.21.15.3:0/940526820 wait complete. 2022-07-21T01:41:12.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36eee0f700 1 Processor -- start 2022-07-21T01:41:12.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36eee0f700 1 -- start start 2022-07-21T01:41:12.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36eee0f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 0x7f36e8112d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36eee0f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36e80b5ed0 0x7f36e8152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36eee0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36e80b6ee0 0x7f36e8152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36eee0f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80b01c0 con 0x7f36e80619b0 2022-07-21T01:41:12.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36eee0f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e8145ce0 con 0x7f36e80b5ed0 2022-07-21T01:41:12.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36eee0f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80afdc0 con 0x7f36e80b6ee0 2022-07-21T01:41:12.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ede0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 0x7f36e8112d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ede0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 0x7f36e8112d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49266/0 (socket says 172.21.15.3:49266) 2022-07-21T01:41:12.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ede0d700 1 -- 172.21.15.3:0/2846571371 learned_addr learned my addr 172.21.15.3:0/2846571371 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:12.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ed60c700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36e80b5ed0 0x7f36e8152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ee60e700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36e80b6ee0 0x7f36e8152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ede0d700 1 -- 172.21.15.3:0/2846571371 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36e80b6ee0 msgr2=0x7f36e8152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ede0d700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36e80b6ee0 0x7f36e8152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ede0d700 1 -- 172.21.15.3:0/2846571371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36e80b5ed0 msgr2=0x7f36e8152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ede0d700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36e80b5ed0 0x7f36e8152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ede0d700 1 -- 172.21.15.3:0/2846571371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc00b180 con 0x7f36e80619b0 2022-07-21T01:41:12.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36ede0d700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 0x7f36e8112d70 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7f36e4001110 tx=0x7f36dc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.344+0000 7f36daffd700 1 -- 172.21.15.3:0/2846571371 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36dc013de0 con 0x7f36e80619b0 2022-07-21T01:41:12.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36daffd700 1 -- 172.21.15.3:0/2846571371 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc002f50 con 0x7f36e80619b0 2022-07-21T01:41:12.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e80af750 con 0x7f36e80619b0 2022-07-21T01:41:12.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36daffd700 1 -- 172.21.15.3:0/2846571371 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36dc01e5f0 con 0x7f36e80619b0 2022-07-21T01:41:12.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e8113260 con 0x7f36e80619b0 2022-07-21T01:41:12.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e80b0330 con 0x7f36e80619b0 2022-07-21T01:41:12.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36daffd700 1 -- 172.21.15.3:0/2846571371 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f36dc025030 con 0x7f36e80619b0 2022-07-21T01:41:12.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36daffd700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36c8053a70 0x7f36c8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36daffd700 1 -- 172.21.15.3:0/2846571371 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f36dc06ce20 con 0x7f36e80619b0 2022-07-21T01:41:12.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36ed60c700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36c8053a70 0x7f36c8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.348+0000 7f36ed60c700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36c8053a70 0x7f36c8055f30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f36e400d000 tx=0x7f36e4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.356+0000 7f36daffd700 1 -- 172.21.15.3:0/2846571371 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f36e80b0330 con 0x7f36e80619b0 2022-07-21T01:41:12.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.480+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f36e80b0800 con 0x7f36e80619b0 2022-07-21T01:41:12.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.480+0000 7f36daffd700 1 -- 172.21.15.3:0/2846571371 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+10 (secure 0 0 0) 0x7f36dc023020 con 0x7f36e80619b0 2022-07-21T01:41:12.484 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:12.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36c8053a70 msgr2=0x7f36c8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36c8053a70 0x7f36c8055f30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f36e400d000 tx=0x7f36e4009040).stop 2022-07-21T01:41:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 msgr2=0x7f36e8112d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 0x7f36e8112d70 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7f36e4001110 tx=0x7f36dc014000).stop 2022-07-21T01:41:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 shutdown_connections 2022-07-21T01:41:12.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36e80b6ee0 0x7f36e8152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f36c8053a70 0x7f36c8055f30 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36e80b5ed0 0x7f36e8152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 --2- 172.21.15.3:0/2846571371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e80619b0 0x7f36e8112d70 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 >> 172.21.15.3:0/2846571371 conn(0x7f36e806af80 msgr2=0x7f36e805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:12.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 shutdown_connections 2022-07-21T01:41:12.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.484+0000 7f36eee0f700 1 -- 172.21.15.3:0/2846571371 wait complete. 2022-07-21T01:41:12.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:41:12.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.604+0000 7fd281558700 1 Processor -- start 2022-07-21T01:41:12.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.604+0000 7fd281558700 1 -- start start 2022-07-21T01:41:12.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.604+0000 7fd281558700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c060290 0x7fd27c064ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.604+0000 7fd281558700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd27c0b3a60 0x7fd27c065390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.604+0000 7fd281558700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27c060670 con 0x7fd27c1445d0 2022-07-21T01:41:12.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.604+0000 7fd281558700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd27c065880 con 0x7fd27c0b3a60 2022-07-21T01:41:12.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.604+0000 7fd281558700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c065e10 con 0x7fd27c060290 2022-07-21T01:41:12.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd27c0b3a60 0x7fd27c065390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c060290 0x7fd27c064ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd27c0b3a60 0x7fd27c065390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49278/0 (socket says 172.21.15.3:49278) 2022-07-21T01:41:12.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd2727fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd27c1445d0 0x7fd27c144b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57556/0 (socket says 172.21.15.3:57556) 2022-07-21T01:41:12.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27b7fe700 1 -- 172.21.15.3:0/3808210837 learned_addr learned my addr 172.21.15.3:0/3808210837 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:12.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27b7fe700 1 -- 172.21.15.3:0/3808210837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c060290 msgr2=0x7fd27c064ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:12.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27b7fe700 1 --2- 172.21.15.3:0/3808210837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c060290 0x7fd27c064ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27b7fe700 1 -- 172.21.15.3:0/3808210837 >> v1:172.21.15.3:6789/0 conn(0x7fd27c1445d0 legacy=0x7fd27c144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27b7fe700 1 -- 172.21.15.3:0/3808210837 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c0b2540 con 0x7fd27c0b3a60 2022-07-21T01:41:12.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27b7fe700 1 --2- 172.21.15.3:0/3808210837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c0b3a60 0x7fd27c065390 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7fd2740032f0 tx=0x7fd274009290).ready entity=mon.0 client_cookie=e086d972e8442b0a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27a7fc700 1 -- 172.21.15.3:0/3808210837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd27400b8a0 con 0x7fd27c0b3a60 2022-07-21T01:41:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27a7fc700 1 -- 172.21.15.3:0/3808210837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27400ba00 con 0x7fd27c0b3a60 2022-07-21T01:41:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd27a7fc700 1 -- 172.21.15.3:0/3808210837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd27400bb70 con 0x7fd27c0b3a60 2022-07-21T01:41:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 -- 172.21.15.3:0/3808210837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c0b3a60 msgr2=0x7fd27c065390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 --2- 172.21.15.3:0/3808210837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c0b3a60 0x7fd27c065390 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7fd2740032f0 tx=0x7fd274009290).stop 2022-07-21T01:41:12.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 -- 172.21.15.3:0/3808210837 shutdown_connections 2022-07-21T01:41:12.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 --2- 172.21.15.3:0/3808210837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c0b3a60 0x7fd27c065390 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 --2- 172.21.15.3:0/3808210837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c060290 0x7fd27c064ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 -- 172.21.15.3:0/3808210837 >> 172.21.15.3:0/3808210837 conn(0x7fd27c05d0f0 msgr2=0x7fd27c05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:12.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 -- 172.21.15.3:0/3808210837 shutdown_connections 2022-07-21T01:41:12.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 -- 172.21.15.3:0/3808210837 wait complete. 2022-07-21T01:41:12.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 Processor -- start 2022-07-21T01:41:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.608+0000 7fd281558700 1 -- start start 2022-07-21T01:41:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd281558700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c060290 0x7fd27c154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd281558700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd27c0b3a60 0x7fd27c141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd281558700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c0ac030 0x7fd27c142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd281558700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c144030 con 0x7fd27c060290 2022-07-21T01:41:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd281558700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c153ca0 con 0x7fd27c0b3a60 2022-07-21T01:41:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd281558700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c154b30 con 0x7fd27c0ac030 2022-07-21T01:41:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c060290 0x7fd27c154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c060290 0x7fd27c154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49280/0 (socket says 172.21.15.3:49280) 2022-07-21T01:41:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27affd700 1 -- 172.21.15.3:0/2504703853 learned_addr learned my addr 172.21.15.3:0/2504703853 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27b7fe700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c0ac030 0x7fd27c142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd2727fc700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd27c0b3a60 0x7fd27c141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27affd700 1 -- 172.21.15.3:0/2504703853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c0ac030 msgr2=0x7fd27c142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27affd700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c0ac030 0x7fd27c142320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27affd700 1 -- 172.21.15.3:0/2504703853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd27c0b3a60 msgr2=0x7fd27c141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27affd700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd27c0b3a60 0x7fd27c141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27affd700 1 -- 172.21.15.3:0/2504703853 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd274009530 con 0x7fd27c060290 2022-07-21T01:41:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd27affd700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c060290 0x7fd27c154640 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7fd26c00c000 tx=0x7fd26c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd278ff9700 1 -- 172.21.15.3:0/2504703853 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd26c010780 con 0x7fd27c060290 2022-07-21T01:41:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd278ff9700 1 -- 172.21.15.3:0/2504703853 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c0108e0 con 0x7fd27c060290 2022-07-21T01:41:12.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27c144260 con 0x7fd27c060290 2022-07-21T01:41:12.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd278ff9700 1 -- 172.21.15.3:0/2504703853 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd26c010c40 con 0x7fd27c060290 2022-07-21T01:41:12.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27c13e480 con 0x7fd27c060290 2022-07-21T01:41:12.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27c0574e0 con 0x7fd27c060290 2022-07-21T01:41:12.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd278ff9700 1 -- 172.21.15.3:0/2504703853 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd26c01c460 con 0x7fd27c060290 2022-07-21T01:41:12.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd278ff9700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd260053aa0 0x7fd260055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.612+0000 7fd278ff9700 1 -- 172.21.15.3:0/2504703853 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd26c06c440 con 0x7fd27c060290 2022-07-21T01:41:12.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.616+0000 7fd2727fc700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd260053aa0 0x7fd260055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.616+0000 7fd2727fc700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd260053aa0 0x7fd260055f60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fd26400d000 tx=0x7fd264007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.620+0000 7fd278ff9700 1 -- 172.21.15.3:0/2504703853 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd26c043100 con 0x7fd27c060290 2022-07-21T01:41:12.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.744+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd27c153e10 con 0x7fd27c060290 2022-07-21T01:41:12.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.744+0000 7fd278ff9700 1 -- 172.21.15.3:0/2504703853 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd26c046720 con 0x7fd27c060290 2022-07-21T01:41:12.749 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:12.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd260053aa0 msgr2=0x7fd260055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd260053aa0 0x7fd260055f60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fd26400d000 tx=0x7fd264007b80).stop 2022-07-21T01:41:12.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c060290 msgr2=0x7fd27c154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c060290 0x7fd27c154640 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7fd26c00c000 tx=0x7fd26c006df0).stop 2022-07-21T01:41:12.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 shutdown_connections 2022-07-21T01:41:12.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd27c0ac030 0x7fd27c142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd260053aa0 0x7fd260055f60 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd27c0b3a60 0x7fd27c141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 --2- 172.21.15.3:0/2504703853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd27c060290 0x7fd27c154640 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 >> 172.21.15.3:0/2504703853 conn(0x7fd27c05d0f0 msgr2=0x7fd27c064dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:12.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 shutdown_connections 2022-07-21T01:41:12.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.748+0000 7fd281558700 1 -- 172.21.15.3:0/2504703853 wait complete. 2022-07-21T01:41:12.761 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:41:12.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:12.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.860+0000 7fb50ce36700 1 Processor -- start 2022-07-21T01:41:12.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.860+0000 7fb50ce36700 1 -- start start 2022-07-21T01:41:12.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.860+0000 7fb50ce36700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5080ad750 0x7fb508061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.860+0000 7fb50ce36700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5081405d0 0x7fb508140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.860+0000 7fb50ce36700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5080aee80 con 0x7fb5080614b0 2022-07-21T01:41:12.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5080b70b0 con 0x7fb5080ad750 2022-07-21T01:41:12.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5080af0b0 con 0x7fb5081405d0 2022-07-21T01:41:12.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb506ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5080ad750 0x7fb508061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb506ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5080ad750 0x7fb508061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49290/0 (socket says 172.21.15.3:49290) 2022-07-21T01:41:12.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb506ffd700 1 -- 172.21.15.3:0/3297391578 learned_addr learned my addr 172.21.15.3:0/3297391578 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:12.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb507fff700 1 --2- 172.21.15.3:0/3297391578 >> v2:172.21.15.143:3300/0 conn(0x7fb5081405d0 0x7fb508140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb506ffd700 1 -- 172.21.15.3:0/3297391578 >> v2:172.21.15.143:3300/0 conn(0x7fb5081405d0 msgr2=0x7fb508140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:12.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb506ffd700 1 --2- 172.21.15.3:0/3297391578 >> v2:172.21.15.143:3300/0 conn(0x7fb5081405d0 0x7fb508140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb506ffd700 1 -- 172.21.15.3:0/3297391578 >> v1:172.21.15.143:6789/0 conn(0x7fb5080614b0 legacy=0x7fb5080b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb506ffd700 1 -- 172.21.15.3:0/3297391578 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5080af350 con 0x7fb5080ad750 2022-07-21T01:41:12.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb506ffd700 1 --2- 172.21.15.3:0/3297391578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5080ad750 0x7fb508061890 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7fb4f8002f70 tx=0x7fb4f8006eb0).ready entity=mon.0 client_cookie=ad7ff65f2212939e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb5067fc700 1 -- 172.21.15.3:0/3297391578 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f800d5c0 con 0x7fb5080ad750 2022-07-21T01:41:12.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb5067fc700 1 -- 172.21.15.3:0/3297391578 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f8000e90 con 0x7fb5080ad750 2022-07-21T01:41:12.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb5067fc700 1 -- 172.21.15.3:0/3297391578 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f800d950 con 0x7fb5080ad750 2022-07-21T01:41:12.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 -- 172.21.15.3:0/3297391578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5080ad750 msgr2=0x7fb508061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 --2- 172.21.15.3:0/3297391578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5080ad750 0x7fb508061890 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7fb4f8002f70 tx=0x7fb4f8006eb0).stop 2022-07-21T01:41:12.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 -- 172.21.15.3:0/3297391578 shutdown_connections 2022-07-21T01:41:12.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 --2- 172.21.15.3:0/3297391578 >> v2:172.21.15.143:3300/0 conn(0x7fb5081405d0 0x7fb508140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 --2- 172.21.15.3:0/3297391578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5080ad750 0x7fb508061890 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 -- 172.21.15.3:0/3297391578 >> 172.21.15.3:0/3297391578 conn(0x7fb508066f70 msgr2=0x7fb508067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:12.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 -- 172.21.15.3:0/3297391578 shutdown_connections 2022-07-21T01:41:12.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.864+0000 7fb50ce36700 1 -- 172.21.15.3:0/3297391578 wait complete. 2022-07-21T01:41:12.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 Processor -- start 2022-07-21T01:41:12.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 -- start start 2022-07-21T01:41:12.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5080614b0 0x7fb5080ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5080ad750 0x7fb5080ac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5081405d0 0x7fb5080b3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5080ac0b0 con 0x7fb5081405d0 2022-07-21T01:41:12.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5080aae50 con 0x7fb5080ad750 2022-07-21T01:41:12.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5081534b0 con 0x7fb5080614b0 2022-07-21T01:41:12.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb507fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5081405d0 0x7fb5080b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb507fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5081405d0 0x7fb5080b3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49298/0 (socket says 172.21.15.3:49298) 2022-07-21T01:41:12.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb507fff700 1 -- 172.21.15.3:0/1402385130 learned_addr learned my addr 172.21.15.3:0/1402385130 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:12.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb506ffd700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5080ad750 0x7fb5080ac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb5077fe700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5080614b0 0x7fb5080ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb507fff700 1 -- 172.21.15.3:0/1402385130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5080614b0 msgr2=0x7fb5080ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb507fff700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5080614b0 0x7fb5080ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb507fff700 1 -- 172.21.15.3:0/1402385130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5080ad750 msgr2=0x7fb5080ac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:12.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb507fff700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5080ad750 0x7fb5080ac2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:12.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb507fff700 1 -- 172.21.15.3:0/1402385130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f800b1e0 con 0x7fb5081405d0 2022-07-21T01:41:12.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb507fff700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5081405d0 0x7fb5080b3840 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7fb500006e00 tx=0x7fb5000091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb504ff9700 1 -- 172.21.15.3:0/1402385130 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb50000c670 con 0x7fb5081405d0 2022-07-21T01:41:12.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb504ff9700 1 -- 172.21.15.3:0/1402385130 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50000c7d0 con 0x7fb5081405d0 2022-07-21T01:41:12.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5080ab080 con 0x7fb5081405d0 2022-07-21T01:41:12.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb504ff9700 1 -- 172.21.15.3:0/1402385130 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb50000cb30 con 0x7fb5081405d0 2022-07-21T01:41:12.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5081530b0 con 0x7fb5081405d0 2022-07-21T01:41:12.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.868+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5080af5f0 con 0x7fb5081405d0 2022-07-21T01:41:12.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.872+0000 7fb504ff9700 1 -- 172.21.15.3:0/1402385130 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb500022070 con 0x7fb5081405d0 2022-07-21T01:41:12.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.872+0000 7fb504ff9700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb4e8053950 0x7fb4e8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:12.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.872+0000 7fb504ff9700 1 -- 172.21.15.3:0/1402385130 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb50006c7c0 con 0x7fb5081405d0 2022-07-21T01:41:12.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.872+0000 7fb5077fe700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb4e8053950 0x7fb4e8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:12.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.872+0000 7fb5077fe700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb4e8053950 0x7fb4e8055e10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fb4fc00b000 tx=0x7fb4fc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:12.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:12.880+0000 7fb504ff9700 1 -- 172.21.15.3:0/1402385130 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb500043410 con 0x7fb5081405d0 2022-07-21T01:41:13.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.040+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5080574e0 con 0x7fb5081405d0 2022-07-21T01:41:13.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.040+0000 7fb504ff9700 1 -- 172.21.15.3:0/1402385130 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2484 (secure 0 0 0) 0x7fb500046a30 con 0x7fb5081405d0 2022-07-21T01:41:13.044 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:13.044 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":54,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1707710855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1707710855},{"type":"v1","addr":"172.21.15.143:6833","nonce":1707710855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/764850063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":764850063},{"type":"v1","addr":"172.21.15.3:6835","nonce":764850063}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:04.591520+0000","modified":"2022-07-21T01:41:05.594363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7254":{"gid":7254,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/151311757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":151311757},{"type":"v1","addr":"172.21.15.3:6837","nonce":151311757}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-21T01:41:13.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb4e8053950 msgr2=0x7fb4e8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:13.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb4e8053950 0x7fb4e8055e10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fb4fc00b000 tx=0x7fb4fc009040).stop 2022-07-21T01:41:13.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5081405d0 msgr2=0x7fb5080b3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:13.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5081405d0 0x7fb5080b3840 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7fb500006e00 tx=0x7fb5000091d0).stop 2022-07-21T01:41:13.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 shutdown_connections 2022-07-21T01:41:13.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5081405d0 0x7fb5080b3840 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:13.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb4e8053950 0x7fb4e8055e10 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:13.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5080ad750 0x7fb5080ac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:13.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 --2- 172.21.15.3:0/1402385130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5080614b0 0x7fb5080ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:13.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 >> 172.21.15.3:0/1402385130 conn(0x7fb508066f70 msgr2=0x7fb508145620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:13.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 shutdown_connections 2022-07-21T01:41:13.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:13.044+0000 7fb50ce36700 1 -- 172.21.15.3:0/1402385130 wait complete. 2022-07-21T01:41:13.049 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 54 2022-07-21T01:41:13.361 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:41:04.591520+0000', 'modified': '2022-07-21T01:41:05.594363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7254': {'gid': 7254, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/151311757', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 151311757}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 151311757}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:41:13.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:41:13.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:41:13.482 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":7254,"want_state":"up:active","state":"up:active","rank_uptime":8.8777767000000001,"mdsmap_epoch":54,"osdmap_epoch":111,"osdmap_epoch_barrier":110,"uptime":13.204999126000001} 2022-07-21T01:41:13.482 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 7254, "mdsmap_epoch": 54, "osdmap_epoch": 111, "osdmap_epoch_barrier": 110, "rank_uptime": 8.8777767, "state": "up:active", "uptime": 13.204999126, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:41:13.483 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:41:13.483 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:41:13.484 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:41:13.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:13.485 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:13.494 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:41:13.552 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:13.553 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:13.563 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:13.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:13.567 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:41:13.574 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:41:13.575 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:13.578 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:41:13.599 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:13.600 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:13.654 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:41:13.653+0000 7fd018902080 -1 init, newargv = 0x5644956a8a00 newargc=9ceph-fuse[ 2022-07-21T01:41:13.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:22706]: starting ceph client 2022-07-21T01:41:13.658 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:13.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:13.661 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:41:13.664 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22706]: starting fuse 2022-07-21T01:41:13.667 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:41:13.667 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:41:13.668 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:13.668 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:41:13.669 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:41:13.670 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:41:13.671 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:41:13.671 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:41:13.672 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:41:13.672 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:41:13.673 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:41:13.673 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:41:13.674 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:41:13.674 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:41:13.675 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:41:13.675 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:41:13.676 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:41:13.676 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:41:13.677 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:41:13.677 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:41:13.678 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:41:13.679 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:41:13.679 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:41:13.680 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:41:13.680 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:41:13.681 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:41:13.751 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.22706.asok 2022-07-21T01:41:13.757 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22706.asok 2022-07-21T01:41:13.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:13.759 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22706.asok status 2022-07-21T01:41:13.849 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:41:13.850 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:41:13.850 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:41:13.851 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:41:13.851 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:41:13.851 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:41:13.851 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:41:13.852 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "22706", 2022-07-21T01:41:13.852 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:41:13.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:41:13.852 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:41:13.853 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:41:13.853 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 7485, 2022-07-21T01:41:13.853 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:41:13.853 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:41:13.854 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:41:13.854 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 7485 2022-07-21T01:41:13.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:41:13.855 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:41:13.855 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:41:13.855 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:41:13.855 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 944477177 2022-07-21T01:41:13.856 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:41:13.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:41:13.856 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:41:13.857 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:41:13.857 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:41:13.857 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 944477177 2022-07-21T01:41:13.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:41:13.858 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.7485 172.21.15.174:0/944477177", 2022-07-21T01:41:13.858 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/944477177", 2022-07-21T01:41:13.858 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:41:13.859 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 54, 2022-07-21T01:41:13.859 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 111, 2022-07-21T01:41:13.859 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:41:13.859 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:41:13.860 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:41:13.860 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:41:13.861 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:13.862 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:13.862 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:13.871 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:41:13.872 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:41:13.873 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:13.873 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:13.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-21T01:41:14.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.044+0000 7fcb7f59e700 1 Processor -- start 2022-07-21T01:41:14.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.044+0000 7fcb7f59e700 1 -- start start 2022-07-21T01:41:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.044+0000 7fcb7f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcb800af450 0x7fcb800af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.044+0000 7fcb7f59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb800ad9e0 0x7fcb800a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb800aff30 con 0x7fcb800ad4b0 2022-07-21T01:41:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8005c7a0 con 0x7fcb800ad9e0 2022-07-21T01:41:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb80140990 con 0x7fcb800af450 2022-07-21T01:41:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7ed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcb800ad9e0 0x7fcb800a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7dd9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcb800ad4b0 0x7fcb801432e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57582/0 (socket says 172.21.15.3:57582) 2022-07-21T01:41:14.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7dd9b700 1 -- 172.21.15.3:0/3112423043 learned_addr learned my addr 172.21.15.3:0/3112423043 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:14.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7e59c700 1 --2- 172.21.15.3:0/3112423043 >> v2:172.21.15.143:3300/0 conn(0x7fcb800af450 0x7fcb800af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7ed9d700 1 -- 172.21.15.3:0/3112423043 >> v2:172.21.15.143:3300/0 conn(0x7fcb800af450 msgr2=0x7fcb800af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:14.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7ed9d700 1 --2- 172.21.15.3:0/3112423043 >> v2:172.21.15.143:3300/0 conn(0x7fcb800af450 0x7fcb800af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7ed9d700 1 -- 172.21.15.3:0/3112423043 >> v1:172.21.15.3:6789/0 conn(0x7fcb800ad4b0 legacy=0x7fcb801432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7ed9d700 1 -- 172.21.15.3:0/3112423043 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb801439f0 con 0x7fcb800ad9e0 2022-07-21T01:41:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7ed9d700 1 --2- 172.21.15.3:0/3112423043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 0x7fcb800a8e40 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7fcb800ad900 tx=0x7fcb780091d0).ready entity=mon.0 client_cookie=27d7929c5456e142 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7d59a700 1 -- 172.21.15.3:0/3112423043 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7800b680 con 0x7fcb800ad9e0 2022-07-21T01:41:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7d59a700 1 -- 172.21.15.3:0/3112423043 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7800b7e0 con 0x7fcb800ad9e0 2022-07-21T01:41:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7d59a700 1 -- 172.21.15.3:0/3112423043 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7800b950 con 0x7fcb800ad9e0 2022-07-21T01:41:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 -- 172.21.15.3:0/3112423043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 msgr2=0x7fcb800a8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 --2- 172.21.15.3:0/3112423043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 0x7fcb800a8e40 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7fcb800ad900 tx=0x7fcb780091d0).stop 2022-07-21T01:41:14.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 -- 172.21.15.3:0/3112423043 shutdown_connections 2022-07-21T01:41:14.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 --2- 172.21.15.3:0/3112423043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 0x7fcb800a8e40 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 --2- 172.21.15.3:0/3112423043 >> v2:172.21.15.143:3300/0 conn(0x7fcb800af450 0x7fcb800af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 -- 172.21.15.3:0/3112423043 >> 172.21.15.3:0/3112423043 conn(0x7fcb800612d0 msgr2=0x7fcb80062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 -- 172.21.15.3:0/3112423043 shutdown_connections 2022-07-21T01:41:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.048+0000 7fcb7f59e700 1 -- 172.21.15.3:0/3112423043 wait complete. 2022-07-21T01:41:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7f59e700 1 Processor -- start 2022-07-21T01:41:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7f59e700 1 -- start start 2022-07-21T01:41:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb800ad4b0 0x7fcb800aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 0x7fcb800aac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb800af450 0x7fcb80133970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb80067e20 con 0x7fcb800ad9e0 2022-07-21T01:41:14.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb800ac6a0 con 0x7fcb800ad4b0 2022-07-21T01:41:14.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb80067bc0 con 0x7fcb800af450 2022-07-21T01:41:14.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 0x7fcb800aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 0x7fcb800aac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49310/0 (socket says 172.21.15.3:49310) 2022-07-21T01:41:14.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb800ad4b0 0x7fcb800aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7dd9b700 1 -- 172.21.15.3:0/2760759052 learned_addr learned my addr 172.21.15.3:0/2760759052 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:14.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7ed9d700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb800af450 0x7fcb80133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7dd9b700 1 -- 172.21.15.3:0/2760759052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb800af450 msgr2=0x7fcb80133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7dd9b700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb800af450 0x7fcb80133970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7dd9b700 1 -- 172.21.15.3:0/2760759052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb800ad4b0 msgr2=0x7fcb800aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7dd9b700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb800ad4b0 0x7fcb800aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7dd9b700 1 -- 172.21.15.3:0/2760759052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb78009470 con 0x7fcb800ad9e0 2022-07-21T01:41:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7ed9d700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb800af450 0x7fcb80133970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.052+0000 7fcb7dd9b700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 0x7fcb800aac00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fcb80062a00 tx=0x7fcb74007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb6f7fe700 1 -- 172.21.15.3:0/2760759052 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb740070d0 con 0x7fcb800ad9e0 2022-07-21T01:41:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb6f7fe700 1 -- 172.21.15.3:0/2760759052 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb74007230 con 0x7fcb800ad9e0 2022-07-21T01:41:14.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb80136cc0 con 0x7fcb800ad9e0 2022-07-21T01:41:14.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb6f7fe700 1 -- 172.21.15.3:0/2760759052 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb74005610 con 0x7fcb800ad9e0 2022-07-21T01:41:14.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb801378c0 con 0x7fcb800ad9e0 2022-07-21T01:41:14.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb800574e0 con 0x7fcb800ad9e0 2022-07-21T01:41:14.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb6f7fe700 1 -- 172.21.15.3:0/2760759052 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb74010070 con 0x7fcb800ad9e0 2022-07-21T01:41:14.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb6f7fe700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb5c053a70 0x7fcb5c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb6f7fe700 1 -- 172.21.15.3:0/2760759052 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcb7406b5c0 con 0x7fcb800ad9e0 2022-07-21T01:41:14.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb7e59c700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb5c053a70 0x7fcb5c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.056+0000 7fcb7e59c700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb5c053a70 0x7fcb5c055f30 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fcb7000b000 tx=0x7fcb70006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:14.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.064+0000 7fcb6f7fe700 1 -- 172.21.15.3:0/2760759052 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb74042380 con 0x7fcb800ad9e0 2022-07-21T01:41:14.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.232+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fcb80061b20 con 0x7fcb800ad9e0 2022-07-21T01:41:14.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.232+0000 7fcb6f7fe700 1 -- 172.21.15.3:0/2760759052 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1936 (secure 0 0 0) 0x7fcb740459a0 con 0x7fcb800ad9e0 2022-07-21T01:41:14.240 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:14.241 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:41:14.241 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:14.242 INFO:teuthology.orchestra.run.smithi003.stdout: "total_bytes": 773094113280, 2022-07-21T01:41:14.243 INFO:teuthology.orchestra.run.smithi003.stdout: "total_avail_bytes": 764493017088, 2022-07-21T01:41:14.243 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_bytes": 11161600, 2022-07-21T01:41:14.244 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_bytes": 8601096192, 2022-07-21T01:41:14.244 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_ratio": 0.011125548742711544, 2022-07-21T01:41:14.244 INFO:teuthology.orchestra.run.smithi003.stdout: "num_osds": 8, 2022-07-21T01:41:14.244 INFO:teuthology.orchestra.run.smithi003.stdout: "num_per_pool_osds": 8, 2022-07-21T01:41:14.245 INFO:teuthology.orchestra.run.smithi003.stdout: "num_per_pool_omap_osds": 8 2022-07-21T01:41:14.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:14.245 INFO:teuthology.orchestra.run.smithi003.stdout: "stats_by_class": { 2022-07-21T01:41:14.245 INFO:teuthology.orchestra.run.smithi003.stdout: "ssd": { 2022-07-21T01:41:14.246 INFO:teuthology.orchestra.run.smithi003.stdout: "total_bytes": 773094113280, 2022-07-21T01:41:14.246 INFO:teuthology.orchestra.run.smithi003.stdout: "total_avail_bytes": 764493017088, 2022-07-21T01:41:14.247 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_bytes": 11161600, 2022-07-21T01:41:14.247 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_bytes": 8601096192, 2022-07-21T01:41:14.248 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_ratio": 0.011125548742711544 2022-07-21T01:41:14.248 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:14.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:14.249 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:41:14.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:14.250 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_health_metrics", 2022-07-21T01:41:14.250 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 1, 2022-07-21T01:41:14.251 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:14.251 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:14.252 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 2, 2022-07-21T01:41:14.252 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:14.252 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:14.253 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:14.253 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362919100416 2022-07-21T01:41:14.254 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:14.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:14.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:14.256 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd", 2022-07-21T01:41:14.257 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 2, 2022-07-21T01:41:14.257 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:14.258 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:14.258 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 0, 2022-07-21T01:41:14.259 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:14.259 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:14.259 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:14.260 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362919100416 2022-07-21T01:41:14.261 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:14.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:14.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:14.262 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephfs_metadata", 2022-07-21T01:41:14.262 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 19, 2022-07-21T01:41:14.262 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:14.262 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 2304, 2022-07-21T01:41:14.263 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 21, 2022-07-21T01:41:14.263 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 64, 2022-07-21T01:41:14.264 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 65536, 2022-07-21T01:41:14.264 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 9.0290086518507451e-08, 2022-07-21T01:41:14.265 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362919100416 2022-07-21T01:41:14.265 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:14.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:14.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:14.267 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephfs_data", 2022-07-21T01:41:14.267 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 20, 2022-07-21T01:41:14.267 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:14.268 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:14.268 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 0, 2022-07-21T01:41:14.269 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:14.269 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:14.269 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:14.270 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362919100416 2022-07-21T01:41:14.270 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:14.271 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:14.271 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:14.272 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:41:14.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb5c053a70 msgr2=0x7fcb5c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb5c053a70 0x7fcb5c055f30 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fcb7000b000 tx=0x7fcb70006df0).stop 2022-07-21T01:41:14.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 msgr2=0x7fcb800aac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 0x7fcb800aac00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fcb80062a00 tx=0x7fcb74007c00).stop 2022-07-21T01:41:14.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 shutdown_connections 2022-07-21T01:41:14.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb800af450 0x7fcb80133970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb800ad9e0 0x7fcb800aac00 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcb5c053a70 0x7fcb5c055f30 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 --2- 172.21.15.3:0/2760759052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb800ad4b0 0x7fcb800aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 >> 172.21.15.3:0/2760759052 conn(0x7fcb800612d0 msgr2=0x7fcb8013d180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:14.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 shutdown_connections 2022-07-21T01:41:14.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.236+0000 7fcb7f59e700 1 -- 172.21.15.3:0/2760759052 wait complete. 2022-07-21T01:41:14.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data size -f json-pretty 2022-07-21T01:41:14.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.396+0000 7f2171f06700 1 Processor -- start 2022-07-21T01:41:14.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.396+0000 7f2171f06700 1 -- start start 2022-07-21T01:41:14.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f216c146ae0 0x7f216c1403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 0x7f216c0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216c143cf0 con 0x7f216c0b4310 2022-07-21T01:41:14.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216c1476a0 con 0x7f216c146ae0 2022-07-21T01:41:14.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c143f90 con 0x7f216c0b4820 2022-07-21T01:41:14.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f216c146ae0 0x7f216c1403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f216c146ae0 0x7f216c1403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49316/0 (socket says 172.21.15.3:49316) 2022-07-21T01:41:14.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216b7fe700 1 -- 172.21.15.3:0/1816676218 learned_addr learned my addr 172.21.15.3:0/1816676218 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:14.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216bfff700 1 --2- 172.21.15.3:0/1816676218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 0x7f216c0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216a7fc700 1 -- 172.21.15.3:0/1816676218 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476631482 0 0) 0x7f216c143cf0 con 0x7f216c0b4310 2022-07-21T01:41:14.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216a7fc700 1 -- 172.21.15.3:0/1816676218 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2158003260 con 0x7f216c0b4310 2022-07-21T01:41:14.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216b7fe700 1 -- 172.21.15.3:0/1816676218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 msgr2=0x7f216c0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:14.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216b7fe700 1 --2- 172.21.15.3:0/1816676218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 0x7f216c0b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216b7fe700 1 -- 172.21.15.3:0/1816676218 >> v1:172.21.15.143:6789/0 conn(0x7f216c0b4310 legacy=0x7f216c1408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216b7fe700 1 -- 172.21.15.3:0/1816676218 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c0b0e20 con 0x7f216c146ae0 2022-07-21T01:41:14.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216b7fe700 1 --2- 172.21.15.3:0/1816676218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c146ae0 0x7f216c1403c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7f2160002f70 tx=0x7f2160006eb0).ready entity=mon.0 client_cookie=1c2d783f69ad9f69 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:14.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216a7fc700 1 -- 172.21.15.3:0/1816676218 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f216000d5c0 con 0x7f216c146ae0 2022-07-21T01:41:14.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216a7fc700 1 -- 172.21.15.3:0/1816676218 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2160000e90 con 0x7f216c146ae0 2022-07-21T01:41:14.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f216a7fc700 1 -- 172.21.15.3:0/1816676218 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f216000d950 con 0x7f216c146ae0 2022-07-21T01:41:14.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 -- 172.21.15.3:0/1816676218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c146ae0 msgr2=0x7f216c1403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 --2- 172.21.15.3:0/1816676218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c146ae0 0x7f216c1403c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7f2160002f70 tx=0x7f2160006eb0).stop 2022-07-21T01:41:14.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 -- 172.21.15.3:0/1816676218 shutdown_connections 2022-07-21T01:41:14.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 --2- 172.21.15.3:0/1816676218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 0x7f216c0b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 --2- 172.21.15.3:0/1816676218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c146ae0 0x7f216c1403c0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.400+0000 7f2171f06700 1 -- 172.21.15.3:0/1816676218 >> 172.21.15.3:0/1816676218 conn(0x7f216c05c620 msgr2=0x7f216c05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:14.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 -- 172.21.15.3:0/1816676218 shutdown_connections 2022-07-21T01:41:14.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 -- 172.21.15.3:0/1816676218 wait complete. 2022-07-21T01:41:14.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 Processor -- start 2022-07-21T01:41:14.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 -- start start 2022-07-21T01:41:14.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c0b4310 0x7f216c0b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 0x7f216c144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c146ae0 0x7f216c144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0b08f0 con 0x7f216c0b4310 2022-07-21T01:41:14.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c145e80 con 0x7f216c146ae0 2022-07-21T01:41:14.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f2171f06700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c146120 con 0x7f216c0b4820 2022-07-21T01:41:14.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f216b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c0b4310 0x7f216c0b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f216b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c0b4310 0x7f216c0b53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49322/0 (socket says 172.21.15.3:49322) 2022-07-21T01:41:14.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f216b7fe700 1 -- 172.21.15.3:0/1582066595 learned_addr learned my addr 172.21.15.3:0/1582066595 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:14.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f216affd700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 0x7f216c144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.404+0000 7f216bfff700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c146ae0 0x7f216c144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f216b7fe700 1 -- 172.21.15.3:0/1582066595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 msgr2=0x7f216c144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f216b7fe700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 0x7f216c144a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f216b7fe700 1 -- 172.21.15.3:0/1582066595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c146ae0 msgr2=0x7f216c144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f216b7fe700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c146ae0 0x7f216c144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f216b7fe700 1 -- 172.21.15.3:0/1582066595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216000b1e0 con 0x7f216c0b4310 2022-07-21T01:41:14.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f216b7fe700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c0b4310 0x7f216c0b53a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f2160002f70 tx=0x7f2160015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:14.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f2168ff9700 1 -- 172.21.15.3:0/1582066595 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21600116e0 con 0x7f216c0b4310 2022-07-21T01:41:14.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f2168ff9700 1 -- 172.21.15.3:0/1582066595 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2160011840 con 0x7f216c0b4310 2022-07-21T01:41:14.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c1463c0 con 0x7f216c0b4310 2022-07-21T01:41:14.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f2168ff9700 1 -- 172.21.15.3:0/1582066595 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2160011ba0 con 0x7f216c0b4310 2022-07-21T01:41:14.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c0b6cd0 con 0x7f216c0b4310 2022-07-21T01:41:14.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c0ad920 con 0x7f216c0b4310 2022-07-21T01:41:14.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.408+0000 7f2168ff9700 1 -- 172.21.15.3:0/1582066595 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2160025030 con 0x7f216c0b4310 2022-07-21T01:41:14.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.412+0000 7f2168ff9700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f214c053a70 0x7f214c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:14.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.412+0000 7f2168ff9700 1 -- 172.21.15.3:0/1582066595 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f216006d040 con 0x7f216c0b4310 2022-07-21T01:41:14.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.412+0000 7f216affd700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f214c053a70 0x7f214c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:14.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.412+0000 7f216affd700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f214c053a70 0x7f214c055f30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f215c00d000 tx=0x7f215c007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:14.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.416+0000 7f2168ff9700 1 -- 172.21.15.3:0/1582066595 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2160043e00 con 0x7f216c0b4310 2022-07-21T01:41:14.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.540+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"} v 0) v1 -- 0x7f216c0574e0 con 0x7f216c0b4310 2022-07-21T01:41:14.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.540+0000 7f2168ff9700 1 -- 172.21.15.3:0/1582066595 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"}]=0 v111) v1 ==== 123+0+64 (secure 0 0 0) 0x7f2160023020 con 0x7f216c0b4310 2022-07-21T01:41:14.544 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:14.544 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:41:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": "cephfs_data", 2022-07-21T01:41:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_id": 20, 2022-07-21T01:41:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2 2022-07-21T01:41:14.545 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:41:14.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f214c053a70 msgr2=0x7f214c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f214c053a70 0x7f214c055f30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f215c00d000 tx=0x7f215c007a20).stop 2022-07-21T01:41:14.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c0b4310 msgr2=0x7f216c0b53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:14.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c0b4310 0x7f216c0b53a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f2160002f70 tx=0x7f2160015000).stop 2022-07-21T01:41:14.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 shutdown_connections 2022-07-21T01:41:14.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f214c053a70 0x7f214c055f30 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c146ae0 0x7f216c144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0b4820 0x7f216c144a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 --2- 172.21.15.3:0/1582066595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f216c0b4310 0x7f216c0b53a0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:14.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 >> 172.21.15.3:0/1582066595 conn(0x7f216c05c620 msgr2=0x7f216c147860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:14.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 shutdown_connections 2022-07-21T01:41:14.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:14.544+0000 7f2171f06700 1 -- 172.21.15.3:0/1582066595 wait complete. 2022-07-21T01:41:14.557 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-21T01:41:14.558 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest/mnt.0 && sudo df . 2022-07-21T01:41:14.588 INFO:teuthology.orchestra.run.smithi171.stdout:Filesystem 1K-blocks Used Available Use% Mounted on 2022-07-21T01:41:14.589 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-fuse 354410496 0 354410496 0% /home/ubuntu/cephtest/mnt.0 2022-07-21T01:41:14.591 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:41:14.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:14.611 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:14.634 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:14.657 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:14.679 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:41:14.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:14.680 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:14.681 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:14.689 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:41:14.690 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:41:14.691 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:41:14.691 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:41:14.691 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:14.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23128]: fuse finished with error 0 and tester_r 0 2022-07-21T01:41:14.709 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:14.710 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:14.710 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:14.763 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:41:14.764 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:41:14.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:41:14.884 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:14.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:14.894 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:14.895 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:14.903 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:41:14.904 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:41:14.905 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:41:14.905 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:41:14.906 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:14.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22706]: fuse finished with error 0 and tester_r 0 2022-07-21T01:41:14.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:14.967 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:14.968 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:14.975 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:41:14.977 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:41:15.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:41:15.051 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:15.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:41:15.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd656681700 1 Processor -- start 2022-07-21T01:41:15.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd656681700 1 -- start start 2022-07-21T01:41:15.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd656681700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500b1f40 0x7fd6500b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd656681700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd6500a7250 0x7fd6501448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd656681700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6500b04e0 con 0x7fd650148530 2022-07-21T01:41:15.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd656681700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6500a9250 con 0x7fd6500a7250 2022-07-21T01:41:15.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd656681700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6500b0780 con 0x7fd6500b1f40 2022-07-21T01:41:15.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd654c1e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd6500a7250 0x7fd6501448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd654c1e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd6500a7250 0x7fd6501448b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49334/0 (socket says 172.21.15.3:49334) 2022-07-21T01:41:15.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd654c1e700 1 -- 172.21.15.3:0/1560815057 learned_addr learned my addr 172.21.15.3:0/1560815057 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:15.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd64ffff700 1 --2- 172.21.15.3:0/1560815057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500b1f40 0x7fd6500b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd654c1e700 1 -- 172.21.15.3:0/1560815057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500b1f40 msgr2=0x7fd6500b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:15.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd654c1e700 1 --2- 172.21.15.3:0/1560815057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500b1f40 0x7fd6500b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd654c1e700 1 -- 172.21.15.3:0/1560815057 >> v1:172.21.15.143:6789/0 conn(0x7fd650148530 legacy=0x7fd6500b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd654c1e700 1 -- 172.21.15.3:0/1560815057 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6500b0a20 con 0x7fd6500a7250 2022-07-21T01:41:15.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd654c1e700 1 --2- 172.21.15.3:0/1560815057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500a7250 0x7fd6501448b0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7fd638001110 tx=0x7fd648009290).ready entity=mon.0 client_cookie=3bb8d8c95a5e874e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:15.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd64f7fe700 1 -- 172.21.15.3:0/1560815057 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd64800b760 con 0x7fd6500a7250 2022-07-21T01:41:15.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd64f7fe700 1 -- 172.21.15.3:0/1560815057 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64800b8c0 con 0x7fd6500a7250 2022-07-21T01:41:15.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.164+0000 7fd64f7fe700 1 -- 172.21.15.3:0/1560815057 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd64800ba30 con 0x7fd6500a7250 2022-07-21T01:41:15.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 -- 172.21.15.3:0/1560815057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500a7250 msgr2=0x7fd6501448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 --2- 172.21.15.3:0/1560815057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500a7250 0x7fd6501448b0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7fd638001110 tx=0x7fd648009290).stop 2022-07-21T01:41:15.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 -- 172.21.15.3:0/1560815057 shutdown_connections 2022-07-21T01:41:15.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 --2- 172.21.15.3:0/1560815057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500a7250 0x7fd6501448b0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 --2- 172.21.15.3:0/1560815057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500b1f40 0x7fd6500b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 -- 172.21.15.3:0/1560815057 >> 172.21.15.3:0/1560815057 conn(0x7fd65005f840 msgr2=0x7fd65005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:15.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 -- 172.21.15.3:0/1560815057 shutdown_connections 2022-07-21T01:41:15.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 -- 172.21.15.3:0/1560815057 wait complete. 2022-07-21T01:41:15.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 Processor -- start 2022-07-21T01:41:15.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.168+0000 7fd656681700 1 -- start start 2022-07-21T01:41:15.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd656681700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500a7250 0x7fd65014ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd656681700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500b1f40 0x7fd6500ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd656681700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd650148530 0x7fd6500b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd656681700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6500b6550 con 0x7fd6500b1f40 2022-07-21T01:41:15.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd656681700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65014b370 con 0x7fd650148530 2022-07-21T01:41:15.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd656681700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65014ae30 con 0x7fd6500a7250 2022-07-21T01:41:15.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd6477fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500b1f40 0x7fd6500ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd6477fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500b1f40 0x7fd6500ae780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49338/0 (socket says 172.21.15.3:49338) 2022-07-21T01:41:15.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd6477fe700 1 -- 172.21.15.3:0/3741146077 learned_addr learned my addr 172.21.15.3:0/3741146077 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:15.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd654c1e700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd650148530 0x7fd6500b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd64ffff700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500a7250 0x7fd65014ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd6477fe700 1 -- 172.21.15.3:0/3741146077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500a7250 msgr2=0x7fd65014ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd6477fe700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500a7250 0x7fd65014ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd6477fe700 1 -- 172.21.15.3:0/3741146077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd650148530 msgr2=0x7fd6500b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd6477fe700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd650148530 0x7fd6500b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd6477fe700 1 -- 172.21.15.3:0/3741146077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd648009530 con 0x7fd6500b1f40 2022-07-21T01:41:15.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd6477fe700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500b1f40 0x7fd6500ae780 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7fd638000bc0 tx=0x7fd638007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:15.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd64dffb700 1 -- 172.21.15.3:0/3741146077 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd63800ce40 con 0x7fd6500b1f40 2022-07-21T01:41:15.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd64dffb700 1 -- 172.21.15.3:0/3741146077 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd638007020 con 0x7fd6500b1f40 2022-07-21T01:41:15.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65014b0d0 con 0x7fd6500b1f40 2022-07-21T01:41:15.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd64dffb700 1 -- 172.21.15.3:0/3741146077 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd638007380 con 0x7fd6500b1f40 2022-07-21T01:41:15.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65014a950 con 0x7fd6500b1f40 2022-07-21T01:41:15.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.172+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6500574e0 con 0x7fd6500b1f40 2022-07-21T01:41:15.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.176+0000 7fd64dffb700 1 -- 172.21.15.3:0/3741146077 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd638010070 con 0x7fd6500b1f40 2022-07-21T01:41:15.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.176+0000 7fd64dffb700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd634053b00 0x7fd634055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.176+0000 7fd64dffb700 1 -- 172.21.15.3:0/3741146077 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd63806b4a0 con 0x7fd6500b1f40 2022-07-21T01:41:15.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.176+0000 7fd64ffff700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd634053b00 0x7fd634055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.176+0000 7fd64ffff700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd634053b00 0x7fd634055fc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7fd64000b000 tx=0x7fd640006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:15.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.180+0000 7fd64dffb700 1 -- 172.21.15.3:0/3741146077 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd638042150 con 0x7fd6500b1f40 2022-07-21T01:41:15.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.308+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd6500447e0 con 0x7fd6500b1f40 2022-07-21T01:41:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd64dffb700 1 -- 172.21.15.3:0/3741146077 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v111) v1 ==== 81+0+34051 (secure 0 0 0) 0x7fd638028520 con 0x7fd6500b1f40 2022-07-21T01:41:15.314 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:15.314 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:41:15.315 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 111, 2022-07-21T01:41:15.315 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:41:15.315 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:41:15.315 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:41:05.595017+0000", 2022-07-21T01:41:15.316 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:41:15.316 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:41:15.316 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:41:15.316 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:41:15.317 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:41:15.317 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:41:15.317 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:41:15.317 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:41:15.317 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:41:15.318 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:15.318 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:41:15.318 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:41:15.318 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:41:15.319 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:41:15.319 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:41:15.319 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 20, 2022-07-21T01:41:15.319 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:41:15.319 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:41:15.320 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:41:15.320 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:41:15.320 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:41:15.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.321 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:41:15.321 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:41:15.321 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:41:15.321 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:41:15.321 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:41:15.322 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:15.322 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:15.322 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:15.322 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:15.323 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:15.323 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:15.323 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:41:15.323 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:41:15.324 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:41:15.324 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:41:15.324 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:41:15.325 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:15.325 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:15.325 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:15.325 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:15.326 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:15.326 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:15.326 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:15.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.327 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:41:15.327 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:15.327 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:15.327 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:15.328 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:15.328 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:15.328 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:15.328 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:15.328 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:15.329 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:15.329 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:15.329 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:15.329 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:15.330 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:15.330 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:15.330 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:15.330 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:15.330 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:15.331 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:15.331 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:15.331 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:15.331 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:15.332 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:15.332 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:15.332 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:15.332 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:15.332 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:15.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.333 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:15.333 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:15.333 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:15.334 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:15.334 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:15.334 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:15.334 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:15.335 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:15.335 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:15.336 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:15.336 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:15.336 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:41:15.336 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:41:15.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.337 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:15.337 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:41:15.337 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.338 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:41:15.338 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:41:15.338 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:41:15.339 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:41:15.339 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:41:15.339 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:15.339 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:15.340 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:15.340 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:15.340 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:15.340 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:15.341 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:41:15.341 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:41:15.341 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:41:15.341 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:41:15.341 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:41:15.342 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:15.342 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:15.342 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:15.342 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:15.343 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:15.343 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:15.343 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:15.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.344 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:41:15.344 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:15.344 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:15.344 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:15.344 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:15.345 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:15.345 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:15.346 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:15.346 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:15.346 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:15.346 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:15.347 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:15.347 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:15.347 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:15.347 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:15.348 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:15.348 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:15.348 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:15.348 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:15.348 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:15.349 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:15.349 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:15.349 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:15.349 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:15.350 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:15.350 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:15.350 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:15.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.350 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:15.351 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:15.351 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:15.351 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:15.351 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:15.352 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:15.352 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:15.352 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:15.352 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:15.352 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:15.353 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:15.353 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:41:15.353 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:15.353 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:41:15.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.354 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 19, 2022-07-21T01:41:15.354 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:41:15.355 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:41:03.035970+0000", 2022-07-21T01:41:15.355 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:41:15.355 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:41:15.355 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:15.357 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:15.357 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:15.357 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:15.357 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:15.357 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:15.358 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:41:15.358 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:41:15.358 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:41:15.358 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:41:15.359 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:41:15.359 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:15.359 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:15.359 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:15.360 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:15.360 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:15.360 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:15.360 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:15.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.361 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "110", 2022-07-21T01:41:15.361 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:15.361 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:15.361 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:15.362 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:15.362 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:15.362 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:15.362 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:15.363 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:15.363 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:15.363 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:15.363 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:15.363 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:15.364 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:15.364 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:15.364 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:15.364 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:15.365 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:15.365 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:15.365 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:15.365 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:15.365 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:15.366 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:15.366 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:15.366 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:15.366 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:15.367 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:15.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.368 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:15.368 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:15.368 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:15.368 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:15.368 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:15.369 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:15.369 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:15.369 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:15.369 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:15.370 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:15.370 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:15.370 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:41:15.370 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:41:15.370 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:41:15.371 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:41:15.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.371 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:15.371 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:41:15.372 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:41:15.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.373 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 20, 2022-07-21T01:41:15.373 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:41:15.373 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:41:03.980079+0000", 2022-07-21T01:41:15.373 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:41:15.374 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:41:15.374 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:15.374 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:15.374 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:15.375 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:15.375 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:15.375 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:15.375 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:41:15.375 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:41:15.376 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:41:15.376 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:41:15.376 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:41:15.376 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:15.377 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:15.377 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:15.377 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:15.378 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:15.378 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:15.378 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:15.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "111", 2022-07-21T01:41:15.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:15.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:15.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:15.380 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:15.380 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:15.380 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:15.380 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:15.380 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:15.381 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:15.381 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:15.381 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:15.381 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:15.382 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:15.382 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:15.382 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:15.382 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:15.383 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:15.383 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:15.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:15.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:15.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:15.384 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:15.384 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:15.384 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:15.384 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:15.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:15.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.385 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:15.385 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:15.385 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:15.386 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:15.386 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:15.386 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:15.386 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:15.387 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:15.387 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:15.387 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:15.387 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:15.388 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:41:15.388 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:15.388 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:41:15.389 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:41:15.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.390 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:15.390 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:41:15.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.390 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:41:15.390 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:41:15.391 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:15.391 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:15.391 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:15.391 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:15.392 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:15.392 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:15.392 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:41:15.392 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:15.392 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:15.393 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:15.393 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:15.393 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.394 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.394 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:41:15.394 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:15.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.395 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.395 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:41:15.395 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:15.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.396 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.396 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:15.396 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.397 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:41:15.397 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:15.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.399 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.399 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:41:15.399 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:15.399 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.399 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.400 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:15.400 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.401 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.401 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:41:15.401 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:15.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.402 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:41:15.402 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:15.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.403 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.403 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:15.403 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.404 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.404 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:41:15.404 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:15.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.405 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.405 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:41:15.405 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:15.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.406 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.406 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:41:15.406 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:41:15.407 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:41:15.407 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:41:15.407 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:15.407 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:15.408 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:15.408 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.409 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:41:15.409 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:41:15.410 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:15.410 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:15.410 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:15.410 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:15.411 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:15.411 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:15.411 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:15.411 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:15.412 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:15.412 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:15.412 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:15.412 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.413 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.413 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:41:15.413 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:15.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.414 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:41:15.414 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:15.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.415 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.415 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:15.416 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.416 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:41:15.416 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:15.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.417 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.417 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:41:15.417 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:15.418 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.418 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.418 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:15.419 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.420 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.420 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:41:15.420 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:15.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.421 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:41:15.421 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:15.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.422 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.422 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:15.423 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.423 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.423 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:41:15.424 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:15.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.424 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:41:15.425 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:15.425 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.425 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.426 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:41:15.426 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:41:15.426 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:41:15.426 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:41:15.427 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:15.427 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:15.427 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:15.427 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.428 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:41:15.428 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:41:15.428 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:15.429 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:15.429 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:15.429 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:15.430 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:15.430 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:15.430 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:41:15.431 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 107, 2022-07-21T01:41:15.431 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:15.431 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:15.431 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:15.432 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.432 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.432 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:41:15.432 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:15.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.433 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:41:15.434 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:15.434 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.434 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.435 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:15.435 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.435 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:41:15.436 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:15.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.437 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:41:15.437 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:15.437 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.437 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.438 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:15.438 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.438 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.439 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:41:15.439 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:15.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.440 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:41:15.440 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:15.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.440 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.441 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:15.441 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.442 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:41:15.442 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:15.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.443 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.443 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:41:15.443 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:15.443 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.444 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.444 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:41:15.444 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:41:15.445 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:41:15.445 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:41:15.445 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:15.445 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:15.445 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:15.446 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:41:15.447 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:41:15.447 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:15.447 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:15.447 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:15.448 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:15.448 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:15.448 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:15.448 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:41:15.448 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:15.449 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:15.449 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:15.449 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:15.449 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.450 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:41:15.450 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:15.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.451 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.451 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:41:15.451 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:15.452 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.452 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.452 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:15.452 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.453 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.453 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:41:15.453 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:15.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.454 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:41:15.454 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:15.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.455 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.455 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:15.456 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.456 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:41:15.456 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:15.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.457 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:41:15.458 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:15.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.458 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:15.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:41:15.460 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:15.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:41:15.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:15.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.461 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.462 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:41:15.462 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:41:15.462 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:41:15.462 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:41:15.463 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:15.463 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:15.463 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:15.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:41:15.464 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:41:15.464 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:15.465 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:15.465 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:15.465 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:15.465 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:15.465 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:15.466 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:15.466 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 107, 2022-07-21T01:41:15.466 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:15.466 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:15.467 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:15.467 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.467 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.467 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:41:15.468 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:15.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.468 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.469 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:41:15.469 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:15.469 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.469 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.470 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:15.470 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.470 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.471 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:41:15.471 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:15.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:41:15.472 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:15.472 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.472 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.473 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:15.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.474 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:41:15.474 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:15.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.475 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:41:15.475 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:15.475 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.476 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.476 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:15.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.477 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:41:15.477 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:15.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:41:15.478 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:15.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.479 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.479 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:41:15.479 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:41:15.480 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:41:15.480 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:41:15.480 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:15.480 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:15.480 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:41:15.481 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:41:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:15.482 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 107, 2022-07-21T01:41:15.483 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:41:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:41:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:15.486 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:41:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:15.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:41:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:15.489 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:15.490 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:41:15.491 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:41:15.492 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:15.493 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.493 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.493 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:15.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:41:15.494 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:15.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.495 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.495 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:41:15.496 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:15.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.496 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.497 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:41:15.497 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:41:15.497 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:41:15.497 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:41:15.497 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:15.498 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:15.498 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:15.498 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.499 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:41:15.499 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:41:15.500 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:15.500 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:15.500 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:15.500 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:15.501 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:15.501 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:15.501 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:15.501 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:15.501 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:15.502 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:15.502 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:15.502 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:41:15.503 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:15.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.504 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:41:15.504 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:15.504 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.505 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.505 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:15.505 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.506 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:41:15.507 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:15.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:41:15.508 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:15.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.508 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.509 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:15.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.509 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:41:15.510 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:15.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.510 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:41:15.511 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:15.511 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.511 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.512 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:15.512 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.512 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.513 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:41:15.513 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:15.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.513 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.514 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:41:15.514 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:15.514 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.514 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.515 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:41:15.515 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:41:15.515 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:41:15.515 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:41:15.516 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:15.516 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:15.516 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:15.516 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.517 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:41:15.517 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:41:15.517 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:15.518 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:15.518 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:15.518 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:15.518 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:15.519 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:15.519 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:15.519 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:15.519 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:15.519 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:15.520 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:15.520 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.520 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.521 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:41:15.521 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:15.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.521 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.522 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:41:15.522 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:15.522 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.522 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.523 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:15.523 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.523 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.524 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:41:15.524 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:15.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.525 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.525 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:41:15.525 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:15.525 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.525 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.526 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:15.526 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.527 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:41:15.527 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:15.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.528 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:41:15.528 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:15.528 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.528 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.529 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:15.529 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:15.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.530 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:15.530 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:41:15.530 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:15.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.531 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:15.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:41:15.531 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:15.531 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.532 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.532 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:41:15.532 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:41:15.532 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:41:15.533 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:41:15.533 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:15.533 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:15.533 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:15.534 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:15.534 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.534 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:15.534 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:41:15.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.535 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:41:15.535 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:15.535 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:15.535 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:15.536 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:15.536 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:15.536 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:41:15.536 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:15.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.537 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:41:15.537 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:15.537 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:15.538 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:15.538 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:15.538 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:15.538 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:41:15.538 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:15.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.539 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:41:15.539 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:15.540 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:15.540 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:15.540 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:15.540 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:15.540 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:41:15.541 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:15.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.541 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:41:15.542 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:15.542 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:15.542 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:15.542 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:15.542 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:15.543 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:41:15.543 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:15.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.544 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:41:15.544 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:15.544 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:15.544 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:15.544 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:15.545 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:15.545 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:41:15.545 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:15.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.546 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:41:15.546 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:15.546 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:15.546 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:15.547 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:15.547 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:15.547 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:41:15.547 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:15.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.548 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:41:15.548 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:15.548 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:15.549 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:15.549 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:15.549 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:15.549 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:41:15.549 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:15.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:15.550 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:41:15.550 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:15.551 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:15.551 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:15.551 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:15.551 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:15.551 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:41:15.552 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:15.552 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.552 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:15.552 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:41:15.553 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:41:15.553 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:41:15.553 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:41:15.553 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:41:15.553 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:41:15.554 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:41:15.554 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:41:15.554 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:41:15.554 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:41:15.555 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:41:15.555 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:41:15.555 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:41:15.555 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:41:15.555 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:41:15.556 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:41:15.556 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:41:15.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.556 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:41:15.556 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:41:15.557 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:41:15.557 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:41:15.557 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:41:15.557 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:41:15.558 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:41:15.558 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:15.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:15.558 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:41:15.558 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:41:15.559 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:41:15.559 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:41:15.559 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:41:15.560 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:41:15.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd634053b00 msgr2=0x7fd634055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd634053b00 0x7fd634055fc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7fd64000b000 tx=0x7fd640006df0).stop 2022-07-21T01:41:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500b1f40 msgr2=0x7fd6500ae780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500b1f40 0x7fd6500ae780 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7fd638000bc0 tx=0x7fd638007bc0).stop 2022-07-21T01:41:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 shutdown_connections 2022-07-21T01:41:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd634053b00 0x7fd634055fc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd650148530 0x7fd6500b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6500b1f40 0x7fd6500ae780 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 --2- 172.21.15.3:0/3741146077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6500a7250 0x7fd65014ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 >> 172.21.15.3:0/3741146077 conn(0x7fd65005f840 msgr2=0x7fd650144470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 shutdown_connections 2022-07-21T01:41:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.312+0000 7fd656681700 1 -- 172.21.15.3:0/3741146077 wait complete. 2022-07-21T01:41:15.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:15.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59af47d700 1 Processor -- start 2022-07-21T01:41:15.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59af47d700 1 -- start start 2022-07-21T01:41:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59af47d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f59a80b0630 0x7f59a80adec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59af47d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 0x7f59a80ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59af47d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59a80b0a10 0x7f59a81438a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59af47d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a80b2c80 con 0x7f59a80b0630 2022-07-21T01:41:15.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59af47d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a80b2900 con 0x7f59a80b0a10 2022-07-21T01:41:15.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59af47d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80b2eb0 con 0x7f59a80aa2b0 2022-07-21T01:41:15.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59ad219700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f59a80b0630 0x7f59a80adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59ad219700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f59a80b0630 0x7f59a80adec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49346/0 (socket says 172.21.15.3:49346) 2022-07-21T01:41:15.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59ada1a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59a80b0a10 0x7f59a81438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59aca18700 1 --2- 172.21.15.3:0/729772907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 0x7f59a80ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59ad219700 1 -- 172.21.15.3:0/729772907 learned_addr learned my addr 172.21.15.3:0/729772907 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:15.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59aca18700 1 -- 172.21.15.3:0/729772907 >> v2:172.21.15.143:3300/0 conn(0x7f59a80b0a10 msgr2=0x7f59a81438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:15.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59aca18700 1 --2- 172.21.15.3:0/729772907 >> v2:172.21.15.143:3300/0 conn(0x7f59a80b0a10 0x7f59a81438a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59aca18700 1 -- 172.21.15.3:0/729772907 >> v2:172.21.15.3:3300/0 conn(0x7f59a80b0630 msgr2=0x7f59a80adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:15.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59aca18700 1 --2- 172.21.15.3:0/729772907 >> v2:172.21.15.3:3300/0 conn(0x7f59a80b0630 0x7f59a80adec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59aca18700 1 -- 172.21.15.3:0/729772907 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a80a9d00 con 0x7f59a80aa2b0 2022-07-21T01:41:15.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59ad219700 1 --2- 172.21.15.3:0/729772907 >> v2:172.21.15.3:3300/0 conn(0x7f59a80b0630 0x7f59a80adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:15.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f59aca18700 1 --2- 172.21.15.3:0/729772907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 0x7f59a80ae3b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f599c002f70 tx=0x7f599c006df0).ready entity=mon.2 client_cookie=8244423c95a26c27 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:15.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f599bfff700 1 -- 172.21.15.3:0/729772907 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f599c00d4a0 con 0x7f59a80aa2b0 2022-07-21T01:41:15.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f599bfff700 1 -- 172.21.15.3:0/729772907 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c000e90 con 0x7f59a80aa2b0 2022-07-21T01:41:15.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.676+0000 7f599bfff700 1 -- 172.21.15.3:0/729772907 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f599c00d7f0 con 0x7f59a80aa2b0 2022-07-21T01:41:15.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 -- 172.21.15.3:0/729772907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 msgr2=0x7f59a80ae3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 --2- 172.21.15.3:0/729772907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 0x7f59a80ae3b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f599c002f70 tx=0x7f599c006df0).stop 2022-07-21T01:41:15.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 -- 172.21.15.3:0/729772907 shutdown_connections 2022-07-21T01:41:15.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 --2- 172.21.15.3:0/729772907 >> v2:172.21.15.143:3300/0 conn(0x7f59a80b0a10 0x7f59a81438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 --2- 172.21.15.3:0/729772907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 0x7f59a80ae3b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 --2- 172.21.15.3:0/729772907 >> v2:172.21.15.3:3300/0 conn(0x7f59a80b0630 0x7f59a80adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 -- 172.21.15.3:0/729772907 >> 172.21.15.3:0/729772907 conn(0x7f59a8060950 msgr2=0x7f59a8060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:15.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 -- 172.21.15.3:0/729772907 shutdown_connections 2022-07-21T01:41:15.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 -- 172.21.15.3:0/729772907 wait complete. 2022-07-21T01:41:15.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 Processor -- start 2022-07-21T01:41:15.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 -- start start 2022-07-21T01:41:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 0x7f59a80ac150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a80b0630 0x7f59a8143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59a80b0a10 0x7f59a813c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8067200 con 0x7f59a80b0a10 2022-07-21T01:41:15.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80aae90 con 0x7f59a80b0630 2022-07-21T01:41:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59af47d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8141c50 con 0x7f59a80aa2b0 2022-07-21T01:41:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59ada1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59a80b0a10 0x7f59a813c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59aca18700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a80b0630 0x7f59a8143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59ada1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59a80b0a10 0x7f59a813c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49354/0 (socket says 172.21.15.3:49354) 2022-07-21T01:41:15.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59ada1a700 1 -- 172.21.15.3:0/1551542170 learned_addr learned my addr 172.21.15.3:0/1551542170 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:15.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.680+0000 7f59ad219700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 0x7f59a80ac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59ada1a700 1 -- 172.21.15.3:0/1551542170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 msgr2=0x7f59a80ac150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59ada1a700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 0x7f59a80ac150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59ada1a700 1 -- 172.21.15.3:0/1551542170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a80b0630 msgr2=0x7f59a8143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59ada1a700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a80b0630 0x7f59a8143fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59ada1a700 1 -- 172.21.15.3:0/1551542170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a400b1e0 con 0x7f59a80b0a10 2022-07-21T01:41:15.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59ada1a700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59a80b0a10 0x7f59a813c6f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7f59a0006e00 tx=0x7f59a00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:15.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f599a7fc700 1 -- 172.21.15.3:0/1551542170 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59a000c670 con 0x7f59a80b0a10 2022-07-21T01:41:15.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f599a7fc700 1 -- 172.21.15.3:0/1551542170 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a000c7d0 con 0x7f59a80b0a10 2022-07-21T01:41:15.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f599a7fc700 1 -- 172.21.15.3:0/1551542170 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59a000cb30 con 0x7f59a80b0a10 2022-07-21T01:41:15.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c00b180 con 0x7f59a80b0a10 2022-07-21T01:41:15.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a80ab8b0 con 0x7f59a80b0a10 2022-07-21T01:41:15.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a80574e0 con 0x7f59a80b0a10 2022-07-21T01:41:15.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f599a7fc700 1 -- 172.21.15.3:0/1551542170 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f59a000cc90 con 0x7f59a80b0a10 2022-07-21T01:41:15.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f599a7fc700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5990053a70 0x7f5990055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:15.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f599a7fc700 1 -- 172.21.15.3:0/1551542170 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f59a006c310 con 0x7f59a80b0a10 2022-07-21T01:41:15.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.684+0000 7f59ad219700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5990053a70 0x7f5990055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:15.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.688+0000 7f59ad219700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5990053a70 0x7f5990055f30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f59a400b000 tx=0x7f59a4006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:15.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.692+0000 7f599a7fc700 1 -- 172.21.15.3:0/1551542170 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f59a00430d0 con 0x7f59a80b0a10 2022-07-21T01:41:15.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.856+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59a80447e0 con 0x7f59a80b0a10 2022-07-21T01:41:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f599a7fc700 1 -- 172.21.15.3:0/1551542170 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2484 (secure 0 0 0) 0x7f59a00466f0 con 0x7f59a80b0a10 2022-07-21T01:41:15.862 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:15.863 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":54,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1707710855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1707710855},{"type":"v1","addr":"172.21.15.143:6833","nonce":1707710855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/764850063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":764850063},{"type":"v1","addr":"172.21.15.3:6835","nonce":764850063}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:04.591520+0000","modified":"2022-07-21T01:41:05.594363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7254":{"gid":7254,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/151311757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":151311757},{"type":"v1","addr":"172.21.15.3:6837","nonce":151311757}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-21T01:41:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5990053a70 msgr2=0x7f5990055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5990053a70 0x7f5990055f30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f59a400b000 tx=0x7f59a4006eb0).stop 2022-07-21T01:41:15.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59a80b0a10 msgr2=0x7f59a813c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59a80b0a10 0x7f59a813c6f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7f59a0006e00 tx=0x7f59a00091d0).stop 2022-07-21T01:41:15.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 shutdown_connections 2022-07-21T01:41:15.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59a80b0a10 0x7f59a813c6f0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5990053a70 0x7f5990055f30 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a80b0630 0x7f59a8143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 --2- 172.21.15.3:0/1551542170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59a80aa2b0 0x7f59a80ac150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:15.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 >> 172.21.15.3:0/1551542170 conn(0x7f59a8060950 msgr2=0x7f59a80ae570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 shutdown_connections 2022-07-21T01:41:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.860+0000 7f59af47d700 1 -- 172.21.15.3:0/1551542170 wait complete. 2022-07-21T01:41:15.875 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 54 2022-07-21T01:41:15.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:41:16.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f7363700 1 Processor -- start 2022-07-21T01:41:16.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f7363700 1 -- start start 2022-07-21T01:41:16.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f7363700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe5f00b07a0 0x7fe5f00af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f7363700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f0147b80 con 0x7fe5f0147730 2022-07-21T01:41:16.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f7363700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f00b5a10 con 0x7fe5f00b5630 2022-07-21T01:41:16.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f7363700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f0145470 con 0x7fe5f00b07a0 2022-07-21T01:41:16.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f6b62700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe5f0147730 0x7fe5f00acdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57640/0 (socket says 172.21.15.3:57640) 2022-07-21T01:41:16.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f6b62700 1 -- 172.21.15.3:0/1667642476 learned_addr learned my addr 172.21.15.3:0/1667642476 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:16.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f5b60700 1 --2- 172.21.15.3:0/1667642476 >> v2:172.21.15.143:3300/0 conn(0x7fe5f00b07a0 0x7fe5f00af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938829996 0 0) 0x7fe5f0147b80 con 0x7fe5f0147730 2022-07-21T01:41:16.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d8003260 con 0x7fe5f0147730 2022-07-21T01:41:16.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566343255 0 0) 0x7fe5d8003260 con 0x7fe5f0147730 2022-07-21T01:41:16.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 >> v2:172.21.15.143:3300/0 conn(0x7fe5f00b07a0 msgr2=0x7fe5f00af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:16.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 --2- 172.21.15.3:0/1667642476 >> v2:172.21.15.143:3300/0 conn(0x7fe5f00b07a0 0x7fe5f00af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 >> v1:172.21.15.143:6789/0 conn(0x7fe5f00b5630 legacy=0x7fe5f0067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f00ac990 con 0x7fe5f0147730 2022-07-21T01:41:16.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fe5e8001660 con 0x7fe5f0147730 2022-07-21T01:41:16.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 >> v1:172.21.15.3:6789/0 conn(0x7fe5f0147730 legacy=0x7fe5f00acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8003750 0x7fe5d8003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8004500 0x7fe5d8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5d8004e70 0x7fe5d800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0145470 con 0x7fe5d8003750 2022-07-21T01:41:16.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d800bd20 con 0x7fe5d8004500 2022-07-21T01:41:16.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d800bee0 con 0x7fe5d8004e70 2022-07-21T01:41:16.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.984+0000 7fe5f6361700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8003750 0x7fe5d8003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f5b60700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8004500 0x7fe5d8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f6b62700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5d8004e70 0x7fe5d800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f6361700 1 -- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5d8004e70 msgr2=0x7fe5d800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:16.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f6361700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5d8004e70 0x7fe5d800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f6361700 1 -- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8004500 msgr2=0x7fe5d8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:16.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f6361700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8004500 0x7fe5d8004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f6361700 1 -- 172.21.15.3:0/1667642476 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8003260 con 0x7fe5d8003750 2022-07-21T01:41:16.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f5b60700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8004500 0x7fe5d8004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:16.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f6b62700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5d8004e70 0x7fe5d800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:16.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f6b62700 1 -- 172.21.15.3:0/1667642476 reap_dead start 2022-07-21T01:41:16.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f6361700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8003750 0x7fe5d8003b90 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7fe5f0069fc0 tx=0x7fe5e0007a00).ready entity=mon.0 client_cookie=9ebd51dc439fed9c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:16.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5e0007d10 con 0x7fe5d8003750 2022-07-21T01:41:16.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e0007030 con 0x7fe5d8003750 2022-07-21T01:41:16.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f535f700 1 -- 172.21.15.3:0/1667642476 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5e0007390 con 0x7fe5d8003750 2022-07-21T01:41:16.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f7363700 1 -- 172.21.15.3:0/1667642476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8003750 msgr2=0x7fe5d8003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f7363700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8003750 0x7fe5d8003b90 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7fe5f0069fc0 tx=0x7fe5e0007a00).stop 2022-07-21T01:41:16.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f7363700 1 -- 172.21.15.3:0/1667642476 shutdown_connections 2022-07-21T01:41:16.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f7363700 1 --2- 172.21.15.3:0/1667642476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5d8003750 0x7fe5d8003b90 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f7363700 1 -- 172.21.15.3:0/1667642476 >> 172.21.15.3:0/1667642476 conn(0x7fe5f00626f0 msgr2=0x7fe5f0062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:16.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f7363700 1 -- 172.21.15.3:0/1667642476 shutdown_connections 2022-07-21T01:41:16.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f7363700 1 -- 172.21.15.3:0/1667642476 wait complete. 2022-07-21T01:41:16.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f7363700 1 Processor -- start 2022-07-21T01:41:16.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.988+0000 7fe5f7363700 1 -- start start 2022-07-21T01:41:16.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f7363700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8003750 0x7fe5f00b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f7363700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f0065490 0x7fe5f015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f7363700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5f015ae00 0x7fe5f00633d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f7363700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f00ae180 con 0x7fe5f0065490 2022-07-21T01:41:16.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f7363700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f00647b0 con 0x7fe5d8003750 2022-07-21T01:41:16.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f7363700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f00649e0 con 0x7fe5f015ae00 2022-07-21T01:41:16.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f5b60700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f0065490 0x7fe5f015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f5b60700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f0065490 0x7fe5f015a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49372/0 (socket says 172.21.15.3:49372) 2022-07-21T01:41:16.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f5b60700 1 -- 172.21.15.3:0/3973856290 learned_addr learned my addr 172.21.15.3:0/3973856290 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:16.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f6b62700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5f015ae00 0x7fe5f00633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f6361700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8003750 0x7fe5f00b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f5b60700 1 -- 172.21.15.3:0/3973856290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5f015ae00 msgr2=0x7fe5f00633d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f5b60700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5f015ae00 0x7fe5f00633d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f5b60700 1 -- 172.21.15.3:0/3973856290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8003750 msgr2=0x7fe5f00b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f5b60700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8003750 0x7fe5f00b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f5b60700 1 -- 172.21.15.3:0/3973856290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f0064c80 con 0x7fe5f0065490 2022-07-21T01:41:16.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f6361700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8003750 0x7fe5f00b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:16.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f5b60700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f0065490 0x7fe5f015a910 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7fe5f00a82f0 tx=0x7fe5ec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:16.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5df7fe700 1 -- 172.21.15.3:0/3973856290 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5ec00f700 con 0x7fe5f0065490 2022-07-21T01:41:16.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5df7fe700 1 -- 172.21.15.3:0/3973856290 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec00f860 con 0x7fe5f0065490 2022-07-21T01:41:16.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f015a470 con 0x7fe5f0065490 2022-07-21T01:41:16.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5df7fe700 1 -- 172.21.15.3:0/3973856290 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5ec00f9d0 con 0x7fe5f0065490 2022-07-21T01:41:16.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f00b4af0 con 0x7fe5f0065490 2022-07-21T01:41:16.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.992+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f015a650 con 0x7fe5f0065490 2022-07-21T01:41:16.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.996+0000 7fe5df7fe700 1 -- 172.21.15.3:0/3973856290 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe5ec025070 con 0x7fe5f0065490 2022-07-21T01:41:16.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.996+0000 7fe5df7fe700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5d0053b00 0x7fe5d0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.996+0000 7fe5df7fe700 1 -- 172.21.15.3:0/3973856290 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe5ec06c880 con 0x7fe5f0065490 2022-07-21T01:41:16.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.996+0000 7fe5f6361700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5d0053b00 0x7fe5d0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:15.996+0000 7fe5f6361700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5d0053b00 0x7fe5d0055fc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fe5e0019000 tx=0x7fe5e0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:16.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.000+0000 7fe5df7fe700 1 -- 172.21.15.3:0/3973856290 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe5f015a650 con 0x7fe5f0065490 2022-07-21T01:41:16.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.164+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe5f015a650 con 0x7fe5f0065490 2022-07-21T01:41:16.579 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:41:16.576+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:41:16.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.592+0000 7fe5df7fe700 1 -- 172.21.15.3:0/3973856290 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe5f015a650 con 0x7fe5f0065490 2022-07-21T01:41:16.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5d0053b00 msgr2=0x7fe5d0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5d0053b00 0x7fe5d0055fc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fe5e0019000 tx=0x7fe5e0015000).stop 2022-07-21T01:41:16.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f0065490 msgr2=0x7fe5f015a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f0065490 0x7fe5f015a910 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7fe5f00a82f0 tx=0x7fe5ec008040).stop 2022-07-21T01:41:16.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 shutdown_connections 2022-07-21T01:41:16.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5f015ae00 0x7fe5f00633d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f0065490 0x7fe5f015a910 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe5d0053b00 0x7fe5d0055fc0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 --2- 172.21.15.3:0/3973856290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5d8003750 0x7fe5f00b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 >> 172.21.15.3:0/3973856290 conn(0x7fe5f00626f0 msgr2=0x7fe5f0067500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:16.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 shutdown_connections 2022-07-21T01:41:16.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.596+0000 7fe5f7363700 1 -- 172.21.15.3:0/3973856290 wait complete. 2022-07-21T01:41:16.610 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:41:16.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:16.619 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:41:16.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.716+0000 7fb0818c2700 1 Processor -- start 2022-07-21T01:41:16.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.716+0000 7fb0818c2700 1 -- start start 2022-07-21T01:41:16.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb0818c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c0ac6a0 0x7fb07c14cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb0818c2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c1453e0 con 0x7fb07c0b0160 2022-07-21T01:41:16.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb0818c2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c145680 con 0x7fb07c0b2390 2022-07-21T01:41:16.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb0818c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c141390 con 0x7fb07c0ac6a0 2022-07-21T01:41:16.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07a7fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb07c0b0160 0x7fb07c141830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57670/0 (socket says 172.21.15.3:57670) 2022-07-21T01:41:16.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07a7fc700 1 -- 172.21.15.3:0/1903856024 learned_addr learned my addr 172.21.15.3:0/1903856024 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:16.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07b7fe700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c0ac6a0 0x7fb07c14cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799534534 0 0) 0x7fb07c1453e0 con 0x7fb07c0b0160 2022-07-21T01:41:16.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb060003260 con 0x7fb07c0b0160 2022-07-21T01:41:16.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415290546 0 0) 0x7fb060003260 con 0x7fb07c0b0160 2022-07-21T01:41:16.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c0ac6a0 msgr2=0x7fb07c14cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:16.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c0ac6a0 0x7fb07c14cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 >> v1:172.21.15.143:6789/0 conn(0x7fb07c0b2390 legacy=0x7fb07c0abf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c0b0670 con 0x7fb07c0b0160 2022-07-21T01:41:16.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fb070001680 con 0x7fb07c0b0160 2022-07-21T01:41:16.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 >> v1:172.21.15.3:6789/0 conn(0x7fb07c0b0160 legacy=0x7fb07c141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb060003780 0x7fb060003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb060004500 0x7fb060004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c0ac6a0 0x7fb060004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c141390 con 0x7fb060003780 2022-07-21T01:41:16.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06000b750 con 0x7fb060004500 2022-07-21T01:41:16.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06000b8e0 con 0x7fb07c0ac6a0 2022-07-21T01:41:16.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07affd700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb060003780 0x7fb060003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07a7fc700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb060004500 0x7fb060004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07b7fe700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c0ac6a0 0x7fb060004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07affd700 1 -- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c0ac6a0 msgr2=0x7fb060004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:16.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07affd700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c0ac6a0 0x7fb060004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07affd700 1 -- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb060004500 msgr2=0x7fb060004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:16.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07affd700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb060004500 0x7fb060004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07affd700 1 -- 172.21.15.3:0/1903856024 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb060003260 con 0x7fb060003780 2022-07-21T01:41:16.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb07affd700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb060003780 0x7fb060003b60 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7fb06c001020 tx=0x7fb06c0079e0).ready entity=mon.0 client_cookie=4a366b49cca2f277 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:16.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb06c007cf0 con 0x7fb060003780 2022-07-21T01:41:16.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c007030 con 0x7fb060003780 2022-07-21T01:41:16.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.720+0000 7fb079ffb700 1 -- 172.21.15.3:0/1903856024 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb06c007390 con 0x7fb060003780 2022-07-21T01:41:16.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.724+0000 7fb0818c2700 1 -- 172.21.15.3:0/1903856024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb060003780 msgr2=0x7fb060003b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.724+0000 7fb0818c2700 1 --2- 172.21.15.3:0/1903856024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb060003780 0x7fb060003b60 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7fb06c001020 tx=0x7fb06c0079e0).stop 2022-07-21T01:41:16.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.724+0000 7fb07b7fe700 1 -- 172.21.15.3:0/1903856024 reap_dead start 2022-07-21T01:41:16.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.724+0000 7fb0818c2700 1 -- 172.21.15.3:0/1903856024 shutdown_connections 2022-07-21T01:41:16.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.724+0000 7fb0818c2700 1 -- 172.21.15.3:0/1903856024 >> 172.21.15.3:0/1903856024 conn(0x7fb07c05f840 msgr2=0x7fb07c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:16.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.724+0000 7fb0818c2700 1 -- 172.21.15.3:0/1903856024 shutdown_connections 2022-07-21T01:41:16.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.724+0000 7fb0818c2700 1 -- 172.21.15.3:0/1903856024 wait complete. 2022-07-21T01:41:16.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.724+0000 7fb0818c2700 1 Processor -- start 2022-07-21T01:41:16.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.724+0000 7fb0818c2700 1 -- start start 2022-07-21T01:41:16.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb0818c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c140790 0x7fb07c13b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb0818c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb07c138c90 0x7fb07c139110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb0818c2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb07c153e70 0x7fb07c1542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb0818c2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c13c0e0 con 0x7fb07c153e70 2022-07-21T01:41:16.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb0818c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c13bec0 con 0x7fb07c138c90 2022-07-21T01:41:16.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb0818c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c140b70 con 0x7fb07c140790 2022-07-21T01:41:16.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb07c153e70 0x7fb07c1542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb07c153e70 0x7fb07c1542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49406/0 (socket says 172.21.15.3:49406) 2022-07-21T01:41:16.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07a7fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb07c138c90 0x7fb07c139110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07b7fe700 1 -- 172.21.15.3:0/741929798 learned_addr learned my addr 172.21.15.3:0/741929798 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:16.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07affd700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c140790 0x7fb07c13b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07b7fe700 1 -- 172.21.15.3:0/741929798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c140790 msgr2=0x7fb07c13b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07b7fe700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c140790 0x7fb07c13b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07b7fe700 1 -- 172.21.15.3:0/741929798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb07c138c90 msgr2=0x7fb07c139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07b7fe700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb07c138c90 0x7fb07c139110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07b7fe700 1 -- 172.21.15.3:0/741929798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c13cc60 con 0x7fb07c153e70 2022-07-21T01:41:16.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb07b7fe700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb07c153e70 0x7fb07c1542f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7fb07c144a70 tx=0x7fb0740091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:16.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb06bfff700 1 -- 172.21.15.3:0/741929798 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb074019070 con 0x7fb07c153e70 2022-07-21T01:41:16.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb06bfff700 1 -- 172.21.15.3:0/741929798 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07400f7d0 con 0x7fb07c153e70 2022-07-21T01:41:16.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07c13cf00 con 0x7fb07c153e70 2022-07-21T01:41:16.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb06bfff700 1 -- 172.21.15.3:0/741929798 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb07400fb30 con 0x7fb07c153e70 2022-07-21T01:41:16.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.728+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07c0640f0 con 0x7fb07c153e70 2022-07-21T01:41:16.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.732+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07c0b2de0 con 0x7fb07c153e70 2022-07-21T01:41:16.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.732+0000 7fb06bfff700 1 -- 172.21.15.3:0/741929798 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb07400fc90 con 0x7fb07c153e70 2022-07-21T01:41:16.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.732+0000 7fb06bfff700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb058053b00 0x7fb058055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:16.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.732+0000 7fb06bfff700 1 -- 172.21.15.3:0/741929798 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fb07406c900 con 0x7fb07c153e70 2022-07-21T01:41:16.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.732+0000 7fb07affd700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb058053b00 0x7fb058055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:16.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.732+0000 7fb07affd700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb058053b00 0x7fb058055fc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fb06c01b000 tx=0x7fb06c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:16.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.740+0000 7fb06bfff700 1 -- 172.21.15.3:0/741929798 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb074043670 con 0x7fb07c153e70 2022-07-21T01:41:16.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.900+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb07c0574e0 con 0x7fb07c153e70 2022-07-21T01:41:16.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb06bfff700 1 -- 172.21.15.3:0/741929798 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2130 (secure 0 0 0) 0x7fb074046c90 con 0x7fb07c153e70 2022-07-21T01:41:16.906 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:16.906 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":55,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1707710855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1707710855},{"type":"v1","addr":"172.21.15.143:6833","nonce":1707710855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/764850063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":764850063},{"type":"v1","addr":"172.21.15.3:6835","nonce":764850063}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":55,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:04.591520+0000","modified":"2022-07-21T01:41:16.579862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-21T01:41:16.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb058053b00 msgr2=0x7fb058055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb058053b00 0x7fb058055fc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fb06c01b000 tx=0x7fb06c015000).stop 2022-07-21T01:41:16.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb07c153e70 msgr2=0x7fb07c1542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:16.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb07c153e70 0x7fb07c1542f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7fb07c144a70 tx=0x7fb0740091d0).stop 2022-07-21T01:41:16.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 shutdown_connections 2022-07-21T01:41:16.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb07c153e70 0x7fb07c1542f0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb058053b00 0x7fb058055fc0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb07c138c90 0x7fb07c139110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 --2- 172.21.15.3:0/741929798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb07c140790 0x7fb07c13b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:16.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 >> 172.21.15.3:0/741929798 conn(0x7fb07c05f840 msgr2=0x7fb07c13ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:16.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 shutdown_connections 2022-07-21T01:41:16.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:16.904+0000 7fb0818c2700 1 -- 172.21.15.3:0/741929798 wait complete. 2022-07-21T01:41:16.911 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 55 2022-07-21T01:41:16.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:41:17.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6bab3da700 1 Processor -- start 2022-07-21T01:41:17.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6bab3da700 1 -- start start 2022-07-21T01:41:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6bab3da700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6ba40ad1c0 0x7f6ba40b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6bab3da700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba40ae650 con 0x7f6ba40b0f90 2022-07-21T01:41:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6bab3da700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba40ae880 con 0x7f6ba40b22b0 2022-07-21T01:41:17.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6bab3da700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba40aeb20 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6ba8975700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6ba40ad1c0 0x7f6ba40b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6ba8975700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6ba40ad1c0 0x7f6ba40b0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49412/0 (socket says 172.21.15.3:49412) 2022-07-21T01:41:17.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6ba8975700 1 -- 172.21.15.3:0/2149888172 learned_addr learned my addr 172.21.15.3:0/2149888172 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:17.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6ba9977700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6ba40b0f90 0x7f6ba40b1400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57694/0 (socket says 172.21.15.3:57694) 2022-07-21T01:41:17.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6ba8975700 1 -- 172.21.15.3:0/2149888172 >> v1:172.21.15.143:6789/0 conn(0x7f6ba40b22b0 legacy=0x7f6ba4143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6ba8975700 1 -- 172.21.15.3:0/2149888172 >> v1:172.21.15.3:6789/0 conn(0x7f6ba40b0f90 legacy=0x7f6ba40b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6b9bfff700 1 -- 172.21.15.3:0/2149888172 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490371634 0 0) 0x7f6ba40ae880 con 0x7f6ba40b22b0 2022-07-21T01:41:17.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6ba8975700 1 -- 172.21.15.3:0/2149888172 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba40aedc0 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6ba8975700 1 --2- 172.21.15.3:0/2149888172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 0x7f6ba40b0140 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f6b9c001110 tx=0x7f6b94006eb0).ready entity=mon.0 client_cookie=2910fa890f1d3406 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:17.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6b9bfff700 1 -- 172.21.15.3:0/2149888172 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b9400d610 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6b9bfff700 1 -- 172.21.15.3:0/2149888172 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b94000e90 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.032+0000 7f6b9bfff700 1 -- 172.21.15.3:0/2149888172 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b9400d960 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.036+0000 7f6bab3da700 1 -- 172.21.15.3:0/2149888172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 msgr2=0x7f6ba40b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.036+0000 7f6bab3da700 1 --2- 172.21.15.3:0/2149888172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 0x7f6ba40b0140 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f6b9c001110 tx=0x7f6b94006eb0).stop 2022-07-21T01:41:17.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.036+0000 7f6bab3da700 1 -- 172.21.15.3:0/2149888172 shutdown_connections 2022-07-21T01:41:17.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.036+0000 7f6bab3da700 1 --2- 172.21.15.3:0/2149888172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 0x7f6ba40b0140 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.036+0000 7f6bab3da700 1 -- 172.21.15.3:0/2149888172 >> 172.21.15.3:0/2149888172 conn(0x7f6ba405f840 msgr2=0x7f6ba405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:17.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.036+0000 7f6bab3da700 1 -- 172.21.15.3:0/2149888172 shutdown_connections 2022-07-21T01:41:17.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.036+0000 7f6bab3da700 1 -- 172.21.15.3:0/2149888172 wait complete. 2022-07-21T01:41:17.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.036+0000 7f6bab3da700 1 Processor -- start 2022-07-21T01:41:17.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.036+0000 7f6bab3da700 1 -- start start 2022-07-21T01:41:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6bab3da700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 0x7f6ba4145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6bab3da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40b0f90 0x7f6ba4153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6bab3da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ba40b22b0 0x7f6ba4153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6bab3da700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba41529e0 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6bab3da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba40a0700 con 0x7f6ba40b0f90 2022-07-21T01:41:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6bab3da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba40a09a0 con 0x7f6ba40b22b0 2022-07-21T01:41:17.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9176700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 0x7f6ba4145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9176700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 0x7f6ba4145e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49416/0 (socket says 172.21.15.3:49416) 2022-07-21T01:41:17.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9176700 1 -- 172.21.15.3:0/2810790890 learned_addr learned my addr 172.21.15.3:0/2810790890 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:17.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9977700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ba40b22b0 0x7f6ba4153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba8975700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40b0f90 0x7f6ba4153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9176700 1 -- 172.21.15.3:0/2810790890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ba40b22b0 msgr2=0x7f6ba4153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9176700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ba40b22b0 0x7f6ba4153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9176700 1 -- 172.21.15.3:0/2810790890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40b0f90 msgr2=0x7f6ba4153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9176700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40b0f90 0x7f6ba4153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9176700 1 -- 172.21.15.3:0/2810790890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9400b1e0 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba8975700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40b0f90 0x7f6ba4153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6ba9176700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 0x7f6ba4145e40 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7f6ba0001250 tx=0x7f6ba00013f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6b9a7fc700 1 -- 172.21.15.3:0/2810790890 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ba000e4a0 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6b9a7fc700 1 -- 172.21.15.3:0/2810790890 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba000e600 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba4152bd0 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6b9a7fc700 1 -- 172.21.15.3:0/2810790890 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ba000e960 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba40a3c70 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.040+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba4152d60 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.044+0000 7f6b9a7fc700 1 -- 172.21.15.3:0/2810790890 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6ba0019030 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.044+0000 7f6b9a7fc700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6b8c053a70 0x7f6b8c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.044+0000 7f6b9a7fc700 1 -- 172.21.15.3:0/2810790890 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f6ba006b670 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.044+0000 7f6ba8975700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6b8c053a70 0x7f6b8c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.044+0000 7f6ba8975700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6b8c053a70 0x7f6b8c055f30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f6b9401a000 tx=0x7f6b94015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.048+0000 7f6b9a7fc700 1 -- 172.21.15.3:0/2810790890 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6ba00423e0 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.208+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6ba40574e0 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.600+0000 7f6b9a7fc700 1 -- 172.21.15.3:0/2810790890 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ba0045a00 con 0x7f6ba40ad1c0 2022-07-21T01:41:17.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6b8c053a70 msgr2=0x7f6b8c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6b8c053a70 0x7f6b8c055f30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f6b9401a000 tx=0x7f6b94015000).stop 2022-07-21T01:41:17.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 msgr2=0x7f6ba4145e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 0x7f6ba4145e40 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7f6ba0001250 tx=0x7f6ba00013f0).stop 2022-07-21T01:41:17.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 shutdown_connections 2022-07-21T01:41:17.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ba40b22b0 0x7f6ba4153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6b8c053a70 0x7f6b8c055f30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40b0f90 0x7f6ba4153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 --2- 172.21.15.3:0/2810790890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6ba40ad1c0 0x7f6ba4145e40 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 >> 172.21.15.3:0/2810790890 conn(0x7f6ba405f840 msgr2=0x7f6ba4142de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:17.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 shutdown_connections 2022-07-21T01:41:17.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.604+0000 7f6bab3da700 1 -- 172.21.15.3:0/2810790890 wait complete. 2022-07-21T01:41:17.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:41:17.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.736+0000 7f32c01b7700 1 Processor -- start 2022-07-21T01:41:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.736+0000 7f32c01b7700 1 -- start start 2022-07-21T01:41:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.736+0000 7f32c01b7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f32b8066e70 0x7f32b814eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.736+0000 7f32c01b7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b8060670 con 0x7f32b8060290 2022-07-21T01:41:17.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.736+0000 7f32c01b7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b8060e00 con 0x7f32b80617e0 2022-07-21T01:41:17.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.736+0000 7f32c01b7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b80672c0 con 0x7f32b8066e70 2022-07-21T01:41:17.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.736+0000 7f32bdf53700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f32b8060290 0x7f32b8065b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57706/0 (socket says 172.21.15.3:57706) 2022-07-21T01:41:17.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.736+0000 7f32bdf53700 1 -- 172.21.15.3:0/2699201152 learned_addr learned my addr 172.21.15.3:0/2699201152 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:17.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32be754700 1 --2- 172.21.15.3:0/2699201152 >> v2:172.21.15.143:3300/0 conn(0x7f32b8066e70 0x7f32b814eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32bcf51700 1 -- 172.21.15.3:0/2699201152 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463101080 0 0) 0x7f32b8060670 con 0x7f32b8060290 2022-07-21T01:41:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32bcf51700 1 -- 172.21.15.3:0/2699201152 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a0003260 con 0x7f32b8060290 2022-07-21T01:41:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32be754700 1 -- 172.21.15.3:0/2699201152 >> v1:172.21.15.143:6789/0 conn(0x7f32b80617e0 legacy=0x7f32b8066760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32be754700 1 -- 172.21.15.3:0/2699201152 >> v1:172.21.15.3:6789/0 conn(0x7f32b8060290 legacy=0x7f32b8065b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32be754700 1 -- 172.21.15.3:0/2699201152 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b80610a0 con 0x7f32b8066e70 2022-07-21T01:41:17.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32be754700 1 --2- 172.21.15.3:0/2699201152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 0x7f32b814eb70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f32b00032f0 tx=0x7f32b00091d0).ready entity=mon.1 client_cookie=9bc96d6e1b3b98c9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:17.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32bcf51700 1 -- 172.21.15.3:0/2699201152 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b000b6c0 con 0x7f32b8066e70 2022-07-21T01:41:17.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32bcf51700 1 -- 172.21.15.3:0/2699201152 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b000b820 con 0x7f32b8066e70 2022-07-21T01:41:17.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32bcf51700 1 -- 172.21.15.3:0/2699201152 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b000b990 con 0x7f32b8066e70 2022-07-21T01:41:17.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32c01b7700 1 -- 172.21.15.3:0/2699201152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 msgr2=0x7f32b814eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32c01b7700 1 --2- 172.21.15.3:0/2699201152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 0x7f32b814eb70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f32b00032f0 tx=0x7f32b00091d0).stop 2022-07-21T01:41:17.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32c01b7700 1 -- 172.21.15.3:0/2699201152 shutdown_connections 2022-07-21T01:41:17.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32c01b7700 1 --2- 172.21.15.3:0/2699201152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 0x7f32b814eb70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32c01b7700 1 -- 172.21.15.3:0/2699201152 >> 172.21.15.3:0/2699201152 conn(0x7f32b805d0c0 msgr2=0x7f32b805f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:17.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32c01b7700 1 -- 172.21.15.3:0/2699201152 shutdown_connections 2022-07-21T01:41:17.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.740+0000 7f32c01b7700 1 -- 172.21.15.3:0/2699201152 wait complete. 2022-07-21T01:41:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 Processor -- start 2022-07-21T01:41:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 -- start start 2022-07-21T01:41:17.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b8060290 0x7f32b80a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b80617e0 0x7f32b80a9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 0x7f32b80abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80a76e0 con 0x7f32b80617e0 2022-07-21T01:41:17.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80a9170 con 0x7f32b8066e70 2022-07-21T01:41:17.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80a8c30 con 0x7f32b8060290 2022-07-21T01:41:17.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bd752700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b80617e0 0x7f32b80a9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bd752700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b80617e0 0x7f32b80a9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49434/0 (socket says 172.21.15.3:49434) 2022-07-21T01:41:17.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bd752700 1 -- 172.21.15.3:0/1700989635 learned_addr learned my addr 172.21.15.3:0/1700989635 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:17.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bdf53700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b8060290 0x7f32b80a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32be754700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 0x7f32b80abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bd752700 1 -- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b8060290 msgr2=0x7f32b80a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bd752700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b8060290 0x7f32b80a82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bd752700 1 -- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 msgr2=0x7f32b80abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:17.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bd752700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 0x7f32b80abc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:17.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bd752700 1 -- 172.21.15.3:0/1700989635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b0009470 con 0x7f32b80617e0 2022-07-21T01:41:17.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32be754700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 0x7f32b80abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:17.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bdf53700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b8060290 0x7f32b80a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:17.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32bd752700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b80617e0 0x7f32b80a9880 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f32b8064000 tx=0x7f32b4007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:17.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32a6ffd700 1 -- 172.21.15.3:0/1700989635 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b400ce30 con 0x7f32b80617e0 2022-07-21T01:41:17.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32a6ffd700 1 -- 172.21.15.3:0/1700989635 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b4007240 con 0x7f32b80617e0 2022-07-21T01:41:17.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b80a8ed0 con 0x7f32b80617e0 2022-07-21T01:41:17.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32a6ffd700 1 -- 172.21.15.3:0/1700989635 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b4006700 con 0x7f32b80617e0 2022-07-21T01:41:17.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.744+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b80a9d70 con 0x7f32b80617e0 2022-07-21T01:41:17.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.748+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b8061340 con 0x7f32b80617e0 2022-07-21T01:41:17.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.748+0000 7f32a6ffd700 1 -- 172.21.15.3:0/1700989635 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f32b4010030 con 0x7f32b80617e0 2022-07-21T01:41:17.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.748+0000 7f32a6ffd700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f32980539e0 0x7f3298055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:17.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.748+0000 7f32a6ffd700 1 -- 172.21.15.3:0/1700989635 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f32b406be00 con 0x7f32b80617e0 2022-07-21T01:41:17.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.748+0000 7f32bdf53700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f32980539e0 0x7f3298055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:17.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.748+0000 7f32bdf53700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f32980539e0 0x7f3298055ea0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f32a800b000 tx=0x7f32a8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:17.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.756+0000 7f32a6ffd700 1 -- 172.21.15.3:0/1700989635 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f32b4042ad0 con 0x7f32b80617e0 2022-07-21T01:41:17.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:17.880+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f32b80574e0 con 0x7f32b80617e0 2022-07-21T01:41:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.616+0000 7f32a6ffd700 1 -- 172.21.15.3:0/1700989635 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v113) v1 ==== 185+0+0 (secure 0 0 0) 0x7f32b40460f0 con 0x7f32b80617e0 2022-07-21T01:41:18.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f32980539e0 msgr2=0x7f3298055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:18.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f32980539e0 0x7f3298055ea0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f32a800b000 tx=0x7f32a8009040).stop 2022-07-21T01:41:18.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b80617e0 msgr2=0x7f32b80a9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:18.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b80617e0 0x7f32b80a9880 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f32b8064000 tx=0x7f32b4007c10).stop 2022-07-21T01:41:18.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 shutdown_connections 2022-07-21T01:41:18.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f32980539e0 0x7f3298055ea0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:18.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32b8066e70 0x7f32b80abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:18.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b80617e0 0x7f32b80a9880 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:18.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 --2- 172.21.15.3:0/1700989635 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b8060290 0x7f32b80a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:18.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 >> 172.21.15.3:0/1700989635 conn(0x7f32b805d0c0 msgr2=0x7f32b8147b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:18.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 shutdown_connections 2022-07-21T01:41:18.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.620+0000 7f32c01b7700 1 -- 172.21.15.3:0/1700989635 wait complete. 2022-07-21T01:41:18.629 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:41:18.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:41:18.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.764+0000 7f9370b8a700 1 Processor -- start 2022-07-21T01:41:18.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.764+0000 7f9370b8a700 1 -- start start 2022-07-21T01:41:18.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f936c0b20a0 0x7f936c0b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:18.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0a9d00 0x7f936c0b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:18.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f936c0aa520 con 0x7f936c0b0250 2022-07-21T01:41:18.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f936c0ab3f0 con 0x7f936c0b20a0 2022-07-21T01:41:18.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936c0b0630 con 0x7f936c0a9d00 2022-07-21T01:41:18.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9363fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f936c0b20a0 0x7f936c0b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:18.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f936ad9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0a9d00 0x7f936c0b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:18.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9363fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f936c0b20a0 0x7f936c0b1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:54888/0 (socket says 172.21.15.3:54888) 2022-07-21T01:41:18.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9363fff700 1 -- 172.21.15.3:0/2539755344 learned_addr learned my addr 172.21.15.3:0/2539755344 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:18.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f936a59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f936c0b0250 0x7f936c0aace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:59838/0 (socket says 172.21.15.3:59838) 2022-07-21T01:41:18.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9363fff700 1 -- 172.21.15.3:0/2539755344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0a9d00 msgr2=0x7f936c0b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:18.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9363fff700 1 --2- 172.21.15.3:0/2539755344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0a9d00 0x7f936c0b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:18.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9363fff700 1 -- 172.21.15.3:0/2539755344 >> v1:172.21.15.143:6789/0 conn(0x7f936c0b0250 legacy=0x7f936c0aace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:18.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9363fff700 1 -- 172.21.15.3:0/2539755344 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c0aa280 con 0x7f936c0b20a0 2022-07-21T01:41:18.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f936ad9d700 1 --2- 172.21.15.3:0/2539755344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0a9d00 0x7f936c0b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:18.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9363fff700 1 --2- 172.21.15.3:0/2539755344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f936c0b20a0 0x7f936c0b1ab0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f9354002f70 tx=0x7f9354006eb0).ready entity=mon.1 client_cookie=f440d99bca1d87c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:18.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9369d9b700 1 -- 172.21.15.3:0/2539755344 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935400d570 con 0x7f936c0b20a0 2022-07-21T01:41:18.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9369d9b700 1 -- 172.21.15.3:0/2539755344 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9354000e90 con 0x7f936c0b20a0 2022-07-21T01:41:18.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9369d9b700 1 -- 172.21.15.3:0/2539755344 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935400d900 con 0x7f936c0b20a0 2022-07-21T01:41:18.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 -- 172.21.15.3:0/2539755344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f936c0b20a0 msgr2=0x7f936c0b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:18.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 --2- 172.21.15.3:0/2539755344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f936c0b20a0 0x7f936c0b1ab0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f9354002f70 tx=0x7f9354006eb0).stop 2022-07-21T01:41:18.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 -- 172.21.15.3:0/2539755344 shutdown_connections 2022-07-21T01:41:18.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 --2- 172.21.15.3:0/2539755344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0a9d00 0x7f936c0b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:18.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 --2- 172.21.15.3:0/2539755344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f936c0b20a0 0x7f936c0b1ab0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:18.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 -- 172.21.15.3:0/2539755344 >> 172.21.15.3:0/2539755344 conn(0x7f936c060950 msgr2=0x7f936c060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 -- 172.21.15.3:0/2539755344 shutdown_connections 2022-07-21T01:41:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.768+0000 7f9370b8a700 1 -- 172.21.15.3:0/2539755344 wait complete. 2022-07-21T01:41:18.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f9370b8a700 1 Processor -- start 2022-07-21T01:41:18.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f9370b8a700 1 -- start start 2022-07-21T01:41:18.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f9370b8a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f936c0a9d00 0x7f936c140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:18.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f9370b8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0b0250 0x7f936c1432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:18.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f9370b8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f936c0b20a0 0x7f936c13d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:18.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f9370b8a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c1437c0 con 0x7f936c0a9d00 2022-07-21T01:41:18.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f9370b8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c142e70 con 0x7f936c0b20a0 2022-07-21T01:41:18.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f9370b8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936c13ebb0 con 0x7f936c0b0250 2022-07-21T01:41:18.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f936c0a9d00 0x7f936c140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:18.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f936c0a9d00 0x7f936c140af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49446/0 (socket says 172.21.15.3:49446) 2022-07-21T01:41:18.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936ad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f936c0b20a0 0x7f936c13d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:18.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f9363fff700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0b0250 0x7f936c1432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:18.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936a59c700 1 -- 172.21.15.3:0/3673142359 learned_addr learned my addr 172.21.15.3:0/3673142359 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:18.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936a59c700 1 -- 172.21.15.3:0/3673142359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0b0250 msgr2=0x7f936c1432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:18.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936a59c700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0b0250 0x7f936c1432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:18.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936a59c700 1 -- 172.21.15.3:0/3673142359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f936c0b20a0 msgr2=0x7f936c13d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:18.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936a59c700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f936c0b20a0 0x7f936c13d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:18.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936a59c700 1 -- 172.21.15.3:0/3673142359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936400d910 con 0x7f936c0a9d00 2022-07-21T01:41:18.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.772+0000 7f936a59c700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f936c0a9d00 0x7f936c140af0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f936c142d90 tx=0x7f935c007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:18.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f93637fe700 1 -- 172.21.15.3:0/3673142359 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935c0070d0 con 0x7f936c0a9d00 2022-07-21T01:41:18.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935400b1e0 con 0x7f936c0a9d00 2022-07-21T01:41:18.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f93637fe700 1 -- 172.21.15.3:0/3673142359 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935c007230 con 0x7f936c0a9d00 2022-07-21T01:41:18.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f93637fe700 1 -- 172.21.15.3:0/3673142359 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935c005550 con 0x7f936c0a9d00 2022-07-21T01:41:18.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936c13e910 con 0x7f936c0a9d00 2022-07-21T01:41:18.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936c143030 con 0x7f936c0a9d00 2022-07-21T01:41:18.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f93637fe700 1 -- 172.21.15.3:0/3673142359 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f935c01a070 con 0x7f936c0a9d00 2022-07-21T01:41:18.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f93637fe700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93500539e0 0x7f9350055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:18.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f9363fff700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93500539e0 0x7f9350055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:18.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.776+0000 7f93637fe700 1 -- 172.21.15.3:0/3673142359 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f935c06b2a0 con 0x7f936c0a9d00 2022-07-21T01:41:18.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.780+0000 7f9363fff700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93500539e0 0x7f9350055ea0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f935401c000 tx=0x7f9354015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:18.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.784+0000 7f93637fe700 1 -- 172.21.15.3:0/3673142359 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f936c143030 con 0x7f936c0a9d00 2022-07-21T01:41:18.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:18.912+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f936c143030 con 0x7f936c0a9d00 2022-07-21T01:41:19.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.620+0000 7f93637fe700 1 -- 172.21.15.3:0/3673142359 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v114) v1 ==== 173+0+0 (secure 0 0 0) 0x7f936c143030 con 0x7f936c0a9d00 2022-07-21T01:41:19.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93500539e0 msgr2=0x7f9350055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:19.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93500539e0 0x7f9350055ea0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f935401c000 tx=0x7f9354015000).stop 2022-07-21T01:41:19.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f936c0a9d00 msgr2=0x7f936c140af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:19.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f936c0a9d00 0x7f936c140af0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f936c142d90 tx=0x7f935c007c40).stop 2022-07-21T01:41:19.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 shutdown_connections 2022-07-21T01:41:19.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93500539e0 0x7f9350055ea0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f936c0b20a0 0x7f936c13d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f936c0b0250 0x7f936c1432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 --2- 172.21.15.3:0/3673142359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f936c0a9d00 0x7f936c140af0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 >> 172.21.15.3:0/3673142359 conn(0x7f936c060950 msgr2=0x7f936c0b4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:19.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 shutdown_connections 2022-07-21T01:41:19.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.624+0000 7f9370b8a700 1 -- 172.21.15.3:0/3673142359 wait complete. 2022-07-21T01:41:19.632 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:41:19.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:41:19.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3883fff700 1 Processor -- start 2022-07-21T01:41:19.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3883fff700 1 -- start start 2022-07-21T01:41:19.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3883fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f38840aec40 0x7f38840af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:19.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3883fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 0x7f3884146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:19.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3883fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38840b0190 con 0x7f3884146e40 2022-07-21T01:41:19.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3883fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38840ae0d0 con 0x7f38840aec40 2022-07-21T01:41:19.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3883fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38840ad230 con 0x7f38840b0610 2022-07-21T01:41:19.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3882ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f38840aec40 0x7f38840af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:19.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3882ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f38840aec40 0x7f38840af020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49454/0 (socket says 172.21.15.3:49454) 2022-07-21T01:41:19.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3882ffd700 1 -- 172.21.15.3:0/353566319 learned_addr learned my addr 172.21.15.3:0/353566319 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:19.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f38837fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3884146e40 0x7f388405f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57736/0 (socket says 172.21.15.3:57736) 2022-07-21T01:41:19.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f387bfff700 1 --2- 172.21.15.3:0/353566319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 0x7f3884146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:19.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3882ffd700 1 -- 172.21.15.3:0/353566319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 msgr2=0x7f3884146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:19.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3882ffd700 1 --2- 172.21.15.3:0/353566319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 0x7f3884146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f3882ffd700 1 -- 172.21.15.3:0/353566319 >> v1:172.21.15.3:6789/0 conn(0x7f3884146e40 legacy=0x7f388405f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:19.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.748+0000 7f38827fc700 1 -- 172.21.15.3:0/353566319 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771185899 0 0) 0x7f38840b0190 con 0x7f3884146e40 2022-07-21T01:41:19.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3882ffd700 1 -- 172.21.15.3:0/353566319 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38840b0b20 con 0x7f38840aec40 2022-07-21T01:41:19.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3882ffd700 1 --2- 172.21.15.3:0/353566319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 0x7f38840af020 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7f3874002f70 tx=0x7f3874006eb0).ready entity=mon.0 client_cookie=92f9ee7ce14b4970 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:19.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f38827fc700 1 -- 172.21.15.3:0/353566319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f387400d570 con 0x7f38840aec40 2022-07-21T01:41:19.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f38827fc700 1 -- 172.21.15.3:0/353566319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3874000e90 con 0x7f38840aec40 2022-07-21T01:41:19.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f38827fc700 1 -- 172.21.15.3:0/353566319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f387400d8c0 con 0x7f38840aec40 2022-07-21T01:41:19.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 -- 172.21.15.3:0/353566319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 msgr2=0x7f38840af020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:19.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 --2- 172.21.15.3:0/353566319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 0x7f38840af020 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7f3874002f70 tx=0x7f3874006eb0).stop 2022-07-21T01:41:19.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 -- 172.21.15.3:0/353566319 shutdown_connections 2022-07-21T01:41:19.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 --2- 172.21.15.3:0/353566319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 0x7f3884146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 --2- 172.21.15.3:0/353566319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 0x7f38840af020 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 -- 172.21.15.3:0/353566319 >> 172.21.15.3:0/353566319 conn(0x7f38840613c0 msgr2=0x7f3884062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:19.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 -- 172.21.15.3:0/353566319 shutdown_connections 2022-07-21T01:41:19.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 -- 172.21.15.3:0/353566319 wait complete. 2022-07-21T01:41:19.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 Processor -- start 2022-07-21T01:41:19.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.752+0000 7f3883fff700 1 -- start start 2022-07-21T01:41:19.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3883fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 0x7f388413acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:19.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3883fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 0x7f3884142830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:19.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3883fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3884146e40 0x7f3884142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:19.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3883fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388413ca80 con 0x7f38840aec40 2022-07-21T01:41:19.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3883fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3884145090 con 0x7f3884146e40 2022-07-21T01:41:19.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3883fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3884143210 con 0x7f38840b0610 2022-07-21T01:41:19.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3882ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 0x7f388413acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:19.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3882ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 0x7f388413acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49462/0 (socket says 172.21.15.3:49462) 2022-07-21T01:41:19.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3882ffd700 1 -- 172.21.15.3:0/2845628563 learned_addr learned my addr 172.21.15.3:0/2845628563 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:19.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f387bfff700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 0x7f3884142830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:19.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f38837fe700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3884146e40 0x7f3884142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:19.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3882ffd700 1 -- 172.21.15.3:0/2845628563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 msgr2=0x7f3884142830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:19.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3882ffd700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 0x7f3884142830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3882ffd700 1 -- 172.21.15.3:0/2845628563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3884146e40 msgr2=0x7f3884142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:19.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3882ffd700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3884146e40 0x7f3884142d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3882ffd700 1 -- 172.21.15.3:0/2845628563 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387400b1e0 con 0x7f38840aec40 2022-07-21T01:41:19.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f38837fe700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3884146e40 0x7f3884142d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:19.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3882ffd700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 0x7f388413acd0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f38841453f0 tx=0x7f3874015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:19.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3880ff9700 1 -- 172.21.15.3:0/2845628563 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38740117e0 con 0x7f38840aec40 2022-07-21T01:41:19.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3880ff9700 1 -- 172.21.15.3:0/2845628563 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3874011940 con 0x7f38840aec40 2022-07-21T01:41:19.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388413fae0 con 0x7f38840aec40 2022-07-21T01:41:19.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3880ff9700 1 -- 172.21.15.3:0/2845628563 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3874011ca0 con 0x7f38840aec40 2022-07-21T01:41:19.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3884142050 con 0x7f38840aec40 2022-07-21T01:41:19.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38841472b0 con 0x7f38840aec40 2022-07-21T01:41:19.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.756+0000 7f3880ff9700 1 -- 172.21.15.3:0/2845628563 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3874025030 con 0x7f38840aec40 2022-07-21T01:41:19.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.760+0000 7f3880ff9700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38680539e0 0x7f3868055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:19.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.760+0000 7f3880ff9700 1 -- 172.21.15.3:0/2845628563 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f387406d060 con 0x7f38840aec40 2022-07-21T01:41:19.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.760+0000 7f387bfff700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38680539e0 0x7f3868055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:19.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.760+0000 7f387bfff700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38680539e0 0x7f3868055ea0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f386c00b000 tx=0x7f386c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:19.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.760+0000 7f3880ff9700 1 -- 172.21.15.3:0/2845628563 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3874044000 con 0x7f38840aec40 2022-07-21T01:41:19.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.888+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f388405f3d0 con 0x7f38840aec40 2022-07-21T01:41:19.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.896+0000 7f3880ff9700 1 -- 172.21.15.3:0/2845628563 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f38740230a0 con 0x7f38840aec40 2022-07-21T01:41:19.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38680539e0 msgr2=0x7f3868055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:19.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38680539e0 0x7f3868055ea0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f386c00b000 tx=0x7f386c006df0).stop 2022-07-21T01:41:19.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 msgr2=0x7f388413acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:19.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 0x7f388413acd0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f38841453f0 tx=0x7f3874015000).stop 2022-07-21T01:41:19.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 shutdown_connections 2022-07-21T01:41:19.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38680539e0 0x7f3868055ea0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3884146e40 0x7f3884142d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38840b0610 0x7f3884142830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 --2- 172.21.15.3:0/2845628563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38840aec40 0x7f388413acd0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:19.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 >> 172.21.15.3:0/2845628563 conn(0x7f38840613c0 msgr2=0x7f388413c120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:19.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 shutdown_connections 2022-07-21T01:41:19.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:19.900+0000 7f3883fff700 1 -- 172.21.15.3:0/2845628563 wait complete. 2022-07-21T01:41:19.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-07-21T01:41:20.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7facf55de700 1 Processor -- start 2022-07-21T01:41:20.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7facf55de700 1 -- start start 2022-07-21T01:41:20.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7facf55de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 0x7facf00a9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7facf55de700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7facf00a8cf0 0x7facf0143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7facf55de700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf00b1650 con 0x7facf01436d0 2022-07-21T01:41:20.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7facf55de700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf00ab570 con 0x7facf00a8cf0 2022-07-21T01:41:20.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7facf55de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf00b1d20 con 0x7facf005f840 2022-07-21T01:41:20.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7face67fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 0x7facf00a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7facef7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7facf00a8cf0 0x7facf0143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7faceeffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7facf01436d0 0x7facf00b0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:59864/0 (socket says 172.21.15.3:59864) 2022-07-21T01:41:20.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.020+0000 7faceeffd700 1 -- 172.21.15.3:0/1726925638 learned_addr learned my addr 172.21.15.3:0/1726925638 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:20.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facef7fe700 1 -- 172.21.15.3:0/1726925638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 msgr2=0x7facf00a9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:20.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facef7fe700 1 --2- 172.21.15.3:0/1726925638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 0x7facf00a9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facef7fe700 1 -- 172.21.15.3:0/1726925638 >> v1:172.21.15.143:6789/0 conn(0x7facf01436d0 legacy=0x7facf00b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facef7fe700 1 -- 172.21.15.3:0/1726925638 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf0144130 con 0x7facf00a8cf0 2022-07-21T01:41:20.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7face67fc700 1 --2- 172.21.15.3:0/1726925638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 0x7facf00a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:20.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facef7fe700 1 --2- 172.21.15.3:0/1726925638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf00a8cf0 0x7facf0143ab0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7face80032f0 tx=0x7face80091d0).ready entity=mon.1 client_cookie=6e2bbfd33bd0253e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:20.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facee7fc700 1 -- 172.21.15.3:0/1726925638 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face800b6c0 con 0x7facf00a8cf0 2022-07-21T01:41:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facee7fc700 1 -- 172.21.15.3:0/1726925638 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face800b820 con 0x7facf00a8cf0 2022-07-21T01:41:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facee7fc700 1 -- 172.21.15.3:0/1726925638 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face800b990 con 0x7facf00a8cf0 2022-07-21T01:41:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 -- 172.21.15.3:0/1726925638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf00a8cf0 msgr2=0x7facf0143ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 --2- 172.21.15.3:0/1726925638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf00a8cf0 0x7facf0143ab0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7face80032f0 tx=0x7face80091d0).stop 2022-07-21T01:41:20.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 -- 172.21.15.3:0/1726925638 shutdown_connections 2022-07-21T01:41:20.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 --2- 172.21.15.3:0/1726925638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf00a8cf0 0x7facf0143ab0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 --2- 172.21.15.3:0/1726925638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 0x7facf00a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 -- 172.21.15.3:0/1726925638 >> 172.21.15.3:0/1726925638 conn(0x7facf00b5570 msgr2=0x7facf00b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:20.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 -- 172.21.15.3:0/1726925638 shutdown_connections 2022-07-21T01:41:20.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 -- 172.21.15.3:0/1726925638 wait complete. 2022-07-21T01:41:20.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 Processor -- start 2022-07-21T01:41:20.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.024+0000 7facf55de700 1 -- start start 2022-07-21T01:41:20.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facf55de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 0x7facf00ad980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facf55de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf00a8cf0 0x7facf015af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facf55de700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facf01436d0 0x7facf00a7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facf55de700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf015a890 con 0x7facf01436d0 2022-07-21T01:41:20.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facf55de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf00adfd0 con 0x7facf00a8cf0 2022-07-21T01:41:20.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facf55de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf00ae270 con 0x7facf005f840 2022-07-21T01:41:20.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facf01436d0 0x7facf00a7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facf01436d0 0x7facf00a7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49478/0 (socket says 172.21.15.3:49478) 2022-07-21T01:41:20.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facef7fe700 1 -- 172.21.15.3:0/1667687199 learned_addr learned my addr 172.21.15.3:0/1667687199 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:20.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7faceeffd700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 0x7facf00ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7face67fc700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf00a8cf0 0x7facf015af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facef7fe700 1 -- 172.21.15.3:0/1667687199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 msgr2=0x7facf00ad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facef7fe700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 0x7facf00ad980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facef7fe700 1 -- 172.21.15.3:0/1667687199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf00a8cf0 msgr2=0x7facf015af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facef7fe700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf00a8cf0 0x7facf015af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facef7fe700 1 -- 172.21.15.3:0/1667687199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd800b180 con 0x7facf01436d0 2022-07-21T01:41:20.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facef7fe700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facf01436d0 0x7facf00a7250 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7face80032f0 tx=0x7face8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:20.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facecff9700 1 -- 172.21.15.3:0/1667687199 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face8011d70 con 0x7facf01436d0 2022-07-21T01:41:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facecff9700 1 -- 172.21.15.3:0/1667687199 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face8021440 con 0x7facf01436d0 2022-07-21T01:41:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facecff9700 1 -- 172.21.15.3:0/1667687199 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face80217a0 con 0x7facf01436d0 2022-07-21T01:41:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face8009470 con 0x7facf01436d0 2022-07-21T01:41:20.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.028+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf015aa80 con 0x7facf01436d0 2022-07-21T01:41:20.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.032+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf00a8ae0 con 0x7facf01436d0 2022-07-21T01:41:20.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.032+0000 7facecff9700 1 -- 172.21.15.3:0/1667687199 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7face8029070 con 0x7facf01436d0 2022-07-21T01:41:20.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.032+0000 7facecff9700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7facd4053aa0 0x7facd4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.032+0000 7facecff9700 1 -- 172.21.15.3:0/1667687199 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5686+0+0 (secure 0 0 0) 0x7face806cca0 con 0x7facf01436d0 2022-07-21T01:41:20.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.032+0000 7faceeffd700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7facd4053aa0 0x7facd4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.032+0000 7faceeffd700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7facd4053aa0 0x7facd4055f60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7face000d000 tx=0x7face0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:20.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.040+0000 7facecff9700 1 -- 172.21.15.3:0/1667687199 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7face8071070 con 0x7facf01436d0 2022-07-21T01:41:20.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.208+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7facf01443d0 con 0x7facf01436d0 2022-07-21T01:41:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.636+0000 7facecff9700 1 -- 172.21.15.3:0/1667687199 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v357) v1 ==== 127+0+0 (secure 0 0 0) 0x7face8015070 con 0x7facf01436d0 2022-07-21T01:41:20.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7facd4053aa0 msgr2=0x7facd4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7facd4053aa0 0x7facd4055f60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7face000d000 tx=0x7face0009040).stop 2022-07-21T01:41:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facf01436d0 msgr2=0x7facf00a7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facf01436d0 0x7facf00a7250 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7face80032f0 tx=0x7face8016000).stop 2022-07-21T01:41:20.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 shutdown_connections 2022-07-21T01:41:20.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facf01436d0 0x7facf00a7250 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7facd4053aa0 0x7facd4055f60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facf00a8cf0 0x7facf015af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 --2- 172.21.15.3:0/1667687199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facf005f840 0x7facf00ad980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 >> 172.21.15.3:0/1667687199 conn(0x7facf00b5570 msgr2=0x7facf00b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:20.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 shutdown_connections 2022-07-21T01:41:20.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.640+0000 7facf55de700 1 -- 172.21.15.3:0/1667687199 wait complete. 2022-07-21T01:41:20.660 INFO:tasks.cephfs_test_runner:test_filtered_df (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-21T01:41:20.661 INFO:tasks.cephfs_test_runner:Starting test: test_fs_new (tasks.cephfs.test_misc.TestMisc) 2022-07-21T01:41:20.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-07-21T01:41:20.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.768+0000 7f75e2e05700 1 Processor -- start 2022-07-21T01:41:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.768+0000 7f75e2e05700 1 -- start start 2022-07-21T01:41:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f75dc060330 0x7f75dc0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc05c6f0 0x7f75dc0a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc0aa630 con 0x7f75dc0af550 2022-07-21T01:41:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75dc0b15e0 con 0x7f75dc060330 2022-07-21T01:41:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0af930 con 0x7f75dc05c6f0 2022-07-21T01:41:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e1602700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f75dc060330 0x7f75dc0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e1602700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f75dc060330 0x7f75dc0b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49486/0 (socket says 172.21.15.3:49486) 2022-07-21T01:41:20.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e1602700 1 -- 172.21.15.3:0/4204058140 learned_addr learned my addr 172.21.15.3:0/4204058140 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:20.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2604700 1 --2- 172.21.15.3:0/4204058140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc05c6f0 0x7f75dc0a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e1602700 1 -- 172.21.15.3:0/4204058140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc05c6f0 msgr2=0x7f75dc0a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e1602700 1 --2- 172.21.15.3:0/4204058140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc05c6f0 0x7f75dc0a8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e1602700 1 -- 172.21.15.3:0/4204058140 >> v1:172.21.15.143:6789/0 conn(0x7f75dc0af550 legacy=0x7f75dc1445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e1602700 1 -- 172.21.15.3:0/4204058140 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc05fd10 con 0x7f75dc060330 2022-07-21T01:41:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e0e01700 1 -- 172.21.15.3:0/4204058140 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832013230 0 0) 0x7f75dc0aa630 con 0x7f75dc0af550 2022-07-21T01:41:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e1602700 1 --2- 172.21.15.3:0/4204058140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc060330 0x7f75dc0b0e40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f75d8002f70 tx=0x7f75d8006eb0).ready entity=mon.0 client_cookie=3ba5468550dd7392 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e0e01700 1 -- 172.21.15.3:0/4204058140 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75d800d570 con 0x7f75dc060330 2022-07-21T01:41:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e0e01700 1 -- 172.21.15.3:0/4204058140 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d8000e90 con 0x7f75dc060330 2022-07-21T01:41:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e0e01700 1 -- 172.21.15.3:0/4204058140 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75d800d8c0 con 0x7f75dc060330 2022-07-21T01:41:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 -- 172.21.15.3:0/4204058140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc060330 msgr2=0x7f75dc0b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 --2- 172.21.15.3:0/4204058140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc060330 0x7f75dc0b0e40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f75d8002f70 tx=0x7f75d8006eb0).stop 2022-07-21T01:41:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 -- 172.21.15.3:0/4204058140 shutdown_connections 2022-07-21T01:41:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 --2- 172.21.15.3:0/4204058140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc05c6f0 0x7f75dc0a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 --2- 172.21.15.3:0/4204058140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc060330 0x7f75dc0b0e40 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 -- 172.21.15.3:0/4204058140 >> 172.21.15.3:0/4204058140 conn(0x7f75dc0612d0 msgr2=0x7f75dc062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 -- 172.21.15.3:0/4204058140 shutdown_connections 2022-07-21T01:41:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.772+0000 7f75e2e05700 1 -- 172.21.15.3:0/4204058140 wait complete. 2022-07-21T01:41:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.776+0000 7f75e2e05700 1 Processor -- start 2022-07-21T01:41:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.776+0000 7f75e2e05700 1 -- start start 2022-07-21T01:41:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2e05700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc05c6f0 0x7f75dc144c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2e05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc060330 0x7f75dc14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2e05700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75dc0af550 0x7f75dc1504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2e05700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc144070 con 0x7f75dc05c6f0 2022-07-21T01:41:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2e05700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc144400 con 0x7f75dc0af550 2022-07-21T01:41:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2e05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc143180 con 0x7f75dc060330 2022-07-21T01:41:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1e03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc05c6f0 0x7f75dc144c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1e03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc05c6f0 0x7f75dc144c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49488/0 (socket says 172.21.15.3:49488) 2022-07-21T01:41:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1e03700 1 -- 172.21.15.3:0/2340099116 learned_addr learned my addr 172.21.15.3:0/2340099116 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1602700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc060330 0x7f75dc14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2604700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75dc0af550 0x7f75dc1504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1e03700 1 -- 172.21.15.3:0/2340099116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc060330 msgr2=0x7f75dc14ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1e03700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc060330 0x7f75dc14ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1e03700 1 -- 172.21.15.3:0/2340099116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75dc0af550 msgr2=0x7f75dc1504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1e03700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75dc0af550 0x7f75dc1504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1e03700 1 -- 172.21.15.3:0/2340099116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d800b1e0 con 0x7f75dc05c6f0 2022-07-21T01:41:20.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e1e03700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc05c6f0 0x7f75dc144c10 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7f75d0000ed0 tx=0x7f75d0007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:20.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75ceffd700 1 -- 172.21.15.3:0/2340099116 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75d000ce80 con 0x7f75dc05c6f0 2022-07-21T01:41:20.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75ceffd700 1 -- 172.21.15.3:0/2340099116 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d0013450 con 0x7f75dc05c6f0 2022-07-21T01:41:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc05f840 con 0x7f75dc05c6f0 2022-07-21T01:41:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75ceffd700 1 -- 172.21.15.3:0/2340099116 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75d00137b0 con 0x7f75dc05c6f0 2022-07-21T01:41:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc1509c0 con 0x7f75dc05c6f0 2022-07-21T01:41:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75dc05fa70 con 0x7f75dc05c6f0 2022-07-21T01:41:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.780+0000 7f75ceffd700 1 -- 172.21.15.3:0/2340099116 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f75d001a060 con 0x7f75dc05c6f0 2022-07-21T01:41:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.784+0000 7f75ceffd700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f75bc053a10 0x7f75bc055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.784+0000 7f75ceffd700 1 -- 172.21.15.3:0/2340099116 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f75d006b6e0 con 0x7f75dc05c6f0 2022-07-21T01:41:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.784+0000 7f75e1602700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f75bc053a10 0x7f75bc055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.784+0000 7f75e1602700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f75bc053a10 0x7f75bc055ed0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f75d801a000 tx=0x7f75d8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.788+0000 7f75ceffd700 1 -- 172.21.15.3:0/2340099116 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75d0042670 con 0x7f75dc05c6f0 2022-07-21T01:41:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:20.956+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7f75dc0574e0 con 0x7f75dc05c6f0 2022-07-21T01:41:21.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.640+0000 7f75ceffd700 1 -- 172.21.15.3:0/2340099116 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v358) v1 ==== 125+0+0 (secure 0 0 0) 0x7f75d0018070 con 0x7f75dc05c6f0 2022-07-21T01:41:21.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f75bc053a10 msgr2=0x7f75bc055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:21.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f75bc053a10 0x7f75bc055ed0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f75d801a000 tx=0x7f75d8015000).stop 2022-07-21T01:41:21.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc05c6f0 msgr2=0x7f75dc144c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:21.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc05c6f0 0x7f75dc144c10 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7f75d0000ed0 tx=0x7f75d0007c20).stop 2022-07-21T01:41:21.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 shutdown_connections 2022-07-21T01:41:21.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f75bc053a10 0x7f75bc055ed0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75dc0af550 0x7f75dc1504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75dc060330 0x7f75dc14ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 --2- 172.21.15.3:0/2340099116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75dc05c6f0 0x7f75dc144c10 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 >> 172.21.15.3:0/2340099116 conn(0x7f75dc0612d0 msgr2=0x7f75dc0b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:21.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 shutdown_connections 2022-07-21T01:41:21.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.644+0000 7f75e2e05700 1 -- 172.21.15.3:0/2340099116 wait complete. 2022-07-21T01:41:21.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:41:21.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.780+0000 7f8f69654700 1 Processor -- start 2022-07-21T01:41:21.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.780+0000 7f8f69654700 1 -- start start 2022-07-21T01:41:21.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.780+0000 7f8f69654700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8f64060330 0x7f8f640b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:21.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.780+0000 7f8f69654700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640aa630 con 0x7f8f6405c6f0 2022-07-21T01:41:21.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.780+0000 7f8f69654700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6405fd10 con 0x7f8f640af550 2022-07-21T01:41:21.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.780+0000 7f8f69654700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f640b15e0 con 0x7f8f64060330 2022-07-21T01:41:21.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.780+0000 7f8f68e53700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8f6405c6f0 0x7f8f640a8d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57780/0 (socket says 172.21.15.3:57780) 2022-07-21T01:41:21.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.780+0000 7f8f68e53700 1 -- 172.21.15.3:0/3629437190 learned_addr learned my addr 172.21.15.3:0/3629437190 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:21.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f637fe700 1 --2- 172.21.15.3:0/3629437190 >> v2:172.21.15.143:3300/0 conn(0x7f8f64060330 0x7f8f640b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:21.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f62ffd700 1 -- 172.21.15.3:0/3629437190 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862974691 0 0) 0x7f8f640aa630 con 0x7f8f6405c6f0 2022-07-21T01:41:21.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f62ffd700 1 -- 172.21.15.3:0/3629437190 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f44003260 con 0x7f8f6405c6f0 2022-07-21T01:41:21.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f62ffd700 1 -- 172.21.15.3:0/3629437190 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881126250 0 0) 0x7f8f6405fd10 con 0x7f8f640af550 2022-07-21T01:41:21.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f62ffd700 1 -- 172.21.15.3:0/3629437190 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f640aa630 con 0x7f8f640af550 2022-07-21T01:41:21.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f62ffd700 1 -- 172.21.15.3:0/3629437190 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2931612169 0 0) 0x7f8f44003260 con 0x7f8f6405c6f0 2022-07-21T01:41:21.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f637fe700 1 -- 172.21.15.3:0/3629437190 >> v1:172.21.15.143:6789/0 conn(0x7f8f640af550 legacy=0x7f8f641445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:21.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f637fe700 1 -- 172.21.15.3:0/3629437190 >> v1:172.21.15.3:6789/0 conn(0x7f8f6405c6f0 legacy=0x7f8f640a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:21.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f637fe700 1 -- 172.21.15.3:0/3629437190 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6405f840 con 0x7f8f64060330 2022-07-21T01:41:21.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f637fe700 1 --2- 172.21.15.3:0/3629437190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f64060330 0x7f8f640b0e40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f8f5c001490 tx=0x7f8f54006df0).ready entity=mon.1 client_cookie=278d49abbb8ab9f1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:21.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f62ffd700 1 -- 172.21.15.3:0/3629437190 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f5c0019e0 con 0x7f8f64060330 2022-07-21T01:41:21.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f62ffd700 1 -- 172.21.15.3:0/3629437190 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54000e90 con 0x7f8f64060330 2022-07-21T01:41:21.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f62ffd700 1 -- 172.21.15.3:0/3629437190 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f54012460 con 0x7f8f64060330 2022-07-21T01:41:21.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f69654700 1 -- 172.21.15.3:0/3629437190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f64060330 msgr2=0x7f8f640b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f69654700 1 --2- 172.21.15.3:0/3629437190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f64060330 0x7f8f640b0e40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f8f5c001490 tx=0x7f8f54006df0).stop 2022-07-21T01:41:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f69654700 1 -- 172.21.15.3:0/3629437190 shutdown_connections 2022-07-21T01:41:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f69654700 1 --2- 172.21.15.3:0/3629437190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f64060330 0x7f8f640b0e40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f69654700 1 -- 172.21.15.3:0/3629437190 >> 172.21.15.3:0/3629437190 conn(0x7f8f640612d0 msgr2=0x7f8f64062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f69654700 1 -- 172.21.15.3:0/3629437190 shutdown_connections 2022-07-21T01:41:21.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.784+0000 7f8f69654700 1 -- 172.21.15.3:0/3629437190 wait complete. 2022-07-21T01:41:21.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.788+0000 7f8f69654700 1 Processor -- start 2022-07-21T01:41:21.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.788+0000 7f8f69654700 1 -- start start 2022-07-21T01:41:21.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.788+0000 7f8f69654700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6405c6f0 0x7f8f64063ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:21.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.788+0000 7f8f69654700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f64060330 0x7f8f64142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:21.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.788+0000 7f8f69654700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f640af550 0x7f8f64066b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:21.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.788+0000 7f8f69654700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64065f80 con 0x7f8f640af550 2022-07-21T01:41:21.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.788+0000 7f8f69654700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f641540e0 con 0x7f8f6405c6f0 2022-07-21T01:41:21.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.788+0000 7f8f69654700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64154380 con 0x7f8f64060330 2022-07-21T01:41:21.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f68e53700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f640af550 0x7f8f64066b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:21.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f68e53700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f640af550 0x7f8f64066b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49506/0 (socket says 172.21.15.3:49506) 2022-07-21T01:41:21.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f68e53700 1 -- 172.21.15.3:0/101680717 learned_addr learned my addr 172.21.15.3:0/101680717 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:21.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f637fe700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f64060330 0x7f8f64142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:21.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f63fff700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6405c6f0 0x7f8f64063ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:21.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f68e53700 1 -- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f64060330 msgr2=0x7f8f64142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:21.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f68e53700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f64060330 0x7f8f64142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f68e53700 1 -- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6405c6f0 msgr2=0x7f8f64063ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:21.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f68e53700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6405c6f0 0x7f8f64063ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f68e53700 1 -- 172.21.15.3:0/101680717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5400b180 con 0x7f8f640af550 2022-07-21T01:41:21.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f637fe700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f64060330 0x7f8f64142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:21.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f63fff700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6405c6f0 0x7f8f64063ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:21.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f68e53700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f640af550 0x7f8f64066b60 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f8f5c001360 tx=0x7f8f5c001c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:21.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f617fa700 1 -- 172.21.15.3:0/101680717 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f5c004ab0 con 0x7f8f640af550 2022-07-21T01:41:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f617fa700 1 -- 172.21.15.3:0/101680717 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c004c10 con 0x7f8f640af550 2022-07-21T01:41:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f64065b50 con 0x7f8f640af550 2022-07-21T01:41:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f617fa700 1 -- 172.21.15.3:0/101680717 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f5c007170 con 0x7f8f640af550 2022-07-21T01:41:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f64142b10 con 0x7f8f640af550 2022-07-21T01:41:21.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.792+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f640574e0 con 0x7f8f640af550 2022-07-21T01:41:21.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.796+0000 7f8f617fa700 1 -- 172.21.15.3:0/101680717 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8f5c01b060 con 0x7f8f640af550 2022-07-21T01:41:21.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.796+0000 7f8f617fa700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8f48053aa0 0x7f8f48055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.796+0000 7f8f617fa700 1 -- 172.21.15.3:0/101680717 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f8f5c06ba80 con 0x7f8f640af550 2022-07-21T01:41:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.796+0000 7f8f63fff700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8f48053aa0 0x7f8f48055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.796+0000 7f8f63fff700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8f48053aa0 0x7f8f48055f60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f8f4c00b000 tx=0x7f8f4c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:21.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.800+0000 7f8f617fa700 1 -- 172.21.15.3:0/101680717 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f5c042b20 con 0x7f8f640af550 2022-07-21T01:41:21.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.932+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8f64069a70 con 0x7f8f640af550 2022-07-21T01:41:21.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.940+0000 7f8f617fa700 1 -- 172.21.15.3:0/101680717 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8f5c019070 con 0x7f8f640af550 2022-07-21T01:41:21.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.940+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8f48053aa0 msgr2=0x7f8f48055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:21.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.940+0000 7f8f69654700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8f48053aa0 0x7f8f48055f60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f8f4c00b000 tx=0x7f8f4c009040).stop 2022-07-21T01:41:21.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.940+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f640af550 msgr2=0x7f8f64066b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:21.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.940+0000 7f8f69654700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f640af550 0x7f8f64066b60 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f8f5c001360 tx=0x7f8f5c001c50).stop 2022-07-21T01:41:21.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.944+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 shutdown_connections 2022-07-21T01:41:21.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.944+0000 7f8f69654700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f640af550 0x7f8f64066b60 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.944+0000 7f8f69654700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f64060330 0x7f8f64142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.944+0000 7f8f69654700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8f48053aa0 0x7f8f48055f60 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.944+0000 7f8f69654700 1 --2- 172.21.15.3:0/101680717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6405c6f0 0x7f8f64063ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:21.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.944+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 >> 172.21.15.3:0/101680717 conn(0x7f8f640612d0 msgr2=0x7f8f640b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:21.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.944+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 shutdown_connections 2022-07-21T01:41:21.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:21.944+0000 7f8f69654700 1 -- 172.21.15.3:0/101680717 wait complete. 2022-07-21T01:41:21.956 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:41:21.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:21.976 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:21.999 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:22.026 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:22.048 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:41:22.048 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:22.049 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:22.049 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:22.057 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:41:22.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:41:22.059 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:41:22.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:22.060 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:22.060 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:22.068 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:41:22.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:41:22.069 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:41:22.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:41:22.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd193b48700 1 Processor -- start 2022-07-21T01:41:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd193b48700 1 -- start start 2022-07-21T01:41:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd193b48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 0x7fd18c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd193b48700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd18c0a7250 0x7fd18c1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd193b48700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd18c0b04e0 con 0x7fd18c148530 2022-07-21T01:41:22.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd193b48700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd18c0a9250 con 0x7fd18c0a7250 2022-07-21T01:41:22.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd193b48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0b0780 con 0x7fd18c0b1f40 2022-07-21T01:41:22.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1920e5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd18c0a7250 0x7fd18c1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1920e5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd18c0a7250 0x7fd18c1448b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49514/0 (socket says 172.21.15.3:49514) 2022-07-21T01:41:22.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1920e5700 1 -- 172.21.15.3:0/1626556941 learned_addr learned my addr 172.21.15.3:0/1626556941 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:22.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1918e4700 1 --2- 172.21.15.3:0/1626556941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 0x7fd18c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1920e5700 1 -- 172.21.15.3:0/1626556941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 msgr2=0x7fd18c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1920e5700 1 --2- 172.21.15.3:0/1626556941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 0x7fd18c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1920e5700 1 -- 172.21.15.3:0/1626556941 >> v1:172.21.15.143:6789/0 conn(0x7fd18c148530 legacy=0x7fd18c0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1920e5700 1 -- 172.21.15.3:0/1626556941 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c0b0a20 con 0x7fd18c0a7250 2022-07-21T01:41:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1920e5700 1 --2- 172.21.15.3:0/1626556941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c0a7250 0x7fd18c1448b0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7fd1840032f0 tx=0x7fd184009290).ready entity=mon.0 client_cookie=eff1f287ab0f39ed server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1908e2700 1 -- 172.21.15.3:0/1626556941 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18400b8a0 con 0x7fd18c0a7250 2022-07-21T01:41:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1908e2700 1 -- 172.21.15.3:0/1626556941 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18400ba00 con 0x7fd18c0a7250 2022-07-21T01:41:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd1908e2700 1 -- 172.21.15.3:0/1626556941 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18400bb70 con 0x7fd18c0a7250 2022-07-21T01:41:22.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd193b48700 1 -- 172.21.15.3:0/1626556941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c0a7250 msgr2=0x7fd18c1448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.180+0000 7fd193b48700 1 --2- 172.21.15.3:0/1626556941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c0a7250 0x7fd18c1448b0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7fd1840032f0 tx=0x7fd184009290).stop 2022-07-21T01:41:22.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.184+0000 7fd193b48700 1 -- 172.21.15.3:0/1626556941 shutdown_connections 2022-07-21T01:41:22.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.184+0000 7fd193b48700 1 --2- 172.21.15.3:0/1626556941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c0a7250 0x7fd18c1448b0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.184+0000 7fd193b48700 1 --2- 172.21.15.3:0/1626556941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 0x7fd18c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.184+0000 7fd193b48700 1 -- 172.21.15.3:0/1626556941 >> 172.21.15.3:0/1626556941 conn(0x7fd18c05f840 msgr2=0x7fd18c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:22.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.184+0000 7fd193b48700 1 -- 172.21.15.3:0/1626556941 shutdown_connections 2022-07-21T01:41:22.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.184+0000 7fd193b48700 1 -- 172.21.15.3:0/1626556941 wait complete. 2022-07-21T01:41:22.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.184+0000 7fd193b48700 1 Processor -- start 2022-07-21T01:41:22.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.184+0000 7fd193b48700 1 -- start start 2022-07-21T01:41:22.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd193b48700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd18c0a7250 0x7fd18c14e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd193b48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 0x7fd18c15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd193b48700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c148530 0x7fd18c15b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd193b48700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c13bb20 con 0x7fd18c148530 2022-07-21T01:41:22.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd193b48700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c15a600 con 0x7fd18c0a7250 2022-07-21T01:41:22.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd193b48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c13ca00 con 0x7fd18c0b1f40 2022-07-21T01:41:22.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1920e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c148530 0x7fd18c15b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1918e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd18c0a7250 0x7fd18c14e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1920e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c148530 0x7fd18c15b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49520/0 (socket says 172.21.15.3:49520) 2022-07-21T01:41:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1920e5700 1 -- 172.21.15.3:0/2541837667 learned_addr learned my addr 172.21.15.3:0/2541837667 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1910e3700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 0x7fd18c15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1920e5700 1 -- 172.21.15.3:0/2541837667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 msgr2=0x7fd18c15abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1920e5700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 0x7fd18c15abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1920e5700 1 -- 172.21.15.3:0/2541837667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd18c0a7250 msgr2=0x7fd18c14e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1920e5700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd18c0a7250 0x7fd18c14e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1920e5700 1 -- 172.21.15.3:0/2541837667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd184009530 con 0x7fd18c148530 2022-07-21T01:41:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd1920e5700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c148530 0x7fd18c15b0e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7fd1840032f0 tx=0x7fd184016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd17affd700 1 -- 172.21.15.3:0/2541837667 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18400b8a0 con 0x7fd18c148530 2022-07-21T01:41:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd17affd700 1 -- 172.21.15.3:0/2541837667 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd184021630 con 0x7fd18c148530 2022-07-21T01:41:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18c13cca0 con 0x7fd18c148530 2022-07-21T01:41:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd17affd700 1 -- 172.21.15.3:0/2541837667 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1840219d0 con 0x7fd18c148530 2022-07-21T01:41:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18c13d1e0 con 0x7fd18c148530 2022-07-21T01:41:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18c13d700 con 0x7fd18c148530 2022-07-21T01:41:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.188+0000 7fd17affd700 1 -- 172.21.15.3:0/2541837667 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd184027020 con 0x7fd18c148530 2022-07-21T01:41:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.192+0000 7fd17affd700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd16c053a70 0x7fd16c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.192+0000 7fd17affd700 1 -- 172.21.15.3:0/2541837667 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fd18406d530 con 0x7fd18c148530 2022-07-21T01:41:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.192+0000 7fd1918e4700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd16c053a70 0x7fd16c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.192+0000 7fd1918e4700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd16c053a70 0x7fd16c055f30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fd17c00c000 tx=0x7fd17c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.196+0000 7fd17affd700 1 -- 172.21.15.3:0/2541837667 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd18c13d700 con 0x7fd18c148530 2022-07-21T01:41:22.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.316+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd18c13d700 con 0x7fd18c148530 2022-07-21T01:41:22.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd17affd700 1 -- 172.21.15.3:0/2541837667 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v114) v1 ==== 81+0+29225 (secure 0 0 0) 0x7fd18c13d700 con 0x7fd18c148530 2022-07-21T01:41:22.323 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:22.323 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:41:22.324 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 114, 2022-07-21T01:41:22.324 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:41:22.324 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:41:22.325 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:41:19.612222+0000", 2022-07-21T01:41:22.325 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:41:22.325 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:41:22.325 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:41:22.326 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:41:22.326 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:41:22.326 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:41:22.327 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:41:22.327 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:41:22.327 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:41:22.327 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:22.328 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:41:22.328 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:41:22.328 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:41:22.328 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:41:22.329 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:41:22.329 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 20, 2022-07-21T01:41:22.329 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:41:22.330 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:41:22.330 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:41:22.330 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:41:22.331 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:41:22.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.331 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:41:22.331 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:41:22.332 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:41:22.332 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:41:22.332 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:41:22.332 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:22.333 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:22.333 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:22.338 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:22.339 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:22.339 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:22.340 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:41:22.340 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:41:22.341 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:41:22.341 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:41:22.342 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:41:22.342 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:22.342 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:41:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:41:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:41:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:41:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.361 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:41:22.362 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:41:22.362 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:41:22.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:41:22.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:41:22.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:22.364 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:22.365 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:22.365 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:22.366 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:22.367 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:22.367 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:41:22.367 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:41:22.368 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:41:22.368 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:41:22.369 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:41:22.370 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:22.370 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:22.370 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:22.370 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:22.371 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:22.371 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:22.372 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:22.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.372 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:41:22.373 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:22.373 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:22.374 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:22.374 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:22.375 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:22.375 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:22.376 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:22.376 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:22.377 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:22.377 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:22.378 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:22.379 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:22.379 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:22.380 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:22.380 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:22.381 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:22.381 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:22.382 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:22.382 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:22.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:22.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:22.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:22.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:22.386 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:22.388 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:22.388 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:22.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.388 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:22.389 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:22.389 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:22.390 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:22.390 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:22.390 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:22.390 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:22.391 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:22.391 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:22.391 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:22.391 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:41:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:41:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.393 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:22.393 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:41:22.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.393 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:41:22.394 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:41:22.394 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:22.394 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:22.394 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:22.395 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:22.395 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:22.395 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:22.395 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:41:22.395 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.397 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:41:22.397 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:22.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.398 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.398 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:41:22.398 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:41:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.402 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:41:22.402 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:22.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.403 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.403 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:22.403 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.404 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.404 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:41:22.405 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:22.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.405 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:41:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:22.408 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.409 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:41:22.410 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:22.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.412 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.412 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:41:22.412 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:22.413 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.413 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.414 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:41:22.414 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:41:22.414 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:41:22.414 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:41:22.415 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:22.415 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:22.415 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:22.415 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.416 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:41:22.416 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:41:22.417 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:22.417 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:22.417 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:22.417 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:22.417 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:22.418 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:22.418 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:22.418 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:22.419 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:22.419 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:22.419 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:22.419 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.420 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.420 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:41:22.421 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:22.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.422 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:41:22.422 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:22.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.422 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.423 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:22.423 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.424 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:41:22.424 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:22.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.425 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.425 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:41:22.425 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:22.426 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.426 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.426 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:22.427 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.427 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.427 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:41:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:41:22.429 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:22.429 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.429 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.430 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:22.430 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.430 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.430 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:41:22.431 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:22.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.431 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:41:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.433 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:41:22.433 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:41:22.433 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:41:22.433 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:41:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.435 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:41:22.435 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:41:22.435 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:22.437 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:41:22.437 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 107, 2022-07-21T01:41:22.437 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:22.437 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:22.438 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:22.438 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.438 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.438 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:41:22.439 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:22.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.439 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:41:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:41:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:41:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:41:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:41:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:41:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:41:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:41:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:41:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:41:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:41:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.452 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:41:22.452 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:41:22.452 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:41:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:41:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.457 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:41:22.457 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:22.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.458 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:22.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:41:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:41:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:22.462 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.462 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:41:22.463 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:22.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.463 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.463 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:41:22.464 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:22.464 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.464 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:41:22.466 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:22.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.466 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.467 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:41:22.467 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:22.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.467 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.468 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:41:22.468 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:41:22.468 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:41:22.468 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:41:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:41:22.470 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:41:22.470 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:22.470 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:22.471 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:22.471 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:22.471 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:22.471 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 107, 2022-07-21T01:41:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:22.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.473 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:41:22.473 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:22.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.474 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:41:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:41:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:41:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:41:22.480 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:22.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.480 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.481 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:41:22.481 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:22.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.481 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.482 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:22.482 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.482 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:41:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.484 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:41:22.484 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:22.484 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.484 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:41:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:41:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:41:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:41:22.486 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:22.486 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:22.486 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:22.486 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.487 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:41:22.487 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:41:22.487 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:22.488 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:22.488 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:22.488 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:22.488 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 107, 2022-07-21T01:41:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:22.490 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:22.490 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:41:22.491 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:22.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.492 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:41:22.492 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:22.492 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.492 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.493 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:22.493 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:41:22.494 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:22.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.495 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.495 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:41:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:22.497 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.497 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.497 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:41:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:41:22.499 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:22.499 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.499 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:41:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:41:22.502 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:22.502 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.502 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.503 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:41:22.503 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:41:22.503 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:41:22.503 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:41:22.503 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:22.504 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:22.504 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:22.504 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.505 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:41:22.505 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:41:22.505 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:22.505 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:22.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:41:22.509 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:22.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.509 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:41:22.510 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:22.510 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.510 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.511 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:22.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.511 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:41:22.512 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:22.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.512 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.513 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:41:22.513 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:22.513 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.513 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.514 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:22.514 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.514 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.515 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:41:22.515 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:22.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.515 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:41:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.517 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:22.517 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.517 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:41:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.519 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:41:22.519 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:22.519 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.519 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.520 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:41:22.520 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:41:22.520 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:41:22.521 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:41:22.521 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:22.521 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:22.521 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:22.522 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.522 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:41:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:41:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 108, 2022-07-21T01:41:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:22.525 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:22.525 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:22.525 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:41:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:41:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.539 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:22.539 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.539 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.540 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:41:22.540 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:22.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:41:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.542 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:22.542 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.543 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:41:22.543 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:22.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.544 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.544 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:41:22.545 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:22.545 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.545 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.546 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:22.546 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:22.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.546 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:22.546 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:41:22.547 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:22.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.547 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:22.548 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:41:22.548 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:22.548 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.548 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.549 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:41:22.549 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:41:22.549 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:41:22.549 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:41:22.549 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:22.550 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:22.550 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:22.550 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:22.550 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.551 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:22.551 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:41:22.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.551 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:41:22.551 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:22.552 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:22.552 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:22.552 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:22.552 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:22.553 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:41:22.553 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:22.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.553 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:41:22.554 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:22.554 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:22.554 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:22.554 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:22.555 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:22.555 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:41:22.555 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:22.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.556 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:41:22.556 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:22.556 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:22.556 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:22.557 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:22.557 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:22.557 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:41:22.557 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:22.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.558 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:41:22.558 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:22.558 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:22.559 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:22.559 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:22.559 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:22.559 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:41:22.559 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:22.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.560 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:41:22.560 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:22.561 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:22.561 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:22.561 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:22.561 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:22.561 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:41:22.562 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:22.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.562 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:41:22.563 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:22.563 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:22.563 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:22.563 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:22.564 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:22.564 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:41:22.564 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:22.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.565 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:41:22.565 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:22.565 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:22.565 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:22.566 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:22.566 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:22.566 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:41:22.566 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:22.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:22.567 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:41:22.567 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:22.567 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:22.568 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:22.568 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:22.568 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:22.568 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:41:22.568 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:22.569 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.569 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:22.569 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:41:22.569 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:41:22.570 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:41:22.570 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:41:22.570 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:41:22.570 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6837/151311757": "2022-07-22T01:41:16.168279+0000", 2022-07-21T01:41:22.571 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6836/151311757": "2022-07-22T01:41:16.168279+0000" 2022-07-21T01:41:22.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.571 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:41:22.571 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:41:22.571 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:41:22.572 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:41:22.572 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:41:22.572 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:41:22.572 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:41:22.572 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:41:22.573 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:41:22.573 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:41:22.573 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:41:22.573 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:41:22.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.574 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:41:22.574 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:41:22.574 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:41:22.575 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:41:22.575 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:41:22.575 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:41:22.575 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:41:22.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:22.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:22.576 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:41:22.576 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:41:22.576 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:41:22.577 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:41:22.577 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:41:22.577 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:41:22.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd16c053a70 msgr2=0x7fd16c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd16c053a70 0x7fd16c055f30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fd17c00c000 tx=0x7fd17c008040).stop 2022-07-21T01:41:22.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c148530 msgr2=0x7fd18c15b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c148530 0x7fd18c15b0e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7fd1840032f0 tx=0x7fd184016000).stop 2022-07-21T01:41:22.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 shutdown_connections 2022-07-21T01:41:22.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd18c148530 0x7fd18c15b0e0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd18c0b1f40 0x7fd18c15abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd16c053a70 0x7fd16c055f30 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 --2- 172.21.15.3:0/2541837667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd18c0a7250 0x7fd18c14e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 >> 172.21.15.3:0/2541837667 conn(0x7fd18c05f840 msgr2=0x7fd18c0aec90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:22.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 shutdown_connections 2022-07-21T01:41:22.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.320+0000 7fd193b48700 1 -- 172.21.15.3:0/2541837667 wait complete. 2022-07-21T01:41:22.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:22.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644dfe0700 1 Processor -- start 2022-07-21T01:41:22.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644dfe0700 1 -- start start 2022-07-21T01:41:22.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644dfe0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480614b0 0x7f6448061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644dfe0700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64480aee80 con 0x7f64481405d0 2022-07-21T01:41:22.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644dfe0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64480b70b0 con 0x7f64480ad750 2022-07-21T01:41:22.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644dfe0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64480af0b0 con 0x7f64480614b0 2022-07-21T01:41:22.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644d7df700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f64481405d0 0x7f64481466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57808/0 (socket says 172.21.15.3:57808) 2022-07-21T01:41:22.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644d7df700 1 -- 172.21.15.3:0/235576990 learned_addr learned my addr 172.21.15.3:0/235576990 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:22.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644cfde700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480614b0 0x7f6448061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206349535 0 0) 0x7f64480aee80 con 0x7f64481405d0 2022-07-21T01:41:22.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f642c003260 con 0x7f64481405d0 2022-07-21T01:41:22.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518749364 0 0) 0x7f64480b70b0 con 0x7f64480ad750 2022-07-21T01:41:22.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64480aee80 con 0x7f64480ad750 2022-07-21T01:41:22.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643832412 0 0) 0x7f642c003260 con 0x7f64481405d0 2022-07-21T01:41:22.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480614b0 msgr2=0x7f6448061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:22.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480614b0 0x7f6448061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 >> v1:172.21.15.143:6789/0 conn(0x7f64480ad750 legacy=0x7f64480b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64480af350 con 0x7f64481405d0 2022-07-21T01:41:22.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f64400019e0 con 0x7f64481405d0 2022-07-21T01:41:22.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644cfde700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480614b0 0x7f6448061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:22.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 >> v1:172.21.15.3:6789/0 conn(0x7f64481405d0 legacy=0x7f64481466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480614b0 0x7f642c003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f642c003e50 0x7f642c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f642c0049f0 0x7f642c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64480af0b0 con 0x7f642c003e50 2022-07-21T01:41:22.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c00b800 con 0x7f642c0049f0 2022-07-21T01:41:22.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c00b9c0 con 0x7f64480614b0 2022-07-21T01:41:22.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f6447fff700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f642c003e50 0x7f642c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644cfde700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480614b0 0x7f642c003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.696+0000 7f644d7df700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f642c0049f0 0x7f642c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f6447fff700 1 -- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480614b0 msgr2=0x7f642c003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:22.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f6447fff700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480614b0 0x7f642c003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f6447fff700 1 -- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f642c0049f0 msgr2=0x7f642c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:22.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f6447fff700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f642c0049f0 0x7f642c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f6447fff700 1 -- 172.21.15.3:0/235576990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c00b180 con 0x7f642c003e50 2022-07-21T01:41:22.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f6447fff700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f642c003e50 0x7f642c004500 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f6434001110 tx=0x7f6434007a00).ready entity=mon.0 client_cookie=d31b9373226fe146 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:22.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6434007d10 con 0x7f642c003e50 2022-07-21T01:41:22.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6434007030 con 0x7f642c003e50 2022-07-21T01:41:22.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f64477fe700 1 -- 172.21.15.3:0/235576990 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6434007390 con 0x7f642c003e50 2022-07-21T01:41:22.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f644dfe0700 1 -- 172.21.15.3:0/235576990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f642c003e50 msgr2=0x7f642c004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f644dfe0700 1 --2- 172.21.15.3:0/235576990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f642c003e50 0x7f642c004500 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f6434001110 tx=0x7f6434007a00).stop 2022-07-21T01:41:22.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f644d7df700 1 -- 172.21.15.3:0/235576990 reap_dead start 2022-07-21T01:41:22.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f644dfe0700 1 -- 172.21.15.3:0/235576990 shutdown_connections 2022-07-21T01:41:22.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f644dfe0700 1 -- 172.21.15.3:0/235576990 >> 172.21.15.3:0/235576990 conn(0x7f6448066f70 msgr2=0x7f6448067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:22.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f644dfe0700 1 -- 172.21.15.3:0/235576990 shutdown_connections 2022-07-21T01:41:22.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f644dfe0700 1 -- 172.21.15.3:0/235576990 wait complete. 2022-07-21T01:41:22.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f644dfe0700 1 Processor -- start 2022-07-21T01:41:22.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.700+0000 7f644dfe0700 1 -- start start 2022-07-21T01:41:22.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644dfe0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64480ab000 0x7f6448145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644dfe0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6448152640 0x7f6448145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644dfe0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480aa550 0x7f64480aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644dfe0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6448146100 con 0x7f64480ab000 2022-07-21T01:41:22.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644dfe0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6448152bc0 con 0x7f6448152640 2022-07-21T01:41:22.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644dfe0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6448152e60 con 0x7f64480aa550 2022-07-21T01:41:22.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644cfde700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64480ab000 0x7f6448145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644cfde700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64480ab000 0x7f6448145720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49536/0 (socket says 172.21.15.3:49536) 2022-07-21T01:41:22.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644cfde700 1 -- 172.21.15.3:0/1787784089 learned_addr learned my addr 172.21.15.3:0/1787784089 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:22.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644d7df700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480aa550 0x7f64480aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f6447fff700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6448152640 0x7f6448145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644cfde700 1 -- 172.21.15.3:0/1787784089 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480aa550 msgr2=0x7f64480aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644cfde700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480aa550 0x7f64480aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644cfde700 1 -- 172.21.15.3:0/1787784089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6448152640 msgr2=0x7f6448145c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644cfde700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6448152640 0x7f6448145c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644cfde700 1 -- 172.21.15.3:0/1787784089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6448153100 con 0x7f64480ab000 2022-07-21T01:41:22.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644cfde700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64480ab000 0x7f6448145720 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f643c00b000 tx=0x7f643c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:22.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f6445ffb700 1 -- 172.21.15.3:0/1787784089 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f643c00f940 con 0x7f64480ab000 2022-07-21T01:41:22.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f6445ffb700 1 -- 172.21.15.3:0/1787784089 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c00faa0 con 0x7f64480ab000 2022-07-21T01:41:22.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64481533a0 con 0x7f64480ab000 2022-07-21T01:41:22.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f6445ffb700 1 -- 172.21.15.3:0/1787784089 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f643c00fc10 con 0x7f64480ab000 2022-07-21T01:41:22.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64480ab3e0 con 0x7f64480ab000 2022-07-21T01:41:22.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.704+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64480af5f0 con 0x7f64480ab000 2022-07-21T01:41:22.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.708+0000 7f6445ffb700 1 -- 172.21.15.3:0/1787784089 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f643c024070 con 0x7f64480ab000 2022-07-21T01:41:22.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.708+0000 7f6445ffb700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64280539e0 0x7f6428055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:22.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.708+0000 7f6445ffb700 1 -- 172.21.15.3:0/1787784089 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f643c06c620 con 0x7f64480ab000 2022-07-21T01:41:22.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.708+0000 7f6447fff700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64280539e0 0x7f6428055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:22.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.708+0000 7f6447fff700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64280539e0 0x7f6428055ea0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f643401b000 tx=0x7f6434016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:22.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.716+0000 7f6445ffb700 1 -- 172.21.15.3:0/1787784089 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f643c043620 con 0x7f64480ab000 2022-07-21T01:41:22.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.880+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6448153560 con 0x7f64480ab000 2022-07-21T01:41:22.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.880+0000 7f6445ffb700 1 -- 172.21.15.3:0/1787784089 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+1550 (secure 0 0 0) 0x7f643c006980 con 0x7f64480ab000 2022-07-21T01:41:22.884 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:22.885 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":56,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1707710855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1707710855},{"type":"v1","addr":"172.21.15.143:6833","nonce":1707710855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/764850063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":764850063},{"type":"v1","addr":"172.21.15.3:6835","nonce":764850063}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52},{"gid":7524,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/13681416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":13681416},{"type":"v1","addr":"172.21.15.3:6837","nonce":13681416}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":56}],"filesystems":[]} 2022-07-21T01:41:22.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.880+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64280539e0 msgr2=0x7f6428055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.880+0000 7f644dfe0700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64280539e0 0x7f6428055ea0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f643401b000 tx=0x7f6434016000).stop 2022-07-21T01:41:22.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.880+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64480ab000 msgr2=0x7f6448145720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:22.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.880+0000 7f644dfe0700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64480ab000 0x7f6448145720 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f643c00b000 tx=0x7f643c006df0).stop 2022-07-21T01:41:22.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.884+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 shutdown_connections 2022-07-21T01:41:22.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.884+0000 7f644dfe0700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64480aa550 0x7f64480aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.884+0000 7f644dfe0700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f64280539e0 0x7f6428055ea0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.884+0000 7f644dfe0700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6448152640 0x7f6448145c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.884+0000 7f644dfe0700 1 --2- 172.21.15.3:0/1787784089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64480ab000 0x7f6448145720 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:22.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.884+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 >> 172.21.15.3:0/1787784089 conn(0x7f6448066f70 msgr2=0x7f6448141f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:22.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.884+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 shutdown_connections 2022-07-21T01:41:22.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.884+0000 7f644dfe0700 1 -- 172.21.15.3:0/1787784089 wait complete. 2022-07-21T01:41:22.895 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 56 2022-07-21T01:41:22.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:23.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 Processor -- start 2022-07-21T01:41:23.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 -- start start 2022-07-21T01:41:23.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b4550 0x7f28d40b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f28d40b6af0 0x7f28d412bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d4144240 con 0x7f28d40b6710 2022-07-21T01:41:23.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d4143d00 con 0x7f28d40b6af0 2022-07-21T01:41:23.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4143fa0 con 0x7f28d40b4550 2022-07-21T01:41:23.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f28d40b6af0 0x7f28d412bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28caffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b4550 0x7f28d40b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d37fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f28d40b6710 0x7f28d4061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:59940/0 (socket says 172.21.15.3:59940) 2022-07-21T01:41:23.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d37fe700 1 -- 172.21.15.3:0/3055651555 learned_addr learned my addr 172.21.15.3:0/3055651555 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d3fff700 1 -- 172.21.15.3:0/3055651555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b4550 msgr2=0x7f28d40b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d3fff700 1 --2- 172.21.15.3:0/3055651555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b4550 0x7f28d40b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d3fff700 1 -- 172.21.15.3:0/3055651555 >> v1:172.21.15.143:6789/0 conn(0x7f28d40b6710 legacy=0x7f28d4061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d3fff700 1 -- 172.21.15.3:0/3055651555 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d4061780 con 0x7f28d40b6af0 2022-07-21T01:41:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28caffd700 1 --2- 172.21.15.3:0/3055651555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b4550 0x7f28d40b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:23.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d3fff700 1 --2- 172.21.15.3:0/3055651555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b6af0 0x7f28d412bb90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f28cc0032f0 tx=0x7f28cc0091d0).ready entity=mon.1 client_cookie=aa95052b90bb7d3d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:23.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d2ffd700 1 -- 172.21.15.3:0/3055651555 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28cc00b6c0 con 0x7f28d40b6af0 2022-07-21T01:41:23.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d2ffd700 1 -- 172.21.15.3:0/3055651555 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc00b820 con 0x7f28d40b6af0 2022-07-21T01:41:23.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d2ffd700 1 -- 172.21.15.3:0/3055651555 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28cc00b990 con 0x7f28d40b6af0 2022-07-21T01:41:23.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 -- 172.21.15.3:0/3055651555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b6af0 msgr2=0x7f28d412bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 --2- 172.21.15.3:0/3055651555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b6af0 0x7f28d412bb90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f28cc0032f0 tx=0x7f28cc0091d0).stop 2022-07-21T01:41:23.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 -- 172.21.15.3:0/3055651555 shutdown_connections 2022-07-21T01:41:23.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 --2- 172.21.15.3:0/3055651555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b6af0 0x7f28d412bb90 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 --2- 172.21.15.3:0/3055651555 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b4550 0x7f28d40b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 -- 172.21.15.3:0/3055651555 >> 172.21.15.3:0/3055651555 conn(0x7f28d405c620 msgr2=0x7f28d405ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:23.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 -- 172.21.15.3:0/3055651555 shutdown_connections 2022-07-21T01:41:23.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:22.996+0000 7f28d9fc6700 1 -- 172.21.15.3:0/3055651555 wait complete. 2022-07-21T01:41:23.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 Processor -- start 2022-07-21T01:41:23.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 -- start start 2022-07-21T01:41:23.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b4550 0x7f28d41416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28d40b6710 0x7f28d41403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b6af0 0x7f28d41408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d412b160 con 0x7f28d40b6710 2022-07-21T01:41:23.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40b3a60 con 0x7f28d40b4550 2022-07-21T01:41:23.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40b3d00 con 0x7f28d40b6af0 2022-07-21T01:41:23.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28d40b6710 0x7f28d41403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28d40b6710 0x7f28d41403d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49552/0 (socket says 172.21.15.3:49552) 2022-07-21T01:41:23.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28caffd700 1 -- 172.21.15.3:0/2620614347 learned_addr learned my addr 172.21.15.3:0/2620614347 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:23.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b4550 0x7f28d41416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d3fff700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b6af0 0x7f28d41408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28caffd700 1 -- 172.21.15.3:0/2620614347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b6af0 msgr2=0x7f28d41408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28caffd700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b6af0 0x7f28d41408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28caffd700 1 -- 172.21.15.3:0/2620614347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b4550 msgr2=0x7f28d41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28caffd700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b4550 0x7f28d41416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28caffd700 1 -- 172.21.15.3:0/2620614347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc00b180 con 0x7f28d40b6710 2022-07-21T01:41:23.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d37fe700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b4550 0x7f28d41416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:23.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28caffd700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28d40b6710 0x7f28d41403d0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f28bc00c000 tx=0x7f28bc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:23.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d17fa700 1 -- 172.21.15.3:0/2620614347 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28bc010780 con 0x7f28d40b6710 2022-07-21T01:41:23.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d17fa700 1 -- 172.21.15.3:0/2620614347 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28bc0108e0 con 0x7f28d40b6710 2022-07-21T01:41:23.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28cc009470 con 0x7f28d40b6710 2022-07-21T01:41:23.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d17fa700 1 -- 172.21.15.3:0/2620614347 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28bc010c40 con 0x7f28d40b6710 2022-07-21T01:41:23.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.000+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d4140db0 con 0x7f28d40b6710 2022-07-21T01:41:23.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.004+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d40b38c0 con 0x7f28d40b6710 2022-07-21T01:41:23.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.004+0000 7f28d17fa700 1 -- 172.21.15.3:0/2620614347 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f28bc01c460 con 0x7f28d40b6710 2022-07-21T01:41:23.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.004+0000 7f28d17fa700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28b8053a70 0x7f28b8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.004+0000 7f28d17fa700 1 -- 172.21.15.3:0/2620614347 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f28bc06c3c0 con 0x7f28d40b6710 2022-07-21T01:41:23.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.004+0000 7f28d37fe700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28b8053a70 0x7f28b8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.004+0000 7f28d37fe700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28b8053a70 0x7f28b8055f30 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f28c400d000 tx=0x7f28c4007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:23.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.012+0000 7f28d17fa700 1 -- 172.21.15.3:0/2620614347 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f28d40b38c0 con 0x7f28d40b6710 2022-07-21T01:41:23.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.172+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28d40b38c0 con 0x7f28d40b6710 2022-07-21T01:41:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.172+0000 7f28d17fa700 1 -- 172.21.15.3:0/2620614347 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+1550 (secure 0 0 0) 0x7f28d40b38c0 con 0x7f28d40b6710 2022-07-21T01:41:23.177 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:23.177 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":56,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/1707710855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1707710855},{"type":"v1","addr":"172.21.15.143:6833","nonce":1707710855}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/764850063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":764850063},{"type":"v1","addr":"172.21.15.3:6835","nonce":764850063}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52},{"gid":7524,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/13681416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":13681416},{"type":"v1","addr":"172.21.15.3:6837","nonce":13681416}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":56}],"filesystems":[]} 2022-07-21T01:41:23.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28b8053a70 msgr2=0x7f28b8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28b8053a70 0x7f28b8055f30 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f28c400d000 tx=0x7f28c4007b80).stop 2022-07-21T01:41:23.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28d40b6710 msgr2=0x7f28d41403d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28d40b6710 0x7f28d41403d0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f28bc00c000 tx=0x7f28bc006df0).stop 2022-07-21T01:41:23.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 shutdown_connections 2022-07-21T01:41:23.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d40b6af0 0x7f28d41408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28d40b6710 0x7f28d41403d0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28b8053a70 0x7f28b8055f30 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 --2- 172.21.15.3:0/2620614347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d40b4550 0x7f28d41416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 >> 172.21.15.3:0/2620614347 conn(0x7f28d405c620 msgr2=0x7f28d4127a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:23.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 shutdown_connections 2022-07-21T01:41:23.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.176+0000 7f28d9fc6700 1 -- 172.21.15.3:0/2620614347 wait complete. 2022-07-21T01:41:23.182 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 56 2022-07-21T01:41:23.190 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:41:23.191 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:41:23.191 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:41:23.192 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:41:23.192 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:41:23.192 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:41:23.193 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:41:23.193 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:41:23.193 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:41:23.194 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:41:23.195 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:41:23.195 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:41:23.210 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:41:23.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:41:23.212 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:41:23.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:41:23.214 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:41:23.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:41:23.217 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:41:23.218 DEBUG:teuthology.parallel:result is None 2022-07-21T01:41:23.258 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:41:23.259 DEBUG:teuthology.parallel:result is None 2022-07-21T01:41:23.260 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:41:23.260 DEBUG:teuthology.parallel:result is None 2022-07-21T01:41:23.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:41:23.292 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:41:23.326 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:41:23.337 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:41:23.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f9920e32700 1 Processor -- start 2022-07-21T01:41:23.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f9920e32700 1 -- start start 2022-07-21T01:41:23.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f9920e32700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f991c1437f0 0x7f991c143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f9920e32700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991c0a8cf0 con 0x7f991c0aad90 2022-07-21T01:41:23.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f9920e32700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991c0a8f50 con 0x7f991c13dad0 2022-07-21T01:41:23.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f9920e32700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f991c1441d0 con 0x7f991c1437f0 2022-07-21T01:41:23.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f9919d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f991c0aad90 0x7f991c13d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57870/0 (socket says 172.21.15.3:57870) 2022-07-21T01:41:23.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f9919d9b700 1 -- 172.21.15.3:0/875135821 learned_addr learned my addr 172.21.15.3:0/875135821 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:23.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f991a59c700 1 --2- 172.21.15.3:0/875135821 >> v2:172.21.15.143:3300/0 conn(0x7f991c1437f0 0x7f991c143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f991959a700 1 -- 172.21.15.3:0/875135821 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731263887 0 0) 0x7f991c0a8cf0 con 0x7f991c0aad90 2022-07-21T01:41:23.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f991959a700 1 -- 172.21.15.3:0/875135821 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9900003260 con 0x7f991c0aad90 2022-07-21T01:41:23.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f991a59c700 1 -- 172.21.15.3:0/875135821 >> v1:172.21.15.143:6789/0 conn(0x7f991c13dad0 legacy=0x7f991c0a9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f991a59c700 1 -- 172.21.15.3:0/875135821 >> v1:172.21.15.3:6789/0 conn(0x7f991c0aad90 legacy=0x7f991c13d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.364+0000 7f991a59c700 1 -- 172.21.15.3:0/875135821 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c05fd50 con 0x7f991c1437f0 2022-07-21T01:41:23.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f991a59c700 1 --2- 172.21.15.3:0/875135821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f991c1437f0 0x7f991c143bd0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f9904002f70 tx=0x7f9904006df0).ready entity=mon.1 client_cookie=cb274c795581a87f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:23.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f991959a700 1 -- 172.21.15.3:0/875135821 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990400d610 con 0x7f991c1437f0 2022-07-21T01:41:23.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f991959a700 1 -- 172.21.15.3:0/875135821 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9904000e90 con 0x7f991c1437f0 2022-07-21T01:41:23.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f991959a700 1 -- 172.21.15.3:0/875135821 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990400d960 con 0x7f991c1437f0 2022-07-21T01:41:23.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f9920e32700 1 -- 172.21.15.3:0/875135821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f991c1437f0 msgr2=0x7f991c143bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f9920e32700 1 --2- 172.21.15.3:0/875135821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f991c1437f0 0x7f991c143bd0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f9904002f70 tx=0x7f9904006df0).stop 2022-07-21T01:41:23.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f9920e32700 1 -- 172.21.15.3:0/875135821 shutdown_connections 2022-07-21T01:41:23.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f9920e32700 1 --2- 172.21.15.3:0/875135821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f991c1437f0 0x7f991c143bd0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f9920e32700 1 -- 172.21.15.3:0/875135821 >> 172.21.15.3:0/875135821 conn(0x7f991c0b5570 msgr2=0x7f991c0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:23.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f9920e32700 1 -- 172.21.15.3:0/875135821 shutdown_connections 2022-07-21T01:41:23.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f9920e32700 1 -- 172.21.15.3:0/875135821 wait complete. 2022-07-21T01:41:23.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f9920e32700 1 Processor -- start 2022-07-21T01:41:23.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.368+0000 7f9920e32700 1 -- start start 2022-07-21T01:41:23.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9920e32700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f991c0aad90 0x7f991c0b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9920e32700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f991c13dad0 0x7f991c1403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9920e32700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f991c1437f0 0x7f991c1408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9920e32700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0adff0 con 0x7f991c13dad0 2022-07-21T01:41:23.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9920e32700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0ad390 con 0x7f991c0aad90 2022-07-21T01:41:23.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9920e32700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c0adb50 con 0x7f991c1437f0 2022-07-21T01:41:23.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9919d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f991c13dad0 0x7f991c1403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9919d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f991c13dad0 0x7f991c1403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49596/0 (socket says 172.21.15.3:49596) 2022-07-21T01:41:23.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9919d9b700 1 -- 172.21.15.3:0/505763765 learned_addr learned my addr 172.21.15.3:0/505763765 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:23.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f991a59c700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f991c0aad90 0x7f991c0b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f991ad9d700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f991c1437f0 0x7f991c1408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9919d9b700 1 -- 172.21.15.3:0/505763765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f991c1437f0 msgr2=0x7f991c1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9919d9b700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f991c1437f0 0x7f991c1408d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9919d9b700 1 -- 172.21.15.3:0/505763765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f991c0aad90 msgr2=0x7f991c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:23.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9919d9b700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f991c0aad90 0x7f991c0b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:23.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9919d9b700 1 -- 172.21.15.3:0/505763765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990400b180 con 0x7f991c13dad0 2022-07-21T01:41:23.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9919d9b700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f991c13dad0 0x7f991c1403e0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f990c001490 tx=0x7f990c007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:23.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f99137fe700 1 -- 172.21.15.3:0/505763765 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990c005ba0 con 0x7f991c13dad0 2022-07-21T01:41:23.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f99137fe700 1 -- 172.21.15.3:0/505763765 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c005d00 con 0x7f991c13dad0 2022-07-21T01:41:23.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991c13eff0 con 0x7f991c13dad0 2022-07-21T01:41:23.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f99137fe700 1 -- 172.21.15.3:0/505763765 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990c0072c0 con 0x7f991c13dad0 2022-07-21T01:41:23.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991c13ece0 con 0x7f991c13dad0 2022-07-21T01:41:23.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f99137fe700 1 -- 172.21.15.3:0/505763765 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f990c010070 con 0x7f991c13dad0 2022-07-21T01:41:23.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c066820 con 0x7f991c13dad0 2022-07-21T01:41:23.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.372+0000 7f99137fe700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f98f4053b00 0x7f98f4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:23.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.376+0000 7f99137fe700 1 -- 172.21.15.3:0/505763765 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f990c06b670 con 0x7f991c13dad0 2022-07-21T01:41:23.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.376+0000 7f991a59c700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f98f4053b00 0x7f98f4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:23.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.376+0000 7f991a59c700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f98f4053b00 0x7f98f4055fc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f990401a000 tx=0x7f9904014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:23.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.380+0000 7f99137fe700 1 -- 172.21.15.3:0/505763765 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f990c042670 con 0x7f991c13dad0 2022-07-21T01:41:23.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:23.516+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f991c0ad550 con 0x7f991c13dad0 2022-07-21T01:41:24.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.368+0000 7f99137fe700 1 -- 172.21.15.3:0/505763765 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v117) v1 ==== 97+0+0 (secure 0 0 0) 0x7f990c045c90 con 0x7f991c13dad0 2022-07-21T01:41:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f98f4053b00 msgr2=0x7f98f4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f98f4053b00 0x7f98f4055fc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f990401a000 tx=0x7f9904014000).stop 2022-07-21T01:41:24.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f991c13dad0 msgr2=0x7f991c1403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f991c13dad0 0x7f991c1403e0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f990c001490 tx=0x7f990c007e50).stop 2022-07-21T01:41:24.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 shutdown_connections 2022-07-21T01:41:24.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f991c1437f0 0x7f991c1408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f991c13dad0 0x7f991c1403e0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f98f4053b00 0x7f98f4055fc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 --2- 172.21.15.3:0/505763765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f991c0aad90 0x7f991c0b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 >> 172.21.15.3:0/505763765 conn(0x7f991c0b5570 msgr2=0x7f991c0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 shutdown_connections 2022-07-21T01:41:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.372+0000 7f9920e32700 1 -- 172.21.15.3:0/505763765 wait complete. 2022-07-21T01:41:24.380 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:41:24.390 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:41:24.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:41:24.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27a35c700 1 Processor -- start 2022-07-21T01:41:24.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27a35c700 1 -- start start 2022-07-21T01:41:24.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27a35c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb2740b28b0 0x7fb27413e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27a35c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2740b1180 0x7fb27413e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27a35c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2740aac20 con 0x7fb274141210 2022-07-21T01:41:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27a35c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb274141db0 con 0x7fb2740b28b0 2022-07-21T01:41:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27a35c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb274140e20 con 0x7fb2740b1180 2022-07-21T01:41:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27935a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb2740b28b0 0x7fb27413e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb278b59700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2740b1180 0x7fb27413e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27935a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb2740b28b0 0x7fb27413e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49606/0 (socket says 172.21.15.3:49606) 2022-07-21T01:41:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.516+0000 7fb27935a700 1 -- 172.21.15.3:0/3566994298 learned_addr learned my addr 172.21.15.3:0/3566994298 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:24.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27935a700 1 -- 172.21.15.3:0/3566994298 >> v2:172.21.15.143:3300/0 conn(0x7fb2740b1180 msgr2=0x7fb27413e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:24.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27935a700 1 --2- 172.21.15.3:0/3566994298 >> v2:172.21.15.143:3300/0 conn(0x7fb2740b1180 0x7fb27413e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27935a700 1 -- 172.21.15.3:0/3566994298 >> v1:172.21.15.143:6789/0 conn(0x7fb274141210 legacy=0x7fb2741416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27935a700 1 -- 172.21.15.3:0/3566994298 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2740b2c90 con 0x7fb2740b28b0 2022-07-21T01:41:24.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27935a700 1 --2- 172.21.15.3:0/3566994298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b28b0 0x7fb27413e1c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7fb264002f70 tx=0x7fb264006eb0).ready entity=mon.0 client_cookie=1214b7cbd40ad92b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:24.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb26bfff700 1 -- 172.21.15.3:0/3566994298 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb26400d5c0 con 0x7fb2740b28b0 2022-07-21T01:41:24.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb26bfff700 1 -- 172.21.15.3:0/3566994298 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb264000e90 con 0x7fb2740b28b0 2022-07-21T01:41:24.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb26bfff700 1 -- 172.21.15.3:0/3566994298 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb26400d950 con 0x7fb2740b28b0 2022-07-21T01:41:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 -- 172.21.15.3:0/3566994298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b28b0 msgr2=0x7fb27413e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 --2- 172.21.15.3:0/3566994298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b28b0 0x7fb27413e1c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7fb264002f70 tx=0x7fb264006eb0).stop 2022-07-21T01:41:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 -- 172.21.15.3:0/3566994298 shutdown_connections 2022-07-21T01:41:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 --2- 172.21.15.3:0/3566994298 >> v2:172.21.15.143:3300/0 conn(0x7fb2740b1180 0x7fb27413e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 --2- 172.21.15.3:0/3566994298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b28b0 0x7fb27413e1c0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 -- 172.21.15.3:0/3566994298 >> 172.21.15.3:0/3566994298 conn(0x7fb274060310 msgr2=0x7fb2740b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 -- 172.21.15.3:0/3566994298 shutdown_connections 2022-07-21T01:41:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 -- 172.21.15.3:0/3566994298 wait complete. 2022-07-21T01:41:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 Processor -- start 2022-07-21T01:41:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.520+0000 7fb27a35c700 1 -- start start 2022-07-21T01:41:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27a35c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b1180 0x7fb2740ad150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27a35c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2740b28b0 0x7fb27413f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27a35c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274141210 0x7fb2741559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27a35c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27413fd70 con 0x7fb2740b1180 2022-07-21T01:41:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27a35c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb274142380 con 0x7fb274141210 2022-07-21T01:41:24.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27a35c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2741425e0 con 0x7fb2740b28b0 2022-07-21T01:41:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27935a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b1180 0x7fb2740ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27935a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b1180 0x7fb2740ad150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49610/0 (socket says 172.21.15.3:49610) 2022-07-21T01:41:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27935a700 1 -- 172.21.15.3:0/1114327368 learned_addr learned my addr 172.21.15.3:0/1114327368 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:24.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb278b59700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2740b28b0 0x7fb27413f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb279b5b700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274141210 0x7fb2741559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27935a700 1 -- 172.21.15.3:0/1114327368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2740b28b0 msgr2=0x7fb27413f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27935a700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2740b28b0 0x7fb27413f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27935a700 1 -- 172.21.15.3:0/1114327368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274141210 msgr2=0x7fb2741559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27935a700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274141210 0x7fb2741559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27935a700 1 -- 172.21.15.3:0/1114327368 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26400b1e0 con 0x7fb2740b1180 2022-07-21T01:41:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb278b59700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2740b28b0 0x7fb27413f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27935a700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b1180 0x7fb2740ad150 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7fb264002f70 tx=0x7fb264015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb26a7fc700 1 -- 172.21.15.3:0/1114327368 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb264011760 con 0x7fb2740b1180 2022-07-21T01:41:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb26a7fc700 1 -- 172.21.15.3:0/1114327368 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2640118c0 con 0x7fb2740b1180 2022-07-21T01:41:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb274142880 con 0x7fb2740b1180 2022-07-21T01:41:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb26a7fc700 1 -- 172.21.15.3:0/1114327368 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb264011c20 con 0x7fb2740b1180 2022-07-21T01:41:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb274142d20 con 0x7fb2740b1180 2022-07-21T01:41:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.524+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2741421c0 con 0x7fb2740b1180 2022-07-21T01:41:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.528+0000 7fb26a7fc700 1 -- 172.21.15.3:0/1114327368 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb26401a040 con 0x7fb2740b1180 2022-07-21T01:41:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.528+0000 7fb26a7fc700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb25c053a70 0x7fb25c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.528+0000 7fb26a7fc700 1 -- 172.21.15.3:0/1114327368 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fb26406ca50 con 0x7fb2740b1180 2022-07-21T01:41:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.528+0000 7fb278b59700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb25c053a70 0x7fb25c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.528+0000 7fb278b59700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb25c053a70 0x7fb25c055f30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fb27000b000 tx=0x7fb270008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.532+0000 7fb26a7fc700 1 -- 172.21.15.3:0/1114327368 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb264043b50 con 0x7fb2740b1180 2022-07-21T01:41:24.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb2740574e0 con 0x7fb2740b1180 2022-07-21T01:41:24.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb26a7fc700 1 -- 172.21.15.3:0/1114327368 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v31) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fb264023070 con 0x7fb2740b1180 2022-07-21T01:41:24.708 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:24.708 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:41:24.709 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:41:24.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.710 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:41:24.710 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:41:24.710 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.711 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:24.711 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.712 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.712 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.712 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.713 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:41:24.713 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:41:24.714 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.714 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:24.716 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.716 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.717 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.717 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.719 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:41:24.719 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:41:24.719 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.719 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:24.720 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.720 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.720 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.721 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.722 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:41:24.723 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:41:24.723 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.725 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.725 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.726 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:41:24.726 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:41:24.726 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.727 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.728 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:41:24.728 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:41:24.729 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.729 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.730 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.730 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:41:24.731 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:41:24.731 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.732 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.732 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.733 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:41:24.733 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:41:24.733 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.734 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.734 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.735 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:41:24.735 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:41:24.735 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.736 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.736 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.737 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:41:24.737 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:41:24.737 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.738 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.738 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.739 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:41:24.739 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:41:24.739 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.740 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.741 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:41:24.741 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:41:24.742 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:24.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:41:24.742 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:41:24.743 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.743 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:41:24.744 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:41:24.744 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:41:24.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:41:24.745 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:41:24.745 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.745 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:41:24.746 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:41:24.746 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:41:24.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:41:24.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:41:24.747 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.747 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.748 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:41:24.748 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:41:24.748 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.749 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:41:24.749 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:41:24.749 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.749 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:24.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:24.750 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:41:24.750 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:41:24.751 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:41:24.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:41:24.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:41:24.751 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:41:24.752 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.752 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:24.752 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:24.752 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:41:24.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb25c053a70 msgr2=0x7fb25c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb25c053a70 0x7fb25c055f30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fb27000b000 tx=0x7fb270008040).stop 2022-07-21T01:41:24.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b1180 msgr2=0x7fb2740ad150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b1180 0x7fb2740ad150 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7fb264002f70 tx=0x7fb264015000).stop 2022-07-21T01:41:24.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 shutdown_connections 2022-07-21T01:41:24.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb25c053a70 0x7fb25c055f30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb274141210 0x7fb2741559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2740b28b0 0x7fb27413f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 --2- 172.21.15.3:0/1114327368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2740b1180 0x7fb2740ad150 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 >> 172.21.15.3:0/1114327368 conn(0x7fb274060310 msgr2=0x7fb2740b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:24.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 shutdown_connections 2022-07-21T01:41:24.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.704+0000 7fb27a35c700 1 -- 172.21.15.3:0/1114327368 wait complete. 2022-07-21T01:41:24.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:24.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef415cb700 1 Processor -- start 2022-07-21T01:41:24.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef415cb700 1 -- start start 2022-07-21T01:41:24.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef415cb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef3c0b01a0 0x7fef3c0a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef415cb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fef3c0afd60 0x7fef3c0a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef415cb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3c0a7250 con 0x7fef3c0b2710 2022-07-21T01:41:24.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef415cb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3c0a7480 con 0x7fef3c0afd60 2022-07-21T01:41:24.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef415cb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3c0a7720 con 0x7fef3c0b01a0 2022-07-21T01:41:24.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef3bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fef3c0b2710 0x7fef3c14afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57898/0 (socket says 172.21.15.3:57898) 2022-07-21T01:41:24.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef3bfff700 1 -- 172.21.15.3:0/4177190309 learned_addr learned my addr 172.21.15.3:0/4177190309 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:24.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef40dca700 1 --2- 172.21.15.3:0/4177190309 >> v2:172.21.15.3:3300/0 conn(0x7fef3c0afd60 0x7fef3c0a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef337fe700 1 --2- 172.21.15.3:0/4177190309 >> v2:172.21.15.143:3300/0 conn(0x7fef3c0b01a0 0x7fef3c0a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef3b7fe700 1 -- 172.21.15.3:0/4177190309 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246181616 0 0) 0x7fef3c0a7250 con 0x7fef3c0b2710 2022-07-21T01:41:24.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef3b7fe700 1 -- 172.21.15.3:0/4177190309 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef20003260 con 0x7fef3c0b2710 2022-07-21T01:41:24.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef40dca700 1 -- 172.21.15.3:0/4177190309 >> v2:172.21.15.143:3300/0 conn(0x7fef3c0b01a0 msgr2=0x7fef3c0a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:24.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef40dca700 1 --2- 172.21.15.3:0/4177190309 >> v2:172.21.15.143:3300/0 conn(0x7fef3c0b01a0 0x7fef3c0a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef40dca700 1 -- 172.21.15.3:0/4177190309 >> v1:172.21.15.3:6789/0 conn(0x7fef3c0b2710 legacy=0x7fef3c14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef40dca700 1 -- 172.21.15.3:0/4177190309 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c0a79c0 con 0x7fef3c0afd60 2022-07-21T01:41:24.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef40dca700 1 --2- 172.21.15.3:0/4177190309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0afd60 0x7fef3c0a8a30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7fef340032f0 tx=0x7fef340091d0).ready entity=mon.0 client_cookie=858b0165bbe7ac5d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:24.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.864+0000 7fef3b7fe700 1 -- 172.21.15.3:0/4177190309 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef3400b600 con 0x7fef3c0afd60 2022-07-21T01:41:24.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef3b7fe700 1 -- 172.21.15.3:0/4177190309 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3400b760 con 0x7fef3c0afd60 2022-07-21T01:41:24.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef3b7fe700 1 -- 172.21.15.3:0/4177190309 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef3400b8d0 con 0x7fef3c0afd60 2022-07-21T01:41:24.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 -- 172.21.15.3:0/4177190309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0afd60 msgr2=0x7fef3c0a8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 --2- 172.21.15.3:0/4177190309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0afd60 0x7fef3c0a8a30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7fef340032f0 tx=0x7fef340091d0).stop 2022-07-21T01:41:24.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 -- 172.21.15.3:0/4177190309 shutdown_connections 2022-07-21T01:41:24.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 --2- 172.21.15.3:0/4177190309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0afd60 0x7fef3c0a8a30 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 --2- 172.21.15.3:0/4177190309 >> v2:172.21.15.143:3300/0 conn(0x7fef3c0b01a0 0x7fef3c0a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 -- 172.21.15.3:0/4177190309 >> 172.21.15.3:0/4177190309 conn(0x7fef3c060310 msgr2=0x7fef3c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:24.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 -- 172.21.15.3:0/4177190309 shutdown_connections 2022-07-21T01:41:24.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 -- 172.21.15.3:0/4177190309 wait complete. 2022-07-21T01:41:24.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 Processor -- start 2022-07-21T01:41:24.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.868+0000 7fef415cb700 1 -- start start 2022-07-21T01:41:24.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef415cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3c0afd60 0x7fef3c14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef415cb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0b01a0 0x7fef3c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef415cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3c0b2710 0x7fef3c1569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef415cb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c15a8b0 con 0x7fef3c0b01a0 2022-07-21T01:41:24.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef415cb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c159a00 con 0x7fef3c0afd60 2022-07-21T01:41:24.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef415cb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c159c30 con 0x7fef3c0b2710 2022-07-21T01:41:24.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0b01a0 0x7fef3c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0b01a0 0x7fef3c1564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49626/0 (socket says 172.21.15.3:49626) 2022-07-21T01:41:24.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef337fe700 1 -- 172.21.15.3:0/2126560425 learned_addr learned my addr 172.21.15.3:0/2126560425 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:24.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef3bfff700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3c0afd60 0x7fef3c14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef40dca700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3c0b2710 0x7fef3c1569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef337fe700 1 -- 172.21.15.3:0/2126560425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3c0b2710 msgr2=0x7fef3c1569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef337fe700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3c0b2710 0x7fef3c1569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef337fe700 1 -- 172.21.15.3:0/2126560425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3c0afd60 msgr2=0x7fef3c14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:24.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef337fe700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3c0afd60 0x7fef3c14d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:24.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef337fe700 1 -- 172.21.15.3:0/2126560425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef34009470 con 0x7fef3c0b01a0 2022-07-21T01:41:24.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef40dca700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3c0b2710 0x7fef3c1569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:24.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef337fe700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0b01a0 0x7fef3c1564b0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7fef2c00c000 tx=0x7fef2c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:24.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef39ffb700 1 -- 172.21.15.3:0/2126560425 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef2c010780 con 0x7fef3c0b01a0 2022-07-21T01:41:24.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef39ffb700 1 -- 172.21.15.3:0/2126560425 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2c0108e0 con 0x7fef3c0b01a0 2022-07-21T01:41:24.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef39ffb700 1 -- 172.21.15.3:0/2126560425 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef2c010c40 con 0x7fef3c0b01a0 2022-07-21T01:41:24.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3c159ed0 con 0x7fef3c0b01a0 2022-07-21T01:41:24.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3c0ae940 con 0x7fef3c0b01a0 2022-07-21T01:41:24.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3c15a060 con 0x7fef3c0b01a0 2022-07-21T01:41:24.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.872+0000 7fef39ffb700 1 -- 172.21.15.3:0/2126560425 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef2c01c460 con 0x7fef3c0b01a0 2022-07-21T01:41:24.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.876+0000 7fef39ffb700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef14053a70 0x7fef14055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:24.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.876+0000 7fef39ffb700 1 -- 172.21.15.3:0/2126560425 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fef2c06c1a0 con 0x7fef3c0b01a0 2022-07-21T01:41:24.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.876+0000 7fef3bfff700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef14053a70 0x7fef14055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:24.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.876+0000 7fef3bfff700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef14053a70 0x7fef14055f30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fef2400d000 tx=0x7fef24009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:24.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:24.880+0000 7fef39ffb700 1 -- 172.21.15.3:0/2126560425 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef2c043240 con 0x7fef3c0b01a0 2022-07-21T01:41:25.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.004+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef3c0574e0 con 0x7fef3c0b01a0 2022-07-21T01:41:25.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef39ffb700 1 -- 172.21.15.3:0/2126560425 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+13424 (secure 0 0 0) 0x7fef2c046860 con 0x7fef3c0b01a0 2022-07-21T01:41:25.012 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:25.013 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":117,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:24.359115+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:25.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef14053a70 msgr2=0x7fef14055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef14053a70 0x7fef14055f30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fef2400d000 tx=0x7fef24009040).stop 2022-07-21T01:41:25.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0b01a0 msgr2=0x7fef3c1564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0b01a0 0x7fef3c1564b0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7fef2c00c000 tx=0x7fef2c006df0).stop 2022-07-21T01:41:25.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 shutdown_connections 2022-07-21T01:41:25.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3c0b2710 0x7fef3c1569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef3c0b01a0 0x7fef3c1564b0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fef14053a70 0x7fef14055f30 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 --2- 172.21.15.3:0/2126560425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3c0afd60 0x7fef3c14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 >> 172.21.15.3:0/2126560425 conn(0x7fef3c060310 msgr2=0x7fef3c144c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:25.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 shutdown_connections 2022-07-21T01:41:25.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.008+0000 7fef415cb700 1 -- 172.21.15.3:0/2126560425 wait complete. 2022-07-21T01:41:25.030 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:41:25.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:41:25.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.148+0000 7fdb85c81700 1 Processor -- start 2022-07-21T01:41:25.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.148+0000 7fdb85c81700 1 -- start start 2022-07-21T01:41:25.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb85c81700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdb800b2b50 0x7fdb800af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb85c81700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb85c81700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb800ad9a0 con 0x7fdb800b5850 2022-07-21T01:41:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb85c81700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb800adc40 con 0x7fdb800b2b50 2022-07-21T01:41:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb85c81700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800adee0 con 0x7fdb800b1770 2022-07-21T01:41:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdb800b5850 0x7fdb80143630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57916/0 (socket says 172.21.15.3:57916) 2022-07-21T01:41:25.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7ffff700 1 -- 172.21.15.3:0/2470637731 learned_addr learned my addr 172.21.15.3:0/2470637731 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7f7fe700 1 --2- 172.21.15.3:0/2470637731 >> v2:172.21.15.143:3300/0 conn(0x7fdb800b2b50 0x7fdb800af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb77fff700 1 --2- 172.21.15.3:0/2470637731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7effd700 1 -- 172.21.15.3:0/2470637731 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930681780 0 0) 0x7fdb800ad9a0 con 0x7fdb800b5850 2022-07-21T01:41:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7effd700 1 -- 172.21.15.3:0/2470637731 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb60003260 con 0x7fdb800b5850 2022-07-21T01:41:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7f7fe700 1 -- 172.21.15.3:0/2470637731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 msgr2=0x7fdb800afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7f7fe700 1 --2- 172.21.15.3:0/2470637731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7f7fe700 1 -- 172.21.15.3:0/2470637731 >> v1:172.21.15.3:6789/0 conn(0x7fdb800b5850 legacy=0x7fdb80143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7f7fe700 1 -- 172.21.15.3:0/2470637731 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb800ae180 con 0x7fdb800b2b50 2022-07-21T01:41:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb77fff700 1 --2- 172.21.15.3:0/2470637731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7f7fe700 1 --2- 172.21.15.3:0/2470637731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b2b50 0x7fdb800af8b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fdb70002f70 tx=0x7fdb70006df0).ready entity=mon.1 client_cookie=a3f674b202f786d4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7effd700 1 -- 172.21.15.3:0/2470637731 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb7000d4a0 con 0x7fdb800b2b50 2022-07-21T01:41:25.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7effd700 1 -- 172.21.15.3:0/2470637731 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb70000e90 con 0x7fdb800b2b50 2022-07-21T01:41:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb7effd700 1 -- 172.21.15.3:0/2470637731 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb7000d7f0 con 0x7fdb800b2b50 2022-07-21T01:41:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb85c81700 1 -- 172.21.15.3:0/2470637731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b2b50 msgr2=0x7fdb800af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.152+0000 7fdb85c81700 1 --2- 172.21.15.3:0/2470637731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b2b50 0x7fdb800af8b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fdb70002f70 tx=0x7fdb70006df0).stop 2022-07-21T01:41:25.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- 172.21.15.3:0/2470637731 shutdown_connections 2022-07-21T01:41:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 --2- 172.21.15.3:0/2470637731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 --2- 172.21.15.3:0/2470637731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b2b50 0x7fdb800af8b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- 172.21.15.3:0/2470637731 >> 172.21.15.3:0/2470637731 conn(0x7fdb8006a2a0 msgr2=0x7fdb8005ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:25.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- 172.21.15.3:0/2470637731 shutdown_connections 2022-07-21T01:41:25.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- 172.21.15.3:0/2470637731 wait complete. 2022-07-21T01:41:25.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 Processor -- start 2022-07-21T01:41:25.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- start start 2022-07-21T01:41:25.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800640f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb800b2b50 0x7fdb801531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b5850 0x7fdb80145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb80065b20 con 0x7fdb800b2b50 2022-07-21T01:41:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb801527f0 con 0x7fdb800b5850 2022-07-21T01:41:25.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb80152a90 con 0x7fdb800b1770 2022-07-21T01:41:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb77fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb800b2b50 0x7fdb801531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb77fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb800b2b50 0x7fdb801531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49640/0 (socket says 172.21.15.3:49640) 2022-07-21T01:41:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb77fff700 1 -- 172.21.15.3:0/3205430451 learned_addr learned my addr 172.21.15.3:0/3205430451 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb7ffff700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b5850 0x7fdb80145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb7f7fe700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb77fff700 1 -- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 msgr2=0x7fdb800640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb77fff700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb77fff700 1 -- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b5850 msgr2=0x7fdb80145a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb77fff700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b5850 0x7fdb80145a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb77fff700 1 -- 172.21.15.3:0/3205430451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6800b180 con 0x7fdb800b2b50 2022-07-21T01:41:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb7f7fe700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:25.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb7ffff700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b5850 0x7fdb80145a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb77fff700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb800b2b50 0x7fdb801531d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7fdb6800c000 tx=0x7fdb68006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb7d7fa700 1 -- 172.21.15.3:0/3205430451 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb6800ca40 con 0x7fdb800b2b50 2022-07-21T01:41:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb7d7fa700 1 -- 172.21.15.3:0/3205430451 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6800cba0 con 0x7fdb800b2b50 2022-07-21T01:41:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb7d7fa700 1 -- 172.21.15.3:0/3205430451 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb6800ee70 con 0x7fdb800b2b50 2022-07-21T01:41:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7000b180 con 0x7fdb800b2b50 2022-07-21T01:41:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.156+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb80064ec0 con 0x7fdb800b2b50 2022-07-21T01:41:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.160+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb800b1480 con 0x7fdb800b2b50 2022-07-21T01:41:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.160+0000 7fdb7d7fa700 1 -- 172.21.15.3:0/3205430451 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdb6801c460 con 0x7fdb800b2b50 2022-07-21T01:41:25.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.160+0000 7fdb7d7fa700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb640539e0 0x7fdb64055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.160+0000 7fdb7d7fa700 1 -- 172.21.15.3:0/3205430451 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fdb6806c3a0 con 0x7fdb800b2b50 2022-07-21T01:41:25.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.160+0000 7fdb7f7fe700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb640539e0 0x7fdb64055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.160+0000 7fdb7f7fe700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb640539e0 0x7fdb64055ea0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fdb70018000 tx=0x7fdb70014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.168+0000 7fdb7d7fa700 1 -- 172.21.15.3:0/3205430451 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb68043440 con 0x7fdb800b2b50 2022-07-21T01:41:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fdb800574e0 con 0x7fdb800b2b50 2022-07-21T01:41:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb7d7fa700 1 -- 172.21.15.3:0/3205430451 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v117) v1 ==== 110+0+10 (secure 0 0 0) 0x7fdb68046a60 con 0x7fdb800b2b50 2022-07-21T01:41:25.299 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:41:25.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb640539e0 msgr2=0x7fdb64055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb640539e0 0x7fdb64055ea0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fdb70018000 tx=0x7fdb70014000).stop 2022-07-21T01:41:25.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb800b2b50 msgr2=0x7fdb801531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb800b2b50 0x7fdb801531d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7fdb6800c000 tx=0x7fdb68006df0).stop 2022-07-21T01:41:25.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 shutdown_connections 2022-07-21T01:41:25.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb640539e0 0x7fdb64055ea0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb800b5850 0x7fdb80145a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb800b2b50 0x7fdb801531d0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 --2- 172.21.15.3:0/3205430451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb800b1770 0x7fdb800640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 >> 172.21.15.3:0/3205430451 conn(0x7fdb8006a2a0 msgr2=0x7fdb801438f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:25.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 shutdown_connections 2022-07-21T01:41:25.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.296+0000 7fdb85c81700 1 -- 172.21.15.3:0/3205430451 wait complete. 2022-07-21T01:41:25.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:41:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.420+0000 7f44effff700 1 Processor -- start 2022-07-21T01:41:25.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.420+0000 7f44effff700 1 -- start start 2022-07-21T01:41:25.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.420+0000 7f44effff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 0x7f44f014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.420+0000 7f44effff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00a7250 con 0x7f44f00a8c50 2022-07-21T01:41:25.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.420+0000 7f44effff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00a7480 con 0x7f44f00b01a0 2022-07-21T01:41:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.420+0000 7f44effff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00a7720 con 0x7f44f00b2710 2022-07-21T01:41:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.420+0000 7f44ef7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f44f00a8c50 0x7f44f0151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57930/0 (socket says 172.21.15.3:57930) 2022-07-21T01:41:25.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.420+0000 7f44ef7fe700 1 -- 172.21.15.3:0/1404712650 learned_addr learned my addr 172.21.15.3:0/1404712650 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:25.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.420+0000 7f44eeffd700 1 --2- 172.21.15.3:0/1404712650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 0x7f44f014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44edffb700 1 -- 172.21.15.3:0/1404712650 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838995983 0 0) 0x7f44f00a7250 con 0x7f44f00a8c50 2022-07-21T01:41:25.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44edffb700 1 -- 172.21.15.3:0/1404712650 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4003260 con 0x7f44f00a8c50 2022-07-21T01:41:25.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44edffb700 1 -- 172.21.15.3:0/1404712650 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738862894 0 0) 0x7f44f00a7480 con 0x7f44f00b01a0 2022-07-21T01:41:25.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44edffb700 1 -- 172.21.15.3:0/1404712650 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f00a7250 con 0x7f44f00b01a0 2022-07-21T01:41:25.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44eeffd700 1 -- 172.21.15.3:0/1404712650 >> v1:172.21.15.143:6789/0 conn(0x7f44f00b01a0 legacy=0x7f44f00a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44eeffd700 1 -- 172.21.15.3:0/1404712650 >> v1:172.21.15.3:6789/0 conn(0x7f44f00a8c50 legacy=0x7f44f0151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44eeffd700 1 -- 172.21.15.3:0/1404712650 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00a79c0 con 0x7f44f00b2710 2022-07-21T01:41:25.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44eeffd700 1 --2- 172.21.15.3:0/1404712650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 0x7f44f014afd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f44e4002f70 tx=0x7f44e4006df0).ready entity=mon.2 client_cookie=b46aae5abe11bd8f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44edffb700 1 -- 172.21.15.3:0/1404712650 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e400d5c0 con 0x7f44f00b2710 2022-07-21T01:41:25.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44edffb700 1 -- 172.21.15.3:0/1404712650 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e4000e90 con 0x7f44f00b2710 2022-07-21T01:41:25.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44edffb700 1 -- 172.21.15.3:0/1404712650 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e400d910 con 0x7f44f00b2710 2022-07-21T01:41:25.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44effff700 1 -- 172.21.15.3:0/1404712650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 msgr2=0x7f44f014afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44effff700 1 --2- 172.21.15.3:0/1404712650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 0x7f44f014afd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f44e4002f70 tx=0x7f44e4006df0).stop 2022-07-21T01:41:25.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44effff700 1 -- 172.21.15.3:0/1404712650 shutdown_connections 2022-07-21T01:41:25.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44effff700 1 --2- 172.21.15.3:0/1404712650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 0x7f44f014afd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44effff700 1 -- 172.21.15.3:0/1404712650 >> 172.21.15.3:0/1404712650 conn(0x7f44f0060310 msgr2=0x7f44f00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:25.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44effff700 1 -- 172.21.15.3:0/1404712650 shutdown_connections 2022-07-21T01:41:25.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.424+0000 7f44effff700 1 -- 172.21.15.3:0/1404712650 wait complete. 2022-07-21T01:41:25.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44effff700 1 Processor -- start 2022-07-21T01:41:25.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44effff700 1 -- start start 2022-07-21T01:41:25.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44f00a8c50 0x7f44f015a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44effff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f00b01a0 0x7f44f00aa050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44effff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 0x7f44f0067400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44effff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00b3620 con 0x7f44f00a8c50 2022-07-21T01:41:25.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44effff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00b30e0 con 0x7f44f00b01a0 2022-07-21T01:41:25.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44effff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00b3380 con 0x7f44f00b2710 2022-07-21T01:41:25.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44eeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44f00a8c50 0x7f44f015a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44eeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44f00a8c50 0x7f44f015a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49652/0 (socket says 172.21.15.3:49652) 2022-07-21T01:41:25.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44eeffd700 1 -- 172.21.15.3:0/3936107518 learned_addr learned my addr 172.21.15.3:0/3936107518 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:25.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44ef7fe700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 0x7f44f0067400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.428+0000 7f44ee7fc700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f00b01a0 0x7f44f00aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44eeffd700 1 -- 172.21.15.3:0/3936107518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 msgr2=0x7f44f0067400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44eeffd700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 0x7f44f0067400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44eeffd700 1 -- 172.21.15.3:0/3936107518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f00b01a0 msgr2=0x7f44f00aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44eeffd700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f00b01a0 0x7f44f00aa050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44eeffd700 1 -- 172.21.15.3:0/3936107518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e400b180 con 0x7f44f00a8c50 2022-07-21T01:41:25.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44eeffd700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44f00a8c50 0x7f44f015a3f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f44e4002f70 tx=0x7f44e4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44dffff700 1 -- 172.21.15.3:0/3936107518 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e4013de0 con 0x7f44f00a8c50 2022-07-21T01:41:25.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44dffff700 1 -- 172.21.15.3:0/3936107518 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e401e400 con 0x7f44f00a8c50 2022-07-21T01:41:25.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44dffff700 1 -- 172.21.15.3:0/3936107518 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e401e760 con 0x7f44f00a8c50 2022-07-21T01:41:25.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f014bbb0 con 0x7f44f00a8c50 2022-07-21T01:41:25.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f00aa540 con 0x7f44f00a8c50 2022-07-21T01:41:25.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f014bd40 con 0x7f44f00a8c50 2022-07-21T01:41:25.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44dffff700 1 -- 172.21.15.3:0/3936107518 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f44e401c070 con 0x7f44f00a8c50 2022-07-21T01:41:25.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44dffff700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f44cc053b00 0x7f44cc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44dffff700 1 -- 172.21.15.3:0/3936107518 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f44e406c680 con 0x7f44f00a8c50 2022-07-21T01:41:25.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.432+0000 7f44ee7fc700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f44cc053b00 0x7f44cc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.436+0000 7f44ee7fc700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f44cc053b00 0x7f44cc055fc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f44e000d000 tx=0x7f44e0007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.440+0000 7f44dffff700 1 -- 172.21.15.3:0/3936107518 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f44e4043780 con 0x7f44f00a8c50 2022-07-21T01:41:25.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f44f00574e0 con 0x7f44f00a8c50 2022-07-21T01:41:25.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44dffff700 1 -- 172.21.15.3:0/3936107518 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v117) v1 ==== 92+0+10 (secure 0 0 0) 0x7f44e40230a0 con 0x7f44f00a8c50 2022-07-21T01:41:25.570 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:25.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f44cc053b00 msgr2=0x7f44cc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f44cc053b00 0x7f44cc055fc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f44e000d000 tx=0x7f44e0007da0).stop 2022-07-21T01:41:25.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44f00a8c50 msgr2=0x7f44f015a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44f00a8c50 0x7f44f015a3f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f44e4002f70 tx=0x7f44e4014000).stop 2022-07-21T01:41:25.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 shutdown_connections 2022-07-21T01:41:25.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f00b2710 0x7f44f0067400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f44cc053b00 0x7f44cc055fc0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f00b01a0 0x7f44f00aa050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 --2- 172.21.15.3:0/3936107518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44f00a8c50 0x7f44f015a3f0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 >> 172.21.15.3:0/3936107518 conn(0x7f44f0060310 msgr2=0x7f44f0144cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:25.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 shutdown_connections 2022-07-21T01:41:25.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.568+0000 7f44effff700 1 -- 172.21.15.3:0/3936107518 wait complete. 2022-07-21T01:41:25.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:41:25.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f032c8bc700 1 Processor -- start 2022-07-21T01:41:25.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f032c8bc700 1 -- start start 2022-07-21T01:41:25.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f032c8bc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f03280b2710 0x7f032814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f032c8bc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03280b01a0 0x7f032814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f032c8bc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03280a7250 con 0x7f03280afd60 2022-07-21T01:41:25.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f032c8bc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03280a7480 con 0x7f03280b2710 2022-07-21T01:41:25.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f032c8bc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03280a7720 con 0x7f03280b01a0 2022-07-21T01:41:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0327fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f03280afd60 0x7f03280a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57942/0 (socket says 172.21.15.3:57942) 2022-07-21T01:41:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0327fff700 1 -- 172.21.15.3:0/1773564632 learned_addr learned my addr 172.21.15.3:0/1773564632 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03277fe700 1 --2- 172.21.15.3:0/1773564632 >> v2:172.21.15.143:3300/0 conn(0x7f03280b2710 0x7f032814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0326ffd700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03280b01a0 0x7f032814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104303070 0 0) 0x7f03280a7250 con 0x7f03280afd60 2022-07-21T01:41:25.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0314003260 con 0x7f03280afd60 2022-07-21T01:41:25.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4111657176 0 0) 0x7f0314003260 con 0x7f03280afd60 2022-07-21T01:41:25.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03280b01a0 msgr2=0x7f032814b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:25.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03280b01a0 0x7f032814b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 >> v2:172.21.15.143:3300/0 conn(0x7f03280b2710 msgr2=0x7f032814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 --2- 172.21.15.3:0/1773564632 >> v2:172.21.15.143:3300/0 conn(0x7f03280b2710 0x7f032814afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03280a79c0 con 0x7f03280afd60 2022-07-21T01:41:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f0320001660 con 0x7f03280afd60 2022-07-21T01:41:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 >> v1:172.21.15.3:6789/0 conn(0x7f03280afd60 legacy=0x7f03280a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03280b01a0 0x7f03140037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0314003d50 0x7f0314004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f0314004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03280a7480 con 0x7f03140049f0 2022-07-21T01:41:25.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03280a7720 con 0x7f0314003d50 2022-07-21T01:41:25.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031400b780 con 0x7f03280b01a0 2022-07-21T01:41:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03277fe700 1 --2- 172.21.15.3:0/1773564632 >> v2:172.21.15.143:3300/0 conn(0x7f03280b2710 0x7f032814afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0327fff700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f0314004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0326ffd700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0314003d50 0x7f0314004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03277fe700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03280b01a0 0x7f03140037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0327fff700 1 -- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03280b01a0 msgr2=0x7f03140037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0327fff700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03280b01a0 0x7f03140037d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0327fff700 1 -- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0314003d50 msgr2=0x7f0314004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:25.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0327fff700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0314003d50 0x7f0314004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0327fff700 1 -- 172.21.15.3:0/1773564632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0314003260 con 0x7f03140049f0 2022-07-21T01:41:25.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03277fe700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03280b01a0 0x7f03140037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:25.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f0327fff700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f0314004e70 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7f0320001370 tx=0x7f03280b15d0).ready entity=mon.0 client_cookie=1633ceca6dcbbe76 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03200070d0 con 0x7f03140049f0 2022-07-21T01:41:25.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0320007230 con 0x7f03140049f0 2022-07-21T01:41:25.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.684+0000 7f03267fc700 1 -- 172.21.15.3:0/1773564632 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f032000c8f0 con 0x7f03140049f0 2022-07-21T01:41:25.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.688+0000 7f032c8bc700 1 -- 172.21.15.3:0/1773564632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 msgr2=0x7f0314004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.688+0000 7f032c8bc700 1 --2- 172.21.15.3:0/1773564632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f0314004e70 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7f0320001370 tx=0x7f03280b15d0).stop 2022-07-21T01:41:25.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.688+0000 7f0327fff700 1 -- 172.21.15.3:0/1773564632 reap_dead start 2022-07-21T01:41:25.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.688+0000 7f032c8bc700 1 -- 172.21.15.3:0/1773564632 shutdown_connections 2022-07-21T01:41:25.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.688+0000 7f032c8bc700 1 -- 172.21.15.3:0/1773564632 >> 172.21.15.3:0/1773564632 conn(0x7f0328060310 msgr2=0x7f03280b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:25.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.688+0000 7f032c8bc700 1 -- 172.21.15.3:0/1773564632 shutdown_connections 2022-07-21T01:41:25.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.688+0000 7f032c8bc700 1 -- 172.21.15.3:0/1773564632 wait complete. 2022-07-21T01:41:25.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 Processor -- start 2022-07-21T01:41:25.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 -- start start 2022-07-21T01:41:25.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f03280b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0328063190 0x7f0328064b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03280ac7e0 0x7f0328065020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03280aa750 con 0x7f03140049f0 2022-07-21T01:41:25.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03280ab240 con 0x7f03280ac7e0 2022-07-21T01:41:25.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03280aa980 con 0x7f0328063190 2022-07-21T01:41:25.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f03277fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f03280b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f03277fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f03280b5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49672/0 (socket says 172.21.15.3:49672) 2022-07-21T01:41:25.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f03277fe700 1 -- 172.21.15.3:0/2997087541 learned_addr learned my addr 172.21.15.3:0/2997087541 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:25.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f0327fff700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03280ac7e0 0x7f0328065020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f0326ffd700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0328063190 0x7f0328064b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f03277fe700 1 -- 172.21.15.3:0/2997087541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0328063190 msgr2=0x7f0328064b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f03277fe700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0328063190 0x7f0328064b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f03277fe700 1 -- 172.21.15.3:0/2997087541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03280ac7e0 msgr2=0x7f0328065020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f03277fe700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03280ac7e0 0x7f0328065020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f03277fe700 1 -- 172.21.15.3:0/2997087541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03280aac20 con 0x7f03140049f0 2022-07-21T01:41:25.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f0327fff700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03280ac7e0 0x7f0328065020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:25.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f03277fe700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f03280b5210 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7f031c00b000 tx=0x7f031c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f0324ff9700 1 -- 172.21.15.3:0/2997087541 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f031c0115a0 con 0x7f03140049f0 2022-07-21T01:41:25.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f0324ff9700 1 -- 172.21.15.3:0/2997087541 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031c000e90 con 0x7f03140049f0 2022-07-21T01:41:25.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03280aaec0 con 0x7f03140049f0 2022-07-21T01:41:25.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f0324ff9700 1 -- 172.21.15.3:0/2997087541 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f031c0118f0 con 0x7f03140049f0 2022-07-21T01:41:25.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.692+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0328159990 con 0x7f03140049f0 2022-07-21T01:41:25.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.696+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03280574e0 con 0x7f03140049f0 2022-07-21T01:41:25.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.696+0000 7f0324ff9700 1 -- 172.21.15.3:0/2997087541 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f031c019020 con 0x7f03140049f0 2022-07-21T01:41:25.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.696+0000 7f0324ff9700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0308053a70 0x7f0308055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.696+0000 7f0324ff9700 1 -- 172.21.15.3:0/2997087541 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f031c06bac0 con 0x7f03140049f0 2022-07-21T01:41:25.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.696+0000 7f0326ffd700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0308053a70 0x7f0308055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.696+0000 7f0326ffd700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0308053a70 0x7f0308055f30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f031800b000 tx=0x7f0318008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.704+0000 7f0324ff9700 1 -- 172.21.15.3:0/2997087541 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f031c042bc0 con 0x7f03140049f0 2022-07-21T01:41:25.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0328069a70 con 0x7f03140049f0 2022-07-21T01:41:25.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f0324ff9700 1 -- 172.21.15.3:0/2997087541 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v117) v1 ==== 84+0+143 (secure 0 0 0) 0x7f031c0461e0 con 0x7f03140049f0 2022-07-21T01:41:25.835 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:25.835 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:41:25.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:25.836 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:41:25.836 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:41:25.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:25.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:25.837 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:41:25.838 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:41:25.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:25.839 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:41:25.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0308053a70 msgr2=0x7f0308055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0308053a70 0x7f0308055f30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f031800b000 tx=0x7f0318008040).stop 2022-07-21T01:41:25.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 msgr2=0x7f03280b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f03280b5210 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7f031c00b000 tx=0x7f031c008040).stop 2022-07-21T01:41:25.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 shutdown_connections 2022-07-21T01:41:25.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0308053a70 0x7f0308055f30 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03280ac7e0 0x7f0328065020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0328063190 0x7f0328064b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 --2- 172.21.15.3:0/2997087541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03140049f0 0x7f03280b5210 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 >> 172.21.15.3:0/2997087541 conn(0x7f0328060310 msgr2=0x7f032805c810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:25.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 shutdown_connections 2022-07-21T01:41:25.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.832+0000 7f032c8bc700 1 -- 172.21.15.3:0/2997087541 wait complete. 2022-07-21T01:41:25.850 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:41:25.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:41:25.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009ebe8700 1 Processor -- start 2022-07-21T01:41:25.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009ebe8700 1 -- start start 2022-07-21T01:41:25.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009ebe8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f00980b13c0 0x7f00980b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009ebe8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00980afd80 0x7f0098142910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009ebe8700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00980ac520 con 0x7f0098142e00 2022-07-21T01:41:25.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009ebe8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00980b17a0 con 0x7f00980b13c0 2022-07-21T01:41:25.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009ebe8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00980b0450 con 0x7f00980afd80 2022-07-21T01:41:25.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009e3e7700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0098142e00 0x7f00980ae870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57964/0 (socket says 172.21.15.3:57964) 2022-07-21T01:41:25.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009e3e7700 1 -- 172.21.15.3:0/770497100 learned_addr learned my addr 172.21.15.3:0/770497100 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:25.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009dbe6700 1 --2- 172.21.15.3:0/770497100 >> v2:172.21.15.143:3300/0 conn(0x7f00980b13c0 0x7f00980b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009d3e5700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00980afd80 0x7f0098142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385080337 0 0) 0x7f00980ac520 con 0x7f0098142e00 2022-07-21T01:41:25.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0080003260 con 0x7f0098142e00 2022-07-21T01:41:25.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197528567 0 0) 0x7f0080003260 con 0x7f0098142e00 2022-07-21T01:41:25.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00980afd80 msgr2=0x7f0098142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:25.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009cbe4700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00980afd80 0x7f0098142910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 >> v2:172.21.15.143:3300/0 conn(0x7f00980b13c0 msgr2=0x7f00980b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:25.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009cbe4700 1 --2- 172.21.15.3:0/770497100 >> v2:172.21.15.143:3300/0 conn(0x7f00980b13c0 0x7f00980b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00980b06f0 con 0x7f0098142e00 2022-07-21T01:41:25.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f00900019e0 con 0x7f0098142e00 2022-07-21T01:41:25.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.960+0000 7f009dbe6700 1 --2- 172.21.15.3:0/770497100 >> v2:172.21.15.143:3300/0 conn(0x7f00980b13c0 0x7f00980b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:25.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 >> v1:172.21.15.3:6789/0 conn(0x7f0098142e00 legacy=0x7f00980ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00980afd80 0x7f0080003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0080003c80 0x7f0080004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800049f0 0x7f0080004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:25.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00980b17a0 con 0x7f00800049f0 2022-07-21T01:41:25.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00980b0450 con 0x7f0080003c80 2022-07-21T01:41:25.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008000b750 con 0x7f00980afd80 2022-07-21T01:41:25.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009e3e7700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800049f0 0x7f0080004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009dbe6700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00980afd80 0x7f0080003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009d3e5700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0080003c80 0x7f0080004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:25.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009dbe6700 1 -- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0080003c80 msgr2=0x7f0080004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:25.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009dbe6700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0080003c80 0x7f0080004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009dbe6700 1 -- 172.21.15.3:0/770497100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800049f0 msgr2=0x7f0080004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:25.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009dbe6700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800049f0 0x7f0080004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:25.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009dbe6700 1 -- 172.21.15.3:0/770497100 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0080003260 con 0x7f00980afd80 2022-07-21T01:41:25.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009d3e5700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0080003c80 0x7f0080004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:25.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009dbe6700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00980afd80 0x7f0080003750 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f008800c000 tx=0x7f0088006df0).ready entity=mon.2 client_cookie=a90a89cfc193bf10 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:25.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f008800e830 con 0x7f00980afd80 2022-07-21T01:41:25.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008800e990 con 0x7f00980afd80 2022-07-21T01:41:25.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009cbe4700 1 -- 172.21.15.3:0/770497100 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f008800ecf0 con 0x7f00980afd80 2022-07-21T01:41:25.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009ebe8700 1 -- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00980afd80 msgr2=0x7f0080003750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:25.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009ebe8700 1 --2- 172.21.15.3:0/770497100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00980afd80 0x7f0080003750 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f008800c000 tx=0x7f0088006df0).stop 2022-07-21T01:41:25.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009e3e7700 1 -- 172.21.15.3:0/770497100 reap_dead start 2022-07-21T01:41:25.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009ebe8700 1 -- 172.21.15.3:0/770497100 shutdown_connections 2022-07-21T01:41:25.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009ebe8700 1 -- 172.21.15.3:0/770497100 >> 172.21.15.3:0/770497100 conn(0x7f00980626f0 msgr2=0x7f0098062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:25.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009ebe8700 1 -- 172.21.15.3:0/770497100 shutdown_connections 2022-07-21T01:41:25.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009ebe8700 1 -- 172.21.15.3:0/770497100 wait complete. 2022-07-21T01:41:25.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009ebe8700 1 Processor -- start 2022-07-21T01:41:25.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.964+0000 7f009ebe8700 1 -- start start 2022-07-21T01:41:26.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009ebe8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00980afd80 0x7f0098156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009ebe8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f009813c770 0x7f009813cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009ebe8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0098145100 0x7f00981455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009ebe8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0098156d10 con 0x7f00980afd80 2022-07-21T01:41:26.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009ebe8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0098147370 con 0x7f0098145100 2022-07-21T01:41:26.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009ebe8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00980abea0 con 0x7f009813c770 2022-07-21T01:41:26.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009dbe6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00980afd80 0x7f0098156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009dbe6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00980afd80 0x7f0098156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49692/0 (socket says 172.21.15.3:49692) 2022-07-21T01:41:26.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009dbe6700 1 -- 172.21.15.3:0/1970024687 learned_addr learned my addr 172.21.15.3:0/1970024687 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:26.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009d3e5700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f009813c770 0x7f009813cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009e3e7700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0098145100 0x7f00981455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009dbe6700 1 -- 172.21.15.3:0/1970024687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f009813c770 msgr2=0x7f009813cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:26.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009dbe6700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f009813c770 0x7f009813cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009dbe6700 1 -- 172.21.15.3:0/1970024687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0098145100 msgr2=0x7f00981455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:26.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009dbe6700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0098145100 0x7f00981455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009dbe6700 1 -- 172.21.15.3:0/1970024687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00981470d0 con 0x7f00980afd80 2022-07-21T01:41:26.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009e3e7700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0098145100 0x7f00981455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:26.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009dbe6700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00980afd80 0x7f0098156180 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7f008800c000 tx=0x7f0088019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:26.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f0086ffd700 1 -- 172.21.15.3:0/1970024687 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f008801b820 con 0x7f00980afd80 2022-07-21T01:41:26.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f0086ffd700 1 -- 172.21.15.3:0/1970024687 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0088000e90 con 0x7f00980afd80 2022-07-21T01:41:26.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f0086ffd700 1 -- 172.21.15.3:0/1970024687 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f008801bb70 con 0x7f00980afd80 2022-07-21T01:41:26.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00981567c0 con 0x7f00980afd80 2022-07-21T01:41:26.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0098146130 con 0x7f00980afd80 2022-07-21T01:41:26.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0098156950 con 0x7f00980afd80 2022-07-21T01:41:26.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.968+0000 7f0086ffd700 1 -- 172.21.15.3:0/1970024687 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f008801bcd0 con 0x7f00980afd80 2022-07-21T01:41:26.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.972+0000 7f0086ffd700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0078053b30 0x7f0078055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.972+0000 7f0086ffd700 1 -- 172.21.15.3:0/1970024687 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f008806d210 con 0x7f00980afd80 2022-07-21T01:41:26.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.972+0000 7f009d3e5700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0078053b30 0x7f0078055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.972+0000 7f009d3e5700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0078053b30 0x7f0078055ff0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f009400b000 tx=0x7f0094006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:26.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:25.976+0000 7f0086ffd700 1 -- 172.21.15.3:0/1970024687 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0088044310 con 0x7f00980afd80 2022-07-21T01:41:26.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.100+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f00980574e0 con 0x7f00980afd80 2022-07-21T01:41:26.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.676+0000 7f0086ffd700 1 -- 172.21.15.3:0/1970024687 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v118) v1 ==== 133+0+0 (secure 0 0 0) 0x7f008802a070 con 0x7f00980afd80 2022-07-21T01:41:26.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0078053b30 msgr2=0x7f0078055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:26.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0078053b30 0x7f0078055ff0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f009400b000 tx=0x7f0094006df0).stop 2022-07-21T01:41:26.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00980afd80 msgr2=0x7f0098156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:26.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00980afd80 0x7f0098156180 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7f008800c000 tx=0x7f0088019000).stop 2022-07-21T01:41:26.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 shutdown_connections 2022-07-21T01:41:26.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0078053b30 0x7f0078055ff0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0098145100 0x7f00981455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f009813c770 0x7f009813cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 --2- 172.21.15.3:0/1970024687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00980afd80 0x7f0098156180 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 >> 172.21.15.3:0/1970024687 conn(0x7f00980626f0 msgr2=0x7f0098143370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:26.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 shutdown_connections 2022-07-21T01:41:26.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.684+0000 7f009ebe8700 1 -- 172.21.15.3:0/1970024687 wait complete. 2022-07-21T01:41:26.692 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:41:26.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:41:26.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.808+0000 7f6e45e28700 1 Processor -- start 2022-07-21T01:41:26.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.808+0000 7f6e45e28700 1 -- start start 2022-07-21T01:41:26.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6e400b0270 0x7f6e40143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6e400ac880 0x7f6e401434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e40067a10 con 0x7f6e400b2710 2022-07-21T01:41:26.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e400ad740 con 0x7f6e400b0270 2022-07-21T01:41:26.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e40067cb0 con 0x7f6e400ac880 2022-07-21T01:41:26.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6e400b0270 0x7f6e40143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6e400b0270 0x7f6e40143000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49702/0 (socket says 172.21.15.3:49702) 2022-07-21T01:41:26.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3ffff700 1 -- 172.21.15.3:0/2061040570 learned_addr learned my addr 172.21.15.3:0/2061040570 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:26.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45627700 1 --2- 172.21.15.3:0/2061040570 >> v2:172.21.15.143:3300/0 conn(0x7f6e400ac880 0x7f6e401434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3ffff700 1 -- 172.21.15.3:0/2061040570 >> v2:172.21.15.143:3300/0 conn(0x7f6e400ac880 msgr2=0x7f6e401434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:26.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3ffff700 1 --2- 172.21.15.3:0/2061040570 >> v2:172.21.15.143:3300/0 conn(0x7f6e400ac880 0x7f6e401434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3ffff700 1 -- 172.21.15.3:0/2061040570 >> v1:172.21.15.143:6789/0 conn(0x7f6e400b2710 legacy=0x7f6e401401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:26.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3ffff700 1 -- 172.21.15.3:0/2061040570 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e40147160 con 0x7f6e400b0270 2022-07-21T01:41:26.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3ffff700 1 --2- 172.21.15.3:0/2061040570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b0270 0x7f6e40143000 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f6e300032f0 tx=0x7f6e30009290).ready entity=mon.0 client_cookie=6486dc4c11b84154 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:26.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3f7fe700 1 -- 172.21.15.3:0/2061040570 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e3000b870 con 0x7f6e400b0270 2022-07-21T01:41:26.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3f7fe700 1 -- 172.21.15.3:0/2061040570 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3000b9d0 con 0x7f6e400b0270 2022-07-21T01:41:26.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e3f7fe700 1 -- 172.21.15.3:0/2061040570 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e3000bb40 con 0x7f6e400b0270 2022-07-21T01:41:26.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 -- 172.21.15.3:0/2061040570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b0270 msgr2=0x7f6e40143000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:26.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 --2- 172.21.15.3:0/2061040570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b0270 0x7f6e40143000 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f6e300032f0 tx=0x7f6e30009290).stop 2022-07-21T01:41:26.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 -- 172.21.15.3:0/2061040570 shutdown_connections 2022-07-21T01:41:26.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 --2- 172.21.15.3:0/2061040570 >> v2:172.21.15.143:3300/0 conn(0x7f6e400ac880 0x7f6e401434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 --2- 172.21.15.3:0/2061040570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b0270 0x7f6e40143000 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 -- 172.21.15.3:0/2061040570 >> 172.21.15.3:0/2061040570 conn(0x7f6e40060310 msgr2=0x7f6e400b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:26.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 -- 172.21.15.3:0/2061040570 shutdown_connections 2022-07-21T01:41:26.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.812+0000 7f6e45e28700 1 -- 172.21.15.3:0/2061040570 wait complete. 2022-07-21T01:41:26.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45e28700 1 Processor -- start 2022-07-21T01:41:26.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45e28700 1 -- start start 2022-07-21T01:41:26.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45e28700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e400ac880 0x7f6e4009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45e28700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e400b0270 0x7f6e400ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45e28700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b2710 0x7f6e40142f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45e28700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40152610 con 0x7f6e400b2710 2022-07-21T01:41:26.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45e28700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e401429f0 con 0x7f6e400b0270 2022-07-21T01:41:26.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45e28700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e40152840 con 0x7f6e400ac880 2022-07-21T01:41:26.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45627700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b2710 0x7f6e40142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45627700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b2710 0x7f6e40142f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49710/0 (socket says 172.21.15.3:49710) 2022-07-21T01:41:26.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e3ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e400b0270 0x7f6e400ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45627700 1 -- 172.21.15.3:0/3522359352 learned_addr learned my addr 172.21.15.3:0/3522359352 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:26.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e44e26700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e400ac880 0x7f6e4009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45627700 1 -- 172.21.15.3:0/3522359352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e400ac880 msgr2=0x7f6e4009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:26.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45627700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e400ac880 0x7f6e4009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45627700 1 -- 172.21.15.3:0/3522359352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e400b0270 msgr2=0x7f6e400ab4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:26.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45627700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e400b0270 0x7f6e400ab4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:26.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45627700 1 -- 172.21.15.3:0/3522359352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e30009530 con 0x7f6e400b2710 2022-07-21T01:41:26.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e3ffff700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e400b0270 0x7f6e400ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:26.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.816+0000 7f6e45627700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b2710 0x7f6e40142f80 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f6e3800a000 tx=0x7f6e38008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:26.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e3dffb700 1 -- 172.21.15.3:0/3522359352 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e38011780 con 0x7f6e400b2710 2022-07-21T01:41:26.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e3dffb700 1 -- 172.21.15.3:0/3522359352 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e380118e0 con 0x7f6e400b2710 2022-07-21T01:41:26.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e40142c90 con 0x7f6e400b2710 2022-07-21T01:41:26.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e3dffb700 1 -- 172.21.15.3:0/3522359352 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e38011c40 con 0x7f6e400b2710 2022-07-21T01:41:26.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e40143470 con 0x7f6e400b2710 2022-07-21T01:41:26.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e40152a00 con 0x7f6e400b2710 2022-07-21T01:41:26.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e3dffb700 1 -- 172.21.15.3:0/3522359352 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6e38012420 con 0x7f6e400b2710 2022-07-21T01:41:26.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e3dffb700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6e20053b30 0x7f6e20055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:26.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e3dffb700 1 -- 172.21.15.3:0/3522359352 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f6e3806bb80 con 0x7f6e400b2710 2022-07-21T01:41:26.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e44e26700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6e20053b30 0x7f6e20055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:26.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.820+0000 7f6e44e26700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6e20053b30 0x7f6e20055ff0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f6e3400d000 tx=0x7f6e34007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:26.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.828+0000 7f6e3dffb700 1 -- 172.21.15.3:0/3522359352 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e38042a30 con 0x7f6e400b2710 2022-07-21T01:41:26.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:26.952+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f6e400574e0 con 0x7f6e400b2710 2022-07-21T01:41:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.372+0000 7f6e3dffb700 1 -- 172.21.15.3:0/3522359352 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v119) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6e38046050 con 0x7f6e400b2710 2022-07-21T01:41:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6e20053b30 msgr2=0x7f6e20055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6e20053b30 0x7f6e20055ff0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f6e3400d000 tx=0x7f6e34007b70).stop 2022-07-21T01:41:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b2710 msgr2=0x7f6e40142f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b2710 0x7f6e40142f80 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f6e3800a000 tx=0x7f6e38008040).stop 2022-07-21T01:41:27.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 shutdown_connections 2022-07-21T01:41:27.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e400b2710 0x7f6e40142f80 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6e20053b30 0x7f6e20055ff0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e400b0270 0x7f6e400ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 --2- 172.21.15.3:0/3522359352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e400ac880 0x7f6e4009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 >> 172.21.15.3:0/3522359352 conn(0x7f6e40060310 msgr2=0x7f6e400b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:27.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 shutdown_connections 2022-07-21T01:41:27.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.380+0000 7f6e45e28700 1 -- 172.21.15.3:0/3522359352 wait complete. 2022-07-21T01:41:27.388 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:41:27.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:41:27.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.504+0000 7f2b9274b700 1 Processor -- start 2022-07-21T01:41:27.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.504+0000 7f2b9274b700 1 -- start start 2022-07-21T01:41:27.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2b8c0617e0 0x7f2b8c066760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c066c50 0x7f2b8c14db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8c060670 con 0x7f2b8c060290 2022-07-21T01:41:27.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8c060e00 con 0x7f2b8c0617e0 2022-07-21T01:41:27.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0610a0 con 0x7f2b8c066c50 2022-07-21T01:41:27.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b8bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2b8c060290 0x7f2b8c065b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57994/0 (socket says 172.21.15.3:57994) 2022-07-21T01:41:27.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b8bfff700 1 -- 172.21.15.3:0/897514118 learned_addr learned my addr 172.21.15.3:0/897514118 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:27.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b90ce8700 1 --2- 172.21.15.3:0/897514118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c066c50 0x7f2b8c14db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b837fe700 1 --2- 172.21.15.3:0/897514118 >> v2:172.21.15.143:3300/0 conn(0x7f2b8c0617e0 0x7f2b8c066760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b8b7fe700 1 -- 172.21.15.3:0/897514118 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298567516 0 0) 0x7f2b8c060670 con 0x7f2b8c060290 2022-07-21T01:41:27.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b8b7fe700 1 -- 172.21.15.3:0/897514118 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b6c003260 con 0x7f2b8c060290 2022-07-21T01:41:27.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b90ce8700 1 -- 172.21.15.3:0/897514118 >> v2:172.21.15.143:3300/0 conn(0x7f2b8c0617e0 msgr2=0x7f2b8c066760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:27.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b90ce8700 1 --2- 172.21.15.3:0/897514118 >> v2:172.21.15.143:3300/0 conn(0x7f2b8c0617e0 0x7f2b8c066760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b90ce8700 1 -- 172.21.15.3:0/897514118 >> v1:172.21.15.3:6789/0 conn(0x7f2b8c060290 legacy=0x7f2b8c065b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b90ce8700 1 -- 172.21.15.3:0/897514118 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c061340 con 0x7f2b8c066c50 2022-07-21T01:41:27.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b90ce8700 1 --2- 172.21.15.3:0/897514118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c066c50 0x7f2b8c14db60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2b840032f0 tx=0x7f2b840091d0).ready entity=mon.2 client_cookie=ed70b7ce50a792e2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:27.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b8b7fe700 1 -- 172.21.15.3:0/897514118 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b8400b600 con 0x7f2b8c066c50 2022-07-21T01:41:27.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b8b7fe700 1 -- 172.21.15.3:0/897514118 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8400b760 con 0x7f2b8c066c50 2022-07-21T01:41:27.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b8b7fe700 1 -- 172.21.15.3:0/897514118 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b8400b8d0 con 0x7f2b8c066c50 2022-07-21T01:41:27.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 -- 172.21.15.3:0/897514118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c066c50 msgr2=0x7f2b8c14db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 --2- 172.21.15.3:0/897514118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c066c50 0x7f2b8c14db60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2b840032f0 tx=0x7f2b840091d0).stop 2022-07-21T01:41:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 -- 172.21.15.3:0/897514118 shutdown_connections 2022-07-21T01:41:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 --2- 172.21.15.3:0/897514118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c066c50 0x7f2b8c14db60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 --2- 172.21.15.3:0/897514118 >> v2:172.21.15.143:3300/0 conn(0x7f2b8c0617e0 0x7f2b8c066760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 -- 172.21.15.3:0/897514118 >> 172.21.15.3:0/897514118 conn(0x7f2b8c05d0c0 msgr2=0x7f2b8c05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:27.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 -- 172.21.15.3:0/897514118 shutdown_connections 2022-07-21T01:41:27.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.508+0000 7f2b9274b700 1 -- 172.21.15.3:0/897514118 wait complete. 2022-07-21T01:41:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b9274b700 1 Processor -- start 2022-07-21T01:41:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b9274b700 1 -- start start 2022-07-21T01:41:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b9274b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c060290 0x7f2b8c146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b9274b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b8c0617e0 0x7f2b8c152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b9274b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b8c066c50 0x7f2b8c0a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b9274b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0a3270 con 0x7f2b8c0617e0 2022-07-21T01:41:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b9274b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c141170 con 0x7f2b8c066c50 2022-07-21T01:41:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b9274b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0a2fd0 con 0x7f2b8c060290 2022-07-21T01:41:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b837fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b8c0617e0 0x7f2b8c152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b837fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b8c0617e0 0x7f2b8c152550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49722/0 (socket says 172.21.15.3:49722) 2022-07-21T01:41:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b837fe700 1 -- 172.21.15.3:0/1562308443 learned_addr learned my addr 172.21.15.3:0/1562308443 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b8bfff700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c060290 0x7f2b8c146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.512+0000 7f2b90ce8700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b8c066c50 0x7f2b8c0a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b837fe700 1 -- 172.21.15.3:0/1562308443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c060290 msgr2=0x7f2b8c146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b837fe700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c060290 0x7f2b8c146400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b837fe700 1 -- 172.21.15.3:0/1562308443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b8c066c50 msgr2=0x7f2b8c0a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b837fe700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b8c066c50 0x7f2b8c0a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b837fe700 1 -- 172.21.15.3:0/1562308443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b84009470 con 0x7f2b8c0617e0 2022-07-21T01:41:27.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b8bfff700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c060290 0x7f2b8c146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:27.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b837fe700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b8c0617e0 0x7f2b8c152550 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f2b7400c000 tx=0x7f2b74006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:27.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b89ffb700 1 -- 172.21.15.3:0/1562308443 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b74010780 con 0x7f2b8c0617e0 2022-07-21T01:41:27.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b89ffb700 1 -- 172.21.15.3:0/1562308443 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b740108e0 con 0x7f2b8c0617e0 2022-07-21T01:41:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c0b83c0 con 0x7f2b8c0617e0 2022-07-21T01:41:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b89ffb700 1 -- 172.21.15.3:0/1562308443 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b74010c40 con 0x7f2b8c0617e0 2022-07-21T01:41:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8c0b4e90 con 0x7f2b8c0617e0 2022-07-21T01:41:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c0574e0 con 0x7f2b8c0617e0 2022-07-21T01:41:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.516+0000 7f2b89ffb700 1 -- 172.21.15.3:0/1562308443 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2b74026430 con 0x7f2b8c0617e0 2022-07-21T01:41:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.520+0000 7f2b89ffb700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b70053a70 0x7f2b70055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.520+0000 7f2b8bfff700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b70053a70 0x7f2b70055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.520+0000 7f2b89ffb700 1 -- 172.21.15.3:0/1562308443 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f2b7406cfd0 con 0x7f2b8c0617e0 2022-07-21T01:41:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.520+0000 7f2b8bfff700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b70053a70 0x7f2b70055f30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f2b7c00d000 tx=0x7f2b7c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.528+0000 7f2b89ffb700 1 -- 172.21.15.3:0/1562308443 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b74043ca0 con 0x7f2b8c0617e0 2022-07-21T01:41:27.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.692+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f2b8c0b2550 con 0x7f2b8c0617e0 2022-07-21T01:41:27.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.704+0000 7f2b89ffb700 1 -- 172.21.15.3:0/1562308443 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 21 and data pool 22 v59) v1 ==== 189+0+0 (secure 0 0 0) 0x7f2b740472c0 con 0x7f2b8c0617e0 2022-07-21T01:41:27.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.708+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b70053a70 msgr2=0x7f2b70055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.708+0000 7f2b9274b700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b70053a70 0x7f2b70055f30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f2b7c00d000 tx=0x7f2b7c009040).stop 2022-07-21T01:41:27.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.708+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b8c0617e0 msgr2=0x7f2b8c152550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.708+0000 7f2b9274b700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b8c0617e0 0x7f2b8c152550 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f2b7400c000 tx=0x7f2b74006df0).stop 2022-07-21T01:41:27.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.712+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 shutdown_connections 2022-07-21T01:41:27.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.712+0000 7f2b9274b700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b70053a70 0x7f2b70055f30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.712+0000 7f2b9274b700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b8c066c50 0x7f2b8c0a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.712+0000 7f2b9274b700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b8c0617e0 0x7f2b8c152550 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.712+0000 7f2b9274b700 1 --2- 172.21.15.3:0/1562308443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b8c060290 0x7f2b8c146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.712+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 >> 172.21.15.3:0/1562308443 conn(0x7f2b8c05d0c0 msgr2=0x7f2b8c0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:27.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.712+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 shutdown_connections 2022-07-21T01:41:27.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.712+0000 7f2b9274b700 1 -- 172.21.15.3:0/1562308443 wait complete. 2022-07-21T01:41:27.724 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 21 and data pool 22 2022-07-21T01:41:27.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:27.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 Processor -- start 2022-07-21T01:41:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 -- start start 2022-07-21T01:41:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff3d80b2710 0x7ff3d814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80a8c50 0x7ff3d814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d80a7250 con 0x7ff3d80b01a0 2022-07-21T01:41:27.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d80a7480 con 0x7ff3d80b2710 2022-07-21T01:41:27.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d80a7720 con 0x7ff3d80a8c50 2022-07-21T01:41:27.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d6ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff3d80b2710 0x7ff3d814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d67fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff3d80b01a0 0x7ff3d80a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58024/0 (socket says 172.21.15.3:58024) 2022-07-21T01:41:27.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d6ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ff3d80b2710 0x7ff3d814afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49742/0 (socket says 172.21.15.3:49742) 2022-07-21T01:41:27.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d67fc700 1 -- 172.21.15.3:0/89545594 learned_addr learned my addr 172.21.15.3:0/89545594 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:27.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d77fe700 1 --2- 172.21.15.3:0/89545594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80a8c50 0x7ff3d814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d6ffd700 1 -- 172.21.15.3:0/89545594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80a8c50 msgr2=0x7ff3d814b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:27.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d6ffd700 1 --2- 172.21.15.3:0/89545594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80a8c50 0x7ff3d814b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d5ffb700 1 -- 172.21.15.3:0/89545594 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524731493 0 0) 0x7ff3d80a7250 con 0x7ff3d80b01a0 2022-07-21T01:41:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d6ffd700 1 -- 172.21.15.3:0/89545594 >> v1:172.21.15.3:6789/0 conn(0x7ff3d80b01a0 legacy=0x7ff3d80a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d6ffd700 1 -- 172.21.15.3:0/89545594 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80a79c0 con 0x7ff3d80b2710 2022-07-21T01:41:27.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d6ffd700 1 --2- 172.21.15.3:0/89545594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 0x7ff3d814afd0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7ff3c0002f70 tx=0x7ff3c0006df0).ready entity=mon.0 client_cookie=86360fb0f5aa7160 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:27.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d5ffb700 1 -- 172.21.15.3:0/89545594 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3c000d570 con 0x7ff3d80b2710 2022-07-21T01:41:27.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d5ffb700 1 -- 172.21.15.3:0/89545594 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c0000e90 con 0x7ff3d80b2710 2022-07-21T01:41:27.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d5ffb700 1 -- 172.21.15.3:0/89545594 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3c000d8c0 con 0x7ff3d80b2710 2022-07-21T01:41:27.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 -- 172.21.15.3:0/89545594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 msgr2=0x7ff3d814afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 --2- 172.21.15.3:0/89545594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 0x7ff3d814afd0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7ff3c0002f70 tx=0x7ff3c0006df0).stop 2022-07-21T01:41:27.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 -- 172.21.15.3:0/89545594 shutdown_connections 2022-07-21T01:41:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 --2- 172.21.15.3:0/89545594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80a8c50 0x7ff3d814b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 --2- 172.21.15.3:0/89545594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 0x7ff3d814afd0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 -- 172.21.15.3:0/89545594 >> 172.21.15.3:0/89545594 conn(0x7ff3d8060310 msgr2=0x7ff3d80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:27.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 -- 172.21.15.3:0/89545594 shutdown_connections 2022-07-21T01:41:27.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.824+0000 7ff3d7fff700 1 -- 172.21.15.3:0/89545594 wait complete. 2022-07-21T01:41:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d7fff700 1 Processor -- start 2022-07-21T01:41:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d7fff700 1 -- start start 2022-07-21T01:41:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3d80a8c50 0x7ff3d81570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80b01a0 0x7ff3d80aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 0x7ff3d8065110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d7fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d811cfb0 con 0x7ff3d80b2710 2022-07-21T01:41:27.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d7fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8157590 con 0x7ff3d80a8c50 2022-07-21T01:41:27.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8067a40 con 0x7ff3d80b01a0 2022-07-21T01:41:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 0x7ff3d8065110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 0x7ff3d8065110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49752/0 (socket says 172.21.15.3:49752) 2022-07-21T01:41:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d77fe700 1 -- 172.21.15.3:0/1299373079 learned_addr learned my addr 172.21.15.3:0/1299373079 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d67fc700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80b01a0 0x7ff3d80aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d6ffd700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3d80a8c50 0x7ff3d81570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d77fe700 1 -- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80b01a0 msgr2=0x7ff3d80aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d77fe700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80b01a0 0x7ff3d80aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d77fe700 1 -- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3d80a8c50 msgr2=0x7ff3d81570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d77fe700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3d80a8c50 0x7ff3d81570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d77fe700 1 -- 172.21.15.3:0/1299373079 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c000b180 con 0x7ff3d80b2710 2022-07-21T01:41:27.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d67fc700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80b01a0 0x7ff3d80aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:27.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.828+0000 7ff3d6ffd700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3d80a8c50 0x7ff3d81570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:27.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3d77fe700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 0x7ff3d8065110 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7ff3c800a000 tx=0x7ff3c8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:27.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3cffff700 1 -- 172.21.15.3:0/1299373079 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3c8011780 con 0x7ff3d80b2710 2022-07-21T01:41:27.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3cffff700 1 -- 172.21.15.3:0/1299373079 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c80118e0 con 0x7ff3d80b2710 2022-07-21T01:41:27.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d8067ce0 con 0x7ff3d80b2710 2022-07-21T01:41:27.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3cffff700 1 -- 172.21.15.3:0/1299373079 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3c8011c40 con 0x7ff3d80b2710 2022-07-21T01:41:27.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d80aaf40 con 0x7ff3d80b2710 2022-07-21T01:41:27.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d80574e0 con 0x7ff3d80b2710 2022-07-21T01:41:27.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3cffff700 1 -- 172.21.15.3:0/1299373079 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff3c8012420 con 0x7ff3d80b2710 2022-07-21T01:41:27.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3cffff700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3bc053950 0x7ff3bc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:27.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3cffff700 1 -- 172.21.15.3:0/1299373079 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff3c806bed0 con 0x7ff3d80b2710 2022-07-21T01:41:27.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.832+0000 7ff3d6ffd700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3bc053950 0x7ff3bc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:27.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.836+0000 7ff3d6ffd700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3bc053950 0x7ff3bc055e10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7ff3c001a000 tx=0x7ff3c0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:27.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.840+0000 7ff3cffff700 1 -- 172.21.15.3:0/1299373079 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff3c8042b90 con 0x7ff3d80b2710 2022-07-21T01:41:27.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.964+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3d8062080 con 0x7ff3d80b2710 2022-07-21T01:41:27.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.964+0000 7ff3cffff700 1 -- 172.21.15.3:0/1299373079 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16298 (secure 0 0 0) 0x7ff3c80461b0 con 0x7ff3d80b2710 2022-07-21T01:41:27.971 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:27.971 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":120,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:27.699564+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:26.156392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:27.012266+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:27.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3bc053950 msgr2=0x7ff3bc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3bc053950 0x7ff3bc055e10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7ff3c001a000 tx=0x7ff3c0015000).stop 2022-07-21T01:41:27.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 msgr2=0x7ff3d8065110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:27.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 0x7ff3d8065110 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7ff3c800a000 tx=0x7ff3c8008040).stop 2022-07-21T01:41:27.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 shutdown_connections 2022-07-21T01:41:27.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3d80b2710 0x7ff3d8065110 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3d80b01a0 0x7ff3d80aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff3bc053950 0x7ff3bc055e10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 --2- 172.21.15.3:0/1299373079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3d80a8c50 0x7ff3d81570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:27.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 >> 172.21.15.3:0/1299373079 conn(0x7ff3d8060310 msgr2=0x7ff3d80af3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:27.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 shutdown_connections 2022-07-21T01:41:27.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:27.968+0000 7ff3d7fff700 1 -- 172.21.15.3:0/1299373079 wait complete. 2022-07-21T01:41:27.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:28.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.104+0000 7f1b51125700 1 Processor -- start 2022-07-21T01:41:28.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.104+0000 7f1b51125700 1 -- start start 2022-07-21T01:41:28.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1b4c0b08a0 0x7f1b4c142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4c143ab0 con 0x7f1b4c0ad000 2022-07-21T01:41:28.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4c0b0c80 con 0x7f1b4c0b3260 2022-07-21T01:41:28.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4c0ae810 con 0x7f1b4c0b08a0 2022-07-21T01:41:28.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1b4c0b08a0 0x7f1b4c142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b50924700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1b4c0ad000 0x7f1b4c13fa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58040/0 (socket says 172.21.15.3:58040) 2022-07-21T01:41:28.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1b4c0b08a0 0x7f1b4c142eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49756/0 (socket says 172.21.15.3:49756) 2022-07-21T01:41:28.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b50924700 1 -- 172.21.15.3:0/4290130273 learned_addr learned my addr 172.21.15.3:0/4290130273 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:28.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4b7fe700 1 -- 172.21.15.3:0/4290130273 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223853452 0 0) 0x7f1b4c143ab0 con 0x7f1b4c0ad000 2022-07-21T01:41:28.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4b7fe700 1 -- 172.21.15.3:0/4290130273 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b2c004370 con 0x7f1b4c0ad000 2022-07-21T01:41:28.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4bfff700 1 -- 172.21.15.3:0/4290130273 >> v1:172.21.15.143:6789/0 conn(0x7f1b4c0b3260 legacy=0x7f1b4c1433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4bfff700 1 -- 172.21.15.3:0/4290130273 >> v1:172.21.15.3:6789/0 conn(0x7f1b4c0ad000 legacy=0x7f1b4c13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4bfff700 1 -- 172.21.15.3:0/4290130273 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c0aeab0 con 0x7f1b4c0b08a0 2022-07-21T01:41:28.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4bfff700 1 --2- 172.21.15.3:0/4290130273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0b08a0 0x7f1b4c142eb0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f1b3c002f70 tx=0x7f1b3c006df0).ready entity=mon.0 client_cookie=32be8f94ae8f35c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:28.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4b7fe700 1 -- 172.21.15.3:0/4290130273 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b3c00d5c0 con 0x7f1b4c0b08a0 2022-07-21T01:41:28.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4b7fe700 1 -- 172.21.15.3:0/4290130273 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c000e90 con 0x7f1b4c0b08a0 2022-07-21T01:41:28.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b4b7fe700 1 -- 172.21.15.3:0/4290130273 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b3c00d910 con 0x7f1b4c0b08a0 2022-07-21T01:41:28.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 -- 172.21.15.3:0/4290130273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0b08a0 msgr2=0x7f1b4c142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 --2- 172.21.15.3:0/4290130273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0b08a0 0x7f1b4c142eb0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f1b3c002f70 tx=0x7f1b3c006df0).stop 2022-07-21T01:41:28.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 -- 172.21.15.3:0/4290130273 shutdown_connections 2022-07-21T01:41:28.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 --2- 172.21.15.3:0/4290130273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0b08a0 0x7f1b4c142eb0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 -- 172.21.15.3:0/4290130273 >> 172.21.15.3:0/4290130273 conn(0x7f1b4c0612d0 msgr2=0x7f1b4c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:28.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 -- 172.21.15.3:0/4290130273 shutdown_connections 2022-07-21T01:41:28.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.108+0000 7f1b51125700 1 -- 172.21.15.3:0/4290130273 wait complete. 2022-07-21T01:41:28.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 Processor -- start 2022-07-21T01:41:28.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 -- start start 2022-07-21T01:41:28.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0ad000 0x7f1b4c063f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b4c0b08a0 0x7f1b4c1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b4c0b3260 0x7f1b4c0abd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c065b40 con 0x7f1b4c0ad000 2022-07-21T01:41:28.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c064630 con 0x7f1b4c0b3260 2022-07-21T01:41:28.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c065d70 con 0x7f1b4c0b08a0 2022-07-21T01:41:28.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0ad000 0x7f1b4c063f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0ad000 0x7f1b4c063f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49762/0 (socket says 172.21.15.3:49762) 2022-07-21T01:41:28.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b4bfff700 1 -- 172.21.15.3:0/2408546789 learned_addr learned my addr 172.21.15.3:0/2408546789 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:28.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b50924700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b4c0b3260 0x7f1b4c0abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b437fe700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b4c0b08a0 0x7f1b4c1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b4bfff700 1 -- 172.21.15.3:0/2408546789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b4c0b08a0 msgr2=0x7f1b4c1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b4bfff700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b4c0b08a0 0x7f1b4c1530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b4bfff700 1 -- 172.21.15.3:0/2408546789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b4c0b3260 msgr2=0x7f1b4c0abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b4bfff700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b4c0b3260 0x7f1b4c0abd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b4bfff700 1 -- 172.21.15.3:0/2408546789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c00b180 con 0x7f1b4c0ad000 2022-07-21T01:41:28.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b4bfff700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0ad000 0x7f1b4c063f40 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7f1b3c002f70 tx=0x7f1b3c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:28.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b49ffb700 1 -- 172.21.15.3:0/2408546789 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b3c013de0 con 0x7f1b4c0ad000 2022-07-21T01:41:28.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b49ffb700 1 -- 172.21.15.3:0/2408546789 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c00de90 con 0x7f1b4c0ad000 2022-07-21T01:41:28.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b49ffb700 1 -- 172.21.15.3:0/2408546789 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b3c01e5f0 con 0x7f1b4c0ad000 2022-07-21T01:41:28.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4c066010 con 0x7f1b4c0ad000 2022-07-21T01:41:28.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.112+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4c1535c0 con 0x7f1b4c0ad000 2022-07-21T01:41:28.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.116+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4c0574e0 con 0x7f1b4c0ad000 2022-07-21T01:41:28.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.116+0000 7f1b49ffb700 1 -- 172.21.15.3:0/2408546789 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1b3c01c070 con 0x7f1b4c0ad000 2022-07-21T01:41:28.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.116+0000 7f1b49ffb700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1b30053aa0 0x7f1b30055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.116+0000 7f1b49ffb700 1 -- 172.21.15.3:0/2408546789 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1b3c06ca00 con 0x7f1b4c0ad000 2022-07-21T01:41:28.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.116+0000 7f1b437fe700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1b30053aa0 0x7f1b30055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.116+0000 7f1b437fe700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1b30053aa0 0x7f1b30055f60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f1b3400d000 tx=0x7f1b34007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:28.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.124+0000 7f1b49ffb700 1 -- 172.21.15.3:0/2408546789 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1b3c0437c0 con 0x7f1b4c0ad000 2022-07-21T01:41:28.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.252+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b4c0b2e00 con 0x7f1b4c0ad000 2022-07-21T01:41:28.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.252+0000 7f1b49ffb700 1 -- 172.21.15.3:0/2408546789 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16298 (secure 0 0 0) 0x7f1b3c023020 con 0x7f1b4c0ad000 2022-07-21T01:41:28.256 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:28.257 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":120,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:27.699564+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:26.156392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:27.012266+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:28.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1b30053aa0 msgr2=0x7f1b30055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1b30053aa0 0x7f1b30055f60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f1b3400d000 tx=0x7f1b34007b90).stop 2022-07-21T01:41:28.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0ad000 msgr2=0x7f1b4c063f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0ad000 0x7f1b4c063f40 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7f1b3c002f70 tx=0x7f1b3c014000).stop 2022-07-21T01:41:28.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 shutdown_connections 2022-07-21T01:41:28.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1b30053aa0 0x7f1b30055f60 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.262 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b4c0b3260 0x7f1b4c0abd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b4c0b08a0 0x7f1b4c1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 --2- 172.21.15.3:0/2408546789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c0ad000 0x7f1b4c063f40 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 >> 172.21.15.3:0/2408546789 conn(0x7f1b4c0612d0 msgr2=0x7f1b4c1421a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:28.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 shutdown_connections 2022-07-21T01:41:28.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.256+0000 7f1b51125700 1 -- 172.21.15.3:0/2408546789 wait complete. 2022-07-21T01:41:28.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:41:28.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.380+0000 7f8ce938e700 1 Processor -- start 2022-07-21T01:41:28.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.380+0000 7f8ce938e700 1 -- start start 2022-07-21T01:41:28.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8ce40b5630 0x7f8ce40af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ce40b07a0 0x7f8ce4147730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 0x7f8ce40acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce4145470 con 0x7f8ce40b5630 2022-07-21T01:41:28.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce40ae180 con 0x7f8ce40b07a0 2022-07-21T01:41:28.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce40b5a10 con 0x7f8ce4067730 2022-07-21T01:41:28.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8ce40b5630 0x7f8ce40af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8ce40b5630 0x7f8ce40af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49772/0 (socket says 172.21.15.3:49772) 2022-07-21T01:41:28.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8cdbfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ce40b07a0 0x7f8ce4147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce3fff700 1 -- 172.21.15.3:0/4222567717 learned_addr learned my addr 172.21.15.3:0/4222567717 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:28.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce8b8d700 1 --2- 172.21.15.3:0/4222567717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 0x7f8ce40acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce3fff700 1 -- 172.21.15.3:0/4222567717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 msgr2=0x7f8ce40acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:28.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce3fff700 1 --2- 172.21.15.3:0/4222567717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 0x7f8ce40acdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce3fff700 1 -- 172.21.15.3:0/4222567717 >> v2:172.21.15.143:3300/0 conn(0x7f8ce40b07a0 msgr2=0x7f8ce4147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:28.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce3fff700 1 --2- 172.21.15.3:0/4222567717 >> v2:172.21.15.143:3300/0 conn(0x7f8ce40b07a0 0x7f8ce4147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce3fff700 1 -- 172.21.15.3:0/4222567717 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce40ac990 con 0x7f8ce40b5630 2022-07-21T01:41:28.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce3fff700 1 --2- 172.21.15.3:0/4222567717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 0x7f8ce40af6e0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0x7f8cd4002f70 tx=0x7f8cd4006eb0).ready entity=mon.0 client_cookie=7d312c0659bfd60a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:28.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce37fe700 1 -- 172.21.15.3:0/4222567717 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cd400d570 con 0x7f8ce40b5630 2022-07-21T01:41:28.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce37fe700 1 -- 172.21.15.3:0/4222567717 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd4000e90 con 0x7f8ce40b5630 2022-07-21T01:41:28.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce37fe700 1 -- 172.21.15.3:0/4222567717 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cd400d900 con 0x7f8ce40b5630 2022-07-21T01:41:28.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 -- 172.21.15.3:0/4222567717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 msgr2=0x7f8ce40af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 --2- 172.21.15.3:0/4222567717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 0x7f8ce40af6e0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0x7f8cd4002f70 tx=0x7f8cd4006eb0).stop 2022-07-21T01:41:28.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 -- 172.21.15.3:0/4222567717 shutdown_connections 2022-07-21T01:41:28.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 --2- 172.21.15.3:0/4222567717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 0x7f8ce40acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 --2- 172.21.15.3:0/4222567717 >> v2:172.21.15.143:3300/0 conn(0x7f8ce40b07a0 0x7f8ce4147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 --2- 172.21.15.3:0/4222567717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 0x7f8ce40af6e0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 -- 172.21.15.3:0/4222567717 >> 172.21.15.3:0/4222567717 conn(0x7f8ce40626f0 msgr2=0x7f8ce4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:28.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 -- 172.21.15.3:0/4222567717 shutdown_connections 2022-07-21T01:41:28.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.384+0000 7f8ce938e700 1 -- 172.21.15.3:0/4222567717 wait complete. 2022-07-21T01:41:28.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 Processor -- start 2022-07-21T01:41:28.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 -- start start 2022-07-21T01:41:28.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 0x7f8ce40b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ce40b07a0 0x7f8ce4156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 0x7f8ce4157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce414f270 con 0x7f8ce40b5630 2022-07-21T01:41:28.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce41476c0 con 0x7f8ce40b07a0 2022-07-21T01:41:28.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce4065af0 con 0x7f8ce4067730 2022-07-21T01:41:28.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce8b8d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 0x7f8ce4157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce8b8d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 0x7f8ce4157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49780/0 (socket says 172.21.15.3:49780) 2022-07-21T01:41:28.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce8b8d700 1 -- 172.21.15.3:0/2586527515 learned_addr learned my addr 172.21.15.3:0/2586527515 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:28.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8cdbfff700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ce40b07a0 0x7f8ce4156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce3fff700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 0x7f8ce40b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce8b8d700 1 -- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 msgr2=0x7f8ce40b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce8b8d700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 0x7f8ce40b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce8b8d700 1 -- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ce40b07a0 msgr2=0x7f8ce4156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce8b8d700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ce40b07a0 0x7f8ce4156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce8b8d700 1 -- 172.21.15.3:0/2586527515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd400b1e0 con 0x7f8ce40b5630 2022-07-21T01:41:28.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8cdbfff700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ce40b07a0 0x7f8ce4156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:28.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce3fff700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 0x7f8ce40b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:28.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce8b8d700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 0x7f8ce4157360 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7f8cdc006e00 tx=0x7f8cdc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:28.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce1ffb700 1 -- 172.21.15.3:0/2586527515 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cdc00c850 con 0x7f8ce40b5630 2022-07-21T01:41:28.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce1ffb700 1 -- 172.21.15.3:0/2586527515 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc00c9b0 con 0x7f8ce40b5630 2022-07-21T01:41:28.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce1ffb700 1 -- 172.21.15.3:0/2586527515 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cdc00cd10 con 0x7f8ce40b5630 2022-07-21T01:41:28.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce4147420 con 0x7f8ce40b5630 2022-07-21T01:41:28.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce4066fe0 con 0x7f8ce40b5630 2022-07-21T01:41:28.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.388+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce40b1680 con 0x7f8ce40b5630 2022-07-21T01:41:28.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.392+0000 7f8ce1ffb700 1 -- 172.21.15.3:0/2586527515 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8cdc0123f0 con 0x7f8ce40b5630 2022-07-21T01:41:28.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.392+0000 7f8ce1ffb700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8cc8053950 0x7f8cc8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.392+0000 7f8ce1ffb700 1 -- 172.21.15.3:0/2586527515 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8cdc06c360 con 0x7f8ce40b5630 2022-07-21T01:41:28.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.392+0000 7f8ce3fff700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8cc8053950 0x7f8cc8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.392+0000 7f8ce3fff700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8cc8053950 0x7f8cc8055e10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f8cd401a000 tx=0x7f8cd4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:28.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.400+0000 7f8ce1ffb700 1 -- 172.21.15.3:0/2586527515 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8cdc0430c0 con 0x7f8ce40b5630 2022-07-21T01:41:28.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.564+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8ce40574e0 con 0x7f8ce40b5630 2022-07-21T01:41:28.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.720+0000 7f8ce1ffb700 1 -- 172.21.15.3:0/2586527515 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8cdc0466e0 con 0x7f8ce40b5630 2022-07-21T01:41:28.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.720+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8cc8053950 msgr2=0x7f8cc8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.720+0000 7f8ce938e700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8cc8053950 0x7f8cc8055e10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f8cd401a000 tx=0x7f8cd4015000).stop 2022-07-21T01:41:28.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.720+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 msgr2=0x7f8ce4157360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.720+0000 7f8ce938e700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 0x7f8ce4157360 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7f8cdc006e00 tx=0x7f8cdc0091d0).stop 2022-07-21T01:41:28.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.724+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 shutdown_connections 2022-07-21T01:41:28.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.724+0000 7f8ce938e700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ce40b5630 0x7f8ce4157360 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.724+0000 7f8ce938e700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8cc8053950 0x7f8cc8055e10 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.724+0000 7f8ce938e700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ce40b07a0 0x7f8ce4156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.724+0000 7f8ce938e700 1 --2- 172.21.15.3:0/2586527515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ce4067730 0x7f8ce40b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.724+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 >> 172.21.15.3:0/2586527515 conn(0x7f8ce40626f0 msgr2=0x7f8ce4148aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:28.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.724+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 shutdown_connections 2022-07-21T01:41:28.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.724+0000 7f8ce938e700 1 -- 172.21.15.3:0/2586527515 wait complete. 2022-07-21T01:41:28.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:28.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa55b59e700 1 Processor -- start 2022-07-21T01:41:28.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa55b59e700 1 -- start start 2022-07-21T01:41:28.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa55b59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa55c0a9da0 0x7fa55c060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa55b59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa55c067030 con 0x7fa55c0b31d0 2022-07-21T01:41:28.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa55b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa55c0b35b0 con 0x7fa55c1402d0 2022-07-21T01:41:28.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa55b59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa55c064a70 con 0x7fa55c0a9da0 2022-07-21T01:41:28.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa559d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa55c0a9da0 0x7fa55c060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa559d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa55c0a9da0 0x7fa55c060950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49786/0 (socket says 172.21.15.3:49786) 2022-07-21T01:41:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa559d9b700 1 -- 172.21.15.3:0/2554095637 learned_addr learned my addr 172.21.15.3:0/2554095637 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa55a59c700 1 --1- 172.21.15.3:0/2554095637 >> v1:172.21.15.3:6789/0 conn(0x7fa55c0b31d0 0x7fa55c066020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58064/0 (socket says 172.21.15.3:58064) 2022-07-21T01:41:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa559d9b700 1 -- 172.21.15.3:0/2554095637 >> v1:172.21.15.143:6789/0 conn(0x7fa55c1402d0 legacy=0x7fa55c140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa559d9b700 1 -- 172.21.15.3:0/2554095637 >> v1:172.21.15.3:6789/0 conn(0x7fa55c0b31d0 legacy=0x7fa55c066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.848+0000 7fa559d9b700 1 -- 172.21.15.3:0/2554095637 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c066af0 con 0x7fa55c0a9da0 2022-07-21T01:41:28.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa559d9b700 1 --2- 172.21.15.3:0/2554095637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 0x7fa55c060950 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7fa55c068010 tx=0x7fa550006eb0).ready entity=mon.0 client_cookie=9f9b741b2f5c11b6 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:28.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55959a700 1 -- 172.21.15.3:0/2554095637 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa55000d610 con 0x7fa55c0a9da0 2022-07-21T01:41:28.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55959a700 1 -- 172.21.15.3:0/2554095637 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa550000e90 con 0x7fa55c0a9da0 2022-07-21T01:41:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55959a700 1 -- 172.21.15.3:0/2554095637 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa55000d960 con 0x7fa55c0a9da0 2022-07-21T01:41:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55b59e700 1 -- 172.21.15.3:0/2554095637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 msgr2=0x7fa55c060950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55b59e700 1 --2- 172.21.15.3:0/2554095637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 0x7fa55c060950 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7fa55c068010 tx=0x7fa550006eb0).stop 2022-07-21T01:41:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55b59e700 1 -- 172.21.15.3:0/2554095637 shutdown_connections 2022-07-21T01:41:28.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55b59e700 1 --2- 172.21.15.3:0/2554095637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 0x7fa55c060950 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55b59e700 1 -- 172.21.15.3:0/2554095637 >> 172.21.15.3:0/2554095637 conn(0x7fa55c063870 msgr2=0x7fa55c063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:28.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55b59e700 1 -- 172.21.15.3:0/2554095637 shutdown_connections 2022-07-21T01:41:28.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55b59e700 1 -- 172.21.15.3:0/2554095637 wait complete. 2022-07-21T01:41:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55b59e700 1 Processor -- start 2022-07-21T01:41:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.852+0000 7fa55b59e700 1 -- start start 2022-07-21T01:41:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 0x7fa55c0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c0b31d0 0x7fa55c1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa55c1402d0 0x7fa55c0acc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c0ab850 con 0x7fa55c0a9da0 2022-07-21T01:41:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c1433e0 con 0x7fa55c0b31d0 2022-07-21T01:41:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c142ea0 con 0x7fa55c1402d0 2022-07-21T01:41:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 0x7fa55c0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 0x7fa55c0b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49790/0 (socket says 172.21.15.3:49790) 2022-07-21T01:41:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55a59c700 1 -- 172.21.15.3:0/2434900979 learned_addr learned my addr 172.21.15.3:0/2434900979 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa559d9b700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c0b31d0 0x7fa55c1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55ad9d700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa55c1402d0 0x7fa55c0acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55a59c700 1 -- 172.21.15.3:0/2434900979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa55c1402d0 msgr2=0x7fa55c0acc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55a59c700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa55c1402d0 0x7fa55c0acc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55a59c700 1 -- 172.21.15.3:0/2434900979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c0b31d0 msgr2=0x7fa55c1424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55a59c700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c0b31d0 0x7fa55c1424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55a59c700 1 -- 172.21.15.3:0/2434900979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55000b1e0 con 0x7fa55c0a9da0 2022-07-21T01:41:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55a59c700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 0x7fa55c0b45d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7fa55c0634e0 tx=0x7fa54c007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa54b7fe700 1 -- 172.21.15.3:0/2434900979 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa54c0070d0 con 0x7fa55c0a9da0 2022-07-21T01:41:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa54b7fe700 1 -- 172.21.15.3:0/2434900979 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54c007230 con 0x7fa55c0a9da0 2022-07-21T01:41:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55c143140 con 0x7fa55c0a9da0 2022-07-21T01:41:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa54b7fe700 1 -- 172.21.15.3:0/2434900979 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa54c005740 con 0x7fa55c0a9da0 2022-07-21T01:41:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55c0ab540 con 0x7fa55c0a9da0 2022-07-21T01:41:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.856+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa55c066d90 con 0x7fa55c0a9da0 2022-07-21T01:41:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.860+0000 7fa54b7fe700 1 -- 172.21.15.3:0/2434900979 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa54c010070 con 0x7fa55c0a9da0 2022-07-21T01:41:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.860+0000 7fa54b7fe700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa538053980 0x7fa538055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.860+0000 7fa54b7fe700 1 -- 172.21.15.3:0/2434900979 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa54c06b470 con 0x7fa55c0a9da0 2022-07-21T01:41:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.860+0000 7fa559d9b700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa538053980 0x7fa538055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.860+0000 7fa559d9b700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa538053980 0x7fa538055e40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fa55001c000 tx=0x7fa550015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:28.864+0000 7fa54b7fe700 1 -- 172.21.15.3:0/2434900979 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa54c042230 con 0x7fa55c0a9da0 2022-07-21T01:41:29.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.032+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa55c0574e0 con 0x7fa55c0a9da0 2022-07-21T01:41:29.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.032+0000 7fa54b7fe700 1 -- 172.21.15.3:0/2434900979 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+2486 (secure 0 0 0) 0x7fa54c018070 con 0x7fa55c0a9da0 2022-07-21T01:41:29.037 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:29.038 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":60,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/297776761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":297776761},{"type":"v1","addr":"172.21.15.143:6833","nonce":297776761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":57},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3673188672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3673188672},{"type":"v1","addr":"172.21.15.3:6835","nonce":3673188672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:27.699850+0000","modified":"2022-07-21T01:41:28.704288+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7611},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7611":{"gid":7611,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/508489181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":508489181},{"type":"v1","addr":"172.21.15.3:6837","nonce":508489181}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-21T01:41:29.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa538053980 msgr2=0x7fa538055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa538053980 0x7fa538055e40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fa55001c000 tx=0x7fa550015000).stop 2022-07-21T01:41:29.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 msgr2=0x7fa55c0b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 0x7fa55c0b45d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7fa55c0634e0 tx=0x7fa54c007c00).stop 2022-07-21T01:41:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 shutdown_connections 2022-07-21T01:41:29.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa55c1402d0 0x7fa55c0acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa538053980 0x7fa538055e40 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c0b31d0 0x7fa55c1424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 --2- 172.21.15.3:0/2434900979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa55c0a9da0 0x7fa55c0b45d0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 >> 172.21.15.3:0/2434900979 conn(0x7fa55c063870 msgr2=0x7fa55c13f760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:29.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 shutdown_connections 2022-07-21T01:41:29.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.036+0000 7fa55b59e700 1 -- 172.21.15.3:0/2434900979 wait complete. 2022-07-21T01:41:29.050 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 60 2022-07-21T01:41:29.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:29.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd7a49f2700 1 Processor -- start 2022-07-21T01:41:29.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd7a49f2700 1 -- start start 2022-07-21T01:41:29.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd7a49f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00b01a0 0x7fd7a00a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd7a49f2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a00a7250 con 0x7fd7a00b2710 2022-07-21T01:41:29.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd7a49f2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a00a7480 con 0x7fd7a00afd60 2022-07-21T01:41:29.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd7a49f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a00a7720 con 0x7fd7a00b01a0 2022-07-21T01:41:29.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd7a00b2710 0x7fd7a014afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58080/0 (socket says 172.21.15.3:58080) 2022-07-21T01:41:29.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79f7fe700 1 -- 172.21.15.3:0/3039052013 learned_addr learned my addr 172.21.15.3:0/3039052013 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:29.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79effd700 1 --2- 172.21.15.3:0/3039052013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00b01a0 0x7fd7a00a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79e7fc700 1 -- 172.21.15.3:0/3039052013 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340217920 0 0) 0x7fd7a00a7250 con 0x7fd7a00b2710 2022-07-21T01:41:29.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79e7fc700 1 -- 172.21.15.3:0/3039052013 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78c003260 con 0x7fd7a00b2710 2022-07-21T01:41:29.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79effd700 1 -- 172.21.15.3:0/3039052013 >> v1:172.21.15.143:6789/0 conn(0x7fd7a00afd60 legacy=0x7fd7a0151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79effd700 1 -- 172.21.15.3:0/3039052013 >> v1:172.21.15.3:6789/0 conn(0x7fd7a00b2710 legacy=0x7fd7a014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79effd700 1 -- 172.21.15.3:0/3039052013 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a00a79c0 con 0x7fd7a00b01a0 2022-07-21T01:41:29.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79effd700 1 --2- 172.21.15.3:0/3039052013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00b01a0 0x7fd7a00a8540 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd790002f70 tx=0x7fd790006df0).ready entity=mon.2 client_cookie=7bbcc402dcdff909 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:29.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79e7fc700 1 -- 172.21.15.3:0/3039052013 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd79000d570 con 0x7fd7a00b01a0 2022-07-21T01:41:29.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79e7fc700 1 -- 172.21.15.3:0/3039052013 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd790000e90 con 0x7fd7a00b01a0 2022-07-21T01:41:29.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.160+0000 7fd79e7fc700 1 -- 172.21.15.3:0/3039052013 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd79000d8c0 con 0x7fd7a00b01a0 2022-07-21T01:41:29.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- 172.21.15.3:0/3039052013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00b01a0 msgr2=0x7fd7a00a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 --2- 172.21.15.3:0/3039052013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00b01a0 0x7fd7a00a8540 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd790002f70 tx=0x7fd790006df0).stop 2022-07-21T01:41:29.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- 172.21.15.3:0/3039052013 shutdown_connections 2022-07-21T01:41:29.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 --2- 172.21.15.3:0/3039052013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00b01a0 0x7fd7a00a8540 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- 172.21.15.3:0/3039052013 >> 172.21.15.3:0/3039052013 conn(0x7fd7a0060310 msgr2=0x7fd7a00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:29.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- 172.21.15.3:0/3039052013 shutdown_connections 2022-07-21T01:41:29.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- 172.21.15.3:0/3039052013 wait complete. 2022-07-21T01:41:29.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 Processor -- start 2022-07-21T01:41:29.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- start start 2022-07-21T01:41:29.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00afd60 0x7fd7a015a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a00b01a0 0x7fd7a00aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7a00b2710 0x7fd7a01569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a015ac30 con 0x7fd7a00b2710 2022-07-21T01:41:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a014d150 con 0x7fd7a00b01a0 2022-07-21T01:41:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a014d3f0 con 0x7fd7a00afd60 2022-07-21T01:41:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7a00b2710 0x7fd7a01569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7a00b2710 0x7fd7a01569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49808/0 (socket says 172.21.15.3:49808) 2022-07-21T01:41:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79ffff700 1 -- 172.21.15.3:0/2397127495 learned_addr learned my addr 172.21.15.3:0/2397127495 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:29.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79effd700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a00b01a0 0x7fd7a00aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79f7fe700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00afd60 0x7fd7a015a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79ffff700 1 -- 172.21.15.3:0/2397127495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00afd60 msgr2=0x7fd7a015a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79ffff700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00afd60 0x7fd7a015a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79ffff700 1 -- 172.21.15.3:0/2397127495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a00b01a0 msgr2=0x7fd7a00aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79ffff700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a00b01a0 0x7fd7a00aefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79ffff700 1 -- 172.21.15.3:0/2397127495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79000b180 con 0x7fd7a00b2710 2022-07-21T01:41:29.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79f7fe700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00afd60 0x7fd7a015a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:29.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79ffff700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7a00b2710 0x7fd7a01569b0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7fd7a0062040 tx=0x7fd798007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:29.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79cff9700 1 -- 172.21.15.3:0/2397127495 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd79800e4a0 con 0x7fd7a00b2710 2022-07-21T01:41:29.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79cff9700 1 -- 172.21.15.3:0/2397127495 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79800e600 con 0x7fd7a00b2710 2022-07-21T01:41:29.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a014d690 con 0x7fd7a00b2710 2022-07-21T01:41:29.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd79cff9700 1 -- 172.21.15.3:0/2397127495 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd79800e960 con 0x7fd7a00b2710 2022-07-21T01:41:29.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.164+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a00b3980 con 0x7fd7a00b2710 2022-07-21T01:41:29.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.168+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a00574e0 con 0x7fd7a00b2710 2022-07-21T01:41:29.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.168+0000 7fd79cff9700 1 -- 172.21.15.3:0/2397127495 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd798019030 con 0x7fd7a00b2710 2022-07-21T01:41:29.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.168+0000 7fd79cff9700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd780053aa0 0x7fd780055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.168+0000 7fd79cff9700 1 -- 172.21.15.3:0/2397127495 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd79806b880 con 0x7fd7a00b2710 2022-07-21T01:41:29.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.168+0000 7fd79f7fe700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd780053aa0 0x7fd780055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.168+0000 7fd79f7fe700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd780053aa0 0x7fd780055f60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7fd79400b000 tx=0x7fd794009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:29.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.176+0000 7fd79cff9700 1 -- 172.21.15.3:0/2397127495 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd798042450 con 0x7fd7a00b2710 2022-07-21T01:41:29.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.304+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7a006a5e0 con 0x7fd7a00b2710 2022-07-21T01:41:29.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.304+0000 7fd79cff9700 1 -- 172.21.15.3:0/2397127495 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16285 (secure 0 0 0) 0x7fd798045a70 con 0x7fd7a00b2710 2022-07-21T01:41:29.307 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:29.308 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":121,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:28.703604+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:26.156392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:27.012266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:29.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd780053aa0 msgr2=0x7fd780055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd780053aa0 0x7fd780055f60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7fd79400b000 tx=0x7fd794009040).stop 2022-07-21T01:41:29.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7a00b2710 msgr2=0x7fd7a01569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7a00b2710 0x7fd7a01569b0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7fd7a0062040 tx=0x7fd798007f40).stop 2022-07-21T01:41:29.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 shutdown_connections 2022-07-21T01:41:29.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7a00b2710 0x7fd7a01569b0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd780053aa0 0x7fd780055f60 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a00b01a0 0x7fd7a00aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 --2- 172.21.15.3:0/2397127495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7a00afd60 0x7fd7a015a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 >> 172.21.15.3:0/2397127495 conn(0x7fd7a0060310 msgr2=0x7fd7a014a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:29.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 shutdown_connections 2022-07-21T01:41:29.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.308+0000 7fd7a49f2700 1 -- 172.21.15.3:0/2397127495 wait complete. 2022-07-21T01:41:29.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:41:29.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.432+0000 7f2ede3ba700 1 Processor -- start 2022-07-21T01:41:29.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.432+0000 7f2ede3ba700 1 -- start start 2022-07-21T01:41:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.432+0000 7f2ede3ba700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2ed80b0d10 0x7f2ed8146df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.432+0000 7f2ede3ba700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed80b12d0 con 0x7f2ed80b0560 2022-07-21T01:41:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.432+0000 7f2ede3ba700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed80a9700 con 0x7f2ed80a9320 2022-07-21T01:41:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.432+0000 7f2ede3ba700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed80b1500 con 0x7f2ed80b0d10 2022-07-21T01:41:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.432+0000 7f2eddbb9700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2ed80b0560 0x7f2ed8140350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58096/0 (socket says 172.21.15.3:58096) 2022-07-21T01:41:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2eddbb9700 1 -- 172.21.15.3:0/3909865828 learned_addr learned my addr 172.21.15.3:0/3909865828 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2edcbb7700 1 --2- 172.21.15.3:0/3909865828 >> v2:172.21.15.143:3300/0 conn(0x7f2ed80b0d10 0x7f2ed8146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ecffff700 1 -- 172.21.15.3:0/3909865828 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532155068 0 0) 0x7f2ed80b12d0 con 0x7f2ed80b0560 2022-07-21T01:41:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ecffff700 1 -- 172.21.15.3:0/3909865828 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ebc003260 con 0x7f2ed80b0560 2022-07-21T01:41:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ecffff700 1 -- 172.21.15.3:0/3909865828 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134882845 0 0) 0x7f2ed80a9700 con 0x7f2ed80a9320 2022-07-21T01:41:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ecffff700 1 -- 172.21.15.3:0/3909865828 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed80b12d0 con 0x7f2ed80a9320 2022-07-21T01:41:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2edcbb7700 1 -- 172.21.15.3:0/3909865828 >> v1:172.21.15.143:6789/0 conn(0x7f2ed80a9320 legacy=0x7f2ed81466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2edcbb7700 1 -- 172.21.15.3:0/3909865828 >> v1:172.21.15.3:6789/0 conn(0x7f2ed80b0560 legacy=0x7f2ed8140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ecffff700 1 -- 172.21.15.3:0/3909865828 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2016080748 0 0) 0x7f2ebc003260 con 0x7f2ed80b0560 2022-07-21T01:41:29.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2edcbb7700 1 -- 172.21.15.3:0/3909865828 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed80b17a0 con 0x7f2ed80b0d10 2022-07-21T01:41:29.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2edcbb7700 1 --2- 172.21.15.3:0/3909865828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 0x7f2ed8146df0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f2ed0001490 tx=0x7f2ec8006df0).ready entity=mon.1 client_cookie=5dd44b3dc1dfe0c4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:29.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ecffff700 1 -- 172.21.15.3:0/3909865828 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ed00019e0 con 0x7f2ed80b0d10 2022-07-21T01:41:29.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ecffff700 1 -- 172.21.15.3:0/3909865828 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec8000e90 con 0x7f2ed80b0d10 2022-07-21T01:41:29.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ecffff700 1 -- 172.21.15.3:0/3909865828 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ec8012460 con 0x7f2ed80b0d10 2022-07-21T01:41:29.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ede3ba700 1 -- 172.21.15.3:0/3909865828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 msgr2=0x7f2ed8146df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ede3ba700 1 --2- 172.21.15.3:0/3909865828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 0x7f2ed8146df0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f2ed0001490 tx=0x7f2ec8006df0).stop 2022-07-21T01:41:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ede3ba700 1 -- 172.21.15.3:0/3909865828 shutdown_connections 2022-07-21T01:41:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ede3ba700 1 --2- 172.21.15.3:0/3909865828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 0x7f2ed8146df0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ede3ba700 1 -- 172.21.15.3:0/3909865828 >> 172.21.15.3:0/3909865828 conn(0x7f2ed80612d0 msgr2=0x7f2ed8062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ede3ba700 1 -- 172.21.15.3:0/3909865828 shutdown_connections 2022-07-21T01:41:29.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.436+0000 7f2ede3ba700 1 -- 172.21.15.3:0/3909865828 wait complete. 2022-07-21T01:41:29.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 Processor -- start 2022-07-21T01:41:29.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 -- start start 2022-07-21T01:41:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ed80a9320 0x7f2ed80aa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ed80b0560 0x7f2ed80af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 0x7f2ed80abba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed813f8b0 con 0x7f2ed80a9320 2022-07-21T01:41:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed813fd80 con 0x7f2ed80b0d10 2022-07-21T01:41:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed813fae0 con 0x7f2ed80b0560 2022-07-21T01:41:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edd3b8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ed80a9320 0x7f2ed80aa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edd3b8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ed80a9320 0x7f2ed80aa3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49818/0 (socket says 172.21.15.3:49818) 2022-07-21T01:41:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edd3b8700 1 -- 172.21.15.3:0/2992801562 learned_addr learned my addr 172.21.15.3:0/2992801562 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2eddbb9700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 0x7f2ed80abba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edcbb7700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ed80b0560 0x7f2ed80af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edd3b8700 1 -- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ed80b0560 msgr2=0x7f2ed80af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edd3b8700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ed80b0560 0x7f2ed80af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edd3b8700 1 -- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 msgr2=0x7f2ed80abba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edd3b8700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 0x7f2ed80abba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edd3b8700 1 -- 172.21.15.3:0/2992801562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec800b180 con 0x7f2ed80a9320 2022-07-21T01:41:29.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2eddbb9700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 0x7f2ed80abba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edcbb7700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ed80b0560 0x7f2ed80af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2edd3b8700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ed80a9320 0x7f2ed80aa3c0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7f2ed4001110 tx=0x7f2ed4007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ece7fc700 1 -- 172.21.15.3:0/2992801562 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ed400ce10 con 0x7f2ed80a9320 2022-07-21T01:41:29.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ece7fc700 1 -- 172.21.15.3:0/2992801562 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed400f470 con 0x7f2ed80a9320 2022-07-21T01:41:29.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed813ee80 con 0x7f2ed80a9320 2022-07-21T01:41:29.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ece7fc700 1 -- 172.21.15.3:0/2992801562 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ed400f7d0 con 0x7f2ed80a9320 2022-07-21T01:41:29.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.440+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed813e760 con 0x7f2ed80a9320 2022-07-21T01:41:29.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.444+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed80574e0 con 0x7f2ed80a9320 2022-07-21T01:41:29.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.444+0000 7f2ece7fc700 1 -- 172.21.15.3:0/2992801562 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2ed4019030 con 0x7f2ed80a9320 2022-07-21T01:41:29.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.444+0000 7f2ece7fc700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ec0053b00 0x7f2ec0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.444+0000 7f2ece7fc700 1 -- 172.21.15.3:0/2992801562 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2ed406b490 con 0x7f2ed80a9320 2022-07-21T01:41:29.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.444+0000 7f2edcbb7700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ec0053b00 0x7f2ec0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.448+0000 7f2edcbb7700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ec0053b00 0x7f2ec0055fc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f2ec8015000 tx=0x7f2ec8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:29.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.456+0000 7f2ece7fc700 1 -- 172.21.15.3:0/2992801562 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2ed4042250 con 0x7f2ed80a9320 2022-07-21T01:41:29.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.620+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2ed8069a70 con 0x7f2ed80a9320 2022-07-21T01:41:29.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.628+0000 7f2ece7fc700 1 -- 172.21.15.3:0/2992801562 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v32) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2ed4045870 con 0x7f2ed80a9320 2022-07-21T01:41:29.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.628+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ec0053b00 msgr2=0x7f2ec0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.628+0000 7f2ede3ba700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ec0053b00 0x7f2ec0055fc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f2ec8015000 tx=0x7f2ec8014000).stop 2022-07-21T01:41:29.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.628+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ed80a9320 msgr2=0x7f2ed80aa3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.628+0000 7f2ede3ba700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ed80a9320 0x7f2ed80aa3c0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7f2ed4001110 tx=0x7f2ed4007ae0).stop 2022-07-21T01:41:29.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.632+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 shutdown_connections 2022-07-21T01:41:29.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.632+0000 7f2ede3ba700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ec0053b00 0x7f2ec0055fc0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.632+0000 7f2ede3ba700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ed80b0d10 0x7f2ed80abba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.632+0000 7f2ede3ba700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ed80b0560 0x7f2ed80af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.632+0000 7f2ede3ba700 1 --2- 172.21.15.3:0/2992801562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ed80a9320 0x7f2ed80aa3c0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.632+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 >> 172.21.15.3:0/2992801562 conn(0x7f2ed80612d0 msgr2=0x7f2ed813c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:29.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.632+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 shutdown_connections 2022-07-21T01:41:29.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.632+0000 7f2ede3ba700 1 -- 172.21.15.3:0/2992801562 wait complete. 2022-07-21T01:41:29.636 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:41:29.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:41:29.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcc50d700 1 Processor -- start 2022-07-21T01:41:29.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcc50d700 1 -- start start 2022-07-21T01:41:29.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcc50d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc40b5d80 0x7f8fc40b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcc50d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc4144240 con 0x7f8fc40b4550 2022-07-21T01:41:29.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcc50d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc40b6fe0 con 0x7f8fc40b6710 2022-07-21T01:41:29.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcc50d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc4143d00 con 0x7f8fc40b5d80 2022-07-21T01:41:29.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc9aa8700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8fc40b4550 0x7f8fc412bb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58108/0 (socket says 172.21.15.3:58108) 2022-07-21T01:41:29.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc9aa8700 1 -- 172.21.15.3:0/3092812283 learned_addr learned my addr 172.21.15.3:0/3092812283 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:29.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcaaaa700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc40b5d80 0x7f8fc40b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859948058 0 0) 0x7f8fc4144240 con 0x7f8fc40b4550 2022-07-21T01:41:29.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fac003260 con 0x7f8fc40b4550 2022-07-21T01:41:29.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569939367 0 0) 0x7f8fc40b6fe0 con 0x7f8fc40b6710 2022-07-21T01:41:29.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc4144240 con 0x7f8fc40b6710 2022-07-21T01:41:29.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3479347216 0 0) 0x7f8fac003260 con 0x7f8fc40b4550 2022-07-21T01:41:29.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc40b5d80 msgr2=0x7f8fc40b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:29.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc40b5d80 0x7f8fc40b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 >> v1:172.21.15.143:6789/0 conn(0x7f8fc40b6710 legacy=0x7f8fc4061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4143fa0 con 0x7f8fc40b4550 2022-07-21T01:41:29.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f8fc00019e0 con 0x7f8fc40b4550 2022-07-21T01:41:29.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 >> v1:172.21.15.3:6789/0 conn(0x7f8fc40b4550 legacy=0x7f8fc412bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc40b5d80 0x7f8fac003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fac003e20 0x7f8fac004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fac0049f0 0x7f8fac004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc4143d00 con 0x7f8fac0049f0 2022-07-21T01:41:29.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac00b780 con 0x7f8fac003e20 2022-07-21T01:41:29.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac00b940 con 0x7f8fc40b5d80 2022-07-21T01:41:29.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcaaaa700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fac0049f0 0x7f8fac004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fca2a9700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc40b5d80 0x7f8fac003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc9aa8700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fac003e20 0x7f8fac004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcaaaa700 1 -- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc40b5d80 msgr2=0x7f8fac003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:29.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcaaaa700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc40b5d80 0x7f8fac003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcaaaa700 1 -- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fac003e20 msgr2=0x7f8fac004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:29.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcaaaa700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fac003e20 0x7f8fac004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fcaaaa700 1 -- 172.21.15.3:0/3092812283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4143fa0 con 0x7f8fac0049f0 2022-07-21T01:41:29.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.768+0000 7f8fc9aa8700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fac003e20 0x7f8fac004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:29.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcaaaa700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fac0049f0 0x7f8fac004e70 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f8fbc00c000 tx=0x7f8fbc006df0).ready entity=mon.0 client_cookie=6ef09034ab5646b7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:29.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fbc00e830 con 0x7f8fac0049f0 2022-07-21T01:41:29.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc00e990 con 0x7f8fac0049f0 2022-07-21T01:41:29.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fc92a7700 1 -- 172.21.15.3:0/3092812283 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fbc00ecf0 con 0x7f8fac0049f0 2022-07-21T01:41:29.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcc50d700 1 -- 172.21.15.3:0/3092812283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fac0049f0 msgr2=0x7f8fac004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcc50d700 1 --2- 172.21.15.3:0/3092812283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fac0049f0 0x7f8fac004e70 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f8fbc00c000 tx=0x7f8fbc006df0).stop 2022-07-21T01:41:29.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcaaaa700 1 -- 172.21.15.3:0/3092812283 reap_dead start 2022-07-21T01:41:29.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcc50d700 1 -- 172.21.15.3:0/3092812283 shutdown_connections 2022-07-21T01:41:29.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcc50d700 1 -- 172.21.15.3:0/3092812283 >> 172.21.15.3:0/3092812283 conn(0x7f8fc405c620 msgr2=0x7f8fc405ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:29.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcc50d700 1 -- 172.21.15.3:0/3092812283 shutdown_connections 2022-07-21T01:41:29.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcc50d700 1 -- 172.21.15.3:0/3092812283 wait complete. 2022-07-21T01:41:29.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcc50d700 1 Processor -- start 2022-07-21T01:41:29.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.772+0000 7f8fcc50d700 1 -- start start 2022-07-21T01:41:29.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcc50d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fc412bd10 0x7f8fc40af830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcc50d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc4126690 0x7f8fc40afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcc50d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc4152700 0x7f8fc4152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcc50d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc4152310 con 0x7f8fc4152700 2022-07-21T01:41:29.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcc50d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc40ade20 con 0x7f8fc412bd10 2022-07-21T01:41:29.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcc50d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc412c0f0 con 0x7f8fc4126690 2022-07-21T01:41:29.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcaaaa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc4152700 0x7f8fc4152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcaaaa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc4152700 0x7f8fc4152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49842/0 (socket says 172.21.15.3:49842) 2022-07-21T01:41:29.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcaaaa700 1 -- 172.21.15.3:0/62101076 learned_addr learned my addr 172.21.15.3:0/62101076 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:29.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fc9aa8700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc4126690 0x7f8fc40afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fca2a9700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fc412bd10 0x7f8fc40af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcaaaa700 1 -- 172.21.15.3:0/62101076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc4126690 msgr2=0x7f8fc40afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcaaaa700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc4126690 0x7f8fc40afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcaaaa700 1 -- 172.21.15.3:0/62101076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fc412bd10 msgr2=0x7f8fc40af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcaaaa700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fc412bd10 0x7f8fc40af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcaaaa700 1 -- 172.21.15.3:0/62101076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc41567a0 con 0x7f8fc4152700 2022-07-21T01:41:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcaaaa700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc4152700 0x7f8fc4152b90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f8fc40ba9d0 tx=0x7f8fbc019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fb77fe700 1 -- 172.21.15.3:0/62101076 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fbc01cd60 con 0x7f8fc4152700 2022-07-21T01:41:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fb77fe700 1 -- 172.21.15.3:0/62101076 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc000e90 con 0x7f8fc4152700 2022-07-21T01:41:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc40b3aa0 con 0x7f8fc4152700 2022-07-21T01:41:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fb77fe700 1 -- 172.21.15.3:0/62101076 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fbc0245e0 con 0x7f8fc4152700 2022-07-21T01:41:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc40b0210 con 0x7f8fc4152700 2022-07-21T01:41:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.776+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc4155c20 con 0x7f8fc4152700 2022-07-21T01:41:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.780+0000 7f8fb77fe700 1 -- 172.21.15.3:0/62101076 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8fbc0123f0 con 0x7f8fc4152700 2022-07-21T01:41:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.780+0000 7f8fb77fe700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fa4053a70 0x7f8fa4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:29.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.780+0000 7f8fb77fe700 1 -- 172.21.15.3:0/62101076 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8fbc06d500 con 0x7f8fc4152700 2022-07-21T01:41:29.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.780+0000 7f8fca2a9700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fa4053a70 0x7f8fa4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:29.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.780+0000 7f8fca2a9700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fa4053a70 0x7f8fa4055f30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f8fb800d000 tx=0x7f8fb8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:29.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.784+0000 7f8fb77fe700 1 -- 172.21.15.3:0/62101076 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8fbc044260 con 0x7f8fc4152700 2022-07-21T01:41:29.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.948+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8fc4061780 con 0x7f8fc4152700 2022-07-21T01:41:29.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fb77fe700 1 -- 172.21.15.3:0/62101076 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v33) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8fbc00be60 con 0x7f8fc4152700 2022-07-21T01:41:29.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fa4053a70 msgr2=0x7f8fa4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fa4053a70 0x7f8fa4055f30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f8fb800d000 tx=0x7f8fb8009040).stop 2022-07-21T01:41:29.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc4152700 msgr2=0x7f8fc4152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:29.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc4152700 0x7f8fc4152b90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f8fc40ba9d0 tx=0x7f8fbc019000).stop 2022-07-21T01:41:29.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 shutdown_connections 2022-07-21T01:41:29.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc4152700 0x7f8fc4152b90 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fc4126690 0x7f8fc40afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fa4053a70 0x7f8fa4055f30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 --2- 172.21.15.3:0/62101076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fc412bd10 0x7f8fc40af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:29.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 >> 172.21.15.3:0/62101076 conn(0x7f8fc405c620 msgr2=0x7f8fc412a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:29.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 shutdown_connections 2022-07-21T01:41:29.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:29.956+0000 7f8fcc50d700 1 -- 172.21.15.3:0/62101076 wait complete. 2022-07-21T01:41:29.967 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:41:29.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:30.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfed94700 1 Processor -- start 2022-07-21T01:41:30.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfed94700 1 -- start start 2022-07-21T01:41:30.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfed94700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdf80b1f40 0x7fbdf80b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfed94700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf80a7250 0x7fbdf81448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfed94700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf80b04e0 con 0x7fbdf8148530 2022-07-21T01:41:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfed94700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf80a9250 con 0x7fbdf80b1f40 2022-07-21T01:41:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfed94700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80b0780 con 0x7fbdf80a7250 2022-07-21T01:41:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf7fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbdf8148530 0x7fbdf80b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58128/0 (socket says 172.21.15.3:58128) 2022-07-21T01:41:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf7fff700 1 -- 172.21.15.3:0/2343465279 learned_addr learned my addr 172.21.15.3:0/2343465279 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfcb30700 1 --2- 172.21.15.3:0/2343465279 >> v2:172.21.15.143:3300/0 conn(0x7fbdf80b1f40 0x7fbdf80b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfd331700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf80a7250 0x7fbdf81448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81296227 0 0) 0x7fbdf80b04e0 con 0x7fbdf8148530 2022-07-21T01:41:30.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8003260 con 0x7fbdf8148530 2022-07-21T01:41:30.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 843168543 0 0) 0x7fbdd8003260 con 0x7fbdf8148530 2022-07-21T01:41:30.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf80a7250 msgr2=0x7fbdf81448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:30.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf80a7250 0x7fbdf81448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 >> v2:172.21.15.143:3300/0 conn(0x7fbdf80b1f40 msgr2=0x7fbdf80b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 --2- 172.21.15.3:0/2343465279 >> v2:172.21.15.143:3300/0 conn(0x7fbdf80b1f40 0x7fbdf80b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf80b0a20 con 0x7fbdf8148530 2022-07-21T01:41:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fbdf00019e0 con 0x7fbdf8148530 2022-07-21T01:41:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 >> v1:172.21.15.3:6789/0 conn(0x7fbdf8148530 legacy=0x7fbdf80b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd8003750 0x7fbdd8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdd8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf80a7250 0x7fbdd8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80b0780 con 0x7fbdd8004500 2022-07-21T01:41:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf7fff700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdd8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd8004320 con 0x7fbdd8003750 2022-07-21T01:41:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd800b780 con 0x7fbdf80a7250 2022-07-21T01:41:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.080+0000 7fbdfcb30700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd8003750 0x7fbdd8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdf7fff700 1 -- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf80a7250 msgr2=0x7fbdd8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdf7fff700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf80a7250 0x7fbdd8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:41:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdf7fff700 1 -- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd8003750 msgr2=0x7fbdd8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdf7fff700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd8003750 0x7fbdd8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdf7fff700 1 -- 172.21.15.3:0/2343465279 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8003260 con 0x7fbdd8004500 2022-07-21T01:41:30.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdf7fff700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdd8004980 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7fbdf81478c0 tx=0x7fbdf80b0bb0).ready entity=mon.0 client_cookie=16cd40319f2e3bd9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:30.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdf000cc40 con 0x7fbdd8004500 2022-07-21T01:41:30.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf000cda0 con 0x7fbdd8004500 2022-07-21T01:41:30.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdf77fe700 1 -- 172.21.15.3:0/2343465279 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdf0005b80 con 0x7fbdd8004500 2022-07-21T01:41:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdfed94700 1 -- 172.21.15.3:0/2343465279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 msgr2=0x7fbdd8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdfed94700 1 --2- 172.21.15.3:0/2343465279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdd8004980 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7fbdf81478c0 tx=0x7fbdf80b0bb0).stop 2022-07-21T01:41:30.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdfd331700 1 -- 172.21.15.3:0/2343465279 reap_dead start 2022-07-21T01:41:30.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdfed94700 1 -- 172.21.15.3:0/2343465279 shutdown_connections 2022-07-21T01:41:30.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdfed94700 1 -- 172.21.15.3:0/2343465279 >> 172.21.15.3:0/2343465279 conn(0x7fbdf805f840 msgr2=0x7fbdf805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:30.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdfed94700 1 -- 172.21.15.3:0/2343465279 shutdown_connections 2022-07-21T01:41:30.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdfed94700 1 -- 172.21.15.3:0/2343465279 wait complete. 2022-07-21T01:41:30.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdfed94700 1 Processor -- start 2022-07-21T01:41:30.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.084+0000 7fbdfed94700 1 -- start start 2022-07-21T01:41:30.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfed94700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdf80adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfed94700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf815b150 0x7fbdf8149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfed94700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf8149580 0x7fbdf813d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfed94700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf814b560 con 0x7fbdd8004500 2022-07-21T01:41:30.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfed94700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80b6290 con 0x7fbdf815b150 2022-07-21T01:41:30.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfed94700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf814b790 con 0x7fbdf8149580 2022-07-21T01:41:30.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfcb30700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdf80adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfcb30700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdf80adbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49858/0 (socket says 172.21.15.3:49858) 2022-07-21T01:41:30.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfcb30700 1 -- 172.21.15.3:0/3464307581 learned_addr learned my addr 172.21.15.3:0/3464307581 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:30.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfd331700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf8149580 0x7fbdf813d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdf7fff700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf815b150 0x7fbdf8149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfcb30700 1 -- 172.21.15.3:0/3464307581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf8149580 msgr2=0x7fbdf813d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfcb30700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf8149580 0x7fbdf813d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfcb30700 1 -- 172.21.15.3:0/3464307581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf815b150 msgr2=0x7fbdf8149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfcb30700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf815b150 0x7fbdf8149090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfcb30700 1 -- 172.21.15.3:0/3464307581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf80ae0b0 con 0x7fbdd8004500 2022-07-21T01:41:30.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfd331700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf8149580 0x7fbdf813d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:30.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfcb30700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdf80adbc0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7fbde000b000 tx=0x7fbde0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:30.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdf5ffb700 1 -- 172.21.15.3:0/3464307581 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbde000f940 con 0x7fbdd8004500 2022-07-21T01:41:30.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdf5ffb700 1 -- 172.21.15.3:0/3464307581 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde000faa0 con 0x7fbdd8004500 2022-07-21T01:41:30.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf8141e10 con 0x7fbdd8004500 2022-07-21T01:41:30.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdf5ffb700 1 -- 172.21.15.3:0/3464307581 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbde000fe00 con 0x7fbdd8004500 2022-07-21T01:41:30.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf8142960 con 0x7fbdd8004500 2022-07-21T01:41:30.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf815af50 con 0x7fbdd8004500 2022-07-21T01:41:30.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.088+0000 7fbdf5ffb700 1 -- 172.21.15.3:0/3464307581 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbde0014440 con 0x7fbdd8004500 2022-07-21T01:41:30.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.092+0000 7fbdf5ffb700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbddc053b30 0x7fbddc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.092+0000 7fbdf5ffb700 1 -- 172.21.15.3:0/3464307581 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fbde006cb10 con 0x7fbdd8004500 2022-07-21T01:41:30.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.092+0000 7fbdf7fff700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbddc053b30 0x7fbddc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.092+0000 7fbdf7fff700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbddc053b30 0x7fbddc055ff0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fbdf0019000 tx=0x7fbdf0013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:30.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.096+0000 7fbdf5ffb700 1 -- 172.21.15.3:0/3464307581 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbde0043870 con 0x7fbdd8004500 2022-07-21T01:41:30.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.260+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdf80b6400 con 0x7fbdd8004500 2022-07-21T01:41:30.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.260+0000 7fbdf5ffb700 1 -- 172.21.15.3:0/3464307581 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+2486 (secure 0 0 0) 0x7fbdf80b6400 con 0x7fbdd8004500 2022-07-21T01:41:30.265 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:30.265 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":60,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/297776761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":297776761},{"type":"v1","addr":"172.21.15.143:6833","nonce":297776761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":57},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3673188672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3673188672},{"type":"v1","addr":"172.21.15.3:6835","nonce":3673188672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:27.699850+0000","modified":"2022-07-21T01:41:28.704288+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7611},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7611":{"gid":7611,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/508489181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":508489181},{"type":"v1","addr":"172.21.15.3:6837","nonce":508489181}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-21T01:41:30.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbddc053b30 msgr2=0x7fbddc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbddc053b30 0x7fbddc055ff0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fbdf0019000 tx=0x7fbdf0013000).stop 2022-07-21T01:41:30.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 msgr2=0x7fbdf80adbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdf80adbc0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7fbde000b000 tx=0x7fbde0006df0).stop 2022-07-21T01:41:30.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 shutdown_connections 2022-07-21T01:41:30.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf8149580 0x7fbdf813d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbddc053b30 0x7fbddc055ff0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf815b150 0x7fbdf8149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 --2- 172.21.15.3:0/3464307581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd8004500 0x7fbdf80adbc0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 >> 172.21.15.3:0/3464307581 conn(0x7fbdf805f840 msgr2=0x7fbdf80af890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:30.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 shutdown_connections 2022-07-21T01:41:30.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.264+0000 7fbdfed94700 1 -- 172.21.15.3:0/3464307581 wait complete. 2022-07-21T01:41:30.271 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 60 2022-07-21T01:41:30.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:41:27.699850+0000', 'modified': '2022-07-21T01:41:28.704288+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7611}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7611': {'gid': 7611, 'name': 'c', 'rank': 0, 'incarnation': 59, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/508489181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 508489181}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 508489181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:41:30.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:41:30.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:41:30.702 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":7611,"want_state":"up:active","state":"up:active","rank_uptime":2.9904582139999998,"mdsmap_epoch":60,"osdmap_epoch":121,"osdmap_epoch_barrier":121,"uptime":7.3537218629999996} 2022-07-21T01:41:30.703 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 7611, "mdsmap_epoch": 60, "osdmap_epoch": 121, "osdmap_epoch_barrier": 121, "rank_uptime": 2.990458214, "state": "up:active", "uptime": 7.353721863, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:41:30.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:30.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.804+0000 7f0fc57d9700 1 Processor -- start 2022-07-21T01:41:30.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.804+0000 7f0fc57d9700 1 -- start start 2022-07-21T01:41:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.804+0000 7f0fc57d9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0fc00b3630 0x7f0fc0143f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.804+0000 7f0fc57d9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc00b3a10 con 0x7f0fc00af910 2022-07-21T01:41:30.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.804+0000 7f0fc57d9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc0146b40 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.804+0000 7f0fc57d9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc01443f0 con 0x7f0fc00b3630 2022-07-21T01:41:30.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbf7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0fc00af910 0x7f0fc01402a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58150/0 (socket says 172.21.15.3:58150) 2022-07-21T01:41:30.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbf7fe700 1 -- 172.21.15.3:0/139850345 learned_addr learned my addr 172.21.15.3:0/139850345 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:30.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbeffd700 1 --2- 172.21.15.3:0/139850345 >> v2:172.21.15.143:3300/0 conn(0x7f0fc00b3630 0x7f0fc0143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbdffb700 1 -- 172.21.15.3:0/139850345 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212096816 0 0) 0x7f0fc00b3a10 con 0x7f0fc00af910 2022-07-21T01:41:30.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbdffb700 1 -- 172.21.15.3:0/139850345 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa4003260 con 0x7f0fc00af910 2022-07-21T01:41:30.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbeffd700 1 -- 172.21.15.3:0/139850345 >> v1:172.21.15.143:6789/0 conn(0x7f0fc00aa0d0 legacy=0x7f0fc00af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbeffd700 1 -- 172.21.15.3:0/139850345 >> v1:172.21.15.3:6789/0 conn(0x7f0fc00af910 legacy=0x7f0fc01402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbeffd700 1 -- 172.21.15.3:0/139850345 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc0146de0 con 0x7f0fc00b3630 2022-07-21T01:41:30.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbeffd700 1 --2- 172.21.15.3:0/139850345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fc00b3630 0x7f0fc0143f00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f0fb0002f70 tx=0x7f0fb0006df0).ready entity=mon.1 client_cookie=82d9d5f4cb7647f0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:30.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbdffb700 1 -- 172.21.15.3:0/139850345 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fb000d610 con 0x7f0fc00b3630 2022-07-21T01:41:30.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbdffb700 1 -- 172.21.15.3:0/139850345 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb0000e90 con 0x7f0fc00b3630 2022-07-21T01:41:30.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fbdffb700 1 -- 172.21.15.3:0/139850345 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fb000d960 con 0x7f0fc00b3630 2022-07-21T01:41:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fc57d9700 1 -- 172.21.15.3:0/139850345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fc00b3630 msgr2=0x7f0fc0143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fc57d9700 1 --2- 172.21.15.3:0/139850345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fc00b3630 0x7f0fc0143f00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f0fb0002f70 tx=0x7f0fb0006df0).stop 2022-07-21T01:41:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fc57d9700 1 -- 172.21.15.3:0/139850345 shutdown_connections 2022-07-21T01:41:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fc57d9700 1 --2- 172.21.15.3:0/139850345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fc00b3630 0x7f0fc0143f00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fc57d9700 1 -- 172.21.15.3:0/139850345 >> 172.21.15.3:0/139850345 conn(0x7f0fc0060950 msgr2=0x7f0fc0060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fc57d9700 1 -- 172.21.15.3:0/139850345 shutdown_connections 2022-07-21T01:41:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.808+0000 7f0fc57d9700 1 -- 172.21.15.3:0/139850345 wait complete. 2022-07-21T01:41:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fc57d9700 1 Processor -- start 2022-07-21T01:41:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fc57d9700 1 -- start start 2022-07-21T01:41:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fc57d9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc00aa0d0 0x7f0fc013e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fc57d9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fc00af910 0x7f0fc0155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fc57d9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fc00b3630 0x7f0fc0156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fc57d9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc0145ee0 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fc57d9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc0145140 con 0x7f0fc00af910 2022-07-21T01:41:30.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fc57d9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00b2e80 con 0x7f0fc00b3630 2022-07-21T01:41:30.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fbeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc00aa0d0 0x7f0fc013e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fbeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc00aa0d0 0x7f0fc013e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49872/0 (socket says 172.21.15.3:49872) 2022-07-21T01:41:30.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fbeffd700 1 -- 172.21.15.3:0/3037265680 learned_addr learned my addr 172.21.15.3:0/3037265680 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:30.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.812+0000 7f0fbe7fc700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fc00af910 0x7f0fc0155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbf7fe700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fc00b3630 0x7f0fc0156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbeffd700 1 -- 172.21.15.3:0/3037265680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fc00b3630 msgr2=0x7f0fc0156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbeffd700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fc00b3630 0x7f0fc0156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbeffd700 1 -- 172.21.15.3:0/3037265680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fc00af910 msgr2=0x7f0fc0155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbeffd700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fc00af910 0x7f0fc0155b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbeffd700 1 -- 172.21.15.3:0/3037265680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb000b180 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbf7fe700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fc00b3630 0x7f0fc0156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:30.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbeffd700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc00aa0d0 0x7f0fc013e360 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f0fb0002f70 tx=0x7f0fb0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:30.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0faffff700 1 -- 172.21.15.3:0/3037265680 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fb000d290 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0faffff700 1 -- 172.21.15.3:0/3037265680 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb0013e90 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc01453b0 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0faffff700 1 -- 172.21.15.3:0/3037265680 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fb001e5f0 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc013e850 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc00b20a0 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0faffff700 1 -- 172.21.15.3:0/3037265680 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0fb0025030 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0faffff700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f9c053b30 0x7f0f9c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0faffff700 1 -- 172.21.15.3:0/3037265680 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0fb006ccd0 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbe7fc700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f9c053b30 0x7f0f9c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.816+0000 7f0fbe7fc700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f9c053b30 0x7f0f9c055ff0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f0fb400d000 tx=0x7f0fb4007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.824+0000 7f0faffff700 1 -- 172.21.15.3:0/3037265680 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0fb00439f0 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.948+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fc0147080 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.948+0000 7f0faffff700 1 -- 172.21.15.3:0/3037265680 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f0fb0023020 con 0x7f0fc00aa0d0 2022-07-21T01:41:30.953 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:30.953 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":121,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:28.703604+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:26.156392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:27.012266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f9c053b30 msgr2=0x7f0f9c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f9c053b30 0x7f0f9c055ff0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f0fb400d000 tx=0x7f0fb4007b90).stop 2022-07-21T01:41:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc00aa0d0 msgr2=0x7f0fc013e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc00aa0d0 0x7f0fc013e360 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f0fb0002f70 tx=0x7f0fb0014000).stop 2022-07-21T01:41:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 shutdown_connections 2022-07-21T01:41:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fc00b3630 0x7f0fc0156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f9c053b30 0x7f0f9c055ff0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fc00af910 0x7f0fc0155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 --2- 172.21.15.3:0/3037265680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc00aa0d0 0x7f0fc013e360 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 >> 172.21.15.3:0/3037265680 conn(0x7f0fc0060950 msgr2=0x7f0fc005fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 shutdown_connections 2022-07-21T01:41:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:30.952+0000 7f0fc57d9700 1 -- 172.21.15.3:0/3037265680 wait complete. 2022-07-21T01:41:30.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:41:26.156392+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:41:27.012266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:41:30.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:41:31.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.088+0000 7fb53f0bd700 1 Processor -- start 2022-07-21T01:41:31.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.088+0000 7fb53f0bd700 1 -- start start 2022-07-21T01:41:31.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.088+0000 7fb53f0bd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb538060eb0 0x7fb53805f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.088+0000 7fb53f0bd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5380b3dd0 con 0x7fb538143e20 2022-07-21T01:41:31.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.088+0000 7fb53f0bd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5380b2720 con 0x7fb5380b2340 2022-07-21T01:41:31.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.088+0000 7fb53f0bd700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb538063f80 con 0x7fb538060eb0 2022-07-21T01:41:31.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.088+0000 7fb53d8ba700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb538060eb0 0x7fb53805f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.088+0000 7fb53e8bc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb538143e20 0x7fb53812c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58164/0 (socket says 172.21.15.3:58164) 2022-07-21T01:41:31.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.088+0000 7fb53e8bc700 1 -- 172.21.15.3:0/798387774 learned_addr learned my addr 172.21.15.3:0/798387774 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:31.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53d8ba700 1 -- 172.21.15.3:0/798387774 >> v1:172.21.15.143:6789/0 conn(0x7fb5380b2340 legacy=0x7fb5380b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53d8ba700 1 -- 172.21.15.3:0/798387774 >> v1:172.21.15.3:6789/0 conn(0x7fb538143e20 legacy=0x7fb53812c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53d8ba700 1 -- 172.21.15.3:0/798387774 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380b2fc0 con 0x7fb538060eb0 2022-07-21T01:41:31.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53d8ba700 1 --2- 172.21.15.3:0/798387774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb538060eb0 0x7fb53805f840 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7fb534002f70 tx=0x7fb534006df0).ready entity=mon.0 client_cookie=a19bf28c0434736e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53d0b9700 1 -- 172.21.15.3:0/798387774 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53400d610 con 0x7fb538060eb0 2022-07-21T01:41:31.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53d0b9700 1 -- 172.21.15.3:0/798387774 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb534000e90 con 0x7fb538060eb0 2022-07-21T01:41:31.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53d0b9700 1 -- 172.21.15.3:0/798387774 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53400d960 con 0x7fb538060eb0 2022-07-21T01:41:31.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53f0bd700 1 -- 172.21.15.3:0/798387774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb538060eb0 msgr2=0x7fb53805f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53f0bd700 1 --2- 172.21.15.3:0/798387774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb538060eb0 0x7fb53805f840 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7fb534002f70 tx=0x7fb534006df0).stop 2022-07-21T01:41:31.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53f0bd700 1 -- 172.21.15.3:0/798387774 shutdown_connections 2022-07-21T01:41:31.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53f0bd700 1 --2- 172.21.15.3:0/798387774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb538060eb0 0x7fb53805f840 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53f0bd700 1 -- 172.21.15.3:0/798387774 >> 172.21.15.3:0/798387774 conn(0x7fb53805c620 msgr2=0x7fb53805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:31.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53f0bd700 1 -- 172.21.15.3:0/798387774 shutdown_connections 2022-07-21T01:41:31.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.092+0000 7fb53f0bd700 1 -- 172.21.15.3:0/798387774 wait complete. 2022-07-21T01:41:31.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 Processor -- start 2022-07-21T01:41:31.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 -- start start 2022-07-21T01:41:31.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538060eb0 0x7fb538064db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b2340 0x7fb538128080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538143e20 0x7fb538128570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5381463e0 con 0x7fb5380b2340 2022-07-21T01:41:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380b03f0 con 0x7fb538060eb0 2022-07-21T01:41:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb538128a60 con 0x7fb538143e20 2022-07-21T01:41:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53d8ba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b2340 0x7fb538128080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53d8ba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b2340 0x7fb538128080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49888/0 (socket says 172.21.15.3:49888) 2022-07-21T01:41:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53d8ba700 1 -- 172.21.15.3:0/3027863762 learned_addr learned my addr 172.21.15.3:0/3027863762 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53e8bc700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538143e20 0x7fb538128570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53e0bb700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538060eb0 0x7fb538064db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53d8ba700 1 -- 172.21.15.3:0/3027863762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538143e20 msgr2=0x7fb538128570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53d8ba700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538143e20 0x7fb538128570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53d8ba700 1 -- 172.21.15.3:0/3027863762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538060eb0 msgr2=0x7fb538064db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53d8ba700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538060eb0 0x7fb538064db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53d8ba700 1 -- 172.21.15.3:0/3027863762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53400b180 con 0x7fb5380b2340 2022-07-21T01:41:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53e0bb700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538060eb0 0x7fb538064db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53d8ba700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b2340 0x7fb538128080 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7fb534002f70 tx=0x7fb534015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb52b7fe700 1 -- 172.21.15.3:0/3027863762 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53400de70 con 0x7fb5380b2340 2022-07-21T01:41:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb52b7fe700 1 -- 172.21.15.3:0/3027863762 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb534011bf0 con 0x7fb5380b2340 2022-07-21T01:41:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb538145a00 con 0x7fb5380b2340 2022-07-21T01:41:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb52b7fe700 1 -- 172.21.15.3:0/3027863762 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53401e460 con 0x7fb5380b2340 2022-07-21T01:41:31.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.096+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb538145f40 con 0x7fb5380b2340 2022-07-21T01:41:31.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.100+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb538146100 con 0x7fb5380b2340 2022-07-21T01:41:31.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.100+0000 7fb52b7fe700 1 -- 172.21.15.3:0/3027863762 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb534025030 con 0x7fb5380b2340 2022-07-21T01:41:31.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.100+0000 7fb52b7fe700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053aa0 0x7fb518055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.100+0000 7fb52b7fe700 1 -- 172.21.15.3:0/3027863762 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb53406d390 con 0x7fb5380b2340 2022-07-21T01:41:31.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.100+0000 7fb53e0bb700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053aa0 0x7fb518055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.100+0000 7fb53e0bb700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053aa0 0x7fb518055f60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7fb52c00b000 tx=0x7fb52c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.108+0000 7fb52b7fe700 1 -- 172.21.15.3:0/3027863762 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb534044150 con 0x7fb5380b2340 2022-07-21T01:41:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.236+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb538061d60 con 0x7fb5380b2340 2022-07-21T01:41:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.236+0000 7fb52b7fe700 1 -- 172.21.15.3:0/3027863762 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v121) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb5340230b0 con 0x7fb5380b2340 2022-07-21T01:41:31.239 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:41:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.236+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053aa0 msgr2=0x7fb518055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.236+0000 7fb53f0bd700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053aa0 0x7fb518055f60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7fb52c00b000 tx=0x7fb52c009040).stop 2022-07-21T01:41:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.236+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b2340 msgr2=0x7fb538128080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.236+0000 7fb53f0bd700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b2340 0x7fb538128080 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7fb534002f70 tx=0x7fb534015000).stop 2022-07-21T01:41:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.240+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 shutdown_connections 2022-07-21T01:41:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.240+0000 7fb53f0bd700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538143e20 0x7fb538128570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.240+0000 7fb53f0bd700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b2340 0x7fb538128080 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.240+0000 7fb53f0bd700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053aa0 0x7fb518055f60 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.240+0000 7fb53f0bd700 1 --2- 172.21.15.3:0/3027863762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538060eb0 0x7fb538064db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.240+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 >> 172.21.15.3:0/3027863762 conn(0x7fb53805c620 msgr2=0x7fb53812b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.240+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 shutdown_connections 2022-07-21T01:41:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.240+0000 7fb53f0bd700 1 -- 172.21.15.3:0/3027863762 wait complete. 2022-07-21T01:41:31.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:41:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f84a5a2f700 1 Processor -- start 2022-07-21T01:41:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f84a5a2f700 1 -- start start 2022-07-21T01:41:31.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f84a5a2f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84a0146ae0 0x7f84a01403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f84a5a2f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f84a00b4310 0x7f84a01408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f84a5a2f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 0x7f84a00b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f84a5a2f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a01476a0 con 0x7f84a0146ae0 2022-07-21T01:41:31.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f84a5a2f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a0143cf0 con 0x7f84a00b4310 2022-07-21T01:41:31.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f84a5a2f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0143f90 con 0x7f84a00b4820 2022-07-21T01:41:31.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84a0146ae0 0x7f84a01403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f84a0146ae0 0x7f84a01403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49894/0 (socket says 172.21.15.3:49894) 2022-07-21T01:41:31.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849effd700 1 -- 172.21.15.3:0/4249795710 learned_addr learned my addr 172.21.15.3:0/4249795710 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:31.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849f7fe700 1 --2- 172.21.15.3:0/4249795710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 0x7f84a00b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849e7fc700 1 --2- 172.21.15.3:0/4249795710 >> v2:172.21.15.143:3300/0 conn(0x7f84a00b4310 0x7f84a01408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849effd700 1 -- 172.21.15.3:0/4249795710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 msgr2=0x7f84a00b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:31.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849effd700 1 --2- 172.21.15.3:0/4249795710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 0x7f84a00b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849effd700 1 -- 172.21.15.3:0/4249795710 >> v2:172.21.15.143:3300/0 conn(0x7f84a00b4310 msgr2=0x7f84a01408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:31.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849effd700 1 --2- 172.21.15.3:0/4249795710 >> v2:172.21.15.143:3300/0 conn(0x7f84a00b4310 0x7f84a01408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849effd700 1 -- 172.21.15.3:0/4249795710 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a00b0e20 con 0x7f84a0146ae0 2022-07-21T01:41:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849effd700 1 --2- 172.21.15.3:0/4249795710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a0146ae0 0x7f84a01403c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f8490002f70 tx=0x7f8490006eb0).ready entity=mon.0 client_cookie=8351f7d0e783a603 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849dffb700 1 -- 172.21.15.3:0/4249795710 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f849000d570 con 0x7f84a0146ae0 2022-07-21T01:41:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849dffb700 1 -- 172.21.15.3:0/4249795710 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8490000e90 con 0x7f84a0146ae0 2022-07-21T01:41:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.364+0000 7f849dffb700 1 -- 172.21.15.3:0/4249795710 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f849000d900 con 0x7f84a0146ae0 2022-07-21T01:41:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- 172.21.15.3:0/4249795710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a0146ae0 msgr2=0x7f84a01403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/4249795710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a0146ae0 0x7f84a01403c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f8490002f70 tx=0x7f8490006eb0).stop 2022-07-21T01:41:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- 172.21.15.3:0/4249795710 shutdown_connections 2022-07-21T01:41:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/4249795710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 0x7f84a00b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/4249795710 >> v2:172.21.15.143:3300/0 conn(0x7f84a00b4310 0x7f84a01408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/4249795710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a0146ae0 0x7f84a01403c0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- 172.21.15.3:0/4249795710 >> 172.21.15.3:0/4249795710 conn(0x7f84a005c620 msgr2=0x7f84a005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- 172.21.15.3:0/4249795710 shutdown_connections 2022-07-21T01:41:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- 172.21.15.3:0/4249795710 wait complete. 2022-07-21T01:41:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 Processor -- start 2022-07-21T01:41:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- start start 2022-07-21T01:41:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b4310 0x7f84a01524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 0x7f84a013e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a0146ae0 0x7f84a0155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00b7c80 con 0x7f84a00b4310 2022-07-21T01:41:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0155f80 con 0x7f84a0146ae0 2022-07-21T01:41:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0156980 con 0x7f84a00b4820 2022-07-21T01:41:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b4310 0x7f84a01524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b4310 0x7f84a01524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49900/0 (socket says 172.21.15.3:49900) 2022-07-21T01:41:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849effd700 1 -- 172.21.15.3:0/3019279221 learned_addr learned my addr 172.21.15.3:0/3019279221 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849e7fc700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 0x7f84a013e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849f7fe700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a0146ae0 0x7f84a0155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849effd700 1 -- 172.21.15.3:0/3019279221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 msgr2=0x7f84a013e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849effd700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 0x7f84a013e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849effd700 1 -- 172.21.15.3:0/3019279221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a0146ae0 msgr2=0x7f84a0155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849effd700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a0146ae0 0x7f84a0155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849effd700 1 -- 172.21.15.3:0/3019279221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849000b1e0 con 0x7f84a00b4310 2022-07-21T01:41:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f849effd700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b4310 0x7f84a01524e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f84a01560c0 tx=0x7f8490015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f848ffff700 1 -- 172.21.15.3:0/3019279221 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8490011760 con 0x7f84a00b4310 2022-07-21T01:41:31.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f848ffff700 1 -- 172.21.15.3:0/3019279221 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84900118c0 con 0x7f84a00b4310 2022-07-21T01:41:31.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a01561a0 con 0x7f84a00b4310 2022-07-21T01:41:31.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f848ffff700 1 -- 172.21.15.3:0/3019279221 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8490011c20 con 0x7f84a00b4310 2022-07-21T01:41:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.368+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a01566e0 con 0x7f84a00b4310 2022-07-21T01:41:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.372+0000 7f848ffff700 1 -- 172.21.15.3:0/3019279221 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8490025030 con 0x7f84a00b4310 2022-07-21T01:41:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.372+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a00b84e0 con 0x7f84a00b4310 2022-07-21T01:41:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.372+0000 7f848ffff700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f847c0539e0 0x7f847c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.372+0000 7f848ffff700 1 -- 172.21.15.3:0/3019279221 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f849006d1e0 con 0x7f84a00b4310 2022-07-21T01:41:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.372+0000 7f849e7fc700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f847c0539e0 0x7f847c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.372+0000 7f849e7fc700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f847c0539e0 0x7f847c055ea0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f849400c000 tx=0x7f8494008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.380+0000 7f848ffff700 1 -- 172.21.15.3:0/3019279221 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8490047930 con 0x7f84a00b4310 2022-07-21T01:41:31.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.500+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f84a00574e0 con 0x7f84a00b4310 2022-07-21T01:41:31.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.500+0000 7f848ffff700 1 -- 172.21.15.3:0/3019279221 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v121) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8490047520 con 0x7f84a00b4310 2022-07-21T01:41:31.505 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:31.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f847c0539e0 msgr2=0x7f847c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f847c0539e0 0x7f847c055ea0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f849400c000 tx=0x7f8494008040).stop 2022-07-21T01:41:31.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b4310 msgr2=0x7f84a01524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b4310 0x7f84a01524e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f84a01560c0 tx=0x7f8490015000).stop 2022-07-21T01:41:31.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 shutdown_connections 2022-07-21T01:41:31.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f847c0539e0 0x7f847c055ea0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a0146ae0 0x7f84a0155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84a00b4820 0x7f84a013e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 --2- 172.21.15.3:0/3019279221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a00b4310 0x7f84a01524e0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 >> 172.21.15.3:0/3019279221 conn(0x7f84a005c620 msgr2=0x7f84a00616e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:31.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 shutdown_connections 2022-07-21T01:41:31.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.504+0000 7f84a5a2f700 1 -- 172.21.15.3:0/3019279221 wait complete. 2022-07-21T01:41:31.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:41:31.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.624+0000 7f7b45bf9700 1 Processor -- start 2022-07-21T01:41:31.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.624+0000 7f7b45bf9700 1 -- start start 2022-07-21T01:41:31.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b45bf9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7b400b66f0 0x7f7b400b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b45bf9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b401440f0 con 0x7f7b4005fd90 2022-07-21T01:41:31.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b45bf9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b40060170 con 0x7f7b400b5b20 2022-07-21T01:41:31.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b45bf9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b400ba840 con 0x7f7b400b66f0 2022-07-21T01:41:31.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7b4005fd90 0x7f7b400ba130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58188/0 (socket says 172.21.15.3:58188) 2022-07-21T01:41:31.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3f7fe700 1 -- 172.21.15.3:0/2699777031 learned_addr learned my addr 172.21.15.3:0/2699777031 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:31.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3effd700 1 --2- 172.21.15.3:0/2699777031 >> v2:172.21.15.143:3300/0 conn(0x7f7b400b66f0 0x7f7b400b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3e7fc700 1 -- 172.21.15.3:0/2699777031 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645117433 0 0) 0x7f7b401440f0 con 0x7f7b4005fd90 2022-07-21T01:41:31.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3e7fc700 1 -- 172.21.15.3:0/2699777031 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b20003260 con 0x7f7b4005fd90 2022-07-21T01:41:31.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3e7fc700 1 -- 172.21.15.3:0/2699777031 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963971582 0 0) 0x7f7b40060170 con 0x7f7b400b5b20 2022-07-21T01:41:31.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3e7fc700 1 -- 172.21.15.3:0/2699777031 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b401440f0 con 0x7f7b400b5b20 2022-07-21T01:41:31.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3e7fc700 1 -- 172.21.15.3:0/2699777031 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298473381 0 0) 0x7f7b20003260 con 0x7f7b4005fd90 2022-07-21T01:41:31.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3effd700 1 -- 172.21.15.3:0/2699777031 >> v1:172.21.15.143:6789/0 conn(0x7f7b400b5b20 legacy=0x7f7b400b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3effd700 1 -- 172.21.15.3:0/2699777031 >> v1:172.21.15.3:6789/0 conn(0x7f7b4005fd90 legacy=0x7f7b400ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3effd700 1 -- 172.21.15.3:0/2699777031 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b40143be0 con 0x7f7b400b66f0 2022-07-21T01:41:31.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3effd700 1 --2- 172.21.15.3:0/2699777031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b400b66f0 0x7f7b400b4340 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f7b30001800 tx=0x7f7b28006df0).ready entity=mon.1 client_cookie=d556e8c1a43db7c1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3e7fc700 1 -- 172.21.15.3:0/2699777031 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b30001660 con 0x7f7b400b66f0 2022-07-21T01:41:31.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3e7fc700 1 -- 172.21.15.3:0/2699777031 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b28000e90 con 0x7f7b400b66f0 2022-07-21T01:41:31.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b3e7fc700 1 -- 172.21.15.3:0/2699777031 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b28012460 con 0x7f7b400b66f0 2022-07-21T01:41:31.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b45bf9700 1 -- 172.21.15.3:0/2699777031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b400b66f0 msgr2=0x7f7b400b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.628+0000 7f7b45bf9700 1 --2- 172.21.15.3:0/2699777031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b400b66f0 0x7f7b400b4340 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f7b30001800 tx=0x7f7b28006df0).stop 2022-07-21T01:41:31.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 -- 172.21.15.3:0/2699777031 shutdown_connections 2022-07-21T01:41:31.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 --2- 172.21.15.3:0/2699777031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b400b66f0 0x7f7b400b4340 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 -- 172.21.15.3:0/2699777031 >> 172.21.15.3:0/2699777031 conn(0x7f7b40062530 msgr2=0x7f7b40066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:31.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 -- 172.21.15.3:0/2699777031 shutdown_connections 2022-07-21T01:41:31.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 -- 172.21.15.3:0/2699777031 wait complete. 2022-07-21T01:41:31.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 Processor -- start 2022-07-21T01:41:31.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 -- start start 2022-07-21T01:41:31.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b4005fd90 0x7f7b40141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b400b5b20 0x7f7b4012b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400b66f0 0x7f7b4012b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b401400a0 con 0x7f7b400b66f0 2022-07-21T01:41:31.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4013fe00 con 0x7f7b4005fd90 2022-07-21T01:41:31.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b45bf9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4013f280 con 0x7f7b400b5b20 2022-07-21T01:41:31.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b3ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400b66f0 0x7f7b4012b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b3ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400b66f0 0x7f7b4012b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49918/0 (socket says 172.21.15.3:49918) 2022-07-21T01:41:31.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b3ffff700 1 -- 172.21.15.3:0/207638569 learned_addr learned my addr 172.21.15.3:0/207638569 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:31.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b3f7fe700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b4005fd90 0x7f7b40141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.632+0000 7f7b3effd700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b400b5b20 0x7f7b4012b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3ffff700 1 -- 172.21.15.3:0/207638569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b400b5b20 msgr2=0x7f7b4012b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3ffff700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b400b5b20 0x7f7b4012b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3ffff700 1 -- 172.21.15.3:0/207638569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b4005fd90 msgr2=0x7f7b40141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3ffff700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b4005fd90 0x7f7b40141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3ffff700 1 -- 172.21.15.3:0/207638569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2800b180 con 0x7f7b400b66f0 2022-07-21T01:41:31.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3f7fe700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b4005fd90 0x7f7b40141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:31.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3ffff700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400b66f0 0x7f7b4012b5d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f7b38001490 tx=0x7f7b38007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3cff9700 1 -- 172.21.15.3:0/207638569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b38002310 con 0x7f7b400b66f0 2022-07-21T01:41:31.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3cff9700 1 -- 172.21.15.3:0/207638569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b38002470 con 0x7f7b400b66f0 2022-07-21T01:41:31.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4013f520 con 0x7f7b400b66f0 2022-07-21T01:41:31.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3cff9700 1 -- 172.21.15.3:0/207638569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b38001840 con 0x7f7b400b66f0 2022-07-21T01:41:31.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4012bac0 con 0x7f7b400b66f0 2022-07-21T01:41:31.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4013f6b0 con 0x7f7b400b66f0 2022-07-21T01:41:31.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.636+0000 7f7b3cff9700 1 -- 172.21.15.3:0/207638569 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7b38010030 con 0x7f7b400b66f0 2022-07-21T01:41:31.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.640+0000 7f7b3cff9700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7b24053b30 0x7f7b24055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.640+0000 7f7b3cff9700 1 -- 172.21.15.3:0/207638569 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7b3806bf40 con 0x7f7b400b66f0 2022-07-21T01:41:31.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.640+0000 7f7b3f7fe700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7b24053b30 0x7f7b24055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.640+0000 7f7b3f7fe700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7b24053b30 0x7f7b24055ff0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f7b3000b000 tx=0x7f7b30001660).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.648+0000 7f7b3cff9700 1 -- 172.21.15.3:0/207638569 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7b38042bf0 con 0x7f7b400b66f0 2022-07-21T01:41:31.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.772+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7b40143e50 con 0x7f7b400b66f0 2022-07-21T01:41:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.772+0000 7f7b3cff9700 1 -- 172.21.15.3:0/207638569 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7b38046210 con 0x7f7b400b66f0 2022-07-21T01:41:31.777 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:31.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7b24053b30 msgr2=0x7f7b24055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7b24053b30 0x7f7b24055ff0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f7b3000b000 tx=0x7f7b30001660).stop 2022-07-21T01:41:31.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400b66f0 msgr2=0x7f7b4012b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400b66f0 0x7f7b4012b5d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f7b38001490 tx=0x7f7b38007dd0).stop 2022-07-21T01:41:31.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 shutdown_connections 2022-07-21T01:41:31.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b400b66f0 0x7f7b4012b5d0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b400b5b20 0x7f7b4012b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7b24053b30 0x7f7b24055ff0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 --2- 172.21.15.3:0/207638569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b4005fd90 0x7f7b40141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 >> 172.21.15.3:0/207638569 conn(0x7f7b40062530 msgr2=0x7f7b4012c620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:31.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 shutdown_connections 2022-07-21T01:41:31.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.776+0000 7f7b45bf9700 1 -- 172.21.15.3:0/207638569 wait complete. 2022-07-21T01:41:31.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:41:31.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.896+0000 7f2b1bfff700 1 Processor -- start 2022-07-21T01:41:31.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.896+0000 7f2b1bfff700 1 -- start start 2022-07-21T01:41:31.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c0614b0 0x7f2b1c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2b1c1405d0 0x7f2b1c140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1c0aee80 con 0x7f2b1c0ad750 2022-07-21T01:41:31.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c0b70b0 con 0x7f2b1c1405d0 2022-07-21T01:41:31.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0af0b0 con 0x7f2b1c0614b0 2022-07-21T01:41:31.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2b1c1405d0 0x7f2b1c140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1a7fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2b1c0ad750 0x7f2b1c0b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58204/0 (socket says 172.21.15.3:58204) 2022-07-21T01:41:31.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1a7fc700 1 -- 172.21.15.3:0/2201103782 learned_addr learned my addr 172.21.15.3:0/2201103782 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:31.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1affd700 1 --2- 172.21.15.3:0/2201103782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c0614b0 0x7f2b1c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1b7fe700 1 -- 172.21.15.3:0/2201103782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c0614b0 msgr2=0x7f2b1c061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:31.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1b7fe700 1 --2- 172.21.15.3:0/2201103782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c0614b0 0x7f2b1c061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1b7fe700 1 -- 172.21.15.3:0/2201103782 >> v1:172.21.15.3:6789/0 conn(0x7f2b1c0ad750 legacy=0x7f2b1c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1b7fe700 1 -- 172.21.15.3:0/2201103782 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c0af350 con 0x7f2b1c1405d0 2022-07-21T01:41:31.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1affd700 1 --2- 172.21.15.3:0/2201103782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c0614b0 0x7f2b1c061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:31.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1b7fe700 1 --2- 172.21.15.3:0/2201103782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c1405d0 0x7f2b1c140a60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f2b140032f0 tx=0x7f2b140091d0).ready entity=mon.0 client_cookie=85e4c9e2c77397db server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b19ffb700 1 -- 172.21.15.3:0/2201103782 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b14009ea0 con 0x7f2b1c1405d0 2022-07-21T01:41:31.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b19ffb700 1 -- 172.21.15.3:0/2201103782 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1400b8a0 con 0x7f2b1c1405d0 2022-07-21T01:41:31.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b19ffb700 1 -- 172.21.15.3:0/2201103782 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b1400ba10 con 0x7f2b1c1405d0 2022-07-21T01:41:31.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 -- 172.21.15.3:0/2201103782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c1405d0 msgr2=0x7f2b1c140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 --2- 172.21.15.3:0/2201103782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c1405d0 0x7f2b1c140a60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f2b140032f0 tx=0x7f2b140091d0).stop 2022-07-21T01:41:31.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 -- 172.21.15.3:0/2201103782 shutdown_connections 2022-07-21T01:41:31.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 --2- 172.21.15.3:0/2201103782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c1405d0 0x7f2b1c140a60 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.900+0000 7f2b1bfff700 1 --2- 172.21.15.3:0/2201103782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c0614b0 0x7f2b1c061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 -- 172.21.15.3:0/2201103782 >> 172.21.15.3:0/2201103782 conn(0x7f2b1c066f70 msgr2=0x7f2b1c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:31.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 -- 172.21.15.3:0/2201103782 shutdown_connections 2022-07-21T01:41:31.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 -- 172.21.15.3:0/2201103782 wait complete. 2022-07-21T01:41:31.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 Processor -- start 2022-07-21T01:41:31.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 -- start start 2022-07-21T01:41:31.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0614b0 0x7f2b1c0b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c0ad750 0x7f2b1c0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c1405d0 0x7f2b1c0b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0b3980 con 0x7f2b1c0ad750 2022-07-21T01:41:31.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0b01b0 con 0x7f2b1c0614b0 2022-07-21T01:41:31.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.904+0000 7f2b1bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c153020 con 0x7f2b1c1405d0 2022-07-21T01:41:31.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c0ad750 0x7f2b1c0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c0ad750 0x7f2b1c0b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49932/0 (socket says 172.21.15.3:49932) 2022-07-21T01:41:31.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1a7fc700 1 -- 172.21.15.3:0/1871758641 learned_addr learned my addr 172.21.15.3:0/1871758641 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:31.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1b7fe700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c1405d0 0x7f2b1c0b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1affd700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0614b0 0x7f2b1c0b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1a7fc700 1 -- 172.21.15.3:0/1871758641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c1405d0 msgr2=0x7f2b1c0b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1a7fc700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c1405d0 0x7f2b1c0b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1a7fc700 1 -- 172.21.15.3:0/1871758641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0614b0 msgr2=0x7f2b1c0b22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:31.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1a7fc700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0614b0 0x7f2b1c0b22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:31.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1a7fc700 1 -- 172.21.15.3:0/1871758641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14009470 con 0x7f2b1c0ad750 2022-07-21T01:41:31.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1b7fe700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c1405d0 0x7f2b1c0b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:31.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1a7fc700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c0ad750 0x7f2b1c0b1af0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7f2b1c0b6c70 tx=0x7f2b04007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b13fff700 1 -- 172.21.15.3:0/1871758641 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b040070d0 con 0x7f2b1c0ad750 2022-07-21T01:41:31.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b13fff700 1 -- 172.21.15.3:0/1871758641 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b04007230 con 0x7f2b1c0ad750 2022-07-21T01:41:31.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c1532c0 con 0x7f2b1c0ad750 2022-07-21T01:41:31.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b13fff700 1 -- 172.21.15.3:0/1871758641 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b04005610 con 0x7f2b1c0ad750 2022-07-21T01:41:31.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c0b14b0 con 0x7f2b1c0ad750 2022-07-21T01:41:31.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c0b0370 con 0x7f2b1c0ad750 2022-07-21T01:41:31.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.908+0000 7f2b13fff700 1 -- 172.21.15.3:0/1871758641 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2b04010070 con 0x7f2b1c0ad750 2022-07-21T01:41:31.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.912+0000 7f2b13fff700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b00053950 0x7f2b00055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:31.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.912+0000 7f2b13fff700 1 -- 172.21.15.3:0/1871758641 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f2b0406b560 con 0x7f2b1c0ad750 2022-07-21T01:41:31.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.912+0000 7f2b1affd700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b00053950 0x7f2b00055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:31.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.912+0000 7f2b1affd700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b00053950 0x7f2b00055e10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f2b0c00d000 tx=0x7f2b0c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:31.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:31.916+0000 7f2b13fff700 1 -- 172.21.15.3:0/1871758641 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b04042320 con 0x7f2b1c0ad750 2022-07-21T01:41:32.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b1c0b1fe0 con 0x7f2b1c0ad750 2022-07-21T01:41:32.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b13fff700 1 -- 172.21.15.3:0/1871758641 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2b04045940 con 0x7f2b1c0ad750 2022-07-21T01:41:32.047 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:32.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b00053950 msgr2=0x7f2b00055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:32.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b00053950 0x7f2b00055e10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f2b0c00d000 tx=0x7f2b0c008040).stop 2022-07-21T01:41:32.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c0ad750 msgr2=0x7f2b1c0b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:32.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c0ad750 0x7f2b1c0b1af0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7f2b1c0b6c70 tx=0x7f2b04007c40).stop 2022-07-21T01:41:32.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 shutdown_connections 2022-07-21T01:41:32.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b1c1405d0 0x7f2b1c0b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b1c0ad750 0x7f2b1c0b1af0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2b00053950 0x7f2b00055e10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 --2- 172.21.15.3:0/1871758641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0614b0 0x7f2b1c0b22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 >> 172.21.15.3:0/1871758641 conn(0x7f2b1c066f70 msgr2=0x7f2b1c0b7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:32.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 shutdown_connections 2022-07-21T01:41:32.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.044+0000 7f2b1bfff700 1 -- 172.21.15.3:0/1871758641 wait complete. 2022-07-21T01:41:32.065 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:41:32.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:32.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f86a484e700 1 Processor -- start 2022-07-21T01:41:32.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f86a484e700 1 -- start start 2022-07-21T01:41:32.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f86a484e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86a00ac880 0x7f86a0143710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:32.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f86a484e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a0067a10 con 0x7f86a00b2710 2022-07-21T01:41:32.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f86a484e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a0067cb0 con 0x7f86a00b0270 2022-07-21T01:41:32.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f86a484e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a00ad740 con 0x7f86a00ac880 2022-07-21T01:41:32.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869ed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f86a00b2710 0x7f86a01401e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58218/0 (socket says 172.21.15.3:58218) 2022-07-21T01:41:32.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869ed9d700 1 -- 172.21.15.3:0/2666440407 learned_addr learned my addr 172.21.15.3:0/2666440407 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:32.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869f59e700 1 --2- 172.21.15.3:0/2666440407 >> v2:172.21.15.143:3300/0 conn(0x7f86a00ac880 0x7f86a0143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:32.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869dd9b700 1 -- 172.21.15.3:0/2666440407 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068298239 0 0) 0x7f86a0067a10 con 0x7f86a00b2710 2022-07-21T01:41:32.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869dd9b700 1 -- 172.21.15.3:0/2666440407 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8684003260 con 0x7f86a00b2710 2022-07-21T01:41:32.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869dd9b700 1 -- 172.21.15.3:0/2666440407 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888083936 0 0) 0x7f86a0067cb0 con 0x7f86a00b0270 2022-07-21T01:41:32.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869dd9b700 1 -- 172.21.15.3:0/2666440407 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a0067a10 con 0x7f86a00b0270 2022-07-21T01:41:32.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869f59e700 1 -- 172.21.15.3:0/2666440407 >> v1:172.21.15.143:6789/0 conn(0x7f86a00b0270 legacy=0x7f86a0143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:32.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869f59e700 1 -- 172.21.15.3:0/2666440407 >> v1:172.21.15.3:6789/0 conn(0x7f86a00b2710 legacy=0x7f86a01401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:32.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.168+0000 7f869f59e700 1 -- 172.21.15.3:0/2666440407 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0147160 con 0x7f86a00ac880 2022-07-21T01:41:32.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f869f59e700 1 --2- 172.21.15.3:0/2666440407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a00ac880 0x7f86a0143710 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f86980032f0 tx=0x7f86980091d0).ready entity=mon.1 client_cookie=76f3bd579c5a440d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:32.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f869dd9b700 1 -- 172.21.15.3:0/2666440407 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869800b600 con 0x7f86a00ac880 2022-07-21T01:41:32.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f869dd9b700 1 -- 172.21.15.3:0/2666440407 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869800b760 con 0x7f86a00ac880 2022-07-21T01:41:32.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f869dd9b700 1 -- 172.21.15.3:0/2666440407 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869800b8d0 con 0x7f86a00ac880 2022-07-21T01:41:32.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f86a484e700 1 -- 172.21.15.3:0/2666440407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a00ac880 msgr2=0x7f86a0143710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:32.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f86a484e700 1 --2- 172.21.15.3:0/2666440407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a00ac880 0x7f86a0143710 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f86980032f0 tx=0x7f86980091d0).stop 2022-07-21T01:41:32.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f86a484e700 1 -- 172.21.15.3:0/2666440407 shutdown_connections 2022-07-21T01:41:32.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f86a484e700 1 --2- 172.21.15.3:0/2666440407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a00ac880 0x7f86a0143710 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f86a484e700 1 -- 172.21.15.3:0/2666440407 >> 172.21.15.3:0/2666440407 conn(0x7f86a0060310 msgr2=0x7f86a00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:32.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f86a484e700 1 -- 172.21.15.3:0/2666440407 shutdown_connections 2022-07-21T01:41:32.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.172+0000 7f86a484e700 1 -- 172.21.15.3:0/2666440407 wait complete. 2022-07-21T01:41:32.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 Processor -- start 2022-07-21T01:41:32.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 -- start start 2022-07-21T01:41:32.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a00ac880 0x7f86a0142b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:32.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86a00b0270 0x7f86a01523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:32.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86a00b2710 0x7f86a01528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:32.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0146300 con 0x7f86a00b2710 2022-07-21T01:41:32.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a01564a0 con 0x7f86a00ac880 2022-07-21T01:41:32.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0156740 con 0x7f86a00b0270 2022-07-21T01:41:32.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86a00b2710 0x7f86a01528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:32.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86a00b2710 0x7f86a01528a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49946/0 (socket says 172.21.15.3:49946) 2022-07-21T01:41:32.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869f59e700 1 -- 172.21.15.3:0/2864485260 learned_addr learned my addr 172.21.15.3:0/2864485260 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:32.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869ed9d700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a00ac880 0x7f86a0142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:32.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869e59c700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86a00b0270 0x7f86a01523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:32.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869f59e700 1 -- 172.21.15.3:0/2864485260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86a00b0270 msgr2=0x7f86a01523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:32.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869f59e700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86a00b0270 0x7f86a01523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869f59e700 1 -- 172.21.15.3:0/2864485260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a00ac880 msgr2=0x7f86a0142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:32.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869f59e700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a00ac880 0x7f86a0142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869f59e700 1 -- 172.21.15.3:0/2864485260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8698009470 con 0x7f86a00b2710 2022-07-21T01:41:32.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f869f59e700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86a00b2710 0x7f86a01528a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f86980032f0 tx=0x7f8698016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:32.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f868ffff700 1 -- 172.21.15.3:0/2864485260 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869800b8d0 con 0x7f86a00b2710 2022-07-21T01:41:32.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f868ffff700 1 -- 172.21.15.3:0/2864485260 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869801f3f0 con 0x7f86a00b2710 2022-07-21T01:41:32.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a0146560 con 0x7f86a00b2710 2022-07-21T01:41:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f868ffff700 1 -- 172.21.15.3:0/2864485260 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869800b600 con 0x7f86a00b2710 2022-07-21T01:41:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.176+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a0146c10 con 0x7f86a00b2710 2022-07-21T01:41:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.180+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a01466f0 con 0x7f86a00b2710 2022-07-21T01:41:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.180+0000 7f868ffff700 1 -- 172.21.15.3:0/2864485260 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8698026030 con 0x7f86a00b2710 2022-07-21T01:41:32.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.180+0000 7f868ffff700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f867c053aa0 0x7f867c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:32.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.180+0000 7f868ffff700 1 -- 172.21.15.3:0/2864485260 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f869806cbd0 con 0x7f86a00b2710 2022-07-21T01:41:32.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.180+0000 7f869ed9d700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f867c053aa0 0x7f867c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:32.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.180+0000 7f869ed9d700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f867c053aa0 0x7f867c055f60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f869400b000 tx=0x7f8694009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:32.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.184+0000 7f868ffff700 1 -- 172.21.15.3:0/2864485260 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8698043990 con 0x7f86a00b2710 2022-07-21T01:41:32.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.348+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86a0143c00 con 0x7f86a00b2710 2022-07-21T01:41:32.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.348+0000 7f868ffff700 1 -- 172.21.15.3:0/2864485260 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+2486 (secure 0 0 0) 0x7f8698024090 con 0x7f86a00b2710 2022-07-21T01:41:32.351 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:32.351 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":60,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/297776761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":297776761},{"type":"v1","addr":"172.21.15.143:6833","nonce":297776761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":57},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3673188672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3673188672},{"type":"v1","addr":"172.21.15.3:6835","nonce":3673188672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:27.699850+0000","modified":"2022-07-21T01:41:28.704288+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7611},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7611":{"gid":7611,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/508489181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":508489181},{"type":"v1","addr":"172.21.15.3:6837","nonce":508489181}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-21T01:41:32.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f867c053aa0 msgr2=0x7f867c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:32.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f867c053aa0 0x7f867c055f60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f869400b000 tx=0x7f8694009040).stop 2022-07-21T01:41:32.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86a00b2710 msgr2=0x7f86a01528a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:32.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86a00b2710 0x7f86a01528a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f86980032f0 tx=0x7f8698016000).stop 2022-07-21T01:41:32.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 shutdown_connections 2022-07-21T01:41:32.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86a00b2710 0x7f86a01528a0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86a00b0270 0x7f86a01523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f867c053aa0 0x7f867c055f60 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 --2- 172.21.15.3:0/2864485260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a00ac880 0x7f86a0142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:32.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 >> 172.21.15.3:0/2864485260 conn(0x7f86a0060310 msgr2=0x7f86a00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 shutdown_connections 2022-07-21T01:41:32.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:32.352+0000 7f86a484e700 1 -- 172.21.15.3:0/2864485260 wait complete. 2022-07-21T01:41:32.363 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 60 2022-07-21T01:41:32.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:41:32.423 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:41:32.426 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:41:32.428 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:41:32.667 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:41:27.699850+0000', 'modified': '2022-07-21T01:41:28.704288+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7611}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7611': {'gid': 7611, 'name': 'c', 'rank': 0, 'incarnation': 59, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/508489181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 508489181}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 508489181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:41:32.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:41:32.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:41:32.788 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":7611,"want_state":"up:active","state":"up:active","rank_uptime":5.0764462699999999,"mdsmap_epoch":60,"osdmap_epoch":121,"osdmap_epoch_barrier":121,"uptime":9.4397076089999992} 2022-07-21T01:41:32.788 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 7611, "mdsmap_epoch": 60, "osdmap_epoch": 121, "osdmap_epoch_barrier": 121, "rank_uptime": 5.07644627, "state": "up:active", "uptime": 9.439707609, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:41:32.789 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:41:32.789 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:41:32.790 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:41:32.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:32.791 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:32.800 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:41:32.857 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:32.857 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:32.867 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:32.869 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:32.870 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:41:32.876 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:41:32.877 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:32.880 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:41:32.897 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:32.898 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:32.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:41:32.952+0000 7f337c75d080 -1 init, newargv = 0x563359605a30 newargc=9ceph-fuse[ 2022-07-21T01:41:32.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:23214]: starting ceph client 2022-07-21T01:41:32.954 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:32.956 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:32.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:32.957 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:41:32.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23214]: starting fuse 2022-07-21T01:41:34.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:41:34.084 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:34.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:34.143 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:34.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:34.148 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:41:34.155 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:41:34.156 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:41:34.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:34.158 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:41:34.159 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:41:34.160 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:41:34.161 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:41:34.162 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:41:34.163 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:41:34.163 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:41:34.164 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:41:34.165 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:41:34.166 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:41:34.167 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:41:34.167 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:41:34.168 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:41:34.168 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:41:34.169 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:41:34.169 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:41:34.169 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:41:34.170 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:41:34.171 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:41:34.173 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:41:34.174 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:41:34.175 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:41:34.176 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:41:34.224 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.23214.asok 2022-07-21T01:41:34.229 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23214.asok 2022-07-21T01:41:34.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:34.230 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23214.asok status 2022-07-21T01:41:34.364 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:41:34.365 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:41:34.365 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:41:34.365 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:41:34.365 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:41:34.366 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:41:34.366 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:41:34.366 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "23214", 2022-07-21T01:41:34.366 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:41:34.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:41:34.367 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:41:34.367 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:41:34.367 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 7738, 2022-07-21T01:41:34.368 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:41:34.368 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:41:34.368 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:41:34.368 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 7738 2022-07-21T01:41:34.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:41:34.369 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:41:34.369 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:41:34.369 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:41:34.370 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3215838684 2022-07-21T01:41:34.370 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:41:34.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:41:34.370 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:41:34.371 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:41:34.371 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:41:34.371 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3215838684 2022-07-21T01:41:34.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:41:34.372 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.7738 172.21.15.171:0/3215838684", 2022-07-21T01:41:34.372 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/3215838684", 2022-07-21T01:41:34.372 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:41:34.372 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 60, 2022-07-21T01:41:34.373 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 121, 2022-07-21T01:41:34.373 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:41:34.373 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:41:34.373 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:41:34.373 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:41:34.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:34.375 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:34.376 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:34.384 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:41:34.386 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:41:34.386 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:34.387 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:34.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:41:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 Processor -- start 2022-07-21T01:41:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 -- start start 2022-07-21T01:41:34.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9080b1f40 0x7fd9080b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd908148530 0x7fd9080b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9080b04e0 con 0x7fd9080a7250 2022-07-21T01:41:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9080a9250 con 0x7fd9080b1f40 2022-07-21T01:41:34.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080b0780 con 0x7fd908148530 2022-07-21T01:41:34.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90c8fd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd9080a7250 0x7fd9081448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58234/0 (socket says 172.21.15.3:58234) 2022-07-21T01:41:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90c8fd700 1 -- 172.21.15.3:0/2359401317 learned_addr learned my addr 172.21.15.3:0/2359401317 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd8ff7fe700 1 --2- 172.21.15.3:0/2359401317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd908148530 0x7fd9080b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd907fff700 1 --2- 172.21.15.3:0/2359401317 >> v2:172.21.15.143:3300/0 conn(0x7fd9080b1f40 0x7fd9080b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd9077fe700 1 -- 172.21.15.3:0/2359401317 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821488232 0 0) 0x7fd9080b04e0 con 0x7fd9080a7250 2022-07-21T01:41:34.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd9077fe700 1 -- 172.21.15.3:0/2359401317 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e8003260 con 0x7fd9080a7250 2022-07-21T01:41:34.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd8ff7fe700 1 -- 172.21.15.3:0/2359401317 >> v2:172.21.15.143:3300/0 conn(0x7fd9080b1f40 msgr2=0x7fd9080b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:34.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd8ff7fe700 1 --2- 172.21.15.3:0/2359401317 >> v2:172.21.15.143:3300/0 conn(0x7fd9080b1f40 0x7fd9080b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd8ff7fe700 1 -- 172.21.15.3:0/2359401317 >> v1:172.21.15.3:6789/0 conn(0x7fd9080a7250 legacy=0x7fd9081448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd8ff7fe700 1 -- 172.21.15.3:0/2359401317 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080b0a20 con 0x7fd908148530 2022-07-21T01:41:34.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd8ff7fe700 1 --2- 172.21.15.3:0/2359401317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd908148530 0x7fd9080b2320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fd8f0002f70 tx=0x7fd8f0006df0).ready entity=mon.2 client_cookie=934053f17b8f3ae5 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:34.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd9077fe700 1 -- 172.21.15.3:0/2359401317 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f000d4a0 con 0x7fd908148530 2022-07-21T01:41:34.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd9077fe700 1 -- 172.21.15.3:0/2359401317 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0000e90 con 0x7fd908148530 2022-07-21T01:41:34.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd9077fe700 1 -- 172.21.15.3:0/2359401317 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f000d7f0 con 0x7fd908148530 2022-07-21T01:41:34.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 -- 172.21.15.3:0/2359401317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd908148530 msgr2=0x7fd9080b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 --2- 172.21.15.3:0/2359401317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd908148530 0x7fd9080b2320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fd8f0002f70 tx=0x7fd8f0006df0).stop 2022-07-21T01:41:34.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 -- 172.21.15.3:0/2359401317 shutdown_connections 2022-07-21T01:41:34.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 --2- 172.21.15.3:0/2359401317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd908148530 0x7fd9080b2320 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 --2- 172.21.15.3:0/2359401317 >> v2:172.21.15.143:3300/0 conn(0x7fd9080b1f40 0x7fd9080b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 -- 172.21.15.3:0/2359401317 >> 172.21.15.3:0/2359401317 conn(0x7fd90805f840 msgr2=0x7fd90805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:34.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 -- 172.21.15.3:0/2359401317 shutdown_connections 2022-07-21T01:41:34.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.516+0000 7fd90e360700 1 -- 172.21.15.3:0/2359401317 wait complete. 2022-07-21T01:41:34.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 Processor -- start 2022-07-21T01:41:34.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 -- start start 2022-07-21T01:41:34.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9080a7250 0x7fd90814b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9080b1f40 0x7fd908064320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd908148530 0x7fd908064810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080ac2f0 con 0x7fd9080a7250 2022-07-21T01:41:34.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080aaa20 con 0x7fd908148530 2022-07-21T01:41:34.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080ab0b0 con 0x7fd9080b1f40 2022-07-21T01:41:34.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd907fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9080a7250 0x7fd90814b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd907fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9080a7250 0x7fd90814b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49956/0 (socket says 172.21.15.3:49956) 2022-07-21T01:41:34.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd907fff700 1 -- 172.21.15.3:0/3834415375 learned_addr learned my addr 172.21.15.3:0/3834415375 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:34.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd8ff7fe700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9080b1f40 0x7fd908064320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90c8fd700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd908148530 0x7fd908064810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd907fff700 1 -- 172.21.15.3:0/3834415375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9080b1f40 msgr2=0x7fd908064320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd907fff700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9080b1f40 0x7fd908064320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd907fff700 1 -- 172.21.15.3:0/3834415375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd908148530 msgr2=0x7fd908064810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd907fff700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd908148530 0x7fd908064810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd907fff700 1 -- 172.21.15.3:0/3834415375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f000b180 con 0x7fd9080a7250 2022-07-21T01:41:34.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90c8fd700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd908148530 0x7fd908064810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:34.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd907fff700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9080a7250 0x7fd90814b0e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7fd9080ab1f0 tx=0x7fd8f8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:34.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd905ffb700 1 -- 172.21.15.3:0/3834415375 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f8010780 con 0x7fd9080a7250 2022-07-21T01:41:34.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd905ffb700 1 -- 172.21.15.3:0/3834415375 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f80108e0 con 0x7fd9080a7250 2022-07-21T01:41:34.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd905ffb700 1 -- 172.21.15.3:0/3834415375 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f8010c40 con 0x7fd9080a7250 2022-07-21T01:41:34.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9080ab330 con 0x7fd9080a7250 2022-07-21T01:41:34.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.520+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9080ab870 con 0x7fd9080a7250 2022-07-21T01:41:34.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.524+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9080aba30 con 0x7fd9080a7250 2022-07-21T01:41:34.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.524+0000 7fd905ffb700 1 -- 172.21.15.3:0/3834415375 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd8f801c460 con 0x7fd9080a7250 2022-07-21T01:41:34.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.524+0000 7fd905ffb700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053950 0x7fd8ec055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.524+0000 7fd905ffb700 1 -- 172.21.15.3:0/3834415375 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd8f806c540 con 0x7fd9080a7250 2022-07-21T01:41:34.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.524+0000 7fd8ff7fe700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053950 0x7fd8ec055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.524+0000 7fd8ff7fe700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053950 0x7fd8ec055e10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fd8f001a000 tx=0x7fd8f0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:34.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.532+0000 7fd905ffb700 1 -- 172.21.15.3:0/3834415375 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd8f8043200 con 0x7fd9080a7250 2022-07-21T01:41:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.660+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9080aadb0 con 0x7fd9080a7250 2022-07-21T01:41:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd905ffb700 1 -- 172.21.15.3:0/3834415375 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16285 (secure 0 0 0) 0x7fd8f8046820 con 0x7fd9080a7250 2022-07-21T01:41:34.668 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:34.669 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":121,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:41:28.703604+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:41:26.156392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-21T01:41:27.012266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:41:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053950 msgr2=0x7fd8ec055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053950 0x7fd8ec055e10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fd8f001a000 tx=0x7fd8f0014000).stop 2022-07-21T01:41:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9080a7250 msgr2=0x7fd90814b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9080a7250 0x7fd90814b0e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7fd9080ab1f0 tx=0x7fd8f8006df0).stop 2022-07-21T01:41:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 shutdown_connections 2022-07-21T01:41:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053950 0x7fd8ec055e10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd908148530 0x7fd908064810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9080b1f40 0x7fd908064320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 --2- 172.21.15.3:0/3834415375 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9080a7250 0x7fd90814b0e0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 >> 172.21.15.3:0/3834415375 conn(0x7fd90805f840 msgr2=0x7fd9080b3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:34.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 shutdown_connections 2022-07-21T01:41:34.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.664+0000 7fd90e360700 1 -- 172.21.15.3:0/3834415375 wait complete. 2022-07-21T01:41:34.685 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:41:26.156392+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:41:27.012266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:41:34.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:41:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.804+0000 7fdcb8d8b700 1 Processor -- start 2022-07-21T01:41:34.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.804+0000 7fdcb8d8b700 1 -- start start 2022-07-21T01:41:34.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.804+0000 7fdcb8d8b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdcb4063a60 0x7fdcb4062050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.804+0000 7fdcb8d8b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcb40b1940 0x7fdcb413e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.804+0000 7fdcb8d8b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb40b10d0 con 0x7fdcb413e5b0 2022-07-21T01:41:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.804+0000 7fdcb8d8b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb4063e40 con 0x7fdcb40b1940 2022-07-21T01:41:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.804+0000 7fdcb8d8b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb4060200 con 0x7fdcb4063a60 2022-07-21T01:41:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb2ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcb40b1940 0x7fdcb413e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb3fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fdcb413e5b0 0x7fdcb41414c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:60364/0 (socket says 172.21.15.3:60364) 2022-07-21T01:41:34.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb37fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdcb4063a60 0x7fdcb4062050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb3fff700 1 -- 172.21.15.3:0/1987047283 learned_addr learned my addr 172.21.15.3:0/1987047283 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:34.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb2ffd700 1 -- 172.21.15.3:0/1987047283 >> v2:172.21.15.143:3300/0 conn(0x7fdcb4063a60 msgr2=0x7fdcb4062050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb2ffd700 1 --2- 172.21.15.3:0/1987047283 >> v2:172.21.15.143:3300/0 conn(0x7fdcb4063a60 0x7fdcb4062050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb27fc700 1 -- 172.21.15.3:0/1987047283 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268486147 0 0) 0x7fdcb40b10d0 con 0x7fdcb413e5b0 2022-07-21T01:41:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb2ffd700 1 -- 172.21.15.3:0/1987047283 >> v1:172.21.15.143:6789/0 conn(0x7fdcb413e5b0 legacy=0x7fdcb41414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb2ffd700 1 -- 172.21.15.3:0/1987047283 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb40b1300 con 0x7fdcb40b1940 2022-07-21T01:41:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb2ffd700 1 --2- 172.21.15.3:0/1987047283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 0x7fdcb413e0c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7fdca8002f70 tx=0x7fdca8006df0).ready entity=mon.0 client_cookie=4f9df70ad9156272 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb27fc700 1 -- 172.21.15.3:0/1987047283 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdca800d570 con 0x7fdcb40b1940 2022-07-21T01:41:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb27fc700 1 -- 172.21.15.3:0/1987047283 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca8000e90 con 0x7fdcb40b1940 2022-07-21T01:41:34.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb27fc700 1 -- 172.21.15.3:0/1987047283 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdca800d8c0 con 0x7fdcb40b1940 2022-07-21T01:41:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/1987047283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 msgr2=0x7fdcb413e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb8d8b700 1 --2- 172.21.15.3:0/1987047283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 0x7fdcb413e0c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7fdca8002f70 tx=0x7fdca8006df0).stop 2022-07-21T01:41:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/1987047283 shutdown_connections 2022-07-21T01:41:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb8d8b700 1 --2- 172.21.15.3:0/1987047283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 0x7fdcb413e0c0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb8d8b700 1 --2- 172.21.15.3:0/1987047283 >> v2:172.21.15.143:3300/0 conn(0x7fdcb4063a60 0x7fdcb4062050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/1987047283 >> 172.21.15.3:0/1987047283 conn(0x7fdcb405c620 msgr2=0x7fdcb405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/1987047283 shutdown_connections 2022-07-21T01:41:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.808+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/1987047283 wait complete. 2022-07-21T01:41:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb8d8b700 1 Processor -- start 2022-07-21T01:41:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb8d8b700 1 -- start start 2022-07-21T01:41:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb8d8b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcb4063a60 0x7fdcb4135210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb8d8b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 0x7fdcb4146900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb8d8b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcb413e5b0 0x7fdcb4150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb8d8b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb4136870 con 0x7fdcb40b1940 2022-07-21T01:41:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb8d8b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb4150500 con 0x7fdcb4063a60 2022-07-21T01:41:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb8d8b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb41507a0 con 0x7fdcb413e5b0 2022-07-21T01:41:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 0x7fdcb4146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 0x7fdcb4146900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49972/0 (socket says 172.21.15.3:49972) 2022-07-21T01:41:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb2ffd700 1 -- 172.21.15.3:0/2049447673 learned_addr learned my addr 172.21.15.3:0/2049447673 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb37fe700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcb4063a60 0x7fdcb4135210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.812+0000 7fdcb3fff700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcb413e5b0 0x7fdcb4150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb2ffd700 1 -- 172.21.15.3:0/2049447673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcb413e5b0 msgr2=0x7fdcb4150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb2ffd700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcb413e5b0 0x7fdcb4150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb2ffd700 1 -- 172.21.15.3:0/2049447673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcb4063a60 msgr2=0x7fdcb4135210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb2ffd700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcb4063a60 0x7fdcb4135210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb2ffd700 1 -- 172.21.15.3:0/2049447673 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca800b180 con 0x7fdcb40b1940 2022-07-21T01:41:34.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb37fe700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcb4063a60 0x7fdcb4135210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:34.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb2ffd700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 0x7fdcb4146900 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7fdca8002f70 tx=0x7fdca8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:34.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb0ff9700 1 -- 172.21.15.3:0/2049447673 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdca800dea0 con 0x7fdcb40b1940 2022-07-21T01:41:34.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb0ff9700 1 -- 172.21.15.3:0/2049447673 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca801e400 con 0x7fdcb40b1940 2022-07-21T01:41:34.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb0ff9700 1 -- 172.21.15.3:0/2049447673 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdca801e760 con 0x7fdcb40b1940 2022-07-21T01:41:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb4146df0 con 0x7fdcb40b1940 2022-07-21T01:41:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb4151130 con 0x7fdcb40b1940 2022-07-21T01:41:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb0ff9700 1 -- 172.21.15.3:0/2049447673 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdca801c070 con 0x7fdcb40b1940 2022-07-21T01:41:34.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb4136a70 con 0x7fdcb40b1940 2022-07-21T01:41:34.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb0ff9700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc94053b00 0x7fdc94055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:34.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb0ff9700 1 -- 172.21.15.3:0/2049447673 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdca806cc90 con 0x7fdcb40b1940 2022-07-21T01:41:34.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb37fe700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc94053b00 0x7fdc94055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:34.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.816+0000 7fdcb37fe700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc94053b00 0x7fdc94055fc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7fdca400b000 tx=0x7fdca4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:34.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.820+0000 7fdcb0ff9700 1 -- 172.21.15.3:0/2049447673 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdca8043940 con 0x7fdcb40b1940 2022-07-21T01:41:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.948+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fdcb4150960 con 0x7fdcb40b1940 2022-07-21T01:41:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.948+0000 7fdcb0ff9700 1 -- 172.21.15.3:0/2049447673 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v121) v1 ==== 110+0+10 (secure 0 0 0) 0x7fdca8023070 con 0x7fdcb40b1940 2022-07-21T01:41:34.952 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:41:34.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc94053b00 msgr2=0x7fdc94055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc94053b00 0x7fdc94055fc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7fdca400b000 tx=0x7fdca4008040).stop 2022-07-21T01:41:34.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 msgr2=0x7fdcb4146900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:34.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 0x7fdcb4146900 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7fdca8002f70 tx=0x7fdca8014000).stop 2022-07-21T01:41:34.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 shutdown_connections 2022-07-21T01:41:34.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcb413e5b0 0x7fdcb4150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcb40b1940 0x7fdcb4146900 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc94053b00 0x7fdc94055fc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 --2- 172.21.15.3:0/2049447673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcb4063a60 0x7fdcb4135210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:34.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 >> 172.21.15.3:0/2049447673 conn(0x7fdcb405c620 msgr2=0x7fdcb4139d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:34.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 shutdown_connections 2022-07-21T01:41:34.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:34.952+0000 7fdcb8d8b700 1 -- 172.21.15.3:0/2049447673 wait complete. 2022-07-21T01:41:34.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:41:35.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.076+0000 7fb6cc990700 1 Processor -- start 2022-07-21T01:41:35.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.076+0000 7fb6cc990700 1 -- start start 2022-07-21T01:41:35.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.076+0000 7fb6cc990700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb6c8146ae0 0x7fb6c81403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.076+0000 7fb6cc990700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4310 0x7fb6c81408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb6c80b4820 0x7fb6c80b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c81476a0 con 0x7fb6c80b4820 2022-07-21T01:41:35.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c8143cf0 con 0x7fb6c8146ae0 2022-07-21T01:41:35.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8143f90 con 0x7fb6c80b4310 2022-07-21T01:41:35.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c6d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb6c80b4820 0x7fb6c80b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c6d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb6c80b4820 0x7fb6c80b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49982/0 (socket says 172.21.15.3:49982) 2022-07-21T01:41:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c6d9d700 1 -- 172.21.15.3:0/1396498187 learned_addr learned my addr 172.21.15.3:0/1396498187 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c659c700 1 --2- 172.21.15.3:0/1396498187 >> v2:172.21.15.143:3300/0 conn(0x7fb6c8146ae0 0x7fb6c81403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c5d9b700 1 --2- 172.21.15.3:0/1396498187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4310 0x7fb6c81408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c6d9d700 1 -- 172.21.15.3:0/1396498187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4310 msgr2=0x7fb6c81408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:35.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c6d9d700 1 --2- 172.21.15.3:0/1396498187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4310 0x7fb6c81408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c6d9d700 1 -- 172.21.15.3:0/1396498187 >> v2:172.21.15.143:3300/0 conn(0x7fb6c8146ae0 msgr2=0x7fb6c81403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c6d9d700 1 --2- 172.21.15.3:0/1396498187 >> v2:172.21.15.143:3300/0 conn(0x7fb6c8146ae0 0x7fb6c81403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c6d9d700 1 -- 172.21.15.3:0/1396498187 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c80b0e20 con 0x7fb6c80b4820 2022-07-21T01:41:35.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c5d9b700 1 --2- 172.21.15.3:0/1396498187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4310 0x7fb6c81408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:35.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c6d9d700 1 --2- 172.21.15.3:0/1396498187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4820 0x7fb6c80b5060 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7fb6c00032f0 tx=0x7fb6c0009290).ready entity=mon.0 client_cookie=fd272ef33a5e02b2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c559a700 1 -- 172.21.15.3:0/1396498187 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6c000b870 con 0x7fb6c80b4820 2022-07-21T01:41:35.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c559a700 1 -- 172.21.15.3:0/1396498187 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c000b9d0 con 0x7fb6c80b4820 2022-07-21T01:41:35.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6c559a700 1 -- 172.21.15.3:0/1396498187 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6c000bb40 con 0x7fb6c80b4820 2022-07-21T01:41:35.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 -- 172.21.15.3:0/1396498187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4820 msgr2=0x7fb6c80b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 --2- 172.21.15.3:0/1396498187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4820 0x7fb6c80b5060 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7fb6c00032f0 tx=0x7fb6c0009290).stop 2022-07-21T01:41:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 -- 172.21.15.3:0/1396498187 shutdown_connections 2022-07-21T01:41:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 --2- 172.21.15.3:0/1396498187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4820 0x7fb6c80b5060 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 --2- 172.21.15.3:0/1396498187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4310 0x7fb6c81408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 --2- 172.21.15.3:0/1396498187 >> v2:172.21.15.143:3300/0 conn(0x7fb6c8146ae0 0x7fb6c81403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 -- 172.21.15.3:0/1396498187 >> 172.21.15.3:0/1396498187 conn(0x7fb6c805c620 msgr2=0x7fb6c805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 -- 172.21.15.3:0/1396498187 shutdown_connections 2022-07-21T01:41:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.080+0000 7fb6cc990700 1 -- 172.21.15.3:0/1396498187 wait complete. 2022-07-21T01:41:35.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 Processor -- start 2022-07-21T01:41:35.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 -- start start 2022-07-21T01:41:35.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4310 0x7fb6c814a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4820 0x7fb6c813f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6c8146ae0 0x7fb6c813f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c813fcf0 con 0x7fb6c80b4310 2022-07-21T01:41:35.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c81529b0 con 0x7fb6c8146ae0 2022-07-21T01:41:35.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8152c50 con 0x7fb6c80b4820 2022-07-21T01:41:35.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4310 0x7fb6c814a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4310 0x7fb6c814a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49984/0 (socket says 172.21.15.3:49984) 2022-07-21T01:41:35.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c659c700 1 -- 172.21.15.3:0/46609248 learned_addr learned my addr 172.21.15.3:0/46609248 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:35.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c6d9d700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6c8146ae0 0x7fb6c813f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c5d9b700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4820 0x7fb6c813f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c659c700 1 -- 172.21.15.3:0/46609248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4820 msgr2=0x7fb6c813f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c659c700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4820 0x7fb6c813f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c659c700 1 -- 172.21.15.3:0/46609248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6c8146ae0 msgr2=0x7fb6c813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c659c700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6c8146ae0 0x7fb6c813f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c659c700 1 -- 172.21.15.3:0/46609248 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c0009530 con 0x7fb6c80b4310 2022-07-21T01:41:35.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6c659c700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4310 0x7fb6c814a560 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7fb6bc0030b0 tx=0x7fb6bc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6b77fe700 1 -- 172.21.15.3:0/46609248 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6bc011780 con 0x7fb6c80b4310 2022-07-21T01:41:35.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6b77fe700 1 -- 172.21.15.3:0/46609248 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc0118e0 con 0x7fb6c80b4310 2022-07-21T01:41:35.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c8152ef0 con 0x7fb6c80b4310 2022-07-21T01:41:35.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6b77fe700 1 -- 172.21.15.3:0/46609248 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6bc011c40 con 0x7fb6c80b4310 2022-07-21T01:41:35.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.084+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c8112b60 con 0x7fb6c80b4310 2022-07-21T01:41:35.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.088+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c80574e0 con 0x7fb6c80b4310 2022-07-21T01:41:35.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.088+0000 7fb6b77fe700 1 -- 172.21.15.3:0/46609248 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb6bc01c460 con 0x7fb6c80b4310 2022-07-21T01:41:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.088+0000 7fb6b77fe700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6a4053aa0 0x7fb6a4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.088+0000 7fb6b77fe700 1 -- 172.21.15.3:0/46609248 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb6bc06c390 con 0x7fb6c80b4310 2022-07-21T01:41:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.088+0000 7fb6c5d9b700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6a4053aa0 0x7fb6a4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.088+0000 7fb6c5d9b700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6a4053aa0 0x7fb6a4055f60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7fb6b800c000 tx=0x7fb6b8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.092+0000 7fb6b77fe700 1 -- 172.21.15.3:0/46609248 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb6bc0430f0 con 0x7fb6c80b4310 2022-07-21T01:41:35.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.216+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb6c8153390 con 0x7fb6c80b4310 2022-07-21T01:41:35.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6b77fe700 1 -- 172.21.15.3:0/46609248 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v121) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb6bc046710 con 0x7fb6c80b4310 2022-07-21T01:41:35.222 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:35.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6a4053aa0 msgr2=0x7fb6a4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6a4053aa0 0x7fb6a4055f60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7fb6b800c000 tx=0x7fb6b8008040).stop 2022-07-21T01:41:35.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4310 msgr2=0x7fb6c814a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4310 0x7fb6c814a560 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7fb6bc0030b0 tx=0x7fb6bc008040).stop 2022-07-21T01:41:35.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 shutdown_connections 2022-07-21T01:41:35.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6a4053aa0 0x7fb6a4055f60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6c8146ae0 0x7fb6c813f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6c80b4820 0x7fb6c813f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 --2- 172.21.15.3:0/46609248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c80b4310 0x7fb6c814a560 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 >> 172.21.15.3:0/46609248 conn(0x7fb6c805c620 msgr2=0x7fb6c80b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:35.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 shutdown_connections 2022-07-21T01:41:35.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.220+0000 7fb6cc990700 1 -- 172.21.15.3:0/46609248 wait complete. 2022-07-21T01:41:35.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:41:35.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.344+0000 7f55029aa700 1 Processor -- start 2022-07-21T01:41:35.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.344+0000 7f55029aa700 1 -- start start 2022-07-21T01:41:35.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.344+0000 7f55029aa700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54fc0619b0 0x7f54fc0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.344+0000 7f55029aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b6ee0 0x7f54fc143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f54fc0b5ed0 0x7f54fc0b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc142f60 con 0x7f54fc0619b0 2022-07-21T01:41:35.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc061d90 con 0x7f54fc0b5ed0 2022-07-21T01:41:35.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0ba5a0 con 0x7f54fc0b6ee0 2022-07-21T01:41:35.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55019a8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54fc0619b0 0x7f54fc0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55019a8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f54fc0619b0 0x7f54fc0b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49992/0 (socket says 172.21.15.3:49992) 2022-07-21T01:41:35.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55019a8700 1 -- 172.21.15.3:0/247228446 learned_addr learned my addr 172.21.15.3:0/247228446 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:35.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55011a7700 1 --2- 172.21.15.3:0/247228446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b6ee0 0x7f54fc143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55021a9700 1 --2- 172.21.15.3:0/247228446 >> v2:172.21.15.143:3300/0 conn(0x7f54fc0b5ed0 0x7f54fc0b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55019a8700 1 -- 172.21.15.3:0/247228446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b6ee0 msgr2=0x7f54fc143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:35.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55019a8700 1 --2- 172.21.15.3:0/247228446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b6ee0 0x7f54fc143890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55019a8700 1 -- 172.21.15.3:0/247228446 >> v2:172.21.15.143:3300/0 conn(0x7f54fc0b5ed0 msgr2=0x7f54fc0b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:35.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55019a8700 1 --2- 172.21.15.3:0/247228446 >> v2:172.21.15.143:3300/0 conn(0x7f54fc0b5ed0 0x7f54fc0b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55019a8700 1 -- 172.21.15.3:0/247228446 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc143150 con 0x7f54fc0619b0 2022-07-21T01:41:35.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55019a8700 1 --2- 172.21.15.3:0/247228446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 0x7f54fc0b48a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7f54f8002f70 tx=0x7f54f8006eb0).ready entity=mon.0 client_cookie=6a171eac7f62d86e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55009a6700 1 -- 172.21.15.3:0/247228446 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54f800d570 con 0x7f54fc0619b0 2022-07-21T01:41:35.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55009a6700 1 -- 172.21.15.3:0/247228446 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f8000e90 con 0x7f54fc0619b0 2022-07-21T01:41:35.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55009a6700 1 -- 172.21.15.3:0/247228446 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54f800d900 con 0x7f54fc0619b0 2022-07-21T01:41:35.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 -- 172.21.15.3:0/247228446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 msgr2=0x7f54fc0b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 --2- 172.21.15.3:0/247228446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 0x7f54fc0b48a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7f54f8002f70 tx=0x7f54f8006eb0).stop 2022-07-21T01:41:35.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 -- 172.21.15.3:0/247228446 shutdown_connections 2022-07-21T01:41:35.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 --2- 172.21.15.3:0/247228446 >> v2:172.21.15.143:3300/0 conn(0x7f54fc0b5ed0 0x7f54fc0b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 --2- 172.21.15.3:0/247228446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b6ee0 0x7f54fc143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 --2- 172.21.15.3:0/247228446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 0x7f54fc0b48a0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 -- 172.21.15.3:0/247228446 >> 172.21.15.3:0/247228446 conn(0x7f54fc06af80 msgr2=0x7f54fc05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:35.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 -- 172.21.15.3:0/247228446 shutdown_connections 2022-07-21T01:41:35.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.348+0000 7f55029aa700 1 -- 172.21.15.3:0/247228446 wait complete. 2022-07-21T01:41:35.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55029aa700 1 Processor -- start 2022-07-21T01:41:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55029aa700 1 -- start start 2022-07-21T01:41:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55029aa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 0x7f54fc0b9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55029aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b5ed0 0x7f54fc0ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55029aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54fc0b6ee0 0x7f54fc0afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55029aa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc149f90 con 0x7f54fc0619b0 2022-07-21T01:41:35.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55029aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc149140 con 0x7f54fc0b6ee0 2022-07-21T01:41:35.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55029aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc1498c0 con 0x7f54fc0b5ed0 2022-07-21T01:41:35.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55019a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 0x7f54fc0b9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55019a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 0x7f54fc0b9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50000/0 (socket says 172.21.15.3:50000) 2022-07-21T01:41:35.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55019a8700 1 -- 172.21.15.3:0/3951374331 learned_addr learned my addr 172.21.15.3:0/3951374331 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:35.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55011a7700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b5ed0 0x7f54fc0ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55021a9700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54fc0b6ee0 0x7f54fc0afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55019a8700 1 -- 172.21.15.3:0/3951374331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b5ed0 msgr2=0x7f54fc0ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55019a8700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b5ed0 0x7f54fc0ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55019a8700 1 -- 172.21.15.3:0/3951374331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54fc0b6ee0 msgr2=0x7f54fc0afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55019a8700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54fc0b6ee0 0x7f54fc0afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.352+0000 7f55019a8700 1 -- 172.21.15.3:0/3951374331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f800b1e0 con 0x7f54fc0619b0 2022-07-21T01:41:35.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f55019a8700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 0x7f54fc0b9570 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f54fc149640 tx=0x7f54f8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f54eaffd700 1 -- 172.21.15.3:0/3951374331 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54f80068b0 con 0x7f54fc0619b0 2022-07-21T01:41:35.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f54eaffd700 1 -- 172.21.15.3:0/3951374331 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f8011b40 con 0x7f54fc0619b0 2022-07-21T01:41:35.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc149370 con 0x7f54fc0619b0 2022-07-21T01:41:35.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f54eaffd700 1 -- 172.21.15.3:0/3951374331 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54f8011ea0 con 0x7f54fc0619b0 2022-07-21T01:41:35.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54fc0b9f40 con 0x7f54fc0619b0 2022-07-21T01:41:35.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f54eaffd700 1 -- 172.21.15.3:0/3951374331 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f54f8025030 con 0x7f54fc0619b0 2022-07-21T01:41:35.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54fc0574e0 con 0x7f54fc0619b0 2022-07-21T01:41:35.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f54eaffd700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54dc053980 0x7f54dc055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f54eaffd700 1 -- 172.21.15.3:0/3951374331 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f54f806d170 con 0x7f54fc0619b0 2022-07-21T01:41:35.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f55011a7700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54dc053980 0x7f54dc055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.356+0000 7f55011a7700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54dc053980 0x7f54dc055e40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f54ec00b000 tx=0x7f54ec006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.360+0000 7f54eaffd700 1 -- 172.21.15.3:0/3951374331 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54f8043e20 con 0x7f54fc0619b0 2022-07-21T01:41:35.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.488+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f54fc05db60 con 0x7f54fc0619b0 2022-07-21T01:41:35.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.488+0000 7f54eaffd700 1 -- 172.21.15.3:0/3951374331 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v121) v1 ==== 104+0+10 (secure 0 0 0) 0x7f54f8023020 con 0x7f54fc0619b0 2022-07-21T01:41:35.493 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:35.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54dc053980 msgr2=0x7f54dc055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54dc053980 0x7f54dc055e40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f54ec00b000 tx=0x7f54ec006df0).stop 2022-07-21T01:41:35.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 msgr2=0x7f54fc0b9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 0x7f54fc0b9570 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f54fc149640 tx=0x7f54f8015000).stop 2022-07-21T01:41:35.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 shutdown_connections 2022-07-21T01:41:35.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54dc053980 0x7f54dc055e40 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54fc0b6ee0 0x7f54fc0afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54fc0b5ed0 0x7f54fc0ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 --2- 172.21.15.3:0/3951374331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc0619b0 0x7f54fc0b9570 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 >> 172.21.15.3:0/3951374331 conn(0x7f54fc06af80 msgr2=0x7f54fc140e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:35.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 shutdown_connections 2022-07-21T01:41:35.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.492+0000 7f55029aa700 1 -- 172.21.15.3:0/3951374331 wait complete. 2022-07-21T01:41:35.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:41:35.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f3847e12700 1 Processor -- start 2022-07-21T01:41:35.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f3847e12700 1 -- start start 2022-07-21T01:41:35.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f3847e12700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f38400b1250 0x7f38400b1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f3847e12700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400acd00 0x7f3840140d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f3847e12700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f384013d5f0 0x7f384013da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f3847e12700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384013df50 con 0x7f384013d5f0 2022-07-21T01:41:35.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f3847e12700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3840141220 con 0x7f38400b1250 2022-07-21T01:41:35.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f3847e12700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400ac950 con 0x7f38400acd00 2022-07-21T01:41:35.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38463af700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f384013d5f0 0x7f384013da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38463af700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f384013d5f0 0x7f384013da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50010/0 (socket says 172.21.15.3:50010) 2022-07-21T01:41:35.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38463af700 1 -- 172.21.15.3:0/2540992725 learned_addr learned my addr 172.21.15.3:0/2540992725 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:35.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38453ad700 1 --2- 172.21.15.3:0/2540992725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400acd00 0x7f3840140d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f3845bae700 1 --2- 172.21.15.3:0/2540992725 >> v2:172.21.15.143:3300/0 conn(0x7f38400b1250 0x7f38400b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38463af700 1 -- 172.21.15.3:0/2540992725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400acd00 msgr2=0x7f3840140d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:35.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38463af700 1 --2- 172.21.15.3:0/2540992725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400acd00 0x7f3840140d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38463af700 1 -- 172.21.15.3:0/2540992725 >> v2:172.21.15.143:3300/0 conn(0x7f38400b1250 msgr2=0x7f38400b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:35.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38463af700 1 --2- 172.21.15.3:0/2540992725 >> v2:172.21.15.143:3300/0 conn(0x7f38400b1250 0x7f38400b1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38463af700 1 -- 172.21.15.3:0/2540992725 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400ac1b0 con 0x7f384013d5f0 2022-07-21T01:41:35.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.608+0000 7f38463af700 1 --2- 172.21.15.3:0/2540992725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f384013d5f0 0x7f384013da60 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f38380032f0 tx=0x7f3838009290).ready entity=mon.0 client_cookie=521bfba60dde0ace server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3844bac700 1 -- 172.21.15.3:0/2540992725 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383800b680 con 0x7f384013d5f0 2022-07-21T01:41:35.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3844bac700 1 -- 172.21.15.3:0/2540992725 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383800b7e0 con 0x7f384013d5f0 2022-07-21T01:41:35.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3844bac700 1 -- 172.21.15.3:0/2540992725 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383800b950 con 0x7f384013d5f0 2022-07-21T01:41:35.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3847e12700 1 -- 172.21.15.3:0/2540992725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f384013d5f0 msgr2=0x7f384013da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3847e12700 1 --2- 172.21.15.3:0/2540992725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f384013d5f0 0x7f384013da60 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f38380032f0 tx=0x7f3838009290).stop 2022-07-21T01:41:35.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3847e12700 1 -- 172.21.15.3:0/2540992725 shutdown_connections 2022-07-21T01:41:35.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3847e12700 1 --2- 172.21.15.3:0/2540992725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f384013d5f0 0x7f384013da60 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3847e12700 1 --2- 172.21.15.3:0/2540992725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400acd00 0x7f3840140d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3847e12700 1 --2- 172.21.15.3:0/2540992725 >> v2:172.21.15.143:3300/0 conn(0x7f38400b1250 0x7f38400b1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3847e12700 1 -- 172.21.15.3:0/2540992725 >> 172.21.15.3:0/2540992725 conn(0x7f384005f840 msgr2=0x7f384005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:35.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3847e12700 1 -- 172.21.15.3:0/2540992725 shutdown_connections 2022-07-21T01:41:35.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.612+0000 7f3847e12700 1 -- 172.21.15.3:0/2540992725 wait complete. 2022-07-21T01:41:35.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3847e12700 1 Processor -- start 2022-07-21T01:41:35.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3847e12700 1 -- start start 2022-07-21T01:41:35.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3847e12700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38400acd00 0x7f3840135500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3847e12700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400b1250 0x7f384013c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3847e12700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384013d5f0 0x7f384013cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3847e12700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840062f40 con 0x7f38400acd00 2022-07-21T01:41:35.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3847e12700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384014b630 con 0x7f384013d5f0 2022-07-21T01:41:35.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3847e12700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384014b860 con 0x7f38400b1250 2022-07-21T01:41:35.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3845bae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38400acd00 0x7f3840135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3845bae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38400acd00 0x7f3840135500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50012/0 (socket says 172.21.15.3:50012) 2022-07-21T01:41:35.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3845bae700 1 -- 172.21.15.3:0/767085283 learned_addr learned my addr 172.21.15.3:0/767085283 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:35.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f38463af700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384013d5f0 0x7f384013cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f38453ad700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400b1250 0x7f384013c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f38463af700 1 -- 172.21.15.3:0/767085283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400b1250 msgr2=0x7f384013c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f38463af700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400b1250 0x7f384013c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f38463af700 1 -- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38400acd00 msgr2=0x7f3840135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f38463af700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38400acd00 0x7f3840135500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f38463af700 1 -- 172.21.15.3:0/767085283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3838009530 con 0x7f384013d5f0 2022-07-21T01:41:35.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f3845bae700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38400acd00 0x7f3840135500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:35.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f38463af700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384013d5f0 0x7f384013cb90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f38380032f0 tx=0x7f3838017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f382effd700 1 -- 172.21.15.3:0/767085283 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383800b680 con 0x7f384013d5f0 2022-07-21T01:41:35.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.616+0000 7f382effd700 1 -- 172.21.15.3:0/767085283 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383800be80 con 0x7f384013d5f0 2022-07-21T01:41:35.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.620+0000 7f382effd700 1 -- 172.21.15.3:0/767085283 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383801b630 con 0x7f384013d5f0 2022-07-21T01:41:35.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.620+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384013fc80 con 0x7f384013d5f0 2022-07-21T01:41:35.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.620+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38401359f0 con 0x7f384013d5f0 2022-07-21T01:41:35.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.620+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38400ac410 con 0x7f384013d5f0 2022-07-21T01:41:35.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.620+0000 7f382effd700 1 -- 172.21.15.3:0/767085283 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f383800f070 con 0x7f384013d5f0 2022-07-21T01:41:35.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.620+0000 7f382effd700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3820053a10 0x7f3820055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.620+0000 7f382effd700 1 -- 172.21.15.3:0/767085283 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f383806cea0 con 0x7f384013d5f0 2022-07-21T01:41:35.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.620+0000 7f3845bae700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3820053a10 0x7f3820055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.620+0000 7f3845bae700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3820053a10 0x7f3820055ed0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f383c00b000 tx=0x7f383c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.632+0000 7f382effd700 1 -- 172.21.15.3:0/767085283 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3838043b50 con 0x7f384013d5f0 2022-07-21T01:41:35.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.760+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f38400574e0 con 0x7f384013d5f0 2022-07-21T01:41:35.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.760+0000 7f382effd700 1 -- 172.21.15.3:0/767085283 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3838047170 con 0x7f384013d5f0 2022-07-21T01:41:35.762 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:41:35.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.760+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3820053a10 msgr2=0x7f3820055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.760+0000 7f3847e12700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3820053a10 0x7f3820055ed0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f383c00b000 tx=0x7f383c006df0).stop 2022-07-21T01:41:35.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.760+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384013d5f0 msgr2=0x7f384013cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.760+0000 7f3847e12700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384013d5f0 0x7f384013cb90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f38380032f0 tx=0x7f3838017000).stop 2022-07-21T01:41:35.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.764+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 shutdown_connections 2022-07-21T01:41:35.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.764+0000 7f3847e12700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3820053a10 0x7f3820055ed0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.764+0000 7f3847e12700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384013d5f0 0x7f384013cb90 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.764+0000 7f3847e12700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38400b1250 0x7f384013c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.764+0000 7f3847e12700 1 --2- 172.21.15.3:0/767085283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38400acd00 0x7f3840135500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.764+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 >> 172.21.15.3:0/767085283 conn(0x7f384005f840 msgr2=0x7f3840133650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:35.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.764+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 shutdown_connections 2022-07-21T01:41:35.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.764+0000 7f3847e12700 1 -- 172.21.15.3:0/767085283 wait complete. 2022-07-21T01:41:35.776 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:41:35.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:35.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe83ce46700 1 Processor -- start 2022-07-21T01:41:35.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe83ce46700 1 -- start start 2022-07-21T01:41:35.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe83ce46700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe8380b01a0 0x7fe8380a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe83ce46700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8380a7250 con 0x7fe8380afd60 2022-07-21T01:41:35.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe83ce46700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8380a7480 con 0x7fe8380b2710 2022-07-21T01:41:35.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe83ce46700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8380a7720 con 0x7fe8380b01a0 2022-07-21T01:41:35.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe836ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe8380b01a0 0x7fe8380a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe837fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe8380afd60 0x7fe838151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58304/0 (socket says 172.21.15.3:58304) 2022-07-21T01:41:35.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe837fff700 1 -- 172.21.15.3:0/3246274676 learned_addr learned my addr 172.21.15.3:0/3246274676 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:35.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe836ffd700 1 -- 172.21.15.3:0/3246274676 >> v1:172.21.15.143:6789/0 conn(0x7fe8380b2710 legacy=0x7fe83814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:35.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe836ffd700 1 -- 172.21.15.3:0/3246274676 >> v1:172.21.15.3:6789/0 conn(0x7fe8380afd60 legacy=0x7fe838151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe836ffd700 1 -- 172.21.15.3:0/3246274676 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8380a79c0 con 0x7fe8380b01a0 2022-07-21T01:41:35.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe836ffd700 1 --2- 172.21.15.3:0/3246274676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380b01a0 0x7fe8380a8540 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7fe838061c10 tx=0x7fe8280091d0).ready entity=mon.0 client_cookie=584eecbde9f70a1a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.876+0000 7fe8367fc700 1 -- 172.21.15.3:0/3246274676 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe828009ea0 con 0x7fe8380b01a0 2022-07-21T01:41:35.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe8367fc700 1 -- 172.21.15.3:0/3246274676 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82800b860 con 0x7fe8380b01a0 2022-07-21T01:41:35.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe8367fc700 1 -- 172.21.15.3:0/3246274676 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe82800b9d0 con 0x7fe8380b01a0 2022-07-21T01:41:35.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 -- 172.21.15.3:0/3246274676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380b01a0 msgr2=0x7fe8380a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 --2- 172.21.15.3:0/3246274676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380b01a0 0x7fe8380a8540 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7fe838061c10 tx=0x7fe8280091d0).stop 2022-07-21T01:41:35.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 -- 172.21.15.3:0/3246274676 shutdown_connections 2022-07-21T01:41:35.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 --2- 172.21.15.3:0/3246274676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380b01a0 0x7fe8380a8540 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 -- 172.21.15.3:0/3246274676 >> 172.21.15.3:0/3246274676 conn(0x7fe838060310 msgr2=0x7fe8380b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:35.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 -- 172.21.15.3:0/3246274676 shutdown_connections 2022-07-21T01:41:35.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 -- 172.21.15.3:0/3246274676 wait complete. 2022-07-21T01:41:35.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 Processor -- start 2022-07-21T01:41:35.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 -- start start 2022-07-21T01:41:35.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380afd60 0x7fe83811cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8380b01a0 0x7fe838153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8380b2710 0x7fe838154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe838063320 con 0x7fe8380afd60 2022-07-21T01:41:35.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe838064ad0 con 0x7fe8380b2710 2022-07-21T01:41:35.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe83ce46700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83815a990 con 0x7fe8380b01a0 2022-07-21T01:41:35.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe8377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380afd60 0x7fe83811cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe8377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380afd60 0x7fe83811cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50026/0 (socket says 172.21.15.3:50026) 2022-07-21T01:41:35.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe8377fe700 1 -- 172.21.15.3:0/328919569 learned_addr learned my addr 172.21.15.3:0/328919569 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:35.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe837fff700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8380b2710 0x7fe838154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.880+0000 7fe836ffd700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8380b01a0 0x7fe838153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe8377fe700 1 -- 172.21.15.3:0/328919569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8380b01a0 msgr2=0x7fe838153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe8377fe700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8380b01a0 0x7fe838153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe8377fe700 1 -- 172.21.15.3:0/328919569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8380b2710 msgr2=0x7fe838154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe8377fe700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8380b2710 0x7fe838154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:35.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe8377fe700 1 -- 172.21.15.3:0/328919569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe828009470 con 0x7fe8380afd60 2022-07-21T01:41:35.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe836ffd700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8380b01a0 0x7fe838153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe8377fe700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380afd60 0x7fe83811cfb0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7fe820002f30 tx=0x7fe820006d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe834ff9700 1 -- 172.21.15.3:0/328919569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe820009e20 con 0x7fe8380afd60 2022-07-21T01:41:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe834ff9700 1 -- 172.21.15.3:0/328919569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe820002320 con 0x7fe8380afd60 2022-07-21T01:41:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe834ff9700 1 -- 172.21.15.3:0/328919569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe820002680 con 0x7fe8380afd60 2022-07-21T01:41:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83815ac30 con 0x7fe8380afd60 2022-07-21T01:41:35.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8380647c0 con 0x7fe8380afd60 2022-07-21T01:41:35.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8380574e0 con 0x7fe8380afd60 2022-07-21T01:41:35.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe834ff9700 1 -- 172.21.15.3:0/328919569 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe82000f030 con 0x7fe8380afd60 2022-07-21T01:41:35.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe834ff9700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe81c053a70 0x7fe81c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:35.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe834ff9700 1 -- 172.21.15.3:0/328919569 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe82006bcc0 con 0x7fe8380afd60 2022-07-21T01:41:35.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.884+0000 7fe836ffd700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe81c053a70 0x7fe81c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:35.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.888+0000 7fe836ffd700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe81c053a70 0x7fe81c055f30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fe828019000 tx=0x7fe828016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:35.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:35.892+0000 7fe834ff9700 1 -- 172.21.15.3:0/328919569 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe820042a20 con 0x7fe8380afd60 2022-07-21T01:41:36.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.056+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe838069a70 con 0x7fe8380afd60 2022-07-21T01:41:36.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.056+0000 7fe834ff9700 1 -- 172.21.15.3:0/328919569 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+2486 (secure 0 0 0) 0x7fe820046040 con 0x7fe8380afd60 2022-07-21T01:41:36.060 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:36.060 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":60,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/297776761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":297776761},{"type":"v1","addr":"172.21.15.143:6833","nonce":297776761}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":57},{"gid":7602,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3673188672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3673188672},{"type":"v1","addr":"172.21.15.3:6835","nonce":3673188672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:27.699850+0000","modified":"2022-07-21T01:41:28.704288+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7611},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7611":{"gid":7611,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/508489181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":508489181},{"type":"v1","addr":"172.21.15.3:6837","nonce":508489181}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-21T01:41:36.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe81c053a70 msgr2=0x7fe81c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:36.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe81c053a70 0x7fe81c055f30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fe828019000 tx=0x7fe828016000).stop 2022-07-21T01:41:36.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380afd60 msgr2=0x7fe83811cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:36.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380afd60 0x7fe83811cfb0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7fe820002f30 tx=0x7fe820006d70).stop 2022-07-21T01:41:36.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 shutdown_connections 2022-07-21T01:41:36.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe81c053a70 0x7fe81c055f30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:36.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8380b2710 0x7fe838154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:36.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8380b01a0 0x7fe838153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:36.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 --2- 172.21.15.3:0/328919569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8380afd60 0x7fe83811cfb0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:36.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 >> 172.21.15.3:0/328919569 conn(0x7fe838060310 msgr2=0x7fe83814ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:36.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 shutdown_connections 2022-07-21T01:41:36.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:36.060+0000 7fe83ce46700 1 -- 172.21.15.3:0/328919569 wait complete. 2022-07-21T01:41:36.065 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 60 2022-07-21T01:41:36.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:41:27.699850+0000', 'modified': '2022-07-21T01:41:28.704288+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7611}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7611': {'gid': 7611, 'name': 'c', 'rank': 0, 'incarnation': 59, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/508489181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 508489181}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 508489181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:41:36.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:41:36.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:41:36.498 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":7611,"want_state":"up:active","state":"up:active","rank_uptime":8.7858842359999993,"mdsmap_epoch":60,"osdmap_epoch":121,"osdmap_epoch_barrier":121,"uptime":13.149138831} 2022-07-21T01:41:36.498 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 7611, "mdsmap_epoch": 60, "osdmap_epoch": 121, "osdmap_epoch_barrier": 121, "rank_uptime": 8.785884236, "state": "up:active", "uptime": 13.149138831, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:41:36.499 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:41:36.499 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:41:36.500 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:41:36.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:36.501 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:36.516 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:41:36.534 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:36.535 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:36.603 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:36.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:36.604 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:41:36.612 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:41:36.613 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:36.658 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:41:36.678 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:36.679 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:36.690 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:36.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:36.692 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:41:36.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:41:36.732+0000 7ff5e0f5f080 -1 init, newargv = 0x559dca760860 newargc=9 2022-07-21T01:41:36.734 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22788]: starting ceph client 2022-07-21T01:41:36.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22788]: starting fuse 2022-07-21T01:41:37.843 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:41:37.863 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:41:37.864 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:41:37.919 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:41:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:41:37.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:37.922 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:41:37.932 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:41:37.932 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:41:37.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:37.934 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:41:37.935 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:41:37.936 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:41:37.937 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:41:37.938 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:41:37.938 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:41:37.939 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:41:37.940 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:41:37.941 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:41:37.942 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:41:37.943 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:41:37.943 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:41:37.944 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:41:37.945 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:41:37.946 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:41:37.946 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:41:37.947 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:41:37.948 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:41:37.949 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:41:37.949 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:41:37.950 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:41:37.951 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:41:37.952 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:41:37.996 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.22788.asok 2022-07-21T01:41:38.001 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22788.asok 2022-07-21T01:41:38.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:38.002 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22788.asok status 2022-07-21T01:41:38.095 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:41:38.095 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:41:38.096 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:41:38.096 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:41:38.097 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:41:38.097 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:41:38.099 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:41:38.100 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "22788", 2022-07-21T01:41:38.100 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:41:38.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:41:38.101 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:41:38.101 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:41:38.101 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 6773, 2022-07-21T01:41:38.107 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:41:38.107 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:41:38.108 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:41:38.108 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 6773 2022-07-21T01:41:38.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:41:38.109 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:41:38.109 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:41:38.109 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:41:38.110 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3445305223 2022-07-21T01:41:38.110 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:41:38.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:41:38.111 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:41:38.112 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:41:38.112 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:41:38.113 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3445305223 2022-07-21T01:41:38.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:41:38.113 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.6773 172.21.15.174:0/3445305223", 2022-07-21T01:41:38.114 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/3445305223", 2022-07-21T01:41:38.114 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:41:38.114 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 60, 2022-07-21T01:41:38.115 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 121, 2022-07-21T01:41:38.115 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:41:38.115 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:41:38.116 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:41:38.116 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:41:38.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:38.118 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:38.119 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:38.131 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:41:38.132 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:41:38.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:38.133 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:38.153 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:38.155 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:38.156 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:38.165 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:41:38.166 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:41:38.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:38.168 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:38.168 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:38.181 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:41:38.182 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:41:38.183 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:41:38.183 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:41:38.184 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:38.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23214]: fuse finished with error 0 and tester_r 0 2022-07-21T01:41:38.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:38.204 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:38.205 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:38.255 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:41:38.256 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:41:38.257 INFO:teuthology.orchestra.run:waiting for 900 2022-07-21T01:41:38.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:41:38.353 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:38.360 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:38.360 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:38.361 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:38.369 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:41:38.371 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:41:38.371 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:38.372 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:38.372 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:38.383 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:41:38.383 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:41:38.384 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:41:38.385 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:41:38.385 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:38.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22788]: fuse finished with error 0 and tester_r 0 2022-07-21T01:41:38.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:38.446 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:38.447 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:38.455 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:41:38.457 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:41:38.458 INFO:teuthology.orchestra.run:waiting for 900 2022-07-21T01:41:38.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:41:38.531 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:38.542 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:41:38.543 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:41:38.544 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:41:38.544 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:41:38.545 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:41:38.546 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:41:38.561 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:41:38.562 DEBUG:teuthology.parallel:result is None 2022-07-21T01:41:38.563 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:41:38.564 DEBUG:teuthology.parallel:result is None 2022-07-21T01:41:38.566 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:41:38.567 DEBUG:teuthology.parallel:result is None 2022-07-21T01:41:38.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2022-07-21T01:41:38.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2022-07-21T01:41:38.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2022-07-21T01:41:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.668+0000 7fcf05845700 1 Processor -- start 2022-07-21T01:41:38.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.668+0000 7fcf05845700 1 -- start start 2022-07-21T01:41:38.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.668+0000 7fa9031a8700 1 Processor -- start 2022-07-21T01:41:38.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa9031a8700 1 -- start start 2022-07-21T01:41:38.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcf000b0d10 0x7fcf00146df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcf000b0560 0x7fcf00140350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf000b12d0 con 0x7fcf000a9320 2022-07-21T01:41:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf000a9700 con 0x7fcf000b0560 2022-07-21T01:41:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf000b1500 con 0x7fcf000b0d10 2022-07-21T01:41:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa9031a8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa8fc140e80 0x7fa8fc0ae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05044700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcf000b0560 0x7fcf00140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa9031a8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa8fc0a8e00 0x7fa8fc0aed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa9031a8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8fc0af220 con 0x7fa8fc0b1410 2022-07-21T01:41:38.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa9031a8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8fc1404a0 con 0x7fa8fc140e80 2022-07-21T01:41:38.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa9031a8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8fc140740 con 0x7fa8fc0a8e00 2022-07-21T01:41:38.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05044700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcf000b0560 0x7fcf00140350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50038/0 (socket says 172.21.15.3:50038) 2022-07-21T01:41:38.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05044700 1 -- 172.21.15.3:0/2670473868 learned_addr learned my addr 172.21.15.3:0/2670473868 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:38.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa900f44700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa8fc0b1410 0x7fa8fc0b1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:60436/0 (socket says 172.21.15.3:60436) 2022-07-21T01:41:38.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa900f44700 1 -- 172.21.15.3:0/1842289766 learned_addr learned my addr 172.21.15.3:0/1842289766 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:38.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcefffff700 1 --2- 172.21.15.3:0/2670473868 >> v2:172.21.15.143:3300/0 conn(0x7fcf000b0d10 0x7fcf00146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa8fbfff700 1 --2- 172.21.15.3:0/1842289766 >> v2:172.21.15.3:3300/0 conn(0x7fa8fc140e80 0x7fa8fc0ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa901745700 1 --2- 172.21.15.3:0/1842289766 >> v2:172.21.15.143:3300/0 conn(0x7fa8fc0a8e00 0x7fa8fc0aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05044700 1 -- 172.21.15.3:0/2670473868 >> v2:172.21.15.143:3300/0 conn(0x7fcf000b0d10 msgr2=0x7fcf00146df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:38.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05044700 1 --2- 172.21.15.3:0/2670473868 >> v2:172.21.15.143:3300/0 conn(0x7fcf000b0d10 0x7fcf00146df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05044700 1 -- 172.21.15.3:0/2670473868 >> v1:172.21.15.143:6789/0 conn(0x7fcf000a9320 legacy=0x7fcf001466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05044700 1 -- 172.21.15.3:0/2670473868 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf000b17a0 con 0x7fcf000b0560 2022-07-21T01:41:38.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05044700 1 --2- 172.21.15.3:0/2670473868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 0x7fcf00140350 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7fcef80032f0 tx=0x7fcef8009290).ready entity=mon.0 client_cookie=51897fc9ee24a8a0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:38.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa8fbfff700 1 -- 172.21.15.3:0/1842289766 >> v2:172.21.15.143:3300/0 conn(0x7fa8fc0a8e00 msgr2=0x7fa8fc0aed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:38.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa8fbfff700 1 --2- 172.21.15.3:0/1842289766 >> v2:172.21.15.143:3300/0 conn(0x7fa8fc0a8e00 0x7fa8fc0aed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa8fbfff700 1 -- 172.21.15.3:0/1842289766 >> v1:172.21.15.143:6789/0 conn(0x7fa8fc0b1410 legacy=0x7fa8fc0b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa8fbfff700 1 -- 172.21.15.3:0/1842289766 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc1409e0 con 0x7fa8fc140e80 2022-07-21T01:41:38.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fceff7fe700 1 -- 172.21.15.3:0/2670473868 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcef800b870 con 0x7fcf000b0560 2022-07-21T01:41:38.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fceff7fe700 1 -- 172.21.15.3:0/2670473868 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef800b9d0 con 0x7fcf000b0560 2022-07-21T01:41:38.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fceff7fe700 1 -- 172.21.15.3:0/2670473868 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcef800bb40 con 0x7fcf000b0560 2022-07-21T01:41:38.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3ccbd27700 1 Processor -- start 2022-07-21T01:41:38.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3ccbd27700 1 -- start start 2022-07-21T01:41:38.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 -- 172.21.15.3:0/2670473868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 msgr2=0x7fcf00140350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 --2- 172.21.15.3:0/2670473868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 0x7fcf00140350 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7fcef80032f0 tx=0x7fcef8009290).stop 2022-07-21T01:41:38.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fa8fbfff700 1 --2- 172.21.15.3:0/1842289766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 0x7fa8fc0ae840 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7fa8e8002f70 tx=0x7fa8e8006df0).ready entity=mon.0 client_cookie=d83f4488e25f58e8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:38.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 -- 172.21.15.3:0/2670473868 shutdown_connections 2022-07-21T01:41:38.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 --2- 172.21.15.3:0/2670473868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 0x7fcf00140350 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 --2- 172.21.15.3:0/2670473868 >> v2:172.21.15.143:3300/0 conn(0x7fcf000b0d10 0x7fcf00146df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 -- 172.21.15.3:0/2670473868 >> 172.21.15.3:0/2670473868 conn(0x7fcf000612d0 msgr2=0x7fcf00062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:38.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 -- 172.21.15.3:0/2670473868 shutdown_connections 2022-07-21T01:41:38.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7fcf05845700 1 -- 172.21.15.3:0/2670473868 wait complete. 2022-07-21T01:41:38.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3ccbd27700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3cc40b1410 0x7f3cc40b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3ccbd27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc4140e80 0x7f3cc40b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3ccbd27700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc41404a0 con 0x7f3cc40a8e00 2022-07-21T01:41:38.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3ccbd27700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc4140740 con 0x7f3cc40b1410 2022-07-21T01:41:38.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3ccbd27700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc41409e0 con 0x7f3cc4140e80 2022-07-21T01:41:38.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3cca2c4700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3cc40a8e00 0x7f3cc40ae840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58330/0 (socket says 172.21.15.3:58330) 2022-07-21T01:41:38.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3cca2c4700 1 -- 172.21.15.3:0/2429305817 learned_addr learned my addr 172.21.15.3:0/2429305817 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:38.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3cc9ac3700 1 --2- 172.21.15.3:0/2429305817 >> v2:172.21.15.143:3300/0 conn(0x7f3cc40b1410 0x7f3cc40b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3cc92c2700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc4140e80 0x7f3cc40b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330092473 0 0) 0x7f3cc41404a0 con 0x7f3cc40a8e00 2022-07-21T01:41:38.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.672+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cac003260 con 0x7f3cc40a8e00 2022-07-21T01:41:38.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 532863392 0 0) 0x7f3cac003260 con 0x7f3cc40a8e00 2022-07-21T01:41:38.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc4140e80 msgr2=0x7f3cc40b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:38.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa8fb7fe700 1 -- 172.21.15.3:0/1842289766 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8e800d570 con 0x7fa8fc140e80 2022-07-21T01:41:38.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc4140e80 0x7f3cc40b22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 >> v2:172.21.15.143:3300/0 conn(0x7f3cc40b1410 msgr2=0x7f3cc40b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:38.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 --2- 172.21.15.3:0/2429305817 >> v2:172.21.15.143:3300/0 conn(0x7f3cc40b1410 0x7f3cc40b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa8fb7fe700 1 -- 172.21.15.3:0/1842289766 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e8000e90 con 0x7fa8fc140e80 2022-07-21T01:41:38.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa8fb7fe700 1 -- 172.21.15.3:0/1842289766 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8e800d8c0 con 0x7fa8fc140e80 2022-07-21T01:41:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 Processor -- start 2022-07-21T01:41:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 -- start start 2022-07-21T01:41:38.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 -- 172.21.15.3:0/1842289766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 msgr2=0x7fa8fc0ae840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc4140190 con 0x7f3cc40a8e00 2022-07-21T01:41:38.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f3cbc0019e0 con 0x7f3cc40a8e00 2022-07-21T01:41:38.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 --2- 172.21.15.3:0/1842289766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 0x7fa8fc0ae840 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7fa8e8002f70 tx=0x7fa8e8006df0).stop 2022-07-21T01:41:38.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 >> v1:172.21.15.3:6789/0 conn(0x7f3cc40a8e00 legacy=0x7f3cc40ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc4140e80 0x7f3cac003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cac004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cac0049f0 0x7f3cac004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc4140740 con 0x7f3cac003c80 2022-07-21T01:41:38.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc41409e0 con 0x7f3cac0049f0 2022-07-21T01:41:38.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac00b750 con 0x7f3cc4140e80 2022-07-21T01:41:38.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc92c2700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cac004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc9ac3700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc4140e80 0x7f3cac003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cca2c4700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cac0049f0 0x7f3cac004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc92c2700 1 -- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc4140e80 msgr2=0x7f3cac003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:38.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc92c2700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc4140e80 0x7f3cac003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc92c2700 1 -- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cac0049f0 msgr2=0x7f3cac004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:38.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc92c2700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cac0049f0 0x7f3cac004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc92c2700 1 -- 172.21.15.3:0/2429305817 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac003260 con 0x7f3cac003c80 2022-07-21T01:41:38.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cca2c4700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cac0049f0 0x7f3cac004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:38.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 -- 172.21.15.3:0/1842289766 shutdown_connections 2022-07-21T01:41:38.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 --2- 172.21.15.3:0/1842289766 >> v2:172.21.15.143:3300/0 conn(0x7fa8fc0a8e00 0x7fa8fc0aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 --2- 172.21.15.3:0/1842289766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 0x7fa8fc0ae840 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 -- 172.21.15.3:0/1842289766 >> 172.21.15.3:0/1842289766 conn(0x7fa8fc05f840 msgr2=0x7fa8fc05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:38.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc92c2700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cac004500 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f3cac0054d0 tx=0x7f3cb4006df0).ready entity=mon.0 client_cookie=c4264ea4cd5442bd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:38.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 -- 172.21.15.3:0/1842289766 shutdown_connections 2022-07-21T01:41:38.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 -- 172.21.15.3:0/1842289766 wait complete. 2022-07-21T01:41:38.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf000a9320 0x7fcf00145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 0x7fcf00156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf000b0d10 0x7fcf000a99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf00146440 con 0x7fcf000b0560 2022-07-21T01:41:38.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf00155b90 con 0x7fcf000b0d10 2022-07-21T01:41:38.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000ac780 con 0x7fcf000a9320 2022-07-21T01:41:38.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 0x7fcf00156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 0x7fcf00156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50062/0 (socket says 172.21.15.3:50062) 2022-07-21T01:41:38.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefffff700 1 -- 172.21.15.3:0/2955566220 learned_addr learned my addr 172.21.15.3:0/2955566220 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:38.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05044700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf000b0d10 0x7fcf000a99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf04843700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf000a9320 0x7fcf00145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefffff700 1 -- 172.21.15.3:0/2955566220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf000a9320 msgr2=0x7fcf00145c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefffff700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf000a9320 0x7fcf00145c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefffff700 1 -- 172.21.15.3:0/2955566220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf000b0d10 msgr2=0x7fcf000a99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefffff700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf000b0d10 0x7fcf000a99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefffff700 1 -- 172.21.15.3:0/2955566220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef8009530 con 0x7fcf000b0560 2022-07-21T01:41:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefffff700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 0x7fcf00156290 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fceec0030b0 tx=0x7fceec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cb400e830 con 0x7f3cac003c80 2022-07-21T01:41:38.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb400be90 con 0x7f3cac003c80 2022-07-21T01:41:38.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cc8ac1700 1 -- 172.21.15.3:0/2429305817 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cb400eb80 con 0x7f3cac003c80 2022-07-21T01:41:38.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 Processor -- start 2022-07-21T01:41:38.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3ccbd27700 1 -- 172.21.15.3:0/2429305817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 msgr2=0x7f3cac004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3ccbd27700 1 --2- 172.21.15.3:0/2429305817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cac004500 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f3cac0054d0 tx=0x7f3cb4006df0).stop 2022-07-21T01:41:38.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3cca2c4700 1 -- 172.21.15.3:0/2429305817 reap_dead start 2022-07-21T01:41:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefdffb700 1 -- 172.21.15.3:0/2955566220 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceec011780 con 0x7fcf000b0560 2022-07-21T01:41:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3ccbd27700 1 -- 172.21.15.3:0/2429305817 shutdown_connections 2022-07-21T01:41:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3ccbd27700 1 -- 172.21.15.3:0/2429305817 >> 172.21.15.3:0/2429305817 conn(0x7f3cc405f840 msgr2=0x7f3cc405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fa9031a8700 1 -- start start 2022-07-21T01:41:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefdffb700 1 -- 172.21.15.3:0/2955566220 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec0118e0 con 0x7fcf000b0560 2022-07-21T01:41:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf00155df0 con 0x7fcf000b0560 2022-07-21T01:41:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcefdffb700 1 -- 172.21.15.3:0/2955566220 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceec011c40 con 0x7fcf000b0560 2022-07-21T01:41:38.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf00146180 con 0x7fcf000b0560 2022-07-21T01:41:38.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3ccbd27700 1 -- 172.21.15.3:0/2429305817 shutdown_connections 2022-07-21T01:41:38.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.676+0000 7f3ccbd27700 1 -- 172.21.15.3:0/2429305817 wait complete. 2022-07-21T01:41:38.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa9031a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8fc0a8e00 0x7fa8fc130600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa9031a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8fc0b1410 0x7fa8fc0abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa9031a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 0x7fa8fc0aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa9031a8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc0b3f50 con 0x7fa8fc140e80 2022-07-21T01:41:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa9031a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc0b41f0 con 0x7fa8fc0a8e00 2022-07-21T01:41:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa9031a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc0ac2f0 con 0x7fa8fc0b1410 2022-07-21T01:41:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3ccbd27700 1 Processor -- start 2022-07-21T01:41:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa901745700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 0x7fa8fc0aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa901745700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 0x7fa8fc0aaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50068/0 (socket says 172.21.15.3:50068) 2022-07-21T01:41:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa901745700 1 -- 172.21.15.3:0/610083080 learned_addr learned my addr 172.21.15.3:0/610083080 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa900f44700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8fc0a8e00 0x7fa8fc130600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3ccbd27700 1 -- start start 2022-07-21T01:41:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa8fbfff700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8fc0b1410 0x7fa8fc0abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf00155f80 con 0x7fcf000b0560 2022-07-21T01:41:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa901745700 1 -- 172.21.15.3:0/610083080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8fc0b1410 msgr2=0x7fa8fc0abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fcefdffb700 1 -- 172.21.15.3:0/2955566220 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fceec012420 con 0x7fcf000b0560 2022-07-21T01:41:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa901745700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8fc0b1410 0x7fa8fc0abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa901745700 1 -- 172.21.15.3:0/610083080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8fc0a8e00 msgr2=0x7fa8fc130600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa901745700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8fc0a8e00 0x7fa8fc130600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa901745700 1 -- 172.21.15.3:0/610083080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e800b180 con 0x7fa8fc140e80 2022-07-21T01:41:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fa901745700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 0x7fa8fc0aaf00 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7fa8f4006e00 tx=0x7fa8f40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fcefdffb700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcee0053a70 0x7fcee0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fcefdffb700 1 -- 172.21.15.3:0/2955566220 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fceec06bdf0 con 0x7fcf000b0560 2022-07-21T01:41:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fcf04843700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcee0053a70 0x7fcee0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3ccbd27700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cc40aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3ccbd27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc40ae2d0 0x7f3cc4135920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3ccbd27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cc4135e10 0x7f3cc4143dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3ccbd27700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc40ad910 con 0x7f3cac003c80 2022-07-21T01:41:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3ccbd27700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc40ac480 con 0x7f3cc4135e10 2022-07-21T01:41:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3ccbd27700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc4134cb0 con 0x7f3cc40ae2d0 2022-07-21T01:41:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc9ac3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cc40aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc9ac3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cc40aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50072/0 (socket says 172.21.15.3:50072) 2022-07-21T01:41:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc9ac3700 1 -- 172.21.15.3:0/114552756 learned_addr learned my addr 172.21.15.3:0/114552756 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cca2c4700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cc4135e10 0x7f3cc4143dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc92c2700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc40ae2d0 0x7f3cc4135920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7fcf04843700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcee0053a70 0x7fcee0055f30 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fcef400d000 tx=0x7fcef4007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:38.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc9ac3700 1 -- 172.21.15.3:0/114552756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc40ae2d0 msgr2=0x7f3cc4135920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc9ac3700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc40ae2d0 0x7f3cc4135920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc9ac3700 1 -- 172.21.15.3:0/114552756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cc4135e10 msgr2=0x7f3cc4143dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:38.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc9ac3700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cc4135e10 0x7f3cc4143dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:38.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc9ac3700 1 -- 172.21.15.3:0/114552756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc40ac1e0 con 0x7f3cac003c80 2022-07-21T01:41:38.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cca2c4700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cc4135e10 0x7f3cc4143dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:38.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.680+0000 7f3cc9ac3700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cc40aefc0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f3cc4142120 tx=0x7f3cc0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:38.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7f3cb2ffd700 1 -- 172.21.15.3:0/114552756 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc000b5f0 con 0x7f3cac003c80 2022-07-21T01:41:38.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7fa8f9ffb700 1 -- 172.21.15.3:0/610083080 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8f400c640 con 0x7fa8fc140e80 2022-07-21T01:41:38.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7fa8f9ffb700 1 -- 172.21.15.3:0/610083080 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f400c7a0 con 0x7fa8fc140e80 2022-07-21T01:41:38.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8fc0ac6d0 con 0x7fa8fc140e80 2022-07-21T01:41:38.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7fa8f9ffb700 1 -- 172.21.15.3:0/610083080 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8f400cb00 con 0x7fa8fc140e80 2022-07-21T01:41:38.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc40adb90 con 0x7f3cac003c80 2022-07-21T01:41:38.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc4142be0 con 0x7f3cac003c80 2022-07-21T01:41:38.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7f3cb2ffd700 1 -- 172.21.15.3:0/114552756 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc000f940 con 0x7f3cac003c80 2022-07-21T01:41:38.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7fcefdffb700 1 -- 172.21.15.3:0/2955566220 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fceec042b50 con 0x7fcf000b0560 2022-07-21T01:41:38.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7f3cb2ffd700 1 -- 172.21.15.3:0/114552756 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc000fca0 con 0x7f3cac003c80 2022-07-21T01:41:38.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8fc0ab590 con 0x7fa8fc140e80 2022-07-21T01:41:38.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.684+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc40aa520 con 0x7f3cac003c80 2022-07-21T01:41:38.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7f3cb2ffd700 1 -- 172.21.15.3:0/114552756 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3cc0014440 con 0x7f3cac003c80 2022-07-21T01:41:38.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7f3cb2ffd700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca4053b00 0x7f3ca4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7f3cb2ffd700 1 -- 172.21.15.3:0/114552756 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3cc006d2e0 con 0x7f3cac003c80 2022-07-21T01:41:38.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7f3cc92c2700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca4053b00 0x7f3ca4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7f3cc92c2700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca4053b00 0x7f3ca4055fc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f3cb401c000 tx=0x7f3cb4019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:38.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7fa8f9ffb700 1 -- 172.21.15.3:0/610083080 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa8f400cc60 con 0x7fa8fc140e80 2022-07-21T01:41:38.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8fc0aa8c0 con 0x7fa8fc140e80 2022-07-21T01:41:38.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7fa8f9ffb700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8dc053b00 0x7fa8dc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:38.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7fa900f44700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8dc053b00 0x7fa8dc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:38.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.688+0000 7fa900f44700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8dc053b00 0x7fa8dc055fc0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7fa8f000d000 tx=0x7fa8f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:38.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.692+0000 7f3cb2ffd700 1 -- 172.21.15.3:0/114552756 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3cc0044040 con 0x7f3cac003c80 2022-07-21T01:41:38.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.692+0000 7fa8f9ffb700 1 -- 172.21.15.3:0/610083080 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa8f40465f0 con 0x7fa8fc140e80 2022-07-21T01:41:38.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.696+0000 7fa8f9ffb700 1 -- 172.21.15.3:0/610083080 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa8fc0aa8c0 con 0x7fa8fc140e80 2022-07-21T01:41:38.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.852+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fcf000574e0 con 0x7fcf000b0560 2022-07-21T01:41:38.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.856+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f3cc40574e0 con 0x7f3cac003c80 2022-07-21T01:41:38.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:38.864+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7fa8fc140190 con 0x7fa8fc140e80 2022-07-21T01:41:39.746 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:41:39.740+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:41:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.756+0000 7fcefdffb700 1 -- 172.21.15.3:0/2955566220 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7573 v61) v1 ==== 95+0+0 (secure 0 0 0) 0x7fceec046170 con 0x7fcf000b0560 2022-07-21T01:41:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.756+0000 7f3cb2ffd700 1 -- 172.21.15.3:0/114552756 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7602 v61) v1 ==== 95+0+0 (secure 0 0 0) 0x7f3cc0018090 con 0x7f3cac003c80 2022-07-21T01:41:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.756+0000 7fa8f9ffb700 1 -- 172.21.15.3:0/610083080 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7611 v61) v1 ==== 95+0+0 (secure 0 0 0) 0x7fa8f4042990 con 0x7fa8fc140e80 2022-07-21T01:41:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8dc053b00 msgr2=0x7fa8dc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:39.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8dc053b00 0x7fa8dc055fc0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7fa8f000d000 tx=0x7fa8f0009040).stop 2022-07-21T01:41:39.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca4053b00 msgr2=0x7f3ca4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:39.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca4053b00 0x7f3ca4055fc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f3cb401c000 tx=0x7f3cb4019000).stop 2022-07-21T01:41:39.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 msgr2=0x7fa8fc0aaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:39.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 0x7fa8fc0aaf00 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7fa8f4006e00 tx=0x7fa8f40091d0).stop 2022-07-21T01:41:39.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 msgr2=0x7f3cc40aefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:39.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cc40aefc0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f3cc4142120 tx=0x7f3cc0006df0).stop 2022-07-21T01:41:39.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 shutdown_connections 2022-07-21T01:41:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8fc140e80 0x7fa8fc0aaf00 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8fc0b1410 0x7fa8fc0abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa8dc053b00 0x7fa8dc055fc0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 --2- 172.21.15.3:0/610083080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8fc0a8e00 0x7fa8fc130600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 >> 172.21.15.3:0/610083080 conn(0x7fa8fc05f840 msgr2=0x7fa8fc0aeae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 shutdown_connections 2022-07-21T01:41:39.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ca4053b00 0x7f3ca4055fc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cc4135e10 0x7f3cc4143dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cc40ae2d0 0x7f3cc4135920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 --2- 172.21.15.3:0/114552756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cac003c80 0x7f3cc40aefc0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 >> 172.21.15.3:0/114552756 conn(0x7f3cc405f840 msgr2=0x7f3cc40b4670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcee0053a70 msgr2=0x7fcee0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 shutdown_connections 2022-07-21T01:41:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fa9031a8700 1 -- 172.21.15.3:0/610083080 wait complete. 2022-07-21T01:41:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcee0053a70 0x7fcee0055f30 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fcef400d000 tx=0x7fcef4007b90).stop 2022-07-21T01:41:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 msgr2=0x7fcf00156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:39.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 0x7fcf00156290 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fceec0030b0 tx=0x7fceec008040).stop 2022-07-21T01:41:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 shutdown_connections 2022-07-21T01:41:39.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7f3ccbd27700 1 -- 172.21.15.3:0/114552756 wait complete. 2022-07-21T01:41:39.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 shutdown_connections 2022-07-21T01:41:39.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcee0053a70 0x7fcee0055f30 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf000b0d10 0x7fcf000a99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf000b0560 0x7fcf00156290 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 --2- 172.21.15.3:0/2955566220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf000a9320 0x7fcf00145c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 >> 172.21.15.3:0/2955566220 conn(0x7fcf000612d0 msgr2=0x7fcf000b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:39.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 shutdown_connections 2022-07-21T01:41:39.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.760+0000 7fcf05845700 1 -- 172.21.15.3:0/2955566220 wait complete. 2022-07-21T01:41:39.780 INFO:teuthology.orchestra.run.smithi003.stderr:failed mds gid 7611 2022-07-21T01:41:39.780 INFO:teuthology.orchestra.run.smithi003.stderr:failed mds gid 7602 2022-07-21T01:41:39.781 INFO:teuthology.orchestra.run.smithi003.stderr:failed mds gid 7573 2022-07-21T01:41:39.784 DEBUG:teuthology.parallel:result is '' 2022-07-21T01:41:39.786 DEBUG:teuthology.parallel:result is '' 2022-07-21T01:41:39.801 DEBUG:teuthology.parallel:result is '' 2022-07-21T01:41:39.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:41:39.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 Processor -- start 2022-07-21T01:41:39.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 -- start start 2022-07-21T01:41:39.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad400b2710 0x7fad4014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:39.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad400a8c50 0x7fad4014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:39.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad400a7250 con 0x7fad400b01a0 2022-07-21T01:41:39.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad400a7480 con 0x7fad400a8c50 2022-07-21T01:41:39.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad400a7720 con 0x7fad400b2710 2022-07-21T01:41:39.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad464f7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad400a8c50 0x7fad4014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:39.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad464f7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fad400a8c50 0x7fad4014b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50086/0 (socket says 172.21.15.3:50086) 2022-07-21T01:41:39.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad464f7700 1 -- 172.21.15.3:0/1570747010 learned_addr learned my addr 172.21.15.3:0/1570747010 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:39.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad45cf6700 1 --2- 172.21.15.3:0/1570747010 >> v2:172.21.15.143:3300/0 conn(0x7fad400b2710 0x7fad4014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:39.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad464f7700 1 -- 172.21.15.3:0/1570747010 >> v2:172.21.15.143:3300/0 conn(0x7fad400b2710 msgr2=0x7fad4014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:39.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad464f7700 1 --2- 172.21.15.3:0/1570747010 >> v2:172.21.15.143:3300/0 conn(0x7fad400b2710 0x7fad4014afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad464f7700 1 -- 172.21.15.3:0/1570747010 >> v1:172.21.15.143:6789/0 conn(0x7fad400b01a0 legacy=0x7fad400a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:39.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad464f7700 1 -- 172.21.15.3:0/1570747010 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad400a79c0 con 0x7fad400a8c50 2022-07-21T01:41:39.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad464f7700 1 --2- 172.21.15.3:0/1570747010 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400a8c50 0x7fad4014b4c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7fad4006b950 tx=0x7fad38009290).ready entity=mon.0 client_cookie=bf8ca3fc56946e29 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:39.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad44cf4700 1 -- 172.21.15.3:0/1570747010 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad3800b830 con 0x7fad400a8c50 2022-07-21T01:41:39.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad44cf4700 1 -- 172.21.15.3:0/1570747010 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3800b990 con 0x7fad400a8c50 2022-07-21T01:41:39.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad44cf4700 1 -- 172.21.15.3:0/1570747010 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad3800bb40 con 0x7fad400a8c50 2022-07-21T01:41:39.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 -- 172.21.15.3:0/1570747010 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400a8c50 msgr2=0x7fad4014b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:39.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 --2- 172.21.15.3:0/1570747010 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400a8c50 0x7fad4014b4c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7fad4006b950 tx=0x7fad38009290).stop 2022-07-21T01:41:39.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 -- 172.21.15.3:0/1570747010 shutdown_connections 2022-07-21T01:41:39.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 --2- 172.21.15.3:0/1570747010 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400a8c50 0x7fad4014b4c0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 --2- 172.21.15.3:0/1570747010 >> v2:172.21.15.143:3300/0 conn(0x7fad400b2710 0x7fad4014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 -- 172.21.15.3:0/1570747010 >> 172.21.15.3:0/1570747010 conn(0x7fad40060310 msgr2=0x7fad400b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:39.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 -- 172.21.15.3:0/1570747010 shutdown_connections 2022-07-21T01:41:39.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.932+0000 7fad46cf8700 1 -- 172.21.15.3:0/1570747010 wait complete. 2022-07-21T01:41:39.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 Processor -- start 2022-07-21T01:41:39.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 -- start start 2022-07-21T01:41:39.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad400a8c50 0x7fad400ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:39.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad400b01a0 0x7fad40159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:39.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400b2710 0x7fad400669d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:39.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad400ac430 con 0x7fad400b2710 2022-07-21T01:41:39.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4015a240 con 0x7fad400b01a0 2022-07-21T01:41:39.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad400ac6a0 con 0x7fad400a8c50 2022-07-21T01:41:39.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad464f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400b2710 0x7fad400669d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:39.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad464f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400b2710 0x7fad400669d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50092/0 (socket says 172.21.15.3:50092) 2022-07-21T01:41:39.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad464f7700 1 -- 172.21.15.3:0/2242623942 learned_addr learned my addr 172.21.15.3:0/2242623942 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:39.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad454f5700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad400b01a0 0x7fad40159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:39.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad45cf6700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad400a8c50 0x7fad400ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:39.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad464f7700 1 -- 172.21.15.3:0/2242623942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad400a8c50 msgr2=0x7fad400ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:39.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad464f7700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad400a8c50 0x7fad400ae070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad464f7700 1 -- 172.21.15.3:0/2242623942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad400b01a0 msgr2=0x7fad40159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:39.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad464f7700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad400b01a0 0x7fad40159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:39.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad464f7700 1 -- 172.21.15.3:0/2242623942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38009530 con 0x7fad400b2710 2022-07-21T01:41:39.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad464f7700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400b2710 0x7fad400669d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7fad4006b950 tx=0x7fad38016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:39.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad2effd700 1 -- 172.21.15.3:0/2242623942 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad380032f0 con 0x7fad400b2710 2022-07-21T01:41:39.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad2effd700 1 -- 172.21.15.3:0/2242623942 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad38021440 con 0x7fad400b2710 2022-07-21T01:41:39.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad2effd700 1 -- 172.21.15.3:0/2242623942 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad380217a0 con 0x7fad400b2710 2022-07-21T01:41:39.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad400aaeb0 con 0x7fad400b2710 2022-07-21T01:41:39.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.936+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad40066ec0 con 0x7fad400b2710 2022-07-21T01:41:39.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.940+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad400574e0 con 0x7fad400b2710 2022-07-21T01:41:39.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.940+0000 7fad2effd700 1 -- 172.21.15.3:0/2242623942 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fad3800f070 con 0x7fad400b2710 2022-07-21T01:41:39.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.940+0000 7fad2effd700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad20053aa0 0x7fad20055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:39.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.940+0000 7fad2effd700 1 -- 172.21.15.3:0/2242623942 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fad3806cdf0 con 0x7fad400b2710 2022-07-21T01:41:39.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.940+0000 7fad45cf6700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad20053aa0 0x7fad20055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:39.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.940+0000 7fad45cf6700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad20053aa0 0x7fad20055f60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fad3c00c000 tx=0x7fad3c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:39.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:39.948+0000 7fad2effd700 1 -- 172.21.15.3:0/2242623942 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad38043ba0 con 0x7fad400b2710 2022-07-21T01:41:40.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.112+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fad400ab4d0 con 0x7fad400b2710 2022-07-21T01:41:40.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.884+0000 7fad2effd700 1 -- 172.21.15.3:0/2242623942 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7fad38015070 con 0x7fad400b2710 2022-07-21T01:41:40.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad20053aa0 msgr2=0x7fad20055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:40.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad20053aa0 0x7fad20055f60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fad3c00c000 tx=0x7fad3c008040).stop 2022-07-21T01:41:40.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400b2710 msgr2=0x7fad400669d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400b2710 0x7fad400669d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7fad4006b950 tx=0x7fad38016000).stop 2022-07-21T01:41:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 shutdown_connections 2022-07-21T01:41:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad400b2710 0x7fad400669d0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad20053aa0 0x7fad20055f60 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:40.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad400b01a0 0x7fad40159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:40.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 --2- 172.21.15.3:0/2242623942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad400a8c50 0x7fad400ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:40.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 >> 172.21.15.3:0/2242623942 conn(0x7fad40060310 msgr2=0x7fad400673e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:40.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 shutdown_connections 2022-07-21T01:41:40.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:40.888+0000 7fad46cf8700 1 -- 172.21.15.3:0/2242623942 wait complete. 2022-07-21T01:41:40.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:41:41.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.024+0000 7fbf421ad700 1 Processor -- start 2022-07-21T01:41:41.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.024+0000 7fbf421ad700 1 -- start start 2022-07-21T01:41:41.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbf3c0b6710 0x7fbf3c0b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:41.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6af0 0x7fbf3c12bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:41.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3c144240 con 0x7fbf3c0b4550 2022-07-21T01:41:41.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c143d00 con 0x7fbf3c0b6710 2022-07-21T01:41:41.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c143fa0 con 0x7fbf3c0b6af0 2022-07-21T01:41:41.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fbf3c0b6710 0x7fbf3c0b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:41.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3affd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbf3c0b4550 0x7fbf3c061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58380/0 (socket says 172.21.15.3:58380) 2022-07-21T01:41:41.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3affd700 1 -- 172.21.15.3:0/2510451695 learned_addr learned my addr 172.21.15.3:0/2510451695 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:41.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3bfff700 1 --2- 172.21.15.3:0/2510451695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6af0 0x7fbf3c12bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:41.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3b7fe700 1 -- 172.21.15.3:0/2510451695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6af0 msgr2=0x7fbf3c12bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:41.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3b7fe700 1 --2- 172.21.15.3:0/2510451695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6af0 0x7fbf3c12bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:41.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3b7fe700 1 -- 172.21.15.3:0/2510451695 >> v1:172.21.15.3:6789/0 conn(0x7fbf3c0b4550 legacy=0x7fbf3c061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:41.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3a7fc700 1 -- 172.21.15.3:0/2510451695 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191288096 0 0) 0x7fbf3c144240 con 0x7fbf3c0b4550 2022-07-21T01:41:41.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3b7fe700 1 -- 172.21.15.3:0/2510451695 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c061780 con 0x7fbf3c0b6710 2022-07-21T01:41:41.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3bfff700 1 --2- 172.21.15.3:0/2510451695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6af0 0x7fbf3c12bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:41.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3b7fe700 1 --2- 172.21.15.3:0/2510451695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b6710 0x7fbf3c0b5d80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7fbf3c1432b0 tx=0x7fbf2c006df0).ready entity=mon.0 client_cookie=89a5247c4b6a5893 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:41.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3a7fc700 1 -- 172.21.15.3:0/2510451695 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf2c00d570 con 0x7fbf3c0b6710 2022-07-21T01:41:41.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3a7fc700 1 -- 172.21.15.3:0/2510451695 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c000e90 con 0x7fbf3c0b6710 2022-07-21T01:41:41.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf3a7fc700 1 -- 172.21.15.3:0/2510451695 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf2c00d8c0 con 0x7fbf3c0b6710 2022-07-21T01:41:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 -- 172.21.15.3:0/2510451695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b6710 msgr2=0x7fbf3c0b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:41.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 --2- 172.21.15.3:0/2510451695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b6710 0x7fbf3c0b5d80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7fbf3c1432b0 tx=0x7fbf2c006df0).stop 2022-07-21T01:41:41.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 -- 172.21.15.3:0/2510451695 shutdown_connections 2022-07-21T01:41:41.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 --2- 172.21.15.3:0/2510451695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6af0 0x7fbf3c12bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:41.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 --2- 172.21.15.3:0/2510451695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b6710 0x7fbf3c0b5d80 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:41.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 -- 172.21.15.3:0/2510451695 >> 172.21.15.3:0/2510451695 conn(0x7fbf3c05c620 msgr2=0x7fbf3c05ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:41.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 -- 172.21.15.3:0/2510451695 shutdown_connections 2022-07-21T01:41:41.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 -- 172.21.15.3:0/2510451695 wait complete. 2022-07-21T01:41:41.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 Processor -- start 2022-07-21T01:41:41.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.028+0000 7fbf421ad700 1 -- start start 2022-07-21T01:41:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf421ad700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b4550 0x7fbf3c127180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf421ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6710 0x7fbf3c147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf421ad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf3c0b6af0 0x7fbf3c123990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf421ad700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0b1e50 con 0x7fbf3c0b4550 2022-07-21T01:41:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf421ad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0b2080 con 0x7fbf3c0b6af0 2022-07-21T01:41:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf421ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0b2320 con 0x7fbf3c0b6710 2022-07-21T01:41:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b4550 0x7fbf3c127180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b4550 0x7fbf3c127180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50104/0 (socket says 172.21.15.3:50104) 2022-07-21T01:41:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3b7fe700 1 -- 172.21.15.3:0/401512594 learned_addr learned my addr 172.21.15.3:0/401512594 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3affd700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6710 0x7fbf3c147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3bfff700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf3c0b6af0 0x7fbf3c123990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3b7fe700 1 -- 172.21.15.3:0/401512594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6710 msgr2=0x7fbf3c147580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3b7fe700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6710 0x7fbf3c147580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:41.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3b7fe700 1 -- 172.21.15.3:0/401512594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf3c0b6af0 msgr2=0x7fbf3c123990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:41.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3b7fe700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf3c0b6af0 0x7fbf3c123990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:41.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3b7fe700 1 -- 172.21.15.3:0/401512594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c00b180 con 0x7fbf3c0b4550 2022-07-21T01:41:41.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3bfff700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf3c0b6af0 0x7fbf3c123990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:41.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf3b7fe700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b4550 0x7fbf3c127180 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7fbf3c1432b0 tx=0x7fbf2c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:41.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf38ff9700 1 -- 172.21.15.3:0/401512594 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf2c011790 con 0x7fbf3c0b4550 2022-07-21T01:41:41.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf38ff9700 1 -- 172.21.15.3:0/401512594 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c0118f0 con 0x7fbf3c0b4550 2022-07-21T01:41:41.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf38ff9700 1 -- 172.21.15.3:0/401512594 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf2c011c50 con 0x7fbf3c0b4550 2022-07-21T01:41:41.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c0b25c0 con 0x7fbf3c0b4550 2022-07-21T01:41:41.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.032+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c127670 con 0x7fbf3c0b4550 2022-07-21T01:41:41.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.036+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c061a20 con 0x7fbf3c0b4550 2022-07-21T01:41:41.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.036+0000 7fbf38ff9700 1 -- 172.21.15.3:0/401512594 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbf2c01a040 con 0x7fbf3c0b4550 2022-07-21T01:41:41.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.036+0000 7fbf38ff9700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbf20053b30 0x7fbf20055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:41.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.036+0000 7fbf38ff9700 1 -- 172.21.15.3:0/401512594 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fbf2c06cfe0 con 0x7fbf3c0b4550 2022-07-21T01:41:41.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.036+0000 7fbf3affd700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbf20053b30 0x7fbf20055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:41.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.036+0000 7fbf3affd700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbf20053b30 0x7fbf20055ff0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fbf2400d000 tx=0x7fbf24007c00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:41.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.040+0000 7fbf38ff9700 1 -- 172.21.15.3:0/401512594 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbf2c043c40 con 0x7fbf3c0b4550 2022-07-21T01:41:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.168+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbf3c0574e0 con 0x7fbf3c0b4550 2022-07-21T01:41:41.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.888+0000 7fbf38ff9700 1 -- 172.21.15.3:0/401512594 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v125) v1 ==== 185+0+0 (secure 0 0 0) 0x7fbf2c0230e0 con 0x7fbf3c0b4550 2022-07-21T01:41:41.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbf20053b30 msgr2=0x7fbf20055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:41.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbf20053b30 0x7fbf20055ff0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fbf2400d000 tx=0x7fbf24007c00).stop 2022-07-21T01:41:41.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b4550 msgr2=0x7fbf3c127180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:41.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b4550 0x7fbf3c127180 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7fbf3c1432b0 tx=0x7fbf2c015000).stop 2022-07-21T01:41:41.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 shutdown_connections 2022-07-21T01:41:41.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbf20053b30 0x7fbf20055ff0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:41.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf3c0b6af0 0x7fbf3c123990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:41.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf3c0b6710 0x7fbf3c147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:41.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 --2- 172.21.15.3:0/401512594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf3c0b4550 0x7fbf3c127180 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:41.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 >> 172.21.15.3:0/401512594 conn(0x7fbf3c05c620 msgr2=0x7fbf3c1432f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:41.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 shutdown_connections 2022-07-21T01:41:41.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:41.892+0000 7fbf421ad700 1 -- 172.21.15.3:0/401512594 wait complete. 2022-07-21T01:41:41.901 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:41:41.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:41:42.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f3895af6700 1 Processor -- start 2022-07-21T01:41:42.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f3895af6700 1 -- start start 2022-07-21T01:41:42.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f3895af6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f389014ad10 0x7f389005f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f3895af6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f38900acb40 0x7f389014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f3895af6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900614f0 con 0x7f38900b00a0 2022-07-21T01:41:42.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f3895af6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389014b770 con 0x7f38900acb40 2022-07-21T01:41:42.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f3895af6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389014ba10 con 0x7f389014ad10 2022-07-21T01:41:42.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f38900acb40 0x7f389014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388effd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f38900b00a0 0x7f38900af350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58390/0 (socket says 172.21.15.3:58390) 2022-07-21T01:41:42.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388effd700 1 -- 172.21.15.3:0/1132300795 learned_addr learned my addr 172.21.15.3:0/1132300795 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:42.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388f7fe700 1 --2- 172.21.15.3:0/1132300795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f389014ad10 0x7f389005f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388e7fc700 1 -- 172.21.15.3:0/1132300795 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471025911 0 0) 0x7f38900614f0 con 0x7f38900b00a0 2022-07-21T01:41:42.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388e7fc700 1 -- 172.21.15.3:0/1132300795 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3870003260 con 0x7f38900b00a0 2022-07-21T01:41:42.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388ffff700 1 -- 172.21.15.3:0/1132300795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f389014ad10 msgr2=0x7f389005f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:42.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388ffff700 1 --2- 172.21.15.3:0/1132300795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f389014ad10 0x7f389005f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388ffff700 1 -- 172.21.15.3:0/1132300795 >> v1:172.21.15.3:6789/0 conn(0x7f38900b00a0 legacy=0x7f38900af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388ffff700 1 -- 172.21.15.3:0/1132300795 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38900aeeb0 con 0x7f38900acb40 2022-07-21T01:41:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388ffff700 1 --2- 172.21.15.3:0/1132300795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38900acb40 0x7f389014b0f0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f3888002f70 tx=0x7f3888006df0).ready entity=mon.0 client_cookie=9e3321d7e51bbe41 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:42.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388e7fc700 1 -- 172.21.15.3:0/1132300795 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f388800d570 con 0x7f38900acb40 2022-07-21T01:41:42.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388e7fc700 1 -- 172.21.15.3:0/1132300795 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3888000e90 con 0x7f38900acb40 2022-07-21T01:41:42.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.020+0000 7f388e7fc700 1 -- 172.21.15.3:0/1132300795 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f388800d8c0 con 0x7f38900acb40 2022-07-21T01:41:42.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.024+0000 7f3895af6700 1 -- 172.21.15.3:0/1132300795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38900acb40 msgr2=0x7f389014b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:42.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.024+0000 7f3895af6700 1 --2- 172.21.15.3:0/1132300795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38900acb40 0x7f389014b0f0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f3888002f70 tx=0x7f3888006df0).stop 2022-07-21T01:41:42.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.024+0000 7f3895af6700 1 -- 172.21.15.3:0/1132300795 shutdown_connections 2022-07-21T01:41:42.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.024+0000 7f3895af6700 1 --2- 172.21.15.3:0/1132300795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38900acb40 0x7f389014b0f0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.024+0000 7f3895af6700 1 --2- 172.21.15.3:0/1132300795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f389014ad10 0x7f389005f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.024+0000 7f3895af6700 1 -- 172.21.15.3:0/1132300795 >> 172.21.15.3:0/1132300795 conn(0x7f38900b5700 msgr2=0x7f38900b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:42.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.024+0000 7f3895af6700 1 -- 172.21.15.3:0/1132300795 shutdown_connections 2022-07-21T01:41:42.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.024+0000 7f3895af6700 1 -- 172.21.15.3:0/1132300795 wait complete. 2022-07-21T01:41:42.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.024+0000 7f3895af6700 1 Processor -- start 2022-07-21T01:41:42.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f3895af6700 1 -- start start 2022-07-21T01:41:42.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f3895af6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38900acb40 0x7f3890147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f3895af6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38900b00a0 0x7f38901454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f3895af6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389014ad10 0x7f38901459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f3895af6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3890156a10 con 0x7f389014ad10 2022-07-21T01:41:42.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f3895af6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3890156cb0 con 0x7f38900acb40 2022-07-21T01:41:42.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f3895af6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3890156f50 con 0x7f38900b00a0 2022-07-21T01:41:42.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389014ad10 0x7f38901459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389014ad10 0x7f38901459a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50120/0 (socket says 172.21.15.3:50120) 2022-07-21T01:41:42.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38900acb40 0x7f3890147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388ffff700 1 -- 172.21.15.3:0/1734988053 learned_addr learned my addr 172.21.15.3:0/1734988053 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:42.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388effd700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38900b00a0 0x7f38901454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388ffff700 1 -- 172.21.15.3:0/1734988053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38900b00a0 msgr2=0x7f38901454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:42.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388ffff700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38900b00a0 0x7f38901454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388ffff700 1 -- 172.21.15.3:0/1734988053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38900acb40 msgr2=0x7f3890147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:42.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388ffff700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38900acb40 0x7f3890147cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388ffff700 1 -- 172.21.15.3:0/1734988053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388800b180 con 0x7f389014ad10 2022-07-21T01:41:42.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388ffff700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389014ad10 0x7f38901459a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7f3888002f70 tx=0x7f3888014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:42.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388cff9700 1 -- 172.21.15.3:0/1734988053 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38880139c0 con 0x7f389014ad10 2022-07-21T01:41:42.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388cff9700 1 -- 172.21.15.3:0/1734988053 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3888013b20 con 0x7f389014ad10 2022-07-21T01:41:42.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38901571f0 con 0x7f389014ad10 2022-07-21T01:41:42.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f388cff9700 1 -- 172.21.15.3:0/1734988053 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3888013e80 con 0x7f389014ad10 2022-07-21T01:41:42.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.028+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38900b3370 con 0x7f389014ad10 2022-07-21T01:41:42.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.032+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3890157380 con 0x7f389014ad10 2022-07-21T01:41:42.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.032+0000 7f388cff9700 1 -- 172.21.15.3:0/1734988053 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3888025030 con 0x7f389014ad10 2022-07-21T01:41:42.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.032+0000 7f388cff9700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38740539e0 0x7f3874055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.032+0000 7f388cff9700 1 -- 172.21.15.3:0/1734988053 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f388806ce10 con 0x7f389014ad10 2022-07-21T01:41:42.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.032+0000 7f388f7fe700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38740539e0 0x7f3874055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.032+0000 7f388f7fe700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38740539e0 0x7f3874055ea0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f387800c000 tx=0x7f3878008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:42.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.040+0000 7f388cff9700 1 -- 172.21.15.3:0/1734988053 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3888070030 con 0x7f389014ad10 2022-07-21T01:41:42.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.172+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f38900afa60 con 0x7f389014ad10 2022-07-21T01:41:42.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.892+0000 7f388cff9700 1 -- 172.21.15.3:0/1734988053 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v126) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3888023070 con 0x7f389014ad10 2022-07-21T01:41:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38740539e0 msgr2=0x7f3874055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38740539e0 0x7f3874055ea0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f387800c000 tx=0x7f3878008040).stop 2022-07-21T01:41:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389014ad10 msgr2=0x7f38901459a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389014ad10 0x7f38901459a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7f3888002f70 tx=0x7f3888014000).stop 2022-07-21T01:41:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 shutdown_connections 2022-07-21T01:41:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389014ad10 0x7f38901459a0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38900b00a0 0x7f38901454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f38740539e0 0x7f3874055ea0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 --2- 172.21.15.3:0/1734988053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38900acb40 0x7f3890147cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 >> 172.21.15.3:0/1734988053 conn(0x7f38900b5700 msgr2=0x7f3890140500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 shutdown_connections 2022-07-21T01:41:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:42.896+0000 7f3895af6700 1 -- 172.21.15.3:0/1734988053 wait complete. 2022-07-21T01:41:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:41:42.915 DEBUG:teuthology.orchestra.run.smithi171:> sudo rados -p cephfs_metadata put key /etc/fstab 2022-07-21T01:41:42.971 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.956+0000 7f68df9e1b80 1 Processor -- start 2022-07-21T01:41:42.971 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.956+0000 7f68df9e1b80 1 -- start start 2022-07-21T01:41:42.972 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.956+0000 7f68df9e1b80 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfcfd470 0x5620dfd84620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.972 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.956+0000 7f68df9e1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfd93040 0x5620dfd84b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.972 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.956+0000 7f68df9e1b80 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd95920 0x5620dfd85000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.973 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.956+0000 7f68df9e1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620dfc796f0 con 0x5620dfd93040 2022-07-21T01:41:42.973 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.956+0000 7f68df9e1b80 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620dfd9f3d0 con 0x5620dfcfd470 2022-07-21T01:41:42.973 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.956+0000 7f68df9e1b80 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x5620dfd9f510 con 0x5620dfd95920 2022-07-21T01:41:42.974 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dde1e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfcfd470 0x5620dfd84620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.974 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dd61d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfd93040 0x5620dfd84b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.974 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68de61f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd95920 0x5620dfd85000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.975 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dde1e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfcfd470 0x5620dfd84620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.171:59068/0 (socket says 172.21.15.171:59068) 2022-07-21T01:41:42.975 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dde1e700 1 -- 172.21.15.171:0/1860366327 learned_addr learned my addr 172.21.15.171:0/1860366327 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-07-21T01:41:42.975 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dd61d700 1 -- 172.21.15.171:0/1860366327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd95920 msgr2=0x5620dfd85000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:42.976 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dd61d700 1 --2- 172.21.15.171:0/1860366327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd95920 0x5620dfd85000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.976 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dd61d700 1 -- 172.21.15.171:0/1860366327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfcfd470 msgr2=0x5620dfd84620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:42.976 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dd61d700 1 --2- 172.21.15.171:0/1860366327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfcfd470 0x5620dfd84620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.976 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dd61d700 1 -- 172.21.15.171:0/1860366327 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d000a0e0 con 0x5620dfd93040 2022-07-21T01:41:42.977 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dde1e700 1 --2- 172.21.15.171:0/1860366327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfcfd470 0x5620dfd84620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:42.977 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dd61d700 1 --2- 172.21.15.171:0/1860366327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfd93040 0x5620dfd84b10 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f68c800b000 tx=0x7f68c800c040).ready entity=mon.0 client_cookie=3068997c84c7c871 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:42.978 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68d77fe700 1 -- 172.21.15.171:0/1860366327 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c800f040 con 0x5620dfd93040 2022-07-21T01:41:42.979 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68d77fe700 1 -- 172.21.15.171:0/1860366327 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c8008890 con 0x5620dfd93040 2022-07-21T01:41:42.980 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68d77fe700 1 -- 172.21.15.171:0/1860366327 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c8008bf0 con 0x5620dfd93040 2022-07-21T01:41:42.980 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1860366327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfd93040 msgr2=0x5620dfd84b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:42.980 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 --2- 172.21.15.171:0/1860366327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfd93040 0x5620dfd84b10 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f68c800b000 tx=0x7f68c800c040).stop 2022-07-21T01:41:42.980 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1860366327 shutdown_connections 2022-07-21T01:41:42.981 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 --2- 172.21.15.171:0/1860366327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd95920 0x5620dfd85000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.981 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 --2- 172.21.15.171:0/1860366327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfd93040 0x5620dfd84b10 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.981 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 --2- 172.21.15.171:0/1860366327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfcfd470 0x5620dfd84620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.981 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1860366327 >> 172.21.15.171:0/1860366327 conn(0x5620dfd83850 msgr2=0x5620dfd85c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:42.982 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1860366327 shutdown_connections 2022-07-21T01:41:42.982 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1860366327 wait complete. 2022-07-21T01:41:42.982 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 Processor -- start 2022-07-21T01:41:42.982 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 -- start start 2022-07-21T01:41:42.983 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfcfd470 0x5620dfd9ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.983 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd93040 0x5620dfd99f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.983 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfd95920 0x5620dfd9a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.983 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620dfd99e00 con 0x5620dfcfd470 2022-07-21T01:41:42.984 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620dfd9ab70 con 0x5620dfd95920 2022-07-21T01:41:42.984 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68df9e1b80 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x5620dfd9ad30 con 0x5620dfd93040 2022-07-21T01:41:42.984 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68de61f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfd95920 0x5620dfd9a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.984 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dd61d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd93040 0x5620dfd99f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.985 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dde1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfcfd470 0x5620dfd9ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.985 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68de61f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfd95920 0x5620dfd9a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.171:59076/0 (socket says 172.21.15.171:59076) 2022-07-21T01:41:42.985 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dd61d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd93040 0x5620dfd99f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.171:53662/0 (socket says 172.21.15.171:53662) 2022-07-21T01:41:42.985 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68dde1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfcfd470 0x5620dfd9ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.171:50036/0 (socket says 172.21.15.171:50036) 2022-07-21T01:41:42.986 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.960+0000 7f68de61f700 1 -- 172.21.15.171:0/1292714287 learned_addr learned my addr 172.21.15.171:0/1292714287 (peer_addr_for_me v2:172.21.15.171:0/0) 2022-07-21T01:41:42.986 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68de61f700 1 -- 172.21.15.171:0/1292714287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd93040 msgr2=0x5620dfd99f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:42.986 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68de61f700 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x5620dfd93040 0x5620dfd99f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.986 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68de61f700 1 -- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfcfd470 msgr2=0x5620dfd9ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:42.986 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68de61f700 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfcfd470 0x5620dfd9ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:42.987 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68de61f700 1 -- 172.21.15.171:0/1292714287 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc001e10 con 0x5620dfd95920 2022-07-21T01:41:42.987 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68dde1e700 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620dfcfd470 0x5620dfd9ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:42.987 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68de61f700 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfd95920 0x5620dfd9a430 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f68d0019000 tx=0x7f68d0014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:42.987 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68d6ffd700 1 -- 172.21.15.171:0/1292714287 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68d0005ea0 con 0x5620dfd95920 2022-07-21T01:41:42.988 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68d6ffd700 1 -- 172.21.15.171:0/1292714287 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d000d070 con 0x5620dfd95920 2022-07-21T01:41:42.988 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c8009430 con 0x5620dfd95920 2022-07-21T01:41:42.988 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68d6ffd700 1 -- 172.21.15.171:0/1292714287 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68d001d430 con 0x5620dfd95920 2022-07-21T01:41:42.988 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.964+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5620dfc9e5d0 con 0x5620dfd95920 2022-07-21T01:41:42.990 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.968+0000 7f68d6ffd700 1 -- 172.21.15.171:0/1292714287 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f68d002d070 con 0x5620dfd95920 2022-07-21T01:41:42.991 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.968+0000 7f68d6ffd700 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68c00525f0 0x7f68c0054ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.991 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.968+0000 7f68d6ffd700 1 -- 172.21.15.171:0/1292714287 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6422+0+0 (secure 0 0 0) 0x7f68d006c9e0 con 0x5620dfd95920 2022-07-21T01:41:42.992 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.968+0000 7f68dde1e700 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68c00525f0 0x7f68c0054ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.992 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.968+0000 7f68df9e1b80 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x5620dfda7b40 0x5620dfda9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:42.992 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.968+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 --> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] -- osd_op(unknown.0.0:1 23.0 23:0620e6f9:::key:head [writefull 0~617 in=617b] snapc 0=[] ondisk+write+known_if_redirected e126) v8 -- 0x5620dfdaa4c0 con 0x5620dfda7b40 2022-07-21T01:41:42.992 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.968+0000 7f68dd61d700 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x5620dfda7b40 0x5620dfda9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:42.993 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.968+0000 7f68dde1e700 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68c00525f0 0x7f68c0054ab0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f68d8009000 tx=0x7f68d8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:42.993 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:42.968+0000 7f68dd61d700 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x5620dfda7b40 0x5620dfda9f30 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:43.917 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.916+0000 7f68d6ffd700 1 -- 172.21.15.171:0/1292714287 <== osd.0 v2:172.21.15.3:6813/22896 1 ==== osd_map(127..127 src has 1..127) v4 ==== 320+0+0 (crc 0 0 0) 0x7f68c8008bb0 con 0x5620dfda7b40 2022-07-21T01:41:43.924 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68dd61d700 1 -- 172.21.15.171:0/1292714287 <== osd.0 v2:172.21.15.3:6813/22896 2 ==== osd_op_reply(1 key [writefull 0~617] v127'1 uv1 ondisk = 0) v8 ==== 147+0+0 (crc 0 0 0) 0x7f68c8008d70 con 0x5620dfda7b40 2022-07-21T01:41:43.924 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x5620dfda7b40 msgr2=0x5620dfda9f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:43.924 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x5620dfda7b40 0x5620dfda9f30 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:43.925 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68c00525f0 msgr2=0x7f68c0054ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:43.925 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f68c00525f0 0x7f68c0054ab0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f68d8009000 tx=0x7f68d8000bc0).stop 2022-07-21T01:41:43.925 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfd95920 msgr2=0x5620dfd9a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:43.925 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 --2- 172.21.15.171:0/1292714287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5620dfd95920 0x5620dfd9a430 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f68d0019000 tx=0x7f68d0014000).stop 2022-07-21T01:41:43.930 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68de61f700 1 -- 172.21.15.171:0/1292714287 reap_dead start 2022-07-21T01:41:43.930 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 shutdown_connections 2022-07-21T01:41:43.930 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 >> 172.21.15.171:0/1292714287 conn(0x5620dfd83850 msgr2=0x5620dfd84830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:43.930 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 shutdown_connections 2022-07-21T01:41:43.931 INFO:teuthology.orchestra.run.smithi171.stderr:2022-07-21T01:41:43.920+0000 7f68df9e1b80 1 -- 172.21.15.171:0/1292714287 wait complete. 2022-07-21T01:41:43.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-21T01:41:44.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.032+0000 7f49b6df0700 1 Processor -- start 2022-07-21T01:41:44.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.032+0000 7f49b6df0700 1 -- start start 2022-07-21T01:41:44.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.032+0000 7f49b6df0700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f49b0060290 0x7f49b00b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:44.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.032+0000 7f49b6df0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b00b4dd0 0x7f49b014b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:44.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.032+0000 7f49b6df0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00aad40 con 0x7f49b0067490 2022-07-21T01:41:44.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.032+0000 7f49b6df0700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b0060670 con 0x7f49b0060290 2022-07-21T01:41:44.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.032+0000 7f49b6df0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b0067030 con 0x7f49b00b4dd0 2022-07-21T01:41:44.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.032+0000 7f49b4b8c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f49b0060290 0x7f49b00b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b4b8c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f49b0060290 0x7f49b00b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50124/0 (socket says 172.21.15.3:50124) 2022-07-21T01:41:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b4b8c700 1 -- 172.21.15.3:0/2742011831 learned_addr learned my addr 172.21.15.3:0/2742011831 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:44.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b538d700 1 --2- 172.21.15.3:0/2742011831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b00b4dd0 0x7f49b014b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:44.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b4b8c700 1 -- 172.21.15.3:0/2742011831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b00b4dd0 msgr2=0x7f49b014b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:44.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b4b8c700 1 --2- 172.21.15.3:0/2742011831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b00b4dd0 0x7f49b014b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:44.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b4b8c700 1 -- 172.21.15.3:0/2742011831 >> v1:172.21.15.143:6789/0 conn(0x7f49b0067490 legacy=0x7f49b0143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:44.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b4b8c700 1 -- 172.21.15.3:0/2742011831 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b00b3540 con 0x7f49b0060290 2022-07-21T01:41:44.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b4b8c700 1 --2- 172.21.15.3:0/2742011831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 0x7f49b00b48e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f49a0002f70 tx=0x7f49a0006eb0).ready entity=mon.0 client_cookie=e445cf2335308567 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:44.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49af7fe700 1 -- 172.21.15.3:0/2742011831 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49a000d5c0 con 0x7f49b0060290 2022-07-21T01:41:44.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49af7fe700 1 -- 172.21.15.3:0/2742011831 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a0000e90 con 0x7f49b0060290 2022-07-21T01:41:44.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49af7fe700 1 -- 172.21.15.3:0/2742011831 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49a000d950 con 0x7f49b0060290 2022-07-21T01:41:44.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 -- 172.21.15.3:0/2742011831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 msgr2=0x7f49b00b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:44.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 --2- 172.21.15.3:0/2742011831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 0x7f49b00b48e0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f49a0002f70 tx=0x7f49a0006eb0).stop 2022-07-21T01:41:44.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 -- 172.21.15.3:0/2742011831 shutdown_connections 2022-07-21T01:41:44.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 --2- 172.21.15.3:0/2742011831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b00b4dd0 0x7f49b014b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:44.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 --2- 172.21.15.3:0/2742011831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 0x7f49b00b48e0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:44.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 -- 172.21.15.3:0/2742011831 >> 172.21.15.3:0/2742011831 conn(0x7f49b005d0c0 msgr2=0x7f49b005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:44.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 -- 172.21.15.3:0/2742011831 shutdown_connections 2022-07-21T01:41:44.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 -- 172.21.15.3:0/2742011831 wait complete. 2022-07-21T01:41:44.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 Processor -- start 2022-07-21T01:41:44.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.036+0000 7f49b6df0700 1 -- start start 2022-07-21T01:41:44.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b6df0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 0x7f49b00b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:44.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b6df0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b0067490 0x7f49b00a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:44.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b6df0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b00b4dd0 0x7f49b00ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:44.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b6df0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00ae360 con 0x7f49b0060290 2022-07-21T01:41:44.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b6df0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00ae600 con 0x7f49b00b4dd0 2022-07-21T01:41:44.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b6df0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00b0cd0 con 0x7f49b0067490 2022-07-21T01:41:44.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b4b8c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 0x7f49b00b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:44.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b4b8c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 0x7f49b00b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50130/0 (socket says 172.21.15.3:50130) 2022-07-21T01:41:44.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b4b8c700 1 -- 172.21.15.3:0/4113855147 learned_addr learned my addr 172.21.15.3:0/4113855147 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:44.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b538d700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b00b4dd0 0x7f49b00ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:44.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49affff700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b0067490 0x7f49b00a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:44.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b4b8c700 1 -- 172.21.15.3:0/4113855147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b0067490 msgr2=0x7f49b00a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:44.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b4b8c700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b0067490 0x7f49b00a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:44.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b4b8c700 1 -- 172.21.15.3:0/4113855147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b00b4dd0 msgr2=0x7f49b00ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:44.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b4b8c700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b00b4dd0 0x7f49b00ab5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:44.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b4b8c700 1 -- 172.21.15.3:0/4113855147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a000b1e0 con 0x7f49b0060290 2022-07-21T01:41:44.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b4b8c700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 0x7f49b00b0270 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f49a0002f70 tx=0x7f49a0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:44.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49adffb700 1 -- 172.21.15.3:0/4113855147 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49a0011860 con 0x7f49b0060290 2022-07-21T01:41:44.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49adffb700 1 -- 172.21.15.3:0/4113855147 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a00119c0 con 0x7f49b0060290 2022-07-21T01:41:44.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b00ad290 con 0x7f49b0060290 2022-07-21T01:41:44.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49adffb700 1 -- 172.21.15.3:0/4113855147 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49a0011d20 con 0x7f49b0060290 2022-07-21T01:41:44.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b00a0db0 con 0x7f49b0060290 2022-07-21T01:41:44.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b00ae7c0 con 0x7f49b0060290 2022-07-21T01:41:44.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.040+0000 7f49adffb700 1 -- 172.21.15.3:0/4113855147 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f49a0025030 con 0x7f49b0060290 2022-07-21T01:41:44.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.044+0000 7f49adffb700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4990053b00 0x7f4990055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:44.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.044+0000 7f49adffb700 1 -- 172.21.15.3:0/4113855147 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6422+0+0 (secure 0 0 0) 0x7f49a006d330 con 0x7f49b0060290 2022-07-21T01:41:44.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.044+0000 7f49affff700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4990053b00 0x7f4990055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:44.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.044+0000 7f49affff700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4990053b00 0x7f4990055fc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f49a400b000 tx=0x7f49a4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:44.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.048+0000 7f49adffb700 1 -- 172.21.15.3:0/4113855147 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f49a00440f0 con 0x7f49b0060290 2022-07-21T01:41:44.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.220+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f49b00574e0 con 0x7f49b0060290 2022-07-21T01:41:44.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.220+0000 7f49adffb700 1 -- 172.21.15.3:0/4113855147 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1904 (secure 0 0 0) 0x7f49a0023020 con 0x7f49b0060290 2022-07-21T01:41:44.223 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:44.223 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:41:44.224 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:44.224 INFO:teuthology.orchestra.run.smithi003.stdout: "total_bytes": 773094113280, 2022-07-21T01:41:44.225 INFO:teuthology.orchestra.run.smithi003.stdout: "total_avail_bytes": 764491632640, 2022-07-21T01:41:44.225 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_bytes": 12546048, 2022-07-21T01:41:44.225 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_bytes": 8602480640, 2022-07-21T01:41:44.225 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_ratio": 0.01112733967602253, 2022-07-21T01:41:44.226 INFO:teuthology.orchestra.run.smithi003.stdout: "num_osds": 8, 2022-07-21T01:41:44.226 INFO:teuthology.orchestra.run.smithi003.stdout: "num_per_pool_osds": 8, 2022-07-21T01:41:44.226 INFO:teuthology.orchestra.run.smithi003.stdout: "num_per_pool_omap_osds": 8 2022-07-21T01:41:44.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:44.227 INFO:teuthology.orchestra.run.smithi003.stdout: "stats_by_class": { 2022-07-21T01:41:44.227 INFO:teuthology.orchestra.run.smithi003.stdout: "ssd": { 2022-07-21T01:41:44.227 INFO:teuthology.orchestra.run.smithi003.stdout: "total_bytes": 773094113280, 2022-07-21T01:41:44.227 INFO:teuthology.orchestra.run.smithi003.stdout: "total_avail_bytes": 764491632640, 2022-07-21T01:41:44.228 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_bytes": 12546048, 2022-07-21T01:41:44.228 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_bytes": 8602480640, 2022-07-21T01:41:44.228 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_ratio": 0.01112733967602253 2022-07-21T01:41:44.229 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:44.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:44.229 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:41:44.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:44.230 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_health_metrics", 2022-07-21T01:41:44.230 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 1, 2022-07-21T01:41:44.230 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:44.230 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:44.230 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 2, 2022-07-21T01:41:44.231 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:44.231 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:44.231 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:44.231 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362918379520 2022-07-21T01:41:44.232 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:44.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:44.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:44.232 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd", 2022-07-21T01:41:44.232 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 2, 2022-07-21T01:41:44.233 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:44.233 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:44.233 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 0, 2022-07-21T01:41:44.233 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:44.234 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:44.234 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:44.234 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362918379520 2022-07-21T01:41:44.234 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:44.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:44.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:44.235 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephfs_data", 2022-07-21T01:41:44.235 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 22, 2022-07-21T01:41:44.235 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:44.236 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:44.236 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 0, 2022-07-21T01:41:44.236 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:44.236 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:44.237 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:44.237 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362918379520 2022-07-21T01:41:44.237 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:44.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:44.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:44.238 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephfs_metadata", 2022-07-21T01:41:44.238 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 23, 2022-07-21T01:41:44.238 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:44.238 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:44.239 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 0, 2022-07-21T01:41:44.239 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:44.240 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:44.240 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:44.241 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362918379520 2022-07-21T01:41:44.241 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:44.242 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:44.242 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:44.242 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:41:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.220+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4990053b00 msgr2=0x7f4990055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:44.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.220+0000 7f49b6df0700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4990053b00 0x7f4990055fc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f49a400b000 tx=0x7f49a4009040).stop 2022-07-21T01:41:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.220+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 msgr2=0x7f49b00b0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.220+0000 7f49b6df0700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 0x7f49b00b0270 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f49a0002f70 tx=0x7f49a0015000).stop 2022-07-21T01:41:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.224+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 shutdown_connections 2022-07-21T01:41:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.224+0000 7f49b6df0700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4990053b00 0x7f4990055fc0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.224+0000 7f49b6df0700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b00b4dd0 0x7f49b00ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.224+0000 7f49b6df0700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49b0067490 0x7f49b00a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.224+0000 7f49b6df0700 1 --2- 172.21.15.3:0/4113855147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49b0060290 0x7f49b00b0270 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:44.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.224+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 >> 172.21.15.3:0/4113855147 conn(0x7f49b005d0c0 msgr2=0x7f49b00b4660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.224+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 shutdown_connections 2022-07-21T01:41:44.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:44.224+0000 7f49b6df0700 1 -- 172.21.15.3:0/4113855147 wait complete. 2022-07-21T01:41:44.249 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30)... 2022-07-21T01:41:49.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-21T01:41:49.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b40c3700 1 Processor -- start 2022-07-21T01:41:49.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b40c3700 1 -- start start 2022-07-21T01:41:49.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b40c3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89ac139c70 0x7f89ac142f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b40c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac143430 0x7f89ac0ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b40c3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f89ac0b1de0 0x7f89ac0abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b40c3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0b31a0 con 0x7f89ac139c70 2022-07-21T01:41:49.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b40c3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0af4b0 con 0x7f89ac0b1de0 2022-07-21T01:41:49.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b40c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac0ac600 con 0x7f89ac143430 2022-07-21T01:41:49.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b1e5f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89ac139c70 0x7f89ac142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b1e5f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f89ac139c70 0x7f89ac142f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50140/0 (socket says 172.21.15.3:50140) 2022-07-21T01:41:49.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b1e5f700 1 -- 172.21.15.3:0/278791749 learned_addr learned my addr 172.21.15.3:0/278791749 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:49.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b165e700 1 --2- 172.21.15.3:0/278791749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac143430 0x7f89ac0ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.360+0000 7f89b2660700 1 --2- 172.21.15.3:0/278791749 >> v2:172.21.15.143:3300/0 conn(0x7f89ac0b1de0 0x7f89ac0abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b1e5f700 1 -- 172.21.15.3:0/278791749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac143430 msgr2=0x7f89ac0ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:49.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b1e5f700 1 --2- 172.21.15.3:0/278791749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac143430 0x7f89ac0ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b1e5f700 1 -- 172.21.15.3:0/278791749 >> v2:172.21.15.143:3300/0 conn(0x7f89ac0b1de0 msgr2=0x7f89ac0abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:49.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b1e5f700 1 --2- 172.21.15.3:0/278791749 >> v2:172.21.15.143:3300/0 conn(0x7f89ac0b1de0 0x7f89ac0abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b1e5f700 1 -- 172.21.15.3:0/278791749 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac1438c0 con 0x7f89ac139c70 2022-07-21T01:41:49.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b165e700 1 --2- 172.21.15.3:0/278791749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac143430 0x7f89ac0ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:49.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b2660700 1 --2- 172.21.15.3:0/278791749 >> v2:172.21.15.143:3300/0 conn(0x7f89ac0b1de0 0x7f89ac0abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:49.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b1e5f700 1 --2- 172.21.15.3:0/278791749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac139c70 0x7f89ac142f40 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0x7f89ac060480 tx=0x7f89a8006eb0).ready entity=mon.0 client_cookie=b4e610c505186eb1 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:49.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b0e5d700 1 -- 172.21.15.3:0/278791749 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89a800e610 con 0x7f89ac139c70 2022-07-21T01:41:49.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b0e5d700 1 -- 172.21.15.3:0/278791749 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a8000e90 con 0x7f89ac139c70 2022-07-21T01:41:49.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b0e5d700 1 -- 172.21.15.3:0/278791749 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89a800e9a0 con 0x7f89ac139c70 2022-07-21T01:41:49.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b40c3700 1 -- 172.21.15.3:0/278791749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac139c70 msgr2=0x7f89ac142f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:49.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b40c3700 1 --2- 172.21.15.3:0/278791749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac139c70 0x7f89ac142f40 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0x7f89ac060480 tx=0x7f89a8006eb0).stop 2022-07-21T01:41:49.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b40c3700 1 -- 172.21.15.3:0/278791749 shutdown_connections 2022-07-21T01:41:49.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b40c3700 1 --2- 172.21.15.3:0/278791749 >> v2:172.21.15.143:3300/0 conn(0x7f89ac0b1de0 0x7f89ac0abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b40c3700 1 --2- 172.21.15.3:0/278791749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac143430 0x7f89ac0ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b40c3700 1 --2- 172.21.15.3:0/278791749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac139c70 0x7f89ac142f40 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b40c3700 1 -- 172.21.15.3:0/278791749 >> 172.21.15.3:0/278791749 conn(0x7f89ac05f840 msgr2=0x7f89ac05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:49.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b40c3700 1 -- 172.21.15.3:0/278791749 shutdown_connections 2022-07-21T01:41:49.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.364+0000 7f89b40c3700 1 -- 172.21.15.3:0/278791749 wait complete. 2022-07-21T01:41:49.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b40c3700 1 Processor -- start 2022-07-21T01:41:49.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b40c3700 1 -- start start 2022-07-21T01:41:49.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b40c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac0b1de0 0x7f89ac146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b40c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac1450a0 0x7f89ac0670d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b40c3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac1526e0 0x7f89ac152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b40c3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac14bc00 con 0x7f89ac1526e0 2022-07-21T01:41:49.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b40c3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac14bea0 con 0x7f89ac1450a0 2022-07-21T01:41:49.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b40c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac146870 con 0x7f89ac0b1de0 2022-07-21T01:41:49.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b2660700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac1526e0 0x7f89ac152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b2660700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac1526e0 0x7f89ac152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50148/0 (socket says 172.21.15.3:50148) 2022-07-21T01:41:49.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b2660700 1 -- 172.21.15.3:0/2426229403 learned_addr learned my addr 172.21.15.3:0/2426229403 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:49.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b1e5f700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac0b1de0 0x7f89ac146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b165e700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac1450a0 0x7f89ac0670d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b2660700 1 -- 172.21.15.3:0/2426229403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac0b1de0 msgr2=0x7f89ac146fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:49.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b2660700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac0b1de0 0x7f89ac146fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b2660700 1 -- 172.21.15.3:0/2426229403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac1450a0 msgr2=0x7f89ac0670d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:49.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b2660700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac1450a0 0x7f89ac0670d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b2660700 1 -- 172.21.15.3:0/2426229403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4009470 con 0x7f89ac1526e0 2022-07-21T01:41:49.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b1e5f700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac0b1de0 0x7f89ac146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:49.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.368+0000 7f89b2660700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac1526e0 0x7f89ac152b90 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f89a400c000 tx=0x7f89a40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:49.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f899affd700 1 -- 172.21.15.3:0/2426229403 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89a400c850 con 0x7f89ac1526e0 2022-07-21T01:41:49.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f899affd700 1 -- 172.21.15.3:0/2426229403 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a400c9b0 con 0x7f89ac1526e0 2022-07-21T01:41:49.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899c00b180 con 0x7f89ac1526e0 2022-07-21T01:41:49.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f899affd700 1 -- 172.21.15.3:0/2426229403 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89a400cd10 con 0x7f89ac1526e0 2022-07-21T01:41:49.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a800b1e0 con 0x7f89ac1526e0 2022-07-21T01:41:49.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac14b970 con 0x7f89ac1526e0 2022-07-21T01:41:49.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f899affd700 1 -- 172.21.15.3:0/2426229403 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f89a4022070 con 0x7f89ac1526e0 2022-07-21T01:41:49.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f899affd700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f898c053980 0x7f898c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f899affd700 1 -- 172.21.15.3:0/2426229403 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6422+0+0 (secure 0 0 0) 0x7f89a406c7b0 con 0x7f89ac1526e0 2022-07-21T01:41:49.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.372+0000 7f89b1e5f700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f898c053980 0x7f898c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.376+0000 7f89b1e5f700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f898c053980 0x7f898c055e40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f89a801b000 tx=0x7f89a8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:49.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.380+0000 7f899affd700 1 -- 172.21.15.3:0/2426229403 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f89a4043570 con 0x7f89ac1526e0 2022-07-21T01:41:49.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.548+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f89ac143b60 con 0x7f89ac1526e0 2022-07-21T01:41:49.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.548+0000 7f899affd700 1 -- 172.21.15.3:0/2426229403 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1932 (secure 0 0 0) 0x7f89a4016020 con 0x7f89ac1526e0 2022-07-21T01:41:49.554 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:49.555 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:41:49.555 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:49.556 INFO:teuthology.orchestra.run.smithi003.stdout: "total_bytes": 773094113280, 2022-07-21T01:41:49.556 INFO:teuthology.orchestra.run.smithi003.stdout: "total_avail_bytes": 764491456512, 2022-07-21T01:41:49.557 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_bytes": 12722176, 2022-07-21T01:41:49.557 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_bytes": 8602656768, 2022-07-21T01:41:49.558 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_ratio": 0.011127566918730736, 2022-07-21T01:41:49.558 INFO:teuthology.orchestra.run.smithi003.stdout: "num_osds": 8, 2022-07-21T01:41:49.559 INFO:teuthology.orchestra.run.smithi003.stdout: "num_per_pool_osds": 8, 2022-07-21T01:41:49.559 INFO:teuthology.orchestra.run.smithi003.stdout: "num_per_pool_omap_osds": 8 2022-07-21T01:41:49.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:49.560 INFO:teuthology.orchestra.run.smithi003.stdout: "stats_by_class": { 2022-07-21T01:41:49.561 INFO:teuthology.orchestra.run.smithi003.stdout: "ssd": { 2022-07-21T01:41:49.562 INFO:teuthology.orchestra.run.smithi003.stdout: "total_bytes": 773094113280, 2022-07-21T01:41:49.562 INFO:teuthology.orchestra.run.smithi003.stdout: "total_avail_bytes": 764491456512, 2022-07-21T01:41:49.562 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_bytes": 12722176, 2022-07-21T01:41:49.563 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_bytes": 8602656768, 2022-07-21T01:41:49.563 INFO:teuthology.orchestra.run.smithi003.stdout: "total_used_raw_ratio": 0.011127566918730736 2022-07-21T01:41:49.564 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:49.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:49.565 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:41:49.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:49.566 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_health_metrics", 2022-07-21T01:41:49.567 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 1, 2022-07-21T01:41:49.567 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:49.567 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:49.568 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 2, 2022-07-21T01:41:49.568 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:49.568 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:49.568 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:49.569 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362918281216 2022-07-21T01:41:49.569 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:49.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:49.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:49.570 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd", 2022-07-21T01:41:49.570 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 2, 2022-07-21T01:41:49.570 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:49.570 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:49.570 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 0, 2022-07-21T01:41:49.571 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:49.571 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:49.571 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:49.571 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362918281216 2022-07-21T01:41:49.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:49.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:49.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:49.573 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephfs_data", 2022-07-21T01:41:49.573 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 22, 2022-07-21T01:41:49.573 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:49.574 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 0, 2022-07-21T01:41:49.574 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 0, 2022-07-21T01:41:49.574 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 0, 2022-07-21T01:41:49.574 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 0, 2022-07-21T01:41:49.576 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 0, 2022-07-21T01:41:49.576 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362918281216 2022-07-21T01:41:49.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:49.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:49.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:49.577 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephfs_metadata", 2022-07-21T01:41:49.577 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 23, 2022-07-21T01:41:49.577 INFO:teuthology.orchestra.run.smithi003.stdout: "stats": { 2022-07-21T01:41:49.578 INFO:teuthology.orchestra.run.smithi003.stdout: "stored": 308, 2022-07-21T01:41:49.578 INFO:teuthology.orchestra.run.smithi003.stdout: "objects": 1, 2022-07-21T01:41:49.578 INFO:teuthology.orchestra.run.smithi003.stdout: "kb_used": 4, 2022-07-21T01:41:49.578 INFO:teuthology.orchestra.run.smithi003.stdout: "bytes_used": 4096, 2022-07-21T01:41:49.578 INFO:teuthology.orchestra.run.smithi003.stdout: "percent_used": 5.6431437300830112e-09, 2022-07-21T01:41:49.579 INFO:teuthology.orchestra.run.smithi003.stdout: "max_avail": 362918281216 2022-07-21T01:41:49.579 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:49.579 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:49.579 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:49.580 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:41:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f898c053980 msgr2=0x7f898c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f898c053980 0x7f898c055e40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f89a801b000 tx=0x7f89a8015000).stop 2022-07-21T01:41:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac1526e0 msgr2=0x7f89ac152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac1526e0 0x7f89ac152b90 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f89a400c000 tx=0x7f89a40091d0).stop 2022-07-21T01:41:49.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 shutdown_connections 2022-07-21T01:41:49.586 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89ac1526e0 0x7f89ac152b90 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f898c053980 0x7f898c055e40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac1450a0 0x7f89ac0670d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 --2- 172.21.15.3:0/2426229403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89ac0b1de0 0x7f89ac146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 >> 172.21.15.3:0/2426229403 conn(0x7f89ac05f840 msgr2=0x7f89ac0b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:49.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 shutdown_connections 2022-07-21T01:41:49.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.552+0000 7f89b40c3700 1 -- 172.21.15.3:0/2426229403 wait complete. 2022-07-21T01:41:49.590 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2022-07-21T01:41:49.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-07-21T01:41:49.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4d4bd7700 1 Processor -- start 2022-07-21T01:41:49.707 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4d4bd7700 1 -- start start 2022-07-21T01:41:49.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4d4bd7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4d00aebd0 0x7fa4d0143940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4d4bd7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4d00b0610 0x7fa4d0143e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4d4bd7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d00b0b20 con 0x7fa4d00ad190 2022-07-21T01:41:49.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4d4bd7700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d00adde0 con 0x7fa4d00aebd0 2022-07-21T01:41:49.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4d4bd7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d00aefb0 con 0x7fa4d00b0610 2022-07-21T01:41:49.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4cf7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4d00aebd0 0x7fa4d0143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4cf7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa4d00aebd0 0x7fa4d0143940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50152/0 (socket says 172.21.15.3:50152) 2022-07-21T01:41:49.713 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4cf7fe700 1 -- 172.21.15.3:0/58238429 learned_addr learned my addr 172.21.15.3:0/58238429 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:49.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4cf7fe700 1 -- 172.21.15.3:0/58238429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4d00b0610 msgr2=0x7fa4d0143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:49.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4cf7fe700 1 --2- 172.21.15.3:0/58238429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4d00b0610 0x7fa4d0143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:41:49.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4cf7fe700 1 -- 172.21.15.3:0/58238429 >> v1:172.21.15.3:6789/0 conn(0x7fa4d00ad190 legacy=0x7fa4d0146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:49.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4cf7fe700 1 -- 172.21.15.3:0/58238429 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d0146890 con 0x7fa4d00aebd0 2022-07-21T01:41:49.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4cf7fe700 1 --2- 172.21.15.3:0/58238429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4d00aebd0 0x7fa4d0143940 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7fa4c0002f70 tx=0x7fa4c0006eb0).ready entity=mon.0 client_cookie=564e999bab549a39 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:49.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4ceffd700 1 -- 172.21.15.3:0/58238429 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4c000d5c0 con 0x7fa4d00aebd0 2022-07-21T01:41:49.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.704+0000 7fa4ceffd700 1 -- 172.21.15.3:0/58238429 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c0000e90 con 0x7fa4d00aebd0 2022-07-21T01:41:49.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4ceffd700 1 -- 172.21.15.3:0/58238429 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4c000d950 con 0x7fa4d00aebd0 2022-07-21T01:41:49.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 -- 172.21.15.3:0/58238429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4d00aebd0 msgr2=0x7fa4d0143940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:49.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 --2- 172.21.15.3:0/58238429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4d00aebd0 0x7fa4d0143940 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7fa4c0002f70 tx=0x7fa4c0006eb0).stop 2022-07-21T01:41:49.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 -- 172.21.15.3:0/58238429 shutdown_connections 2022-07-21T01:41:49.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 --2- 172.21.15.3:0/58238429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4d00b0610 0x7fa4d0143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:41:49.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 --2- 172.21.15.3:0/58238429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4d00aebd0 0x7fa4d0143940 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 -- 172.21.15.3:0/58238429 >> 172.21.15.3:0/58238429 conn(0x7fa4d00613c0 msgr2=0x7fa4d0062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:49.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 -- 172.21.15.3:0/58238429 shutdown_connections 2022-07-21T01:41:49.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 -- 172.21.15.3:0/58238429 wait complete. 2022-07-21T01:41:49.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 Processor -- start 2022-07-21T01:41:49.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.708+0000 7fa4d4bd7700 1 -- start start 2022-07-21T01:41:49.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4d4bd7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4d00ad190 0x7fa4d0142af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4d4bd7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4d00aebd0 0x7fa4d0155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4d4bd7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4d00b0610 0x7fa4d0156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4d4bd7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0142310 con 0x7fa4d00ad190 2022-07-21T01:41:49.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4d4bd7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0145f50 con 0x7fa4d00b0610 2022-07-21T01:41:49.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4d4bd7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d01461f0 con 0x7fa4d00aebd0 2022-07-21T01:41:49.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4d00ad190 0x7fa4d0142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4d00ad190 0x7fa4d0142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50158/0 (socket says 172.21.15.3:50158) 2022-07-21T01:41:49.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cf7fe700 1 -- 172.21.15.3:0/805404374 learned_addr learned my addr 172.21.15.3:0/805404374 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:49.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4c7fff700 1 --2- 172.21.15.3:0/805404374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4d00aebd0 0x7fa4d0155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cffff700 1 --2- 172.21.15.3:0/805404374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4d00b0610 0x7fa4d0156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cf7fe700 1 -- 172.21.15.3:0/805404374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4d00aebd0 msgr2=0x7fa4d0155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:49.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cf7fe700 1 --2- 172.21.15.3:0/805404374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4d00aebd0 0x7fa4d0155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cf7fe700 1 -- 172.21.15.3:0/805404374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4d00b0610 msgr2=0x7fa4d0156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:49.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cf7fe700 1 --2- 172.21.15.3:0/805404374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4d00b0610 0x7fa4d0156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:49.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cf7fe700 1 -- 172.21.15.3:0/805404374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c000b1e0 con 0x7fa4d00ad190 2022-07-21T01:41:49.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4c7fff700 1 --2- 172.21.15.3:0/805404374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4d00aebd0 0x7fa4d0155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:49.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cf7fe700 1 --2- 172.21.15.3:0/805404374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4d00ad190 0x7fa4d0142af0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7fa4c0002f70 tx=0x7fa4c0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:49.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cd7fa700 1 -- 172.21.15.3:0/805404374 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4c00118a0 con 0x7fa4d00ad190 2022-07-21T01:41:49.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cd7fa700 1 -- 172.21.15.3:0/805404374 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c0011a00 con 0x7fa4d00ad190 2022-07-21T01:41:49.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4d4bd7700 1 -- 172.21.15.3:0/805404374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d01412e0 con 0x7fa4d00ad190 2022-07-21T01:41:49.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cd7fa700 1 -- 172.21.15.3:0/805404374 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4c0011d60 con 0x7fa4d00ad190 2022-07-21T01:41:49.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4d4bd7700 1 -- 172.21.15.3:0/805404374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d0156780 con 0x7fa4d00ad190 2022-07-21T01:41:49.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4d4bd7700 1 -- 172.21.15.3:0/805404374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d01463b0 con 0x7fa4d00ad190 2022-07-21T01:41:49.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.712+0000 7fa4cd7fa700 1 -- 172.21.15.3:0/805404374 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa4c0025030 con 0x7fa4d00ad190 2022-07-21T01:41:49.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.716+0000 7fa4cd7fa700 1 --2- 172.21.15.3:0/805404374 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa4b4053980 0x7fa4b4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:49.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.716+0000 7fa4cd7fa700 1 -- 172.21.15.3:0/805404374 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6422+0+0 (secure 0 0 0) 0x7fa4c006d420 con 0x7fa4d00ad190 2022-07-21T01:41:49.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.716+0000 7fa4c7fff700 1 --2- 172.21.15.3:0/805404374 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa4b4053980 0x7fa4b4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:49.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.716+0000 7fa4c7fff700 1 --2- 172.21.15.3:0/805404374 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa4b4053980 0x7fa4b4055e40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fa4b800b000 tx=0x7fa4b8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:49.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.720+0000 7fa4cd7fa700 1 -- 172.21.15.3:0/805404374 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa4c00441e0 con 0x7fa4d00ad190 2022-07-21T01:41:49.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.880+0000 7fa4d4bd7700 1 -- 172.21.15.3:0/805404374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa4d00b0d50 con 0x7fa4d00ad190 2022-07-21T01:41:49.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.880+0000 7fa4cd7fa700 1 -- 172.21.15.3:0/805404374 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=-22 pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. v62) v1 ==== 209+0+0 (secure 0 0 0) 0x7fa4c0023020 con 0x7fa4d00ad190 2022-07-21T01:41:49.886 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. 2022-07-21T01:41:49.887 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-07-21T01:41:49.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:41:49.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.992+0000 7f12a92ca700 1 Processor -- start 2022-07-21T01:41:49.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.992+0000 7f12a92ca700 1 -- start start 2022-07-21T01:41:50.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f12a40af450 0x7f12a40af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f12a40ad4b0 0x7f12a41432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 0x7f12a40a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a40aff30 con 0x7f12a40af450 2022-07-21T01:41:50.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a405c7a0 con 0x7f12a40ad4b0 2022-07-21T01:41:50.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a4140990 con 0x7f12a41437d0 2022-07-21T01:41:50.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f12a40af450 0x7f12a40af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f12a40af450 0x7f12a40af830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50166/0 (socket says 172.21.15.3:50166) 2022-07-21T01:41:50.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a3fff700 1 -- 172.21.15.3:0/1134355764 learned_addr learned my addr 172.21.15.3:0/1134355764 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:50.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f129b7fe700 1 --2- 172.21.15.3:0/1134355764 >> v2:172.21.15.143:3300/0 conn(0x7f12a40ad4b0 0x7f12a41432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a8ac9700 1 --2- 172.21.15.3:0/1134355764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 0x7f12a40a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a3fff700 1 -- 172.21.15.3:0/1134355764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 msgr2=0x7f12a40a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:50.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a3fff700 1 --2- 172.21.15.3:0/1134355764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 0x7f12a40a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a3fff700 1 -- 172.21.15.3:0/1134355764 >> v2:172.21.15.143:3300/0 conn(0x7f12a40ad4b0 msgr2=0x7f12a41432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:50.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a3fff700 1 --2- 172.21.15.3:0/1134355764 >> v2:172.21.15.143:3300/0 conn(0x7f12a40ad4b0 0x7f12a41432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a3fff700 1 -- 172.21.15.3:0/1134355764 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a40ad9e0 con 0x7f12a40af450 2022-07-21T01:41:50.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a3fff700 1 --2- 172.21.15.3:0/1134355764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40af450 0x7f12a40af830 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f12a40ad900 tx=0x7f1294006eb0).ready entity=mon.0 client_cookie=1f8d3f7be6512fdc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:50.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a37fe700 1 -- 172.21.15.3:0/1134355764 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f129400e610 con 0x7f12a40af450 2022-07-21T01:41:50.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a37fe700 1 -- 172.21.15.3:0/1134355764 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1294000e90 con 0x7f12a40af450 2022-07-21T01:41:50.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a37fe700 1 -- 172.21.15.3:0/1134355764 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f129400e9a0 con 0x7f12a40af450 2022-07-21T01:41:50.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 -- 172.21.15.3:0/1134355764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40af450 msgr2=0x7f12a40af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 --2- 172.21.15.3:0/1134355764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40af450 0x7f12a40af830 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f12a40ad900 tx=0x7f1294006eb0).stop 2022-07-21T01:41:50.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 -- 172.21.15.3:0/1134355764 shutdown_connections 2022-07-21T01:41:50.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 --2- 172.21.15.3:0/1134355764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 0x7f12a40a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 --2- 172.21.15.3:0/1134355764 >> v2:172.21.15.143:3300/0 conn(0x7f12a40ad4b0 0x7f12a41432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 --2- 172.21.15.3:0/1134355764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40af450 0x7f12a40af830 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 -- 172.21.15.3:0/1134355764 >> 172.21.15.3:0/1134355764 conn(0x7f12a40612d0 msgr2=0x7f12a4062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:50.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:49.996+0000 7f12a92ca700 1 -- 172.21.15.3:0/1134355764 shutdown_connections 2022-07-21T01:41:50.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 -- 172.21.15.3:0/1134355764 wait complete. 2022-07-21T01:41:50.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 Processor -- start 2022-07-21T01:41:50.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 -- start start 2022-07-21T01:41:50.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40ad4b0 0x7f12a4142830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12a40af450 0x7f12a413f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 0x7f12a40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a40b4270 con 0x7f12a40ad4b0 2022-07-21T01:41:50.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a40b3d70 con 0x7f12a40af450 2022-07-21T01:41:50.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a4142df0 con 0x7f12a41437d0 2022-07-21T01:41:50.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40ad4b0 0x7f12a4142830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40ad4b0 0x7f12a4142830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50172/0 (socket says 172.21.15.3:50172) 2022-07-21T01:41:50.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a3fff700 1 -- 172.21.15.3:0/2211483882 learned_addr learned my addr 172.21.15.3:0/2211483882 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:50.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f129b7fe700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12a40af450 0x7f12a413f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a8ac9700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 0x7f12a40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a3fff700 1 -- 172.21.15.3:0/2211483882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 msgr2=0x7f12a40678e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a3fff700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 0x7f12a40678e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a3fff700 1 -- 172.21.15.3:0/2211483882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12a40af450 msgr2=0x7f12a413f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a3fff700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12a40af450 0x7f12a413f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a3fff700 1 -- 172.21.15.3:0/2211483882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129400b1e0 con 0x7f12a40ad4b0 2022-07-21T01:41:50.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a3fff700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40ad4b0 0x7f12a4142830 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f12a4142f80 tx=0x7f1294015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:50.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a1ffb700 1 -- 172.21.15.3:0/2211483882 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1294011760 con 0x7f12a40ad4b0 2022-07-21T01:41:50.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a40b3fd0 con 0x7f12a40ad4b0 2022-07-21T01:41:50.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a1ffb700 1 -- 172.21.15.3:0/2211483882 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12940118c0 con 0x7f12a40ad4b0 2022-07-21T01:41:50.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a1ffb700 1 -- 172.21.15.3:0/2211483882 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1294011c20 con 0x7f12a40ad4b0 2022-07-21T01:41:50.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.000+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a413fa50 con 0x7f12a40ad4b0 2022-07-21T01:41:50.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.004+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a40574e0 con 0x7f12a40ad4b0 2022-07-21T01:41:50.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.004+0000 7f12a1ffb700 1 -- 172.21.15.3:0/2211483882 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1294025030 con 0x7f12a40ad4b0 2022-07-21T01:41:50.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.004+0000 7f12a1ffb700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1288053a10 0x7f1288055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.004+0000 7f12a1ffb700 1 -- 172.21.15.3:0/2211483882 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6422+0+0 (secure 0 0 0) 0x7f129406d330 con 0x7f12a40ad4b0 2022-07-21T01:41:50.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.004+0000 7f129b7fe700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1288053a10 0x7f1288055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.004+0000 7f129b7fe700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1288053a10 0x7f1288055ed0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f128c00b000 tx=0x7f128c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:50.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.008+0000 7f12a1ffb700 1 -- 172.21.15.3:0/2211483882 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1294044050 con 0x7f12a40ad4b0 2022-07-21T01:41:50.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.164+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f12a4061b20 con 0x7f12a40ad4b0 2022-07-21T01:41:50.169 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:41:50.164+0000 7f6e564ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:41:50.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a1ffb700 1 -- 172.21.15.3:0/2211483882 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 23 and data pool 22 v63) v1 ==== 189+0+0 (secure 0 0 0) 0x7f12940230d0 con 0x7f12a40ad4b0 2022-07-21T01:41:50.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1288053a10 msgr2=0x7f1288055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1288053a10 0x7f1288055ed0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f128c00b000 tx=0x7f128c006df0).stop 2022-07-21T01:41:50.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40ad4b0 msgr2=0x7f12a4142830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40ad4b0 0x7f12a4142830 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f12a4142f80 tx=0x7f1294015000).stop 2022-07-21T01:41:50.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 shutdown_connections 2022-07-21T01:41:50.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12a41437d0 0x7f12a40678e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1288053a10 0x7f1288055ed0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12a40af450 0x7f12a413f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 --2- 172.21.15.3:0/2211483882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12a40ad4b0 0x7f12a4142830 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 >> 172.21.15.3:0/2211483882 conn(0x7f12a40612d0 msgr2=0x7f12a40a8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:50.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 shutdown_connections 2022-07-21T01:41:50.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.176+0000 7f12a92ca700 1 -- 172.21.15.3:0/2211483882 wait complete. 2022-07-21T01:41:50.184 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 23 and data pool 22 2022-07-21T01:41:50.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:41:50.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.292+0000 7f9f1ea5c700 1 Processor -- start 2022-07-21T01:41:50.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.292+0000 7f9f1ea5c700 1 -- start start 2022-07-21T01:41:50.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.292+0000 7f9f1ea5c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f180b28b0 0x7f9f1813e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.292+0000 7f9f1ea5c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f180aac20 con 0x7f9f180b1180 2022-07-21T01:41:50.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.292+0000 7f9f1ea5c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f180b2c90 con 0x7f9f18141920 2022-07-21T01:41:50.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.292+0000 7f9f1ea5c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18141db0 con 0x7f9f180b28b0 2022-07-21T01:41:50.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.292+0000 7f9f1d259700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9f180b1180 0x7f9f18141210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58462/0 (socket says 172.21.15.3:58462) 2022-07-21T01:41:50.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.292+0000 7f9f1d259700 1 -- 172.21.15.3:0/3909032883 learned_addr learned my addr 172.21.15.3:0/3909032883 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:50.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.292+0000 7f9f1da5a700 1 --2- 172.21.15.3:0/3909032883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f180b28b0 0x7f9f1813e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ca58700 1 -- 172.21.15.3:0/3909032883 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346826786 0 0) 0x7f9f08001110 con 0x7f9f180b1180 2022-07-21T01:41:50.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ca58700 1 -- 172.21.15.3:0/3909032883 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f00003260 con 0x7f9f180b1180 2022-07-21T01:41:50.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ca58700 1 -- 172.21.15.3:0/3909032883 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454024270 0 0) 0x7f9f180b2c90 con 0x7f9f18141920 2022-07-21T01:41:50.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ca58700 1 -- 172.21.15.3:0/3909032883 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f08001110 con 0x7f9f18141920 2022-07-21T01:41:50.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1da5a700 1 -- 172.21.15.3:0/3909032883 >> v1:172.21.15.143:6789/0 conn(0x7f9f18141920 legacy=0x7f9f181443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ca58700 1 -- 172.21.15.3:0/3909032883 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638135380 0 0) 0x7f9f00003260 con 0x7f9f180b1180 2022-07-21T01:41:50.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1da5a700 1 -- 172.21.15.3:0/3909032883 >> v1:172.21.15.3:6789/0 conn(0x7f9f180b1180 legacy=0x7f9f18141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1da5a700 1 -- 172.21.15.3:0/3909032883 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180aa440 con 0x7f9f180b28b0 2022-07-21T01:41:50.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1da5a700 1 --2- 172.21.15.3:0/3909032883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f180b28b0 0x7f9f1813e1c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f9f080012a0 tx=0x7f9f100091d0).ready entity=mon.2 client_cookie=fdcc9fed9300db45 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:50.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ca58700 1 -- 172.21.15.3:0/3909032883 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f080017f0 con 0x7f9f180b28b0 2022-07-21T01:41:50.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ca58700 1 -- 172.21.15.3:0/3909032883 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1000f510 con 0x7f9f180b28b0 2022-07-21T01:41:50.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ca58700 1 -- 172.21.15.3:0/3909032883 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f1000f680 con 0x7f9f180b28b0 2022-07-21T01:41:50.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3909032883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f180b28b0 msgr2=0x7f9f1813e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ea5c700 1 --2- 172.21.15.3:0/3909032883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f180b28b0 0x7f9f1813e1c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f9f080012a0 tx=0x7f9f100091d0).stop 2022-07-21T01:41:50.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3909032883 shutdown_connections 2022-07-21T01:41:50.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ea5c700 1 --2- 172.21.15.3:0/3909032883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f180b28b0 0x7f9f1813e1c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3909032883 >> 172.21.15.3:0/3909032883 conn(0x7f9f18060310 msgr2=0x7f9f180b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:50.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3909032883 shutdown_connections 2022-07-21T01:41:50.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3909032883 wait complete. 2022-07-21T01:41:50.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ea5c700 1 Processor -- start 2022-07-21T01:41:50.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.296+0000 7f9f1ea5c700 1 -- start start 2022-07-21T01:41:50.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1ea5c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f180b1180 0x7f9f18147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1ea5c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f180b28b0 0x7f9f180b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1ea5c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f18141920 0x7f9f180b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1ea5c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180b3ce0 con 0x7f9f180b28b0 2022-07-21T01:41:50.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1ea5c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180b3ae0 con 0x7f9f180b1180 2022-07-21T01:41:50.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1ea5c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18145280 con 0x7f9f18141920 2022-07-21T01:41:50.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1d259700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f180b28b0 0x7f9f180b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1d259700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f180b28b0 0x7f9f180b3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50186/0 (socket says 172.21.15.3:50186) 2022-07-21T01:41:50.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1d259700 1 -- 172.21.15.3:0/3690528145 learned_addr learned my addr 172.21.15.3:0/3690528145 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:50.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1da5a700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f180b1180 0x7f9f18147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1e25b700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f18141920 0x7f9f180b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1d259700 1 -- 172.21.15.3:0/3690528145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f18141920 msgr2=0x7f9f180b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1d259700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f18141920 0x7f9f180b35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1d259700 1 -- 172.21.15.3:0/3690528145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f180b1180 msgr2=0x7f9f18147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:50.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1d259700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f180b1180 0x7f9f18147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:50.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1d259700 1 -- 172.21.15.3:0/3690528145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f10009470 con 0x7f9f180b28b0 2022-07-21T01:41:50.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1d259700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f180b28b0 0x7f9f180b3100 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f9f18144e10 tx=0x7f9f08001cc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:50.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f06ffd700 1 -- 172.21.15.3:0/3690528145 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f08007170 con 0x7f9f180b28b0 2022-07-21T01:41:50.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f06ffd700 1 -- 172.21.15.3:0/3690528145 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f080072d0 con 0x7f9f180b28b0 2022-07-21T01:41:50.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f18153160 con 0x7f9f180b28b0 2022-07-21T01:41:50.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f06ffd700 1 -- 172.21.15.3:0/3690528145 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f08004920 con 0x7f9f180b28b0 2022-07-21T01:41:50.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f18152e80 con 0x7f9f180b28b0 2022-07-21T01:41:50.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.300+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f18148c10 con 0x7f9f180b28b0 2022-07-21T01:41:50.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.304+0000 7f9f06ffd700 1 -- 172.21.15.3:0/3690528145 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9f08004a80 con 0x7f9f180b28b0 2022-07-21T01:41:50.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.304+0000 7f9f06ffd700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ef8053aa0 0x7f9ef8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:50.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.304+0000 7f9f1da5a700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ef8053aa0 0x7f9ef8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:50.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.304+0000 7f9f06ffd700 1 -- 172.21.15.3:0/3690528145 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f9f0806bba0 con 0x7f9f180b28b0 2022-07-21T01:41:50.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.304+0000 7f9f1da5a700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ef8053aa0 0x7f9ef8055f60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f9f1001b000 tx=0x7f9f10016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:50.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.312+0000 7f9f06ffd700 1 -- 172.21.15.3:0/3690528145 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f08042910 con 0x7f9f180b28b0 2022-07-21T01:41:50.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:50.468+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9f180aac20 con 0x7f9f180b28b0 2022-07-21T01:41:51.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.183+0000 7f9f06ffd700 1 -- 172.21.15.3:0/3690528145 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9f180aac20 con 0x7f9f180b28b0 2022-07-21T01:41:51.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ef8053aa0 msgr2=0x7f9ef8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:51.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ef8053aa0 0x7f9ef8055f60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f9f1001b000 tx=0x7f9f10016000).stop 2022-07-21T01:41:51.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f180b28b0 msgr2=0x7f9f180b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:51.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f180b28b0 0x7f9f180b3100 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f9f18144e10 tx=0x7f9f08001cc0).stop 2022-07-21T01:41:51.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 shutdown_connections 2022-07-21T01:41:51.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f18141920 0x7f9f180b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:51.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f180b28b0 0x7f9f180b3100 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:51.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9ef8053aa0 0x7f9ef8055f60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:51.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 --2- 172.21.15.3:0/3690528145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f180b1180 0x7f9f18147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:51.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 >> 172.21.15.3:0/3690528145 conn(0x7f9f18060310 msgr2=0x7f9f18145510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:51.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 shutdown_connections 2022-07-21T01:41:51.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.187+0000 7f9f1ea5c700 1 -- 172.21.15.3:0/3690528145 wait complete. 2022-07-21T01:41:51.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:41:51.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f977db93700 1 Processor -- start 2022-07-21T01:41:51.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f977db93700 1 -- start start 2022-07-21T01:41:51.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f977db93700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b4550 0x7f97780b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:51.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f977db93700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f97780b6af0 0x7f977812bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:51.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f977db93700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9778144240 con 0x7f97780b6710 2022-07-21T01:41:51.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f977db93700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9778143d00 con 0x7f97780b6af0 2022-07-21T01:41:51.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f977db93700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9778143fa0 con 0x7f97780b4550 2022-07-21T01:41:51.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f97777fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f97780b6710 0x7f9778061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58476/0 (socket says 172.21.15.3:58476) 2022-07-21T01:41:51.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f97777fe700 1 -- 172.21.15.3:0/2407694866 learned_addr learned my addr 172.21.15.3:0/2407694866 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:51.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f9777fff700 1 --2- 172.21.15.3:0/2407694866 >> v2:172.21.15.3:3300/0 conn(0x7f97780b6af0 0x7f977812bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:51.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.327+0000 7f9776ffd700 1 --2- 172.21.15.3:0/2407694866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b4550 0x7f97780b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:51.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f97767fc700 1 -- 172.21.15.3:0/2407694866 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331709302 0 0) 0x7f9778144240 con 0x7f97780b6710 2022-07-21T01:41:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f97767fc700 1 -- 172.21.15.3:0/2407694866 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9764003260 con 0x7f97780b6710 2022-07-21T01:41:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f9777fff700 1 -- 172.21.15.3:0/2407694866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b4550 msgr2=0x7f97780b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f9777fff700 1 --2- 172.21.15.3:0/2407694866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b4550 0x7f97780b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f9777fff700 1 -- 172.21.15.3:0/2407694866 >> v1:172.21.15.3:6789/0 conn(0x7f97780b6710 legacy=0x7f9778061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f9777fff700 1 -- 172.21.15.3:0/2407694866 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778061780 con 0x7f97780b6af0 2022-07-21T01:41:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f9777fff700 1 --2- 172.21.15.3:0/2407694866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 0x7f977812bb90 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7f97700032f0 tx=0x7f97700091d0).ready entity=mon.0 client_cookie=4def3d25f3737586 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f97767fc700 1 -- 172.21.15.3:0/2407694866 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f977000b600 con 0x7f97780b6af0 2022-07-21T01:41:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f97767fc700 1 -- 172.21.15.3:0/2407694866 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977000b760 con 0x7f97780b6af0 2022-07-21T01:41:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f97767fc700 1 -- 172.21.15.3:0/2407694866 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f977000b8d0 con 0x7f97780b6af0 2022-07-21T01:41:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 -- 172.21.15.3:0/2407694866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 msgr2=0x7f977812bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 --2- 172.21.15.3:0/2407694866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 0x7f977812bb90 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7f97700032f0 tx=0x7f97700091d0).stop 2022-07-21T01:41:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 -- 172.21.15.3:0/2407694866 shutdown_connections 2022-07-21T01:41:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 --2- 172.21.15.3:0/2407694866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 0x7f977812bb90 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:51.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 --2- 172.21.15.3:0/2407694866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b4550 0x7f97780b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:51.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 -- 172.21.15.3:0/2407694866 >> 172.21.15.3:0/2407694866 conn(0x7f977805c620 msgr2=0x7f977805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:51.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 -- 172.21.15.3:0/2407694866 shutdown_connections 2022-07-21T01:41:51.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 -- 172.21.15.3:0/2407694866 wait complete. 2022-07-21T01:41:51.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 Processor -- start 2022-07-21T01:41:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.331+0000 7f977db93700 1 -- start start 2022-07-21T01:41:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f977db93700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97780b4550 0x7f97780ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f977db93700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b6710 0x7f97780af830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f977db93700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 0x7f97780b9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:51.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f977db93700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97780b9aa0 con 0x7f97780b6af0 2022-07-21T01:41:51.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f977db93700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9778128b50 con 0x7f97780b4550 2022-07-21T01:41:51.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f977db93700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9778128370 con 0x7f97780b6710 2022-07-21T01:41:51.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9777fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 0x7f97780b9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:51.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9777fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 0x7f97780b9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50204/0 (socket says 172.21.15.3:50204) 2022-07-21T01:41:51.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9777fff700 1 -- 172.21.15.3:0/1893161890 learned_addr learned my addr 172.21.15.3:0/1893161890 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:51.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f97777fe700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97780b4550 0x7f97780ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9776ffd700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b6710 0x7f97780af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9777fff700 1 -- 172.21.15.3:0/1893161890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b6710 msgr2=0x7f97780af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9777fff700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b6710 0x7f97780af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9777fff700 1 -- 172.21.15.3:0/1893161890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97780b4550 msgr2=0x7f97780ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9777fff700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97780b4550 0x7f97780ae5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9777fff700 1 -- 172.21.15.3:0/1893161890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9770009470 con 0x7f97780b6af0 2022-07-21T01:41:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9777fff700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 0x7f97780b9f00 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f97700032f0 tx=0x7f9770016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9774ff9700 1 -- 172.21.15.3:0/1893161890 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f977000b600 con 0x7f97780b6af0 2022-07-21T01:41:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9774ff9700 1 -- 172.21.15.3:0/1893161890 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9770011b60 con 0x7f97780b6af0 2022-07-21T01:41:51.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9778128610 con 0x7f97780b6af0 2022-07-21T01:41:51.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f9774ff9700 1 -- 172.21.15.3:0/1893161890 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f977001f3f0 con 0x7f97780b6af0 2022-07-21T01:41:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.335+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97780ba3f0 con 0x7f97780b6af0 2022-07-21T01:41:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.339+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9778061a20 con 0x7f97780b6af0 2022-07-21T01:41:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.339+0000 7f9774ff9700 1 -- 172.21.15.3:0/1893161890 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9770026030 con 0x7f97780b6af0 2022-07-21T01:41:51.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.339+0000 7f9774ff9700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9758053aa0 0x7f9758055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:51.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.339+0000 7f9774ff9700 1 -- 172.21.15.3:0/1893161890 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f977006c9c0 con 0x7f97780b6af0 2022-07-21T01:41:51.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.339+0000 7f97777fe700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9758053aa0 0x7f9758055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:51.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.339+0000 7f97777fe700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9758053aa0 0x7f9758055f60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f976800d000 tx=0x7f9768009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:51.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.343+0000 7f9774ff9700 1 -- 172.21.15.3:0/1893161890 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f97700437f0 con 0x7f97780b6af0 2022-07-21T01:41:51.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:51.467+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f97780574e0 con 0x7f97780b6af0 2022-07-21T01:41:52.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.187+0000 7f9774ff9700 1 -- 172.21.15.3:0/1893161890 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v130) v1 ==== 185+0+0 (secure 0 0 0) 0x7f97700150e0 con 0x7f97780b6af0 2022-07-21T01:41:52.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9758053aa0 msgr2=0x7f9758055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:52.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9758053aa0 0x7f9758055f60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f976800d000 tx=0x7f9768009040).stop 2022-07-21T01:41:52.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 msgr2=0x7f97780b9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:52.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 0x7f97780b9f00 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f97700032f0 tx=0x7f9770016000).stop 2022-07-21T01:41:52.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 shutdown_connections 2022-07-21T01:41:52.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97780b6af0 0x7f97780b9f00 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97780b6710 0x7f97780af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9758053aa0 0x7f9758055f60 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:52.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 --2- 172.21.15.3:0/1893161890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97780b4550 0x7f97780ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:52.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 >> 172.21.15.3:0/1893161890 conn(0x7f977805c620 msgr2=0x7f97781432f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:52.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 shutdown_connections 2022-07-21T01:41:52.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.191+0000 7f977db93700 1 -- 172.21.15.3:0/1893161890 wait complete. 2022-07-21T01:41:52.200 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:41:52.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:41:52.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee32e3f700 1 Processor -- start 2022-07-21T01:41:52.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee32e3f700 1 -- start start 2022-07-21T01:41:52.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee32e3f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee2c0b5490 0x7fee2c13e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:52.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee32e3f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fee2c0b15f0 0x7fee2c13e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:52.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee32e3f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c0af0b0 con 0x7fee2c1411d0 2022-07-21T01:41:52.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee32e3f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee2c0ac490 con 0x7fee2c0b5490 2022-07-21T01:41:52.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee32e3f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee2c0af2e0 con 0x7fee2c0b15f0 2022-07-21T01:41:52.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee31e3d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee2c0b5490 0x7fee2c13e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:52.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee31e3d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee2c0b5490 0x7fee2c13e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50208/0 (socket says 172.21.15.3:50208) 2022-07-21T01:41:52.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee31e3d700 1 -- 172.21.15.3:0/3439385575 learned_addr learned my addr 172.21.15.3:0/3439385575 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:52.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee3163c700 1 --2- 172.21.15.3:0/3439385575 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0b15f0 0x7fee2c13e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:52.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee31e3d700 1 -- 172.21.15.3:0/3439385575 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0b15f0 msgr2=0x7fee2c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:52.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee31e3d700 1 --2- 172.21.15.3:0/3439385575 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0b15f0 0x7fee2c13e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:52.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee31e3d700 1 -- 172.21.15.3:0/3439385575 >> v1:172.21.15.143:6789/0 conn(0x7fee2c1411d0 legacy=0x7fee2c141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:52.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee31e3d700 1 -- 172.21.15.3:0/3439385575 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c0af580 con 0x7fee2c0b5490 2022-07-21T01:41:52.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee31e3d700 1 --2- 172.21.15.3:0/3439385575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 0x7fee2c13e240 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7fee2c0af900 tx=0x7fee28006eb0).ready entity=mon.0 client_cookie=29f9c3707135be60 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:52.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee30e3b700 1 -- 172.21.15.3:0/3439385575 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee2800d5c0 con 0x7fee2c0b5490 2022-07-21T01:41:52.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee30e3b700 1 -- 172.21.15.3:0/3439385575 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee28000e90 con 0x7fee2c0b5490 2022-07-21T01:41:52.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.323+0000 7fee30e3b700 1 -- 172.21.15.3:0/3439385575 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee2800d910 con 0x7fee2c0b5490 2022-07-21T01:41:52.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 -- 172.21.15.3:0/3439385575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 msgr2=0x7fee2c13e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:52.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 --2- 172.21.15.3:0/3439385575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 0x7fee2c13e240 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7fee2c0af900 tx=0x7fee28006eb0).stop 2022-07-21T01:41:52.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 -- 172.21.15.3:0/3439385575 shutdown_connections 2022-07-21T01:41:52.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 --2- 172.21.15.3:0/3439385575 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0b15f0 0x7fee2c13e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:52.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 --2- 172.21.15.3:0/3439385575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 0x7fee2c13e240 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:52.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 -- 172.21.15.3:0/3439385575 >> 172.21.15.3:0/3439385575 conn(0x7fee2c0626f0 msgr2=0x7fee2c062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:52.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 -- 172.21.15.3:0/3439385575 shutdown_connections 2022-07-21T01:41:52.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 -- 172.21.15.3:0/3439385575 wait complete. 2022-07-21T01:41:52.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 Processor -- start 2022-07-21T01:41:52.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.327+0000 7fee32e3f700 1 -- start start 2022-07-21T01:41:52.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee32e3f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0b15f0 0x7fee2c13bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:52.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee32e3f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 0x7fee2c0aad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:52.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee32e3f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c1411d0 0x7fee2c0ab220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:52.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee32e3f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c0b3c90 con 0x7fee2c0b5490 2022-07-21T01:41:52.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee32e3f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c0b4820 con 0x7fee2c1411d0 2022-07-21T01:41:52.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee32e3f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c0b4ac0 con 0x7fee2c0b15f0 2022-07-21T01:41:52.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3163c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 0x7fee2c0aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:52.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3163c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 0x7fee2c0aad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50216/0 (socket says 172.21.15.3:50216) 2022-07-21T01:41:52.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3163c700 1 -- 172.21.15.3:0/1350154932 learned_addr learned my addr 172.21.15.3:0/1350154932 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:52.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3263e700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c1411d0 0x7fee2c0ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:52.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee31e3d700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0b15f0 0x7fee2c13bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:52.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3163c700 1 -- 172.21.15.3:0/1350154932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0b15f0 msgr2=0x7fee2c13bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:52.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3163c700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0b15f0 0x7fee2c13bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:52.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3163c700 1 -- 172.21.15.3:0/1350154932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c1411d0 msgr2=0x7fee2c0ab220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:52.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3163c700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c1411d0 0x7fee2c0ab220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:52.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3163c700 1 -- 172.21.15.3:0/1350154932 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2800b1e0 con 0x7fee2c0b5490 2022-07-21T01:41:52.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee3163c700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 0x7fee2c0aad30 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7fee1c00a000 tx=0x7fee1c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:52.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee1affd700 1 -- 172.21.15.3:0/1350154932 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee1c011780 con 0x7fee2c0b5490 2022-07-21T01:41:52.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee1affd700 1 -- 172.21.15.3:0/1350154932 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c00de90 con 0x7fee2c0b5490 2022-07-21T01:41:52.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c0b5090 con 0x7fee2c0b5490 2022-07-21T01:41:52.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee1affd700 1 -- 172.21.15.3:0/1350154932 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee1c011ad0 con 0x7fee2c0b5490 2022-07-21T01:41:52.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.331+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c0b3940 con 0x7fee2c0b5490 2022-07-21T01:41:52.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.335+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c0b3e90 con 0x7fee2c0b5490 2022-07-21T01:41:52.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.335+0000 7fee1affd700 1 -- 172.21.15.3:0/1350154932 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fee1c011c30 con 0x7fee2c0b5490 2022-07-21T01:41:52.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.335+0000 7fee1affd700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee0c053b30 0x7fee0c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:52.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.335+0000 7fee1affd700 1 -- 172.21.15.3:0/1350154932 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6068+0+0 (secure 0 0 0) 0x7fee1c06bc70 con 0x7fee2c0b5490 2022-07-21T01:41:52.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.335+0000 7fee31e3d700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee0c053b30 0x7fee0c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:52.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.335+0000 7fee31e3d700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee0c053b30 0x7fee0c055ff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fee2801c000 tx=0x7fee28015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:52.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.339+0000 7fee1affd700 1 -- 172.21.15.3:0/1350154932 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fee1c042b90 con 0x7fee2c0b5490 2022-07-21T01:41:52.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:52.463+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fee2c141d70 con 0x7fee2c0b5490 2022-07-21T01:41:53.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.191+0000 7fee1affd700 1 -- 172.21.15.3:0/1350154932 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v131) v1 ==== 133+0+0 (secure 0 0 0) 0x7fee1c0461b0 con 0x7fee2c0b5490 2022-07-21T01:41:53.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee0c053b30 msgr2=0x7fee0c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee0c053b30 0x7fee0c055ff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fee2801c000 tx=0x7fee28015000).stop 2022-07-21T01:41:53.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 msgr2=0x7fee2c0aad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 0x7fee2c0aad30 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7fee1c00a000 tx=0x7fee1c008040).stop 2022-07-21T01:41:53.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 shutdown_connections 2022-07-21T01:41:53.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee0c053b30 0x7fee0c055ff0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c1411d0 0x7fee2c0ab220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee2c0b5490 0x7fee2c0aad30 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 --2- 172.21.15.3:0/1350154932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0b15f0 0x7fee2c13bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 >> 172.21.15.3:0/1350154932 conn(0x7fee2c0626f0 msgr2=0x7fee2c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 shutdown_connections 2022-07-21T01:41:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.195+0000 7fee32e3f700 1 -- 172.21.15.3:0/1350154932 wait complete. 2022-07-21T01:41:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:41:53.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-07-21T01:41:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.319+0000 7f010c3b1700 1 Processor -- start 2022-07-21T01:41:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.319+0000 7f010c3b1700 1 -- start start 2022-07-21T01:41:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f01040b4550 0x7f010413e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01040b4180 con 0x7f01040af910 2022-07-21T01:41:53.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01040afcf0 con 0x7f01040b0920 2022-07-21T01:41:53.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f010413eeb0 con 0x7f01040b4550 2022-07-21T01:41:53.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010a94e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f01040b4550 0x7f010413e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010a14d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f01040af910 0x7f01040b1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58502/0 (socket says 172.21.15.3:58502) 2022-07-21T01:41:53.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010a14d700 1 -- 172.21.15.3:0/119393733 learned_addr learned my addr 172.21.15.3:0/119393733 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:53.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010914b700 1 -- 172.21.15.3:0/119393733 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983533675 0 0) 0x7f01040afcf0 con 0x7f01040b0920 2022-07-21T01:41:53.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010a94e700 1 -- 172.21.15.3:0/119393733 >> v1:172.21.15.143:6789/0 conn(0x7f01040b0920 legacy=0x7f010413e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010a94e700 1 -- 172.21.15.3:0/119393733 >> v1:172.21.15.3:6789/0 conn(0x7f01040af910 legacy=0x7f01040b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010a94e700 1 -- 172.21.15.3:0/119393733 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01040b3f20 con 0x7f01040b4550 2022-07-21T01:41:53.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010a94e700 1 --2- 172.21.15.3:0/119393733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b4550 0x7f010413e840 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7f0100001110 tx=0x7f00fc0091d0).ready entity=mon.0 client_cookie=9b9e3427706db516 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:53.332 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010914b700 1 -- 172.21.15.3:0/119393733 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00fc009ea0 con 0x7f01040b4550 2022-07-21T01:41:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010914b700 1 -- 172.21.15.3:0/119393733 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00fc00b730 con 0x7f01040b4550 2022-07-21T01:41:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010914b700 1 -- 172.21.15.3:0/119393733 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00fc00b8a0 con 0x7f01040b4550 2022-07-21T01:41:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 -- 172.21.15.3:0/119393733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b4550 msgr2=0x7f010413e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 --2- 172.21.15.3:0/119393733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b4550 0x7f010413e840 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7f0100001110 tx=0x7f00fc0091d0).stop 2022-07-21T01:41:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 -- 172.21.15.3:0/119393733 shutdown_connections 2022-07-21T01:41:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 --2- 172.21.15.3:0/119393733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b4550 0x7f010413e840 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 -- 172.21.15.3:0/119393733 >> 172.21.15.3:0/119393733 conn(0x7f010406a2a0 msgr2=0x7f010405ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 -- 172.21.15.3:0/119393733 shutdown_connections 2022-07-21T01:41:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.323+0000 7f010c3b1700 1 -- 172.21.15.3:0/119393733 wait complete. 2022-07-21T01:41:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 Processor -- start 2022-07-21T01:41:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 -- start start 2022-07-21T01:41:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01040af910 0x7f010409f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b0920 0x7f01040ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01040b4550 0x7f01040ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01040aced0 con 0x7f01040b0920 2022-07-21T01:41:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0104151170 con 0x7f01040af910 2022-07-21T01:41:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0104150c60 con 0x7f01040b4550 2022-07-21T01:41:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010994c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b0920 0x7f01040ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010994c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b0920 0x7f01040ac4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50230/0 (socket says 172.21.15.3:50230) 2022-07-21T01:41:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010994c700 1 -- 172.21.15.3:0/1354847352 learned_addr learned my addr 172.21.15.3:0/1354847352 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010a94e700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01040b4550 0x7f01040ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010a14d700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01040af910 0x7f010409f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010994c700 1 -- 172.21.15.3:0/1354847352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01040b4550 msgr2=0x7f01040ac9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010994c700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01040b4550 0x7f01040ac9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010994c700 1 -- 172.21.15.3:0/1354847352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01040af910 msgr2=0x7f010409f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010994c700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01040af910 0x7f010409f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010994c700 1 -- 172.21.15.3:0/1354847352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc009470 con 0x7f01040b0920 2022-07-21T01:41:53.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010994c700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b0920 0x7f01040ac4f0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f010409ef40 tx=0x7f00f4007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:53.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f00f37fe700 1 -- 172.21.15.3:0/1354847352 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00f4012400 con 0x7f01040b0920 2022-07-21T01:41:53.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f00f37fe700 1 -- 172.21.15.3:0/1354847352 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f4012560 con 0x7f01040b0920 2022-07-21T01:41:53.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0104150ed0 con 0x7f01040b0920 2022-07-21T01:41:53.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f00f37fe700 1 -- 172.21.15.3:0/1354847352 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00f4012900 con 0x7f01040b0920 2022-07-21T01:41:53.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.327+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0104141970 con 0x7f01040b0920 2022-07-21T01:41:53.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.331+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01040574e0 con 0x7f01040b0920 2022-07-21T01:41:53.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.331+0000 7f00f37fe700 1 -- 172.21.15.3:0/1354847352 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f00f4019030 con 0x7f01040b0920 2022-07-21T01:41:53.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.331+0000 7f00f37fe700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f00e40539e0 0x7f00e4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.331+0000 7f00f37fe700 1 -- 172.21.15.3:0/1354847352 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6422+0+0 (secure 0 0 0) 0x7f00f406b230 con 0x7f01040b0920 2022-07-21T01:41:53.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.331+0000 7f010a14d700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f00e40539e0 0x7f00e4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.331+0000 7f010a14d700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f00e40539e0 0x7f00e4055ea0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f010000d000 tx=0x7f0100009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.339+0000 7f00f37fe700 1 -- 172.21.15.3:0/1354847352 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f00f40420a0 con 0x7f01040b0920 2022-07-21T01:41:53.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.507+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f01040447e0 con 0x7f01040b0920 2022-07-21T01:41:53.509 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:41:53.507+0000 7f6e564ae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:41:53.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.515+0000 7f00f37fe700 1 -- 172.21.15.3:0/1354847352 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 22 v65) v1 ==== 174+0+0 (secure 0 0 0) 0x7f00f40456c0 con 0x7f01040b0920 2022-07-21T01:41:53.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f00e40539e0 msgr2=0x7f00e4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f00e40539e0 0x7f00e4055ea0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f010000d000 tx=0x7f0100009040).stop 2022-07-21T01:41:53.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b0920 msgr2=0x7f01040ac4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b0920 0x7f01040ac4f0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f010409ef40 tx=0x7f00f4007ac0).stop 2022-07-21T01:41:53.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 shutdown_connections 2022-07-21T01:41:53.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01040b4550 0x7f01040ac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01040b0920 0x7f01040ac4f0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f00e40539e0 0x7f00e4055ea0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 --2- 172.21.15.3:0/1354847352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01040af910 0x7f010409f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 >> 172.21.15.3:0/1354847352 conn(0x7f010406a2a0 msgr2=0x7f01041446e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:53.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 shutdown_connections 2022-07-21T01:41:53.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.519+0000 7f010c3b1700 1 -- 172.21.15.3:0/1354847352 wait complete. 2022-07-21T01:41:53.530 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 24 and data pool 22 2022-07-21T01:41:53.537 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:41:53.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:53.555 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:53.575 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:53.599 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:41:53.620 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:41:53.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:53.621 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:53.621 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:53.628 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:41:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:41:53.630 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:41:53.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:41:53.631 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:41:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:41:53.682 INFO:teuthology.orchestra.run.smithi171.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:41:53.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:41:53.683 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:41:53.684 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:53.691 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:41:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:41:53.693 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:41:53.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:41:53.694 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:41:53.700 INFO:teuthology.orchestra.run.smithi174.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:41:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:41:53.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:41:53.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 Processor -- start 2022-07-21T01:41:53.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 -- start start 2022-07-21T01:41:53.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9100b2710 0x7fd91014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd9100b01a0 0x7fd91014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9100a7250 con 0x7fd9100afd60 2022-07-21T01:41:53.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9100a7480 con 0x7fd9100b01a0 2022-07-21T01:41:53.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9100a7720 con 0x7fd9100b2710 2022-07-21T01:41:53.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90e7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd9100b01a0 0x7fd91014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd9100afd60 0x7fd9100a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58518/0 (socket says 172.21.15.3:58518) 2022-07-21T01:41:53.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90f7fe700 1 -- 172.21.15.3:0/2553549249 learned_addr learned my addr 172.21.15.3:0/2553549249 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:53.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90effd700 1 --2- 172.21.15.3:0/2553549249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9100b2710 0x7fd91014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90e7fc700 1 -- 172.21.15.3:0/2553549249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9100b2710 msgr2=0x7fd91014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:53.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90e7fc700 1 --2- 172.21.15.3:0/2553549249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9100b2710 0x7fd91014afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90e7fc700 1 -- 172.21.15.3:0/2553549249 >> v1:172.21.15.3:6789/0 conn(0x7fd9100afd60 legacy=0x7fd9100a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90e7fc700 1 -- 172.21.15.3:0/2553549249 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9100a79c0 con 0x7fd9100b01a0 2022-07-21T01:41:53.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90e7fc700 1 --2- 172.21.15.3:0/2553549249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 0x7fd91014b4c0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7fd900002f70 tx=0x7fd900006df0).ready entity=mon.0 client_cookie=d58faae63ffec5ae server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:53.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90dffb700 1 -- 172.21.15.3:0/2553549249 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd90000d570 con 0x7fd9100b01a0 2022-07-21T01:41:53.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90dffb700 1 -- 172.21.15.3:0/2553549249 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd900000e90 con 0x7fd9100b01a0 2022-07-21T01:41:53.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90dffb700 1 -- 172.21.15.3:0/2553549249 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd90000d8c0 con 0x7fd9100b01a0 2022-07-21T01:41:53.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 -- 172.21.15.3:0/2553549249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 msgr2=0x7fd91014b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 --2- 172.21.15.3:0/2553549249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 0x7fd91014b4c0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7fd900002f70 tx=0x7fd900006df0).stop 2022-07-21T01:41:53.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 -- 172.21.15.3:0/2553549249 shutdown_connections 2022-07-21T01:41:53.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 --2- 172.21.15.3:0/2553549249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 0x7fd91014b4c0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 --2- 172.21.15.3:0/2553549249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9100b2710 0x7fd91014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 -- 172.21.15.3:0/2553549249 >> 172.21.15.3:0/2553549249 conn(0x7fd910060310 msgr2=0x7fd9100b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:53.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.815+0000 7fd90ffff700 1 -- 172.21.15.3:0/2553549249 shutdown_connections 2022-07-21T01:41:53.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90ffff700 1 -- 172.21.15.3:0/2553549249 wait complete. 2022-07-21T01:41:53.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90ffff700 1 Processor -- start 2022-07-21T01:41:53.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90ffff700 1 -- start start 2022-07-21T01:41:53.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9100afd60 0x7fd9100b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 0x7fd91005c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9100b2710 0x7fd910156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90ffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd910156700 con 0x7fd9100b01a0 2022-07-21T01:41:53.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9100b4d50 con 0x7fd9100b2710 2022-07-21T01:41:53.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9100b4810 con 0x7fd9100afd60 2022-07-21T01:41:53.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 0x7fd91005c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 0x7fd91005c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50244/0 (socket says 172.21.15.3:50244) 2022-07-21T01:41:53.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90e7fc700 1 -- 172.21.15.3:0/1497763111 learned_addr learned my addr 172.21.15.3:0/1497763111 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:53.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90f7fe700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9100b2710 0x7fd910156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90e7fc700 1 -- 172.21.15.3:0/1497763111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9100afd60 msgr2=0x7fd9100b5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90e7fc700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9100afd60 0x7fd9100b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:41:53.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90e7fc700 1 -- 172.21.15.3:0/1497763111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9100b2710 msgr2=0x7fd910156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:53.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90e7fc700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9100b2710 0x7fd910156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:53.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90e7fc700 1 -- 172.21.15.3:0/1497763111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90000b180 con 0x7fd9100b01a0 2022-07-21T01:41:53.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.819+0000 7fd90f7fe700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9100b2710 0x7fd910156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:53.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd90e7fc700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 0x7fd91005c170 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7fd900002f70 tx=0x7fd900015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:53.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd8fffff700 1 -- 172.21.15.3:0/1497763111 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd90000bed0 con 0x7fd9100b01a0 2022-07-21T01:41:53.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd8fffff700 1 -- 172.21.15.3:0/1497763111 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90000de50 con 0x7fd9100b01a0 2022-07-21T01:41:53.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9100b4ab0 con 0x7fd9100b01a0 2022-07-21T01:41:53.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd8fffff700 1 -- 172.21.15.3:0/1497763111 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd90001e460 con 0x7fd9100b01a0 2022-07-21T01:41:53.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9100b2f60 con 0x7fd9100b01a0 2022-07-21T01:41:53.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9100574e0 con 0x7fd9100b01a0 2022-07-21T01:41:53.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd8fffff700 1 -- 172.21.15.3:0/1497763111 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd900025030 con 0x7fd9100b01a0 2022-07-21T01:41:53.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd8fffff700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053b30 0x7fd8ec055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:53.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd8fffff700 1 -- 172.21.15.3:0/1497763111 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fd90006d420 con 0x7fd9100b01a0 2022-07-21T01:41:53.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd90effd700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053b30 0x7fd8ec055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:53.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.823+0000 7fd90effd700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053b30 0x7fd8ec055ff0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fd90400a000 tx=0x7fd904008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:53.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.831+0000 7fd8fffff700 1 -- 172.21.15.3:0/1497763111 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd900044080 con 0x7fd9100b01a0 2022-07-21T01:41:53.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.959+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd910069a70 con 0x7fd9100b01a0 2022-07-21T01:41:53.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd8fffff700 1 -- 172.21.15.3:0/1497763111 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v132) v1 ==== 81+0+34214 (secure 0 0 0) 0x7fd900023090 con 0x7fd9100b01a0 2022-07-21T01:41:53.966 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:53.967 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:41:53.967 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 132, 2022-07-21T01:41:53.967 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:41:53.968 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:41:53.968 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:41:53.510119+0000", 2022-07-21T01:41:53.968 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:41:53.968 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:41:53.969 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:41:53.969 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:41:53.969 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:41:53.969 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:41:53.970 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:41:53.970 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:41:53.970 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:41:53.971 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:53.971 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:41:53.971 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:41:53.971 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:41:53.972 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:41:53.972 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:41:53.972 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 24, 2022-07-21T01:41:53.972 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:41:53.973 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:41:53.973 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:41:53.973 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:41:53.973 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:41:53.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:53.974 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:41:53.974 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:41:53.974 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:41:53.975 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:41:53.975 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:41:53.975 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:53.975 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:53.976 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:53.976 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:53.976 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:53.976 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:53.977 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:41:53.977 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:41:53.977 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:41:53.978 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:41:53.978 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:41:53.978 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:53.979 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:53.979 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:53.979 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:53.979 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:53.980 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:53.980 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:53.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:53.980 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:41:53.981 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:53.981 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:53.981 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:53.981 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:53.982 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:53.982 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:53.982 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:53.982 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:53.983 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:53.983 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:53.983 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:53.983 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:53.984 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:53.984 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:53.984 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:53.984 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:53.985 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:53.985 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:53.985 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:53.985 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:53.986 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:53.986 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:53.986 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:53.986 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:53.987 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:53.987 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:53.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:53.987 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:53.988 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:53.988 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:53.988 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:53.988 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:53.989 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:53.989 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:53.989 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:53.989 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:53.990 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:53.990 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:53.990 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:41:53.990 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:41:53.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:53.991 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:53.991 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:41:53.991 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:53.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:53.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:53.992 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:41:53.992 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:41:53.993 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:41:53.993 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:41:53.993 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:41:53.993 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:53.994 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:53.994 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:53.994 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:53.994 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:53.995 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:53.995 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:41:53.995 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:41:53.995 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:41:53.996 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:41:53.996 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:41:53.996 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:53.996 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:53.997 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:53.997 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:53.997 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:53.997 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:53.998 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:53.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:53.998 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:41:53.998 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:53.999 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:53.999 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:53.999 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:53.999 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:54.000 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:54.000 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:54.000 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:54.000 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:54.001 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:54.001 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:54.001 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:54.001 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:54.002 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:54.002 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:54.002 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:54.002 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:54.003 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:54.003 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:54.003 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:54.003 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:54.004 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:54.004 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:54.004 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:54.004 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:54.005 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:54.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.005 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:54.005 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:54.006 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:54.006 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:54.006 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:54.006 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:54.007 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:54.007 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:54.008 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:54.008 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:54.008 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:54.009 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:41:54.009 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:54.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:41:54.010 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.012 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 22, 2022-07-21T01:41:54.012 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:41:54.013 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:41:27.012266+0000", 2022-07-21T01:41:54.013 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:41:54.013 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:41:54.014 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:54.014 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:54.015 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:54.015 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:54.016 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:54.017 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:54.017 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:41:54.018 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:41:54.018 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:41:54.018 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:41:54.019 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:41:54.019 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:54.020 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:54.020 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:54.020 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:54.021 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:54.021 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:54.022 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:54.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.022 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "132", 2022-07-21T01:41:54.023 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:54.023 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:54.024 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:54.024 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:54.024 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:54.027 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:54.027 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:54.027 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:54.027 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:54.028 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:54.028 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:54.028 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:54.028 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:54.029 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:54.029 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:54.029 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:54.029 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:54.030 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:54.030 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:54.030 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:54.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.031 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:54.031 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:54.031 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:54.031 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:54.032 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:54.032 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:54.032 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:54.032 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:54.033 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:54.033 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:54.034 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:54.034 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:41:54.034 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:54.035 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:41:54.035 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:41:54.036 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.036 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.038 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 24, 2022-07-21T01:41:54.038 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:41:54.038 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:41:52.523908+0000", 2022-07-21T01:41:54.039 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 32769, 2022-07-21T01:41:54.039 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool,creating", 2022-07-21T01:41:54.040 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:41:54.040 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:41:54.041 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:41:54.041 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:41:54.042 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:41:54.042 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:41:54.043 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:41:54.043 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:41:54.044 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:41:54.044 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:41:54.044 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:41:54.045 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:41:54.045 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:41:54.046 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:41:54.046 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:41:54.047 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:41:54.047 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:41:54.047 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:41:54.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.048 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "132", 2022-07-21T01:41:54.048 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:41:54.049 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:41:54.049 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:41:54.049 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:41:54.049 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:41:54.050 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:41:54.050 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:41:54.050 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:41:54.051 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:41:54.051 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:41:54.052 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:41:54.052 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:41:54.052 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:41:54.052 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:41:54.053 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:41:54.053 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:41:54.053 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:41:54.053 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:41:54.054 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:41:54.054 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:41:54.054 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:41:54.054 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:41:54.054 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:41:54.055 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:41:54.055 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:41:54.055 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:41:54.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.056 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:41:54.056 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:41:54.056 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:41:54.056 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:41:54.057 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:41:54.057 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:41:54.057 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:41:54.057 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:41:54.058 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:41:54.058 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:41:54.058 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:41:54.058 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:41:54.059 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:41:54.059 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:41:54.059 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:41:54.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.060 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:41:54.060 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:41:54.060 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:41:54.060 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.061 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.061 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.061 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:54.061 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:41:54.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.062 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:41:54.062 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:41:54.062 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:54.062 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:54.063 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:54.063 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:54.063 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:54.063 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:54.064 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:41:54.064 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:41:54.064 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:54.064 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:54.065 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:54.065 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.065 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:41:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.067 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.067 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:41:54.067 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:54.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.068 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.068 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:54.068 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.069 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.069 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:41:54.069 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:54.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.070 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.070 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:41:54.070 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:54.071 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.071 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.071 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:54.072 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.072 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.072 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:41:54.072 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:54.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.073 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.074 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:41:54.074 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:54.074 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.075 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.075 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:54.075 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.076 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.076 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:41:54.076 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:54.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.077 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.077 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:41:54.077 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:41:54.077 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.078 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.078 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:41:54.079 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:41:54.079 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:41:54.079 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:41:54.079 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:54.080 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:54.080 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:54.080 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.081 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:41:54.081 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:41:54.081 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:54.081 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:54.082 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:54.082 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:54.082 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:54.082 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:54.083 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:54.083 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:41:54.083 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:54.083 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:54.084 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:54.084 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.084 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.085 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:41:54.085 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:54.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.085 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.086 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:41:54.086 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:54.086 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.086 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.087 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:54.087 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.088 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.088 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:41:54.088 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:54.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.089 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.089 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:41:54.089 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:54.090 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.090 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.090 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:54.091 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.091 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.091 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:41:54.092 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:54.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.092 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.093 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:41:54.093 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:54.093 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.093 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.095 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.095 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:41:54.095 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:54.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.096 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.096 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:41:54.096 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:41:54.096 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.097 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.097 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:41:54.097 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:41:54.098 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:41:54.098 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:41:54.098 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:54.098 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:54.099 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:54.099 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.100 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:41:54.100 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:41:54.100 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:54.100 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:54.100 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:54.101 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:54.101 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:54.101 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:54.101 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:41:54.102 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 126, 2022-07-21T01:41:54.102 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:54.102 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:54.102 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:54.103 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.103 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.103 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:41:54.103 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:54.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.104 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.104 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:41:54.105 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:54.105 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.105 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.106 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:54.106 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.106 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.107 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:41:54.107 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:54.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.108 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.108 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:41:54.109 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:54.109 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.109 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.110 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:54.110 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.111 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.111 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:41:54.112 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:54.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.113 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.113 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:41:54.113 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:54.113 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.113 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.114 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:54.115 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.115 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.115 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:41:54.116 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:54.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.117 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.117 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:41:54.117 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:41:54.118 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.118 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.118 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:41:54.119 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:41:54.119 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:41:54.119 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:41:54.119 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:54.119 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:54.120 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:54.120 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.121 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:41:54.121 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:41:54.121 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:54.121 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:54.122 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:54.122 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:54.122 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:41:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:41:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:54.124 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:54.124 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.125 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.125 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:41:54.125 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:54.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.126 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.126 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:41:54.126 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:54.127 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.127 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.128 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:54.128 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.129 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.129 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:41:54.129 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:54.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.130 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.130 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:41:54.131 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:54.131 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.132 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.133 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:54.133 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.134 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.134 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:41:54.135 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:54.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.136 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.137 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:41:54.137 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:54.138 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.138 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.139 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:54.139 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.140 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.140 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:41:54.140 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:54.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.142 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.142 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:41:54.143 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:41:54.143 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.143 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.144 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:41:54.144 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:41:54.144 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:41:54.144 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:41:54.145 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:54.145 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:54.145 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:54.146 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.147 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:41:54.148 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:41:54.148 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:54.149 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:54.149 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:54.150 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:54.150 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:54.150 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:54.151 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:54.151 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 126, 2022-07-21T01:41:54.152 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:54.152 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:54.153 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:54.153 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.154 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.155 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:41:54.155 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:54.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.157 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.157 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:41:54.157 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:54.158 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.158 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.159 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:54.160 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.161 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:41:54.161 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:54.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.162 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.162 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:41:54.162 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:54.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.163 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.163 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:54.164 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.164 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.164 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:41:54.164 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:54.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.165 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.165 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:41:54.166 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:54.166 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.166 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.166 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:54.167 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.167 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.167 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:41:54.168 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:54.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:41:54.169 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:41:54.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.169 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.170 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:41:54.170 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:41:54.170 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:41:54.170 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:41:54.171 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:54.171 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:54.171 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:54.171 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.172 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:41:54.172 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:41:54.173 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:54.173 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:54.173 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:54.173 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:54.173 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:54.174 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:54.174 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:54.174 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:41:54.174 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:54.175 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:54.175 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:54.175 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.175 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.176 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:41:54.176 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:54.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.177 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.177 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:41:54.177 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:54.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.178 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.178 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:54.178 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.179 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:41:54.179 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:54.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.180 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.180 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:41:54.181 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:54.181 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.181 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.182 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:54.182 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.182 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.182 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:41:54.183 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:54.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.183 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.184 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:41:54.184 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:54.184 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.184 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.185 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:54.185 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.185 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.186 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:41:54.186 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:54.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.187 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:41:54.187 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:41:54.187 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.187 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.188 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:41:54.188 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:41:54.188 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:41:54.189 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:41:54.189 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:54.189 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:54.189 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:54.189 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.190 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:41:54.190 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:41:54.191 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:54.191 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:54.191 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:54.191 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:54.192 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:54.192 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:54.192 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:54.192 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:41:54.192 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:54.193 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:54.193 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:54.193 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.193 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.194 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:41:54.194 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:54.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.195 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.195 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:41:54.195 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:54.195 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.196 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.196 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:54.196 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.197 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:41:54.197 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:54.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.198 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.198 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:41:54.198 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:54.199 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.199 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.200 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:54.200 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.200 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.200 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:41:54.201 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:54.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.201 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.202 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:41:54.202 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:54.202 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.202 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.203 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:54.203 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.203 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.204 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:41:54.204 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:54.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.205 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.205 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:41:54.205 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:41:54.205 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.205 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.206 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:41:54.206 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:41:54.206 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:41:54.207 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:41:54.207 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:54.207 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:54.207 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:54.208 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.208 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:41:54.208 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:41:54.209 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:41:54.209 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:41:54.209 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:41:54.209 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:41:54.210 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:41:54.210 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:41:54.210 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:41:54.210 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:41:54.211 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:41:54.211 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:41:54.211 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:41:54.211 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.212 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.212 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:41:54.213 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:54.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.213 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.214 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:41:54.214 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:54.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.214 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.215 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:41:54.215 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.215 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.216 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:41:54.216 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:54.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.217 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.217 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:41:54.217 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:54.218 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.218 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.218 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:41:54.219 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.219 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.219 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:41:54.220 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:54.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.220 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.220 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:41:54.221 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:54.221 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.221 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.222 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:41:54.222 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:41:54.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.222 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:41:54.223 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:41:54.223 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:54.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.223 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:41:54.224 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:41:54.224 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:41:54.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.224 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.225 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:41:54.225 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:41:54.225 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:41:54.225 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:41:54.226 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:41:54.226 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:41:54.226 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:41:54.226 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:41:54.227 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.227 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:54.227 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:41:54.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.227 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:41:54.228 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:54.228 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:54.228 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:54.228 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:41:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.230 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:41:54.230 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:54.230 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:54.230 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:54.231 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:54.231 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:54.231 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:41:54.231 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:54.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.232 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:41:54.232 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:54.232 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:54.233 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:54.233 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:54.233 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:54.233 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:41:54.233 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:54.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.234 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:41:54.234 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:54.235 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:54.235 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:54.235 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:54.235 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:54.236 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:41:54.236 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:54.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.236 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:41:54.237 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:54.237 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:54.237 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:54.237 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:54.238 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:54.238 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:41:54.238 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:54.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.239 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:41:54.239 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:54.239 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:54.239 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:54.240 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:54.240 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:54.240 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:41:54.240 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:54.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.241 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:41:54.241 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:54.241 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:41:54.243 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:54.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:41:54.243 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:41:54.243 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:41:54.244 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:41:54.244 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:41:54.244 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:41:54.244 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:41:54.245 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:41:54.245 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:41:54.245 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.245 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:41:54.245 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:41:54.246 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:41:54.246 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:41:54.246 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:41:54.246 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:41:54.247 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6837/508489181": "2022-07-22T01:41:38.873743+0000", 2022-07-21T01:41:54.247 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6836/508489181": "2022-07-22T01:41:38.873743+0000" 2022-07-21T01:41:54.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.247 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:41:54.247 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:41:54.248 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:41:54.248 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:41:54.248 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:41:54.248 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:41:54.249 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:41:54.249 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:41:54.249 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:41:54.250 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:41:54.250 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:41:54.250 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:41:54.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.251 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:41:54.251 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:41:54.251 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:41:54.252 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:41:54.252 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:41:54.252 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:41:54.253 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:41:54.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:41:54.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:41:54.253 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:41:54.254 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:41:54.254 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:41:54.254 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:41:54.254 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:41:54.255 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:41:54.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053b30 msgr2=0x7fd8ec055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053b30 0x7fd8ec055ff0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fd90400a000 tx=0x7fd904008040).stop 2022-07-21T01:41:54.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 msgr2=0x7fd91005c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 0x7fd91005c170 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7fd900002f70 tx=0x7fd900015000).stop 2022-07-21T01:41:54.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 shutdown_connections 2022-07-21T01:41:54.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd8ec053b30 0x7fd8ec055ff0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9100b2710 0x7fd910156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9100b01a0 0x7fd91005c170 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 --2- 172.21.15.3:0/1497763111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9100afd60 0x7fd9100b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T01:41:54.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 >> 172.21.15.3:0/1497763111 conn(0x7fd910060310 msgr2=0x7fd91014a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:54.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 shutdown_connections 2022-07-21T01:41:54.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:53.963+0000 7fd90ffff700 1 -- 172.21.15.3:0/1497763111 wait complete. 2022-07-21T01:41:54.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:54.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 Processor -- start 2022-07-21T01:41:54.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 -- start start 2022-07-21T01:41:54.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340b2710 0x7f213414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21340a8c50 0x7f213414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21340a7250 con 0x7f21340b01a0 2022-07-21T01:41:54.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21340a7480 con 0x7f21340a8c50 2022-07-21T01:41:54.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21340a7720 con 0x7f21340b2710 2022-07-21T01:41:54.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f21337fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21340a8c50 0x7f213414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f21327fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f21340b01a0 0x7f21340a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58532/0 (socket says 172.21.15.3:58532) 2022-07-21T01:41:54.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f21327fc700 1 -- 172.21.15.3:0/3663166763 learned_addr learned my addr 172.21.15.3:0/3663166763 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:54.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2132ffd700 1 --2- 172.21.15.3:0/3663166763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340b2710 0x7f213414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f21337fe700 1 -- 172.21.15.3:0/3663166763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340b2710 msgr2=0x7f213414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:54.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f21337fe700 1 --2- 172.21.15.3:0/3663166763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340b2710 0x7f213414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f21337fe700 1 -- 172.21.15.3:0/3663166763 >> v1:172.21.15.3:6789/0 conn(0x7f21340b01a0 legacy=0x7f21340a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f21337fe700 1 -- 172.21.15.3:0/3663166763 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21340a79c0 con 0x7f21340a8c50 2022-07-21T01:41:54.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f21337fe700 1 --2- 172.21.15.3:0/3663166763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340a8c50 0x7f213414b4c0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f212c0032f0 tx=0x7f212c0091d0).ready entity=mon.0 client_cookie=b9cdbe8f7dff4fc2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:54.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2131ffb700 1 -- 172.21.15.3:0/3663166763 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f212c00b6c0 con 0x7f21340a8c50 2022-07-21T01:41:54.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2131ffb700 1 -- 172.21.15.3:0/3663166763 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c00b820 con 0x7f21340a8c50 2022-07-21T01:41:54.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2131ffb700 1 -- 172.21.15.3:0/3663166763 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f212c00b990 con 0x7f21340a8c50 2022-07-21T01:41:54.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 -- 172.21.15.3:0/3663166763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340a8c50 msgr2=0x7f213414b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 --2- 172.21.15.3:0/3663166763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340a8c50 0x7f213414b4c0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f212c0032f0 tx=0x7f212c0091d0).stop 2022-07-21T01:41:54.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 -- 172.21.15.3:0/3663166763 shutdown_connections 2022-07-21T01:41:54.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 --2- 172.21.15.3:0/3663166763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340a8c50 0x7f213414b4c0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 --2- 172.21.15.3:0/3663166763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340b2710 0x7f213414afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 -- 172.21.15.3:0/3663166763 >> 172.21.15.3:0/3663166763 conn(0x7f2134060310 msgr2=0x7f21340b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:54.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 -- 172.21.15.3:0/3663166763 shutdown_connections 2022-07-21T01:41:54.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.375+0000 7f2133fff700 1 -- 172.21.15.3:0/3663166763 wait complete. 2022-07-21T01:41:54.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 Processor -- start 2022-07-21T01:41:54.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 -- start start 2022-07-21T01:41:54.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340a8c50 0x7f21340b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21340b01a0 0x7f21340b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340b2710 0x7f21340655d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2134063980 con 0x7f21340b2710 2022-07-21T01:41:54.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21340634d0 con 0x7f21340b01a0 2022-07-21T01:41:54.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2134063be0 con 0x7f21340a8c50 2022-07-21T01:41:54.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340b2710 0x7f21340655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340b2710 0x7f21340655d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50260/0 (socket says 172.21.15.3:50260) 2022-07-21T01:41:54.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21337fe700 1 -- 172.21.15.3:0/377776835 learned_addr learned my addr 172.21.15.3:0/377776835 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:54.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2132ffd700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340a8c50 0x7f21340b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21327fc700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21340b01a0 0x7f21340b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21337fe700 1 -- 172.21.15.3:0/377776835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340a8c50 msgr2=0x7f21340b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21337fe700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340a8c50 0x7f21340b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21337fe700 1 -- 172.21.15.3:0/377776835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21340b01a0 msgr2=0x7f21340b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21337fe700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21340b01a0 0x7f21340b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21337fe700 1 -- 172.21.15.3:0/377776835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c009470 con 0x7f21340b2710 2022-07-21T01:41:54.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f21337fe700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340b2710 0x7f21340655d0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f212c0032f0 tx=0x7f212c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:54.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f212bfff700 1 -- 172.21.15.3:0/377776835 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f212c00b6c0 con 0x7f21340b2710 2022-07-21T01:41:54.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f212bfff700 1 -- 172.21.15.3:0/377776835 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c021630 con 0x7f21340b2710 2022-07-21T01:41:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21340b5270 con 0x7f21340b2710 2022-07-21T01:41:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f212bfff700 1 -- 172.21.15.3:0/377776835 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f212c021990 con 0x7f21340b2710 2022-07-21T01:41:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.379+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21341565b0 con 0x7f21340b2710 2022-07-21T01:41:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.383+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21340574e0 con 0x7f21340b2710 2022-07-21T01:41:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.383+0000 7f212bfff700 1 -- 172.21.15.3:0/377776835 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f212c00f070 con 0x7f21340b2710 2022-07-21T01:41:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.383+0000 7f212bfff700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2118053b00 0x7f2118055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.383+0000 7f212bfff700 1 -- 172.21.15.3:0/377776835 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f212c06d3f0 con 0x7f21340b2710 2022-07-21T01:41:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.383+0000 7f2132ffd700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2118053b00 0x7f2118055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.383+0000 7f2132ffd700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2118053b00 0x7f2118055fc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f211c00b000 tx=0x7f211c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.387+0000 7f212bfff700 1 -- 172.21.15.3:0/377776835 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f212c044220 con 0x7f21340b2710 2022-07-21T01:41:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2134069a70 con 0x7f21340b2710 2022-07-21T01:41:54.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f212bfff700 1 -- 172.21.15.3:0/377776835 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+1426 (secure 0 0 0) 0x7f212c015020 con 0x7f21340b2710 2022-07-21T01:41:54.559 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:54.559 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":65,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:53.510287+0000","modified":"2022-07-21T01:41:53.510352+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-07-21T01:41:54.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2118053b00 msgr2=0x7f2118055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2118053b00 0x7f2118055fc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f211c00b000 tx=0x7f211c006df0).stop 2022-07-21T01:41:54.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340b2710 msgr2=0x7f21340655d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340b2710 0x7f21340655d0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f212c0032f0 tx=0x7f212c016000).stop 2022-07-21T01:41:54.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 shutdown_connections 2022-07-21T01:41:54.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21340b2710 0x7f21340655d0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2118053b00 0x7f2118055fc0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21340b01a0 0x7f21340b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 --2- 172.21.15.3:0/377776835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21340a8c50 0x7f21340b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 >> 172.21.15.3:0/377776835 conn(0x7f2134060310 msgr2=0x7f213414cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:54.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 shutdown_connections 2022-07-21T01:41:54.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.555+0000 7f2133fff700 1 -- 172.21.15.3:0/377776835 wait complete. 2022-07-21T01:41:54.563 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 65 2022-07-21T01:41:54.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:41:54.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 Processor -- start 2022-07-21T01:41:54.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 -- start start 2022-07-21T01:41:54.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da00b29e0 0x7f7da00b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7da005c620 0x7f7da013c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7da005ca60 0x7f7da013cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da00b2dc0 con 0x7f7da005c620 2022-07-21T01:41:54.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da00b17f0 con 0x7f7da005ca60 2022-07-21T01:41:54.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da00b1a90 con 0x7f7da00b29e0 2022-07-21T01:41:54.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9dd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7da005c620 0x7f7da013c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9dd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7da005c620 0x7f7da013c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50264/0 (socket says 172.21.15.3:50264) 2022-07-21T01:41:54.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9dd9b700 1 -- 172.21.15.3:0/413766674 learned_addr learned my addr 172.21.15.3:0/413766674 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:54.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9e59c700 1 --2- 172.21.15.3:0/413766674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da00b29e0 0x7f7da00b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9ed9d700 1 --2- 172.21.15.3:0/413766674 >> v2:172.21.15.143:3300/0 conn(0x7f7da005ca60 0x7f7da013cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9dd9b700 1 -- 172.21.15.3:0/413766674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da00b29e0 msgr2=0x7f7da00b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:54.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9dd9b700 1 --2- 172.21.15.3:0/413766674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da00b29e0 0x7f7da00b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9dd9b700 1 -- 172.21.15.3:0/413766674 >> v2:172.21.15.143:3300/0 conn(0x7f7da005ca60 msgr2=0x7f7da013cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:54.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9dd9b700 1 --2- 172.21.15.3:0/413766674 >> v2:172.21.15.143:3300/0 conn(0x7f7da005ca60 0x7f7da013cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9dd9b700 1 -- 172.21.15.3:0/413766674 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da00b1d30 con 0x7f7da005c620 2022-07-21T01:41:54.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9dd9b700 1 --2- 172.21.15.3:0/413766674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005c620 0x7f7da013c870 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f7d980032f0 tx=0x7f7d98009290).ready entity=mon.0 client_cookie=edbfe5b36c070665 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:54.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9d59a700 1 -- 172.21.15.3:0/413766674 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d9800b680 con 0x7f7da005c620 2022-07-21T01:41:54.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9d59a700 1 -- 172.21.15.3:0/413766674 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9800b7e0 con 0x7f7da005c620 2022-07-21T01:41:54.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9d59a700 1 -- 172.21.15.3:0/413766674 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d9800b950 con 0x7f7da005c620 2022-07-21T01:41:54.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 -- 172.21.15.3:0/413766674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005c620 msgr2=0x7f7da013c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/413766674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005c620 0x7f7da013c870 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f7d980032f0 tx=0x7f7d98009290).stop 2022-07-21T01:41:54.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 -- 172.21.15.3:0/413766674 shutdown_connections 2022-07-21T01:41:54.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/413766674 >> v2:172.21.15.143:3300/0 conn(0x7f7da005ca60 0x7f7da013cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/413766674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005c620 0x7f7da013c870 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/413766674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da00b29e0 0x7f7da00b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 -- 172.21.15.3:0/413766674 >> 172.21.15.3:0/413766674 conn(0x7f7da00613c0 msgr2=0x7f7da0062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:54.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 -- 172.21.15.3:0/413766674 shutdown_connections 2022-07-21T01:41:54.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.679+0000 7f7d9f59e700 1 -- 172.21.15.3:0/413766674 wait complete. 2022-07-21T01:41:54.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 Processor -- start 2022-07-21T01:41:54.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 -- start start 2022-07-21T01:41:54.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da005c620 0x7f7da0142b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005ca60 0x7f7da01524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7da00b29e0 0x7f7da01529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da01437b0 con 0x7f7da005ca60 2022-07-21T01:41:54.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da01443e0 con 0x7f7da00b29e0 2022-07-21T01:41:54.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da0141620 con 0x7f7da005c620 2022-07-21T01:41:54.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005ca60 0x7f7da01524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da005c620 0x7f7da0142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005ca60 0x7f7da01524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50272/0 (socket says 172.21.15.3:50272) 2022-07-21T01:41:54.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9dd9b700 1 -- 172.21.15.3:0/3242972257 learned_addr learned my addr 172.21.15.3:0/3242972257 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:54.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9ed9d700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7da00b29e0 0x7f7da01529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9e59c700 1 -- 172.21.15.3:0/3242972257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7da00b29e0 msgr2=0x7f7da01529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9e59c700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7da00b29e0 0x7f7da01529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9e59c700 1 -- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005ca60 msgr2=0x7f7da01524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:54.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9e59c700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005ca60 0x7f7da01524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:54.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9e59c700 1 -- 172.21.15.3:0/3242972257 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98009530 con 0x7f7da005c620 2022-07-21T01:41:54.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9dd9b700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005ca60 0x7f7da01524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:41:54.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9e59c700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da005c620 0x7f7da0142b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7da01430f0 tx=0x7f7d90006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:54.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d977fe700 1 -- 172.21.15.3:0/3242972257 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d90016070 con 0x7f7da005c620 2022-07-21T01:41:54.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da0152f00 con 0x7f7da005c620 2022-07-21T01:41:54.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d977fe700 1 -- 172.21.15.3:0/3242972257 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d90011750 con 0x7f7da005c620 2022-07-21T01:41:54.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d977fe700 1 -- 172.21.15.3:0/3242972257 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d90011ab0 con 0x7f7da005c620 2022-07-21T01:41:54.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.683+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da01448b0 con 0x7f7da005c620 2022-07-21T01:41:54.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.687+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da0144a70 con 0x7f7da005c620 2022-07-21T01:41:54.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.687+0000 7f7d977fe700 1 -- 172.21.15.3:0/3242972257 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7d90011c10 con 0x7f7da005c620 2022-07-21T01:41:54.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.687+0000 7f7d977fe700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d84053950 0x7f7d84055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:54.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.687+0000 7f7d977fe700 1 -- 172.21.15.3:0/3242972257 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f7d9006c0a0 con 0x7f7da005c620 2022-07-21T01:41:54.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.687+0000 7f7d9dd9b700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d84053950 0x7f7d84055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:54.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.687+0000 7f7d9dd9b700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d84053950 0x7f7d84055e10 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f7d9801a000 tx=0x7f7d98017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:54.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.695+0000 7f7d977fe700 1 -- 172.21.15.3:0/3242972257 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7d90042d70 con 0x7f7da005c620 2022-07-21T01:41:54.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:54.859+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7da005f3d0 con 0x7f7da005c620 2022-07-21T01:41:55.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.655+0000 7f7d977fe700 1 -- 172.21.15.3:0/3242972257 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v66) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7d90046390 con 0x7f7da005c620 2022-07-21T01:41:55.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d84053950 msgr2=0x7f7d84055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:55.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d84053950 0x7f7d84055e10 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f7d9801a000 tx=0x7f7d98017000).stop 2022-07-21T01:41:55.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da005c620 msgr2=0x7f7da0142b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:55.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da005c620 0x7f7da0142b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7da01430f0 tx=0x7f7d90006df0).stop 2022-07-21T01:41:55.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 shutdown_connections 2022-07-21T01:41:55.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d84053950 0x7f7d84055e10 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7da00b29e0 0x7f7da01529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7da005ca60 0x7f7da01524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 --2- 172.21.15.3:0/3242972257 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7da005c620 0x7f7da0142b30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 >> 172.21.15.3:0/3242972257 conn(0x7f7da00613c0 msgr2=0x7f7da00acfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:55.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 shutdown_connections 2022-07-21T01:41:55.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.659+0000 7f7d9f59e700 1 -- 172.21.15.3:0/3242972257 wait complete. 2022-07-21T01:41:55.684 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:41:55.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:41:55.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aad27a700 1 Processor -- start 2022-07-21T01:41:55.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aad27a700 1 -- start start 2022-07-21T01:41:55.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aad27a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 0x7f7aa80abca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:55.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aad27a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7aa80b0470 0x7f7aa80ac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:55.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aad27a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa80adf10 con 0x7f7aa80b13f0 2022-07-21T01:41:55.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aad27a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa80ae1b0 con 0x7f7aa80b0470 2022-07-21T01:41:55.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aad27a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80a9520 con 0x7f7aa80b2710 2022-07-21T01:41:55.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aaca79700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7aa80b0470 0x7f7aa80ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:55.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aaca79700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f7aa80b0470 0x7f7aa80ac190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50282/0 (socket says 172.21.15.3:50282) 2022-07-21T01:41:55.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aaca79700 1 -- 172.21.15.3:0/2721358920 learned_addr learned my addr 172.21.15.3:0/2721358920 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aa7fff700 1 --2- 172.21.15.3:0/2721358920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 0x7f7aa80abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aaca79700 1 -- 172.21.15.3:0/2721358920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 msgr2=0x7f7aa80abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:55.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aaca79700 1 --2- 172.21.15.3:0/2721358920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 0x7f7aa80abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aaca79700 1 -- 172.21.15.3:0/2721358920 >> v1:172.21.15.143:6789/0 conn(0x7f7aa80b13f0 legacy=0x7f7aa80afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aaca79700 1 -- 172.21.15.3:0/2721358920 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80a99c0 con 0x7f7aa80b0470 2022-07-21T01:41:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.799+0000 7f7aaca79700 1 --2- 172.21.15.3:0/2721358920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 0x7f7aa80ac190 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0x7f7aa00032f0 tx=0x7f7aa0009290).ready entity=mon.0 client_cookie=e044de7b91c53a61 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:55.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aa6ffd700 1 -- 172.21.15.3:0/2721358920 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7aa000b8a0 con 0x7f7aa80b0470 2022-07-21T01:41:55.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aa6ffd700 1 -- 172.21.15.3:0/2721358920 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa000ba00 con 0x7f7aa80b0470 2022-07-21T01:41:55.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aa6ffd700 1 -- 172.21.15.3:0/2721358920 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7aa000bb70 con 0x7f7aa80b0470 2022-07-21T01:41:55.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 -- 172.21.15.3:0/2721358920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 msgr2=0x7f7aa80ac190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:55.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 --2- 172.21.15.3:0/2721358920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 0x7f7aa80ac190 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0x7f7aa00032f0 tx=0x7f7aa0009290).stop 2022-07-21T01:41:55.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 -- 172.21.15.3:0/2721358920 shutdown_connections 2022-07-21T01:41:55.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 --2- 172.21.15.3:0/2721358920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 0x7f7aa80ac190 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 --2- 172.21.15.3:0/2721358920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 0x7f7aa80abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 -- 172.21.15.3:0/2721358920 >> 172.21.15.3:0/2721358920 conn(0x7f7aa8060310 msgr2=0x7f7aa80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:55.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 -- 172.21.15.3:0/2721358920 shutdown_connections 2022-07-21T01:41:55.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 -- 172.21.15.3:0/2721358920 wait complete. 2022-07-21T01:41:55.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 Processor -- start 2022-07-21T01:41:55.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.803+0000 7f7aad27a700 1 -- start start 2022-07-21T01:41:55.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aad27a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 0x7f7aa80abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:55.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aad27a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aa80b13f0 0x7f7aa8155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:55.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aad27a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 0x7f7aa81456c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:55.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aad27a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80a26b0 con 0x7f7aa80b0470 2022-07-21T01:41:55.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aad27a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa8152390 con 0x7f7aa80b13f0 2022-07-21T01:41:55.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aad27a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa8152630 con 0x7f7aa80b2710 2022-07-21T01:41:55.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 0x7f7aa80abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:55.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 0x7f7aa80abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50284/0 (socket says 172.21.15.3:50284) 2022-07-21T01:41:55.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa7fff700 1 -- 172.21.15.3:0/2772591727 learned_addr learned my addr 172.21.15.3:0/2772591727 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:55.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aaca79700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 0x7f7aa81456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:55.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa77fe700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aa80b13f0 0x7f7aa8155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:55.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa7fff700 1 -- 172.21.15.3:0/2772591727 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 msgr2=0x7f7aa81456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:55.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa7fff700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 0x7f7aa81456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa7fff700 1 -- 172.21.15.3:0/2772591727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aa80b13f0 msgr2=0x7f7aa8155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa7fff700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aa80b13f0 0x7f7aa8155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa7fff700 1 -- 172.21.15.3:0/2772591727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa0009530 con 0x7f7aa80b0470 2022-07-21T01:41:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa7fff700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 0x7f7aa80abd90 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f7a980030b0 tx=0x7f7a98008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa57fa700 1 -- 172.21.15.3:0/2772591727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a98011780 con 0x7f7aa80b0470 2022-07-21T01:41:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa57fa700 1 -- 172.21.15.3:0/2772591727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a980118e0 con 0x7f7aa80b0470 2022-07-21T01:41:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa80a2950 con 0x7f7aa80b0470 2022-07-21T01:41:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aa57fa700 1 -- 172.21.15.3:0/2772591727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a98011c40 con 0x7f7aa80b0470 2022-07-21T01:41:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa80a2190 con 0x7f7aa80b0470 2022-07-21T01:41:55.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.807+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa81527f0 con 0x7f7aa80b0470 2022-07-21T01:41:55.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.811+0000 7f7aa57fa700 1 -- 172.21.15.3:0/2772591727 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7a98012420 con 0x7f7aa80b0470 2022-07-21T01:41:55.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.811+0000 7f7aa57fa700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7a88053b30 0x7f7a88055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:55.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.811+0000 7f7aa57fa700 1 -- 172.21.15.3:0/2772591727 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f7a9806bdb0 con 0x7f7aa80b0470 2022-07-21T01:41:55.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.811+0000 7f7aa77fe700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7a88053b30 0x7f7a88055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:55.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.811+0000 7f7aa77fe700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7a88053b30 0x7f7a88055ff0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f7a9c00d000 tx=0x7f7a9c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:55.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.815+0000 7f7aa57fa700 1 -- 172.21.15.3:0/2772591727 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7a98042ac0 con 0x7f7aa80b0470 2022-07-21T01:41:55.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.979+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aa80574e0 con 0x7f7aa80b0470 2022-07-21T01:41:55.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.979+0000 7f7aa57fa700 1 -- 172.21.15.3:0/2772591727 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+1426 (secure 0 0 0) 0x7f7a980460e0 con 0x7f7aa80b0470 2022-07-21T01:41:55.983 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:41:55.983 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":66,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:41:53.510287+0000","modified":"2022-07-21T01:41:55.647439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-07-21T01:41:55.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.979+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7a88053b30 msgr2=0x7f7a88055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:55.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.979+0000 7f7aad27a700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7a88053b30 0x7f7a88055ff0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f7a9c00d000 tx=0x7f7a9c007b90).stop 2022-07-21T01:41:55.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.979+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 msgr2=0x7f7aa80abd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:55.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.979+0000 7f7aad27a700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 0x7f7aa80abd90 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f7a980030b0 tx=0x7f7a98008040).stop 2022-07-21T01:41:55.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.983+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 shutdown_connections 2022-07-21T01:41:55.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.983+0000 7f7aad27a700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7aa80b2710 0x7f7aa81456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.983+0000 7f7aad27a700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7a88053b30 0x7f7a88055ff0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.983+0000 7f7aad27a700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aa80b13f0 0x7f7aa8155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.983+0000 7f7aad27a700 1 --2- 172.21.15.3:0/2772591727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aa80b0470 0x7f7aa80abd90 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:55.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.983+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 >> 172.21.15.3:0/2772591727 conn(0x7f7aa8060310 msgr2=0x7f7aa805ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:55.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.983+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 shutdown_connections 2022-07-21T01:41:55.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:55.983+0000 7f7aad27a700 1 -- 172.21.15.3:0/2772591727 wait complete. 2022-07-21T01:41:55.991 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 66 2022-07-21T01:41:55.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:41:56.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.103+0000 7f99f768c700 1 Processor -- start 2022-07-21T01:41:56.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.103+0000 7f99f768c700 1 -- start start 2022-07-21T01:41:56.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f99f01445d0 0x7f99f0144b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f00655b0 0x7f99f0144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f01454e0 con 0x7f99f0060290 2022-07-21T01:41:56.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f0065e10 con 0x7f99f01445d0 2022-07-21T01:41:56.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0060670 con 0x7f99f00655b0 2022-07-21T01:41:56.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f4c27700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f99f01445d0 0x7f99f0144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f5428700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f99f0060290 0x7f99f0064ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:60688/0 (socket says 172.21.15.3:60688) 2022-07-21T01:41:56.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f5428700 1 -- 172.21.15.3:0/2653049121 learned_addr learned my addr 172.21.15.3:0/2653049121 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:56.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f5c29700 1 --2- 172.21.15.3:0/2653049121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f00655b0 0x7f99f0144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f4c27700 1 -- 172.21.15.3:0/2653049121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f00655b0 msgr2=0x7f99f0144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:56.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f4c27700 1 --2- 172.21.15.3:0/2653049121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f00655b0 0x7f99f0144ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f4c27700 1 -- 172.21.15.3:0/2653049121 >> v1:172.21.15.143:6789/0 conn(0x7f99f0060290 legacy=0x7f99f0064ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f4c27700 1 -- 172.21.15.3:0/2653049121 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00b2540 con 0x7f99f01445d0 2022-07-21T01:41:56.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f4c27700 1 --2- 172.21.15.3:0/2653049121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01445d0 0x7f99f0144b00 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f99e0002f70 tx=0x7f99e0006df0).ready entity=mon.0 client_cookie=adedb9ce6ccf20ca server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:56.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99e7fff700 1 -- 172.21.15.3:0/2653049121 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99e000d570 con 0x7f99f01445d0 2022-07-21T01:41:56.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99e7fff700 1 -- 172.21.15.3:0/2653049121 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e0000e90 con 0x7f99f01445d0 2022-07-21T01:41:56.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99e7fff700 1 -- 172.21.15.3:0/2653049121 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99e000d8c0 con 0x7f99f01445d0 2022-07-21T01:41:56.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 -- 172.21.15.3:0/2653049121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01445d0 msgr2=0x7f99f0144b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 --2- 172.21.15.3:0/2653049121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01445d0 0x7f99f0144b00 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f99e0002f70 tx=0x7f99e0006df0).stop 2022-07-21T01:41:56.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 -- 172.21.15.3:0/2653049121 shutdown_connections 2022-07-21T01:41:56.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 --2- 172.21.15.3:0/2653049121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f00655b0 0x7f99f0144ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 --2- 172.21.15.3:0/2653049121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01445d0 0x7f99f0144b00 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 -- 172.21.15.3:0/2653049121 >> 172.21.15.3:0/2653049121 conn(0x7f99f005d0f0 msgr2=0x7f99f005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:56.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 -- 172.21.15.3:0/2653049121 shutdown_connections 2022-07-21T01:41:56.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.107+0000 7f99f768c700 1 -- 172.21.15.3:0/2653049121 wait complete. 2022-07-21T01:41:56.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 Processor -- start 2022-07-21T01:41:56.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 -- start start 2022-07-21T01:41:56.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f0060290 0x7f99f00b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99f00655b0 0x7f99f0150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01545c0 0x7f99f0150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f01539b0 con 0x7f99f01545c0 2022-07-21T01:41:56.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f013ec10 con 0x7f99f00655b0 2022-07-21T01:41:56.131 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0153be0 con 0x7f99f0060290 2022-07-21T01:41:56.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5c29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01545c0 0x7f99f0150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5c29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01545c0 0x7f99f0150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50302/0 (socket says 172.21.15.3:50302) 2022-07-21T01:41:56.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5c29700 1 -- 172.21.15.3:0/1680984786 learned_addr learned my addr 172.21.15.3:0/1680984786 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:56.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f4c27700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99f00655b0 0x7f99f0150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5428700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f0060290 0x7f99f00b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5c29700 1 -- 172.21.15.3:0/1680984786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f0060290 msgr2=0x7f99f00b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5c29700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f0060290 0x7f99f00b5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5c29700 1 -- 172.21.15.3:0/1680984786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99f00655b0 msgr2=0x7f99f0150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5c29700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99f00655b0 0x7f99f0150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5c29700 1 -- 172.21.15.3:0/1680984786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e000b180 con 0x7f99f01545c0 2022-07-21T01:41:56.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5428700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f0060290 0x7f99f00b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:56.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f5c29700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01545c0 0x7f99f0150950 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0x7f99e800c000 tx=0x7f99e80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:56.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99e67fc700 1 -- 172.21.15.3:0/1680984786 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99e800c610 con 0x7f99f01545c0 2022-07-21T01:41:56.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99e67fc700 1 -- 172.21.15.3:0/1680984786 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e800c770 con 0x7f99f01545c0 2022-07-21T01:41:56.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99e67fc700 1 -- 172.21.15.3:0/1680984786 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99e800cad0 con 0x7f99f01545c0 2022-07-21T01:41:56.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f0153e80 con 0x7f99f01545c0 2022-07-21T01:41:56.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.111+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f00abfa0 con 0x7f99f01545c0 2022-07-21T01:41:56.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.115+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f0154320 con 0x7f99f01545c0 2022-07-21T01:41:56.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.115+0000 7f99e67fc700 1 -- 172.21.15.3:0/1680984786 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f99e800cc30 con 0x7f99f01545c0 2022-07-21T01:41:56.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.115+0000 7f99e67fc700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f99d8053980 0x7f99d8055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.115+0000 7f99e67fc700 1 -- 172.21.15.3:0/1680984786 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f99e806c910 con 0x7f99f01545c0 2022-07-21T01:41:56.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.115+0000 7f99f5428700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f99d8053980 0x7f99d8055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.115+0000 7f99f5428700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f99d8053980 0x7f99d8055e40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f99ec00b000 tx=0x7f99ec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:56.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.119+0000 7f99e67fc700 1 -- 172.21.15.3:0/1680984786 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f99f0154320 con 0x7f99f01545c0 2022-07-21T01:41:56.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.279+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f99f0154320 con 0x7f99f01545c0 2022-07-21T01:41:56.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.667+0000 7f99e67fc700 1 -- 172.21.15.3:0/1680984786 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v67) v1 ==== 104+0+0 (secure 0 0 0) 0x7f99f0154320 con 0x7f99f01545c0 2022-07-21T01:41:56.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f99d8053980 msgr2=0x7f99d8055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f99d8053980 0x7f99d8055e40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f99ec00b000 tx=0x7f99ec009040).stop 2022-07-21T01:41:56.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01545c0 msgr2=0x7f99f0150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01545c0 0x7f99f0150950 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0x7f99e800c000 tx=0x7f99e80091d0).stop 2022-07-21T01:41:56.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 shutdown_connections 2022-07-21T01:41:56.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f01545c0 0x7f99f0150950 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f99d8053980 0x7f99d8055e40 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99f00655b0 0x7f99f0150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 --2- 172.21.15.3:0/1680984786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99f0060290 0x7f99f00b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 >> 172.21.15.3:0/1680984786 conn(0x7f99f005d0f0 msgr2=0x7f99f013abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:56.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 shutdown_connections 2022-07-21T01:41:56.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.671+0000 7f99f768c700 1 -- 172.21.15.3:0/1680984786 wait complete. 2022-07-21T01:41:56.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:41:56.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.807+0000 7feae9651700 1 Processor -- start 2022-07-21T01:41:56.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.807+0000 7feae9651700 1 -- start start 2022-07-21T01:41:56.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feae4060aa0 0x7feae40acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7feae40aa760 0x7feae413c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae40b6e70 con 0x7feae40b6a90 2022-07-21T01:41:56.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae4061370 con 0x7feae4060aa0 2022-07-21T01:41:56.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae4061610 con 0x7feae40aa760 2022-07-21T01:41:56.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feadbfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feae4060aa0 0x7feae40acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feadbfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7feae4060aa0 0x7feae40acee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50308/0 (socket says 172.21.15.3:50308) 2022-07-21T01:41:56.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feadbfff700 1 -- 172.21.15.3:0/837543473 learned_addr learned my addr 172.21.15.3:0/837543473 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:56.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae37fe700 1 --2- 172.21.15.3:0/837543473 >> v2:172.21.15.143:3300/0 conn(0x7feae40aa760 0x7feae413c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feadbfff700 1 -- 172.21.15.3:0/837543473 >> v2:172.21.15.143:3300/0 conn(0x7feae40aa760 msgr2=0x7feae413c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:56.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feadbfff700 1 --2- 172.21.15.3:0/837543473 >> v2:172.21.15.143:3300/0 conn(0x7feae40aa760 0x7feae413c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feadbfff700 1 -- 172.21.15.3:0/837543473 >> v1:172.21.15.143:6789/0 conn(0x7feae40b6a90 legacy=0x7feae40aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feadbfff700 1 -- 172.21.15.3:0/837543473 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae40618b0 con 0x7feae4060aa0 2022-07-21T01:41:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feadbfff700 1 --2- 172.21.15.3:0/837543473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae4060aa0 0x7feae40acee0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7feacc002f70 tx=0x7feacc006f40).ready entity=mon.0 client_cookie=15d2a62e19edeb55 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae27fc700 1 -- 172.21.15.3:0/837543473 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feacc00da80 con 0x7feae4060aa0 2022-07-21T01:41:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae27fc700 1 -- 172.21.15.3:0/837543473 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc000e90 con 0x7feae4060aa0 2022-07-21T01:41:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae27fc700 1 -- 172.21.15.3:0/837543473 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feacc00de10 con 0x7feae4060aa0 2022-07-21T01:41:56.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 -- 172.21.15.3:0/837543473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae4060aa0 msgr2=0x7feae40acee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 --2- 172.21.15.3:0/837543473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae4060aa0 0x7feae40acee0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7feacc002f70 tx=0x7feacc006f40).stop 2022-07-21T01:41:56.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 -- 172.21.15.3:0/837543473 shutdown_connections 2022-07-21T01:41:56.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 --2- 172.21.15.3:0/837543473 >> v2:172.21.15.143:3300/0 conn(0x7feae40aa760 0x7feae413c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 --2- 172.21.15.3:0/837543473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae4060aa0 0x7feae40acee0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 -- 172.21.15.3:0/837543473 >> 172.21.15.3:0/837543473 conn(0x7feae40b3610 msgr2=0x7feae40b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:56.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 -- 172.21.15.3:0/837543473 shutdown_connections 2022-07-21T01:41:56.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.811+0000 7feae9651700 1 -- 172.21.15.3:0/837543473 wait complete. 2022-07-21T01:41:56.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.815+0000 7feae9651700 1 Processor -- start 2022-07-21T01:41:56.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.815+0000 7feae9651700 1 -- start start 2022-07-21T01:41:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae9651700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4060aa0 0x7feae40670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae9651700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae40aa760 0x7feae40631c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae9651700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae40b6a90 0x7feae40636b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae9651700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae40b05c0 con 0x7feae40aa760 2022-07-21T01:41:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae9651700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae414e560 con 0x7feae4060aa0 2022-07-21T01:41:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae9651700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae414e800 con 0x7feae40b6a90 2022-07-21T01:41:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feadbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae40aa760 0x7feae40631c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4060aa0 0x7feae40670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feadbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae40aa760 0x7feae40631c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50314/0 (socket says 172.21.15.3:50314) 2022-07-21T01:41:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4060aa0 0x7feae40670e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:55760/0 (socket says 172.21.15.3:55760) 2022-07-21T01:41:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feadbfff700 1 -- 172.21.15.3:0/27717134 learned_addr learned my addr 172.21.15.3:0/27717134 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae37fe700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae40b6a90 0x7feae40636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feadbfff700 1 -- 172.21.15.3:0/27717134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae40b6a90 msgr2=0x7feae40636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feadbfff700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae40b6a90 0x7feae40636b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feadbfff700 1 -- 172.21.15.3:0/27717134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4060aa0 msgr2=0x7feae40670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feadbfff700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4060aa0 0x7feae40670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feadbfff700 1 -- 172.21.15.3:0/27717134 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc00b460 con 0x7feae40aa760 2022-07-21T01:41:56.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feadbfff700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae40aa760 0x7feae40631c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7feacc002f70 tx=0x7feae405ff40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:56.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae0ff9700 1 -- 172.21.15.3:0/27717134 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feacc010c40 con 0x7feae40aa760 2022-07-21T01:41:56.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae0ff9700 1 -- 172.21.15.3:0/27717134 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc010da0 con 0x7feae40aa760 2022-07-21T01:41:56.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae414eaa0 con 0x7feae40aa760 2022-07-21T01:41:56.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae0ff9700 1 -- 172.21.15.3:0/27717134 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feacc0245f0 con 0x7feae40aa760 2022-07-21T01:41:56.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae4066660 con 0x7feae40aa760 2022-07-21T01:41:56.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.819+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae4061b50 con 0x7feae40aa760 2022-07-21T01:41:56.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.823+0000 7feae0ff9700 1 -- 172.21.15.3:0/27717134 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feacc02b070 con 0x7feae40aa760 2022-07-21T01:41:56.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.823+0000 7feae0ff9700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feac80539e0 0x7feac8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:56.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.823+0000 7feae0ff9700 1 -- 172.21.15.3:0/27717134 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6506+0+0 (secure 0 0 0) 0x7feacc06f850 con 0x7feae40aa760 2022-07-21T01:41:56.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.827+0000 7feae2ffd700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feac80539e0 0x7feac8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:56.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.827+0000 7feae2ffd700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feac80539e0 0x7feac8055ea0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7fead400d000 tx=0x7fead4007b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:56.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.827+0000 7feae0ff9700 1 -- 172.21.15.3:0/27717134 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feacc0464b0 con 0x7feae40aa760 2022-07-21T01:41:56.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:56.951+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feae40574e0 con 0x7feae40aa760 2022-07-21T01:41:57.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.675+0000 7feae0ff9700 1 -- 172.21.15.3:0/27717134 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v134) v1 ==== 185+0+0 (secure 0 0 0) 0x7feacc049ad0 con 0x7feae40aa760 2022-07-21T01:41:57.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feac80539e0 msgr2=0x7feac8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:57.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feac80539e0 0x7feac8055ea0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7fead400d000 tx=0x7fead4007b50).stop 2022-07-21T01:41:57.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae40aa760 msgr2=0x7feae40631c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:57.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae40aa760 0x7feae40631c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7feacc002f70 tx=0x7feae405ff40).stop 2022-07-21T01:41:57.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 shutdown_connections 2022-07-21T01:41:57.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae40b6a90 0x7feae40636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:57.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feae40aa760 0x7feae40631c0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:57.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7feac80539e0 0x7feac8055ea0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:57.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 --2- 172.21.15.3:0/27717134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4060aa0 0x7feae40670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:57.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 >> 172.21.15.3:0/27717134 conn(0x7feae40b3610 msgr2=0x7feae405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:57.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 shutdown_connections 2022-07-21T01:41:57.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.679+0000 7feae9651700 1 -- 172.21.15.3:0/27717134 wait complete. 2022-07-21T01:41:57.686 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:41:57.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:41:57.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.803+0000 7f6a994f6700 1 Processor -- start 2022-07-21T01:41:57.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 -- start start 2022-07-21T01:41:57.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6a940ad750 0x7f6a94061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:57.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6a941405d0 0x7f6a94140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:57.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a940aee80 con 0x7f6a940614b0 2022-07-21T01:41:57.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a940b70b0 con 0x7f6a940ad750 2022-07-21T01:41:57.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a940af0b0 con 0x7f6a941405d0 2022-07-21T01:41:57.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a937fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6a940ad750 0x7f6a94061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:57.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a937fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6a940ad750 0x7f6a94061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50322/0 (socket says 172.21.15.3:50322) 2022-07-21T01:41:57.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a937fe700 1 -- 172.21.15.3:0/3059415957 learned_addr learned my addr 172.21.15.3:0/3059415957 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:57.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a93fff700 1 --1- 172.21.15.3:0/3059415957 >> v1:172.21.15.3:6789/0 conn(0x7f6a940614b0 0x7f6a940b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58600/0 (socket says 172.21.15.3:58600) 2022-07-21T01:41:57.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a98cf5700 1 --2- 172.21.15.3:0/3059415957 >> v2:172.21.15.143:3300/0 conn(0x7f6a941405d0 0x7f6a94140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:57.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a937fe700 1 -- 172.21.15.3:0/3059415957 >> v2:172.21.15.143:3300/0 conn(0x7f6a941405d0 msgr2=0x7f6a94140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:41:57.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a937fe700 1 --2- 172.21.15.3:0/3059415957 >> v2:172.21.15.143:3300/0 conn(0x7f6a941405d0 0x7f6a94140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:57.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a937fe700 1 -- 172.21.15.3:0/3059415957 >> v1:172.21.15.3:6789/0 conn(0x7f6a940614b0 legacy=0x7f6a940b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:57.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a937fe700 1 -- 172.21.15.3:0/3059415957 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a940af350 con 0x7f6a940ad750 2022-07-21T01:41:57.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a937fe700 1 --2- 172.21.15.3:0/3059415957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 0x7f6a94061890 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f6a84002f70 tx=0x7f6a84006eb0).ready entity=mon.0 client_cookie=f97c8277d4c4f0bc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:57.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a92ffd700 1 -- 172.21.15.3:0/3059415957 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8400d610 con 0x7f6a940ad750 2022-07-21T01:41:57.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a92ffd700 1 -- 172.21.15.3:0/3059415957 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a84000e90 con 0x7f6a940ad750 2022-07-21T01:41:57.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a92ffd700 1 -- 172.21.15.3:0/3059415957 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8400d9a0 con 0x7f6a940ad750 2022-07-21T01:41:57.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 -- 172.21.15.3:0/3059415957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 msgr2=0x7f6a94061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:57.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 --2- 172.21.15.3:0/3059415957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 0x7f6a94061890 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f6a84002f70 tx=0x7f6a84006eb0).stop 2022-07-21T01:41:57.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 -- 172.21.15.3:0/3059415957 shutdown_connections 2022-07-21T01:41:57.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 --2- 172.21.15.3:0/3059415957 >> v2:172.21.15.143:3300/0 conn(0x7f6a941405d0 0x7f6a94140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:57.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 --2- 172.21.15.3:0/3059415957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 0x7f6a94061890 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:57.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 -- 172.21.15.3:0/3059415957 >> 172.21.15.3:0/3059415957 conn(0x7f6a94066f70 msgr2=0x7f6a94067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:57.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 -- 172.21.15.3:0/3059415957 shutdown_connections 2022-07-21T01:41:57.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.807+0000 7f6a994f6700 1 -- 172.21.15.3:0/3059415957 wait complete. 2022-07-21T01:41:57.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 Processor -- start 2022-07-21T01:41:57.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 -- start start 2022-07-21T01:41:57.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a940614b0 0x7f6a940acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:57.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 0x7f6a94152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:57.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a941405d0 0x7f6a94153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:57.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940b2eb0 con 0x7f6a940ad750 2022-07-21T01:41:57.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940b2110 con 0x7f6a941405d0 2022-07-21T01:41:57.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940b23b0 con 0x7f6a940614b0 2022-07-21T01:41:57.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a937fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 0x7f6a94152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:57.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a937fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 0x7f6a94152ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50326/0 (socket says 172.21.15.3:50326) 2022-07-21T01:41:57.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a937fe700 1 -- 172.21.15.3:0/3020221607 learned_addr learned my addr 172.21.15.3:0/3020221607 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:57.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a98cf5700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a941405d0 0x7f6a94153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:57.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a93fff700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a940614b0 0x7f6a940acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:57.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a937fe700 1 -- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a940614b0 msgr2=0x7f6a940acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:57.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a937fe700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a940614b0 0x7f6a940acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:57.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a937fe700 1 -- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a941405d0 msgr2=0x7f6a94153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:57.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a937fe700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a941405d0 0x7f6a94153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:57.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a937fe700 1 -- 172.21.15.3:0/3020221607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8400b1e0 con 0x7f6a940ad750 2022-07-21T01:41:57.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a93fff700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a940614b0 0x7f6a940acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:57.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a98cf5700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a941405d0 0x7f6a94153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:41:57.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a937fe700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 0x7f6a94152ba0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f6a84002f70 tx=0x7f6a84015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:57.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a917fa700 1 -- 172.21.15.3:0/3020221607 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8400d290 con 0x7f6a940ad750 2022-07-21T01:41:57.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a917fa700 1 -- 172.21.15.3:0/3020221607 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8400d990 con 0x7f6a940ad750 2022-07-21T01:41:57.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a917fa700 1 -- 172.21.15.3:0/3020221607 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8401e460 con 0x7f6a940ad750 2022-07-21T01:41:57.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9409fa70 con 0x7f6a940ad750 2022-07-21T01:41:57.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.811+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a940b16a0 con 0x7f6a940ad750 2022-07-21T01:41:57.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.815+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a940b2570 con 0x7f6a940ad750 2022-07-21T01:41:57.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.815+0000 7f6a917fa700 1 -- 172.21.15.3:0/3020221607 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6a8401a040 con 0x7f6a940ad750 2022-07-21T01:41:57.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.815+0000 7f6a917fa700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053950 0x7f6a74055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:57.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.815+0000 7f6a917fa700 1 -- 172.21.15.3:0/3020221607 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f6a8406ceb0 con 0x7f6a940ad750 2022-07-21T01:41:57.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.815+0000 7f6a93fff700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053950 0x7f6a74055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:57.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.815+0000 7f6a93fff700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053950 0x7f6a74055e10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f6a8800b000 tx=0x7f6a880070e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:57.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.819+0000 7f6a917fa700 1 -- 172.21.15.3:0/3020221607 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6a84043e10 con 0x7f6a940ad750 2022-07-21T01:41:57.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:57.947+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6a940574e0 con 0x7f6a940ad750 2022-07-21T01:41:58.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.683+0000 7f6a917fa700 1 -- 172.21.15.3:0/3020221607 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v135) v1 ==== 173+0+0 (secure 0 0 0) 0x7f6a840473f0 con 0x7f6a940ad750 2022-07-21T01:41:58.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053950 msgr2=0x7f6a74055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:58.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053950 0x7f6a74055e10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f6a8800b000 tx=0x7f6a880070e0).stop 2022-07-21T01:41:58.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 msgr2=0x7f6a94152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:58.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 0x7f6a94152ba0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f6a84002f70 tx=0x7f6a84015000).stop 2022-07-21T01:41:58.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 shutdown_connections 2022-07-21T01:41:58.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a74053950 0x7f6a74055e10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a941405d0 0x7f6a94153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a940ad750 0x7f6a94152ba0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 --2- 172.21.15.3:0/3020221607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a940614b0 0x7f6a940acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 >> 172.21.15.3:0/3020221607 conn(0x7f6a94066f70 msgr2=0x7f6a94143970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:58.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 shutdown_connections 2022-07-21T01:41:58.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.691+0000 7f6a994f6700 1 -- 172.21.15.3:0/3020221607 wait complete. 2022-07-21T01:41:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:41:58.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:41:58.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2d538e700 1 Processor -- start 2022-07-21T01:41:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2d538e700 1 -- start start 2022-07-21T01:41:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2d538e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa2d00af6e0 0x7fa2d0147730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2d538e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d00b5a10 con 0x7fa2d00b07a0 2022-07-21T01:41:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2d538e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d00ac990 con 0x7fa2d00b5630 2022-07-21T01:41:58.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2d538e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d0145470 con 0x7fa2d00af6e0 2022-07-21T01:41:58.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2d4b8d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa2d00af6e0 0x7fa2d0147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:58.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2cf7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa2d00b07a0 0x7fa2d00acdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58616/0 (socket says 172.21.15.3:58616) 2022-07-21T01:41:58.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2cf7fe700 1 -- 172.21.15.3:0/2995293401 learned_addr learned my addr 172.21.15.3:0/2995293401 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:58.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.815+0000 7fa2ceffd700 1 -- 172.21.15.3:0/2995293401 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475527569 0 0) 0x7fa2d00b5a10 con 0x7fa2d00b07a0 2022-07-21T01:41:58.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2ceffd700 1 -- 172.21.15.3:0/2995293401 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2bc003260 con 0x7fa2d00b07a0 2022-07-21T01:41:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d4b8d700 1 -- 172.21.15.3:0/2995293401 >> v1:172.21.15.143:6789/0 conn(0x7fa2d00b5630 legacy=0x7fa2d0067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d4b8d700 1 -- 172.21.15.3:0/2995293401 >> v1:172.21.15.3:6789/0 conn(0x7fa2d00b07a0 legacy=0x7fa2d00acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d4b8d700 1 -- 172.21.15.3:0/2995293401 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d00b1bc0 con 0x7fa2d00af6e0 2022-07-21T01:41:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d4b8d700 1 --2- 172.21.15.3:0/2995293401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00af6e0 0x7fa2d0147730 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7fa2c8002f70 tx=0x7fa2c8006df0).ready entity=mon.0 client_cookie=f6f08c0f79383a6a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2ceffd700 1 -- 172.21.15.3:0/2995293401 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2c800d610 con 0x7fa2d00af6e0 2022-07-21T01:41:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2ceffd700 1 -- 172.21.15.3:0/2995293401 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c8000e90 con 0x7fa2d00af6e0 2022-07-21T01:41:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2ceffd700 1 -- 172.21.15.3:0/2995293401 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2c800d960 con 0x7fa2d00af6e0 2022-07-21T01:41:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d538e700 1 -- 172.21.15.3:0/2995293401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00af6e0 msgr2=0x7fa2d0147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d538e700 1 --2- 172.21.15.3:0/2995293401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00af6e0 0x7fa2d0147730 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7fa2c8002f70 tx=0x7fa2c8006df0).stop 2022-07-21T01:41:58.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d538e700 1 -- 172.21.15.3:0/2995293401 shutdown_connections 2022-07-21T01:41:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d538e700 1 --2- 172.21.15.3:0/2995293401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00af6e0 0x7fa2d0147730 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d538e700 1 -- 172.21.15.3:0/2995293401 >> 172.21.15.3:0/2995293401 conn(0x7fa2d00626f0 msgr2=0x7fa2d0062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d538e700 1 -- 172.21.15.3:0/2995293401 shutdown_connections 2022-07-21T01:41:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d538e700 1 -- 172.21.15.3:0/2995293401 wait complete. 2022-07-21T01:41:58.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d538e700 1 Processor -- start 2022-07-21T01:41:58.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.819+0000 7fa2d538e700 1 -- start start 2022-07-21T01:41:58.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d538e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2d00af6e0 0x7fa2d0067620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:58.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d538e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00b07a0 0x7fa2d0157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:58.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d538e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d00b5630 0x7fa2d0157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:58.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d538e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00ae180 con 0x7fa2d00b07a0 2022-07-21T01:41:58.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d538e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0064980 con 0x7fa2d00af6e0 2022-07-21T01:41:58.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d538e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00635d0 con 0x7fa2d00b5630 2022-07-21T01:41:58.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00b07a0 0x7fa2d0157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:58.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00b07a0 0x7fa2d0157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50342/0 (socket says 172.21.15.3:50342) 2022-07-21T01:41:58.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cf7fe700 1 -- 172.21.15.3:0/2141432111 learned_addr learned my addr 172.21.15.3:0/2141432111 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:58.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cffff700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2d00af6e0 0x7fa2d0067620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:58.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d4b8d700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d00b5630 0x7fa2d0157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:58.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cf7fe700 1 -- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d00b5630 msgr2=0x7fa2d0157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:58.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cf7fe700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d00b5630 0x7fa2d0157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cf7fe700 1 -- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2d00af6e0 msgr2=0x7fa2d0067620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:58.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cf7fe700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2d00af6e0 0x7fa2d0067620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cf7fe700 1 -- 172.21.15.3:0/2141432111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c800b180 con 0x7fa2d00b07a0 2022-07-21T01:41:58.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d4b8d700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d00b5630 0x7fa2d0157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:58.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cffff700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2d00af6e0 0x7fa2d0067620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:41:58.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cf7fe700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00b07a0 0x7fa2d0157090 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fa2c0001490 tx=0x7fa2c0007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:58.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cd7fa700 1 -- 172.21.15.3:0/2141432111 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2c0005ba0 con 0x7fa2d00b07a0 2022-07-21T01:41:58.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cd7fa700 1 -- 172.21.15.3:0/2141432111 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c0005d00 con 0x7fa2d00b07a0 2022-07-21T01:41:58.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d0147820 con 0x7fa2d00b07a0 2022-07-21T01:41:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2cd7fa700 1 -- 172.21.15.3:0/2141432111 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2c0007300 con 0x7fa2d00b07a0 2022-07-21T01:41:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d0063830 con 0x7fa2d00b07a0 2022-07-21T01:41:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.823+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d0063c70 con 0x7fa2d00b07a0 2022-07-21T01:41:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.827+0000 7fa2cd7fa700 1 -- 172.21.15.3:0/2141432111 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa2c0010030 con 0x7fa2d00b07a0 2022-07-21T01:41:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.827+0000 7fa2cd7fa700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa2b0053a70 0x7fa2b0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.827+0000 7fa2cd7fa700 1 -- 172.21.15.3:0/2141432111 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fa2c006bdd0 con 0x7fa2d00b07a0 2022-07-21T01:41:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.827+0000 7fa2cffff700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa2b0053a70 0x7fa2b0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:58.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.827+0000 7fa2cffff700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa2b0053a70 0x7fa2b0055f30 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fa2c4008000 tx=0x7fa2c4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:58.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.831+0000 7fa2cd7fa700 1 -- 172.21.15.3:0/2141432111 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa2c0042e70 con 0x7fa2d00b07a0 2022-07-21T01:41:58.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.955+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa2d00b1680 con 0x7fa2d00b07a0 2022-07-21T01:41:58.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2cd7fa700 1 -- 172.21.15.3:0/2141432111 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa2c0046490 con 0x7fa2d00b07a0 2022-07-21T01:41:58.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa2b0053a70 msgr2=0x7fa2b0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:58.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa2b0053a70 0x7fa2b0055f30 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fa2c4008000 tx=0x7fa2c4009040).stop 2022-07-21T01:41:58.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00b07a0 msgr2=0x7fa2d0157090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:58.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00b07a0 0x7fa2d0157090 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fa2c0001490 tx=0x7fa2c0007e50).stop 2022-07-21T01:41:58.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 shutdown_connections 2022-07-21T01:41:58.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d00b5630 0x7fa2d0157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2d00b07a0 0x7fa2d0157090 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa2b0053a70 0x7fa2b0055f30 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 --2- 172.21.15.3:0/2141432111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2d00af6e0 0x7fa2d0067620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:58.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 >> 172.21.15.3:0/2141432111 conn(0x7fa2d00626f0 msgr2=0x7fa2d0148d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:58.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 shutdown_connections 2022-07-21T01:41:58.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:58.963+0000 7fa2d538e700 1 -- 172.21.15.3:0/2141432111 wait complete. 2022-07-21T01:41:58.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-07-21T01:41:59.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.079+0000 7f73da612700 1 Processor -- start 2022-07-21T01:41:59.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.079+0000 7f73da612700 1 -- start start 2022-07-21T01:41:59.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.079+0000 7f73da612700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 0x7f73d40a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.079+0000 7f73da612700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d40a7250 con 0x7f73d40afd60 2022-07-21T01:41:59.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.079+0000 7f73da612700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d40a7480 con 0x7f73d40b2710 2022-07-21T01:41:59.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.079+0000 7f73da612700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d40a7720 con 0x7f73d40b01a0 2022-07-21T01:41:59.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.079+0000 7f73d9e11700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f73d40afd60 0x7f73d4151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58632/0 (socket says 172.21.15.3:58632) 2022-07-21T01:41:59.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.079+0000 7f73d9e11700 1 -- 172.21.15.3:0/540544339 learned_addr learned my addr 172.21.15.3:0/540544339 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:59.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.079+0000 7f73d8e0f700 1 --2- 172.21.15.3:0/540544339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 0x7f73d40a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73c3fff700 1 -- 172.21.15.3:0/540544339 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185177660 0 0) 0x7f73d40a7250 con 0x7f73d40afd60 2022-07-21T01:41:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73c3fff700 1 -- 172.21.15.3:0/540544339 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73bc003260 con 0x7f73d40afd60 2022-07-21T01:41:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73c3fff700 1 -- 172.21.15.3:0/540544339 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532949622 0 0) 0x7f73d40a7480 con 0x7f73d40b2710 2022-07-21T01:41:59.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73c3fff700 1 -- 172.21.15.3:0/540544339 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d40a7250 con 0x7f73d40b2710 2022-07-21T01:41:59.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73d8e0f700 1 -- 172.21.15.3:0/540544339 >> v1:172.21.15.143:6789/0 conn(0x7f73d40b2710 legacy=0x7f73d414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73d8e0f700 1 -- 172.21.15.3:0/540544339 >> v1:172.21.15.3:6789/0 conn(0x7f73d40afd60 legacy=0x7f73d4151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73d8e0f700 1 -- 172.21.15.3:0/540544339 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d40a79c0 con 0x7f73d40b01a0 2022-07-21T01:41:59.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73c3fff700 1 -- 172.21.15.3:0/540544339 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2891117358 0 0) 0x7f73bc003260 con 0x7f73d40afd60 2022-07-21T01:41:59.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73d8e0f700 1 --2- 172.21.15.3:0/540544339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 0x7f73d40a8540 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f73cc001b80 tx=0x7f73c4006df0).ready entity=mon.2 client_cookie=2be8d574d6894329 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:59.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73c3fff700 1 -- 172.21.15.3:0/540544339 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73cc0019e0 con 0x7f73d40b01a0 2022-07-21T01:41:59.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73c3fff700 1 -- 172.21.15.3:0/540544339 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c4000e90 con 0x7f73d40b01a0 2022-07-21T01:41:59.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73c3fff700 1 -- 172.21.15.3:0/540544339 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c4012460 con 0x7f73d40b01a0 2022-07-21T01:41:59.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73da612700 1 -- 172.21.15.3:0/540544339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 msgr2=0x7f73d40a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73da612700 1 --2- 172.21.15.3:0/540544339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 0x7f73d40a8540 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f73cc001b80 tx=0x7f73c4006df0).stop 2022-07-21T01:41:59.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73da612700 1 -- 172.21.15.3:0/540544339 shutdown_connections 2022-07-21T01:41:59.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73da612700 1 --2- 172.21.15.3:0/540544339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 0x7f73d40a8540 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73da612700 1 -- 172.21.15.3:0/540544339 >> 172.21.15.3:0/540544339 conn(0x7f73d4060310 msgr2=0x7f73d40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:59.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73da612700 1 -- 172.21.15.3:0/540544339 shutdown_connections 2022-07-21T01:41:59.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73da612700 1 -- 172.21.15.3:0/540544339 wait complete. 2022-07-21T01:41:59.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73da612700 1 Processor -- start 2022-07-21T01:41:59.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.083+0000 7f73da612700 1 -- start start 2022-07-21T01:41:59.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73da612700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d40afd60 0x7f73d40aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73da612700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 0x7f73d4156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73da612700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73d40b2710 0x7f73d4157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73da612700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d414baa0 con 0x7f73d40b2710 2022-07-21T01:41:59.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73da612700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d414d440 con 0x7f73d40afd60 2022-07-21T01:41:59.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73da612700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d414d6e0 con 0x7f73d40b01a0 2022-07-21T01:41:59.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9e11700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73d40b2710 0x7f73d4157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9e11700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73d40b2710 0x7f73d4157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50358/0 (socket says 172.21.15.3:50358) 2022-07-21T01:41:59.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9e11700 1 -- 172.21.15.3:0/4122278735 learned_addr learned my addr 172.21.15.3:0/4122278735 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:59.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d8e0f700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 0x7f73d4156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9610700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d40afd60 0x7f73d40aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9e11700 1 -- 172.21.15.3:0/4122278735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 msgr2=0x7f73d4156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9e11700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 0x7f73d4156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9e11700 1 -- 172.21.15.3:0/4122278735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d40afd60 msgr2=0x7f73d40aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9e11700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d40afd60 0x7f73d40aeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9e11700 1 -- 172.21.15.3:0/4122278735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c400b180 con 0x7f73d40b2710 2022-07-21T01:41:59.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73d9e11700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73d40b2710 0x7f73d4157220 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7f73cc001360 tx=0x7f73cc001ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:59.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73c27fc700 1 -- 172.21.15.3:0/4122278735 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73cc004ab0 con 0x7f73d40b2710 2022-07-21T01:41:59.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73c27fc700 1 -- 172.21.15.3:0/4122278735 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc004c10 con 0x7f73d40b2710 2022-07-21T01:41:59.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d414d980 con 0x7f73d40b2710 2022-07-21T01:41:59.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73c27fc700 1 -- 172.21.15.3:0/4122278735 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73cc007170 con 0x7f73d40b2710 2022-07-21T01:41:59.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d4157710 con 0x7f73d40b2710 2022-07-21T01:41:59.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d40a9f10 con 0x7f73d40b2710 2022-07-21T01:41:59.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.087+0000 7f73c27fc700 1 -- 172.21.15.3:0/4122278735 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f73cc0072d0 con 0x7f73d40b2710 2022-07-21T01:41:59.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.091+0000 7f73c27fc700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f73b4053a70 0x7f73b4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.091+0000 7f73c27fc700 1 -- 172.21.15.3:0/4122278735 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f73cc06bbe0 con 0x7f73d40b2710 2022-07-21T01:41:59.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.091+0000 7f73d9610700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f73b4053a70 0x7f73b4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.091+0000 7f73d9610700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f73b4053a70 0x7f73b4055f30 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f73d000d000 tx=0x7f73d0007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:59.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.095+0000 7f73c27fc700 1 -- 172.21.15.3:0/4122278735 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f73cc042b10 con 0x7f73d40b2710 2022-07-21T01:41:59.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.259+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7f73d4156a40 con 0x7f73d40b2710 2022-07-21T01:41:59.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.687+0000 7f73c27fc700 1 -- 172.21.15.3:0/4122278735 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v393) v1 ==== 122+0+0 (secure 0 0 0) 0x7f73cc046130 con 0x7f73d40b2710 2022-07-21T01:41:59.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f73b4053a70 msgr2=0x7f73b4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f73b4053a70 0x7f73b4055f30 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f73d000d000 tx=0x7f73d0007a20).stop 2022-07-21T01:41:59.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73d40b2710 msgr2=0x7f73d4157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73d40b2710 0x7f73d4157220 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7f73cc001360 tx=0x7f73cc001ca0).stop 2022-07-21T01:41:59.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 shutdown_connections 2022-07-21T01:41:59.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73d40b2710 0x7f73d4157220 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d40b01a0 0x7f73d4156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f73b4053a70 0x7f73b4055f30 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 --2- 172.21.15.3:0/4122278735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d40afd60 0x7f73d40aeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 >> 172.21.15.3:0/4122278735 conn(0x7f73d4060310 msgr2=0x7f73d4148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:59.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 shutdown_connections 2022-07-21T01:41:59.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.691+0000 7f73da612700 1 -- 172.21.15.3:0/4122278735 wait complete. 2022-07-21T01:41:59.710 INFO:tasks.cephfs_test_runner:test_fs_new (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-21T01:41:59.711 INFO:tasks.cephfs_test_runner:Starting test: test_getattr_caps (tasks.cephfs.test_misc.TestMisc) 2022-07-21T01:41:59.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-07-21T01:41:59.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.827+0000 7f1ada3da700 1 Processor -- start 2022-07-21T01:41:59.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.827+0000 7f1ada3da700 1 -- start start 2022-07-21T01:41:59.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.827+0000 7f1ada3da700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ad40b4550 0x7f1ad40b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.827+0000 7f1ada3da700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad4144240 con 0x7f1ad40b6af0 2022-07-21T01:41:59.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.827+0000 7f1ada3da700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad4143d00 con 0x7f1ad40b6710 2022-07-21T01:41:59.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.827+0000 7f1ada3da700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad4143fa0 con 0x7f1ad40b4550 2022-07-21T01:41:59.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1ad40b4550 0x7f1ad40b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad8977700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1ad40b6af0 0x7f1ad412bb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58646/0 (socket says 172.21.15.3:58646) 2022-07-21T01:41:59.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad8977700 1 -- 172.21.15.3:0/385445688 learned_addr learned my addr 172.21.15.3:0/385445688 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:41:59.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad37fe700 1 -- 172.21.15.3:0/385445688 >> v1:172.21.15.143:6789/0 conn(0x7f1ad40b6710 legacy=0x7f1ad4061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad2ffd700 1 -- 172.21.15.3:0/385445688 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584844319 0 0) 0x7f1ad4143d00 con 0x7f1ad40b6710 2022-07-21T01:41:59.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad37fe700 1 -- 172.21.15.3:0/385445688 >> v1:172.21.15.3:6789/0 conn(0x7f1ad40b6af0 legacy=0x7f1ad412bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad37fe700 1 -- 172.21.15.3:0/385445688 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad4061780 con 0x7f1ad40b4550 2022-07-21T01:41:59.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad37fe700 1 --2- 172.21.15.3:0/385445688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b4550 0x7f1ad40b5d80 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7f1acc001490 tx=0x7f1ac8006df0).ready entity=mon.0 client_cookie=6755bc8f3e476093 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:59.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad2ffd700 1 -- 172.21.15.3:0/385445688 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ac800d5c0 con 0x7f1ad40b4550 2022-07-21T01:41:59.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad2ffd700 1 -- 172.21.15.3:0/385445688 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac8000e90 con 0x7f1ad40b4550 2022-07-21T01:41:59.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ad2ffd700 1 -- 172.21.15.3:0/385445688 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ac800d910 con 0x7f1ad40b4550 2022-07-21T01:41:59.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ada3da700 1 -- 172.21.15.3:0/385445688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b4550 msgr2=0x7f1ad40b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ada3da700 1 --2- 172.21.15.3:0/385445688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b4550 0x7f1ad40b5d80 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7f1acc001490 tx=0x7f1ac8006df0).stop 2022-07-21T01:41:59.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ada3da700 1 -- 172.21.15.3:0/385445688 shutdown_connections 2022-07-21T01:41:59.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ada3da700 1 --2- 172.21.15.3:0/385445688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b4550 0x7f1ad40b5d80 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ada3da700 1 -- 172.21.15.3:0/385445688 >> 172.21.15.3:0/385445688 conn(0x7f1ad405c620 msgr2=0x7f1ad405ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:41:59.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ada3da700 1 -- 172.21.15.3:0/385445688 shutdown_connections 2022-07-21T01:41:59.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.831+0000 7f1ada3da700 1 -- 172.21.15.3:0/385445688 wait complete. 2022-07-21T01:41:59.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 Processor -- start 2022-07-21T01:41:59.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 -- start start 2022-07-21T01:41:59.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ad40b4550 0x7f1ad414a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ad40b6710 0x7f1ad40b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b6af0 0x7f1ad40af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad40adbe0 con 0x7f1ad40b6af0 2022-07-21T01:41:59.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad40b2c50 con 0x7f1ad40b6710 2022-07-21T01:41:59.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad40ade10 con 0x7f1ad40b4550 2022-07-21T01:41:59.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad8977700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b6af0 0x7f1ad40af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad8977700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b6af0 0x7f1ad40af7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50370/0 (socket says 172.21.15.3:50370) 2022-07-21T01:41:59.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad8977700 1 -- 172.21.15.3:0/960047749 learned_addr learned my addr 172.21.15.3:0/960047749 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:41:59.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad37fe700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ad40b6710 0x7f1ad40b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad3fff700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ad40b4550 0x7f1ad414a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad8977700 1 -- 172.21.15.3:0/960047749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ad40b4550 msgr2=0x7f1ad414a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad8977700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ad40b4550 0x7f1ad414a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad8977700 1 -- 172.21.15.3:0/960047749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ad40b6710 msgr2=0x7f1ad40b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:41:59.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad8977700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ad40b6710 0x7f1ad40b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:41:59.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad8977700 1 -- 172.21.15.3:0/960047749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac800b180 con 0x7f1ad40b6af0 2022-07-21T01:41:59.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad8977700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b6af0 0x7f1ad40af7e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7f1acc001360 tx=0x7f1acc007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:59.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad17fa700 1 -- 172.21.15.3:0/960047749 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1acc005ba0 con 0x7f1ad40b6af0 2022-07-21T01:41:59.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad17fa700 1 -- 172.21.15.3:0/960047749 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1acc005d00 con 0x7f1ad40b6af0 2022-07-21T01:41:59.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad40af340 con 0x7f1ad40b6af0 2022-07-21T01:41:59.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ad17fa700 1 -- 172.21.15.3:0/960047749 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1acc0072c0 con 0x7f1ad40b6af0 2022-07-21T01:41:59.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.835+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad40ae0b0 con 0x7f1ad40b6af0 2022-07-21T01:41:59.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.839+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad40af4d0 con 0x7f1ad40b6af0 2022-07-21T01:41:59.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.839+0000 7f1ad17fa700 1 -- 172.21.15.3:0/960047749 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1acc010070 con 0x7f1ad40b6af0 2022-07-21T01:41:59.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.839+0000 7f1ad17fa700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ab4053b00 0x7f1ab4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:41:59.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.839+0000 7f1ad3fff700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ab4053b00 0x7f1ab4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:41:59.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.839+0000 7f1ad17fa700 1 -- 172.21.15.3:0/960047749 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f1acc06b6b0 con 0x7f1ad40b6af0 2022-07-21T01:41:59.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.839+0000 7f1ad3fff700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ab4053b00 0x7f1ab4055fc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f1ac400d000 tx=0x7f1ac4007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:41:59.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:41:59.847+0000 7f1ad17fa700 1 -- 172.21.15.3:0/960047749 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1acc042750 con 0x7f1ad40b6af0 2022-07-21T01:42:00.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.015+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7f1ad4061a20 con 0x7f1ad40b6af0 2022-07-21T01:42:00.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.695+0000 7f1ad17fa700 1 -- 172.21.15.3:0/960047749 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v394) v1 ==== 131+0+0 (secure 0 0 0) 0x7f1acc045d70 con 0x7f1ad40b6af0 2022-07-21T01:42:00.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ab4053b00 msgr2=0x7f1ab4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:00.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ab4053b00 0x7f1ab4055fc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f1ac400d000 tx=0x7f1ac4007a00).stop 2022-07-21T01:42:00.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b6af0 msgr2=0x7f1ad40af7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:00.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b6af0 0x7f1ad40af7e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7f1acc001360 tx=0x7f1acc007f40).stop 2022-07-21T01:42:00.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 shutdown_connections 2022-07-21T01:42:00.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ad40b6af0 0x7f1ad40af7e0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1ab4053b00 0x7f1ab4055fc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ad40b6710 0x7f1ad40b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 --2- 172.21.15.3:0/960047749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ad40b4550 0x7f1ad414a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 >> 172.21.15.3:0/960047749 conn(0x7f1ad405c620 msgr2=0x7f1ad41288c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:00.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 shutdown_connections 2022-07-21T01:42:00.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.699+0000 7f1ada3da700 1 -- 172.21.15.3:0/960047749 wait complete. 2022-07-21T01:42:00.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:42:00.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.823+0000 7fceb62b1700 1 Processor -- start 2022-07-21T01:42:00.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.823+0000 7fceb62b1700 1 -- start start 2022-07-21T01:42:00.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb62b1700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fceb0066780 0x7fceb0066b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:00.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb62b1700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb005f8e0 con 0x7fceb0140e40 2022-07-21T01:42:00.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb62b1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb00b1630 con 0x7fceb00b1bc0 2022-07-21T01:42:00.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb62b1700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb005fb10 con 0x7fceb0066780 2022-07-21T01:42:00.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb52af700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fceb0066780 0x7fceb0066b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:00.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb52af700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fceb0066780 0x7fceb0066b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50376/0 (socket says 172.21.15.3:50376) 2022-07-21T01:42:00.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb5ab0700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fceb0140e40 0x7fceb013e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58660/0 (socket says 172.21.15.3:58660) 2022-07-21T01:42:00.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb52af700 1 -- 172.21.15.3:0/1934377510 learned_addr learned my addr 172.21.15.3:0/1934377510 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:00.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb52af700 1 -- 172.21.15.3:0/1934377510 >> v1:172.21.15.143:6789/0 conn(0x7fceb00b1bc0 legacy=0x7fceb005ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:00.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fce9ffff700 1 -- 172.21.15.3:0/1934377510 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568704536 0 0) 0x7fceb00b1630 con 0x7fceb00b1bc0 2022-07-21T01:42:00.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb52af700 1 -- 172.21.15.3:0/1934377510 >> v1:172.21.15.3:6789/0 conn(0x7fceb0140e40 legacy=0x7fceb013e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:00.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb52af700 1 -- 172.21.15.3:0/1934377510 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb00b18d0 con 0x7fceb0066780 2022-07-21T01:42:00.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fceb52af700 1 --2- 172.21.15.3:0/1934377510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 0x7fceb0066b60 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7fcea8001490 tx=0x7fceac006eb0).ready entity=mon.0 client_cookie=a9d07b18c543d9f5 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:00.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fce9ffff700 1 -- 172.21.15.3:0/1934377510 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceac00d610 con 0x7fceb0066780 2022-07-21T01:42:00.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fce9ffff700 1 -- 172.21.15.3:0/1934377510 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac000e90 con 0x7fceb0066780 2022-07-21T01:42:00.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.827+0000 7fce9ffff700 1 -- 172.21.15.3:0/1934377510 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceac00d960 con 0x7fceb0066780 2022-07-21T01:42:00.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.831+0000 7fceb62b1700 1 -- 172.21.15.3:0/1934377510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 msgr2=0x7fceb0066b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:00.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.831+0000 7fceb62b1700 1 --2- 172.21.15.3:0/1934377510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 0x7fceb0066b60 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7fcea8001490 tx=0x7fceac006eb0).stop 2022-07-21T01:42:00.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.831+0000 7fceb62b1700 1 -- 172.21.15.3:0/1934377510 shutdown_connections 2022-07-21T01:42:00.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.831+0000 7fceb62b1700 1 --2- 172.21.15.3:0/1934377510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 0x7fceb0066b60 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.831+0000 7fceb62b1700 1 -- 172.21.15.3:0/1934377510 >> 172.21.15.3:0/1934377510 conn(0x7fceb0063870 msgr2=0x7fceb0063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:00.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.831+0000 7fceb62b1700 1 -- 172.21.15.3:0/1934377510 shutdown_connections 2022-07-21T01:42:00.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.831+0000 7fceb62b1700 1 -- 172.21.15.3:0/1934377510 wait complete. 2022-07-21T01:42:00.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.831+0000 7fceb62b1700 1 Processor -- start 2022-07-21T01:42:00.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.831+0000 7fceb62b1700 1 -- start start 2022-07-21T01:42:00.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb62b1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 0x7fceb013e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:00.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb62b1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceb00b1bc0 0x7fceb0153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:00.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb62b1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceb0140e40 0x7fceb01543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:00.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb62b1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01463f0 con 0x7fceb0066780 2022-07-21T01:42:00.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb62b1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0146620 con 0x7fceb00b1bc0 2022-07-21T01:42:00.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb62b1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01468c0 con 0x7fceb0140e40 2022-07-21T01:42:00.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb52af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 0x7fceb013e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:00.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb52af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 0x7fceb013e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50382/0 (socket says 172.21.15.3:50382) 2022-07-21T01:42:00.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb52af700 1 -- 172.21.15.3:0/2532238558 learned_addr learned my addr 172.21.15.3:0/2532238558 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:00.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb4aae700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceb00b1bc0 0x7fceb0153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:00.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb5ab0700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceb0140e40 0x7fceb01543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:00.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb52af700 1 -- 172.21.15.3:0/2532238558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceb0140e40 msgr2=0x7fceb01543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:00.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb52af700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceb0140e40 0x7fceb01543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb52af700 1 -- 172.21.15.3:0/2532238558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceb00b1bc0 msgr2=0x7fceb0153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:00.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb52af700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceb00b1bc0 0x7fceb0153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb52af700 1 -- 172.21.15.3:0/2532238558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac00b1e0 con 0x7fceb0066780 2022-07-21T01:42:00.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb52af700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 0x7fceb013e420 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7fcea8001490 tx=0x7fceac015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:00.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fce9e7fc700 1 -- 172.21.15.3:0/2532238558 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceac011860 con 0x7fceb0066780 2022-07-21T01:42:00.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fce9e7fc700 1 -- 172.21.15.3:0/2532238558 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac0119c0 con 0x7fceb0066780 2022-07-21T01:42:00.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb0153a20 con 0x7fceb0066780 2022-07-21T01:42:00.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fce9e7fc700 1 -- 172.21.15.3:0/2532238558 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceac011d20 con 0x7fceb0066780 2022-07-21T01:42:00.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.835+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb0145830 con 0x7fceb0066780 2022-07-21T01:42:00.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.839+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb0147250 con 0x7fceb0066780 2022-07-21T01:42:00.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.839+0000 7fce9e7fc700 1 -- 172.21.15.3:0/2532238558 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fceac025030 con 0x7fceb0066780 2022-07-21T01:42:00.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.839+0000 7fce9e7fc700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fce90053a70 0x7fce90055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:00.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.839+0000 7fce9e7fc700 1 -- 172.21.15.3:0/2532238558 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fceac06cfa0 con 0x7fceb0066780 2022-07-21T01:42:00.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.839+0000 7fceb4aae700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fce90053a70 0x7fce90055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:00.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.839+0000 7fceb4aae700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fce90053a70 0x7fce90055f30 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fcea000d000 tx=0x7fcea0007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:00.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.843+0000 7fce9e7fc700 1 -- 172.21.15.3:0/2532238558 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fceb0147250 con 0x7fceb0066780 2022-07-21T01:42:00.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.971+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fceb0147250 con 0x7fceb0066780 2022-07-21T01:42:00.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.979+0000 7fce9e7fc700 1 -- 172.21.15.3:0/2532238558 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fceb0147250 con 0x7fceb0066780 2022-07-21T01:42:00.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fce90053a70 msgr2=0x7fce90055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:00.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fce90053a70 0x7fce90055f30 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fcea000d000 tx=0x7fcea0007a00).stop 2022-07-21T01:42:00.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 msgr2=0x7fceb013e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:00.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 0x7fceb013e420 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7fcea8001490 tx=0x7fceac015000).stop 2022-07-21T01:42:00.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 shutdown_connections 2022-07-21T01:42:00.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceb0140e40 0x7fceb01543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fce90053a70 0x7fce90055f30 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceb00b1bc0 0x7fceb0153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 --2- 172.21.15.3:0/2532238558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fceb0066780 0x7fceb013e420 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:00.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 >> 172.21.15.3:0/2532238558 conn(0x7fceb0063870 msgr2=0x7fceb00b58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:00.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 shutdown_connections 2022-07-21T01:42:00.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:00.983+0000 7fceb62b1700 1 -- 172.21.15.3:0/2532238558 wait complete. 2022-07-21T01:42:01.003 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:42:01.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:01.021 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:01.044 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:01.065 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:01.086 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:42:01.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:01.087 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:01.088 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:01.095 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:42:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:42:01.097 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:42:01.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:01.098 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:01.098 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:01.133 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:42:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:42:01.136 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:01.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:42:01.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f091af88700 1 Processor -- start 2022-07-21T01:42:01.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f091af88700 1 -- start start 2022-07-21T01:42:01.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f091af88700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b6420 0x7f0914061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f091af88700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f09140b41f0 0x7f0914061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f091af88700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09140b60e0 con 0x7f0914061fe0 2022-07-21T01:42:01.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f091af88700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09140b2100 con 0x7f09140b41f0 2022-07-21T01:42:01.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f091af88700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09140b23a0 con 0x7f09140b6420 2022-07-21T01:42:01.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f091a787700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0914061fe0 0x7f09141465c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58672/0 (socket says 172.21.15.3:58672) 2022-07-21T01:42:01.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f091a787700 1 -- 172.21.15.3:0/3542567993 learned_addr learned my addr 172.21.15.3:0/3542567993 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:01.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f0919785700 1 --2- 172.21.15.3:0/3542567993 >> v2:172.21.15.3:3300/0 conn(0x7f09140b41f0 0x7f0914061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f0919f86700 1 --2- 172.21.15.3:0/3542567993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b6420 0x7f0914061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f0918f84700 1 -- 172.21.15.3:0/3542567993 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307677226 0 0) 0x7f090c001110 con 0x7f0914061fe0 2022-07-21T01:42:01.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f0918f84700 1 -- 172.21.15.3:0/3542567993 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0900003260 con 0x7f0914061fe0 2022-07-21T01:42:01.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f0919785700 1 -- 172.21.15.3:0/3542567993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b6420 msgr2=0x7f0914061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:01.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f0919785700 1 --2- 172.21.15.3:0/3542567993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b6420 0x7f0914061600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f0919785700 1 -- 172.21.15.3:0/3542567993 >> v1:172.21.15.3:6789/0 conn(0x7f0914061fe0 legacy=0x7f09141465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f0919785700 1 -- 172.21.15.3:0/3542567993 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09140b2640 con 0x7f09140b41f0 2022-07-21T01:42:01.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.247+0000 7f0919f86700 1 --2- 172.21.15.3:0/3542567993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b6420 0x7f0914061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:01.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f0919785700 1 --2- 172.21.15.3:0/3542567993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b41f0 0x7f0914061af0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f09100032f0 tx=0x7f09100091d0).ready entity=mon.0 client_cookie=dd545f62a736af23 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:01.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f0918f84700 1 -- 172.21.15.3:0/3542567993 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f091000b6c0 con 0x7f09140b41f0 2022-07-21T01:42:01.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f0918f84700 1 -- 172.21.15.3:0/3542567993 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091000b820 con 0x7f09140b41f0 2022-07-21T01:42:01.261 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f0918f84700 1 -- 172.21.15.3:0/3542567993 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f091000b990 con 0x7f09140b41f0 2022-07-21T01:42:01.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f091af88700 1 -- 172.21.15.3:0/3542567993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b41f0 msgr2=0x7f0914061af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f091af88700 1 --2- 172.21.15.3:0/3542567993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b41f0 0x7f0914061af0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f09100032f0 tx=0x7f09100091d0).stop 2022-07-21T01:42:01.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f091af88700 1 -- 172.21.15.3:0/3542567993 shutdown_connections 2022-07-21T01:42:01.265 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f091af88700 1 --2- 172.21.15.3:0/3542567993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b41f0 0x7f0914061af0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f091af88700 1 --2- 172.21.15.3:0/3542567993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b6420 0x7f0914061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.266 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f091af88700 1 -- 172.21.15.3:0/3542567993 >> 172.21.15.3:0/3542567993 conn(0x7f091405c620 msgr2=0x7f091405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:01.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f091af88700 1 -- 172.21.15.3:0/3542567993 shutdown_connections 2022-07-21T01:42:01.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.251+0000 7f091af88700 1 -- 172.21.15.3:0/3542567993 wait complete. 2022-07-21T01:42:01.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091af88700 1 Processor -- start 2022-07-21T01:42:01.267 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091af88700 1 -- start start 2022-07-21T01:42:01.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091af88700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0914061fe0 0x7f0914145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091af88700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b41f0 0x7f0914156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091af88700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b6420 0x7f0914152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091af88700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09140b6d90 con 0x7f09140b6420 2022-07-21T01:42:01.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091af88700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0914156e00 con 0x7f0914061fe0 2022-07-21T01:42:01.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091af88700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09140b0500 con 0x7f09140b41f0 2022-07-21T01:42:01.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091a787700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b6420 0x7f0914152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091a787700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b6420 0x7f0914152ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50398/0 (socket says 172.21.15.3:50398) 2022-07-21T01:42:01.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091a787700 1 -- 172.21.15.3:0/1463109283 learned_addr learned my addr 172.21.15.3:0/1463109283 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:01.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f0919785700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b41f0 0x7f0914156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f0919f86700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0914061fe0 0x7f0914145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091a787700 1 -- 172.21.15.3:0/1463109283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b41f0 msgr2=0x7f0914156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091a787700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b41f0 0x7f0914156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091a787700 1 -- 172.21.15.3:0/1463109283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0914061fe0 msgr2=0x7f0914145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091a787700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0914061fe0 0x7f0914145a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.255+0000 7f091a787700 1 -- 172.21.15.3:0/1463109283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0910009470 con 0x7f09140b6420 2022-07-21T01:42:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f091a787700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b6420 0x7f0914152ee0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f090c0012a0 tx=0x7f090c007880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f090affd700 1 -- 172.21.15.3:0/1463109283 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f090c00ce00 con 0x7f09140b6420 2022-07-21T01:42:01.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f090affd700 1 -- 172.21.15.3:0/1463109283 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090c0070d0 con 0x7f09140b6420 2022-07-21T01:42:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09140b07a0 con 0x7f09140b6420 2022-07-21T01:42:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f090affd700 1 -- 172.21.15.3:0/1463109283 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f090c007430 con 0x7f09140b6420 2022-07-21T01:42:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09140b0a40 con 0x7f09140b6420 2022-07-21T01:42:01.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09140b28e0 con 0x7f09140b6420 2022-07-21T01:42:01.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f090affd700 1 -- 172.21.15.3:0/1463109283 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f090c010070 con 0x7f09140b6420 2022-07-21T01:42:01.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f090affd700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08f40539e0 0x7f08f4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f090affd700 1 -- 172.21.15.3:0/1463109283 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f090c06b250 con 0x7f09140b6420 2022-07-21T01:42:01.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f0919f86700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08f40539e0 0x7f08f4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.259+0000 7f0919f86700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08f40539e0 0x7f08f4055ea0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f090400c000 tx=0x7f0904008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:01.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.263+0000 7f090affd700 1 -- 172.21.15.3:0/1463109283 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f090c0421e0 con 0x7f09140b6420 2022-07-21T01:42:01.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.387+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f09140b60e0 con 0x7f09140b6420 2022-07-21T01:42:01.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.387+0000 7f090affd700 1 -- 172.21.15.3:0/1463109283 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v135) v1 ==== 81+0+29225 (secure 0 0 0) 0x7f09140b60e0 con 0x7f09140b6420 2022-07-21T01:42:01.393 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:01.393 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:42:01.394 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 135, 2022-07-21T01:42:01.394 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:42:01.394 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:42:01.394 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:41:58.674016+0000", 2022-07-21T01:42:01.395 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:01.395 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:01.395 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:42:01.396 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:42:01.396 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:42:01.397 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:42:01.397 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:42:01.397 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:42:01.398 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:42:01.399 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:01.399 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:42:01.402 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:42:01.402 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:42:01.402 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:42:01.402 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:42:01.403 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 24, 2022-07-21T01:42:01.403 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:42:01.403 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:42:01.403 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:42:01.404 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:42:01.404 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:42:01.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.404 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:42:01.405 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:42:01.405 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:42:01.405 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:01.406 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:01.406 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:01.406 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:01.406 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:01.407 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:01.407 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:01.407 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:01.407 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:42:01.408 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:42:01.408 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:42:01.408 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:42:01.409 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:42:01.409 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:01.409 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:01.410 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:01.410 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:01.410 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:01.411 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:01.411 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:01.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.412 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:42:01.412 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:01.412 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:01.415 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:01.415 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:01.415 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:01.416 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:01.416 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:01.416 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:01.416 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:01.417 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:01.417 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:01.417 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:01.418 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:01.418 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:01.418 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:01.418 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:01.419 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:01.419 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:01.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.419 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:01.420 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:01.420 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:01.420 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:01.421 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:01.421 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:01.421 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:01.421 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:01.422 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:01.422 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:01.422 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:01.422 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:42:01.423 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:42:01.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.423 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:01.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:42:01.424 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.425 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:42:01.425 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:42:01.425 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:42:01.425 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:01.426 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:01.426 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:01.426 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:01.426 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:01.427 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:01.427 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:01.427 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:01.427 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:42:01.428 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:42:01.428 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:42:01.428 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:42:01.429 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:42:01.429 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:01.429 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:01.429 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:01.430 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:01.430 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:01.430 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:01.430 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:01.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.431 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:42:01.431 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:01.432 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:01.432 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:01.432 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:01.432 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:01.433 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:01.433 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:01.433 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:01.433 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:01.434 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:01.434 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:01.434 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:01.435 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:01.435 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:01.435 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:01.435 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:01.436 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:01.436 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:01.436 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:01.437 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:01.437 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:01.437 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:01.437 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:01.438 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:01.438 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:01.438 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:01.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.439 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:01.439 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:01.439 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:01.440 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:01.440 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:01.440 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:01.441 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:01.441 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:01.441 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:01.442 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:01.442 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:01.442 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:42:01.442 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:01.443 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:42:01.443 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.443 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.443 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:01.444 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:42:01.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:01.445 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:42:01.446 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:01.446 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:01.446 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:01.446 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:01.447 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:01.447 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:01.447 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:01.447 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:42:01.448 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:01.448 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:01.448 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:01.448 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.449 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.449 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:42:01.449 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:01.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.450 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:42:01.451 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:01.451 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.451 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.452 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:01.452 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.452 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.453 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:42:01.453 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:01.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.454 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:42:01.454 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:01.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.455 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.455 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:01.455 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.456 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:42:01.456 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:01.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.457 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:42:01.458 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:01.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.459 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:01.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:42:01.460 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:01.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.461 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.461 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:42:01.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:01.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.462 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.462 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:42:01.463 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:42:01.463 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:42:01.464 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:42:01.464 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:01.464 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:01.465 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:01.465 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:01.467 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:42:01.468 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:01.468 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:01.469 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:01.469 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:01.470 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:01.470 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:01.471 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:01.471 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:42:01.472 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:01.472 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:01.473 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:01.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:42:01.475 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:01.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.477 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:42:01.478 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:01.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.479 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.480 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:01.480 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.481 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.482 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:42:01.482 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:01.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.484 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.484 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:42:01.485 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:01.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.485 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.486 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:01.486 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.487 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:42:01.487 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:01.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.487 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.488 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:42:01.488 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:01.488 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.488 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.489 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:01.489 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:42:01.491 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:01.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.492 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:42:01.493 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:01.493 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.493 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.494 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:42:01.495 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:42:01.495 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:42:01.495 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:42:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:01.496 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:01.497 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.498 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:42:01.499 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:01.500 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:01.501 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 126, 2022-07-21T01:42:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:01.502 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:42:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:42:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:01.505 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:01.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:42:01.507 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:42:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.510 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:42:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:42:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.512 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:42:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.515 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.515 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:42:01.515 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:01.516 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.516 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.516 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:42:01.517 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:42:01.517 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:42:01.517 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:42:01.517 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:01.517 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:01.518 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:01.518 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.519 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:01.519 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:42:01.519 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:01.519 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:01.520 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:01.520 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:01.520 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:01.520 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:01.521 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:01.521 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:42:01.521 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:01.522 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:01.522 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:01.523 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.523 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.524 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:42:01.524 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:01.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.525 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.525 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:42:01.525 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:01.525 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.526 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.526 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:01.526 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.527 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:42:01.527 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:01.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.528 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:42:01.528 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:01.529 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.529 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.529 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:01.530 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.531 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.531 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:42:01.532 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:01.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:42:01.533 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:01.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.533 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.534 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:01.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.535 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.535 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:42:01.535 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:01.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.536 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:42:01.536 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:01.537 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.537 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.537 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:42:01.538 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:42:01.538 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:42:01.538 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:42:01.538 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:01.538 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:01.539 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:01.539 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.540 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:01.540 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:42:01.540 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:01.540 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:01.540 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:01.541 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:01.541 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:01.541 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:01.541 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:01.542 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 126, 2022-07-21T01:42:01.542 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:01.542 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:01.542 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:01.543 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.543 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:42:01.544 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:01.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.544 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.545 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:42:01.545 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:01.545 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.546 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.547 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:01.547 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.547 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.548 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:42:01.548 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:01.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.549 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:42:01.549 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:01.549 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.549 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.550 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:01.550 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.551 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:42:01.551 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:01.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.552 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.552 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:42:01.552 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:01.552 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.552 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.553 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:01.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.554 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:42:01.555 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:01.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.555 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.556 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:42:01.556 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:01.556 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.556 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.557 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:42:01.557 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:42:01.557 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:42:01.557 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:42:01.558 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:01.558 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:01.558 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:01.558 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.559 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:01.559 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:42:01.559 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:01.560 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:01.560 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:01.560 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:01.560 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:01.561 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:01.561 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:01.561 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:42:01.562 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:01.562 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:01.562 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:01.563 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.563 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.563 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:42:01.564 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:01.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.564 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.565 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:42:01.565 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:01.565 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.565 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.566 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:01.566 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.567 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.567 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:42:01.568 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:01.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.568 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.569 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:42:01.569 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:01.569 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.570 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.570 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:01.571 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.571 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.571 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:42:01.572 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:01.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.573 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:42:01.573 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:01.573 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.573 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.574 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:01.574 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.575 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:42:01.575 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:01.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.576 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.576 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:42:01.576 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:01.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.577 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.577 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:42:01.577 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:42:01.578 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:42:01.578 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:42:01.578 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:01.578 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:01.578 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:01.579 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.579 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:01.580 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:42:01.580 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:01.580 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:01.580 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:01.581 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:01.581 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:01.581 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:01.581 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:01.581 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:42:01.582 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:01.582 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:01.582 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:01.582 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.583 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.583 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:42:01.583 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:01.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.584 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.584 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:42:01.584 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:01.585 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.585 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.585 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:01.586 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.586 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.586 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:42:01.586 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:01.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.587 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.587 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:42:01.588 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:01.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.588 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.589 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:01.589 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.589 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:42:01.590 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:01.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.590 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.591 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:42:01.591 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:01.591 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.591 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.592 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:01.592 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.592 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.593 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:42:01.593 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:01.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.594 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:42:01.594 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:01.594 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.594 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.595 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:42:01.595 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:42:01.595 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:42:01.596 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:42:01.596 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:01.596 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:01.596 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:01.597 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.597 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:01.598 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:42:01.598 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:01.598 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:01.598 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:01.599 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:01.599 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:01.599 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:01.599 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:01.599 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:42:01.600 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:01.600 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:01.600 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:01.600 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.601 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.601 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:42:01.601 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:01.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.602 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.602 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:42:01.603 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:01.603 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.603 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.604 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:01.604 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.604 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.605 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:42:01.605 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:01.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.606 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.606 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:42:01.606 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:01.606 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.606 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.607 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:01.607 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.608 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.608 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:42:01.608 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:01.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.609 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:42:01.609 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:01.610 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.610 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.610 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:01.610 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:01.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.611 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:01.611 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:42:01.611 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:01.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.612 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:01.612 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:42:01.613 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:01.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.613 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.614 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:42:01.614 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:42:01.614 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:42:01.614 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:42:01.615 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:01.615 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:01.615 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:01.615 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:01.616 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.616 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:01.616 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:42:01.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.617 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:01.617 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:01.617 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:01.617 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:01.618 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:01.618 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:01.618 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:42:01.618 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:01.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.619 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:01.619 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:01.620 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:01.620 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:01.620 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:01.620 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:01.621 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:42:01.621 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:01.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.622 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:01.622 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:01.622 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:01.623 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:01.623 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:01.623 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:01.623 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:42:01.624 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:01.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.624 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:01.625 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:01.625 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:01.625 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:01.625 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:01.626 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:01.626 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:42:01.626 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:01.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.627 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:01.627 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:01.627 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:01.628 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:01.628 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:01.628 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:01.628 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:42:01.629 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:01.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.629 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:01.630 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:01.630 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:01.630 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:01.630 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:01.631 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:01.631 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:42:01.631 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:01.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.632 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:01.632 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:01.632 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:01.632 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:01.633 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:01.633 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:01.633 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:42:01.633 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:01.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:01.634 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:01.634 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:01.635 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:01.635 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:01.635 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:01.635 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:01.635 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:42:01.636 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:01.636 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.636 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:01.636 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:42:01.637 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:42:01.637 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:42:01.637 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:42:01.637 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:42:01.637 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6837/508489181": "2022-07-22T01:41:38.873743+0000", 2022-07-21T01:42:01.638 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6836/508489181": "2022-07-22T01:41:38.873743+0000" 2022-07-21T01:42:01.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.638 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:42:01.638 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:42:01.639 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:42:01.639 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:01.639 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:42:01.639 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:42:01.639 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:01.640 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:42:01.640 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:01.640 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:01.640 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:42:01.641 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:42:01.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.641 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:42:01.641 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:01.641 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:01.642 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:42:01.642 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:01.642 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:01.642 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:42:01.643 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:01.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:01.643 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:42:01.643 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:42:01.644 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:42:01.644 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:42:01.644 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:42:01.644 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:42:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08f40539e0 msgr2=0x7f08f4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08f40539e0 0x7f08f4055ea0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f090400c000 tx=0x7f0904008040).stop 2022-07-21T01:42:01.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b6420 msgr2=0x7f0914152ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b6420 0x7f0914152ee0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f090c0012a0 tx=0x7f090c007880).stop 2022-07-21T01:42:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 shutdown_connections 2022-07-21T01:42:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140b6420 0x7f0914152ee0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09140b41f0 0x7f0914156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f08f40539e0 0x7f08f4055ea0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 --2- 172.21.15.3:0/1463109283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0914061fe0 0x7f0914145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 >> 172.21.15.3:0/1463109283 conn(0x7f091405c620 msgr2=0x7f0914146a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 shutdown_connections 2022-07-21T01:42:01.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.391+0000 7f091af88700 1 -- 172.21.15.3:0/1463109283 wait complete. 2022-07-21T01:42:01.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:01.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb919662700 1 Processor -- start 2022-07-21T01:42:01.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb919662700 1 -- start start 2022-07-21T01:42:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb919662700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb9140af450 0x7fb9140af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb919662700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140aff30 con 0x7fb9140ad9e0 2022-07-21T01:42:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb919662700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91405c7a0 con 0x7fb9140ad4b0 2022-07-21T01:42:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb919662700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb914140990 con 0x7fb9140af450 2022-07-21T01:42:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb913fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb9140af450 0x7fb9140af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb918e61700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb9140ad9e0 0x7fb9140a8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58684/0 (socket says 172.21.15.3:58684) 2022-07-21T01:42:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb918e61700 1 -- 172.21.15.3:0/1541511415 learned_addr learned my addr 172.21.15.3:0/1541511415 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:01.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.763+0000 7fb912ffd700 1 -- 172.21.15.3:0/1541511415 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333673055 0 0) 0x7fb9140aff30 con 0x7fb9140ad9e0 2022-07-21T01:42:01.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb912ffd700 1 -- 172.21.15.3:0/1541511415 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb900003260 con 0x7fb9140ad9e0 2022-07-21T01:42:01.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb912ffd700 1 -- 172.21.15.3:0/1541511415 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671586099 0 0) 0x7fb91405c7a0 con 0x7fb9140ad4b0 2022-07-21T01:42:01.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb912ffd700 1 -- 172.21.15.3:0/1541511415 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9140aff30 con 0x7fb9140ad4b0 2022-07-21T01:42:01.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb913fff700 1 -- 172.21.15.3:0/1541511415 >> v1:172.21.15.143:6789/0 conn(0x7fb9140ad4b0 legacy=0x7fb9141432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb913fff700 1 -- 172.21.15.3:0/1541511415 >> v1:172.21.15.3:6789/0 conn(0x7fb9140ad9e0 legacy=0x7fb9140a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb913fff700 1 -- 172.21.15.3:0/1541511415 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9141439f0 con 0x7fb9140af450 2022-07-21T01:42:01.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb913fff700 1 --2- 172.21.15.3:0/1541511415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140af450 0x7fb9140af830 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7fb9140ad900 tx=0x7fb904006df0).ready entity=mon.0 client_cookie=48d63cd5af102786 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:01.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb912ffd700 1 -- 172.21.15.3:0/1541511415 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb90400d610 con 0x7fb9140af450 2022-07-21T01:42:01.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb912ffd700 1 -- 172.21.15.3:0/1541511415 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb904000e90 con 0x7fb9140af450 2022-07-21T01:42:01.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb912ffd700 1 -- 172.21.15.3:0/1541511415 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb90400d960 con 0x7fb9140af450 2022-07-21T01:42:01.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb919662700 1 -- 172.21.15.3:0/1541511415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140af450 msgr2=0x7fb9140af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb919662700 1 --2- 172.21.15.3:0/1541511415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140af450 0x7fb9140af830 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7fb9140ad900 tx=0x7fb904006df0).stop 2022-07-21T01:42:01.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb919662700 1 -- 172.21.15.3:0/1541511415 shutdown_connections 2022-07-21T01:42:01.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb919662700 1 --2- 172.21.15.3:0/1541511415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140af450 0x7fb9140af830 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb919662700 1 -- 172.21.15.3:0/1541511415 >> 172.21.15.3:0/1541511415 conn(0x7fb9140612d0 msgr2=0x7fb914062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:01.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb919662700 1 -- 172.21.15.3:0/1541511415 shutdown_connections 2022-07-21T01:42:01.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb919662700 1 -- 172.21.15.3:0/1541511415 wait complete. 2022-07-21T01:42:01.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb919662700 1 Processor -- start 2022-07-21T01:42:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.767+0000 7fb919662700 1 -- start start 2022-07-21T01:42:01.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb919662700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9140ad4b0 0x7fb914144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb919662700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140ad9e0 0x7fb914152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb919662700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9140af450 0x7fb914152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb919662700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb914142910 con 0x7fb9140ad9e0 2022-07-21T01:42:01.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb919662700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140ab2b0 con 0x7fb9140ad4b0 2022-07-21T01:42:01.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb919662700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb914145390 con 0x7fb9140af450 2022-07-21T01:42:01.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9137fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140ad9e0 0x7fb914152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9137fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140ad9e0 0x7fb914152330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50410/0 (socket says 172.21.15.3:50410) 2022-07-21T01:42:01.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9137fe700 1 -- 172.21.15.3:0/1850872866 learned_addr learned my addr 172.21.15.3:0/1850872866 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:01.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb913fff700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9140ad4b0 0x7fb914144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb918e61700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9140af450 0x7fb914152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9137fe700 1 -- 172.21.15.3:0/1850872866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9140af450 msgr2=0x7fb914152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9137fe700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9140af450 0x7fb914152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9137fe700 1 -- 172.21.15.3:0/1850872866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9140ad4b0 msgr2=0x7fb914144990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9137fe700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9140ad4b0 0x7fb914144990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9137fe700 1 -- 172.21.15.3:0/1850872866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90400b180 con 0x7fb9140ad9e0 2022-07-21T01:42:01.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb913fff700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9140ad4b0 0x7fb914144990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9137fe700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140ad9e0 0x7fb914152330 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fb908001110 tx=0x7fb908007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:01.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9117fa700 1 -- 172.21.15.3:0/1850872866 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb90800ce50 con 0x7fb9140ad9e0 2022-07-21T01:42:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9117fa700 1 -- 172.21.15.3:0/1850872866 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90800f470 con 0x7fb9140ad9e0 2022-07-21T01:42:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb914152d10 con 0x7fb9140ad9e0 2022-07-21T01:42:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb9117fa700 1 -- 172.21.15.3:0/1850872866 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb90800f7d0 con 0x7fb9140ad9e0 2022-07-21T01:42:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9140ab4e0 con 0x7fb9140ad9e0 2022-07-21T01:42:01.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.771+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9140574e0 con 0x7fb9140ad9e0 2022-07-21T01:42:01.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.775+0000 7fb9117fa700 1 -- 172.21.15.3:0/1850872866 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb908019030 con 0x7fb9140ad9e0 2022-07-21T01:42:01.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.775+0000 7fb9117fa700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8f40539e0 0x7fb8f4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:01.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.775+0000 7fb9117fa700 1 -- 172.21.15.3:0/1850872866 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fb90806b200 con 0x7fb9140ad9e0 2022-07-21T01:42:01.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.775+0000 7fb913fff700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8f40539e0 0x7fb8f4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:01.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.775+0000 7fb913fff700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8f40539e0 0x7fb8f4055ea0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fb904018000 tx=0x7fb904014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:01.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.779+0000 7fb9117fa700 1 -- 172.21.15.3:0/1850872866 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb9080422a0 con 0x7fb9140ad9e0 2022-07-21T01:42:01.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.943+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb914061b20 con 0x7fb9140ad9e0 2022-07-21T01:42:01.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.943+0000 7fb9117fa700 1 -- 172.21.15.3:0/1850872866 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+504 (secure 0 0 0) 0x7fb9080458c0 con 0x7fb9140ad9e0 2022-07-21T01:42:01.947 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:01.947 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-07-21T01:42:01.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8f40539e0 msgr2=0x7fb8f4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8f40539e0 0x7fb8f4055ea0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fb904018000 tx=0x7fb904014000).stop 2022-07-21T01:42:01.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140ad9e0 msgr2=0x7fb914152330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:01.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140ad9e0 0x7fb914152330 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fb908001110 tx=0x7fb908007ae0).stop 2022-07-21T01:42:01.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 shutdown_connections 2022-07-21T01:42:01.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9140af450 0x7fb914152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9140ad9e0 0x7fb914152330 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb8f40539e0 0x7fb8f4055ea0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 --2- 172.21.15.3:0/1850872866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9140ad4b0 0x7fb914144990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:01.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 >> 172.21.15.3:0/1850872866 conn(0x7fb9140612d0 msgr2=0x7fb91405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:01.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 shutdown_connections 2022-07-21T01:42:01.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:01.947+0000 7fb919662700 1 -- 172.21.15.3:0/1850872866 wait complete. 2022-07-21T01:42:01.952 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 67 2022-07-21T01:42:01.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:02.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.055+0000 7f4922f6a700 1 Processor -- start 2022-07-21T01:42:02.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.055+0000 7f4922f6a700 1 -- start start 2022-07-21T01:42:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.055+0000 7f4922f6a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f491c0614b0 0x7f491c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.055+0000 7f4922f6a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0ad750 0x7f491c0b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.055+0000 7f4922f6a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491c0aee80 con 0x7f491c1405d0 2022-07-21T01:42:02.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.055+0000 7f4922f6a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491c0b70b0 con 0x7f491c0614b0 2022-07-21T01:42:02.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.055+0000 7f4922f6a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491c0af0b0 con 0x7f491c0ad750 2022-07-21T01:42:02.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.055+0000 7f4922769700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f491c1405d0 0x7f491c1466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58700/0 (socket says 172.21.15.3:58700) 2022-07-21T01:42:02.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.055+0000 7f4922769700 1 -- 172.21.15.3:0/2352535558 learned_addr learned my addr 172.21.15.3:0/2352535558 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:02.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4921767700 1 --2- 172.21.15.3:0/2352535558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0ad750 0x7f491c0b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4921f68700 1 --2- 172.21.15.3:0/2352535558 >> v2:172.21.15.143:3300/0 conn(0x7f491c0614b0 0x7f491c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4920f66700 1 -- 172.21.15.3:0/2352535558 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451537378 0 0) 0x7f491c0aee80 con 0x7f491c1405d0 2022-07-21T01:42:02.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4920f66700 1 -- 172.21.15.3:0/2352535558 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904003260 con 0x7f491c1405d0 2022-07-21T01:42:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4921767700 1 -- 172.21.15.3:0/2352535558 >> v2:172.21.15.143:3300/0 conn(0x7f491c0614b0 msgr2=0x7f491c061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4921767700 1 --2- 172.21.15.3:0/2352535558 >> v2:172.21.15.143:3300/0 conn(0x7f491c0614b0 0x7f491c061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4921767700 1 -- 172.21.15.3:0/2352535558 >> v1:172.21.15.3:6789/0 conn(0x7f491c1405d0 legacy=0x7f491c1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4921767700 1 -- 172.21.15.3:0/2352535558 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c0af350 con 0x7f491c0ad750 2022-07-21T01:42:02.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4921f68700 1 --2- 172.21.15.3:0/2352535558 >> v2:172.21.15.143:3300/0 conn(0x7f491c0614b0 0x7f491c061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:02.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4921767700 1 --2- 172.21.15.3:0/2352535558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0ad750 0x7f491c0b9ff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f4918002f70 tx=0x7f4918006df0).ready entity=mon.2 client_cookie=2ae3b5c96bd9580a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:02.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4920f66700 1 -- 172.21.15.3:0/2352535558 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f491800d4a0 con 0x7f491c0ad750 2022-07-21T01:42:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4920f66700 1 -- 172.21.15.3:0/2352535558 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4918000e90 con 0x7f491c0ad750 2022-07-21T01:42:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4920f66700 1 -- 172.21.15.3:0/2352535558 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f491800d7f0 con 0x7f491c0ad750 2022-07-21T01:42:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4922f6a700 1 -- 172.21.15.3:0/2352535558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0ad750 msgr2=0x7f491c0b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4922f6a700 1 --2- 172.21.15.3:0/2352535558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0ad750 0x7f491c0b9ff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f4918002f70 tx=0x7f4918006df0).stop 2022-07-21T01:42:02.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4922f6a700 1 -- 172.21.15.3:0/2352535558 shutdown_connections 2022-07-21T01:42:02.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4922f6a700 1 --2- 172.21.15.3:0/2352535558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0ad750 0x7f491c0b9ff0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4922f6a700 1 --2- 172.21.15.3:0/2352535558 >> v2:172.21.15.143:3300/0 conn(0x7f491c0614b0 0x7f491c061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4922f6a700 1 -- 172.21.15.3:0/2352535558 >> 172.21.15.3:0/2352535558 conn(0x7f491c066f70 msgr2=0x7f491c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:02.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4922f6a700 1 -- 172.21.15.3:0/2352535558 shutdown_connections 2022-07-21T01:42:02.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.059+0000 7f4922f6a700 1 -- 172.21.15.3:0/2352535558 wait complete. 2022-07-21T01:42:02.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 Processor -- start 2022-07-21T01:42:02.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 -- start start 2022-07-21T01:42:02.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0614b0 0x7f491c0b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f491c0ad750 0x7f491c152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f491c1405d0 0x7f491c156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c0b3440 con 0x7f491c0ad750 2022-07-21T01:42:02.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c152950 con 0x7f491c1405d0 2022-07-21T01:42:02.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491c0b3630 con 0x7f491c0614b0 2022-07-21T01:42:02.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921767700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f491c0ad750 0x7f491c152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921767700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f491c0ad750 0x7f491c152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50424/0 (socket says 172.21.15.3:50424) 2022-07-21T01:42:02.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921767700 1 -- 172.21.15.3:0/3697556798 learned_addr learned my addr 172.21.15.3:0/3697556798 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:02.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922769700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f491c1405d0 0x7f491c156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921f68700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0614b0 0x7f491c0b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921767700 1 -- 172.21.15.3:0/3697556798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0614b0 msgr2=0x7f491c0b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921767700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0614b0 0x7f491c0b2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921767700 1 -- 172.21.15.3:0/3697556798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f491c1405d0 msgr2=0x7f491c156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921767700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f491c1405d0 0x7f491c156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921767700 1 -- 172.21.15.3:0/3697556798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c00b180 con 0x7f491c0ad750 2022-07-21T01:42:02.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922769700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f491c1405d0 0x7f491c156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:02.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4921767700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f491c0ad750 0x7f491c152460 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f4918002f70 tx=0x7f4918014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:02.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f490affd700 1 -- 172.21.15.3:0/3697556798 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4918016070 con 0x7f491c0ad750 2022-07-21T01:42:02.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f490affd700 1 -- 172.21.15.3:0/3697556798 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49180137d0 con 0x7f491c0ad750 2022-07-21T01:42:02.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491800b180 con 0x7f491c0ad750 2022-07-21T01:42:02.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f490affd700 1 -- 172.21.15.3:0/3697556798 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4918013b30 con 0x7f491c0ad750 2022-07-21T01:42:02.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.063+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491c09ed80 con 0x7f491c0ad750 2022-07-21T01:42:02.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.067+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491c0af5f0 con 0x7f491c0ad750 2022-07-21T01:42:02.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.067+0000 7f490affd700 1 -- 172.21.15.3:0/3697556798 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4918025030 con 0x7f491c0ad750 2022-07-21T01:42:02.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.067+0000 7f490affd700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053950 0x7f48fc055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.067+0000 7f490affd700 1 -- 172.21.15.3:0/3697556798 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f491806cbf0 con 0x7f491c0ad750 2022-07-21T01:42:02.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.067+0000 7f4921f68700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053950 0x7f48fc055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.067+0000 7f4921f68700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053950 0x7f48fc055e10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f490c00b000 tx=0x7f490c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:02.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.075+0000 7f490affd700 1 -- 172.21.15.3:0/3697556798 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4918043c90 con 0x7f491c0ad750 2022-07-21T01:42:02.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.239+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f491c0b37f0 con 0x7f491c0ad750 2022-07-21T01:42:02.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f490affd700 1 -- 172.21.15.3:0/3697556798 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+504 (secure 0 0 0) 0x7f491c0b37f0 con 0x7f491c0ad750 2022-07-21T01:42:02.246 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:02.247 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-07-21T01:42:02.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053950 msgr2=0x7f48fc055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053950 0x7f48fc055e10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f490c00b000 tx=0x7f490c006df0).stop 2022-07-21T01:42:02.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f491c0ad750 msgr2=0x7f491c152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f491c0ad750 0x7f491c152460 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f4918002f70 tx=0x7f4918014000).stop 2022-07-21T01:42:02.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 shutdown_connections 2022-07-21T01:42:02.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f48fc053950 0x7f48fc055e10 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f491c1405d0 0x7f491c156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f491c0ad750 0x7f491c152460 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 --2- 172.21.15.3:0/3697556798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f491c0614b0 0x7f491c0b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 >> 172.21.15.3:0/3697556798 conn(0x7f491c066f70 msgr2=0x7f491c05c920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:02.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 shutdown_connections 2022-07-21T01:42:02.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.243+0000 7f4922f6a700 1 -- 172.21.15.3:0/3697556798 wait complete. 2022-07-21T01:42:02.255 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 67 2022-07-21T01:42:02.259 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:42:02.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:42:02.260 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:42:02.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:42:02.261 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:42:02.262 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:42:02.264 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:42:02.265 DEBUG:teuthology.parallel:result is None 2022-07-21T01:42:02.267 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:42:02.268 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:42:02.268 DEBUG:teuthology.parallel:result is None 2022-07-21T01:42:02.269 DEBUG:teuthology.parallel:result is None 2022-07-21T01:42:02.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:42:02.329 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:42:02.334 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:42:02.340 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:42:02.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.367+0000 7f520498e700 1 Processor -- start 2022-07-21T01:42:02.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.367+0000 7f520498e700 1 -- start start 2022-07-21T01:42:02.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5200060270 0x7f52000a9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f52000610d0 0x7f52000a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5200140420 con 0x7f52000aec90 2022-07-21T01:42:02.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52000aa940 con 0x7f52000610d0 2022-07-21T01:42:02.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52001406c0 con 0x7f5200060270 2022-07-21T01:42:02.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fe59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5200060270 0x7f52000a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fe59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5200060270 0x7f52000a9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:57370/0 (socket says 172.21.15.3:57370) 2022-07-21T01:42:02.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fe59c700 1 -- 172.21.15.3:0/1988902308 learned_addr learned my addr 172.21.15.3:0/1988902308 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:02.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fdd9b700 1 --2- 172.21.15.3:0/1988902308 >> v2:172.21.15.143:3300/0 conn(0x7f52000610d0 0x7f52000a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fe59c700 1 -- 172.21.15.3:0/1988902308 >> v2:172.21.15.143:3300/0 conn(0x7f52000610d0 msgr2=0x7f52000a9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:02.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fe59c700 1 --2- 172.21.15.3:0/1988902308 >> v2:172.21.15.143:3300/0 conn(0x7f52000610d0 0x7f52000a9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fe59c700 1 -- 172.21.15.3:0/1988902308 >> v1:172.21.15.143:6789/0 conn(0x7f52000aec90 legacy=0x7f52000af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fe59c700 1 -- 172.21.15.3:0/1988902308 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52000abdc0 con 0x7f5200060270 2022-07-21T01:42:02.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fe59c700 1 --2- 172.21.15.3:0/1988902308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5200060270 0x7f52000a9490 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f51f80032f0 tx=0x7f51f800b530).ready entity=mon.2 client_cookie=94ad7fbfc288918d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:02.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fd59a700 1 -- 172.21.15.3:0/1988902308 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51f80153f0 con 0x7f5200060270 2022-07-21T01:42:02.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fd59a700 1 -- 172.21.15.3:0/1988902308 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f8015550 con 0x7f5200060270 2022-07-21T01:42:02.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f51fd59a700 1 -- 172.21.15.3:0/1988902308 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51f80158b0 con 0x7f5200060270 2022-07-21T01:42:02.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 -- 172.21.15.3:0/1988902308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5200060270 msgr2=0x7f52000a9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 --2- 172.21.15.3:0/1988902308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5200060270 0x7f52000a9490 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f51f80032f0 tx=0x7f51f800b530).stop 2022-07-21T01:42:02.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 -- 172.21.15.3:0/1988902308 shutdown_connections 2022-07-21T01:42:02.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 --2- 172.21.15.3:0/1988902308 >> v2:172.21.15.143:3300/0 conn(0x7f52000610d0 0x7f52000a9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 --2- 172.21.15.3:0/1988902308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5200060270 0x7f52000a9490 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 -- 172.21.15.3:0/1988902308 >> 172.21.15.3:0/1988902308 conn(0x7f52000b5570 msgr2=0x7f52000b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:02.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 -- 172.21.15.3:0/1988902308 shutdown_connections 2022-07-21T01:42:02.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.371+0000 7f520498e700 1 -- 172.21.15.3:0/1988902308 wait complete. 2022-07-21T01:42:02.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f520498e700 1 Processor -- start 2022-07-21T01:42:02.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f520498e700 1 -- start start 2022-07-21T01:42:02.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f520498e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5200060270 0x7f52000b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f520498e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52000610d0 0x7f5200065e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f520498e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52000aec90 0x7f5200151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f520498e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5200066530 con 0x7f5200060270 2022-07-21T01:42:02.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f520498e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52000667d0 con 0x7f52000aec90 2022-07-21T01:42:02.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f520498e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52001519d0 con 0x7f52000610d0 2022-07-21T01:42:02.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f51fe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5200060270 0x7f52000b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f51fe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5200060270 0x7f52000b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50460/0 (socket says 172.21.15.3:50460) 2022-07-21T01:42:02.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f51fe59c700 1 -- 172.21.15.3:0/3639409237 learned_addr learned my addr 172.21.15.3:0/3639409237 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:02.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f51fed9d700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52000aec90 0x7f5200151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.375+0000 7f51fdd9b700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52000610d0 0x7f5200065e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51fe59c700 1 -- 172.21.15.3:0/3639409237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52000610d0 msgr2=0x7f5200065e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51fe59c700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52000610d0 0x7f5200065e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51fe59c700 1 -- 172.21.15.3:0/3639409237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52000aec90 msgr2=0x7f5200151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:02.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51fe59c700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52000aec90 0x7f5200151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:02.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51fe59c700 1 -- 172.21.15.3:0/3639409237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f800b7d0 con 0x7f5200060270 2022-07-21T01:42:02.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51fed9d700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52000aec90 0x7f5200151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:02.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51fe59c700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5200060270 0x7f52000b47b0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7f51f80032f0 tx=0x7f51f8019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:02.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51ef7fe700 1 -- 172.21.15.3:0/3639409237 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51f80153f0 con 0x7f5200060270 2022-07-21T01:42:02.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51ef7fe700 1 -- 172.21.15.3:0/3639409237 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f8014b00 con 0x7f5200060270 2022-07-21T01:42:02.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520014e300 con 0x7f5200060270 2022-07-21T01:42:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51ef7fe700 1 -- 172.21.15.3:0/3639409237 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51f8014e60 con 0x7f5200060270 2022-07-21T01:42:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5200063480 con 0x7f5200060270 2022-07-21T01:42:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5200066990 con 0x7f5200060270 2022-07-21T01:42:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51ef7fe700 1 -- 172.21.15.3:0/3639409237 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f51f8028460 con 0x7f5200060270 2022-07-21T01:42:02.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51ef7fe700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51dc053a10 0x7f51dc055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:02.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51ef7fe700 1 -- 172.21.15.3:0/3639409237 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f51f806fd60 con 0x7f5200060270 2022-07-21T01:42:02.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.379+0000 7f51fdd9b700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51dc053a10 0x7f51dc055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:02.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.383+0000 7f51fdd9b700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51dc053a10 0x7f51dc055ed0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f51f400b000 tx=0x7f51f4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:02.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.387+0000 7f51ef7fe700 1 -- 172.21.15.3:0/3639409237 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f51f8046da0 con 0x7f5200060270 2022-07-21T01:42:02.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:42:02.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:42:02.490 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:42:02.492 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:42:02.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:02.519+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f52000b1d50 con 0x7f5200060270 2022-07-21T01:42:03.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.347+0000 7f51ef7fe700 1 -- 172.21.15.3:0/3639409237 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v136) v1 ==== 97+0+0 (secure 0 0 0) 0x7f51f804a3c0 con 0x7f5200060270 2022-07-21T01:42:03.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51dc053a10 msgr2=0x7f51dc055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51dc053a10 0x7f51dc055ed0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f51f400b000 tx=0x7f51f4006df0).stop 2022-07-21T01:42:03.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5200060270 msgr2=0x7f52000b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5200060270 0x7f52000b47b0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7f51f80032f0 tx=0x7f51f8019000).stop 2022-07-21T01:42:03.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 shutdown_connections 2022-07-21T01:42:03.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f51dc053a10 0x7f51dc055ed0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52000aec90 0x7f5200151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52000610d0 0x7f5200065e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 --2- 172.21.15.3:0/3639409237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5200060270 0x7f52000b47b0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 >> 172.21.15.3:0/3639409237 conn(0x7f52000b5570 msgr2=0x7f5200140c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:03.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 shutdown_connections 2022-07-21T01:42:03.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.351+0000 7f520498e700 1 -- 172.21.15.3:0/3639409237 wait complete. 2022-07-21T01:42:03.361 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:42:03.369 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:42:03.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:42:03.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e8e7a700 1 Processor -- start 2022-07-21T01:42:03.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e8e7a700 1 -- start start 2022-07-21T01:42:03.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e8e7a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b2390 0x7fb6e40abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e8e7a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e41453e0 con 0x7fb6e4141f40 2022-07-21T01:42:03.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e8e7a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e4145680 con 0x7fb6e40b0160 2022-07-21T01:42:03.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e8e7a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e4141390 con 0x7fb6e40b2390 2022-07-21T01:42:03.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e2d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb6e4141f40 0x7fb6e414bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58752/0 (socket says 172.21.15.3:58752) 2022-07-21T01:42:03.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e2d9d700 1 -- 172.21.15.3:0/163311732 learned_addr learned my addr 172.21.15.3:0/163311732 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:03.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e259c700 1 --2- 172.21.15.3:0/163311732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b2390 0x7fb6e40abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e159a700 1 -- 172.21.15.3:0/163311732 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319657493 0 0) 0x7fb6e41453e0 con 0x7fb6e4141f40 2022-07-21T01:42:03.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e159a700 1 -- 172.21.15.3:0/163311732 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c8003260 con 0x7fb6e4141f40 2022-07-21T01:42:03.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.471+0000 7fb6e159a700 1 -- 172.21.15.3:0/163311732 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857820540 0 0) 0x7fb6e4145680 con 0x7fb6e40b0160 2022-07-21T01:42:03.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e159a700 1 -- 172.21.15.3:0/163311732 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e41453e0 con 0x7fb6e40b0160 2022-07-21T01:42:03.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e259c700 1 -- 172.21.15.3:0/163311732 >> v1:172.21.15.143:6789/0 conn(0x7fb6e40b0160 legacy=0x7fb6e4141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e159a700 1 -- 172.21.15.3:0/163311732 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2197617081 0 0) 0x7fb6c8003260 con 0x7fb6e4141f40 2022-07-21T01:42:03.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e259c700 1 -- 172.21.15.3:0/163311732 >> v1:172.21.15.3:6789/0 conn(0x7fb6e4141f40 legacy=0x7fb6e414bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e259c700 1 -- 172.21.15.3:0/163311732 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e40b0670 con 0x7fb6e40b2390 2022-07-21T01:42:03.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e259c700 1 --2- 172.21.15.3:0/163311732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b2390 0x7fb6e40abf90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb6dc001490 tx=0x7fb6d8006df0).ready entity=mon.2 client_cookie=6f902297015eff54 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:03.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e159a700 1 -- 172.21.15.3:0/163311732 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6dc0019e0 con 0x7fb6e40b2390 2022-07-21T01:42:03.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e159a700 1 -- 172.21.15.3:0/163311732 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d8000e90 con 0x7fb6e40b2390 2022-07-21T01:42:03.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e159a700 1 -- 172.21.15.3:0/163311732 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6d8012460 con 0x7fb6e40b2390 2022-07-21T01:42:03.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/163311732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b2390 msgr2=0x7fb6e40abf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e8e7a700 1 --2- 172.21.15.3:0/163311732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b2390 0x7fb6e40abf90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb6dc001490 tx=0x7fb6d8006df0).stop 2022-07-21T01:42:03.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/163311732 shutdown_connections 2022-07-21T01:42:03.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e8e7a700 1 --2- 172.21.15.3:0/163311732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b2390 0x7fb6e40abf90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/163311732 >> 172.21.15.3:0/163311732 conn(0x7fb6e405f840 msgr2=0x7fb6e405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:03.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/163311732 shutdown_connections 2022-07-21T01:42:03.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/163311732 wait complete. 2022-07-21T01:42:03.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e8e7a700 1 Processor -- start 2022-07-21T01:42:03.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.475+0000 7fb6e8e7a700 1 -- start start 2022-07-21T01:42:03.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e8e7a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b0160 0x7fb6e4144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e8e7a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6e40b2390 0x7fb6e413df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e8e7a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6e4141f40 0x7fb6e413e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e8e7a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e4143be0 con 0x7fb6e4141f40 2022-07-21T01:42:03.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e8e7a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e413e910 con 0x7fb6e40b2390 2022-07-21T01:42:03.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e8e7a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e41436a0 con 0x7fb6e40b0160 2022-07-21T01:42:03.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6e4141f40 0x7fb6e413e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6e4141f40 0x7fb6e413e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50478/0 (socket says 172.21.15.3:50478) 2022-07-21T01:42:03.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e2d9d700 1 -- 172.21.15.3:0/735420789 learned_addr learned my addr 172.21.15.3:0/735420789 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:03.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e1d9b700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6e40b2390 0x7fb6e413df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e259c700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b0160 0x7fb6e4144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e2d9d700 1 -- 172.21.15.3:0/735420789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b0160 msgr2=0x7fb6e4144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e2d9d700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b0160 0x7fb6e4144800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e2d9d700 1 -- 172.21.15.3:0/735420789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6e40b2390 msgr2=0x7fb6e413df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e2d9d700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6e40b2390 0x7fb6e413df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e2d9d700 1 -- 172.21.15.3:0/735420789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d800b180 con 0x7fb6e4141f40 2022-07-21T01:42:03.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e2d9d700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6e4141f40 0x7fb6e413e420 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7fb6dc001360 tx=0x7fb6dc001c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:03.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6d37fe700 1 -- 172.21.15.3:0/735420789 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6dc004ab0 con 0x7fb6e4141f40 2022-07-21T01:42:03.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6d37fe700 1 -- 172.21.15.3:0/735420789 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6dc004c10 con 0x7fb6e4141f40 2022-07-21T01:42:03.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6d37fe700 1 -- 172.21.15.3:0/735420789 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6dc007170 con 0x7fb6e4141f40 2022-07-21T01:42:03.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e4143940 con 0x7fb6e4141f40 2022-07-21T01:42:03.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e413dc20 con 0x7fb6e4141f40 2022-07-21T01:42:03.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.479+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e4144e90 con 0x7fb6e4141f40 2022-07-21T01:42:03.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.483+0000 7fb6d37fe700 1 -- 172.21.15.3:0/735420789 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb6dc0072d0 con 0x7fb6e4141f40 2022-07-21T01:42:03.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.483+0000 7fb6d37fe700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6c0053a70 0x7fb6c0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.483+0000 7fb6d37fe700 1 -- 172.21.15.3:0/735420789 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fb6dc06ba70 con 0x7fb6e4141f40 2022-07-21T01:42:03.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.483+0000 7fb6e259c700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6c0053a70 0x7fb6c0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.483+0000 7fb6e259c700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6c0053a70 0x7fb6c0055f30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7fb6d801a000 tx=0x7fb6d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:03.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.487+0000 7fb6d37fe700 1 -- 172.21.15.3:0/735420789 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb6dc042a60 con 0x7fb6e4141f40 2022-07-21T01:42:03.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.651+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb6e40574e0 con 0x7fb6e4141f40 2022-07-21T01:42:03.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.651+0000 7fb6d37fe700 1 -- 172.21.15.3:0/735420789 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v33) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fb6dc046080 con 0x7fb6e4141f40 2022-07-21T01:42:03.656 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:03.656 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:42:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:42:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:42:03.657 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:42:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:42:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:42:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.660 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.661 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:42:03.662 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:42:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.663 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:42:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:42:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.665 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:42:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:42:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.667 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:42:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:42:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.669 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:42:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:42:03.671 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.672 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:42:03.673 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:42:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.674 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:42:03.675 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:42:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:42:03.677 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:42:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.678 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.679 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:42:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:42:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.680 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:42:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:42:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:03.682 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:42:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:42:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:42:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:42:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:03.684 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:42:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:42:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:42:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:42:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.686 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:42:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:42:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:42:03.688 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:42:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:42:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.689 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:42:03.690 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:42:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:42:03.691 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:42:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:03.692 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:03.693 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:42:03.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6c0053a70 msgr2=0x7fb6c0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6c0053a70 0x7fb6c0055f30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7fb6d801a000 tx=0x7fb6d8014000).stop 2022-07-21T01:42:03.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6e4141f40 msgr2=0x7fb6e413e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6e4141f40 0x7fb6e413e420 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7fb6dc001360 tx=0x7fb6dc001c50).stop 2022-07-21T01:42:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 shutdown_connections 2022-07-21T01:42:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6e4141f40 0x7fb6e413e420 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb6c0053a70 0x7fb6c0055f30 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6e40b2390 0x7fb6e413df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 --2- 172.21.15.3:0/735420789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6e40b0160 0x7fb6e4144800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 >> 172.21.15.3:0/735420789 conn(0x7fb6e405f840 msgr2=0x7fb6e413aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 shutdown_connections 2022-07-21T01:42:03.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.655+0000 7fb6e8e7a700 1 -- 172.21.15.3:0/735420789 wait complete. 2022-07-21T01:42:03.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.799+0000 7fc4aa974700 1 Processor -- start 2022-07-21T01:42:03.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.799+0000 7fc4aa974700 1 -- start start 2022-07-21T01:42:03.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc4a4148530 0x7fc4a40b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc4a40a7250 0x7fc4a41448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a40b04e0 con 0x7fc4a40b1f40 2022-07-21T01:42:03.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a40a9250 con 0x7fc4a4148530 2022-07-21T01:42:03.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a40b0780 con 0x7fc4a40a7250 2022-07-21T01:42:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc4a4148530 0x7fc4a40b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc4a4148530 0x7fc4a40b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50482/0 (socket says 172.21.15.3:50482) 2022-07-21T01:42:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a37fe700 1 -- 172.21.15.3:0/2575605238 learned_addr learned my addr 172.21.15.3:0/2575605238 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a8f11700 1 --2- 172.21.15.3:0/2575605238 >> v2:172.21.15.143:3300/0 conn(0x7fc4a40a7250 0x7fc4a41448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a37fe700 1 -- 172.21.15.3:0/2575605238 >> v2:172.21.15.143:3300/0 conn(0x7fc4a40a7250 msgr2=0x7fc4a41448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a37fe700 1 --2- 172.21.15.3:0/2575605238 >> v2:172.21.15.143:3300/0 conn(0x7fc4a40a7250 0x7fc4a41448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a37fe700 1 -- 172.21.15.3:0/2575605238 >> v1:172.21.15.143:6789/0 conn(0x7fc4a40b1f40 legacy=0x7fc4a40b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a37fe700 1 -- 172.21.15.3:0/2575605238 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a40b0a20 con 0x7fc4a4148530 2022-07-21T01:42:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a37fe700 1 --2- 172.21.15.3:0/2575605238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a4148530 0x7fc4a40b1160 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7fc4a40acff0 tx=0x7fc494006eb0).ready entity=mon.0 client_cookie=4267fef569885c88 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a2ffd700 1 -- 172.21.15.3:0/2575605238 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc49400d5c0 con 0x7fc4a4148530 2022-07-21T01:42:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a2ffd700 1 -- 172.21.15.3:0/2575605238 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc494000e90 con 0x7fc4a4148530 2022-07-21T01:42:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4a2ffd700 1 -- 172.21.15.3:0/2575605238 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc49400d950 con 0x7fc4a4148530 2022-07-21T01:42:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 -- 172.21.15.3:0/2575605238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a4148530 msgr2=0x7fc4a40b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 --2- 172.21.15.3:0/2575605238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a4148530 0x7fc4a40b1160 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7fc4a40acff0 tx=0x7fc494006eb0).stop 2022-07-21T01:42:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 -- 172.21.15.3:0/2575605238 shutdown_connections 2022-07-21T01:42:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 --2- 172.21.15.3:0/2575605238 >> v2:172.21.15.143:3300/0 conn(0x7fc4a40a7250 0x7fc4a41448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 --2- 172.21.15.3:0/2575605238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a4148530 0x7fc4a40b1160 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 -- 172.21.15.3:0/2575605238 >> 172.21.15.3:0/2575605238 conn(0x7fc4a405f840 msgr2=0x7fc4a405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:03.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 -- 172.21.15.3:0/2575605238 shutdown_connections 2022-07-21T01:42:03.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.803+0000 7fc4aa974700 1 -- 172.21.15.3:0/2575605238 wait complete. 2022-07-21T01:42:03.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4aa974700 1 Processor -- start 2022-07-21T01:42:03.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4aa974700 1 -- start start 2022-07-21T01:42:03.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4aa974700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4a40a7250 0x7fc4a415a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4aa974700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a40b1f40 0x7fc4a4149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4aa974700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4a4148530 0x7fc4a40aba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4aa974700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40aa400 con 0x7fc4a40b1f40 2022-07-21T01:42:03.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4aa974700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a415ac80 con 0x7fc4a4148530 2022-07-21T01:42:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4aa974700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a415a9e0 con 0x7fc4a40a7250 2022-07-21T01:42:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4a37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a40b1f40 0x7fc4a4149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4a37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a40b1f40 0x7fc4a4149b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50490/0 (socket says 172.21.15.3:50490) 2022-07-21T01:42:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4a37fe700 1 -- 172.21.15.3:0/3519599807 learned_addr learned my addr 172.21.15.3:0/3519599807 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4a3fff700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4a40a7250 0x7fc4a415a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.807+0000 7fc4a8f11700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4a4148530 0x7fc4a40aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a37fe700 1 -- 172.21.15.3:0/3519599807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4a40a7250 msgr2=0x7fc4a415a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a37fe700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4a40a7250 0x7fc4a415a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a37fe700 1 -- 172.21.15.3:0/3519599807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4a4148530 msgr2=0x7fc4a40aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a37fe700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4a4148530 0x7fc4a40aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a37fe700 1 -- 172.21.15.3:0/3519599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49400b1e0 con 0x7fc4a40b1f40 2022-07-21T01:42:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a8f11700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4a4148530 0x7fc4a40aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a37fe700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a40b1f40 0x7fc4a4149b80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7fc4a40acff0 tx=0x7fc494015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a17fa700 1 -- 172.21.15.3:0/3519599807 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc494011860 con 0x7fc4a40b1f40 2022-07-21T01:42:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a40a7c10 con 0x7fc4a40b1f40 2022-07-21T01:42:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a17fa700 1 -- 172.21.15.3:0/3519599807 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4940119c0 con 0x7fc4a40b1f40 2022-07-21T01:42:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a40a9970 con 0x7fc4a40b1f40 2022-07-21T01:42:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a17fa700 1 -- 172.21.15.3:0/3519599807 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc494011d20 con 0x7fc4a40b1f40 2022-07-21T01:42:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a40b0cc0 con 0x7fc4a40b1f40 2022-07-21T01:42:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a17fa700 1 -- 172.21.15.3:0/3519599807 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc494025030 con 0x7fc4a40b1f40 2022-07-21T01:42:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.811+0000 7fc4a17fa700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc484053b30 0x7fc484055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:03.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.815+0000 7fc4a3fff700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc484053b30 0x7fc484055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:03.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.815+0000 7fc4a17fa700 1 -- 172.21.15.3:0/3519599807 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fc49406ce80 con 0x7fc4a40b1f40 2022-07-21T01:42:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.815+0000 7fc4a3fff700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc484053b30 0x7fc484055ff0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fc49800d000 tx=0x7fc498007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:03.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.819+0000 7fc4a17fa700 1 -- 172.21.15.3:0/3519599807 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc494043f90 con 0x7fc4a40b1f40 2022-07-21T01:42:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.947+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4a40574e0 con 0x7fc4a40b1f40 2022-07-21T01:42:03.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.947+0000 7fc4a17fa700 1 -- 172.21.15.3:0/3519599807 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+13424 (secure 0 0 0) 0x7fc494030480 con 0x7fc4a40b1f40 2022-07-21T01:42:03.952 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:03.953 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":136,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:03.343434+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:03.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc484053b30 msgr2=0x7fc484055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc484053b30 0x7fc484055ff0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fc49800d000 tx=0x7fc498007b70).stop 2022-07-21T01:42:03.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a40b1f40 msgr2=0x7fc4a4149b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:03.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a40b1f40 0x7fc4a4149b80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7fc4a40acff0 tx=0x7fc494015000).stop 2022-07-21T01:42:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 shutdown_connections 2022-07-21T01:42:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc484053b30 0x7fc484055ff0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4a4148530 0x7fc4a40aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4a40b1f40 0x7fc4a4149b80 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 --2- 172.21.15.3:0/3519599807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4a40a7250 0x7fc4a415a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 >> 172.21.15.3:0/3519599807 conn(0x7fc4a405f840 msgr2=0x7fc4a4144a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 shutdown_connections 2022-07-21T01:42:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:03.951+0000 7fc4aa974700 1 -- 172.21.15.3:0/3519599807 wait complete. 2022-07-21T01:42:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:42:03.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:42:04.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.079+0000 7f5610b98700 1 Processor -- start 2022-07-21T01:42:04.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.079+0000 7f5610b98700 1 -- start start 2022-07-21T01:42:04.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f560c0619b0 0x7f560c0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f560c0b5ed0 0x7f560c0b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c061d90 con 0x7f560c0b6ee0 2022-07-21T01:42:04.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560c143fa0 con 0x7f560c0b5ed0 2022-07-21T01:42:04.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560c142f60 con 0x7f560c0619b0 2022-07-21T01:42:04.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f560c0b5ed0 0x7f560c0b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560affd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f560c0b6ee0 0x7f560c143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58776/0 (socket says 172.21.15.3:58776) 2022-07-21T01:42:04.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560affd700 1 -- 172.21.15.3:0/2166903938 learned_addr learned my addr 172.21.15.3:0/2166903938 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:04.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560b7fe700 1 --2- 172.21.15.3:0/2166903938 >> v2:172.21.15.143:3300/0 conn(0x7f560c0619b0 0x7f560c0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560a7fc700 1 -- 172.21.15.3:0/2166903938 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189605779 0 0) 0x7f560c061d90 con 0x7f560c0b6ee0 2022-07-21T01:42:04.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560a7fc700 1 -- 172.21.15.3:0/2166903938 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ec003260 con 0x7f560c0b6ee0 2022-07-21T01:42:04.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560bfff700 1 -- 172.21.15.3:0/2166903938 >> v2:172.21.15.143:3300/0 conn(0x7f560c0619b0 msgr2=0x7f560c0b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:04.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560bfff700 1 --2- 172.21.15.3:0/2166903938 >> v2:172.21.15.143:3300/0 conn(0x7f560c0619b0 0x7f560c0b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560bfff700 1 -- 172.21.15.3:0/2166903938 >> v1:172.21.15.3:6789/0 conn(0x7f560c0b6ee0 legacy=0x7f560c143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560a7fc700 1 -- 172.21.15.3:0/2166903938 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1244821430 0 0) 0x7f55ec003260 con 0x7f560c0b6ee0 2022-07-21T01:42:04.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560bfff700 1 -- 172.21.15.3:0/2166903938 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c0ba5a0 con 0x7f560c0b5ed0 2022-07-21T01:42:04.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560b7fe700 1 --2- 172.21.15.3:0/2166903938 >> v2:172.21.15.143:3300/0 conn(0x7f560c0619b0 0x7f560c0b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:04.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560bfff700 1 --2- 172.21.15.3:0/2166903938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0b5ed0 0x7f560c0b5190 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7f55fc001820 tx=0x7f56040091d0).ready entity=mon.0 client_cookie=862484adf675b970 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560a7fc700 1 -- 172.21.15.3:0/2166903938 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55fc001680 con 0x7f560c0b5ed0 2022-07-21T01:42:04.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560a7fc700 1 -- 172.21.15.3:0/2166903938 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560400b520 con 0x7f560c0b5ed0 2022-07-21T01:42:04.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f560a7fc700 1 -- 172.21.15.3:0/2166903938 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f560400b690 con 0x7f560c0b5ed0 2022-07-21T01:42:04.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 -- 172.21.15.3:0/2166903938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0b5ed0 msgr2=0x7f560c0b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 --2- 172.21.15.3:0/2166903938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0b5ed0 0x7f560c0b5190 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7f55fc001820 tx=0x7f56040091d0).stop 2022-07-21T01:42:04.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 -- 172.21.15.3:0/2166903938 shutdown_connections 2022-07-21T01:42:04.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 --2- 172.21.15.3:0/2166903938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0b5ed0 0x7f560c0b5190 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 --2- 172.21.15.3:0/2166903938 >> v2:172.21.15.143:3300/0 conn(0x7f560c0619b0 0x7f560c0b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 -- 172.21.15.3:0/2166903938 >> 172.21.15.3:0/2166903938 conn(0x7f560c06af80 msgr2=0x7f560c05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:04.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 -- 172.21.15.3:0/2166903938 shutdown_connections 2022-07-21T01:42:04.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.083+0000 7f5610b98700 1 -- 172.21.15.3:0/2166903938 wait complete. 2022-07-21T01:42:04.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f5610b98700 1 Processor -- start 2022-07-21T01:42:04.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f5610b98700 1 -- start start 2022-07-21T01:42:04.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f5610b98700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0619b0 0x7f560c1489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f5610b98700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c0b5ed0 0x7f560c13b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f5610b98700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c0b6ee0 0x7f560c13c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f5610b98700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c13bbf0 con 0x7f560c0619b0 2022-07-21T01:42:04.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f5610b98700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c13c630 con 0x7f560c0b5ed0 2022-07-21T01:42:04.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f5610b98700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c1409a0 con 0x7f560c0b6ee0 2022-07-21T01:42:04.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f560b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0619b0 0x7f560c1489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f560b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0619b0 0x7f560c1489b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50502/0 (socket says 172.21.15.3:50502) 2022-07-21T01:42:04.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.087+0000 7f560b7fe700 1 -- 172.21.15.3:0/983843949 learned_addr learned my addr 172.21.15.3:0/983843949 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:04.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f560bfff700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c0b6ee0 0x7f560c13c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f560affd700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c0b5ed0 0x7f560c13b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f560b7fe700 1 -- 172.21.15.3:0/983843949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c0b6ee0 msgr2=0x7f560c13c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f560b7fe700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c0b6ee0 0x7f560c13c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f560b7fe700 1 -- 172.21.15.3:0/983843949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c0b5ed0 msgr2=0x7f560c13b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f560b7fe700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c0b5ed0 0x7f560c13b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f560b7fe700 1 -- 172.21.15.3:0/983843949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604009470 con 0x7f560c0619b0 2022-07-21T01:42:04.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f560b7fe700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0619b0 0x7f560c1489b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f55f4006940 tx=0x7f55f4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f5608ff9700 1 -- 172.21.15.3:0/983843949 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55f4011780 con 0x7f560c0619b0 2022-07-21T01:42:04.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f5608ff9700 1 -- 172.21.15.3:0/983843949 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f40118e0 con 0x7f560c0619b0 2022-07-21T01:42:04.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c0afaa0 con 0x7f560c0619b0 2022-07-21T01:42:04.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f5608ff9700 1 -- 172.21.15.3:0/983843949 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55f4011c40 con 0x7f560c0619b0 2022-07-21T01:42:04.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c13d400 con 0x7f560c0619b0 2022-07-21T01:42:04.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560c0af900 con 0x7f560c0619b0 2022-07-21T01:42:04.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.091+0000 7f5608ff9700 1 -- 172.21.15.3:0/983843949 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f55f401c460 con 0x7f560c0619b0 2022-07-21T01:42:04.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.095+0000 7f5608ff9700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55f0053a70 0x7f55f0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.095+0000 7f5608ff9700 1 -- 172.21.15.3:0/983843949 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f55f406bf90 con 0x7f560c0619b0 2022-07-21T01:42:04.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.095+0000 7f560affd700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55f0053a70 0x7f55f0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.095+0000 7f560affd700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55f0053a70 0x7f55f0055f30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f55fc00d000 tx=0x7f55fc0019a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.099+0000 7f5608ff9700 1 -- 172.21.15.3:0/983843949 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f560c0af900 con 0x7f560c0619b0 2022-07-21T01:42:04.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f560c0af900 con 0x7f560c0619b0 2022-07-21T01:42:04.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5608ff9700 1 -- 172.21.15.3:0/983843949 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v136) v1 ==== 110+0+10 (secure 0 0 0) 0x7f560c0af900 con 0x7f560c0619b0 2022-07-21T01:42:04.231 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:42:04.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55f0053a70 msgr2=0x7f55f0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55f0053a70 0x7f55f0055f30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f55fc00d000 tx=0x7f55fc0019a0).stop 2022-07-21T01:42:04.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0619b0 msgr2=0x7f560c1489b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0619b0 0x7f560c1489b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f55f4006940 tx=0x7f55f4008040).stop 2022-07-21T01:42:04.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 shutdown_connections 2022-07-21T01:42:04.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c0b6ee0 0x7f560c13c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f55f0053a70 0x7f55f0055f30 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c0b5ed0 0x7f560c13b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 --2- 172.21.15.3:0/983843949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f560c0619b0 0x7f560c1489b0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.227+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 >> 172.21.15.3:0/983843949 conn(0x7f560c06af80 msgr2=0x7f560c05ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:04.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.231+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 shutdown_connections 2022-07-21T01:42:04.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.231+0000 7f5610b98700 1 -- 172.21.15.3:0/983843949 wait complete. 2022-07-21T01:42:04.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:42:04.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.351+0000 7faa4c95c700 1 Processor -- start 2022-07-21T01:42:04.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.351+0000 7faa4c95c700 1 -- start start 2022-07-21T01:42:04.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4c95c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480b08a0 0x7faa48142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4c95c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7faa480b3260 0x7faa481433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4c95c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480b0c80 con 0x7faa480ad000 2022-07-21T01:42:04.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4c95c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa480ae810 con 0x7faa480b3260 2022-07-21T01:42:04.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4c95c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa480aeab0 con 0x7faa480b08a0 2022-07-21T01:42:04.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4759e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7faa480ad000 0x7faa4805f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58792/0 (socket says 172.21.15.3:58792) 2022-07-21T01:42:04.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4759e700 1 -- 172.21.15.3:0/1320357755 learned_addr learned my addr 172.21.15.3:0/1320357755 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:04.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa3e59c700 1 --2- 172.21.15.3:0/1320357755 >> v2:172.21.15.3:3300/0 conn(0x7faa480b3260 0x7faa481433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa46d9d700 1 --2- 172.21.15.3:0/1320357755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480b08a0 0x7faa48142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4659c700 1 -- 172.21.15.3:0/1320357755 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758483534 0 0) 0x7faa480b0c80 con 0x7faa480ad000 2022-07-21T01:42:04.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4659c700 1 -- 172.21.15.3:0/1320357755 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa28003260 con 0x7faa480ad000 2022-07-21T01:42:04.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa3e59c700 1 -- 172.21.15.3:0/1320357755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480b08a0 msgr2=0x7faa48142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:04.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa3e59c700 1 --2- 172.21.15.3:0/1320357755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480b08a0 0x7faa48142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa3e59c700 1 -- 172.21.15.3:0/1320357755 >> v1:172.21.15.3:6789/0 conn(0x7faa480ad000 legacy=0x7faa4805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4659c700 1 -- 172.21.15.3:0/1320357755 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2639328351 0 0) 0x7faa28003260 con 0x7faa480ad000 2022-07-21T01:42:04.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa3e59c700 1 -- 172.21.15.3:0/1320357755 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa480aed50 con 0x7faa480b3260 2022-07-21T01:42:04.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa3e59c700 1 --2- 172.21.15.3:0/1320357755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b3260 0x7faa481433a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0x7faa30002f70 tx=0x7faa30006df0).ready entity=mon.0 client_cookie=2434dfe305da890e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4659c700 1 -- 172.21.15.3:0/1320357755 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa3000d4a0 con 0x7faa480b3260 2022-07-21T01:42:04.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4659c700 1 -- 172.21.15.3:0/1320357755 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa30000e90 con 0x7faa480b3260 2022-07-21T01:42:04.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.355+0000 7faa4659c700 1 -- 172.21.15.3:0/1320357755 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa3000d7f0 con 0x7faa480b3260 2022-07-21T01:42:04.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 -- 172.21.15.3:0/1320357755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b3260 msgr2=0x7faa481433a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 --2- 172.21.15.3:0/1320357755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b3260 0x7faa481433a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0x7faa30002f70 tx=0x7faa30006df0).stop 2022-07-21T01:42:04.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 -- 172.21.15.3:0/1320357755 shutdown_connections 2022-07-21T01:42:04.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 --2- 172.21.15.3:0/1320357755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b3260 0x7faa481433a0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 --2- 172.21.15.3:0/1320357755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480b08a0 0x7faa48142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 -- 172.21.15.3:0/1320357755 >> 172.21.15.3:0/1320357755 conn(0x7faa480612d0 msgr2=0x7faa48062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:04.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 -- 172.21.15.3:0/1320357755 shutdown_connections 2022-07-21T01:42:04.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 -- 172.21.15.3:0/1320357755 wait complete. 2022-07-21T01:42:04.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 Processor -- start 2022-07-21T01:42:04.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 -- start start 2022-07-21T01:42:04.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480ad000 0x7faa48063be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b08a0 0x7faa48140ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa480b3260 0x7faa480676e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa48065890 con 0x7faa480b08a0 2022-07-21T01:42:04.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa48065390 con 0x7faa480b3260 2022-07-21T01:42:04.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4c95c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa480655f0 con 0x7faa480ad000 2022-07-21T01:42:04.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa3e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b08a0 0x7faa48140ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa3e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b08a0 0x7faa48140ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50518/0 (socket says 172.21.15.3:50518) 2022-07-21T01:42:04.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa3e59c700 1 -- 172.21.15.3:0/146764274 learned_addr learned my addr 172.21.15.3:0/146764274 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:04.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa4759e700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa480b3260 0x7faa480676e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa46d9d700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480ad000 0x7faa48063be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.377 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa3e59c700 1 -- 172.21.15.3:0/146764274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480ad000 msgr2=0x7faa48063be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa3e59c700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480ad000 0x7faa48063be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa3e59c700 1 -- 172.21.15.3:0/146764274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa480b3260 msgr2=0x7faa480676e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa3e59c700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa480b3260 0x7faa480676e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa3e59c700 1 -- 172.21.15.3:0/146764274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3000b180 con 0x7faa480b08a0 2022-07-21T01:42:04.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.359+0000 7faa46d9d700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480ad000 0x7faa48063be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:04.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa3e59c700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b08a0 0x7faa48140ce0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7faa30002f70 tx=0x7faa30014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa44d99700 1 -- 172.21.15.3:0/146764274 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa300139c0 con 0x7faa480b08a0 2022-07-21T01:42:04.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa44d99700 1 -- 172.21.15.3:0/146764274 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa30013b20 con 0x7faa480b08a0 2022-07-21T01:42:04.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa48151260 con 0x7faa480b08a0 2022-07-21T01:42:04.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa44d99700 1 -- 172.21.15.3:0/146764274 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa30013e80 con 0x7faa480b08a0 2022-07-21T01:42:04.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa48150df0 con 0x7faa480b08a0 2022-07-21T01:42:04.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa44d99700 1 -- 172.21.15.3:0/146764274 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faa30025030 con 0x7faa480b08a0 2022-07-21T01:42:04.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa48064500 con 0x7faa480b08a0 2022-07-21T01:42:04.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa44d99700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa2c053a70 0x7faa2c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa44d99700 1 -- 172.21.15.3:0/146764274 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5600+0+0 (secure 0 0 0) 0x7faa3006ca20 con 0x7faa480b08a0 2022-07-21T01:42:04.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa46d9d700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa2c053a70 0x7faa2c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.363+0000 7faa46d9d700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa2c053a70 0x7faa2c055f30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7faa3800b000 tx=0x7faa38008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.371+0000 7faa44d99700 1 -- 172.21.15.3:0/146764274 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faa30043a10 con 0x7faa480b08a0 2022-07-21T01:42:04.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.499+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7faa480574e0 con 0x7faa480b08a0 2022-07-21T01:42:04.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.499+0000 7faa44d99700 1 -- 172.21.15.3:0/146764274 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v136) v1 ==== 92+0+10 (secure 0 0 0) 0x7faa30023070 con 0x7faa480b08a0 2022-07-21T01:42:04.504 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:04.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa2c053a70 msgr2=0x7faa2c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa2c053a70 0x7faa2c055f30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7faa3800b000 tx=0x7faa38008040).stop 2022-07-21T01:42:04.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b08a0 msgr2=0x7faa48140ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b08a0 0x7faa48140ce0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7faa30002f70 tx=0x7faa30014000).stop 2022-07-21T01:42:04.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 shutdown_connections 2022-07-21T01:42:04.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faa2c053a70 0x7faa2c055f30 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa480b3260 0x7faa480676e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa480b08a0 0x7faa48140ce0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 --2- 172.21.15.3:0/146764274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa480ad000 0x7faa48063be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 >> 172.21.15.3:0/146764274 conn(0x7faa480612d0 msgr2=0x7faa480b6210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:04.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 shutdown_connections 2022-07-21T01:42:04.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.503+0000 7faa4c95c700 1 -- 172.21.15.3:0/146764274 wait complete. 2022-07-21T01:42:04.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:42:04.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbef59e700 1 Processor -- start 2022-07-21T01:42:04.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbef59e700 1 -- start start 2022-07-21T01:42:04.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbef59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffbf00b01a0 0x7ffbf00a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbef59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffbf00afd60 0x7ffbf00a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbef59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf00a7250 con 0x7ffbf00b2710 2022-07-21T01:42:04.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbef59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf00a7480 con 0x7ffbf00b01a0 2022-07-21T01:42:04.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbef59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf00a7720 con 0x7ffbf00afd60 2022-07-21T01:42:04.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbedd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffbf00b01a0 0x7ffbf00a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbedd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7ffbf00b01a0 0x7ffbf00a8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50524/0 (socket says 172.21.15.3:50524) 2022-07-21T01:42:04.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbedd9b700 1 -- 172.21.15.3:0/2072720037 learned_addr learned my addr 172.21.15.3:0/2072720037 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:04.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.619+0000 7ffbeed9d700 1 --2- 172.21.15.3:0/2072720037 >> v2:172.21.15.143:3300/0 conn(0x7ffbf00afd60 0x7ffbf00a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbedd9b700 1 -- 172.21.15.3:0/2072720037 >> v2:172.21.15.143:3300/0 conn(0x7ffbf00afd60 msgr2=0x7ffbf00a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:04.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbedd9b700 1 --2- 172.21.15.3:0/2072720037 >> v2:172.21.15.143:3300/0 conn(0x7ffbf00afd60 0x7ffbf00a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbedd9b700 1 -- 172.21.15.3:0/2072720037 >> v1:172.21.15.143:6789/0 conn(0x7ffbf00b2710 legacy=0x7ffbf014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbedd9b700 1 -- 172.21.15.3:0/2072720037 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf00a79c0 con 0x7ffbf00b01a0 2022-07-21T01:42:04.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbedd9b700 1 --2- 172.21.15.3:0/2072720037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b01a0 0x7ffbf00a8540 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7ffbe00032f0 tx=0x7ffbe0009290).ready entity=mon.0 client_cookie=56ae1c7f26eadade server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbed59a700 1 -- 172.21.15.3:0/2072720037 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbe000b870 con 0x7ffbf00b01a0 2022-07-21T01:42:04.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbed59a700 1 -- 172.21.15.3:0/2072720037 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe000b9d0 con 0x7ffbf00b01a0 2022-07-21T01:42:04.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbed59a700 1 -- 172.21.15.3:0/2072720037 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbe000bb40 con 0x7ffbf00b01a0 2022-07-21T01:42:04.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 -- 172.21.15.3:0/2072720037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b01a0 msgr2=0x7ffbf00a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 --2- 172.21.15.3:0/2072720037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b01a0 0x7ffbf00a8540 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7ffbe00032f0 tx=0x7ffbe0009290).stop 2022-07-21T01:42:04.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 -- 172.21.15.3:0/2072720037 shutdown_connections 2022-07-21T01:42:04.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 --2- 172.21.15.3:0/2072720037 >> v2:172.21.15.143:3300/0 conn(0x7ffbf00afd60 0x7ffbf00a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 --2- 172.21.15.3:0/2072720037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b01a0 0x7ffbf00a8540 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 -- 172.21.15.3:0/2072720037 >> 172.21.15.3:0/2072720037 conn(0x7ffbf0060310 msgr2=0x7ffbf00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:04.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 -- 172.21.15.3:0/2072720037 shutdown_connections 2022-07-21T01:42:04.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 -- 172.21.15.3:0/2072720037 wait complete. 2022-07-21T01:42:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 Processor -- start 2022-07-21T01:42:04.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.623+0000 7ffbef59e700 1 -- start start 2022-07-21T01:42:04.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbef59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf00afd60 0x7ffbf00634d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbef59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbf00b01a0 0x7ffbf0066e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbef59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b2710 0x7ffbf0067320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbef59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf00ac0a0 con 0x7ffbf00b2710 2022-07-21T01:42:04.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbef59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf00abe00 con 0x7ffbf00afd60 2022-07-21T01:42:04.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbef59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf00b37a0 con 0x7ffbf00b01a0 2022-07-21T01:42:04.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbeed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b2710 0x7ffbf0067320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbeed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b2710 0x7ffbf0067320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50534/0 (socket says 172.21.15.3:50534) 2022-07-21T01:42:04.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbeed9d700 1 -- 172.21.15.3:0/2240698107 learned_addr learned my addr 172.21.15.3:0/2240698107 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:04.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbedd9b700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbf00b01a0 0x7ffbf0066e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbee59c700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf00afd60 0x7ffbf00634d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbeed9d700 1 -- 172.21.15.3:0/2240698107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbf00b01a0 msgr2=0x7ffbf0066e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbeed9d700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbf00b01a0 0x7ffbf0066e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbeed9d700 1 -- 172.21.15.3:0/2240698107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf00afd60 msgr2=0x7ffbf00634d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbeed9d700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf00afd60 0x7ffbf00634d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbeed9d700 1 -- 172.21.15.3:0/2240698107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe0009530 con 0x7ffbf00b2710 2022-07-21T01:42:04.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbeed9d700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b2710 0x7ffbf0067320 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7ffbe800c000 tx=0x7ffbe8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbdf7fe700 1 -- 172.21.15.3:0/2240698107 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbe8011750 con 0x7ffbf00b2710 2022-07-21T01:42:04.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf015a810 con 0x7ffbf00b2710 2022-07-21T01:42:04.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbdf7fe700 1 -- 172.21.15.3:0/2240698107 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe80118b0 con 0x7ffbf00b2710 2022-07-21T01:42:04.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbdf7fe700 1 -- 172.21.15.3:0/2240698107 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbe8011c10 con 0x7ffbf00b2710 2022-07-21T01:42:04.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf00aae40 con 0x7ffbf00b2710 2022-07-21T01:42:04.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf00b39a0 con 0x7ffbf00b2710 2022-07-21T01:42:04.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.627+0000 7ffbdf7fe700 1 -- 172.21.15.3:0/2240698107 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ffbe8011d70 con 0x7ffbf00b2710 2022-07-21T01:42:04.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.631+0000 7ffbdf7fe700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffbcc0539e0 0x7ffbcc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.631+0000 7ffbdf7fe700 1 -- 172.21.15.3:0/2240698107 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5600+0+0 (secure 0 0 0) 0x7ffbe806c240 con 0x7ffbf00b2710 2022-07-21T01:42:04.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.631+0000 7ffbee59c700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffbcc0539e0 0x7ffbcc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.631+0000 7ffbee59c700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffbcc0539e0 0x7ffbcc055ea0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7ffbe400d000 tx=0x7ffbe4007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.635+0000 7ffbdf7fe700 1 -- 172.21.15.3:0/2240698107 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffbe80431b0 con 0x7ffbf00b2710 2022-07-21T01:42:04.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ffbf00abba0 con 0x7ffbf00b2710 2022-07-21T01:42:04.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbdf7fe700 1 -- 172.21.15.3:0/2240698107 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v136) v1 ==== 84+0+143 (secure 0 0 0) 0x7ffbe80467d0 con 0x7ffbf00b2710 2022-07-21T01:42:04.770 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:04.770 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:42:04.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:04.770 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:42:04.771 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:42:04.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:04.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:04.771 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:42:04.772 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:42:04.772 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:04.772 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:42:04.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffbcc0539e0 msgr2=0x7ffbcc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffbcc0539e0 0x7ffbcc055ea0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7ffbe400d000 tx=0x7ffbe4007b70).stop 2022-07-21T01:42:04.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b2710 msgr2=0x7ffbf0067320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b2710 0x7ffbf0067320 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7ffbe800c000 tx=0x7ffbe8006df0).stop 2022-07-21T01:42:04.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 shutdown_connections 2022-07-21T01:42:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbf00b2710 0x7ffbf0067320 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbf00b01a0 0x7ffbf0066e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ffbcc0539e0 0x7ffbcc055ea0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 --2- 172.21.15.3:0/2240698107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf00afd60 0x7ffbf00634d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 >> 172.21.15.3:0/2240698107 conn(0x7ffbf0060310 msgr2=0x7ffbf0146b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:04.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 shutdown_connections 2022-07-21T01:42:04.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.767+0000 7ffbef59e700 1 -- 172.21.15.3:0/2240698107 wait complete. 2022-07-21T01:42:04.784 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:42:04.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:42:04.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.891+0000 7f824d737700 1 Processor -- start 2022-07-21T01:42:04.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.891+0000 7f824d737700 1 -- start start 2022-07-21T01:42:04.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f824d737700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82480b5ed0 0x7f82480b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f824d737700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8248061d90 con 0x7f82480b6ee0 2022-07-21T01:42:04.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f824d737700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82480ba5a0 con 0x7f82480619b0 2022-07-21T01:42:04.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f824d737700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8248143fa0 con 0x7f82480b5ed0 2022-07-21T01:42:04.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f82477fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f82480b6ee0 0x7f82480b5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58822/0 (socket says 172.21.15.3:58822) 2022-07-21T01:42:04.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f82477fe700 1 -- 172.21.15.3:0/39946396 learned_addr learned my addr 172.21.15.3:0/39946396 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:04.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f824cf36700 1 --2- 172.21.15.3:0/39946396 >> v2:172.21.15.143:3300/0 conn(0x7f82480b5ed0 0x7f82480b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530828335 0 0) 0x7f8248061d90 con 0x7f82480b6ee0 2022-07-21T01:42:04.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8234003260 con 0x7f82480b6ee0 2022-07-21T01:42:04.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 70450325 0 0) 0x7f8234003260 con 0x7f82480b6ee0 2022-07-21T01:42:04.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 >> v2:172.21.15.143:3300/0 conn(0x7f82480b5ed0 msgr2=0x7f82480b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:04.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 --2- 172.21.15.3:0/39946396 >> v2:172.21.15.143:3300/0 conn(0x7f82480b5ed0 0x7f82480b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 >> v1:172.21.15.143:6789/0 conn(0x7f82480619b0 legacy=0x7f8248143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8248143150 con 0x7f82480b6ee0 2022-07-21T01:42:04.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f8238001680 con 0x7f82480b6ee0 2022-07-21T01:42:04.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 >> v1:172.21.15.3:6789/0 conn(0x7f82480b6ee0 legacy=0x7f82480b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8234003750 0x7f8234003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f8234004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8234004e70 0x7f823400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8248143fa0 con 0x7f8234004500 2022-07-21T01:42:04.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823400bd20 con 0x7f8234004e70 2022-07-21T01:42:04.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823400bee0 con 0x7f8234003750 2022-07-21T01:42:04.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f82477fe700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f8234004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f824cf36700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8234004e70 0x7f823400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8247fff700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8234003750 0x7f8234003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f82477fe700 1 -- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8234003750 msgr2=0x7f8234003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:04.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f82477fe700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8234003750 0x7f8234003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f82477fe700 1 -- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8234004e70 msgr2=0x7f823400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:04.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f82477fe700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8234004e70 0x7f823400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f82477fe700 1 -- 172.21.15.3:0/39946396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8234003260 con 0x7f8234004500 2022-07-21T01:42:04.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f824cf36700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8234004e70 0x7f823400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:04.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f824cf36700 1 -- 172.21.15.3:0/39946396 reap_dead start 2022-07-21T01:42:04.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8247fff700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8234003750 0x7f8234003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:04.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f82477fe700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f8234004980 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f8238001390 tx=0x7f8248067890).ready entity=mon.0 client_cookie=e2f40ab3fc08eef7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8238007170 con 0x7f8234004500 2022-07-21T01:42:04.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82380072d0 con 0x7f8234004500 2022-07-21T01:42:04.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.895+0000 7f8246ffd700 1 -- 172.21.15.3:0/39946396 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f823800c950 con 0x7f8234004500 2022-07-21T01:42:04.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 -- 172.21.15.3:0/39946396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 msgr2=0x7f8234004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f8234004980 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f8238001390 tx=0x7f8248067890).stop 2022-07-21T01:42:04.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 -- 172.21.15.3:0/39946396 shutdown_connections 2022-07-21T01:42:04.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 --2- 172.21.15.3:0/39946396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f8234004980 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 -- 172.21.15.3:0/39946396 >> 172.21.15.3:0/39946396 conn(0x7f824806af80 msgr2=0x7f824805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:04.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 -- 172.21.15.3:0/39946396 shutdown_connections 2022-07-21T01:42:04.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 -- 172.21.15.3:0/39946396 wait complete. 2022-07-21T01:42:04.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 Processor -- start 2022-07-21T01:42:04.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 -- start start 2022-07-21T01:42:04.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f82481420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8248146410 0x7f8248112d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8248146830 0x7f8248113230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824813ee50 con 0x7f8234004500 2022-07-21T01:42:04.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82481473c0 con 0x7f8248146410 2022-07-21T01:42:04.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f824d737700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8248113720 con 0x7f8248146830 2022-07-21T01:42:04.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.899+0000 7f8247fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f82481420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f82477fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8248146410 0x7f8248112d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f8247fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f82481420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50550/0 (socket says 172.21.15.3:50550) 2022-07-21T01:42:04.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f8247fff700 1 -- 172.21.15.3:0/4099330988 learned_addr learned my addr 172.21.15.3:0/4099330988 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:04.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f824cf36700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8248146830 0x7f8248113230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f8247fff700 1 -- 172.21.15.3:0/4099330988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8248146830 msgr2=0x7f8248113230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f8247fff700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8248146830 0x7f8248113230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f8247fff700 1 -- 172.21.15.3:0/4099330988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8248146410 msgr2=0x7f8248112d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:04.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f8247fff700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8248146410 0x7f8248112d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:04.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f8247fff700 1 -- 172.21.15.3:0/4099330988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8248145db0 con 0x7f8234004500 2022-07-21T01:42:04.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f8247fff700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f82481420d0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f823c00b000 tx=0x7f823c009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f82457fa700 1 -- 172.21.15.3:0/4099330988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f823c014620 con 0x7f8234004500 2022-07-21T01:42:04.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8248147120 con 0x7f8234004500 2022-07-21T01:42:04.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f82457fa700 1 -- 172.21.15.3:0/4099330988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c014780 con 0x7f8234004500 2022-07-21T01:42:04.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f82457fa700 1 -- 172.21.15.3:0/4099330988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f823c014ae0 con 0x7f8234004500 2022-07-21T01:42:04.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8248155c40 con 0x7f8234004500 2022-07-21T01:42:04.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82480574e0 con 0x7f8234004500 2022-07-21T01:42:04.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.903+0000 7f82457fa700 1 -- 172.21.15.3:0/4099330988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f823c014c40 con 0x7f8234004500 2022-07-21T01:42:04.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.907+0000 7f82457fa700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8228053a70 0x7f8228055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:04.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.907+0000 7f82457fa700 1 -- 172.21.15.3:0/4099330988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f823c06c140 con 0x7f8234004500 2022-07-21T01:42:04.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.907+0000 7f82477fe700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8228053a70 0x7f8228055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:04.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.907+0000 7f82477fe700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8228053a70 0x7f8228055f30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f823801a000 tx=0x7f8238013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:04.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:04.915+0000 7f82457fa700 1 -- 172.21.15.3:0/4099330988 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f823c0430b0 con 0x7f8234004500 2022-07-21T01:42:05.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.039+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f824805db60 con 0x7f8234004500 2022-07-21T01:42:05.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.727+0000 7f82457fa700 1 -- 172.21.15.3:0/4099330988 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v137) v1 ==== 133+0+0 (secure 0 0 0) 0x7f823c0466d0 con 0x7f8234004500 2022-07-21T01:42:05.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8228053a70 msgr2=0x7f8228055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:05.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8228053a70 0x7f8228055f30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f823801a000 tx=0x7f8238013000).stop 2022-07-21T01:42:05.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 msgr2=0x7f82481420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:05.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f82481420d0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f823c00b000 tx=0x7f823c009040).stop 2022-07-21T01:42:05.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 shutdown_connections 2022-07-21T01:42:05.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8248146830 0x7f8248113230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:05.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8228053a70 0x7f8228055f30 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:05.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8248146410 0x7f8248112d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:05.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 --2- 172.21.15.3:0/4099330988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8234004500 0x7f82481420d0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:05.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 >> 172.21.15.3:0/4099330988 conn(0x7f824806af80 msgr2=0x7f824805fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:05.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 shutdown_connections 2022-07-21T01:42:05.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.731+0000 7f824d737700 1 -- 172.21.15.3:0/4099330988 wait complete. 2022-07-21T01:42:05.740 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:42:05.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:42:05.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f900a19b700 1 Processor -- start 2022-07-21T01:42:05.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f900a19b700 1 -- start start 2022-07-21T01:42:05.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f900a19b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90041445d0 0x7f9004144b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:05.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f900a19b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f90040655b0 0x7f9004144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:05.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f900a19b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90041454e0 con 0x7f9004060290 2022-07-21T01:42:05.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f900a19b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9004065e10 con 0x7f90040655b0 2022-07-21T01:42:05.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f900a19b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9004060670 con 0x7f90041445d0 2022-07-21T01:42:05.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f9003fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f90040655b0 0x7f9004144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:05.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f90037fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9004060290 0x7f9004064ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58838/0 (socket says 172.21.15.3:58838) 2022-07-21T01:42:05.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f90037fe700 1 -- 172.21.15.3:0/1882240396 learned_addr learned my addr 172.21.15.3:0/1882240396 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:05.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f8ffbfff700 1 --2- 172.21.15.3:0/1882240396 >> v2:172.21.15.143:3300/0 conn(0x7f90041445d0 0x7f9004144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:05.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f9002ffd700 1 -- 172.21.15.3:0/1882240396 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224963724 0 0) 0x7f90041454e0 con 0x7f9004060290 2022-07-21T01:42:05.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f9002ffd700 1 -- 172.21.15.3:0/1882240396 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe4003260 con 0x7f9004060290 2022-07-21T01:42:05.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f9003fff700 1 -- 172.21.15.3:0/1882240396 >> v2:172.21.15.143:3300/0 conn(0x7f90041445d0 msgr2=0x7f9004144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:05.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f9002ffd700 1 -- 172.21.15.3:0/1882240396 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2883334524 0 0) 0x7f8fe4003260 con 0x7f9004060290 2022-07-21T01:42:05.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f9003fff700 1 --2- 172.21.15.3:0/1882240396 >> v2:172.21.15.143:3300/0 conn(0x7f90041445d0 0x7f9004144b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:05.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f9003fff700 1 -- 172.21.15.3:0/1882240396 >> v1:172.21.15.3:6789/0 conn(0x7f9004060290 legacy=0x7f9004064ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:05.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f9003fff700 1 -- 172.21.15.3:0/1882240396 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90040b2540 con 0x7f90040655b0 2022-07-21T01:42:05.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.863+0000 7f9003fff700 1 --2- 172.21.15.3:0/1882240396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90040655b0 0x7f9004144ff0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f8ff4001800 tx=0x7f8ffc0091d0).ready entity=mon.0 client_cookie=d0e41a349b5bd76f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:05.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f9002ffd700 1 -- 172.21.15.3:0/1882240396 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ff4001660 con 0x7f90040655b0 2022-07-21T01:42:05.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f9002ffd700 1 -- 172.21.15.3:0/1882240396 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc00f510 con 0x7f90040655b0 2022-07-21T01:42:05.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f9002ffd700 1 -- 172.21.15.3:0/1882240396 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ffc00f680 con 0x7f90040655b0 2022-07-21T01:42:05.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 -- 172.21.15.3:0/1882240396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90040655b0 msgr2=0x7f9004144ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:05.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 --2- 172.21.15.3:0/1882240396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90040655b0 0x7f9004144ff0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f8ff4001800 tx=0x7f8ffc0091d0).stop 2022-07-21T01:42:05.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 -- 172.21.15.3:0/1882240396 shutdown_connections 2022-07-21T01:42:05.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 --2- 172.21.15.3:0/1882240396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90040655b0 0x7f9004144ff0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:05.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 --2- 172.21.15.3:0/1882240396 >> v2:172.21.15.143:3300/0 conn(0x7f90041445d0 0x7f9004144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:05.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 -- 172.21.15.3:0/1882240396 >> 172.21.15.3:0/1882240396 conn(0x7f900405d0f0 msgr2=0x7f900405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:05.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 -- 172.21.15.3:0/1882240396 shutdown_connections 2022-07-21T01:42:05.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 -- 172.21.15.3:0/1882240396 wait complete. 2022-07-21T01:42:05.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 Processor -- start 2022-07-21T01:42:05.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.867+0000 7f900a19b700 1 -- start start 2022-07-21T01:42:05.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f900a19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9004060290 0x7f9004061130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:05.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f900a19b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90040655b0 0x7f900413c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:05.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f900a19b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90040b1710 0x7f9004146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:05.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f900a19b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900413ec80 con 0x7f9004060290 2022-07-21T01:42:05.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f900a19b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040b01d0 con 0x7f90040b1710 2022-07-21T01:42:05.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f900a19b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9004061c70 con 0x7f90040655b0 2022-07-21T01:42:05.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90037fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9004060290 0x7f9004061130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:05.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90037fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9004060290 0x7f9004061130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50566/0 (socket says 172.21.15.3:50566) 2022-07-21T01:42:05.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90037fe700 1 -- 172.21.15.3:0/3438703213 learned_addr learned my addr 172.21.15.3:0/3438703213 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:05.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f9003fff700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90040b1710 0x7f9004146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:05.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f8ffbfff700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90040655b0 0x7f900413c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:05.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90037fe700 1 -- 172.21.15.3:0/3438703213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90040655b0 msgr2=0x7f900413c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:05.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90037fe700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90040655b0 0x7f900413c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:05.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90037fe700 1 -- 172.21.15.3:0/3438703213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90040b1710 msgr2=0x7f9004146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:05.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90037fe700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90040b1710 0x7f9004146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:05.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90037fe700 1 -- 172.21.15.3:0/3438703213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc009470 con 0x7f9004060290 2022-07-21T01:42:05.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90037fe700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9004060290 0x7f9004061130 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f8ff4001840 tx=0x7f8ff4001a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:05.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90017fa700 1 -- 172.21.15.3:0/3438703213 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ff4007170 con 0x7f9004060290 2022-07-21T01:42:05.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90017fa700 1 -- 172.21.15.3:0/3438703213 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff40072d0 con 0x7f9004060290 2022-07-21T01:42:05.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f90017fa700 1 -- 172.21.15.3:0/3438703213 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ff4004920 con 0x7f9004060290 2022-07-21T01:42:05.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900413e740 con 0x7f9004060290 2022-07-21T01:42:05.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90040afba0 con 0x7f9004060290 2022-07-21T01:42:05.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.871+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004060e70 con 0x7f9004060290 2022-07-21T01:42:05.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.875+0000 7f90017fa700 1 -- 172.21.15.3:0/3438703213 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8ff4004a80 con 0x7f9004060290 2022-07-21T01:42:05.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.875+0000 7f90017fa700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fe8053aa0 0x7f8fe8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:05.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.875+0000 7f90017fa700 1 -- 172.21.15.3:0/3438703213 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f8ff406bfe0 con 0x7f9004060290 2022-07-21T01:42:05.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.875+0000 7f8ffbfff700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fe8053aa0 0x7f8fe8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:05.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.875+0000 7f8ffbfff700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fe8053aa0 0x7f8fe8055f60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f8fec00b000 tx=0x7f8fec006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:05.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:05.879+0000 7f90017fa700 1 -- 172.21.15.3:0/3438703213 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8ff4042e00 con 0x7f9004060290 2022-07-21T01:42:06.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.003+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f9004060e70 con 0x7f9004060290 2022-07-21T01:42:06.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.739+0000 7f90017fa700 1 -- 172.21.15.3:0/3438703213 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v138) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9004060e70 con 0x7f9004060290 2022-07-21T01:42:06.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.743+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fe8053aa0 msgr2=0x7f8fe8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:06.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.743+0000 7f900a19b700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fe8053aa0 0x7f8fe8055f60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f8fec00b000 tx=0x7f8fec006df0).stop 2022-07-21T01:42:06.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.743+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9004060290 msgr2=0x7f9004061130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:06.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.743+0000 7f900a19b700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9004060290 0x7f9004061130 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f8ff4001840 tx=0x7f8ff4001a80).stop 2022-07-21T01:42:06.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.751+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 shutdown_connections 2022-07-21T01:42:06.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.751+0000 7f900a19b700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8fe8053aa0 0x7f8fe8055f60 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:06.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.751+0000 7f900a19b700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90040b1710 0x7f9004146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:06.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.751+0000 7f900a19b700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90040655b0 0x7f900413c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:06.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.751+0000 7f900a19b700 1 --2- 172.21.15.3:0/3438703213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9004060290 0x7f9004061130 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:06.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.751+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 >> 172.21.15.3:0/3438703213 conn(0x7f900405d0f0 msgr2=0x7f900413f100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:06.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.751+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 shutdown_connections 2022-07-21T01:42:06.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.751+0000 7f900a19b700 1 -- 172.21.15.3:0/3438703213 wait complete. 2022-07-21T01:42:06.759 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:42:06.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:42:06.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c2adab700 1 Processor -- start 2022-07-21T01:42:06.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c2adab700 1 -- start start 2022-07-21T01:42:06.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c2adab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c240614b0 0x7f1c24061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:06.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c2adab700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1c240ad750 0x7f1c240b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:06.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c2adab700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240aee80 con 0x7f1c241405d0 2022-07-21T01:42:06.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c2adab700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c240b70b0 con 0x7f1c240ad750 2022-07-21T01:42:06.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c2adab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240af0b0 con 0x7f1c240614b0 2022-07-21T01:42:06.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c295a8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1c240ad750 0x7f1c240b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:06.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c295a8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1c240ad750 0x7f1c240b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50576/0 (socket says 172.21.15.3:50576) 2022-07-21T01:42:06.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c295a8700 1 -- 172.21.15.3:0/3440991196 learned_addr learned my addr 172.21.15.3:0/3440991196 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:06.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.871+0000 7f1c29da9700 1 --2- 172.21.15.3:0/3440991196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c240614b0 0x7f1c24061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:06.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c295a8700 1 -- 172.21.15.3:0/3440991196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c240614b0 msgr2=0x7f1c24061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:06.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c295a8700 1 --2- 172.21.15.3:0/3440991196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c240614b0 0x7f1c24061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:06.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c295a8700 1 -- 172.21.15.3:0/3440991196 >> v1:172.21.15.143:6789/0 conn(0x7f1c241405d0 legacy=0x7f1c241466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c295a8700 1 -- 172.21.15.3:0/3440991196 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c240af350 con 0x7f1c240ad750 2022-07-21T01:42:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c295a8700 1 --2- 172.21.15.3:0/3440991196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 0x7f1c240b9ff0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f1c18002f70 tx=0x7f1c18006eb0).ready entity=mon.0 client_cookie=4f73a54a2503fd7c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c28da7700 1 -- 172.21.15.3:0/3440991196 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c1800d570 con 0x7f1c240ad750 2022-07-21T01:42:06.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c28da7700 1 -- 172.21.15.3:0/3440991196 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c18000e90 con 0x7f1c240ad750 2022-07-21T01:42:06.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c28da7700 1 -- 172.21.15.3:0/3440991196 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c1800d900 con 0x7f1c240ad750 2022-07-21T01:42:06.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 -- 172.21.15.3:0/3440991196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 msgr2=0x7f1c240b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:06.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 --2- 172.21.15.3:0/3440991196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 0x7f1c240b9ff0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f1c18002f70 tx=0x7f1c18006eb0).stop 2022-07-21T01:42:06.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 -- 172.21.15.3:0/3440991196 shutdown_connections 2022-07-21T01:42:06.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 --2- 172.21.15.3:0/3440991196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 0x7f1c240b9ff0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:06.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 --2- 172.21.15.3:0/3440991196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c240614b0 0x7f1c24061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:06.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 -- 172.21.15.3:0/3440991196 >> 172.21.15.3:0/3440991196 conn(0x7f1c24066f70 msgr2=0x7f1c24067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:06.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 -- 172.21.15.3:0/3440991196 shutdown_connections 2022-07-21T01:42:06.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 -- 172.21.15.3:0/3440991196 wait complete. 2022-07-21T01:42:06.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 Processor -- start 2022-07-21T01:42:06.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.875+0000 7f1c2adab700 1 -- start start 2022-07-21T01:42:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c2adab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c240614b0 0x7f1c240b0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c2adab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 0x7f1c240b1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c2adab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c241405d0 0x7f1c2413f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c2adab700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c241559f0 con 0x7f1c240ad750 2022-07-21T01:42:06.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c2adab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240b1090 con 0x7f1c240614b0 2022-07-21T01:42:06.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c2adab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24155c20 con 0x7f1c241405d0 2022-07-21T01:42:06.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c295a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 0x7f1c240b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:06.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c295a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 0x7f1c240b1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50582/0 (socket says 172.21.15.3:50582) 2022-07-21T01:42:06.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c295a8700 1 -- 172.21.15.3:0/2221819826 learned_addr learned my addr 172.21.15.3:0/2221819826 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:06.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c29da9700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c240614b0 0x7f1c240b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:06.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c2a5aa700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c241405d0 0x7f1c2413f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:06.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c295a8700 1 -- 172.21.15.3:0/2221819826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c241405d0 msgr2=0x7f1c2413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:06.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c295a8700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c241405d0 0x7f1c2413f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:06.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c295a8700 1 -- 172.21.15.3:0/2221819826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c240614b0 msgr2=0x7f1c240b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:06.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c295a8700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c240614b0 0x7f1c240b0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:06.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c295a8700 1 -- 172.21.15.3:0/2221819826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1800b1e0 con 0x7f1c240ad750 2022-07-21T01:42:06.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c295a8700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 0x7f1c240b1790 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7f1c18002f70 tx=0x7f1c18015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:06.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c177fe700 1 -- 172.21.15.3:0/2221819826 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c180068b0 con 0x7f1c240ad750 2022-07-21T01:42:06.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c177fe700 1 -- 172.21.15.3:0/2221819826 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1800de30 con 0x7f1c240ad750 2022-07-21T01:42:06.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c240b12f0 con 0x7f1c240ad750 2022-07-21T01:42:06.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c177fe700 1 -- 172.21.15.3:0/2221819826 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c18011e20 con 0x7f1c240ad750 2022-07-21T01:42:06.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.879+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c240b2190 con 0x7f1c240ad750 2022-07-21T01:42:06.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.883+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c240b1fc0 con 0x7f1c240ad750 2022-07-21T01:42:06.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.883+0000 7f1c177fe700 1 -- 172.21.15.3:0/2221819826 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1c18025030 con 0x7f1c240ad750 2022-07-21T01:42:06.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.883+0000 7f1c177fe700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c00053a70 0x7f1c00055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:06.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.883+0000 7f1c177fe700 1 -- 172.21.15.3:0/2221819826 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f1c1806d0f0 con 0x7f1c240ad750 2022-07-21T01:42:06.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.883+0000 7f1c29da9700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c00053a70 0x7f1c00055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:06.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.883+0000 7f1c29da9700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c00053a70 0x7f1c00055f30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f1c2000b000 tx=0x7f1c20006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:06.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:06.887+0000 7f1c177fe700 1 -- 172.21.15.3:0/2221819826 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1c18043fb0 con 0x7f1c240ad750 2022-07-21T01:42:07.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.059+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1c240af5f0 con 0x7f1c240ad750 2022-07-21T01:42:07.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.067+0000 7f1c177fe700 1 -- 172.21.15.3:0/2221819826 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 25 and data pool 26 v70) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1c18023020 con 0x7f1c240ad750 2022-07-21T01:42:07.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c00053a70 msgr2=0x7f1c00055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c00053a70 0x7f1c00055f30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f1c2000b000 tx=0x7f1c20006df0).stop 2022-07-21T01:42:07.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 msgr2=0x7f1c240b1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 0x7f1c240b1790 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7f1c18002f70 tx=0x7f1c18015000).stop 2022-07-21T01:42:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 shutdown_connections 2022-07-21T01:42:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c241405d0 0x7f1c2413f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c240ad750 0x7f1c240b1790 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c00053a70 0x7f1c00055f30 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 --2- 172.21.15.3:0/2221819826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c240614b0 0x7f1c240b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 >> 172.21.15.3:0/2221819826 conn(0x7f1c24066f70 msgr2=0x7f1c24067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 shutdown_connections 2022-07-21T01:42:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.071+0000 7f1c2adab700 1 -- 172.21.15.3:0/2221819826 wait complete. 2022-07-21T01:42:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 25 and data pool 26 2022-07-21T01:42:07.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:07.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5869700 1 Processor -- start 2022-07-21T01:42:07.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5869700 1 -- start start 2022-07-21T01:42:07.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5869700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b65c0 0x7fb5b0061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5869700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5b00b41f0 0x7fb5b0061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5869700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5b0147d80 0x7fb5b0144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5869700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b00b1630 con 0x7fb5b00b41f0 2022-07-21T01:42:07.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5869700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b0061fe0 con 0x7fb5b0147d80 2022-07-21T01:42:07.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5869700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b00b2100 con 0x7fb5b00b65c0 2022-07-21T01:42:07.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5affff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5b00b41f0 0x7fb5b0061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5affff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5b00b41f0 0x7fb5b0061af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50600/0 (socket says 172.21.15.3:50600) 2022-07-21T01:42:07.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5affff700 1 -- 172.21.15.3:0/970128937 learned_addr learned my addr 172.21.15.3:0/970128937 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:07.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5068700 1 --2- 172.21.15.3:0/970128937 >> v2:172.21.15.143:3300/0 conn(0x7fb5b0147d80 0x7fb5b0144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b4867700 1 --2- 172.21.15.3:0/970128937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b65c0 0x7fb5b0061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5affff700 1 -- 172.21.15.3:0/970128937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b65c0 msgr2=0x7fb5b0061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:07.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5affff700 1 --2- 172.21.15.3:0/970128937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b65c0 0x7fb5b0061600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5affff700 1 -- 172.21.15.3:0/970128937 >> v2:172.21.15.143:3300/0 conn(0x7fb5b0147d80 msgr2=0x7fb5b0144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5affff700 1 --2- 172.21.15.3:0/970128937 >> v2:172.21.15.143:3300/0 conn(0x7fb5b0147d80 0x7fb5b0144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5affff700 1 -- 172.21.15.3:0/970128937 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b00b69a0 con 0x7fb5b00b41f0 2022-07-21T01:42:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5b5068700 1 --2- 172.21.15.3:0/970128937 >> v2:172.21.15.143:3300/0 conn(0x7fb5b0147d80 0x7fb5b0144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5affff700 1 --2- 172.21.15.3:0/970128937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b00b41f0 0x7fb5b0061af0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7fb5b0062120 tx=0x7fb5a4006eb0).ready entity=mon.0 client_cookie=ad69d808ca278b06 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:07.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5af7fe700 1 -- 172.21.15.3:0/970128937 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a400d570 con 0x7fb5b00b41f0 2022-07-21T01:42:07.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5af7fe700 1 -- 172.21.15.3:0/970128937 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a4000e90 con 0x7fb5b00b41f0 2022-07-21T01:42:07.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.179+0000 7fb5af7fe700 1 -- 172.21.15.3:0/970128937 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a400d8c0 con 0x7fb5b00b41f0 2022-07-21T01:42:07.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 -- 172.21.15.3:0/970128937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b00b41f0 msgr2=0x7fb5b0061af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 --2- 172.21.15.3:0/970128937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b00b41f0 0x7fb5b0061af0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7fb5b0062120 tx=0x7fb5a4006eb0).stop 2022-07-21T01:42:07.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 -- 172.21.15.3:0/970128937 shutdown_connections 2022-07-21T01:42:07.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 --2- 172.21.15.3:0/970128937 >> v2:172.21.15.143:3300/0 conn(0x7fb5b0147d80 0x7fb5b0144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 --2- 172.21.15.3:0/970128937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b00b41f0 0x7fb5b0061af0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 --2- 172.21.15.3:0/970128937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b65c0 0x7fb5b0061600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 -- 172.21.15.3:0/970128937 >> 172.21.15.3:0/970128937 conn(0x7fb5b005c620 msgr2=0x7fb5b005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:07.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 -- 172.21.15.3:0/970128937 shutdown_connections 2022-07-21T01:42:07.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 -- 172.21.15.3:0/970128937 wait complete. 2022-07-21T01:42:07.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 Processor -- start 2022-07-21T01:42:07.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 -- start start 2022-07-21T01:42:07.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b41f0 0x7fb5b00ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b00b65c0 0x7fb5b01567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b0147d80 0x7fb5b0156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b013d550 con 0x7fb5b0147d80 2022-07-21T01:42:07.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b00b8570 con 0x7fb5b00b65c0 2022-07-21T01:42:07.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5869700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b00b8270 con 0x7fb5b00b41f0 2022-07-21T01:42:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5068700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b0147d80 0x7fb5b0156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5affff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b00b65c0 0x7fb5b01567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5affff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b00b65c0 0x7fb5b01567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:56054/0 (socket says 172.21.15.3:56054) 2022-07-21T01:42:07.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b4867700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b41f0 0x7fb5b00ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5affff700 1 -- 172.21.15.3:0/3700914989 learned_addr learned my addr 172.21.15.3:0/3700914989 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:07.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.183+0000 7fb5b5068700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b0147d80 0x7fb5b0156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50608/0 (socket says 172.21.15.3:50608) 2022-07-21T01:42:07.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5b5068700 1 -- 172.21.15.3:0/3700914989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b41f0 msgr2=0x7fb5b00ad900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5b5068700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b41f0 0x7fb5b00ad900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5b5068700 1 -- 172.21.15.3:0/3700914989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b00b65c0 msgr2=0x7fb5b01567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5b5068700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b00b65c0 0x7fb5b01567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5b5068700 1 -- 172.21.15.3:0/3700914989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a400b1e0 con 0x7fb5b0147d80 2022-07-21T01:42:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5affff700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b00b65c0 0x7fb5b01567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5b5068700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b0147d80 0x7fb5b0156c90 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7fb5a800a000 tx=0x7fb5a8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:07.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5adffb700 1 -- 172.21.15.3:0/3700914989 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a80106e0 con 0x7fb5b0147d80 2022-07-21T01:42:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5adffb700 1 -- 172.21.15.3:0/3700914989 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a8010840 con 0x7fb5b0147d80 2022-07-21T01:42:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b00ae280 con 0x7fb5b0147d80 2022-07-21T01:42:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5adffb700 1 -- 172.21.15.3:0/3700914989 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a8010ba0 con 0x7fb5b0147d80 2022-07-21T01:42:07.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b00addf0 con 0x7fb5b0147d80 2022-07-21T01:42:07.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b01482a0 con 0x7fb5b0147d80 2022-07-21T01:42:07.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5adffb700 1 -- 172.21.15.3:0/3700914989 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb5a801a030 con 0x7fb5b0147d80 2022-07-21T01:42:07.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.187+0000 7fb5adffb700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb590053950 0x7fb590055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.191+0000 7fb5adffb700 1 -- 172.21.15.3:0/3700914989 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb5a806be20 con 0x7fb5b0147d80 2022-07-21T01:42:07.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.191+0000 7fb5b4867700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb590053950 0x7fb590055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.191+0000 7fb5b4867700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb590053950 0x7fb590055e10 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fb5a000c000 tx=0x7fb5a0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:07.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.195+0000 7fb5adffb700 1 -- 172.21.15.3:0/3700914989 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5a8042b80 con 0x7fb5b0147d80 2022-07-21T01:42:07.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.319+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5b00574e0 con 0x7fb5b0147d80 2022-07-21T01:42:07.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5adffb700 1 -- 172.21.15.3:0/3700914989 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16298 (secure 0 0 0) 0x7fb5a80461a0 con 0x7fb5b0147d80 2022-07-21T01:42:07.326 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:07.327 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":139,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:07.062569+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:05.096631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:06.060734+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:07.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb590053950 msgr2=0x7fb590055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb590053950 0x7fb590055e10 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fb5a000c000 tx=0x7fb5a0008040).stop 2022-07-21T01:42:07.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b0147d80 msgr2=0x7fb5b0156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.329 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b0147d80 0x7fb5b0156c90 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7fb5a800a000 tx=0x7fb5a8008040).stop 2022-07-21T01:42:07.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 shutdown_connections 2022-07-21T01:42:07.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b0147d80 0x7fb5b0156c90 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb590053950 0x7fb590055e10 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b00b65c0 0x7fb5b01567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 --2- 172.21.15.3:0/3700914989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b00b41f0 0x7fb5b00ad900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 >> 172.21.15.3:0/3700914989 conn(0x7fb5b005c620 msgr2=0x7fb5b00b55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:07.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 shutdown_connections 2022-07-21T01:42:07.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.323+0000 7fb5b5869700 1 -- 172.21.15.3:0/3700914989 wait complete. 2022-07-21T01:42:07.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:07.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.439+0000 7f75bc9a9700 1 Processor -- start 2022-07-21T01:42:07.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.439+0000 7f75bc9a9700 1 -- start start 2022-07-21T01:42:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f75b8066b60 0x7f75b813e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b805f8e0 con 0x7f75b8066780 2022-07-21T01:42:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b80b1630 con 0x7f75b80b1bc0 2022-07-21T01:42:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b805fb10 con 0x7f75b8066b60 2022-07-21T01:42:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b77fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f75b8066780 0x7f75b805ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58892/0 (socket says 172.21.15.3:58892) 2022-07-21T01:42:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b77fe700 1 -- 172.21.15.3:0/2973664657 learned_addr learned my addr 172.21.15.3:0/2973664657 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:07.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b7fff700 1 --2- 172.21.15.3:0/2973664657 >> v2:172.21.15.143:3300/0 conn(0x7f75b8066b60 0x7f75b813e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b67fc700 1 -- 172.21.15.3:0/2973664657 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989101797 0 0) 0x7f75b805f8e0 con 0x7f75b8066780 2022-07-21T01:42:07.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b67fc700 1 -- 172.21.15.3:0/2973664657 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a0003260 con 0x7f75b8066780 2022-07-21T01:42:07.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b7fff700 1 -- 172.21.15.3:0/2973664657 >> v1:172.21.15.143:6789/0 conn(0x7f75b80b1bc0 legacy=0x7f75b8140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b7fff700 1 -- 172.21.15.3:0/2973664657 >> v1:172.21.15.3:6789/0 conn(0x7f75b8066780 legacy=0x7f75b805ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b67fc700 1 -- 172.21.15.3:0/2973664657 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2520776610 0 0) 0x7f75a0003260 con 0x7f75b8066780 2022-07-21T01:42:07.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b7fff700 1 -- 172.21.15.3:0/2973664657 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b80b18d0 con 0x7f75b8066b60 2022-07-21T01:42:07.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b7fff700 1 --2- 172.21.15.3:0/2973664657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75b8066b60 0x7f75b813e2b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f75a4001800 tx=0x7f75b00091d0).ready entity=mon.1 client_cookie=22851b9417975b7d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:07.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b67fc700 1 -- 172.21.15.3:0/2973664657 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75a4001660 con 0x7f75b8066b60 2022-07-21T01:42:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b67fc700 1 -- 172.21.15.3:0/2973664657 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b000b580 con 0x7f75b8066b60 2022-07-21T01:42:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75b67fc700 1 -- 172.21.15.3:0/2973664657 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75b000b6f0 con 0x7f75b8066b60 2022-07-21T01:42:07.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 -- 172.21.15.3:0/2973664657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75b8066b60 msgr2=0x7f75b813e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 --2- 172.21.15.3:0/2973664657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75b8066b60 0x7f75b813e2b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f75a4001800 tx=0x7f75b00091d0).stop 2022-07-21T01:42:07.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 -- 172.21.15.3:0/2973664657 shutdown_connections 2022-07-21T01:42:07.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 --2- 172.21.15.3:0/2973664657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75b8066b60 0x7f75b813e2b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 -- 172.21.15.3:0/2973664657 >> 172.21.15.3:0/2973664657 conn(0x7f75b8063870 msgr2=0x7f75b8063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:07.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 -- 172.21.15.3:0/2973664657 shutdown_connections 2022-07-21T01:42:07.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.443+0000 7f75bc9a9700 1 -- 172.21.15.3:0/2973664657 wait complete. 2022-07-21T01:42:07.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 Processor -- start 2022-07-21T01:42:07.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 -- start start 2022-07-21T01:42:07.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75b8066780 0x7f75b8144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75b8066b60 0x7f75b80aa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75b80b1bc0 0x7f75b80aace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b814ffe0 con 0x7f75b80b1bc0 2022-07-21T01:42:07.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b8143880 con 0x7f75b8066780 2022-07-21T01:42:07.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b80ab1d0 con 0x7f75b8066b60 2022-07-21T01:42:07.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75b80b1bc0 0x7f75b80aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75b8066780 0x7f75b8144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75b80b1bc0 0x7f75b80aace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50620/0 (socket says 172.21.15.3:50620) 2022-07-21T01:42:07.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b7fff700 1 -- 172.21.15.3:0/494021489 learned_addr learned my addr 172.21.15.3:0/494021489 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:07.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b6ffd700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75b8066b60 0x7f75b80aa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b7fff700 1 -- 172.21.15.3:0/494021489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75b8066b60 msgr2=0x7f75b80aa7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b7fff700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75b8066b60 0x7f75b80aa7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b7fff700 1 -- 172.21.15.3:0/494021489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75b8066780 msgr2=0x7f75b8144840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b7fff700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75b8066780 0x7f75b8144840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b7fff700 1 -- 172.21.15.3:0/494021489 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b0009470 con 0x7f75b80b1bc0 2022-07-21T01:42:07.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b7fff700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75b80b1bc0 0x7f75b80aace0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7f75a4001800 tx=0x7f75b0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:07.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b4ff9700 1 -- 172.21.15.3:0/494021489 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75b00032d0 con 0x7f75b80b1bc0 2022-07-21T01:42:07.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b4ff9700 1 -- 172.21.15.3:0/494021489 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b0011970 con 0x7f75b80b1bc0 2022-07-21T01:42:07.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b81501d0 con 0x7f75b80b1bc0 2022-07-21T01:42:07.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75b4ff9700 1 -- 172.21.15.3:0/494021489 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75b0011cd0 con 0x7f75b80b1bc0 2022-07-21T01:42:07.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.447+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b81509b0 con 0x7f75b80b1bc0 2022-07-21T01:42:07.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.451+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b8150b70 con 0x7f75b80b1bc0 2022-07-21T01:42:07.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.451+0000 7f75b4ff9700 1 -- 172.21.15.3:0/494021489 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f75b0026030 con 0x7f75b80b1bc0 2022-07-21T01:42:07.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.451+0000 7f75b4ff9700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7594053a70 0x7f7594055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.451+0000 7f75b4ff9700 1 -- 172.21.15.3:0/494021489 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f75b006c940 con 0x7f75b80b1bc0 2022-07-21T01:42:07.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.451+0000 7f75b77fe700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7594053a70 0x7f7594055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.451+0000 7f75b77fe700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7594053a70 0x7f7594055f30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f75a400d000 tx=0x7f75a4000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:07.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.459+0000 7f75b4ff9700 1 -- 172.21.15.3:0/494021489 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75b0043700 con 0x7f75b80b1bc0 2022-07-21T01:42:07.585 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.583+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75b8150360 con 0x7f75b80b1bc0 2022-07-21T01:42:07.587 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.583+0000 7f75b4ff9700 1 -- 172.21.15.3:0/494021489 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16298 (secure 0 0 0) 0x7f75b0015090 con 0x7f75b80b1bc0 2022-07-21T01:42:07.588 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:07.589 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":139,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:07.062569+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:05.096631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:06.060734+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7594053a70 msgr2=0x7f7594055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7594053a70 0x7f7594055f30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f75a400d000 tx=0x7f75a4000bc0).stop 2022-07-21T01:42:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75b80b1bc0 msgr2=0x7f75b80aace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75b80b1bc0 0x7f75b80aace0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7f75a4001800 tx=0x7f75b0016000).stop 2022-07-21T01:42:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 shutdown_connections 2022-07-21T01:42:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75b80b1bc0 0x7f75b80aace0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75b8066b60 0x7f75b80aa7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7594053a70 0x7f7594055f30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 --2- 172.21.15.3:0/494021489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75b8066780 0x7f75b8144840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 >> 172.21.15.3:0/494021489 conn(0x7f75b8063870 msgr2=0x7f75b8140c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 shutdown_connections 2022-07-21T01:42:07.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.587+0000 7f75bc9a9700 1 -- 172.21.15.3:0/494021489 wait complete. 2022-07-21T01:42:07.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:42:07.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.711+0000 7fd7ef59e700 1 Processor -- start 2022-07-21T01:42:07.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.711+0000 7fd7ef59e700 1 -- start start 2022-07-21T01:42:07.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd7f0063a60 0x7fd7f0143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd7f0140b10 0x7fd7f0128e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f0063e40 con 0x7fd7f0061ab0 2022-07-21T01:42:07.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f013ff40 con 0x7fd7f0140b10 2022-07-21T01:42:07.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f0062180 con 0x7fd7f0063a60 2022-07-21T01:42:07.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7eed9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd7f0140b10 0x7fd7f0128e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7edd9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd7f0061ab0 0x7fd7f00b1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58908/0 (socket says 172.21.15.3:58908) 2022-07-21T01:42:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7edd9b700 1 -- 172.21.15.3:0/1038757893 learned_addr learned my addr 172.21.15.3:0/1038757893 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ee59c700 1 --2- 172.21.15.3:0/1038757893 >> v2:172.21.15.143:3300/0 conn(0x7fd7f0063a60 0x7fd7f0143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ed59a700 1 -- 172.21.15.3:0/1038757893 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020248526 0 0) 0x7fd7f0063e40 con 0x7fd7f0061ab0 2022-07-21T01:42:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ed59a700 1 -- 172.21.15.3:0/1038757893 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d4003260 con 0x7fd7f0061ab0 2022-07-21T01:42:07.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ee59c700 1 -- 172.21.15.3:0/1038757893 >> v2:172.21.15.3:3300/0 conn(0x7fd7f0140b10 msgr2=0x7fd7f0128e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:07.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ee59c700 1 --2- 172.21.15.3:0/1038757893 >> v2:172.21.15.3:3300/0 conn(0x7fd7f0140b10 0x7fd7f0128e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ee59c700 1 -- 172.21.15.3:0/1038757893 >> v1:172.21.15.3:6789/0 conn(0x7fd7f0061ab0 legacy=0x7fd7f00b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ee59c700 1 -- 172.21.15.3:0/1038757893 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f0140130 con 0x7fd7f0063a60 2022-07-21T01:42:07.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7eed9d700 1 --2- 172.21.15.3:0/1038757893 >> v2:172.21.15.3:3300/0 conn(0x7fd7f0140b10 0x7fd7f0128e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:07.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ee59c700 1 --2- 172.21.15.3:0/1038757893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f0063a60 0x7fd7f0143ee0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fd7e00032f0 tx=0x7fd7e00091d0).ready entity=mon.1 client_cookie=a406fd667db36892 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:07.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ed59a700 1 -- 172.21.15.3:0/1038757893 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e000b600 con 0x7fd7f0063a60 2022-07-21T01:42:07.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ed59a700 1 -- 172.21.15.3:0/1038757893 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e000b760 con 0x7fd7f0063a60 2022-07-21T01:42:07.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ed59a700 1 -- 172.21.15.3:0/1038757893 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e000b8d0 con 0x7fd7f0063a60 2022-07-21T01:42:07.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 -- 172.21.15.3:0/1038757893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f0063a60 msgr2=0x7fd7f0143ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 --2- 172.21.15.3:0/1038757893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f0063a60 0x7fd7f0143ee0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fd7e00032f0 tx=0x7fd7e00091d0).stop 2022-07-21T01:42:07.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 -- 172.21.15.3:0/1038757893 shutdown_connections 2022-07-21T01:42:07.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 --2- 172.21.15.3:0/1038757893 >> v2:172.21.15.3:3300/0 conn(0x7fd7f0140b10 0x7fd7f0128e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 --2- 172.21.15.3:0/1038757893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f0063a60 0x7fd7f0143ee0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 -- 172.21.15.3:0/1038757893 >> 172.21.15.3:0/1038757893 conn(0x7fd7f005c620 msgr2=0x7fd7f005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 -- 172.21.15.3:0/1038757893 shutdown_connections 2022-07-21T01:42:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.715+0000 7fd7ef59e700 1 -- 172.21.15.3:0/1038757893 wait complete. 2022-07-21T01:42:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ef59e700 1 Processor -- start 2022-07-21T01:42:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ef59e700 1 -- start start 2022-07-21T01:42:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ef59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7f0061ab0 0x7fd7f0142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ef59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f0063a60 0x7fd7f00b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ef59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7f0140b10 0x7fd7f0155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ef59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f00b0d10 con 0x7fd7f0140b10 2022-07-21T01:42:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ef59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f00b1da0 con 0x7fd7f0063a60 2022-07-21T01:42:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ef59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f00b2040 con 0x7fd7f0061ab0 2022-07-21T01:42:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7eed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7f0140b10 0x7fd7f0155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7eed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7f0140b10 0x7fd7f0155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50636/0 (socket says 172.21.15.3:50636) 2022-07-21T01:42:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7eed9d700 1 -- 172.21.15.3:0/335185454 learned_addr learned my addr 172.21.15.3:0/335185454 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7edd9b700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f0063a60 0x7fd7f00b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ee59c700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7f0061ab0 0x7fd7f0142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7eed9d700 1 -- 172.21.15.3:0/335185454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7f0061ab0 msgr2=0x7fd7f0142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7eed9d700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7f0061ab0 0x7fd7f0142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7eed9d700 1 -- 172.21.15.3:0/335185454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f0063a60 msgr2=0x7fd7f00b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7eed9d700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f0063a60 0x7fd7f00b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7eed9d700 1 -- 172.21.15.3:0/335185454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e800b180 con 0x7fd7f0140b10 2022-07-21T01:42:07.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7ee59c700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7f0061ab0 0x7fd7f0142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:07.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7eed9d700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7f0140b10 0x7fd7f0155a90 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fd7e800c000 tx=0x7fd7e8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:07.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.719+0000 7fd7df7fe700 1 -- 172.21.15.3:0/335185454 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e8010590 con 0x7fd7f0140b10 2022-07-21T01:42:07.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7df7fe700 1 -- 172.21.15.3:0/335185454 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e80106f0 con 0x7fd7f0140b10 2022-07-21T01:42:07.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e0009470 con 0x7fd7f0140b10 2022-07-21T01:42:07.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7df7fe700 1 -- 172.21.15.3:0/335185454 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e8010a50 con 0x7fd7f0140b10 2022-07-21T01:42:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f013f1b0 con 0x7fd7f0140b10 2022-07-21T01:42:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f00574e0 con 0x7fd7f0140b10 2022-07-21T01:42:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7df7fe700 1 -- 172.21.15.3:0/335185454 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd7e8010bb0 con 0x7fd7f0140b10 2022-07-21T01:42:07.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7df7fe700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7cc0539e0 0x7fd7cc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:07.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7df7fe700 1 -- 172.21.15.3:0/335185454 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd7e806c4e0 con 0x7fd7f0140b10 2022-07-21T01:42:07.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7ee59c700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7cc0539e0 0x7fd7cc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:07.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.723+0000 7fd7ee59c700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7cc0539e0 0x7fd7cc055ea0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7fd7e0019000 tx=0x7fd7e0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:07.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.731+0000 7fd7df7fe700 1 -- 172.21.15.3:0/335185454 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd7e8043200 con 0x7fd7f0140b10 2022-07-21T01:42:07.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:07.891+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd7f0067530 con 0x7fd7f0140b10 2022-07-21T01:42:08.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.067+0000 7fd7df7fe700 1 -- 172.21.15.3:0/335185454 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v71) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd7e8046820 con 0x7fd7f0140b10 2022-07-21T01:42:08.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.071+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7cc0539e0 msgr2=0x7fd7cc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.071+0000 7fd7ef59e700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7cc0539e0 0x7fd7cc055ea0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7fd7e0019000 tx=0x7fd7e0016000).stop 2022-07-21T01:42:08.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7f0140b10 msgr2=0x7fd7f0155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7f0140b10 0x7fd7f0155a90 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fd7e800c000 tx=0x7fd7e8006df0).stop 2022-07-21T01:42:08.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 shutdown_connections 2022-07-21T01:42:08.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7f0140b10 0x7fd7f0155a90 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd7cc0539e0 0x7fd7cc055ea0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f0063a60 0x7fd7f00b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 --2- 172.21.15.3:0/335185454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7f0061ab0 0x7fd7f0142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 >> 172.21.15.3:0/335185454 conn(0x7fd7f005c620 msgr2=0x7fd7f01278e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:08.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 shutdown_connections 2022-07-21T01:42:08.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.075+0000 7fd7ef59e700 1 -- 172.21.15.3:0/335185454 wait complete. 2022-07-21T01:42:08.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:08.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6883c700 1 Processor -- start 2022-07-21T01:42:08.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6883c700 1 -- start start 2022-07-21T01:42:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6883c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a6413e8d0 0x7f6a64141210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6883c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6a640b1180 0x7f6a64141700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6883c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a640aac20 con 0x7f6a640b28b0 2022-07-21T01:42:08.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6883c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a64140e20 con 0x7f6a640b1180 2022-07-21T01:42:08.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6883c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a640b2c90 con 0x7f6a6413e8d0 2022-07-21T01:42:08.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6a640b1180 0x7f6a64141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6a640b1180 0x7f6a64141700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50644/0 (socket says 172.21.15.3:50644) 2022-07-21T01:42:08.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6359e700 1 -- 172.21.15.3:0/1265868030 learned_addr learned my addr 172.21.15.3:0/1265868030 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:08.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a62d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6a640b28b0 0x7f6a6413e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58920/0 (socket says 172.21.15.3:58920) 2022-07-21T01:42:08.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.199+0000 7f6a6259c700 1 --2- 172.21.15.3:0/1265868030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a6413e8d0 0x7f6a64141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6359e700 1 -- 172.21.15.3:0/1265868030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a6413e8d0 msgr2=0x7f6a64141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:08.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6359e700 1 --2- 172.21.15.3:0/1265868030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a6413e8d0 0x7f6a64141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6359e700 1 -- 172.21.15.3:0/1265868030 >> v1:172.21.15.3:6789/0 conn(0x7f6a640b28b0 legacy=0x7f6a6413e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a61d9b700 1 -- 172.21.15.3:0/1265868030 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944193706 0 0) 0x7f6a58001110 con 0x7f6a640b28b0 2022-07-21T01:42:08.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6359e700 1 -- 172.21.15.3:0/1265868030 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a640aa440 con 0x7f6a640b1180 2022-07-21T01:42:08.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6359e700 1 --2- 172.21.15.3:0/1265868030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 0x7f6a64141700 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f6a5c0032f0 tx=0x7f6a5c009290).ready entity=mon.0 client_cookie=dd5c2d4c27d6b12a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:08.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a61d9b700 1 -- 172.21.15.3:0/1265868030 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a5c00b8a0 con 0x7f6a640b1180 2022-07-21T01:42:08.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a61d9b700 1 -- 172.21.15.3:0/1265868030 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c00ba00 con 0x7f6a640b1180 2022-07-21T01:42:08.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a61d9b700 1 -- 172.21.15.3:0/1265868030 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a5c00bb70 con 0x7f6a640b1180 2022-07-21T01:42:08.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 -- 172.21.15.3:0/1265868030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 msgr2=0x7f6a64141700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 --2- 172.21.15.3:0/1265868030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 0x7f6a64141700 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f6a5c0032f0 tx=0x7f6a5c009290).stop 2022-07-21T01:42:08.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 -- 172.21.15.3:0/1265868030 shutdown_connections 2022-07-21T01:42:08.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 --2- 172.21.15.3:0/1265868030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 0x7f6a64141700 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 --2- 172.21.15.3:0/1265868030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a6413e8d0 0x7f6a64141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 -- 172.21.15.3:0/1265868030 >> 172.21.15.3:0/1265868030 conn(0x7f6a64060310 msgr2=0x7f6a640b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:08.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 -- 172.21.15.3:0/1265868030 shutdown_connections 2022-07-21T01:42:08.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 -- 172.21.15.3:0/1265868030 wait complete. 2022-07-21T01:42:08.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 Processor -- start 2022-07-21T01:42:08.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.203+0000 7f6a6883c700 1 -- start start 2022-07-21T01:42:08.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6883c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 0x7f6a64063190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6883c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a640b28b0 0x7f6a640ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6883c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a64144380 0x7f6a64155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6883c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a64142560 con 0x7f6a640b1180 2022-07-21T01:42:08.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6883c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a64143060 con 0x7f6a64144380 2022-07-21T01:42:08.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6883c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a64143300 con 0x7f6a640b28b0 2022-07-21T01:42:08.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a62d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 0x7f6a64063190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a640b28b0 0x7f6a640ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a62d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 0x7f6a64063190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50646/0 (socket says 172.21.15.3:50646) 2022-07-21T01:42:08.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a62d9d700 1 -- 172.21.15.3:0/184828503 learned_addr learned my addr 172.21.15.3:0/184828503 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:08.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6359e700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a64144380 0x7f6a64155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a62d9d700 1 -- 172.21.15.3:0/184828503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a640b28b0 msgr2=0x7f6a640ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a62d9d700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a640b28b0 0x7f6a640ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a62d9d700 1 -- 172.21.15.3:0/184828503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a64144380 msgr2=0x7f6a64155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a62d9d700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a64144380 0x7f6a64155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a62d9d700 1 -- 172.21.15.3:0/184828503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c009530 con 0x7f6a640b1180 2022-07-21T01:42:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a62d9d700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 0x7f6a64063190 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f6a58000ed0 tx=0x7f6a58007920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a53fff700 1 -- 172.21.15.3:0/184828503 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a5800ce40 con 0x7f6a640b1180 2022-07-21T01:42:08.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a53fff700 1 -- 172.21.15.3:0/184828503 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a580072e0 con 0x7f6a640b1180 2022-07-21T01:42:08.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a641435a0 con 0x7f6a640b1180 2022-07-21T01:42:08.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a53fff700 1 -- 172.21.15.3:0/184828503 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a58005740 con 0x7f6a640b1180 2022-07-21T01:42:08.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.207+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a64156290 con 0x7f6a640b1180 2022-07-21T01:42:08.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.211+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a640aa620 con 0x7f6a640b1180 2022-07-21T01:42:08.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.211+0000 7f6a53fff700 1 -- 172.21.15.3:0/184828503 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6a5801a060 con 0x7f6a640b1180 2022-07-21T01:42:08.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.211+0000 7f6a53fff700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a40053a70 0x7f6a40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.211+0000 7f6a53fff700 1 -- 172.21.15.3:0/184828503 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f6a5806b3c0 con 0x7f6a640b1180 2022-07-21T01:42:08.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.211+0000 7f6a6259c700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a40053a70 0x7f6a40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.211+0000 7f6a6259c700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a40053a70 0x7f6a40055f30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f6a5400b000 tx=0x7f6a54006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:08.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.215+0000 7f6a53fff700 1 -- 172.21.15.3:0/184828503 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6a58042070 con 0x7f6a640b1180 2022-07-21T01:42:08.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.379+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a640aac20 con 0x7f6a640b1180 2022-07-21T01:42:08.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.379+0000 7f6a53fff700 1 -- 172.21.15.3:0/184828503 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f6a640aac20 con 0x7f6a640b1180 2022-07-21T01:42:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:08.385 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":71,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/567682837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":567682837},{"type":"v1","addr":"172.21.15.143:6833","nonce":567682837}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/140242740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140242740},{"type":"v1","addr":"172.21.15.3:6835","nonce":140242740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:07.063040+0000","modified":"2022-07-21T01:42:08.065747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8049":{"gid":8049,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/506442168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":506442168},{"type":"v1","addr":"172.21.15.3:6837","nonce":506442168}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-21T01:42:08.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a40053a70 msgr2=0x7f6a40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a40053a70 0x7f6a40055f30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f6a5400b000 tx=0x7f6a54006df0).stop 2022-07-21T01:42:08.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 msgr2=0x7f6a64063190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 0x7f6a64063190 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f6a58000ed0 tx=0x7f6a58007920).stop 2022-07-21T01:42:08.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 shutdown_connections 2022-07-21T01:42:08.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6a40053a70 0x7f6a40055f30 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a64144380 0x7f6a64155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a640b28b0 0x7f6a640ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 --2- 172.21.15.3:0/184828503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a640b1180 0x7f6a64063190 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 >> 172.21.15.3:0/184828503 conn(0x7f6a64060310 msgr2=0x7f6a640af730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:08.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 shutdown_connections 2022-07-21T01:42:08.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.383+0000 7f6a6883c700 1 -- 172.21.15.3:0/184828503 wait complete. 2022-07-21T01:42:08.390 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 71 2022-07-21T01:42:08.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:08.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.495+0000 7f3f096bf700 1 Processor -- start 2022-07-21T01:42:08.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.495+0000 7f3f096bf700 1 -- start start 2022-07-21T01:42:08.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.495+0000 7f3f096bf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f040ad670 0x7f3f040b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.495+0000 7f3f096bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f040af120 0x7f3f040b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.495+0000 7f3f096bf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f040b0c40 con 0x7f3f0405f840 2022-07-21T01:42:08.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.495+0000 7f3f096bf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f040b0ee0 con 0x7f3f040ad670 2022-07-21T01:42:08.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.495+0000 7f3f096bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f040ae9e0 con 0x7f3f040af120 2022-07-21T01:42:08.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f037fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f040af120 0x7f3f040b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f03fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f040ad670 0x7f3f040b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f037fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f040af120 0x7f3f040b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:57574/0 (socket says 172.21.15.3:57574) 2022-07-21T01:42:08.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f08ebe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3f0405f840 0x7f3f0405fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:32820/0 (socket says 172.21.15.3:32820) 2022-07-21T01:42:08.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f037fe700 1 -- 172.21.15.3:0/3324665365 learned_addr learned my addr 172.21.15.3:0/3324665365 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:08.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f03fff700 1 -- 172.21.15.3:0/3324665365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f040af120 msgr2=0x7f3f040b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:08.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f03fff700 1 --2- 172.21.15.3:0/3324665365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f040af120 0x7f3f040b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f03fff700 1 -- 172.21.15.3:0/3324665365 >> v1:172.21.15.143:6789/0 conn(0x7f3f0405f840 legacy=0x7f3f0405fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f03fff700 1 -- 172.21.15.3:0/3324665365 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f040aec80 con 0x7f3f040ad670 2022-07-21T01:42:08.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f03fff700 1 --2- 172.21.15.3:0/3324665365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040ad670 0x7f3f040b0570 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f3efc0032f0 tx=0x7f3efc0091d0).ready entity=mon.1 client_cookie=8188f782084161fe server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:08.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f02ffd700 1 -- 172.21.15.3:0/3324665365 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3efc00b8b0 con 0x7f3f040ad670 2022-07-21T01:42:08.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f02ffd700 1 -- 172.21.15.3:0/3324665365 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3efc00ba10 con 0x7f3f040ad670 2022-07-21T01:42:08.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f02ffd700 1 -- 172.21.15.3:0/3324665365 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3efc00bd70 con 0x7f3f040ad670 2022-07-21T01:42:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f096bf700 1 -- 172.21.15.3:0/3324665365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040ad670 msgr2=0x7f3f040b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f096bf700 1 --2- 172.21.15.3:0/3324665365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040ad670 0x7f3f040b0570 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f3efc0032f0 tx=0x7f3efc0091d0).stop 2022-07-21T01:42:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f096bf700 1 -- 172.21.15.3:0/3324665365 shutdown_connections 2022-07-21T01:42:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f096bf700 1 --2- 172.21.15.3:0/3324665365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f040af120 0x7f3f040b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f096bf700 1 --2- 172.21.15.3:0/3324665365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040ad670 0x7f3f040b0570 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f096bf700 1 -- 172.21.15.3:0/3324665365 >> 172.21.15.3:0/3324665365 conn(0x7f3f04061360 msgr2=0x7f3f04062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f096bf700 1 -- 172.21.15.3:0/3324665365 shutdown_connections 2022-07-21T01:42:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.499+0000 7f3f096bf700 1 -- 172.21.15.3:0/3324665365 wait complete. 2022-07-21T01:42:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f096bf700 1 Processor -- start 2022-07-21T01:42:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f096bf700 1 -- start start 2022-07-21T01:42:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f096bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0405f840 0x7f3f040aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f096bf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f040ad670 0x7f3f040ac290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f096bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040af120 0x7f3f04142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f096bf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0413f8d0 con 0x7f3f040ad670 2022-07-21T01:42:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f096bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f04065960 con 0x7f3f040af120 2022-07-21T01:42:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f096bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f04151800 con 0x7f3f0405f840 2022-07-21T01:42:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f037fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f040ad670 0x7f3f040ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f037fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f040ad670 0x7f3f040ac290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50662/0 (socket says 172.21.15.3:50662) 2022-07-21T01:42:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f037fe700 1 -- 172.21.15.3:0/1574144675 learned_addr learned my addr 172.21.15.3:0/1574144675 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f08ebe700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040af120 0x7f3f04142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.503+0000 7f3f03fff700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0405f840 0x7f3f040aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f037fe700 1 -- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0405f840 msgr2=0x7f3f040aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f037fe700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0405f840 0x7f3f040aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f037fe700 1 -- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040af120 msgr2=0x7f3f04142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f037fe700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040af120 0x7f3f04142510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f037fe700 1 -- 172.21.15.3:0/1574144675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc009470 con 0x7f3f040ad670 2022-07-21T01:42:08.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f03fff700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0405f840 0x7f3f040aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:08.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f08ebe700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040af120 0x7f3f04142510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:08.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f037fe700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f040ad670 0x7f3f040ac290 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7f3ef800c000 tx=0x7f3ef8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:08.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f017fa700 1 -- 172.21.15.3:0/1574144675 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ef8013610 con 0x7f3f040ad670 2022-07-21T01:42:08.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f017fa700 1 -- 172.21.15.3:0/1574144675 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef8013770 con 0x7f3f040ad670 2022-07-21T01:42:08.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f040aad30 con 0x7f3f040ad670 2022-07-21T01:42:08.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f017fa700 1 -- 172.21.15.3:0/1574144675 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ef8013ad0 con 0x7f3f040ad670 2022-07-21T01:42:08.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f040ab270 con 0x7f3f040ad670 2022-07-21T01:42:08.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f040b34e0 con 0x7f3f040ad670 2022-07-21T01:42:08.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f017fa700 1 -- 172.21.15.3:0/1574144675 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ef8013c30 con 0x7f3f040ad670 2022-07-21T01:42:08.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f017fa700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ee4053980 0x7f3ee4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f017fa700 1 -- 172.21.15.3:0/1574144675 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3ef806c0e0 con 0x7f3f040ad670 2022-07-21T01:42:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.507+0000 7f3f03fff700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ee4053980 0x7f3ee4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.511+0000 7f3f03fff700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ee4053980 0x7f3ee4055e40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f3efc01b000 tx=0x7f3efc018000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.519+0000 7f3f017fa700 1 -- 172.21.15.3:0/1574144675 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ef8042ea0 con 0x7f3f040ad670 2022-07-21T01:42:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.647+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f040574e0 con 0x7f3f040ad670 2022-07-21T01:42:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.647+0000 7f3f017fa700 1 -- 172.21.15.3:0/1574144675 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f3ef80464c0 con 0x7f3f040ad670 2022-07-21T01:42:08.652 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:08.652 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":140,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:08.066592+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:05.096631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:06.060734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ee4053980 msgr2=0x7f3ee4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ee4053980 0x7f3ee4055e40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f3efc01b000 tx=0x7f3efc018000).stop 2022-07-21T01:42:08.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f040ad670 msgr2=0x7f3f040ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f040ad670 0x7f3f040ac290 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7f3ef800c000 tx=0x7f3ef8008040).stop 2022-07-21T01:42:08.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 shutdown_connections 2022-07-21T01:42:08.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ee4053980 0x7f3ee4055e40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f040af120 0x7f3f04142510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f040ad670 0x7f3f040ac290 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 --2- 172.21.15.3:0/1574144675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f0405f840 0x7f3f040aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 >> 172.21.15.3:0/1574144675 conn(0x7f3f04061360 msgr2=0x7f3f0413e760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:08.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 shutdown_connections 2022-07-21T01:42:08.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.651+0000 7f3f096bf700 1 -- 172.21.15.3:0/1574144675 wait complete. 2022-07-21T01:42:08.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:42:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0fa1d29700 1 Processor -- start 2022-07-21T01:42:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0fa1d29700 1 -- start start 2022-07-21T01:42:08.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0fa1d29700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0f9c0b6710 0x7f0f9c0b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0fa1d29700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0f9c0b6af0 0x7f0f9c12bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0fa1d29700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c144240 con 0x7f0f9c0b4550 2022-07-21T01:42:08.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0fa1d29700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c143d00 con 0x7f0f9c0b6af0 2022-07-21T01:42:08.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0fa1d29700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c143fa0 con 0x7f0f9c0b6710 2022-07-21T01:42:08.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0f9c0b6af0 0x7f0f9c12bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0f9c0b6af0 0x7f0f9c12bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50672/0 (socket says 172.21.15.3:50672) 2022-07-21T01:42:08.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9bfff700 1 -- 172.21.15.3:0/3998218282 learned_addr learned my addr 172.21.15.3:0/3998218282 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:08.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9b7fe700 1 --2- 172.21.15.3:0/3998218282 >> v2:172.21.15.143:3300/0 conn(0x7f0f9c0b6710 0x7f0f9c0b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9bfff700 1 -- 172.21.15.3:0/3998218282 >> v2:172.21.15.143:3300/0 conn(0x7f0f9c0b6710 msgr2=0x7f0f9c0b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:08.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9bfff700 1 --2- 172.21.15.3:0/3998218282 >> v2:172.21.15.143:3300/0 conn(0x7f0f9c0b6710 0x7f0f9c0b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9bfff700 1 -- 172.21.15.3:0/3998218282 >> v1:172.21.15.143:6789/0 conn(0x7f0f9c0b4550 legacy=0x7f0f9c061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9bfff700 1 -- 172.21.15.3:0/3998218282 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c061780 con 0x7f0f9c0b6af0 2022-07-21T01:42:08.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9bfff700 1 --2- 172.21.15.3:0/3998218282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6af0 0x7f0f9c12bb90 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f0f9c1432b0 tx=0x7f0f94006eb0).ready entity=mon.0 client_cookie=c0ba04e066475eed server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:08.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9a7fc700 1 -- 172.21.15.3:0/3998218282 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f9400d610 con 0x7f0f9c0b6af0 2022-07-21T01:42:08.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9a7fc700 1 -- 172.21.15.3:0/3998218282 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f94000e90 con 0x7f0f9c0b6af0 2022-07-21T01:42:08.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.779+0000 7f0f9a7fc700 1 -- 172.21.15.3:0/3998218282 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f9400d9a0 con 0x7f0f9c0b6af0 2022-07-21T01:42:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 -- 172.21.15.3:0/3998218282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6af0 msgr2=0x7f0f9c12bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 --2- 172.21.15.3:0/3998218282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6af0 0x7f0f9c12bb90 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f0f9c1432b0 tx=0x7f0f94006eb0).stop 2022-07-21T01:42:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 -- 172.21.15.3:0/3998218282 shutdown_connections 2022-07-21T01:42:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 --2- 172.21.15.3:0/3998218282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6af0 0x7f0f9c12bb90 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 --2- 172.21.15.3:0/3998218282 >> v2:172.21.15.143:3300/0 conn(0x7f0f9c0b6710 0x7f0f9c0b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 -- 172.21.15.3:0/3998218282 >> 172.21.15.3:0/3998218282 conn(0x7f0f9c05c620 msgr2=0x7f0f9c05ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 -- 172.21.15.3:0/3998218282 shutdown_connections 2022-07-21T01:42:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 -- 172.21.15.3:0/3998218282 wait complete. 2022-07-21T01:42:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 Processor -- start 2022-07-21T01:42:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 -- start start 2022-07-21T01:42:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f9c0b4550 0x7f0f9c1492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6710 0x7f0f9c155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f9c0b6af0 0x7f0f9c1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c12b0b0 con 0x7f0f9c0b6710 2022-07-21T01:42:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c126f20 con 0x7f0f9c0b4550 2022-07-21T01:42:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.783+0000 7f0fa1d29700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c1269e0 con 0x7f0f9c0b6af0 2022-07-21T01:42:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6710 0x7f0f9c155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6710 0x7f0f9c155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50674/0 (socket says 172.21.15.3:50674) 2022-07-21T01:42:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9affd700 1 -- 172.21.15.3:0/186824785 learned_addr learned my addr 172.21.15.3:0/186824785 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9bfff700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f9c0b6af0 0x7f0f9c1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9b7fe700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f9c0b4550 0x7f0f9c1492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9affd700 1 -- 172.21.15.3:0/186824785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f9c0b6af0 msgr2=0x7f0f9c1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9affd700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f9c0b6af0 0x7f0f9c1563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9affd700 1 -- 172.21.15.3:0/186824785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f9c0b4550 msgr2=0x7f0f9c1492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9affd700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f9c0b4550 0x7f0f9c1492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9affd700 1 -- 172.21.15.3:0/186824785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9400b1e0 con 0x7f0f9c0b6710 2022-07-21T01:42:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f9affd700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6710 0x7f0f9c155eb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f0f9c0b1e10 tx=0x7f0f84007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:08.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f98ff9700 1 -- 172.21.15.3:0/186824785 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f84012400 con 0x7f0f9c0b6710 2022-07-21T01:42:08.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f98ff9700 1 -- 172.21.15.3:0/186824785 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f84012560 con 0x7f0f9c0b6710 2022-07-21T01:42:08.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c126c80 con 0x7f0f9c0b6710 2022-07-21T01:42:08.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f98ff9700 1 -- 172.21.15.3:0/186824785 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f840128c0 con 0x7f0f9c0b6710 2022-07-21T01:42:08.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c12b6f0 con 0x7f0f9c0b6710 2022-07-21T01:42:08.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c061a20 con 0x7f0f9c0b6710 2022-07-21T01:42:08.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.787+0000 7f0f98ff9700 1 -- 172.21.15.3:0/186824785 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0f84019030 con 0x7f0f9c0b6710 2022-07-21T01:42:08.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.791+0000 7f0f98ff9700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f80053b00 0x7f0f80055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:08.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.791+0000 7f0f98ff9700 1 -- 172.21.15.3:0/186824785 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0f8406b340 con 0x7f0f9c0b6710 2022-07-21T01:42:08.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.791+0000 7f0f9b7fe700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f80053b00 0x7f0f80055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:08.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.791+0000 7f0f9b7fe700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f80053b00 0x7f0f80055fc0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f0f8c00b000 tx=0x7f0f8c0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:08.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.795+0000 7f0f98ff9700 1 -- 172.21.15.3:0/186824785 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0f84042100 con 0x7f0f9c0b6710 2022-07-21T01:42:08.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.959+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0f9c0574e0 con 0x7f0f9c0b6710 2022-07-21T01:42:08.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.967+0000 7f0f98ff9700 1 -- 172.21.15.3:0/186824785 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v34) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0f84045720 con 0x7f0f9c0b6710 2022-07-21T01:42:08.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f80053b00 msgr2=0x7f0f80055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f80053b00 0x7f0f80055fc0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f0f8c00b000 tx=0x7f0f8c0091d0).stop 2022-07-21T01:42:08.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6710 msgr2=0x7f0f9c155eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6710 0x7f0f9c155eb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f0f9c0b1e10 tx=0x7f0f84007c20).stop 2022-07-21T01:42:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 shutdown_connections 2022-07-21T01:42:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f9c0b6af0 0x7f0f9c1563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c0b6710 0x7f0f9c155eb0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0f80053b00 0x7f0f80055fc0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 --2- 172.21.15.3:0/186824785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f9c0b4550 0x7f0f9c1492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 >> 172.21.15.3:0/186824785 conn(0x7f0f9c05c620 msgr2=0x7f0f9c142490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 shutdown_connections 2022-07-21T01:42:08.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:08.971+0000 7f0fa1d29700 1 -- 172.21.15.3:0/186824785 wait complete. 2022-07-21T01:42:08.977 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:42:08.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:42:09.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.095+0000 7ff0989ca700 1 Processor -- start 2022-07-21T01:42:09.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.095+0000 7ff0989ca700 1 -- start start 2022-07-21T01:42:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.095+0000 7ff0989ca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff0940b6710 0x7ff0940b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.095+0000 7ff0989ca700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff094144240 con 0x7ff0940b6af0 2022-07-21T01:42:09.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.095+0000 7ff0989ca700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff094143d00 con 0x7ff0940b4550 2022-07-21T01:42:09.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.095+0000 7ff0989ca700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff094143fa0 con 0x7ff0940b6710 2022-07-21T01:42:09.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff092d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff0940b6af0 0x7ff09412bb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58964/0 (socket says 172.21.15.3:58964) 2022-07-21T01:42:09.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff092d9d700 1 -- 172.21.15.3:0/2454973611 learned_addr learned my addr 172.21.15.3:0/2454973611 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:09.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09259c700 1 --2- 172.21.15.3:0/2454973611 >> v2:172.21.15.143:3300/0 conn(0x7ff0940b6710 0x7ff0940b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09159a700 1 -- 172.21.15.3:0/2454973611 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860025096 0 0) 0x7ff094144240 con 0x7ff0940b6af0 2022-07-21T01:42:09.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09159a700 1 -- 172.21.15.3:0/2454973611 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff074003260 con 0x7ff0940b6af0 2022-07-21T01:42:09.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09259c700 1 -- 172.21.15.3:0/2454973611 >> v1:172.21.15.143:6789/0 conn(0x7ff0940b4550 legacy=0x7ff094061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09259c700 1 -- 172.21.15.3:0/2454973611 >> v1:172.21.15.3:6789/0 conn(0x7ff0940b6af0 legacy=0x7ff09412bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09259c700 1 -- 172.21.15.3:0/2454973611 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff094061780 con 0x7ff0940b6710 2022-07-21T01:42:09.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09259c700 1 --2- 172.21.15.3:0/2454973611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0940b6710 0x7ff0940b5d80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7ff0941432b0 tx=0x7ff0840091d0).ready entity=mon.1 client_cookie=9b955d3067df2ca7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:09.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09159a700 1 -- 172.21.15.3:0/2454973611 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff08400b680 con 0x7ff0940b6710 2022-07-21T01:42:09.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09159a700 1 -- 172.21.15.3:0/2454973611 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08400b7e0 con 0x7ff0940b6710 2022-07-21T01:42:09.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff09159a700 1 -- 172.21.15.3:0/2454973611 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff08400b950 con 0x7ff0940b6710 2022-07-21T01:42:09.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff0989ca700 1 -- 172.21.15.3:0/2454973611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0940b6710 msgr2=0x7ff0940b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff0989ca700 1 --2- 172.21.15.3:0/2454973611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0940b6710 0x7ff0940b5d80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7ff0941432b0 tx=0x7ff0840091d0).stop 2022-07-21T01:42:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff0989ca700 1 -- 172.21.15.3:0/2454973611 shutdown_connections 2022-07-21T01:42:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff0989ca700 1 --2- 172.21.15.3:0/2454973611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0940b6710 0x7ff0940b5d80 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff0989ca700 1 -- 172.21.15.3:0/2454973611 >> 172.21.15.3:0/2454973611 conn(0x7ff09405c620 msgr2=0x7ff09405ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:09.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff0989ca700 1 -- 172.21.15.3:0/2454973611 shutdown_connections 2022-07-21T01:42:09.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.099+0000 7ff0989ca700 1 -- 172.21.15.3:0/2454973611 wait complete. 2022-07-21T01:42:09.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 Processor -- start 2022-07-21T01:42:09.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 -- start start 2022-07-21T01:42:09.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0940b4550 0x7ff094152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0940b6710 0x7ff09412c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0940b6af0 0x7ff094128970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0940b0010 con 0x7ff0940b6af0 2022-07-21T01:42:09.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0940b04a0 con 0x7ff0940b4550 2022-07-21T01:42:09.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff094152980 con 0x7ff0940b6710 2022-07-21T01:42:09.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff092d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0940b6af0 0x7ff094128970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff092d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0940b6af0 0x7ff094128970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50692/0 (socket says 172.21.15.3:50692) 2022-07-21T01:42:09.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff092d9d700 1 -- 172.21.15.3:0/2165309108 learned_addr learned my addr 172.21.15.3:0/2165309108 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:09.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff09259c700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0940b4550 0x7ff094152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff091d9b700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0940b6710 0x7ff09412c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff092d9d700 1 -- 172.21.15.3:0/2165309108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0940b6710 msgr2=0x7ff09412c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff092d9d700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0940b6710 0x7ff09412c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff092d9d700 1 -- 172.21.15.3:0/2165309108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0940b4550 msgr2=0x7ff094152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff092d9d700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0940b4550 0x7ff094152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff092d9d700 1 -- 172.21.15.3:0/2165309108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff084009470 con 0x7ff0940b6af0 2022-07-21T01:42:09.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff092d9d700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0940b6af0 0x7ff094128970 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7ff08c001110 tx=0x7ff08c007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:09.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff08b7fe700 1 -- 172.21.15.3:0/2165309108 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff08c00cdf0 con 0x7ff0940b6af0 2022-07-21T01:42:09.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff08b7fe700 1 -- 172.21.15.3:0/2165309108 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c0070d0 con 0x7ff0940b6af0 2022-07-21T01:42:09.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff08b7fe700 1 -- 172.21.15.3:0/2165309108 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff08c007430 con 0x7ff0940b6af0 2022-07-21T01:42:09.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0940b0200 con 0x7ff0940b6af0 2022-07-21T01:42:09.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.103+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09412c840 con 0x7ff0940b6af0 2022-07-21T01:42:09.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.107+0000 7ff08b7fe700 1 -- 172.21.15.3:0/2165309108 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff08c010070 con 0x7ff0940b6af0 2022-07-21T01:42:09.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.107+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff094061a20 con 0x7ff0940b6af0 2022-07-21T01:42:09.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.107+0000 7ff08b7fe700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff078053b00 0x7ff078055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.107+0000 7ff09259c700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff078053b00 0x7ff078055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.107+0000 7ff08b7fe700 1 -- 172.21.15.3:0/2165309108 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff08c06b750 con 0x7ff0940b6af0 2022-07-21T01:42:09.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.107+0000 7ff09259c700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff078053b00 0x7ff078055fc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7ff08401b000 tx=0x7ff084016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:09.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.115+0000 7ff08b7fe700 1 -- 172.21.15.3:0/2165309108 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff08c042400 con 0x7ff0940b6af0 2022-07-21T01:42:09.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.279+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff0940574e0 con 0x7ff0940b6af0 2022-07-21T01:42:09.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff08b7fe700 1 -- 172.21.15.3:0/2165309108 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v35) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff08c007bc0 con 0x7ff0940b6af0 2022-07-21T01:42:09.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff078053b00 msgr2=0x7ff078055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff078053b00 0x7ff078055fc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7ff08401b000 tx=0x7ff084016000).stop 2022-07-21T01:42:09.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0940b6af0 msgr2=0x7ff094128970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0940b6af0 0x7ff094128970 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7ff08c001110 tx=0x7ff08c007a80).stop 2022-07-21T01:42:09.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 shutdown_connections 2022-07-21T01:42:09.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0940b6af0 0x7ff094128970 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0940b6710 0x7ff09412c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff078053b00 0x7ff078055fc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 --2- 172.21.15.3:0/2165309108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0940b4550 0x7ff094152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 >> 172.21.15.3:0/2165309108 conn(0x7ff09405c620 msgr2=0x7ff094141120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 shutdown_connections 2022-07-21T01:42:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.287+0000 7ff0989ca700 1 -- 172.21.15.3:0/2165309108 wait complete. 2022-07-21T01:42:09.295 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:42:09.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:09.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad771fd700 1 Processor -- start 2022-07-21T01:42:09.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad771fd700 1 -- start start 2022-07-21T01:42:09.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad771fd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700614b0 0x7fad70061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad771fd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad700aee80 con 0x7fad700ad750 2022-07-21T01:42:09.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad771fd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad700b70b0 con 0x7fad701405d0 2022-07-21T01:42:09.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad771fd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad700af0b0 con 0x7fad700614b0 2022-07-21T01:42:09.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad759fa700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fad700ad750 0x7fad700b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:58976/0 (socket says 172.21.15.3:58976) 2022-07-21T01:42:09.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad759fa700 1 -- 172.21.15.3:0/2990812368 learned_addr learned my addr 172.21.15.3:0/2990812368 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:09.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad761fb700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700614b0 0x7fad70061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146011332 0 0) 0x7fad700aee80 con 0x7fad700ad750 2022-07-21T01:42:09.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad58003260 con 0x7fad700ad750 2022-07-21T01:42:09.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.403+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092392552 0 0) 0x7fad700b70b0 con 0x7fad701405d0 2022-07-21T01:42:09.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad700aee80 con 0x7fad701405d0 2022-07-21T01:42:09.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941297795 0 0) 0x7fad58003260 con 0x7fad700ad750 2022-07-21T01:42:09.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700614b0 msgr2=0x7fad70061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:09.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700614b0 0x7fad70061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 >> v1:172.21.15.143:6789/0 conn(0x7fad701405d0 legacy=0x7fad701466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad700af350 con 0x7fad700ad750 2022-07-21T01:42:09.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fad6c0019e0 con 0x7fad700ad750 2022-07-21T01:42:09.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 >> v1:172.21.15.3:6789/0 conn(0x7fad700ad750 legacy=0x7fad700b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad58003910 0x7fad58003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad58004500 0x7fad58004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700614b0 0x7fad58004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad700af0b0 con 0x7fad58003910 2022-07-21T01:42:09.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5800b800 con 0x7fad58004500 2022-07-21T01:42:09.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5800b990 con 0x7fad700614b0 2022-07-21T01:42:09.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad761fb700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad58003910 0x7fad58003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad759fa700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad58004500 0x7fad58004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad769fc700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700614b0 0x7fad58004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad761fb700 1 -- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700614b0 msgr2=0x7fad58004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:09.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad761fb700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700614b0 0x7fad58004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad761fb700 1 -- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad58004500 msgr2=0x7fad58004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:09.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad761fb700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad58004500 0x7fad58004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad761fb700 1 -- 172.21.15.3:0/2990812368 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad58003260 con 0x7fad58003910 2022-07-21T01:42:09.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad769fc700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700614b0 0x7fad58004e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:09.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad761fb700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad58003910 0x7fad58003d10 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7fad580054d0 tx=0x7fad60006df0).ready entity=mon.0 client_cookie=2090cbc682221f95 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:09.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad6000e830 con 0x7fad58003910 2022-07-21T01:42:09.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6000be90 con 0x7fad58003910 2022-07-21T01:42:09.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad751f9700 1 -- 172.21.15.3:0/2990812368 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad6000eb80 con 0x7fad58003910 2022-07-21T01:42:09.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad771fd700 1 -- 172.21.15.3:0/2990812368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad58003910 msgr2=0x7fad58003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad771fd700 1 --2- 172.21.15.3:0/2990812368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad58003910 0x7fad58003d10 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7fad580054d0 tx=0x7fad60006df0).stop 2022-07-21T01:42:09.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad769fc700 1 -- 172.21.15.3:0/2990812368 reap_dead start 2022-07-21T01:42:09.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad771fd700 1 -- 172.21.15.3:0/2990812368 shutdown_connections 2022-07-21T01:42:09.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad771fd700 1 -- 172.21.15.3:0/2990812368 >> 172.21.15.3:0/2990812368 conn(0x7fad70066f70 msgr2=0x7fad70067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:09.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad771fd700 1 -- 172.21.15.3:0/2990812368 shutdown_connections 2022-07-21T01:42:09.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.407+0000 7fad771fd700 1 -- 172.21.15.3:0/2990812368 wait complete. 2022-07-21T01:42:09.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 Processor -- start 2022-07-21T01:42:09.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 -- start start 2022-07-21T01:42:09.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700ac950 0x7fad700b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad700afac0 0x7fad700ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700ac190 0x7fad70145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad700acd30 con 0x7fad700ac190 2022-07-21T01:42:09.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad700aff60 con 0x7fad700afac0 2022-07-21T01:42:09.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad700b0200 con 0x7fad700ac950 2022-07-21T01:42:09.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad769fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700ac190 0x7fad70145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad769fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700ac190 0x7fad70145470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50712/0 (socket says 172.21.15.3:50712) 2022-07-21T01:42:09.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad769fc700 1 -- 172.21.15.3:0/2551505556 learned_addr learned my addr 172.21.15.3:0/2551505556 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:09.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad761fb700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700ac950 0x7fad700b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad759fa700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad700afac0 0x7fad700ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad769fc700 1 -- 172.21.15.3:0/2551505556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700ac950 msgr2=0x7fad700b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad769fc700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700ac950 0x7fad700b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad769fc700 1 -- 172.21.15.3:0/2551505556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad700afac0 msgr2=0x7fad700ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad769fc700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad700afac0 0x7fad700ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad769fc700 1 -- 172.21.15.3:0/2551505556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad700b1490 con 0x7fad700ac190 2022-07-21T01:42:09.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad769fc700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700ac190 0x7fad70145470 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7fad6c001bc0 tx=0x7fad68009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:09.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad5f7fe700 1 -- 172.21.15.3:0/2551505556 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad6800dd30 con 0x7fad700ac190 2022-07-21T01:42:09.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad5f7fe700 1 -- 172.21.15.3:0/2551505556 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6800de90 con 0x7fad700ac190 2022-07-21T01:42:09.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad700b1730 con 0x7fad700ac190 2022-07-21T01:42:09.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad5f7fe700 1 -- 172.21.15.3:0/2551505556 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad68013610 con 0x7fad700ac190 2022-07-21T01:42:09.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.411+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad700ab090 con 0x7fad700ac190 2022-07-21T01:42:09.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.415+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad700b18c0 con 0x7fad700ac190 2022-07-21T01:42:09.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.415+0000 7fad5f7fe700 1 -- 172.21.15.3:0/2551505556 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fad68013770 con 0x7fad700ac190 2022-07-21T01:42:09.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.415+0000 7fad5f7fe700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad50053a70 0x7fad50055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:09.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.415+0000 7fad5f7fe700 1 -- 172.21.15.3:0/2551505556 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fad6806b9a0 con 0x7fad700ac190 2022-07-21T01:42:09.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.415+0000 7fad761fb700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad50053a70 0x7fad50055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:09.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.415+0000 7fad761fb700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad50053a70 0x7fad50055f30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fad60007000 tx=0x7fad60019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:09.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.427+0000 7fad5f7fe700 1 -- 172.21.15.3:0/2551505556 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad68042700 con 0x7fad700ac190 2022-07-21T01:42:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.587+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad700ab090 con 0x7fad700ac190 2022-07-21T01:42:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.587+0000 7fad5f7fe700 1 -- 172.21.15.3:0/2551505556 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2483 (secure 0 0 0) 0x7fad700ab090 con 0x7fad700ac190 2022-07-21T01:42:09.592 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:09.592 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":71,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/567682837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":567682837},{"type":"v1","addr":"172.21.15.143:6833","nonce":567682837}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/140242740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140242740},{"type":"v1","addr":"172.21.15.3:6835","nonce":140242740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:07.063040+0000","modified":"2022-07-21T01:42:08.065747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8049":{"gid":8049,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/506442168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":506442168},{"type":"v1","addr":"172.21.15.3:6837","nonce":506442168}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-21T01:42:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad50053a70 msgr2=0x7fad50055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad50053a70 0x7fad50055f30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fad60007000 tx=0x7fad60019000).stop 2022-07-21T01:42:09.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700ac190 msgr2=0x7fad70145470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:09.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700ac190 0x7fad70145470 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7fad6c001bc0 tx=0x7fad68009040).stop 2022-07-21T01:42:09.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 shutdown_connections 2022-07-21T01:42:09.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad700ac190 0x7fad70145470 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad50053a70 0x7fad50055f30 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad700afac0 0x7fad700ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 --2- 172.21.15.3:0/2551505556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad700ac950 0x7fad700b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:09.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 >> 172.21.15.3:0/2551505556 conn(0x7fad70066f70 msgr2=0x7fad7005cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:09.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 shutdown_connections 2022-07-21T01:42:09.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:09.591+0000 7fad771fd700 1 -- 172.21.15.3:0/2551505556 wait complete. 2022-07-21T01:42:09.600 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 71 2022-07-21T01:42:09.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:07.063040+0000', 'modified': '2022-07-21T01:42:08.065747+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8049}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8049': {'gid': 8049, 'name': 'a', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/506442168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 506442168}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 506442168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:42:09.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:42:09.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:42:10.030 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8049,"want_state":"up:active","state":"up:active","rank_uptime":2.9551283540000002,"mdsmap_epoch":71,"osdmap_epoch":140,"osdmap_epoch_barrier":139,"uptime":7.6847476190000004} 2022-07-21T01:42:10.031 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8049, "mdsmap_epoch": 71, "osdmap_epoch": 140, "osdmap_epoch_barrier": 139, "rank_uptime": 2.955128354, "state": "up:active", "uptime": 7.684747619, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:42:10.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:10.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.127+0000 7f992a30f700 1 Processor -- start 2022-07-21T01:42:10.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.127+0000 7f992a30f700 1 -- start start 2022-07-21T01:42:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f99240b1f40 0x7f99240b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9924148530 0x7f99240b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99240b04e0 con 0x7f99240a7250 2022-07-21T01:42:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99240a9250 con 0x7f9924148530 2022-07-21T01:42:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99240b0780 con 0x7f99240b1f40 2022-07-21T01:42:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f991bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9924148530 0x7f99240b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f991bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9924148530 0x7f99240b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50716/0 (socket says 172.21.15.3:50716) 2022-07-21T01:42:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f991bfff700 1 -- 172.21.15.3:0/1450569622 learned_addr learned my addr 172.21.15.3:0/1450569622 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f991bfff700 1 -- 172.21.15.3:0/1450569622 >> v2:172.21.15.143:3300/0 conn(0x7f99240b1f40 msgr2=0x7f99240b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f991bfff700 1 --2- 172.21.15.3:0/1450569622 >> v2:172.21.15.143:3300/0 conn(0x7f99240b1f40 0x7f99240b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:42:10.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f991bfff700 1 -- 172.21.15.3:0/1450569622 >> v1:172.21.15.143:6789/0 conn(0x7f99240a7250 legacy=0x7f99241448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:10.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f991bfff700 1 -- 172.21.15.3:0/1450569622 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99240b0a20 con 0x7f9924148530 2022-07-21T01:42:10.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f991bfff700 1 --2- 172.21.15.3:0/1450569622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9924148530 0x7f99240b2320 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f99240acff0 tx=0x7f990c006eb0).ready entity=mon.0 client_cookie=e292c8fa9b4a028 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f99237fe700 1 -- 172.21.15.3:0/1450569622 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990c00d5c0 con 0x7f9924148530 2022-07-21T01:42:10.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f99237fe700 1 -- 172.21.15.3:0/1450569622 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c000e90 con 0x7f9924148530 2022-07-21T01:42:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f99237fe700 1 -- 172.21.15.3:0/1450569622 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990c00d950 con 0x7f9924148530 2022-07-21T01:42:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 -- 172.21.15.3:0/1450569622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9924148530 msgr2=0x7f99240b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 --2- 172.21.15.3:0/1450569622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9924148530 0x7f99240b2320 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f99240acff0 tx=0x7f990c006eb0).stop 2022-07-21T01:42:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 -- 172.21.15.3:0/1450569622 shutdown_connections 2022-07-21T01:42:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 --2- 172.21.15.3:0/1450569622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9924148530 0x7f99240b2320 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 --2- 172.21.15.3:0/1450569622 >> v2:172.21.15.143:3300/0 conn(0x7f99240b1f40 0x7f99240b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:42:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 -- 172.21.15.3:0/1450569622 >> 172.21.15.3:0/1450569622 conn(0x7f992405f840 msgr2=0x7f992405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 -- 172.21.15.3:0/1450569622 shutdown_connections 2022-07-21T01:42:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 -- 172.21.15.3:0/1450569622 wait complete. 2022-07-21T01:42:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 Processor -- start 2022-07-21T01:42:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.131+0000 7f992a30f700 1 -- start start 2022-07-21T01:42:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f992a30f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99240a7250 0x7f99240ab010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f992a30f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99240b1f40 0x7f9924143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f992a30f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9924148530 0x7f9924143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f992a30f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99240ad910 con 0x7f99240a7250 2022-07-21T01:42:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f992a30f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99240a9950 con 0x7f99240b1f40 2022-07-21T01:42:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f992a30f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99240ae5b0 con 0x7f9924148530 2022-07-21T01:42:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9923fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99240a7250 0x7f99240ab010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9923fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99240a7250 0x7f99240ab010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50722/0 (socket says 172.21.15.3:50722) 2022-07-21T01:42:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9923fff700 1 -- 172.21.15.3:0/1808250473 learned_addr learned my addr 172.21.15.3:0/1808250473 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f991bfff700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99240b1f40 0x7f9924143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f99288ac700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9924148530 0x7f9924143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9923fff700 1 -- 172.21.15.3:0/1808250473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9924148530 msgr2=0x7f9924143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9923fff700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9924148530 0x7f9924143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9923fff700 1 -- 172.21.15.3:0/1808250473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99240b1f40 msgr2=0x7f9924143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9923fff700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99240b1f40 0x7f9924143a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9923fff700 1 -- 172.21.15.3:0/1808250473 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c00b1e0 con 0x7f99240a7250 2022-07-21T01:42:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f99288ac700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9924148530 0x7f9924143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9923fff700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99240a7250 0x7f99240ab010 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f9914002f70 tx=0x7f9914006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9921ffb700 1 -- 172.21.15.3:0/1808250473 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9914013460 con 0x7f99240a7250 2022-07-21T01:42:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9921ffb700 1 -- 172.21.15.3:0/1808250473 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99140135c0 con 0x7f99240a7250 2022-07-21T01:42:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99240adb40 con 0x7f99240a7250 2022-07-21T01:42:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f9921ffb700 1 -- 172.21.15.3:0/1808250473 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9914013920 con 0x7f99240a7250 2022-07-21T01:42:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99240a95d0 con 0x7f99240a7250 2022-07-21T01:42:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.135+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99240b0cc0 con 0x7f99240a7250 2022-07-21T01:42:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.139+0000 7f9921ffb700 1 -- 172.21.15.3:0/1808250473 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f991401a060 con 0x7f99240a7250 2022-07-21T01:42:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.139+0000 7f9921ffb700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9908053b00 0x7f9908055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.139+0000 7f9921ffb700 1 -- 172.21.15.3:0/1808250473 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f991406baa0 con 0x7f99240a7250 2022-07-21T01:42:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.139+0000 7f991bfff700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9908053b00 0x7f9908055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.139+0000 7f991bfff700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9908053b00 0x7f9908055fc0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f990c01a000 tx=0x7f990c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.143+0000 7f9921ffb700 1 -- 172.21.15.3:0/1808250473 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9914042860 con 0x7f99240a7250 2022-07-21T01:42:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.267+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99240574e0 con 0x7f99240a7250 2022-07-21T01:42:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f9921ffb700 1 -- 172.21.15.3:0/1808250473 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f9914045e80 con 0x7f99240a7250 2022-07-21T01:42:10.274 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:10.275 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":140,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:08.066592+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:05.096631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:06.060734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9908053b00 msgr2=0x7f9908055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9908053b00 0x7f9908055fc0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f990c01a000 tx=0x7f990c015000).stop 2022-07-21T01:42:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99240a7250 msgr2=0x7f99240ab010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99240a7250 0x7f99240ab010 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f9914002f70 tx=0x7f9914006ec0).stop 2022-07-21T01:42:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 shutdown_connections 2022-07-21T01:42:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9924148530 0x7f9924143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9908053b00 0x7f9908055fc0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99240b1f40 0x7f9924143a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 --2- 172.21.15.3:0/1808250473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99240a7250 0x7f99240ab010 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 >> 172.21.15.3:0/1808250473 conn(0x7f992405f840 msgr2=0x7f99240b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 shutdown_connections 2022-07-21T01:42:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.271+0000 7f992a30f700 1 -- 172.21.15.3:0/1808250473 wait complete. 2022-07-21T01:42:10.288 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:42:05.096631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:42:06.060734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:42:10.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:42:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.395+0000 7f320ffff700 1 Processor -- start 2022-07-21T01:42:10.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.395+0000 7f320ffff700 1 -- start start 2022-07-21T01:42:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3210061ab0 0x7f3210143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3210063e40 con 0x7f3210063a60 2022-07-21T01:42:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3210062180 con 0x7f3210140b10 2022-07-21T01:42:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321013ff40 con 0x7f3210061ab0 2022-07-21T01:42:10.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320effd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3210063a60 0x7f32100b1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59010/0 (socket says 172.21.15.3:59010) 2022-07-21T01:42:10.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320effd700 1 -- 172.21.15.3:0/2646801576 learned_addr learned my addr 172.21.15.3:0/2646801576 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:10.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f32067fc700 1 --2- 172.21.15.3:0/2646801576 >> v2:172.21.15.143:3300/0 conn(0x7f3210061ab0 0x7f3210143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320e7fc700 1 -- 172.21.15.3:0/2646801576 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019195614 0 0) 0x7f3210063e40 con 0x7f3210063a60 2022-07-21T01:42:10.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320e7fc700 1 -- 172.21.15.3:0/2646801576 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f0003260 con 0x7f3210063a60 2022-07-21T01:42:10.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320e7fc700 1 -- 172.21.15.3:0/2646801576 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637464094 0 0) 0x7f3210062180 con 0x7f3210140b10 2022-07-21T01:42:10.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320e7fc700 1 -- 172.21.15.3:0/2646801576 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3210063e40 con 0x7f3210140b10 2022-07-21T01:42:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f32067fc700 1 -- 172.21.15.3:0/2646801576 >> v1:172.21.15.143:6789/0 conn(0x7f3210140b10 legacy=0x7f3210128e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f32067fc700 1 -- 172.21.15.3:0/2646801576 >> v1:172.21.15.3:6789/0 conn(0x7f3210063a60 legacy=0x7f32100b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f32067fc700 1 -- 172.21.15.3:0/2646801576 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210140130 con 0x7f3210061ab0 2022-07-21T01:42:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f32067fc700 1 --2- 172.21.15.3:0/2646801576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210061ab0 0x7f3210143ee0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f31f8002f70 tx=0x7f31f8006df0).ready entity=mon.1 client_cookie=aeec205a64d26a47 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320e7fc700 1 -- 172.21.15.3:0/2646801576 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31f800d540 con 0x7f3210061ab0 2022-07-21T01:42:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320e7fc700 1 -- 172.21.15.3:0/2646801576 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f8000e90 con 0x7f3210061ab0 2022-07-21T01:42:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320e7fc700 1 -- 172.21.15.3:0/2646801576 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31f800d890 con 0x7f3210061ab0 2022-07-21T01:42:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 -- 172.21.15.3:0/2646801576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210061ab0 msgr2=0x7f3210143ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 --2- 172.21.15.3:0/2646801576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210061ab0 0x7f3210143ee0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f31f8002f70 tx=0x7f31f8006df0).stop 2022-07-21T01:42:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 -- 172.21.15.3:0/2646801576 shutdown_connections 2022-07-21T01:42:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 --2- 172.21.15.3:0/2646801576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210061ab0 0x7f3210143ee0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 -- 172.21.15.3:0/2646801576 >> 172.21.15.3:0/2646801576 conn(0x7f321005c620 msgr2=0x7f321005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 -- 172.21.15.3:0/2646801576 shutdown_connections 2022-07-21T01:42:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.399+0000 7f320ffff700 1 -- 172.21.15.3:0/2646801576 wait complete. 2022-07-21T01:42:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 Processor -- start 2022-07-21T01:42:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 -- start start 2022-07-21T01:42:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3210061ab0 0x7f32100af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210063a60 0x7f32100b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3210140b10 0x7f3210154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3210065660 con 0x7f3210140b10 2022-07-21T01:42:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3210064ad0 con 0x7f3210063a60 2022-07-21T01:42:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32100afbb0 con 0x7f3210061ab0 2022-07-21T01:42:10.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3210140b10 0x7f3210154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f32067fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210063a60 0x7f32100b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3210061ab0 0x7f32100af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f32067fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210063a60 0x7f32100b0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:56186/0 (socket says 172.21.15.3:56186) 2022-07-21T01:42:10.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f32067fc700 1 -- 172.21.15.3:0/2236233175 learned_addr learned my addr 172.21.15.3:0/2236233175 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320f7fe700 1 -- 172.21.15.3:0/2236233175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3210061ab0 msgr2=0x7f32100af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320f7fe700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3210061ab0 0x7f32100af1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320f7fe700 1 -- 172.21.15.3:0/2236233175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210063a60 msgr2=0x7f32100b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320f7fe700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210063a60 0x7f32100b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320f7fe700 1 -- 172.21.15.3:0/2236233175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f800b180 con 0x7f3210140b10 2022-07-21T01:42:10.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320f7fe700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3210140b10 0x7f3210154200 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f3208001490 tx=0x7f3208007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320cff9700 1 -- 172.21.15.3:0/2236233175 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f320800ce20 con 0x7f3210140b10 2022-07-21T01:42:10.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320cff9700 1 -- 172.21.15.3:0/2236233175 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320800f470 con 0x7f3210140b10 2022-07-21T01:42:10.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32100afe50 con 0x7f3210140b10 2022-07-21T01:42:10.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320cff9700 1 -- 172.21.15.3:0/2236233175 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f320800f7d0 con 0x7f3210140b10 2022-07-21T01:42:10.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.403+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3210066bc0 con 0x7f3210140b10 2022-07-21T01:42:10.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.407+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32100574e0 con 0x7f3210140b10 2022-07-21T01:42:10.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.407+0000 7f320cff9700 1 -- 172.21.15.3:0/2236233175 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3208019030 con 0x7f3210140b10 2022-07-21T01:42:10.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.407+0000 7f320cff9700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31f4053b30 0x7f31f4055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.407+0000 7f320cff9700 1 -- 172.21.15.3:0/2236233175 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f320806b540 con 0x7f3210140b10 2022-07-21T01:42:10.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.407+0000 7f320effd700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31f4053b30 0x7f31f4055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.407+0000 7f320effd700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31f4053b30 0x7f31f4055ff0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f320000d000 tx=0x7f3200009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.415+0000 7f320cff9700 1 -- 172.21.15.3:0/2236233175 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3208042300 con 0x7f3210140b10 2022-07-21T01:42:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f321013dbc0 con 0x7f3210140b10 2022-07-21T01:42:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320cff9700 1 -- 172.21.15.3:0/2236233175 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v140) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3208045920 con 0x7f3210140b10 2022-07-21T01:42:10.546 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:42:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31f4053b30 msgr2=0x7f31f4055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31f4053b30 0x7f31f4055ff0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f320000d000 tx=0x7f3200009040).stop 2022-07-21T01:42:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3210140b10 msgr2=0x7f3210154200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3210140b10 0x7f3210154200 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f3208001490 tx=0x7f3208007d40).stop 2022-07-21T01:42:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 shutdown_connections 2022-07-21T01:42:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3210140b10 0x7f3210154200 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f31f4053b30 0x7f31f4055ff0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210063a60 0x7f32100b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 --2- 172.21.15.3:0/2236233175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3210061ab0 0x7f32100af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 >> 172.21.15.3:0/2236233175 conn(0x7f321005c620 msgr2=0x7f3210128be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:10.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 shutdown_connections 2022-07-21T01:42:10.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.543+0000 7f320ffff700 1 -- 172.21.15.3:0/2236233175 wait complete. 2022-07-21T01:42:10.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:42:10.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.659+0000 7fad3cba7700 1 Processor -- start 2022-07-21T01:42:10.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.659+0000 7fad3cba7700 1 -- start start 2022-07-21T01:42:10.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380b31d0 0x7fad38066020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad380a9da0 0x7fad38060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad38067030 con 0x7fad381434b0 2022-07-21T01:42:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad38064a70 con 0x7fad380a9da0 2022-07-21T01:42:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad38061f60 con 0x7fad380b31d0 2022-07-21T01:42:10.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad377fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380b31d0 0x7fad38066020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad2ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad380a9da0 0x7fad38060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad37fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fad381434b0 0x7fad38143940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:32912/0 (socket says 172.21.15.3:32912) 2022-07-21T01:42:10.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad37fff700 1 -- 172.21.15.3:0/1705001729 learned_addr learned my addr 172.21.15.3:0/1705001729 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:10.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad377fe700 1 -- 172.21.15.3:0/1705001729 >> v2:172.21.15.143:3300/0 conn(0x7fad380a9da0 msgr2=0x7fad38060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:10.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad377fe700 1 --2- 172.21.15.3:0/1705001729 >> v2:172.21.15.143:3300/0 conn(0x7fad380a9da0 0x7fad38060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad377fe700 1 -- 172.21.15.3:0/1705001729 >> v1:172.21.15.143:6789/0 conn(0x7fad381434b0 legacy=0x7fad38143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad36ffd700 1 -- 172.21.15.3:0/1705001729 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675019548 0 0) 0x7fad38067030 con 0x7fad381434b0 2022-07-21T01:42:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad377fe700 1 -- 172.21.15.3:0/1705001729 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad380b35b0 con 0x7fad380b31d0 2022-07-21T01:42:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad377fe700 1 --2- 172.21.15.3:0/1705001729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380b31d0 0x7fad38066020 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fad28002f70 tx=0x7fad28006df0).ready entity=mon.2 client_cookie=8326515684b51893 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad36ffd700 1 -- 172.21.15.3:0/1705001729 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad2800d570 con 0x7fad380b31d0 2022-07-21T01:42:10.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad36ffd700 1 -- 172.21.15.3:0/1705001729 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad28000e90 con 0x7fad380b31d0 2022-07-21T01:42:10.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad36ffd700 1 -- 172.21.15.3:0/1705001729 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad2800d8c0 con 0x7fad380b31d0 2022-07-21T01:42:10.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 -- 172.21.15.3:0/1705001729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380b31d0 msgr2=0x7fad38066020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 --2- 172.21.15.3:0/1705001729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380b31d0 0x7fad38066020 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fad28002f70 tx=0x7fad28006df0).stop 2022-07-21T01:42:10.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 -- 172.21.15.3:0/1705001729 shutdown_connections 2022-07-21T01:42:10.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 --2- 172.21.15.3:0/1705001729 >> v2:172.21.15.143:3300/0 conn(0x7fad380a9da0 0x7fad38060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 --2- 172.21.15.3:0/1705001729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380b31d0 0x7fad38066020 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 -- 172.21.15.3:0/1705001729 >> 172.21.15.3:0/1705001729 conn(0x7fad38063870 msgr2=0x7fad38063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:10.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 -- 172.21.15.3:0/1705001729 shutdown_connections 2022-07-21T01:42:10.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.663+0000 7fad3cba7700 1 -- 172.21.15.3:0/1705001729 wait complete. 2022-07-21T01:42:10.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.667+0000 7fad3cba7700 1 Processor -- start 2022-07-21T01:42:10.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.667+0000 7fad3cba7700 1 -- start start 2022-07-21T01:42:10.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad3cba7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380a9da0 0x7fad380b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad3cba7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad380b31d0 0x7fad3814fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad3cba7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad381434b0 0x7fad3813db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad3cba7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380adc50 con 0x7fad380b31d0 2022-07-21T01:42:10.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad3cba7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380adef0 con 0x7fad381434b0 2022-07-21T01:42:10.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad3cba7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380ae190 con 0x7fad380a9da0 2022-07-21T01:42:10.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad2ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad380b31d0 0x7fad3814fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad2ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad380b31d0 0x7fad3814fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50752/0 (socket says 172.21.15.3:50752) 2022-07-21T01:42:10.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad377fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380a9da0 0x7fad380b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad2ffff700 1 -- 172.21.15.3:0/3210265734 learned_addr learned my addr 172.21.15.3:0/3210265734 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:10.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad37fff700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad381434b0 0x7fad3813db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad2ffff700 1 -- 172.21.15.3:0/3210265734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380a9da0 msgr2=0x7fad380b65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad2ffff700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380a9da0 0x7fad380b65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad2ffff700 1 -- 172.21.15.3:0/3210265734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad381434b0 msgr2=0x7fad3813db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad2ffff700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad381434b0 0x7fad3813db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad2ffff700 1 -- 172.21.15.3:0/3210265734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2800b180 con 0x7fad380b31d0 2022-07-21T01:42:10.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad2ffff700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad380b31d0 0x7fad3814fb00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7fad2000c000 tx=0x7fad20006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad357fa700 1 -- 172.21.15.3:0/3210265734 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad20011920 con 0x7fad380b31d0 2022-07-21T01:42:10.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad357fa700 1 -- 172.21.15.3:0/3210265734 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad20011a80 con 0x7fad380b31d0 2022-07-21T01:42:10.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3814fff0 con 0x7fad380b31d0 2022-07-21T01:42:10.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad357fa700 1 -- 172.21.15.3:0/3210265734 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad20011de0 con 0x7fad380b31d0 2022-07-21T01:42:10.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad380ad910 con 0x7fad380b31d0 2022-07-21T01:42:10.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3813e360 con 0x7fad380b31d0 2022-07-21T01:42:10.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.671+0000 7fad357fa700 1 -- 172.21.15.3:0/3210265734 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fad20021070 con 0x7fad380b31d0 2022-07-21T01:42:10.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.675+0000 7fad357fa700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad1c053950 0x7fad1c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.675+0000 7fad357fa700 1 -- 172.21.15.3:0/3210265734 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fad2006dcd0 con 0x7fad380b31d0 2022-07-21T01:42:10.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.675+0000 7fad377fe700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad1c053950 0x7fad1c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.675+0000 7fad377fe700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad1c053950 0x7fad1c055e10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fad2801a000 tx=0x7fad28014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.683+0000 7fad357fa700 1 -- 172.21.15.3:0/3210265734 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad20044a30 con 0x7fad380b31d0 2022-07-21T01:42:10.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fad380ae350 con 0x7fad380b31d0 2022-07-21T01:42:10.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad357fa700 1 -- 172.21.15.3:0/3210265734 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v140) v1 ==== 92+0+10 (secure 0 0 0) 0x7fad20048050 con 0x7fad380b31d0 2022-07-21T01:42:10.815 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:10.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad1c053950 msgr2=0x7fad1c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad1c053950 0x7fad1c055e10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fad2801a000 tx=0x7fad28014000).stop 2022-07-21T01:42:10.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad380b31d0 msgr2=0x7fad3814fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad380b31d0 0x7fad3814fb00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7fad2000c000 tx=0x7fad20006df0).stop 2022-07-21T01:42:10.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 shutdown_connections 2022-07-21T01:42:10.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fad1c053950 0x7fad1c055e10 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad381434b0 0x7fad3813db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad380b31d0 0x7fad3814fb00 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 --2- 172.21.15.3:0/3210265734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad380a9da0 0x7fad380b65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 >> 172.21.15.3:0/3210265734 conn(0x7fad38063870 msgr2=0x7fad38065f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.811+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 shutdown_connections 2022-07-21T01:42:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.815+0000 7fad3cba7700 1 -- 172.21.15.3:0/3210265734 wait complete. 2022-07-21T01:42:10.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:42:10.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6333fff700 1 Processor -- start 2022-07-21T01:42:10.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6333fff700 1 -- start start 2022-07-21T01:42:10.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6333fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 0x7f63340af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6333fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6334147730 0x7f63340acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6333fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63340b5a10 con 0x7f63340b5630 2022-07-21T01:42:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6333fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6334145470 con 0x7f6334147730 2022-07-21T01:42:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6333fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63340ae180 con 0x7f63340b07a0 2022-07-21T01:42:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f63337fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f6334147730 0x7f63340acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6332ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f63340b5630 0x7f6334067730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59040/0 (socket says 172.21.15.3:59040) 2022-07-21T01:42:10.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6332ffd700 1 -- 172.21.15.3:0/1819855446 learned_addr learned my addr 172.21.15.3:0/1819855446 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:10.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f63327fc700 1 --2- 172.21.15.3:0/1819855446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 0x7f63340af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f63337fe700 1 -- 172.21.15.3:0/1819855446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 msgr2=0x7f63340af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:10.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f63337fe700 1 --2- 172.21.15.3:0/1819855446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 0x7f63340af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f63337fe700 1 -- 172.21.15.3:0/1819855446 >> v1:172.21.15.3:6789/0 conn(0x7f63340b5630 legacy=0x7f6334067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f63337fe700 1 -- 172.21.15.3:0/1819855446 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63340ac990 con 0x7f6334147730 2022-07-21T01:42:10.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f63327fc700 1 --2- 172.21.15.3:0/1819855446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 0x7f63340af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:10.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f63337fe700 1 --2- 172.21.15.3:0/1819855446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 0x7f63340acdf0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7f632c002f70 tx=0x7f632c006df0).ready entity=mon.0 client_cookie=775f0d2aa538921f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6331ffb700 1 -- 172.21.15.3:0/1819855446 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f632c00d570 con 0x7f6334147730 2022-07-21T01:42:10.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6331ffb700 1 -- 172.21.15.3:0/1819855446 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632c000e90 con 0x7f6334147730 2022-07-21T01:42:10.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.935+0000 7f6331ffb700 1 -- 172.21.15.3:0/1819855446 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f632c00d8c0 con 0x7f6334147730 2022-07-21T01:42:10.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 -- 172.21.15.3:0/1819855446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 msgr2=0x7f63340acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 --2- 172.21.15.3:0/1819855446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 0x7f63340acdf0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7f632c002f70 tx=0x7f632c006df0).stop 2022-07-21T01:42:10.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 -- 172.21.15.3:0/1819855446 shutdown_connections 2022-07-21T01:42:10.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 --2- 172.21.15.3:0/1819855446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 0x7f63340acdf0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 --2- 172.21.15.3:0/1819855446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 0x7f63340af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 -- 172.21.15.3:0/1819855446 >> 172.21.15.3:0/1819855446 conn(0x7f63340626f0 msgr2=0x7f6334062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:10.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 -- 172.21.15.3:0/1819855446 shutdown_connections 2022-07-21T01:42:10.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 -- 172.21.15.3:0/1819855446 wait complete. 2022-07-21T01:42:10.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 Processor -- start 2022-07-21T01:42:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 -- start start 2022-07-21T01:42:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 0x7f63340663c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63340b5630 0x7f633414ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 0x7f6334065a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63340b3d30 con 0x7f6334147730 2022-07-21T01:42:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63340b3fd0 con 0x7f63340b5630 2022-07-21T01:42:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6333fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63340b4270 con 0x7f63340b07a0 2022-07-21T01:42:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f63337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 0x7f6334065a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f63337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 0x7f6334065a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50768/0 (socket says 172.21.15.3:50768) 2022-07-21T01:42:10.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f63337fe700 1 -- 172.21.15.3:0/96081388 learned_addr learned my addr 172.21.15.3:0/96081388 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f6332ffd700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 0x7f63340663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.939+0000 7f63327fc700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63340b5630 0x7f633414ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f63337fe700 1 -- 172.21.15.3:0/96081388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 msgr2=0x7f63340663c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f63337fe700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 0x7f63340663c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f63337fe700 1 -- 172.21.15.3:0/96081388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63340b5630 msgr2=0x7f633414ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:10.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f63337fe700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63340b5630 0x7f633414ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:10.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f63337fe700 1 -- 172.21.15.3:0/96081388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c00b180 con 0x7f6334147730 2022-07-21T01:42:10.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f63337fe700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 0x7f6334065a60 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7f632c002f70 tx=0x7f632c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f6323fff700 1 -- 172.21.15.3:0/96081388 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f632c011760 con 0x7f6334147730 2022-07-21T01:42:10.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f6323fff700 1 -- 172.21.15.3:0/96081388 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632c0118c0 con 0x7f6334147730 2022-07-21T01:42:10.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63340b3a10 con 0x7f6334147730 2022-07-21T01:42:10.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f6323fff700 1 -- 172.21.15.3:0/96081388 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f632c011c20 con 0x7f6334147730 2022-07-21T01:42:10.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63340660f0 con 0x7f6334147730 2022-07-21T01:42:10.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63340b4430 con 0x7f6334147730 2022-07-21T01:42:10.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f6323fff700 1 -- 172.21.15.3:0/96081388 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f632c025030 con 0x7f6334147730 2022-07-21T01:42:10.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.943+0000 7f6323fff700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6310053a70 0x7f6310055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:10.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.947+0000 7f6332ffd700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6310053a70 0x7f6310055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.947+0000 7f6323fff700 1 -- 172.21.15.3:0/96081388 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f632c06d270 con 0x7f6334147730 2022-07-21T01:42:10.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.947+0000 7f6332ffd700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6310053a70 0x7f6310055f30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f632800d000 tx=0x7f6328009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:10.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:10.951+0000 7f6323fff700 1 -- 172.21.15.3:0/96081388 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f632c043f20 con 0x7f6334147730 2022-07-21T01:42:11.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f633405f3d0 con 0x7f6334147730 2022-07-21T01:42:11.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6323fff700 1 -- 172.21.15.3:0/96081388 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+10 (secure 0 0 0) 0x7f632c047540 con 0x7f6334147730 2022-07-21T01:42:11.083 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6310053a70 msgr2=0x7f6310055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6310053a70 0x7f6310055f30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f632800d000 tx=0x7f6328009040).stop 2022-07-21T01:42:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 msgr2=0x7f6334065a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 0x7f6334065a60 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7f632c002f70 tx=0x7f632c015000).stop 2022-07-21T01:42:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 shutdown_connections 2022-07-21T01:42:11.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6334147730 0x7f6334065a60 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6310053a70 0x7f6310055f30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63340b5630 0x7f633414ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 --2- 172.21.15.3:0/96081388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63340b07a0 0x7f63340663c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 >> 172.21.15.3:0/96081388 conn(0x7f63340626f0 msgr2=0x7f6334067720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 shutdown_connections 2022-07-21T01:42:11.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.075+0000 7f6333fff700 1 -- 172.21.15.3:0/96081388 wait complete. 2022-07-21T01:42:11.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:42:11.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.207+0000 7f0becee4700 1 Processor -- start 2022-07-21T01:42:11.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.207+0000 7f0becee4700 1 -- start start 2022-07-21T01:42:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0be80b0740 0x7f0be80b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0be80a9da0 0x7f0be80b1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be8143ab0 con 0x7f0be80b38a0 2022-07-21T01:42:11.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be8143d50 con 0x7f0be80b0740 2022-07-21T01:42:11.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be8143ff0 con 0x7f0be80a9da0 2022-07-21T01:42:11.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0bdffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0be80b38a0 0x7f0be813d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59054/0 (socket says 172.21.15.3:59054) 2022-07-21T01:42:11.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0bdffff700 1 -- 172.21.15.3:0/3372102085 learned_addr learned my addr 172.21.15.3:0/3372102085 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be659c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0be80b0740 0x7f0be80b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be6d9d700 1 --2- 172.21.15.3:0/3372102085 >> v2:172.21.15.143:3300/0 conn(0x7f0be80a9da0 0x7f0be80b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be5d9b700 1 -- 172.21.15.3:0/3372102085 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534948566 0 0) 0x7f0be8143ab0 con 0x7f0be80b38a0 2022-07-21T01:42:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be659c700 1 -- 172.21.15.3:0/3372102085 >> v2:172.21.15.143:3300/0 conn(0x7f0be80a9da0 msgr2=0x7f0be80b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:11.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be659c700 1 --2- 172.21.15.3:0/3372102085 >> v2:172.21.15.143:3300/0 conn(0x7f0be80a9da0 0x7f0be80b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be659c700 1 -- 172.21.15.3:0/3372102085 >> v1:172.21.15.3:6789/0 conn(0x7f0be80b38a0 legacy=0x7f0be813d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be659c700 1 -- 172.21.15.3:0/3372102085 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be8144290 con 0x7f0be80b0740 2022-07-21T01:42:11.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be659c700 1 --2- 172.21.15.3:0/3372102085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80b0740 0x7f0be80b0b20 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f0bd8002f70 tx=0x7f0bd8006df0).ready entity=mon.0 client_cookie=77b394ddff186533 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:11.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be5d9b700 1 -- 172.21.15.3:0/3372102085 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bd800d570 con 0x7f0be80b0740 2022-07-21T01:42:11.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be5d9b700 1 -- 172.21.15.3:0/3372102085 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd8000e90 con 0x7f0be80b0740 2022-07-21T01:42:11.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0be5d9b700 1 -- 172.21.15.3:0/3372102085 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bd800d8c0 con 0x7f0be80b0740 2022-07-21T01:42:11.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 -- 172.21.15.3:0/3372102085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80b0740 msgr2=0x7f0be80b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 --2- 172.21.15.3:0/3372102085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80b0740 0x7f0be80b0b20 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f0bd8002f70 tx=0x7f0bd8006df0).stop 2022-07-21T01:42:11.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 -- 172.21.15.3:0/3372102085 shutdown_connections 2022-07-21T01:42:11.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 --2- 172.21.15.3:0/3372102085 >> v2:172.21.15.143:3300/0 conn(0x7f0be80a9da0 0x7f0be80b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 --2- 172.21.15.3:0/3372102085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80b0740 0x7f0be80b0b20 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 -- 172.21.15.3:0/3372102085 >> 172.21.15.3:0/3372102085 conn(0x7f0be8060950 msgr2=0x7f0be8060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 -- 172.21.15.3:0/3372102085 shutdown_connections 2022-07-21T01:42:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.211+0000 7f0becee4700 1 -- 172.21.15.3:0/3372102085 wait complete. 2022-07-21T01:42:11.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.215+0000 7f0becee4700 1 Processor -- start 2022-07-21T01:42:11.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.215+0000 7f0becee4700 1 -- start start 2022-07-21T01:42:11.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.215+0000 7f0becee4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80a9da0 0x7f0be8067530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.215+0000 7f0becee4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be80b0740 0x7f0be8150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.215+0000 7f0becee4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be80b38a0 0x7f0be8150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.215+0000 7f0becee4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be8063a60 con 0x7f0be80a9da0 2022-07-21T01:42:11.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.215+0000 7f0becee4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be81434a0 con 0x7f0be80b38a0 2022-07-21T01:42:11.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.215+0000 7f0becee4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be8063c90 con 0x7f0be80b0740 2022-07-21T01:42:11.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.215+0000 7f0be659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80a9da0 0x7f0be8067530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80a9da0 0x7f0be8067530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50778/0 (socket says 172.21.15.3:50778) 2022-07-21T01:42:11.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be659c700 1 -- 172.21.15.3:0/3935250497 learned_addr learned my addr 172.21.15.3:0/3935250497 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:11.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be6d9d700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be80b38a0 0x7f0be8150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0bdffff700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be80b0740 0x7f0be8150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be659c700 1 -- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be80b0740 msgr2=0x7f0be8150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be659c700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be80b0740 0x7f0be8150470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be659c700 1 -- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be80b38a0 msgr2=0x7f0be8150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be659c700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be80b38a0 0x7f0be8150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be659c700 1 -- 172.21.15.3:0/3935250497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd800b180 con 0x7f0be80a9da0 2022-07-21T01:42:11.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0bdffff700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be80b0740 0x7f0be8150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:11.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be6d9d700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be80b38a0 0x7f0be8150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0be659c700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80a9da0 0x7f0be8067530 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7f0bd8002f70 tx=0x7f0bd8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0bdf7fe700 1 -- 172.21.15.3:0/3935250497 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bd800de40 con 0x7f0be80a9da0 2022-07-21T01:42:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0bdf7fe700 1 -- 172.21.15.3:0/3935250497 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd801e400 con 0x7f0be80a9da0 2022-07-21T01:42:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be8065b90 con 0x7f0be80a9da0 2022-07-21T01:42:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0bdf7fe700 1 -- 172.21.15.3:0/3935250497 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bd801e760 con 0x7f0be80a9da0 2022-07-21T01:42:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be80aa2c0 con 0x7f0be80a9da0 2022-07-21T01:42:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0bdf7fe700 1 -- 172.21.15.3:0/3935250497 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0bd8025030 con 0x7f0be80a9da0 2022-07-21T01:42:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be80574e0 con 0x7f0be80a9da0 2022-07-21T01:42:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0bdf7fe700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bcc0539e0 0x7f0bcc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0bdf7fe700 1 -- 172.21.15.3:0/3935250497 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f0bd806cd70 con 0x7f0be80a9da0 2022-07-21T01:42:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.219+0000 7f0bdffff700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bcc0539e0 0x7f0bcc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.223+0000 7f0bdffff700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bcc0539e0 0x7f0bcc055ea0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f0bd0008000 tx=0x7f0bd0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.227+0000 7f0bdf7fe700 1 -- 172.21.15.3:0/3935250497 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0bd8043b30 con 0x7f0be80a9da0 2022-07-21T01:42:11.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.351+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0be80447e0 con 0x7f0be80a9da0 2022-07-21T01:42:11.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.351+0000 7f0bdf7fe700 1 -- 172.21.15.3:0/3935250497 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0bd8023070 con 0x7f0be80a9da0 2022-07-21T01:42:11.359 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:11.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bcc0539e0 msgr2=0x7f0bcc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bcc0539e0 0x7f0bcc055ea0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f0bd0008000 tx=0x7f0bd0009040).stop 2022-07-21T01:42:11.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80a9da0 msgr2=0x7f0be8067530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80a9da0 0x7f0be8067530 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7f0bd8002f70 tx=0x7f0bd8014000).stop 2022-07-21T01:42:11.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 shutdown_connections 2022-07-21T01:42:11.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0bcc0539e0 0x7f0bcc055ea0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be80b38a0 0x7f0be8150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be80b0740 0x7f0be8150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 --2- 172.21.15.3:0/3935250497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0be80a9da0 0x7f0be8067530 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 >> 172.21.15.3:0/3935250497 conn(0x7f0be8060950 msgr2=0x7f0be81458c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:11.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 shutdown_connections 2022-07-21T01:42:11.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.355+0000 7f0becee4700 1 -- 172.21.15.3:0/3935250497 wait complete. 2022-07-21T01:42:11.369 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:42:11.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:11.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90f6b69700 1 Processor -- start 2022-07-21T01:42:11.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90f6b69700 1 -- start start 2022-07-21T01:42:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90f6b69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90f00b1f40 0x7f90f00b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90f6b69700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f00b04e0 con 0x7f90f0148530 2022-07-21T01:42:11.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90f6b69700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f00b0780 con 0x7f90f00a7250 2022-07-21T01:42:11.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90f6b69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00a9250 con 0x7f90f00b1f40 2022-07-21T01:42:11.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90effff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f90f0148530 0x7f90f00b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59066/0 (socket says 172.21.15.3:59066) 2022-07-21T01:42:11.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90effff700 1 -- 172.21.15.3:0/2339501274 learned_addr learned my addr 172.21.15.3:0/2339501274 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:11.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90f4905700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90f00b1f40 0x7f90f00b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984839748 0 0) 0x7f90f00b04e0 con 0x7f90f0148530 2022-07-21T01:42:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90dc003260 con 0x7f90f0148530 2022-07-21T01:42:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425439529 0 0) 0x7f90f00b0780 con 0x7f90f00a7250 2022-07-21T01:42:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90f00b04e0 con 0x7f90f00a7250 2022-07-21T01:42:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1633268581 0 0) 0x7f90dc003260 con 0x7f90f0148530 2022-07-21T01:42:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90f00b1f40 msgr2=0x7f90f00b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90f00b1f40 0x7f90f00b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 >> v1:172.21.15.143:6789/0 conn(0x7f90f00a7250 legacy=0x7f90f01448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f00b0a20 con 0x7f90f0148530 2022-07-21T01:42:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f90e80019e0 con 0x7f90f0148530 2022-07-21T01:42:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 >> v1:172.21.15.3:6789/0 conn(0x7f90f0148530 legacy=0x7f90f00b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90dc003910 0x7f90dc003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90f00b1f40 0x7f90dc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90dc0049f0 0x7f90dc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc00b750 con 0x7f90dc003910 2022-07-21T01:42:11.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc00b8c0 con 0x7f90dc0049f0 2022-07-21T01:42:11.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc00ba80 con 0x7f90f00b1f40 2022-07-21T01:42:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90f4905700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90dc003910 0x7f90dc003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90f5106700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90dc0049f0 0x7f90dc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.479+0000 7f90effff700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90f00b1f40 0x7f90dc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f4905700 1 -- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90f00b1f40 msgr2=0x7f90dc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f4905700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90f00b1f40 0x7f90dc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f4905700 1 -- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90dc0049f0 msgr2=0x7f90dc004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:11.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f4905700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90dc0049f0 0x7f90dc004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f4905700 1 -- 172.21.15.3:0/2339501274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc003260 con 0x7f90dc003910 2022-07-21T01:42:11.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f5106700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90dc0049f0 0x7f90dc004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:11.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f4905700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90dc003910 0x7f90dc003d10 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0x7f90e400c000 tx=0x7f90e4006df0).ready entity=mon.0 client_cookie=19f8a4f33e48a52e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:11.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e400e830 con 0x7f90dc003910 2022-07-21T01:42:11.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e400e990 con 0x7f90dc003910 2022-07-21T01:42:11.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90ef7fe700 1 -- 172.21.15.3:0/2339501274 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e400ecf0 con 0x7f90dc003910 2022-07-21T01:42:11.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f6b69700 1 -- 172.21.15.3:0/2339501274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90dc003910 msgr2=0x7f90dc003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f6b69700 1 --2- 172.21.15.3:0/2339501274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90dc003910 0x7f90dc003d10 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0x7f90e400c000 tx=0x7f90e4006df0).stop 2022-07-21T01:42:11.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f5106700 1 -- 172.21.15.3:0/2339501274 reap_dead start 2022-07-21T01:42:11.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f6b69700 1 -- 172.21.15.3:0/2339501274 shutdown_connections 2022-07-21T01:42:11.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f6b69700 1 -- 172.21.15.3:0/2339501274 >> 172.21.15.3:0/2339501274 conn(0x7f90f005f840 msgr2=0x7f90f005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:11.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f6b69700 1 -- 172.21.15.3:0/2339501274 shutdown_connections 2022-07-21T01:42:11.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.483+0000 7f90f6b69700 1 -- 172.21.15.3:0/2339501274 wait complete. 2022-07-21T01:42:11.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f6b69700 1 Processor -- start 2022-07-21T01:42:11.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f6b69700 1 -- start start 2022-07-21T01:42:11.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f6b69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90dc003910 0x7f90f00ac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f6b69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f00aad00 0x7f90f0064590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f6b69700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90f00ab490 0x7f90f0064a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f6b69700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00afa30 con 0x7f90f00aad00 2022-07-21T01:42:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f6b69700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00aefb0 con 0x7f90f00ab490 2022-07-21T01:42:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f6b69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00af250 con 0x7f90dc003910 2022-07-21T01:42:11.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f00aad00 0x7f90f0064590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f00aad00 0x7f90f0064590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50800/0 (socket says 172.21.15.3:50800) 2022-07-21T01:42:11.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90effff700 1 -- 172.21.15.3:0/2921452840 learned_addr learned my addr 172.21.15.3:0/2921452840 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:11.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f5106700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90f00ab490 0x7f90f0064a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90f4905700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90dc003910 0x7f90f00ac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90effff700 1 -- 172.21.15.3:0/2921452840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90dc003910 msgr2=0x7f90f00ac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90effff700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90dc003910 0x7f90f00ac2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90effff700 1 -- 172.21.15.3:0/2921452840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90f00ab490 msgr2=0x7f90f0064a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90effff700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90f00ab490 0x7f90f0064a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90effff700 1 -- 172.21.15.3:0/2921452840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f00af4f0 con 0x7f90f00aad00 2022-07-21T01:42:11.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90effff700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f00aad00 0x7f90f0064590 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f90e800d000 tx=0x7f90f00b0910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:11.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.487+0000 7f90edffb700 1 -- 172.21.15.3:0/2921452840 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e800f880 con 0x7f90f00aad00 2022-07-21T01:42:11.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90edffb700 1 -- 172.21.15.3:0/2921452840 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e800f9e0 con 0x7f90f00aad00 2022-07-21T01:42:11.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f00af790 con 0x7f90f00aad00 2022-07-21T01:42:11.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90edffb700 1 -- 172.21.15.3:0/2921452840 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e800fd40 con 0x7f90f00aad00 2022-07-21T01:42:11.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f0159fc0 con 0x7f90f00aad00 2022-07-21T01:42:11.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f0064280 con 0x7f90f00aad00 2022-07-21T01:42:11.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90edffb700 1 -- 172.21.15.3:0/2921452840 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f90e801c360 con 0x7f90f00aad00 2022-07-21T01:42:11.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90edffb700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90d0053b30 0x7f90d0055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:11.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90edffb700 1 -- 172.21.15.3:0/2921452840 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f90e806bbe0 con 0x7f90f00aad00 2022-07-21T01:42:11.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90f4905700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90d0053b30 0x7f90d0055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:11.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.491+0000 7f90f4905700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90d0053b30 0x7f90d0055ff0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f90e401e000 tx=0x7f90e4019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:11.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.499+0000 7f90edffb700 1 -- 172.21.15.3:0/2921452840 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90e8042940 con 0x7f90f00aad00 2022-07-21T01:42:11.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.659+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90f00574e0 con 0x7f90f00aad00 2022-07-21T01:42:11.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90edffb700 1 -- 172.21.15.3:0/2921452840 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f90e8019030 con 0x7f90f00aad00 2022-07-21T01:42:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:11.669 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":71,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/567682837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":567682837},{"type":"v1","addr":"172.21.15.143:6833","nonce":567682837}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/140242740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140242740},{"type":"v1","addr":"172.21.15.3:6835","nonce":140242740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:07.063040+0000","modified":"2022-07-21T01:42:08.065747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8049":{"gid":8049,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/506442168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":506442168},{"type":"v1","addr":"172.21.15.3:6837","nonce":506442168}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-21T01:42:11.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90d0053b30 msgr2=0x7f90d0055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90d0053b30 0x7f90d0055ff0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f90e401e000 tx=0x7f90e4019000).stop 2022-07-21T01:42:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f00aad00 msgr2=0x7f90f0064590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f00aad00 0x7f90f0064590 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f90e800d000 tx=0x7f90f00b0910).stop 2022-07-21T01:42:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 shutdown_connections 2022-07-21T01:42:11.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f90d0053b30 0x7f90d0055ff0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90f00ab490 0x7f90f0064a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90f00aad00 0x7f90f0064590 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 --2- 172.21.15.3:0/2921452840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90dc003910 0x7f90f00ac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:11.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 >> 172.21.15.3:0/2921452840 conn(0x7f90f005f840 msgr2=0x7f90f01470e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:11.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 shutdown_connections 2022-07-21T01:42:11.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:11.663+0000 7f90f6b69700 1 -- 172.21.15.3:0/2921452840 wait complete. 2022-07-21T01:42:11.675 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 71 2022-07-21T01:42:11.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:07.063040+0000', 'modified': '2022-07-21T01:42:08.065747+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8049}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8049': {'gid': 8049, 'name': 'a', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/506442168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 506442168}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 506442168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:42:11.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:42:11.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:42:12.094 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8049,"want_state":"up:active","state":"up:active","rank_uptime":5.019791144,"mdsmap_epoch":71,"osdmap_epoch":140,"osdmap_epoch_barrier":139,"uptime":9.7494078460000004} 2022-07-21T01:42:12.095 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8049, "mdsmap_epoch": 71, "osdmap_epoch": 140, "osdmap_epoch_barrier": 139, "rank_uptime": 5.019791144, "state": "up:active", "uptime": 9.749407846, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:42:12.096 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:42:12.097 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:42:12.098 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:42:12.099 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:12.100 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:12.110 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:42:12.169 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:12.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:12.184 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:12.185 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:12.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:12.188 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:42:12.235 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:42:12.236 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:12.238 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:42:12.256 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:12.257 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:12.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:42:12.311+0000 7fd873209080 -1 init, newargv = 0x55e02309f980 newargc=9ceph-fuse[23330]: starting ceph client 2022-07-21T01:42:12.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr: 2022-07-21T01:42:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:12.317 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:12.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:12.318 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:42:12.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23330]: starting fuse 2022-07-21T01:42:12.324 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:42:12.325 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:42:12.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:12.326 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:42:12.326 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:42:12.327 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:42:12.327 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:42:12.327 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:42:12.328 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:42:12.328 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:42:12.329 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:42:12.329 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:42:12.330 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:42:12.330 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:42:12.331 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:42:12.331 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:42:12.331 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:42:12.332 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:42:12.332 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:42:12.333 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:42:12.333 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:42:12.333 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:42:12.334 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:42:12.334 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:42:12.335 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:42:12.335 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:42:12.378 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.23330.asok 2022-07-21T01:42:12.382 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23330.asok 2022-07-21T01:42:12.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:12.384 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23330.asok status 2022-07-21T01:42:12.515 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:42:12.515 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:42:12.516 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:42:12.516 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:42:12.517 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:42:12.517 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:42:12.518 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:42:12.518 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "23330", 2022-07-21T01:42:12.518 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:42:12.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:42:12.519 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:42:12.519 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:42:12.519 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 7028, 2022-07-21T01:42:12.519 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:42:12.520 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:42:12.520 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:42:12.520 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 7028 2022-07-21T01:42:12.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:42:12.521 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:42:12.521 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:42:12.521 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:42:12.521 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3432337665 2022-07-21T01:42:12.522 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:42:12.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:42:12.522 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:42:12.522 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:42:12.523 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:42:12.523 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3432337665 2022-07-21T01:42:12.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:42:12.523 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.7028 172.21.15.171:0/3432337665", 2022-07-21T01:42:12.523 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/3432337665", 2022-07-21T01:42:12.524 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:42:12.524 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 71, 2022-07-21T01:42:12.524 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 140, 2022-07-21T01:42:12.524 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:42:12.525 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:42:12.525 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:42:12.525 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:42:12.526 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:12.527 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:12.527 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:12.537 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:42:12.537 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:42:12.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:12.538 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:12.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:12.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.663+0000 7fdc3ccfc700 1 Processor -- start 2022-07-21T01:42:12.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.663+0000 7fdc3ccfc700 1 -- start start 2022-07-21T01:42:12.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380614b0 0x7fdc38061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdc380ad750 0x7fdc380b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc381405d0 0x7fdc38140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc380aee80 con 0x7fdc381405d0 2022-07-21T01:42:12.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc380b70b0 con 0x7fdc380ad750 2022-07-21T01:42:12.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380af0b0 con 0x7fdc380614b0 2022-07-21T01:42:12.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc37fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdc381405d0 0x7fdc38140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380614b0 0x7fdc38061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380614b0 0x7fdc38061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:57722/0 (socket says 172.21.15.3:57722) 2022-07-21T01:42:12.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc377fe700 1 -- 172.21.15.3:0/3079159449 learned_addr learned my addr 172.21.15.3:0/3079159449 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:12.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc36ffd700 1 --2- 172.21.15.3:0/3079159449 >> v2:172.21.15.143:3300/0 conn(0x7fdc380ad750 0x7fdc380b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc377fe700 1 -- 172.21.15.3:0/3079159449 >> v2:172.21.15.143:3300/0 conn(0x7fdc380ad750 msgr2=0x7fdc380b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:12.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc377fe700 1 --2- 172.21.15.3:0/3079159449 >> v2:172.21.15.143:3300/0 conn(0x7fdc380ad750 0x7fdc380b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc377fe700 1 -- 172.21.15.3:0/3079159449 >> v2:172.21.15.3:3300/0 conn(0x7fdc381405d0 msgr2=0x7fdc38140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:12.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc377fe700 1 --2- 172.21.15.3:0/3079159449 >> v2:172.21.15.3:3300/0 conn(0x7fdc381405d0 0x7fdc38140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc377fe700 1 -- 172.21.15.3:0/3079159449 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc380af350 con 0x7fdc380614b0 2022-07-21T01:42:12.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc37fff700 1 --2- 172.21.15.3:0/3079159449 >> v2:172.21.15.3:3300/0 conn(0x7fdc381405d0 0x7fdc38140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:12.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc377fe700 1 --2- 172.21.15.3:0/3079159449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380614b0 0x7fdc38061890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fdc280032f0 tx=0x7fdc28009290).ready entity=mon.2 client_cookie=13c3211f905b0787 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:12.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc367fc700 1 -- 172.21.15.3:0/3079159449 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2800b680 con 0x7fdc380614b0 2022-07-21T01:42:12.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc367fc700 1 -- 172.21.15.3:0/3079159449 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2800b7e0 con 0x7fdc380614b0 2022-07-21T01:42:12.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc367fc700 1 -- 172.21.15.3:0/3079159449 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2800b950 con 0x7fdc380614b0 2022-07-21T01:42:12.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/3079159449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380614b0 msgr2=0x7fdc38061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:12.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/3079159449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380614b0 0x7fdc38061890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fdc280032f0 tx=0x7fdc28009290).stop 2022-07-21T01:42:12.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/3079159449 shutdown_connections 2022-07-21T01:42:12.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/3079159449 >> v2:172.21.15.3:3300/0 conn(0x7fdc381405d0 0x7fdc38140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/3079159449 >> v2:172.21.15.143:3300/0 conn(0x7fdc380ad750 0x7fdc380b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/3079159449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380614b0 0x7fdc38061890 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/3079159449 >> 172.21.15.3:0/3079159449 conn(0x7fdc38066f70 msgr2=0x7fdc38067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:12.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/3079159449 shutdown_connections 2022-07-21T01:42:12.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.667+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/3079159449 wait complete. 2022-07-21T01:42:12.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc3ccfc700 1 Processor -- start 2022-07-21T01:42:12.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc3ccfc700 1 -- start start 2022-07-21T01:42:12.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc3ccfc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc380614b0 0x7fdc38144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc3ccfc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380ad750 0x7fdc380afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc3ccfc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc381405d0 0x7fdc380affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc3ccfc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38152420 con 0x7fdc380614b0 2022-07-21T01:42:12.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc3ccfc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380b04a0 con 0x7fdc381405d0 2022-07-21T01:42:12.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc3ccfc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38152610 con 0x7fdc380ad750 2022-07-21T01:42:12.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc380614b0 0x7fdc38144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc380614b0 0x7fdc38144850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50812/0 (socket says 172.21.15.3:50812) 2022-07-21T01:42:12.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc377fe700 1 -- 172.21.15.3:0/515111414 learned_addr learned my addr 172.21.15.3:0/515111414 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:12.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc37fff700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc381405d0 0x7fdc380affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc36ffd700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380ad750 0x7fdc380afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc377fe700 1 -- 172.21.15.3:0/515111414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380ad750 msgr2=0x7fdc380afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc377fe700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380ad750 0x7fdc380afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc377fe700 1 -- 172.21.15.3:0/515111414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc381405d0 msgr2=0x7fdc380affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc377fe700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc381405d0 0x7fdc380affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc377fe700 1 -- 172.21.15.3:0/515111414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3000b180 con 0x7fdc380614b0 2022-07-21T01:42:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.671+0000 7fdc377fe700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc380614b0 0x7fdc38144850 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7fdc280032f0 tx=0x7fdc28017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc34ff9700 1 -- 172.21.15.3:0/515111414 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2800b680 con 0x7fdc380614b0 2022-07-21T01:42:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc34ff9700 1 -- 172.21.15.3:0/515111414 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2800b950 con 0x7fdc380614b0 2022-07-21T01:42:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc34ff9700 1 -- 172.21.15.3:0/515111414 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2801b600 con 0x7fdc380614b0 2022-07-21T01:42:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc28009530 con 0x7fdc380614b0 2022-07-21T01:42:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc38152b50 con 0x7fdc380614b0 2022-07-21T01:42:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc38152f90 con 0x7fdc380614b0 2022-07-21T01:42:12.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc34ff9700 1 -- 172.21.15.3:0/515111414 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc2800f070 con 0x7fdc380614b0 2022-07-21T01:42:12.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc34ff9700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc1c053a70 0x7fdc1c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc34ff9700 1 -- 172.21.15.3:0/515111414 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdc2806d480 con 0x7fdc380614b0 2022-07-21T01:42:12.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.675+0000 7fdc36ffd700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc1c053a70 0x7fdc1c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.679+0000 7fdc36ffd700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc1c053a70 0x7fdc1c055f30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fdc2000b000 tx=0x7fdc20006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:12.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.683+0000 7fdc34ff9700 1 -- 172.21.15.3:0/515111414 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc28044260 con 0x7fdc380614b0 2022-07-21T01:42:12.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.811+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc380574e0 con 0x7fdc380614b0 2022-07-21T01:42:12.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc34ff9700 1 -- 172.21.15.3:0/515111414 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16285 (secure 0 0 0) 0x7fdc2801f070 con 0x7fdc380614b0 2022-07-21T01:42:12.819 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:12.819 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":140,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:08.066592+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:05.096631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:06.060734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc1c053a70 msgr2=0x7fdc1c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:12.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc1c053a70 0x7fdc1c055f30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fdc2000b000 tx=0x7fdc20006df0).stop 2022-07-21T01:42:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc380614b0 msgr2=0x7fdc38144850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc380614b0 0x7fdc38144850 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7fdc280032f0 tx=0x7fdc28017000).stop 2022-07-21T01:42:12.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 shutdown_connections 2022-07-21T01:42:12.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc1c053a70 0x7fdc1c055f30 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc381405d0 0x7fdc380affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc380ad750 0x7fdc380afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 --2- 172.21.15.3:0/515111414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc380614b0 0x7fdc38144850 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 >> 172.21.15.3:0/515111414 conn(0x7fdc38066f70 msgr2=0x7fdc38141ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 shutdown_connections 2022-07-21T01:42:12.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.815+0000 7fdc3ccfc700 1 -- 172.21.15.3:0/515111414 wait complete. 2022-07-21T01:42:12.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:42:05.096631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:42:06.060734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:42:12.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:42:12.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f45ab7700 1 Processor -- start 2022-07-21T01:42:12.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f45ab7700 1 -- start start 2022-07-21T01:42:12.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f45ab7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4f400619b0 0x7f4f400b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f45ab7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f400b5ed0 0x7f4f400b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f45ab7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f40061d90 con 0x7f4f400b6ee0 2022-07-21T01:42:12.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f45ab7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f40143fa0 con 0x7f4f400619b0 2022-07-21T01:42:12.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f45ab7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f40142f60 con 0x7f4f400b5ed0 2022-07-21T01:42:12.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f3ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4f400b6ee0 0x7f4f40143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59104/0 (socket says 172.21.15.3:59104) 2022-07-21T01:42:12.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f3ffff700 1 -- 172.21.15.3:0/2567113613 learned_addr learned my addr 172.21.15.3:0/2567113613 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:12.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f44ab5700 1 --2- 172.21.15.3:0/2567113613 >> v2:172.21.15.143:3300/0 conn(0x7f4f400619b0 0x7f4f400b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.943+0000 7f4f452b6700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f400b5ed0 0x7f4f400b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372784632 0 0) 0x7f4f40061d90 con 0x7f4f400b6ee0 2022-07-21T01:42:12.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f20003260 con 0x7f4f400b6ee0 2022-07-21T01:42:12.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129115609 0 0) 0x7f4f20003260 con 0x7f4f400b6ee0 2022-07-21T01:42:12.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f400b5ed0 msgr2=0x7f4f400b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:12.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f400b5ed0 0x7f4f400b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 >> v2:172.21.15.143:3300/0 conn(0x7f4f400619b0 msgr2=0x7f4f400b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:12.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 --2- 172.21.15.3:0/2567113613 >> v2:172.21.15.143:3300/0 conn(0x7f4f400619b0 0x7f4f400b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f400ba5a0 con 0x7f4f400b6ee0 2022-07-21T01:42:12.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f4f30001680 con 0x7f4f400b6ee0 2022-07-21T01:42:12.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 >> v1:172.21.15.3:6789/0 conn(0x7f4f400b6ee0 legacy=0x7f4f40143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:12.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f20003780 0x7f4f20003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f400b5ed0 0x7f4f20004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f200049f0 0x7f4f20004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f40143fa0 con 0x7f4f20003780 2022-07-21T01:42:12.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2000b750 con 0x7f4f200049f0 2022-07-21T01:42:12.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2000b890 con 0x7f4f400b5ed0 2022-07-21T01:42:12.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f44ab5700 1 --2- 172.21.15.3:0/2567113613 >> v2:172.21.15.143:3300/0 conn(0x7f4f400619b0 0x7f4f400b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:12.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f44ab5700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f20003780 0x7f4f20003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3ffff700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f400b5ed0 0x7f4f20004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f452b6700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f200049f0 0x7f4f20004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f44ab5700 1 -- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f400b5ed0 msgr2=0x7f4f20004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:12.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f44ab5700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f400b5ed0 0x7f4f20004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f44ab5700 1 -- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f200049f0 msgr2=0x7f4f20004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:12.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f44ab5700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f200049f0 0x7f4f20004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f44ab5700 1 -- 172.21.15.3:0/2567113613 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f20003260 con 0x7f4f20003780 2022-07-21T01:42:12.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f44ab5700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f20003780 0x7f4f20003ba0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f4f2800c000 tx=0x7f4f28006df0).ready entity=mon.0 client_cookie=81ab0a2301a327ab server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:12.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f2800e830 con 0x7f4f20003780 2022-07-21T01:42:12.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2800e990 con 0x7f4f20003780 2022-07-21T01:42:12.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f3f7fe700 1 -- 172.21.15.3:0/2567113613 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f2800ecf0 con 0x7f4f20003780 2022-07-21T01:42:12.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2567113613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f20003780 msgr2=0x7f4f20003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:12.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f45ab7700 1 --2- 172.21.15.3:0/2567113613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f20003780 0x7f4f20003ba0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f4f2800c000 tx=0x7f4f28006df0).stop 2022-07-21T01:42:12.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f452b6700 1 -- 172.21.15.3:0/2567113613 reap_dead start 2022-07-21T01:42:12.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2567113613 shutdown_connections 2022-07-21T01:42:12.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2567113613 >> 172.21.15.3:0/2567113613 conn(0x7f4f4006af80 msgr2=0x7f4f4005fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:12.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2567113613 shutdown_connections 2022-07-21T01:42:12.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.947+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2567113613 wait complete. 2022-07-21T01:42:12.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f45ab7700 1 Processor -- start 2022-07-21T01:42:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f45ab7700 1 -- start start 2022-07-21T01:42:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f45ab7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f20003780 0x7f4f400b9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f45ab7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f400b9390 0x7f4f40144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f45ab7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f4013d780 0x7f4f4013af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f45ab7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f400af910 con 0x7f4f4013d780 2022-07-21T01:42:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f45ab7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f400b01c0 con 0x7f4f400b9390 2022-07-21T01:42:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f45ab7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f40144750 con 0x7f4f20003780 2022-07-21T01:42:12.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f452b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f4013d780 0x7f4f4013af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f452b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f4013d780 0x7f4f4013af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50834/0 (socket says 172.21.15.3:50834) 2022-07-21T01:42:12.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f452b6700 1 -- 172.21.15.3:0/2129899701 learned_addr learned my addr 172.21.15.3:0/2129899701 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:12.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f3ffff700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f400b9390 0x7f4f40144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f44ab5700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f20003780 0x7f4f400b9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f452b6700 1 -- 172.21.15.3:0/2129899701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f20003780 msgr2=0x7f4f400b9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:12.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f452b6700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f20003780 0x7f4f400b9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f452b6700 1 -- 172.21.15.3:0/2129899701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f400b9390 msgr2=0x7f4f40144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:12.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f452b6700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f400b9390 0x7f4f40144260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:12.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f452b6700 1 -- 172.21.15.3:0/2129899701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f400add30 con 0x7f4f4013d780 2022-07-21T01:42:12.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.951+0000 7f4f452b6700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f4013d780 0x7f4f4013af70 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f4f400b0850 tx=0x7f4f380091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:12.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f3dffb700 1 -- 172.21.15.3:0/2129899701 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f38014720 con 0x7f4f4013d780 2022-07-21T01:42:12.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f400b03f0 con 0x7f4f4013d780 2022-07-21T01:42:12.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f3dffb700 1 -- 172.21.15.3:0/2129899701 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f38014880 con 0x7f4f4013d780 2022-07-21T01:42:12.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f3dffb700 1 -- 172.21.15.3:0/2129899701 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f38014be0 con 0x7f4f4013d780 2022-07-21T01:42:12.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f400adfd0 con 0x7f4f4013d780 2022-07-21T01:42:12.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4013ddd0 con 0x7f4f4013d780 2022-07-21T01:42:12.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f3dffb700 1 -- 172.21.15.3:0/2129899701 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4f38014d40 con 0x7f4f4013d780 2022-07-21T01:42:12.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f3dffb700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f24053b30 0x7f4f24055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:12.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f3dffb700 1 -- 172.21.15.3:0/2129899701 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f4f3806c8a0 con 0x7f4f4013d780 2022-07-21T01:42:12.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.955+0000 7f4f44ab5700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f24053b30 0x7f4f24055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:12.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.959+0000 7f4f44ab5700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f24053b30 0x7f4f24055ff0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f4f2801e000 tx=0x7f4f28019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:12.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:12.963+0000 7f4f3dffb700 1 -- 172.21.15.3:0/2129899701 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f38043550 con 0x7f4f4013d780 2022-07-21T01:42:13.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.087+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4f400574e0 con 0x7f4f4013d780 2022-07-21T01:42:13.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f3dffb700 1 -- 172.21.15.3:0/2129899701 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v140) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4f38018020 con 0x7f4f4013d780 2022-07-21T01:42:13.096 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:42:13.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f24053b30 msgr2=0x7f4f24055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f24053b30 0x7f4f24055ff0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f4f2801e000 tx=0x7f4f28019000).stop 2022-07-21T01:42:13.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f4013d780 msgr2=0x7f4f4013af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f4013d780 0x7f4f4013af70 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f4f400b0850 tx=0x7f4f380091d0).stop 2022-07-21T01:42:13.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 shutdown_connections 2022-07-21T01:42:13.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f4013d780 0x7f4f4013af70 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4f24053b30 0x7f4f24055ff0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f400b9390 0x7f4f40144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 --2- 172.21.15.3:0/2129899701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f20003780 0x7f4f400b9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 >> 172.21.15.3:0/2129899701 conn(0x7f4f4006af80 msgr2=0x7f4f4005ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 shutdown_connections 2022-07-21T01:42:13.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.091+0000 7f4f45ab7700 1 -- 172.21.15.3:0/2129899701 wait complete. 2022-07-21T01:42:13.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:42:13.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99e545700 1 Processor -- start 2022-07-21T01:42:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99e545700 1 -- start start 2022-07-21T01:42:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99e545700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9980b01a0 0x7fd9980a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99e545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 0x7fd9980a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99e545700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9980a7250 con 0x7fd9980b2710 2022-07-21T01:42:13.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99e545700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9980a7480 con 0x7fd9980b01a0 2022-07-21T01:42:13.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99e545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9980a7720 con 0x7fd9980afd60 2022-07-21T01:42:13.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99d543700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd9980b2710 0x7fd99814afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59120/0 (socket says 172.21.15.3:59120) 2022-07-21T01:42:13.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99d543700 1 -- 172.21.15.3:0/3166648065 learned_addr learned my addr 172.21.15.3:0/3166648065 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:13.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99dd44700 1 --2- 172.21.15.3:0/3166648065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 0x7fd9980a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99cd42700 1 --2- 172.21.15.3:0/3166648065 >> v2:172.21.15.143:3300/0 conn(0x7fd9980b01a0 0x7fd9980a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd997fff700 1 -- 172.21.15.3:0/3166648065 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203018751 0 0) 0x7fd9980a7250 con 0x7fd9980b2710 2022-07-21T01:42:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd997fff700 1 -- 172.21.15.3:0/3166648065 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd984003260 con 0x7fd9980b2710 2022-07-21T01:42:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99dd44700 1 -- 172.21.15.3:0/3166648065 >> v2:172.21.15.143:3300/0 conn(0x7fd9980b01a0 msgr2=0x7fd9980a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99dd44700 1 --2- 172.21.15.3:0/3166648065 >> v2:172.21.15.143:3300/0 conn(0x7fd9980b01a0 0x7fd9980a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99dd44700 1 -- 172.21.15.3:0/3166648065 >> v1:172.21.15.3:6789/0 conn(0x7fd9980b2710 legacy=0x7fd99814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99dd44700 1 -- 172.21.15.3:0/3166648065 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9980a79c0 con 0x7fd9980afd60 2022-07-21T01:42:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd99dd44700 1 --2- 172.21.15.3:0/3166648065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 0x7fd9980a8a30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fd990002f70 tx=0x7fd990006df0).ready entity=mon.2 client_cookie=dbb92b28c49fe6b9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd997fff700 1 -- 172.21.15.3:0/3166648065 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd99000d4a0 con 0x7fd9980afd60 2022-07-21T01:42:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd997fff700 1 -- 172.21.15.3:0/3166648065 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd990000e90 con 0x7fd9980afd60 2022-07-21T01:42:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.211+0000 7fd997fff700 1 -- 172.21.15.3:0/3166648065 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd99000d7f0 con 0x7fd9980afd60 2022-07-21T01:42:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 -- 172.21.15.3:0/3166648065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 msgr2=0x7fd9980a8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 --2- 172.21.15.3:0/3166648065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 0x7fd9980a8a30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fd990002f70 tx=0x7fd990006df0).stop 2022-07-21T01:42:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 -- 172.21.15.3:0/3166648065 shutdown_connections 2022-07-21T01:42:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 --2- 172.21.15.3:0/3166648065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 0x7fd9980a8a30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 --2- 172.21.15.3:0/3166648065 >> v2:172.21.15.143:3300/0 conn(0x7fd9980b01a0 0x7fd9980a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 -- 172.21.15.3:0/3166648065 >> 172.21.15.3:0/3166648065 conn(0x7fd998060310 msgr2=0x7fd9980b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 -- 172.21.15.3:0/3166648065 shutdown_connections 2022-07-21T01:42:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 -- 172.21.15.3:0/3166648065 wait complete. 2022-07-21T01:42:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 Processor -- start 2022-07-21T01:42:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.215+0000 7fd99e545700 1 -- start start 2022-07-21T01:42:13.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99e545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 0x7fd99814cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99e545700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9980b01a0 0x7fd9980a93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99e545700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9980b2710 0x7fd998159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99e545700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99814d7f0 con 0x7fd9980b01a0 2022-07-21T01:42:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99e545700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9980af080 con 0x7fd9980b2710 2022-07-21T01:42:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99e545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9980aee20 con 0x7fd9980afd60 2022-07-21T01:42:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99cd42700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9980b01a0 0x7fd9980a93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99cd42700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9980b01a0 0x7fd9980a93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50848/0 (socket says 172.21.15.3:50848) 2022-07-21T01:42:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99cd42700 1 -- 172.21.15.3:0/2455360362 learned_addr learned my addr 172.21.15.3:0/2455360362 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99d543700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 0x7fd99814cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99dd44700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9980b2710 0x7fd998159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99cd42700 1 -- 172.21.15.3:0/2455360362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 msgr2=0x7fd99814cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99cd42700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 0x7fd99814cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99cd42700 1 -- 172.21.15.3:0/2455360362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9980b2710 msgr2=0x7fd998159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99cd42700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9980b2710 0x7fd998159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99cd42700 1 -- 172.21.15.3:0/2455360362 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99000b180 con 0x7fd9980b01a0 2022-07-21T01:42:13.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99dd44700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9980b2710 0x7fd998159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:13.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99cd42700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9980b01a0 0x7fd9980a93b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7fd98800c000 tx=0x7fd9880091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:13.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd9967fc700 1 -- 172.21.15.3:0/2455360362 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd98800c850 con 0x7fd9980b01a0 2022-07-21T01:42:13.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd9967fc700 1 -- 172.21.15.3:0/2455360362 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98800c9b0 con 0x7fd9980b01a0 2022-07-21T01:42:13.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd9967fc700 1 -- 172.21.15.3:0/2455360362 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd98800cd10 con 0x7fd9980b01a0 2022-07-21T01:42:13.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd998157080 con 0x7fd9980b01a0 2022-07-21T01:42:13.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.219+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9980af390 con 0x7fd9980b01a0 2022-07-21T01:42:13.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.223+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd998157210 con 0x7fd9980b01a0 2022-07-21T01:42:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.223+0000 7fd9967fc700 1 -- 172.21.15.3:0/2455360362 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd988022070 con 0x7fd9980b01a0 2022-07-21T01:42:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.223+0000 7fd9967fc700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd978053a70 0x7fd978055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.223+0000 7fd9967fc700 1 -- 172.21.15.3:0/2455360362 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd98806c7c0 con 0x7fd9980b01a0 2022-07-21T01:42:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.223+0000 7fd99d543700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd978053a70 0x7fd978055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.223+0000 7fd99d543700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd978053a70 0x7fd978055f30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fd98c00b000 tx=0x7fd98c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.231+0000 7fd9967fc700 1 -- 172.21.15.3:0/2455360362 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd988043520 con 0x7fd9980b01a0 2022-07-21T01:42:13.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.359+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd9980574e0 con 0x7fd9980b01a0 2022-07-21T01:42:13.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.359+0000 7fd9967fc700 1 -- 172.21.15.3:0/2455360362 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v140) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd988046b40 con 0x7fd9980b01a0 2022-07-21T01:42:13.364 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:13.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd978053a70 msgr2=0x7fd978055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd978053a70 0x7fd978055f30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fd98c00b000 tx=0x7fd98c009040).stop 2022-07-21T01:42:13.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9980b01a0 msgr2=0x7fd9980a93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9980b01a0 0x7fd9980a93b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7fd98800c000 tx=0x7fd9880091d0).stop 2022-07-21T01:42:13.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 shutdown_connections 2022-07-21T01:42:13.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd978053a70 0x7fd978055f30 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9980b2710 0x7fd998159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9980b01a0 0x7fd9980a93b0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 --2- 172.21.15.3:0/2455360362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9980afd60 0x7fd99814cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 >> 172.21.15.3:0/2455360362 conn(0x7fd998060310 msgr2=0x7fd998144b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 shutdown_connections 2022-07-21T01:42:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.363+0000 7fd99e545700 1 -- 172.21.15.3:0/2455360362 wait complete. 2022-07-21T01:42:13.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:42:13.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc059af6700 1 Processor -- start 2022-07-21T01:42:13.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc059af6700 1 -- start start 2022-07-21T01:42:13.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc059af6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540acbc0 0x7fc0540b5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc059af6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc0540b15c0 0x7fc054067190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc059af6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0540acfa0 con 0x7fc0540ada00 2022-07-21T01:42:13.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc059af6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0540b52f0 con 0x7fc0540b15c0 2022-07-21T01:42:13.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc059af6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540ae070 con 0x7fc0540acbc0 2022-07-21T01:42:13.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc053fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc0540ada00 0x7fc054143650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59134/0 (socket says 172.21.15.3:59134) 2022-07-21T01:42:13.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc053fff700 1 -- 172.21.15.3:0/3656170435 learned_addr learned my addr 172.21.15.3:0/3656170435 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:13.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc058af4700 1 --2- 172.21.15.3:0/3656170435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540acbc0 0x7fc0540b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.487+0000 7fc0592f5700 1 --2- 172.21.15.3:0/3656170435 >> v2:172.21.15.143:3300/0 conn(0x7fc0540b15c0 0x7fc054067190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc0537fe700 1 -- 172.21.15.3:0/3656170435 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281883311 0 0) 0x7fc0540acfa0 con 0x7fc0540ada00 2022-07-21T01:42:13.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc0537fe700 1 -- 172.21.15.3:0/3656170435 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc034003260 con 0x7fc0540ada00 2022-07-21T01:42:13.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc058af4700 1 -- 172.21.15.3:0/3656170435 >> v2:172.21.15.143:3300/0 conn(0x7fc0540b15c0 msgr2=0x7fc054067190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:13.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc058af4700 1 --2- 172.21.15.3:0/3656170435 >> v2:172.21.15.143:3300/0 conn(0x7fc0540b15c0 0x7fc054067190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc058af4700 1 -- 172.21.15.3:0/3656170435 >> v1:172.21.15.3:6789/0 conn(0x7fc0540ada00 legacy=0x7fc054143650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc0537fe700 1 -- 172.21.15.3:0/3656170435 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4214238689 0 0) 0x7fc034003260 con 0x7fc0540ada00 2022-07-21T01:42:13.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc058af4700 1 -- 172.21.15.3:0/3656170435 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0540ae310 con 0x7fc0540acbc0 2022-07-21T01:42:13.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc0592f5700 1 --2- 172.21.15.3:0/3656170435 >> v2:172.21.15.143:3300/0 conn(0x7fc0540b15c0 0x7fc054067190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:13.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc058af4700 1 --2- 172.21.15.3:0/3656170435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540acbc0 0x7fc0540b5750 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc03c001800 tx=0x7fc0440091d0).ready entity=mon.2 client_cookie=fdd886212d3dc0b4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:13.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc0537fe700 1 -- 172.21.15.3:0/3656170435 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc03c001660 con 0x7fc0540acbc0 2022-07-21T01:42:13.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc0537fe700 1 -- 172.21.15.3:0/3656170435 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04400f510 con 0x7fc0540acbc0 2022-07-21T01:42:13.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc0537fe700 1 -- 172.21.15.3:0/3656170435 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc04400f680 con 0x7fc0540acbc0 2022-07-21T01:42:13.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc059af6700 1 -- 172.21.15.3:0/3656170435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540acbc0 msgr2=0x7fc0540b5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc059af6700 1 --2- 172.21.15.3:0/3656170435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540acbc0 0x7fc0540b5750 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc03c001800 tx=0x7fc0440091d0).stop 2022-07-21T01:42:13.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc059af6700 1 -- 172.21.15.3:0/3656170435 shutdown_connections 2022-07-21T01:42:13.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc059af6700 1 --2- 172.21.15.3:0/3656170435 >> v2:172.21.15.143:3300/0 conn(0x7fc0540b15c0 0x7fc054067190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc059af6700 1 --2- 172.21.15.3:0/3656170435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540acbc0 0x7fc0540b5750 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc059af6700 1 -- 172.21.15.3:0/3656170435 >> 172.21.15.3:0/3656170435 conn(0x7fc0540626f0 msgr2=0x7fc054062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc059af6700 1 -- 172.21.15.3:0/3656170435 shutdown_connections 2022-07-21T01:42:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc059af6700 1 -- 172.21.15.3:0/3656170435 wait complete. 2022-07-21T01:42:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.491+0000 7fc059af6700 1 Processor -- start 2022-07-21T01:42:13.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc059af6700 1 -- start start 2022-07-21T01:42:13.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc059af6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0540acbc0 0x7fc054065e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc059af6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540ada00 0x7fc05414e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc059af6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0540b15c0 0x7fc0540afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc059af6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540665d0 con 0x7fc0540b15c0 2022-07-21T01:42:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc059af6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc054066800 con 0x7fc0540acbc0 2022-07-21T01:42:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc059af6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc054151930 con 0x7fc0540ada00 2022-07-21T01:42:13.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc0592f5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0540b15c0 0x7fc0540afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc0592f5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0540b15c0 0x7fc0540afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50862/0 (socket says 172.21.15.3:50862) 2022-07-21T01:42:13.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc0592f5700 1 -- 172.21.15.3:0/1011176876 learned_addr learned my addr 172.21.15.3:0/1011176876 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:13.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc058af4700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0540acbc0 0x7fc054065e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc053fff700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540ada00 0x7fc05414e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc0592f5700 1 -- 172.21.15.3:0/1011176876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540ada00 msgr2=0x7fc05414e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc0592f5700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540ada00 0x7fc05414e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc0592f5700 1 -- 172.21.15.3:0/1011176876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0540acbc0 msgr2=0x7fc054065e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc0592f5700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0540acbc0 0x7fc054065e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc0592f5700 1 -- 172.21.15.3:0/1011176876 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c00b180 con 0x7fc0540b15c0 2022-07-21T01:42:13.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc0592f5700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0540b15c0 0x7fc0540afc20 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7fc04c00c000 tx=0x7fc04c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:13.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc051ffb700 1 -- 172.21.15.3:0/1011176876 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc04c010780 con 0x7fc0540b15c0 2022-07-21T01:42:13.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc051ffb700 1 -- 172.21.15.3:0/1011176876 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04c0108e0 con 0x7fc0540b15c0 2022-07-21T01:42:13.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc044009470 con 0x7fc0540b15c0 2022-07-21T01:42:13.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc051ffb700 1 -- 172.21.15.3:0/1011176876 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc04c010c40 con 0x7fc0540b15c0 2022-07-21T01:42:13.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.495+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05414e6d0 con 0x7fc0540b15c0 2022-07-21T01:42:13.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.499+0000 7fc051ffb700 1 -- 172.21.15.3:0/1011176876 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc04c01c460 con 0x7fc0540b15c0 2022-07-21T01:42:13.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.499+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc054066a00 con 0x7fc0540b15c0 2022-07-21T01:42:13.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.499+0000 7fc051ffb700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc038053980 0x7fc038055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.499+0000 7fc051ffb700 1 -- 172.21.15.3:0/1011176876 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc04c06c5c0 con 0x7fc0540b15c0 2022-07-21T01:42:13.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.499+0000 7fc058af4700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc038053980 0x7fc038055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.499+0000 7fc058af4700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc038053980 0x7fc038055e40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7fc044019000 tx=0x7fc044016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:13.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.507+0000 7fc051ffb700 1 -- 172.21.15.3:0/1011176876 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc04c043320 con 0x7fc0540b15c0 2022-07-21T01:42:13.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.631+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0540ae5b0 con 0x7fc0540b15c0 2022-07-21T01:42:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.631+0000 7fc051ffb700 1 -- 172.21.15.3:0/1011176876 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc04c046940 con 0x7fc0540b15c0 2022-07-21T01:42:13.637 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc038053980 msgr2=0x7fc038055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc038053980 0x7fc038055e40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7fc044019000 tx=0x7fc044016000).stop 2022-07-21T01:42:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0540b15c0 msgr2=0x7fc0540afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0540b15c0 0x7fc0540afc20 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7fc04c00c000 tx=0x7fc04c006df0).stop 2022-07-21T01:42:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 shutdown_connections 2022-07-21T01:42:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0540b15c0 0x7fc0540afc20 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0540ada00 0x7fc05414e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc038053980 0x7fc038055e40 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 --2- 172.21.15.3:0/1011176876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0540acbc0 0x7fc054065e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 >> 172.21.15.3:0/1011176876 conn(0x7fc0540626f0 msgr2=0x7fc054067090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 shutdown_connections 2022-07-21T01:42:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.635+0000 7fc059af6700 1 -- 172.21.15.3:0/1011176876 wait complete. 2022-07-21T01:42:13.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:42:13.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.755+0000 7f3421f0b700 1 Processor -- start 2022-07-21T01:42:13.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.755+0000 7f3421f0b700 1 -- start start 2022-07-21T01:42:13.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f3421f0b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f341c0af530 0x7f341c0b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f3421f0b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c145310 con 0x7f341c144e00 2022-07-21T01:42:13.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f3421f0b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c0aecc0 con 0x7f341c0b52f0 2022-07-21T01:42:13.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f3421f0b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341c13e8f0 con 0x7f341c0af530 2022-07-21T01:42:13.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f3413fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f341c144e00 0x7f341c141090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59148/0 (socket says 172.21.15.3:59148) 2022-07-21T01:42:13.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f3413fff700 1 -- 172.21.15.3:0/1481681077 learned_addr learned my addr 172.21.15.3:0/1481681077 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:13.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f342170a700 1 --2- 172.21.15.3:0/1481681077 >> v2:172.21.15.143:3300/0 conn(0x7f341c0af530 0x7f341c0b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f34137fe700 1 -- 172.21.15.3:0/1481681077 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413839832 0 0) 0x7f341c145310 con 0x7f341c144e00 2022-07-21T01:42:13.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f34137fe700 1 -- 172.21.15.3:0/1481681077 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3408003260 con 0x7f341c144e00 2022-07-21T01:42:13.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f342170a700 1 -- 172.21.15.3:0/1481681077 >> v1:172.21.15.143:6789/0 conn(0x7f341c0b52f0 legacy=0x7f341c067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f342170a700 1 -- 172.21.15.3:0/1481681077 >> v1:172.21.15.3:6789/0 conn(0x7f341c144e00 legacy=0x7f341c141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f342170a700 1 -- 172.21.15.3:0/1481681077 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c0aef60 con 0x7f341c0af530 2022-07-21T01:42:13.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.759+0000 7f342170a700 1 --2- 172.21.15.3:0/1481681077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0af530 0x7f341c0b56d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f34140032f0 tx=0x7f34140091d0).ready entity=mon.1 client_cookie=8f9c37b273aacff3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:13.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.763+0000 7f34137fe700 1 -- 172.21.15.3:0/1481681077 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f341400b6c0 con 0x7f341c0af530 2022-07-21T01:42:13.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.763+0000 7f34137fe700 1 -- 172.21.15.3:0/1481681077 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341400b820 con 0x7f341c0af530 2022-07-21T01:42:13.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.763+0000 7f34137fe700 1 -- 172.21.15.3:0/1481681077 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f341400b990 con 0x7f341c0af530 2022-07-21T01:42:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.763+0000 7f3421f0b700 1 -- 172.21.15.3:0/1481681077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0af530 msgr2=0x7f341c0b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.763+0000 7f3421f0b700 1 --2- 172.21.15.3:0/1481681077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0af530 0x7f341c0b56d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f34140032f0 tx=0x7f34140091d0).stop 2022-07-21T01:42:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.763+0000 7f3421f0b700 1 -- 172.21.15.3:0/1481681077 shutdown_connections 2022-07-21T01:42:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.763+0000 7f3421f0b700 1 --2- 172.21.15.3:0/1481681077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0af530 0x7f341c0b56d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.763+0000 7f3421f0b700 1 -- 172.21.15.3:0/1481681077 >> 172.21.15.3:0/1481681077 conn(0x7f341c05f840 msgr2=0x7f341c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.763+0000 7f3421f0b700 1 -- 172.21.15.3:0/1481681077 shutdown_connections 2022-07-21T01:42:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 -- 172.21.15.3:0/1481681077 wait complete. 2022-07-21T01:42:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 Processor -- start 2022-07-21T01:42:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 -- start start 2022-07-21T01:42:13.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0af530 0x7f341c0b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f341c0b52f0 0x7f341c063770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f341c144e00 0x7f341c063c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0ae160 con 0x7f341c144e00 2022-07-21T01:42:13.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0a8200 con 0x7f341c0af530 2022-07-21T01:42:13.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0a84a0 con 0x7f341c0b52f0 2022-07-21T01:42:13.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f342170a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f341c144e00 0x7f341c063c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3413fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f341c0b52f0 0x7f341c063770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3413fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f341c0b52f0 0x7f341c063770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:57788/0 (socket says 172.21.15.3:57788) 2022-07-21T01:42:13.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3413fff700 1 -- 172.21.15.3:0/44701205 learned_addr learned my addr 172.21.15.3:0/44701205 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:13.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3420f09700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0af530 0x7f341c0b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3413fff700 1 -- 172.21.15.3:0/44701205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0af530 msgr2=0x7f341c0b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3413fff700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0af530 0x7f341c0b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3413fff700 1 -- 172.21.15.3:0/44701205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f341c144e00 msgr2=0x7f341c063c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3413fff700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f341c144e00 0x7f341c063c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3413fff700 1 -- 172.21.15.3:0/44701205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3414009470 con 0x7f341c0b52f0 2022-07-21T01:42:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3413fff700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f341c0b52f0 0x7f341c063770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f340c001110 tx=0x7f340c007ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3411ffb700 1 -- 172.21.15.3:0/44701205 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f340c0070d0 con 0x7f341c0b52f0 2022-07-21T01:42:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3411ffb700 1 -- 172.21.15.3:0/44701205 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c007230 con 0x7f341c0b52f0 2022-07-21T01:42:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c0ae3c0 con 0x7f341c0b52f0 2022-07-21T01:42:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3411ffb700 1 -- 172.21.15.3:0/44701205 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f340c005610 con 0x7f341c0b52f0 2022-07-21T01:42:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.767+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c0a8740 con 0x7f341c0b52f0 2022-07-21T01:42:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.771+0000 7f3411ffb700 1 -- 172.21.15.3:0/44701205 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f340c010070 con 0x7f341c0b52f0 2022-07-21T01:42:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.771+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c0ae550 con 0x7f341c0b52f0 2022-07-21T01:42:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.771+0000 7f3411ffb700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f33fc0539e0 0x7f33fc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.771+0000 7f3411ffb700 1 -- 172.21.15.3:0/44701205 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f340c06b6b0 con 0x7f341c0b52f0 2022-07-21T01:42:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.771+0000 7f3420f09700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f33fc0539e0 0x7f33fc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.771+0000 7f3420f09700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f33fc0539e0 0x7f33fc055ea0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f341800d000 tx=0x7f3418007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.775+0000 7f3411ffb700 1 -- 172.21.15.3:0/44701205 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f340c0423f0 con 0x7f341c0b52f0 2022-07-21T01:42:13.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.903+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f341c0574e0 con 0x7f341c0b52f0 2022-07-21T01:42:13.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.903+0000 7f3411ffb700 1 -- 172.21.15.3:0/44701205 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+10 (secure 0 0 0) 0x7f340c045a10 con 0x7f341c0b52f0 2022-07-21T01:42:13.908 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f33fc0539e0 msgr2=0x7f33fc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f33fc0539e0 0x7f33fc055ea0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f341800d000 tx=0x7f3418007b90).stop 2022-07-21T01:42:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f341c0b52f0 msgr2=0x7f341c063770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f341c0b52f0 0x7f341c063770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f340c001110 tx=0x7f340c007ad0).stop 2022-07-21T01:42:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 shutdown_connections 2022-07-21T01:42:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f341c144e00 0x7f341c063c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f341c0b52f0 0x7f341c063770 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f33fc0539e0 0x7f33fc055ea0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 --2- 172.21.15.3:0/44701205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f341c0af530 0x7f341c0b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:13.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 >> 172.21.15.3:0/44701205 conn(0x7f341c05f840 msgr2=0x7f341c0b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:13.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 shutdown_connections 2022-07-21T01:42:13.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:13.907+0000 7f3421f0b700 1 -- 172.21.15.3:0/44701205 wait complete. 2022-07-21T01:42:13.926 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:42:13.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:14.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.027+0000 7fb53e479700 1 Processor -- start 2022-07-21T01:42:14.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.027+0000 7fb53e479700 1 -- start start 2022-07-21T01:42:14.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb538066780 0x7fb538066b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:14.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5380b1bc0 0x7fb53805ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:14.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53805f8e0 con 0x7fb538140e40 2022-07-21T01:42:14.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5380b1630 con 0x7fb538066780 2022-07-21T01:42:14.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53805fb10 con 0x7fb5380b1bc0 2022-07-21T01:42:14.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53cc76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5380b1bc0 0x7fb53805ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:14.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53d477700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb538066780 0x7fb538066b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:14.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53dc78700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb538140e40 0x7fb53813e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:33046/0 (socket says 172.21.15.3:33046) 2022-07-21T01:42:14.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53cc76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5380b1bc0 0x7fb53805ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:57798/0 (socket says 172.21.15.3:57798) 2022-07-21T01:42:14.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53d477700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb538066780 0x7fb538066b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:56328/0 (socket says 172.21.15.3:56328) 2022-07-21T01:42:14.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53dc78700 1 -- 172.21.15.3:0/2550399171 learned_addr learned my addr 172.21.15.3:0/2550399171 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:14.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53d477700 1 -- 172.21.15.3:0/2550399171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5380b1bc0 msgr2=0x7fb53805ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:14.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53d477700 1 --2- 172.21.15.3:0/2550399171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5380b1bc0 0x7fb53805ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:14.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53d477700 1 -- 172.21.15.3:0/2550399171 >> v1:172.21.15.143:6789/0 conn(0x7fb538140e40 legacy=0x7fb53813e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:14.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53d477700 1 -- 172.21.15.3:0/2550399171 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380604a0 con 0x7fb538066780 2022-07-21T01:42:14.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53d477700 1 --2- 172.21.15.3:0/2550399171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538066780 0x7fb538066b60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fb534002f70 tx=0x7fb534006df0).ready entity=mon.1 client_cookie=cdde6cded404f91f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:14.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb527fff700 1 -- 172.21.15.3:0/2550399171 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53400d570 con 0x7fb538066780 2022-07-21T01:42:14.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb527fff700 1 -- 172.21.15.3:0/2550399171 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb534000e90 con 0x7fb538066780 2022-07-21T01:42:14.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb527fff700 1 -- 172.21.15.3:0/2550399171 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53400d8c0 con 0x7fb538066780 2022-07-21T01:42:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 -- 172.21.15.3:0/2550399171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538066780 msgr2=0x7fb538066b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 --2- 172.21.15.3:0/2550399171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538066780 0x7fb538066b60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fb534002f70 tx=0x7fb534006df0).stop 2022-07-21T01:42:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 -- 172.21.15.3:0/2550399171 shutdown_connections 2022-07-21T01:42:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 --2- 172.21.15.3:0/2550399171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5380b1bc0 0x7fb53805ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 --2- 172.21.15.3:0/2550399171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538066780 0x7fb538066b60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 -- 172.21.15.3:0/2550399171 >> 172.21.15.3:0/2550399171 conn(0x7fb538063870 msgr2=0x7fb538063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 -- 172.21.15.3:0/2550399171 shutdown_connections 2022-07-21T01:42:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.031+0000 7fb53e479700 1 -- 172.21.15.3:0/2550399171 wait complete. 2022-07-21T01:42:14.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 Processor -- start 2022-07-21T01:42:14.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 -- start start 2022-07-21T01:42:14.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538066780 0x7fb5380af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b1bc0 0x7fb538152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538140e40 0x7fb538152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380af780 con 0x7fb5380b1bc0 2022-07-21T01:42:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380aeae0 con 0x7fb538140e40 2022-07-21T01:42:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53813fdb0 con 0x7fb538066780 2022-07-21T01:42:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53cc76700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b1bc0 0x7fb538152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53d477700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538066780 0x7fb5380af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:14.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53dc78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538140e40 0x7fb538152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:14.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53d477700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538066780 0x7fb5380af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:57802/0 (socket says 172.21.15.3:57802) 2022-07-21T01:42:14.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53d477700 1 -- 172.21.15.3:0/86250219 learned_addr learned my addr 172.21.15.3:0/86250219 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53cc76700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b1bc0 0x7fb538152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50888/0 (socket says 172.21.15.3:50888) 2022-07-21T01:42:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53cc76700 1 -- 172.21.15.3:0/86250219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538066780 msgr2=0x7fb5380af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53cc76700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538066780 0x7fb5380af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53cc76700 1 -- 172.21.15.3:0/86250219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538140e40 msgr2=0x7fb538152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53cc76700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538140e40 0x7fb538152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53cc76700 1 -- 172.21.15.3:0/86250219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53400b180 con 0x7fb5380b1bc0 2022-07-21T01:42:14.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53cc76700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b1bc0 0x7fb538152230 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7fb52800c000 tx=0x7fb528006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:14.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb5267fc700 1 -- 172.21.15.3:0/86250219 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb528011750 con 0x7fb5380b1bc0 2022-07-21T01:42:14.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb5267fc700 1 -- 172.21.15.3:0/86250219 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5280118b0 con 0x7fb5380b1bc0 2022-07-21T01:42:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb538140050 con 0x7fb5380b1bc0 2022-07-21T01:42:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb5267fc700 1 -- 172.21.15.3:0/86250219 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb528011c10 con 0x7fb5380b1bc0 2022-07-21T01:42:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.035+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5380662f0 con 0x7fb5380b1bc0 2022-07-21T01:42:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.039+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5381401e0 con 0x7fb5380b1bc0 2022-07-21T01:42:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.039+0000 7fb5267fc700 1 -- 172.21.15.3:0/86250219 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb528011d70 con 0x7fb5380b1bc0 2022-07-21T01:42:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.039+0000 7fb5267fc700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053a70 0x7fb518055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.039+0000 7fb5267fc700 1 -- 172.21.15.3:0/86250219 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb52806c380 con 0x7fb5380b1bc0 2022-07-21T01:42:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.039+0000 7fb53d477700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053a70 0x7fb518055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.039+0000 7fb53d477700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053a70 0x7fb518055f30 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fb53401a000 tx=0x7fb534014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.047+0000 7fb5267fc700 1 -- 172.21.15.3:0/86250219 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb528043140 con 0x7fb5380b1bc0 2022-07-21T01:42:14.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.215+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5380b18d0 con 0x7fb5380b1bc0 2022-07-21T01:42:14.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.215+0000 7fb5267fc700 1 -- 172.21.15.3:0/86250219 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2483 (secure 0 0 0) 0x7fb528046760 con 0x7fb5380b1bc0 2022-07-21T01:42:14.220 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:14.220 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":71,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/567682837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":567682837},{"type":"v1","addr":"172.21.15.143:6833","nonce":567682837}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/140242740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140242740},{"type":"v1","addr":"172.21.15.3:6835","nonce":140242740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:07.063040+0000","modified":"2022-07-21T01:42:08.065747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8049":{"gid":8049,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/506442168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":506442168},{"type":"v1","addr":"172.21.15.3:6837","nonce":506442168}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-21T01:42:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053a70 msgr2=0x7fb518055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053a70 0x7fb518055f30 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fb53401a000 tx=0x7fb534014000).stop 2022-07-21T01:42:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b1bc0 msgr2=0x7fb538152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b1bc0 0x7fb538152230 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7fb52800c000 tx=0x7fb528006df0).stop 2022-07-21T01:42:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 shutdown_connections 2022-07-21T01:42:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb518053a70 0x7fb518055f30 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb538140e40 0x7fb538152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5380b1bc0 0x7fb538152230 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 --2- 172.21.15.3:0/86250219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb538066780 0x7fb5380af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 >> 172.21.15.3:0/86250219 conn(0x7fb538063870 msgr2=0x7fb53813ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 shutdown_connections 2022-07-21T01:42:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:14.219+0000 7fb53e479700 1 -- 172.21.15.3:0/86250219 wait complete. 2022-07-21T01:42:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 71 2022-07-21T01:42:14.536 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:07.063040+0000', 'modified': '2022-07-21T01:42:08.065747+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8049}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8049': {'gid': 8049, 'name': 'a', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/506442168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 506442168}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 506442168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:42:14.537 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:42:14.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:42:14.661 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8049,"want_state":"up:active","state":"up:active","rank_uptime":7.5861259719999996,"mdsmap_epoch":71,"osdmap_epoch":140,"osdmap_epoch_barrier":139,"uptime":12.315745810999999} 2022-07-21T01:42:14.662 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8049, "mdsmap_epoch": 71, "osdmap_epoch": 140, "osdmap_epoch_barrier": 139, "rank_uptime": 7.586125972, "state": "up:active", "uptime": 12.315745811, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:42:14.662 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:42:14.663 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:42:14.663 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:42:14.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:14.664 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:14.673 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:42:14.734 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:14.735 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:14.748 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:14.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:14.750 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:42:14.756 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:42:14.757 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:14.801 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:42:14.820 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:14.821 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:14.832 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:14.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:14.835 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:42:14.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:42:14.876+0000 7f221ccaa080 -1 init, newargv = 0x560a6190a900 newargc=9ceph-fuse[ 2022-07-21T01:42:14.877 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:22880]: starting ceph client 2022-07-21T01:42:14.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22880]: starting fuse 2022-07-21T01:42:15.942 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:42:15.961 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:15.962 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:16.019 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:16.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:16.021 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:42:16.026 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:42:16.027 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:42:16.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:16.027 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:42:16.028 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:42:16.028 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:42:16.029 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:42:16.029 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:42:16.030 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:16.030 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:42:16.031 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:42:16.031 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:42:16.031 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:16.032 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:42:16.033 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:42:16.034 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:42:16.035 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:42:16.035 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:16.036 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:42:16.037 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:42:16.037 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:42:16.038 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:42:16.039 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:42:16.039 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:16.040 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:42:16.040 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:42:16.083 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.22880.asok 2022-07-21T01:42:16.088 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22880.asok 2022-07-21T01:42:16.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:16.089 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22880.asok status 2022-07-21T01:42:16.226 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:42:16.227 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:42:16.227 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:42:16.228 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:42:16.228 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:42:16.229 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:42:16.229 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:42:16.230 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "22880", 2022-07-21T01:42:16.230 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:42:16.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:16.231 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:42:16.231 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:42:16.232 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 8152, 2022-07-21T01:42:16.233 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:42:16.234 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:42:16.234 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:42:16.235 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 8152 2022-07-21T01:42:16.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:16.236 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:42:16.236 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:42:16.237 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:42:16.237 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3537529186 2022-07-21T01:42:16.237 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:42:16.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:16.239 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:42:16.240 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:42:16.240 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:42:16.241 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3537529186 2022-07-21T01:42:16.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:16.241 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.8152 172.21.15.174:0/3537529186", 2022-07-21T01:42:16.242 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/3537529186", 2022-07-21T01:42:16.242 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:42:16.243 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 71, 2022-07-21T01:42:16.243 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 140, 2022-07-21T01:42:16.244 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:42:16.244 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:42:16.244 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:42:16.245 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:42:16.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:16.248 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:16.248 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:16.256 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:42:16.258 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:16.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:16.259 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:16.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:16.318 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:16.319 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:16.327 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:42:16.330 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:16.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:16.333 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:16.334 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:16.341 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:42:16.343 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:16.344 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:42:16.344 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:42:16.345 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:16.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22880]: fuse finished with error 0 and tester_r 0 2022-07-21T01:42:16.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:16.401 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:16.401 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:16.410 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:42:16.410 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:42:16.410 INFO:teuthology.orchestra.run:waiting for 900 2022-07-21T01:42:16.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:42:16.475 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:16.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:42:16.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:42:16.490 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:42:16.492 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:42:16.519 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2022-07-21T01:42:16.520 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2022-07-21T01:42:16.520 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:42:16.520 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2022-07-21T01:42:16.521 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.521 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2022-07-21T01:42:16.521 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.521 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2022-07-21T01:42:16.522 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.522 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2022-07-21T01:42:16.522 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2022-07-21T01:42:16.522 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.523 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2022-07-21T01:42:16.523 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2022-07-21T01:42:16.523 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.523 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:42:16.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2022-07-21T01:42:16.524 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.524 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:42:16.524 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:42:16.525 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.525 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2022-07-21T01:42:16.526 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.526 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2022-07-21T01:42:16.526 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2022-07-21T01:42:16.527 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:42:16.527 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:42:16.527 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2022-07-21T01:42:16.527 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:42:16.528 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2022-07-21T01:42:16.528 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:42:16.528 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:42:16.529 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.529 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:42:16.529 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.529 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2022-07-21T01:42:16.529 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.530 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2022-07-21T01:42:16.530 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.530 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2022-07-21T01:42:16.531 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2022-07-21T01:42:16.531 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2022-07-21T01:42:16.531 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:42:16.531 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:42:16.532 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.532 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2022-07-21T01:42:16.532 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2022-07-21T01:42:16.532 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.532 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2022-07-21T01:42:16.533 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2022-07-21T01:42:16.533 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.534 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2022-07-21T01:42:16.534 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2022-07-21T01:42:16.534 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.535 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2022-07-21T01:42:16.535 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2022-07-21T01:42:16.535 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2022-07-21T01:42:16.536 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.536 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2022-07-21T01:42:16.536 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2022-07-21T01:42:16.536 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.537 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:42:16.537 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.537 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2022-07-21T01:42:16.537 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.538 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2022-07-21T01:42:16.538 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.538 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:42:16.539 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2022-07-21T01:42:16.539 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.539 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2022-07-21T01:42:16.539 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2022-07-21T01:42:16.540 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2022-07-21T01:42:16.540 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore compression mode = aggressive 2022-07-21T01:42:16.540 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2022-07-21T01:42:16.540 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 20 2022-07-21T01:42:16.541 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 20 2022-07-21T01:42:16.541 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:16.541 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2022-07-21T01:42:16.541 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 10 2022-07-21T01:42:16.541 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:42:16.542 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2022-07-21T01:42:16.542 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:42:16.542 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:42:16.542 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2022-07-21T01:42:16.543 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2022-07-21T01:42:16.543 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.543 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2022-07-21T01:42:16.543 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:16.544 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2022-07-21T01:42:16.544 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2022-07-21T01:42:16.544 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2022-07-21T01:42:16.545 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:16.545 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:16.545 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2022-07-21T01:42:16.545 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.545 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2022-07-21T01:42:16.546 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:16.546 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2022-07-21T01:42:16.546 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2022-07-21T01:42:16.547 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2022-07-21T01:42:16.547 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2022-07-21T01:42:16.547 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2022-07-21T01:42:16.547 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:16.548 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2022-07-21T01:42:16.548 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2022-07-21T01:42:16.548 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:16.548 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.548 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:42:16.549 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:42:16.549 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:42:16.549 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.549 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:42:16.550 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:42:16.550 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:42:16.550 INFO:teuthology.orchestra.run.smithi143.stdout: mon op complaint time = 120 2022-07-21T01:42:16.550 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:16.551 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2022-07-21T01:42:16.551 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2022-07-21T01:42:16.551 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2022-07-21T01:42:16.551 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2022-07-21T01:42:16.552 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:42:16.552 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:42:16.552 INFO:teuthology.orchestra.run.smithi143.stdout: client mount timeout = 600 2022-07-21T01:42:16.552 INFO:teuthology.orchestra.run.smithi143.stdout: debug client = 20 2022-07-21T01:42:16.553 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:16.553 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 15m 2022-07-21T01:42:16.553 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 15m 2022-07-21T01:42:16.553 INFO:teuthology.orchestra.run.smithi143.stdout: client debug getattr caps = true 2022-07-21T01:42:16.554 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2022-07-21T01:42:16.554 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2022-07-21T01:42:16.554 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2022-07-21T01:42:16.554 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.a] 2022-07-21T01:42:16.555 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.c] 2022-07-21T01:42:16.555 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.b] 2022-07-21T01:42:16.555 INFO:teuthology.orchestra.run.smithi143.stdout:[mds] 2022-07-21T01:42:16.555 INFO:teuthology.orchestra.run.smithi143.stdout: debug mds = 20 2022-07-21T01:42:16.555 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:16.556 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal frag = True 2022-07-21T01:42:16.556 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal fragment size max = 10000 2022-07-21T01:42:16.556 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal merge size = 5 2022-07-21T01:42:16.556 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal split bits = 3 2022-07-21T01:42:16.557 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal split size = 100 2022-07-21T01:42:16.557 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug frag = True 2022-07-21T01:42:16.557 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug scatterstat = True 2022-07-21T01:42:16.557 INFO:teuthology.orchestra.run.smithi143.stdout: mds op complaint time = 180 2022-07-21T01:42:16.557 INFO:teuthology.orchestra.run.smithi143.stdout: mds verify scatter = True 2022-07-21T01:42:16.558 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2022-07-21T01:42:16.558 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 15m 2022-07-21T01:42:16.558 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 15m 2022-07-21T01:42:16.562 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2022-07-21T01:42:16.562 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2022-07-21T01:42:16.562 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:42:16.563 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = cephx 2022-07-21T01:42:16.563 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.563 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2022-07-21T01:42:16.563 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.564 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2022-07-21T01:42:16.564 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.564 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2022-07-21T01:42:16.564 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2022-07-21T01:42:16.565 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.565 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2022-07-21T01:42:16.565 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2022-07-21T01:42:16.566 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.566 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:42:16.566 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2022-07-21T01:42:16.566 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.566 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:42:16.567 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:42:16.567 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.567 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2022-07-21T01:42:16.567 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.568 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2022-07-21T01:42:16.568 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2022-07-21T01:42:16.568 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:42:16.568 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:42:16.569 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2022-07-21T01:42:16.569 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:42:16.569 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2022-07-21T01:42:16.569 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:42:16.569 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:42:16.570 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.570 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:42:16.570 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.571 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2022-07-21T01:42:16.571 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.571 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2022-07-21T01:42:16.571 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.571 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2022-07-21T01:42:16.572 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2022-07-21T01:42:16.572 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2022-07-21T01:42:16.572 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:42:16.572 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:42:16.573 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.573 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2022-07-21T01:42:16.573 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2022-07-21T01:42:16.573 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.573 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2022-07-21T01:42:16.574 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 600 2022-07-21T01:42:16.574 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.574 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2022-07-21T01:42:16.574 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2022-07-21T01:42:16.575 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.575 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2022-07-21T01:42:16.576 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2022-07-21T01:42:16.576 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2022-07-21T01:42:16.576 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.576 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2022-07-21T01:42:16.576 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2022-07-21T01:42:16.577 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.577 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:42:16.577 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.577 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2022-07-21T01:42:16.578 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.578 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2022-07-21T01:42:16.578 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.578 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:42:16.579 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2022-07-21T01:42:16.579 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.579 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2022-07-21T01:42:16.579 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2022-07-21T01:42:16.580 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore block size = 96636764160 2022-07-21T01:42:16.580 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore compression mode = aggressive 2022-07-21T01:42:16.580 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore fsck on mount = True 2022-07-21T01:42:16.580 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluefs = 20 2022-07-21T01:42:16.581 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluestore = 20 2022-07-21T01:42:16.581 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:16.581 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 20 2022-07-21T01:42:16.581 INFO:teuthology.orchestra.run.smithi174.stdout: debug rocksdb = 10 2022-07-21T01:42:16.582 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:42:16.582 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd full ratio = 0.9 2022-07-21T01:42:16.582 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:42:16.582 INFO:teuthology.orchestra.run.smithi174.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:42:16.583 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = bluestore 2022-07-21T01:42:16.583 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2022-07-21T01:42:16.583 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.583 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2022-07-21T01:42:16.584 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:16.584 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2022-07-21T01:42:16.584 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2022-07-21T01:42:16.584 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2022-07-21T01:42:16.585 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:16.585 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:16.585 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2022-07-21T01:42:16.586 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.586 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2022-07-21T01:42:16.586 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:16.586 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2022-07-21T01:42:16.587 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 20 2022-07-21T01:42:16.587 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2022-07-21T01:42:16.587 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2022-07-21T01:42:16.587 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2022-07-21T01:42:16.588 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:16.588 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2022-07-21T01:42:16.588 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2022-07-21T01:42:16.588 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:16.589 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.589 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:42:16.589 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:42:16.589 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:42:16.589 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.590 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:42:16.590 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:42:16.590 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:42:16.591 INFO:teuthology.orchestra.run.smithi174.stdout: mon op complaint time = 120 2022-07-21T01:42:16.591 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:16.591 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2022-07-21T01:42:16.592 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2022-07-21T01:42:16.592 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2022-07-21T01:42:16.592 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2022-07-21T01:42:16.592 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:42:16.592 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:42:16.593 INFO:teuthology.orchestra.run.smithi174.stdout: client mount timeout = 600 2022-07-21T01:42:16.593 INFO:teuthology.orchestra.run.smithi174.stdout: debug client = 20 2022-07-21T01:42:16.593 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:16.593 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 15m 2022-07-21T01:42:16.594 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 15m 2022-07-21T01:42:16.594 INFO:teuthology.orchestra.run.smithi174.stdout: client debug getattr caps = true 2022-07-21T01:42:16.594 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2022-07-21T01:42:16.594 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.b] 2022-07-21T01:42:16.595 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.c] 2022-07-21T01:42:16.595 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.a] 2022-07-21T01:42:16.595 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.c] 2022-07-21T01:42:16.595 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.b] 2022-07-21T01:42:16.595 INFO:teuthology.orchestra.run.smithi174.stdout:[mds] 2022-07-21T01:42:16.596 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds = 20 2022-07-21T01:42:16.596 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:16.596 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal frag = True 2022-07-21T01:42:16.596 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal fragment size max = 10000 2022-07-21T01:42:16.597 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal merge size = 5 2022-07-21T01:42:16.597 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split bits = 3 2022-07-21T01:42:16.597 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split size = 100 2022-07-21T01:42:16.597 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug frag = True 2022-07-21T01:42:16.598 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug scatterstat = True 2022-07-21T01:42:16.598 INFO:teuthology.orchestra.run.smithi174.stdout: mds op complaint time = 180 2022-07-21T01:42:16.598 INFO:teuthology.orchestra.run.smithi174.stdout: mds verify scatter = True 2022-07-21T01:42:16.599 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2022-07-21T01:42:16.599 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 15m 2022-07-21T01:42:16.599 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 15m 2022-07-21T01:42:16.599 INFO:teuthology.orchestra.run.smithi171.stdout:[global] 2022-07-21T01:42:16.600 INFO:teuthology.orchestra.run.smithi171.stdout: chdir = "" 2022-07-21T01:42:16.600 INFO:teuthology.orchestra.run.smithi171.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:42:16.600 INFO:teuthology.orchestra.run.smithi171.stdout: auth supported = cephx 2022-07-21T01:42:16.600 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.601 INFO:teuthology.orchestra.run.smithi171.stdout: filestore xattr use omap = true 2022-07-21T01:42:16.601 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.601 INFO:teuthology.orchestra.run.smithi171.stdout: mon clock drift allowed = 1.000 2022-07-21T01:42:16.601 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.602 INFO:teuthology.orchestra.run.smithi171.stdout: osd crush chooseleaf type = 0 2022-07-21T01:42:16.602 INFO:teuthology.orchestra.run.smithi171.stdout: auth debug = true 2022-07-21T01:42:16.602 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.602 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on old message = true 2022-07-21T01:42:16.603 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on bug = true 2022-07-21T01:42:16.603 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.603 INFO:teuthology.orchestra.run.smithi171.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:42:16.603 INFO:teuthology.orchestra.run.smithi171.stdout: mon pg warn max object skew = 0 2022-07-21T01:42:16.603 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.604 INFO:teuthology.orchestra.run.smithi171.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:42:16.604 INFO:teuthology.orchestra.run.smithi171.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:42:16.604 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.604 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default size = 2 2022-07-21T01:42:16.605 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.605 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow primary affinity = true 2022-07-21T01:42:16.605 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow pg remap = true 2022-07-21T01:42:16.605 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:42:16.606 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:42:16.606 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on no sortbitwise = false 2022-07-21T01:42:16.606 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:42:16.606 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on too few osds = false 2022-07-21T01:42:16.607 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:42:16.607 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:42:16.607 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.607 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:42:16.608 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.608 INFO:teuthology.orchestra.run.smithi171.stdout: osd default data pool replay window = 5 2022-07-21T01:42:16.608 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.608 INFO:teuthology.orchestra.run.smithi171.stdout: mon allow pool delete = true 2022-07-21T01:42:16.608 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.609 INFO:teuthology.orchestra.run.smithi171.stdout: mon cluster log file level = debug 2022-07-21T01:42:16.609 INFO:teuthology.orchestra.run.smithi171.stdout: debug asserts on shutdown = true 2022-07-21T01:42:16.609 INFO:teuthology.orchestra.run.smithi171.stdout: mon health detail to clog = false 2022-07-21T01:42:16.609 INFO:teuthology.orchestra.run.smithi171.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:42:16.610 INFO:teuthology.orchestra.run.smithi171.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:42:16.610 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.610 INFO:teuthology.orchestra.run.smithi171.stdout:[osd] 2022-07-21T01:42:16.610 INFO:teuthology.orchestra.run.smithi171.stdout: osd journal size = 100 2022-07-21T01:42:16.611 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.611 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub load threshold = 5.0 2022-07-21T01:42:16.611 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub max interval = 600 2022-07-21T01:42:16.611 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.611 INFO:teuthology.orchestra.run.smithi171.stdout: osd recover clone overlap = true 2022-07-21T01:42:16.612 INFO:teuthology.orchestra.run.smithi171.stdout: osd recovery max chunk = 1048576 2022-07-21T01:42:16.612 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.612 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug shutdown = true 2022-07-21T01:42:16.612 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug op order = true 2022-07-21T01:42:16.613 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify stray on activate = true 2022-07-21T01:42:16.613 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.613 INFO:teuthology.orchestra.run.smithi171.stdout: osd open classes on start = true 2022-07-21T01:42:16.613 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug pg log writeout = true 2022-07-21T01:42:16.613 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.614 INFO:teuthology.orchestra.run.smithi171.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:42:16.614 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.614 INFO:teuthology.orchestra.run.smithi171.stdout: osd map max advance = 10 2022-07-21T01:42:16.614 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.615 INFO:teuthology.orchestra.run.smithi171.stdout: journal zero on create = true 2022-07-21T01:42:16.615 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.615 INFO:teuthology.orchestra.run.smithi171.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:42:16.615 INFO:teuthology.orchestra.run.smithi171.stdout: filestore apply finisher threads = 3 2022-07-21T01:42:16.616 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.616 INFO:teuthology.orchestra.run.smithi171.stdout: bdev debug aio = true 2022-07-21T01:42:16.616 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug misdirected ops = true 2022-07-21T01:42:16.616 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore block size = 96636764160 2022-07-21T01:42:16.616 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore compression mode = aggressive 2022-07-21T01:42:16.617 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore fsck on mount = True 2022-07-21T01:42:16.617 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluefs = 20 2022-07-21T01:42:16.618 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluestore = 20 2022-07-21T01:42:16.618 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:16.619 INFO:teuthology.orchestra.run.smithi171.stdout: debug osd = 20 2022-07-21T01:42:16.619 INFO:teuthology.orchestra.run.smithi171.stdout: debug rocksdb = 10 2022-07-21T01:42:16.620 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:42:16.620 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd full ratio = 0.9 2022-07-21T01:42:16.621 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:42:16.621 INFO:teuthology.orchestra.run.smithi171.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:42:16.621 INFO:teuthology.orchestra.run.smithi171.stdout: osd objectstore = bluestore 2022-07-21T01:42:16.622 INFO:teuthology.orchestra.run.smithi171.stdout: osd op complaint time = 180 2022-07-21T01:42:16.622 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.623 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr] 2022-07-21T01:42:16.623 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:16.624 INFO:teuthology.orchestra.run.smithi171.stdout: debug mgr = 20 2022-07-21T01:42:16.624 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2022-07-21T01:42:16.625 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2022-07-21T01:42:16.625 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:16.626 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:16.626 INFO:teuthology.orchestra.run.smithi171.stdout: mgr/telemetry/nag = false 2022-07-21T01:42:16.626 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.627 INFO:teuthology.orchestra.run.smithi171.stdout:[mon] 2022-07-21T01:42:16.627 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:16.627 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2022-07-21T01:42:16.627 INFO:teuthology.orchestra.run.smithi171.stdout: debug paxos = 20 2022-07-21T01:42:16.628 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2022-07-21T01:42:16.628 INFO:teuthology.orchestra.run.smithi171.stdout: mon data avail warn = 5 2022-07-21T01:42:16.628 INFO:teuthology.orchestra.run.smithi171.stdout: mon mgr mkfs grace = 240 2022-07-21T01:42:16.628 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:16.629 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd reporter subtree level = osd 2022-07-21T01:42:16.629 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd prime pg temp = true 2022-07-21T01:42:16.629 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:16.629 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.630 INFO:teuthology.orchestra.run.smithi171.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:42:16.630 INFO:teuthology.orchestra.run.smithi171.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:42:16.630 INFO:teuthology.orchestra.run.smithi171.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:42:16.630 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.630 INFO:teuthology.orchestra.run.smithi171.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:42:16.631 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:42:16.631 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:42:16.631 INFO:teuthology.orchestra.run.smithi171.stdout: mon op complaint time = 120 2022-07-21T01:42:16.631 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:16.632 INFO:teuthology.orchestra.run.smithi171.stdout:[client] 2022-07-21T01:42:16.632 INFO:teuthology.orchestra.run.smithi171.stdout: rgw cache enabled = true 2022-07-21T01:42:16.632 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable ops log = true 2022-07-21T01:42:16.633 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable usage log = true 2022-07-21T01:42:16.633 INFO:teuthology.orchestra.run.smithi171.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:42:16.633 INFO:teuthology.orchestra.run.smithi171.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:42:16.634 INFO:teuthology.orchestra.run.smithi171.stdout: client mount timeout = 600 2022-07-21T01:42:16.634 INFO:teuthology.orchestra.run.smithi171.stdout: debug client = 20 2022-07-21T01:42:16.635 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:16.635 INFO:teuthology.orchestra.run.smithi171.stdout: rados mon op timeout = 15m 2022-07-21T01:42:16.635 INFO:teuthology.orchestra.run.smithi171.stdout: rados osd op timeout = 15m 2022-07-21T01:42:16.636 INFO:teuthology.orchestra.run.smithi171.stdout: client debug getattr caps = true 2022-07-21T01:42:16.636 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.a] 2022-07-21T01:42:16.637 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.b] 2022-07-21T01:42:16.637 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.c] 2022-07-21T01:42:16.638 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.a] 2022-07-21T01:42:16.638 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.c] 2022-07-21T01:42:16.638 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.b] 2022-07-21T01:42:16.639 INFO:teuthology.orchestra.run.smithi171.stdout:[mds] 2022-07-21T01:42:16.639 INFO:teuthology.orchestra.run.smithi171.stdout: debug mds = 20 2022-07-21T01:42:16.639 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:16.640 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal frag = True 2022-07-21T01:42:16.640 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal fragment size max = 10000 2022-07-21T01:42:16.641 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal merge size = 5 2022-07-21T01:42:16.641 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal split bits = 3 2022-07-21T01:42:16.641 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal split size = 100 2022-07-21T01:42:16.642 INFO:teuthology.orchestra.run.smithi171.stdout: mds debug frag = True 2022-07-21T01:42:16.642 INFO:teuthology.orchestra.run.smithi171.stdout: mds debug scatterstat = True 2022-07-21T01:42:16.643 INFO:teuthology.orchestra.run.smithi171.stdout: mds op complaint time = 180 2022-07-21T01:42:16.643 INFO:teuthology.orchestra.run.smithi171.stdout: mds verify scatter = True 2022-07-21T01:42:16.643 INFO:teuthology.orchestra.run.smithi171.stdout: osd op complaint time = 180 2022-07-21T01:42:16.644 INFO:teuthology.orchestra.run.smithi171.stdout: rados mon op timeout = 15m 2022-07-21T01:42:16.644 INFO:teuthology.orchestra.run.smithi171.stdout: rados osd op timeout = 15m 2022-07-21T01:42:16.645 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2022-07-21T01:42:16.645 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2022-07-21T01:42:16.646 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:42:16.646 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2022-07-21T01:42:16.646 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.647 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2022-07-21T01:42:16.647 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.648 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2022-07-21T01:42:16.648 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.648 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2022-07-21T01:42:16.649 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2022-07-21T01:42:16.649 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.649 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2022-07-21T01:42:16.650 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2022-07-21T01:42:16.650 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.650 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:42:16.651 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2022-07-21T01:42:16.651 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.651 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:42:16.651 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:42:16.652 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.652 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2022-07-21T01:42:16.652 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.652 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2022-07-21T01:42:16.653 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2022-07-21T01:42:16.653 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:42:16.653 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:42:16.653 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2022-07-21T01:42:16.654 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:42:16.654 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2022-07-21T01:42:16.654 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:42:16.654 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:42:16.655 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.655 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:42:16.655 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.655 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2022-07-21T01:42:16.656 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.656 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2022-07-21T01:42:16.656 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.656 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2022-07-21T01:42:16.657 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2022-07-21T01:42:16.657 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2022-07-21T01:42:16.657 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:42:16.657 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:42:16.658 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.658 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2022-07-21T01:42:16.658 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2022-07-21T01:42:16.658 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.659 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2022-07-21T01:42:16.659 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2022-07-21T01:42:16.659 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.659 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2022-07-21T01:42:16.660 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2022-07-21T01:42:16.660 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.660 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2022-07-21T01:42:16.660 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2022-07-21T01:42:16.661 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2022-07-21T01:42:16.661 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.661 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2022-07-21T01:42:16.661 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2022-07-21T01:42:16.661 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.662 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:42:16.662 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.662 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2022-07-21T01:42:16.662 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.663 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2022-07-21T01:42:16.663 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.663 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:42:16.663 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2022-07-21T01:42:16.664 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.664 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2022-07-21T01:42:16.664 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2022-07-21T01:42:16.664 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2022-07-21T01:42:16.665 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression mode = aggressive 2022-07-21T01:42:16.665 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2022-07-21T01:42:16.665 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 20 2022-07-21T01:42:16.665 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 20 2022-07-21T01:42:16.666 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:16.666 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2022-07-21T01:42:16.666 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 10 2022-07-21T01:42:16.666 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:42:16.666 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2022-07-21T01:42:16.667 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:42:16.667 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:42:16.667 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2022-07-21T01:42:16.667 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2022-07-21T01:42:16.668 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.668 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2022-07-21T01:42:16.668 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:16.668 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2022-07-21T01:42:16.668 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2022-07-21T01:42:16.669 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2022-07-21T01:42:16.669 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:16.669 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:16.669 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2022-07-21T01:42:16.670 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.670 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2022-07-21T01:42:16.670 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:16.670 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2022-07-21T01:42:16.670 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2022-07-21T01:42:16.671 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2022-07-21T01:42:16.671 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2022-07-21T01:42:16.671 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2022-07-21T01:42:16.671 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:16.672 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2022-07-21T01:42:16.672 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2022-07-21T01:42:16.672 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:16.672 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.673 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:42:16.673 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:42:16.673 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:42:16.673 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.674 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:42:16.674 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:42:16.674 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:42:16.674 INFO:teuthology.orchestra.run.smithi003.stdout: mon op complaint time = 120 2022-07-21T01:42:16.675 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.675 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2022-07-21T01:42:16.675 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2022-07-21T01:42:16.675 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2022-07-21T01:42:16.676 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2022-07-21T01:42:16.676 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:42:16.676 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:42:16.676 INFO:teuthology.orchestra.run.smithi003.stdout: client mount timeout = 600 2022-07-21T01:42:16.676 INFO:teuthology.orchestra.run.smithi003.stdout: debug client = 20 2022-07-21T01:42:16.677 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:16.677 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 15m 2022-07-21T01:42:16.677 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 15m 2022-07-21T01:42:16.677 INFO:teuthology.orchestra.run.smithi003.stdout: client debug getattr caps = true 2022-07-21T01:42:16.678 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2022-07-21T01:42:16.678 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2022-07-21T01:42:16.678 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2022-07-21T01:42:16.678 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2022-07-21T01:42:16.679 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2022-07-21T01:42:16.679 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2022-07-21T01:42:16.679 INFO:teuthology.orchestra.run.smithi003.stdout:[mds] 2022-07-21T01:42:16.679 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds = 20 2022-07-21T01:42:16.680 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:16.680 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal frag = True 2022-07-21T01:42:16.680 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal fragment size max = 10000 2022-07-21T01:42:16.680 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal merge size = 5 2022-07-21T01:42:16.681 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal split bits = 3 2022-07-21T01:42:16.681 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal split size = 100 2022-07-21T01:42:16.681 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug frag = True 2022-07-21T01:42:16.681 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug scatterstat = True 2022-07-21T01:42:16.682 INFO:teuthology.orchestra.run.smithi003.stdout: mds op complaint time = 180 2022-07-21T01:42:16.682 INFO:teuthology.orchestra.run.smithi003.stdout: mds verify scatter = True 2022-07-21T01:42:16.682 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2022-07-21T01:42:16.682 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 15m 2022-07-21T01:42:16.683 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 15m 2022-07-21T01:42:16.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:16.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 Processor -- start 2022-07-21T01:42:16.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 -- start start 2022-07-21T01:42:16.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc6c80b4500 0x7fc6c80b9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:16.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c8146a70 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c80b4b20 con 0x7fc6c80b72f0 2022-07-21T01:42:16.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c80b4dc0 con 0x7fc6c80b4500 2022-07-21T01:42:16.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ccb1e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc6c80b9aa0 0x7fc6c8147170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59176/0 (socket says 172.21.15.3:59176) 2022-07-21T01:42:16.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ccb1e700 1 -- 172.21.15.3:0/228645293 learned_addr learned my addr 172.21.15.3:0/228645293 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bffff700 1 --2- 172.21.15.3:0/228645293 >> v2:172.21.15.143:3300/0 conn(0x7fc6c80b4500 0x7fc6c80b9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bf7fe700 1 -- 172.21.15.3:0/228645293 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500757813 0 0) 0x7fc6c8146a70 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bf7fe700 1 -- 172.21.15.3:0/228645293 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b4003260 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bffff700 1 -- 172.21.15.3:0/228645293 >> v1:172.21.15.143:6789/0 conn(0x7fc6c80b72f0 legacy=0x7fc6c813cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:16.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bffff700 1 -- 172.21.15.3:0/228645293 >> v1:172.21.15.3:6789/0 conn(0x7fc6c80b9aa0 legacy=0x7fc6c8147170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bffff700 1 -- 172.21.15.3:0/228645293 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c8146cd0 con 0x7fc6c80b4500 2022-07-21T01:42:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bffff700 1 --2- 172.21.15.3:0/228645293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b4500 0x7fc6c80b9e80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fc6b8002f70 tx=0x7fc6b8006df0).ready entity=mon.1 client_cookie=1d63f212a3c6613 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bf7fe700 1 -- 172.21.15.3:0/228645293 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6b800d610 con 0x7fc6c80b4500 2022-07-21T01:42:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bf7fe700 1 -- 172.21.15.3:0/228645293 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b8000e90 con 0x7fc6c80b4500 2022-07-21T01:42:16.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6bf7fe700 1 -- 172.21.15.3:0/228645293 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6b800d960 con 0x7fc6c80b4500 2022-07-21T01:42:16.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 -- 172.21.15.3:0/228645293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b4500 msgr2=0x7fc6c80b9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 --2- 172.21.15.3:0/228645293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b4500 0x7fc6c80b9e80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fc6b8002f70 tx=0x7fc6b8006df0).stop 2022-07-21T01:42:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 -- 172.21.15.3:0/228645293 shutdown_connections 2022-07-21T01:42:16.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 --2- 172.21.15.3:0/228645293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b4500 0x7fc6c80b9e80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:16.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 -- 172.21.15.3:0/228645293 >> 172.21.15.3:0/228645293 conn(0x7fc6c80600f0 msgr2=0x7fc6c8061830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 -- 172.21.15.3:0/228645293 shutdown_connections 2022-07-21T01:42:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.787+0000 7fc6ced82700 1 -- 172.21.15.3:0/228645293 wait complete. 2022-07-21T01:42:16.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 Processor -- start 2022-07-21T01:42:16.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 -- start start 2022-07-21T01:42:16.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6c80b4500 0x7fc6c8144720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:16.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b72f0 0x7fc6c8146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:16.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6c80b9aa0 0x7fc6c8149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:16.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c81530f0 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c80ad970 con 0x7fc6c80b72f0 2022-07-21T01:42:16.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c8153320 con 0x7fc6c80b4500 2022-07-21T01:42:16.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6cd31f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6c80b9aa0 0x7fc6c8149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:16.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6cd31f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6c80b9aa0 0x7fc6c8149d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50904/0 (socket says 172.21.15.3:50904) 2022-07-21T01:42:16.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6cd31f700 1 -- 172.21.15.3:0/1614931001 learned_addr learned my addr 172.21.15.3:0/1614931001 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:16.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6bffff700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b72f0 0x7fc6c8146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:16.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ccb1e700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6c80b4500 0x7fc6c8144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:16.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6cd31f700 1 -- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6c80b4500 msgr2=0x7fc6c8144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:16.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6cd31f700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6c80b4500 0x7fc6c8144720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:16.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6cd31f700 1 -- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b72f0 msgr2=0x7fc6c8146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:16.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6cd31f700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b72f0 0x7fc6c8146110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:16.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6cd31f700 1 -- 172.21.15.3:0/1614931001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b800b180 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6bffff700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b72f0 0x7fc6c8146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:16.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ccb1e700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6c80b4500 0x7fc6c8144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:16.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6cd31f700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6c80b9aa0 0x7fc6c8149d40 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fc6c8147050 tx=0x7fc6c0007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:16.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6bdffb700 1 -- 172.21.15.3:0/1614931001 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6c000e4a0 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6bdffb700 1 -- 172.21.15.3:0/1614931001 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c000e600 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c8146600 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6bdffb700 1 -- 172.21.15.3:0/1614931001 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6c000e960 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.791+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c814a680 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.795+0000 7fc6bdffb700 1 -- 172.21.15.3:0/1614931001 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc6c0019030 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.795+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c80b2d00 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.795+0000 7fc6bdffb700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6a8053aa0 0x7fc6a8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:16.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.795+0000 7fc6bdffb700 1 -- 172.21.15.3:0/1614931001 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc6c006b7c0 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.795+0000 7fc6ccb1e700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6a8053aa0 0x7fc6a8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:16.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.795+0000 7fc6ccb1e700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6a8053aa0 0x7fc6a8055f60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fc6c400b000 tx=0x7fc6c4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:16.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.803+0000 7fc6bdffb700 1 -- 172.21.15.3:0/1614931001 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc6c80b2d00 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.931+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6c80b2d00 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6bdffb700 1 -- 172.21.15.3:0/1614931001 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16285 (secure 0 0 0) 0x7fc6c80b2d00 con 0x7fc6c80b9aa0 2022-07-21T01:42:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:16.938 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":140,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:08.066592+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:05.096631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:06.060734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:16.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6a8053aa0 msgr2=0x7fc6a8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:16.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6a8053aa0 0x7fc6a8055f60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fc6c400b000 tx=0x7fc6c4009040).stop 2022-07-21T01:42:16.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6c80b9aa0 msgr2=0x7fc6c8149d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:16.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6c80b9aa0 0x7fc6c8149d40 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fc6c8147050 tx=0x7fc6c0007f40).stop 2022-07-21T01:42:16.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 shutdown_connections 2022-07-21T01:42:16.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6c80b9aa0 0x7fc6c8149d40 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6a8053aa0 0x7fc6a8055f60 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c80b72f0 0x7fc6c8146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 --2- 172.21.15.3:0/1614931001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6c80b4500 0x7fc6c8144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:16.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 >> 172.21.15.3:0/1614931001 conn(0x7fc6c80600f0 msgr2=0x7fc6c80b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:16.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 shutdown_connections 2022-07-21T01:42:16.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:16.935+0000 7fc6ced82700 1 -- 172.21.15.3:0/1614931001 wait complete. 2022-07-21T01:42:16.957 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:42:05.096631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:42:06.060734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:42:16.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:42:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.063+0000 7fd1aaf70700 1 Processor -- start 2022-07-21T01:42:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.063+0000 7fd1aaf70700 1 -- start start 2022-07-21T01:42:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.063+0000 7fd1aaf70700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd1a4060f40 0x7fd1a4144570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.063+0000 7fd1aaf70700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1a413a8b0 0x7fd1a413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.063+0000 7fd1aaf70700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a4061320 con 0x7fd1a405fb60 2022-07-21T01:42:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.063+0000 7fd1aaf70700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a40b09b0 con 0x7fd1a413a8b0 2022-07-21T01:42:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.063+0000 7fd1aaf70700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a40b0c50 con 0x7fd1a4060f40 2022-07-21T01:42:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aa76f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1a413a8b0 0x7fd1a413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aa76f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1a413a8b0 0x7fd1a413ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50912/0 (socket says 172.21.15.3:50912) 2022-07-21T01:42:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aa76f700 1 -- 172.21.15.3:0/3044734756 learned_addr learned my addr 172.21.15.3:0/3044734756 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1a9f6e700 1 --2- 172.21.15.3:0/3044734756 >> v2:172.21.15.143:3300/0 conn(0x7fd1a4060f40 0x7fd1a4144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aa76f700 1 -- 172.21.15.3:0/3044734756 >> v2:172.21.15.143:3300/0 conn(0x7fd1a4060f40 msgr2=0x7fd1a4144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aa76f700 1 --2- 172.21.15.3:0/3044734756 >> v2:172.21.15.143:3300/0 conn(0x7fd1a4060f40 0x7fd1a4144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aa76f700 1 -- 172.21.15.3:0/3044734756 >> v1:172.21.15.143:6789/0 conn(0x7fd1a405fb60 legacy=0x7fd1a4144a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aa76f700 1 -- 172.21.15.3:0/3044734756 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40aebd0 con 0x7fd1a413a8b0 2022-07-21T01:42:17.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aa76f700 1 --2- 172.21.15.3:0/3044734756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a413a8b0 0x7fd1a413ad00 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fd19c0032f0 tx=0x7fd19c009290).ready entity=mon.0 client_cookie=c2526984d7d4a59d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1a8f6c700 1 -- 172.21.15.3:0/3044734756 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd19c00b870 con 0x7fd1a413a8b0 2022-07-21T01:42:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1a8f6c700 1 -- 172.21.15.3:0/3044734756 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c00b9d0 con 0x7fd1a413a8b0 2022-07-21T01:42:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1a8f6c700 1 -- 172.21.15.3:0/3044734756 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd19c00bb40 con 0x7fd1a413a8b0 2022-07-21T01:42:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 -- 172.21.15.3:0/3044734756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a413a8b0 msgr2=0x7fd1a413ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 --2- 172.21.15.3:0/3044734756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a413a8b0 0x7fd1a413ad00 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fd19c0032f0 tx=0x7fd19c009290).stop 2022-07-21T01:42:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 -- 172.21.15.3:0/3044734756 shutdown_connections 2022-07-21T01:42:17.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 --2- 172.21.15.3:0/3044734756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a413a8b0 0x7fd1a413ad00 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 --2- 172.21.15.3:0/3044734756 >> v2:172.21.15.143:3300/0 conn(0x7fd1a4060f40 0x7fd1a4144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 -- 172.21.15.3:0/3044734756 >> 172.21.15.3:0/3044734756 conn(0x7fd1a40b5560 msgr2=0x7fd1a40b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:17.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 -- 172.21.15.3:0/3044734756 shutdown_connections 2022-07-21T01:42:17.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 -- 172.21.15.3:0/3044734756 wait complete. 2022-07-21T01:42:17.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 Processor -- start 2022-07-21T01:42:17.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.067+0000 7fd1aaf70700 1 -- start start 2022-07-21T01:42:17.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1aaf70700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1a405fb60 0x7fd1a4066c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1aaf70700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a4060f40 0x7fd1a414d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1aaf70700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1a413a8b0 0x7fd1a414dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1aaf70700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40ac4f0 con 0x7fd1a4060f40 2022-07-21T01:42:17.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1aaf70700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a414e090 con 0x7fd1a413a8b0 2022-07-21T01:42:17.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1aaf70700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40b5d60 con 0x7fd1a405fb60 2022-07-21T01:42:17.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a976d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a4060f40 0x7fd1a414d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a976d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a4060f40 0x7fd1a414d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50916/0 (socket says 172.21.15.3:50916) 2022-07-21T01:42:17.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a976d700 1 -- 172.21.15.3:0/885412742 learned_addr learned my addr 172.21.15.3:0/885412742 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:17.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1aa76f700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1a413a8b0 0x7fd1a414dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a9f6e700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1a405fb60 0x7fd1a4066c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a976d700 1 -- 172.21.15.3:0/885412742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1a405fb60 msgr2=0x7fd1a4066c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a976d700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1a405fb60 0x7fd1a4066c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a976d700 1 -- 172.21.15.3:0/885412742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1a413a8b0 msgr2=0x7fd1a414dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a976d700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1a413a8b0 0x7fd1a414dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a976d700 1 -- 172.21.15.3:0/885412742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c009530 con 0x7fd1a4060f40 2022-07-21T01:42:17.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1a976d700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a4060f40 0x7fd1a414d6b0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7fd1a40a95c0 tx=0x7fd1a0007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd192ffd700 1 -- 172.21.15.3:0/885412742 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1a000ce80 con 0x7fd1a4060f40 2022-07-21T01:42:17.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd192ffd700 1 -- 172.21.15.3:0/885412742 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a0007030 con 0x7fd1a4060f40 2022-07-21T01:42:17.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a4064990 con 0x7fd1a4060f40 2022-07-21T01:42:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd192ffd700 1 -- 172.21.15.3:0/885412742 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1a0007390 con 0x7fd1a4060f40 2022-07-21T01:42:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.071+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a40655d0 con 0x7fd1a4060f40 2022-07-21T01:42:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.075+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a40aee70 con 0x7fd1a4060f40 2022-07-21T01:42:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.075+0000 7fd192ffd700 1 -- 172.21.15.3:0/885412742 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd1a0010030 con 0x7fd1a4060f40 2022-07-21T01:42:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.075+0000 7fd192ffd700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd184053980 0x7fd184055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.075+0000 7fd192ffd700 1 -- 172.21.15.3:0/885412742 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd1a006bae0 con 0x7fd1a4060f40 2022-07-21T01:42:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.075+0000 7fd1a9f6e700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd184053980 0x7fd184055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.075+0000 7fd1a9f6e700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd184053980 0x7fd184055e40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fd19400c000 tx=0x7fd194008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.079+0000 7fd192ffd700 1 -- 172.21.15.3:0/885412742 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd1a00428a0 con 0x7fd1a4060f40 2022-07-21T01:42:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.207+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd1a40574e0 con 0x7fd1a4060f40 2022-07-21T01:42:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd192ffd700 1 -- 172.21.15.3:0/885412742 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v140) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd1a0045ec0 con 0x7fd1a4060f40 2022-07-21T01:42:17.213 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:42:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd184053980 msgr2=0x7fd184055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd184053980 0x7fd184055e40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fd19400c000 tx=0x7fd194008040).stop 2022-07-21T01:42:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a4060f40 msgr2=0x7fd1a414d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a4060f40 0x7fd1a414d6b0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7fd1a40a95c0 tx=0x7fd1a0007c10).stop 2022-07-21T01:42:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 shutdown_connections 2022-07-21T01:42:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd184053980 0x7fd184055e40 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1a413a8b0 0x7fd1a414dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1a4060f40 0x7fd1a414d6b0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 --2- 172.21.15.3:0/885412742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1a405fb60 0x7fd1a4066c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 >> 172.21.15.3:0/885412742 conn(0x7fd1a40b5560 msgr2=0x7fd1a4138700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:17.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 shutdown_connections 2022-07-21T01:42:17.218 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.211+0000 7fd1aaf70700 1 -- 172.21.15.3:0/885412742 wait complete. 2022-07-21T01:42:17.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:42:17.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.335+0000 7ff875d71700 1 Processor -- start 2022-07-21T01:42:17.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.335+0000 7ff875d71700 1 -- start start 2022-07-21T01:42:17.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8700b1790 0x7ff8700604f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff870147260 0x7ff870140370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff870146880 con 0x7ff87005f9e0 2022-07-21T01:42:17.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8701476d0 con 0x7ff8700b1790 2022-07-21T01:42:17.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff870146b20 con 0x7ff870147260 2022-07-21T01:42:17.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff86ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8700b1790 0x7ff8700604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875570700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff870147260 0x7ff870140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff86ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8700b1790 0x7ff8700604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:56372/0 (socket says 172.21.15.3:56372) 2022-07-21T01:42:17.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff86ffff700 1 -- 172.21.15.3:0/793259968 learned_addr learned my addr 172.21.15.3:0/793259968 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:17.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875570700 1 -- 172.21.15.3:0/793259968 >> v2:172.21.15.143:3300/0 conn(0x7ff8700b1790 msgr2=0x7ff8700604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:17.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875570700 1 --2- 172.21.15.3:0/793259968 >> v2:172.21.15.143:3300/0 conn(0x7ff8700b1790 0x7ff8700604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875570700 1 -- 172.21.15.3:0/793259968 >> v1:172.21.15.143:6789/0 conn(0x7ff87005f9e0 legacy=0x7ff870143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875570700 1 -- 172.21.15.3:0/793259968 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff870146dc0 con 0x7ff870147260 2022-07-21T01:42:17.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875570700 1 --2- 172.21.15.3:0/793259968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff870147260 0x7ff870140370 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff8680032f0 tx=0x7ff8680091d0).ready entity=mon.2 client_cookie=f592fcf1b3f0c1fd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff86f7fe700 1 -- 172.21.15.3:0/793259968 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff86800b6c0 con 0x7ff870147260 2022-07-21T01:42:17.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff86f7fe700 1 -- 172.21.15.3:0/793259968 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86800b820 con 0x7ff870147260 2022-07-21T01:42:17.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff86f7fe700 1 -- 172.21.15.3:0/793259968 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff86800b990 con 0x7ff870147260 2022-07-21T01:42:17.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 -- 172.21.15.3:0/793259968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff870147260 msgr2=0x7ff870140370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 --2- 172.21.15.3:0/793259968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff870147260 0x7ff870140370 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff8680032f0 tx=0x7ff8680091d0).stop 2022-07-21T01:42:17.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 -- 172.21.15.3:0/793259968 shutdown_connections 2022-07-21T01:42:17.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 --2- 172.21.15.3:0/793259968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff870147260 0x7ff870140370 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 --2- 172.21.15.3:0/793259968 >> v2:172.21.15.143:3300/0 conn(0x7ff8700b1790 0x7ff8700604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 -- 172.21.15.3:0/793259968 >> 172.21.15.3:0/793259968 conn(0x7ff8700b54f0 msgr2=0x7ff8700b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:17.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 -- 172.21.15.3:0/793259968 shutdown_connections 2022-07-21T01:42:17.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.339+0000 7ff875d71700 1 -- 172.21.15.3:0/793259968 wait complete. 2022-07-21T01:42:17.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 Processor -- start 2022-07-21T01:42:17.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 -- start start 2022-07-21T01:42:17.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87005f9e0 0x7ff8701526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8700b1790 0x7ff8700abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff870147260 0x7ff870143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff870155c80 con 0x7ff870147260 2022-07-21T01:42:17.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700aac20 con 0x7ff8700b1790 2022-07-21T01:42:17.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700aae80 con 0x7ff87005f9e0 2022-07-21T01:42:17.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875570700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff870147260 0x7ff870143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff86ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8700b1790 0x7ff8700abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875570700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff870147260 0x7ff870143a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50932/0 (socket says 172.21.15.3:50932) 2022-07-21T01:42:17.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875570700 1 -- 172.21.15.3:0/361332425 learned_addr learned my addr 172.21.15.3:0/361332425 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:17.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff874d6f700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87005f9e0 0x7ff8701526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875570700 1 -- 172.21.15.3:0/361332425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87005f9e0 msgr2=0x7ff8701526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875570700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87005f9e0 0x7ff8701526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875570700 1 -- 172.21.15.3:0/361332425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8700b1790 msgr2=0x7ff8700abf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875570700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8700b1790 0x7ff8700abf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875570700 1 -- 172.21.15.3:0/361332425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff868009470 con 0x7ff870147260 2022-07-21T01:42:17.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875570700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff870147260 0x7ff870143a60 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7ff8680032f0 tx=0x7ff868016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff86dffb700 1 -- 172.21.15.3:0/361332425 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff86800b6c0 con 0x7ff870147260 2022-07-21T01:42:17.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff86dffb700 1 -- 172.21.15.3:0/361332425 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8680205e0 con 0x7ff870147260 2022-07-21T01:42:17.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff870155eb0 con 0x7ff870147260 2022-07-21T01:42:17.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff86dffb700 1 -- 172.21.15.3:0/361332425 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff868020940 con 0x7ff870147260 2022-07-21T01:42:17.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.343+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff870143f50 con 0x7ff870147260 2022-07-21T01:42:17.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.347+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff870156040 con 0x7ff870147260 2022-07-21T01:42:17.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.347+0000 7ff86dffb700 1 -- 172.21.15.3:0/361332425 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff86801e070 con 0x7ff870147260 2022-07-21T01:42:17.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.347+0000 7ff86dffb700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff850053a70 0x7ff850055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.347+0000 7ff874d6f700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff850053a70 0x7ff850055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.347+0000 7ff86dffb700 1 -- 172.21.15.3:0/361332425 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7ff86806d020 con 0x7ff870147260 2022-07-21T01:42:17.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.347+0000 7ff874d6f700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff850053a70 0x7ff850055f30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7ff86400d000 tx=0x7ff864000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.355+0000 7ff86dffb700 1 -- 172.21.15.3:0/361332425 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff868047810 con 0x7ff870147260 2022-07-21T01:42:17.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff8700ae470 con 0x7ff870147260 2022-07-21T01:42:17.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff86dffb700 1 -- 172.21.15.3:0/361332425 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v140) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff868015020 con 0x7ff870147260 2022-07-21T01:42:17.487 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:17.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff850053a70 msgr2=0x7ff850055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff850053a70 0x7ff850055f30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7ff86400d000 tx=0x7ff864000bc0).stop 2022-07-21T01:42:17.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff870147260 msgr2=0x7ff870143a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff870147260 0x7ff870143a60 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7ff8680032f0 tx=0x7ff868016000).stop 2022-07-21T01:42:17.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 shutdown_connections 2022-07-21T01:42:17.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff870147260 0x7ff870143a60 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7ff850053a70 0x7ff850055f30 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8700b1790 0x7ff8700abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 --2- 172.21.15.3:0/361332425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87005f9e0 0x7ff8701526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 >> 172.21.15.3:0/361332425 conn(0x7ff8700b54f0 msgr2=0x7ff8700af390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:17.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 shutdown_connections 2022-07-21T01:42:17.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.483+0000 7ff875d71700 1 -- 172.21.15.3:0/361332425 wait complete. 2022-07-21T01:42:17.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.607+0000 7faf47fff700 1 Processor -- start 2022-07-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.607+0000 7faf47fff700 1 -- start start 2022-07-21T01:42:17.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.607+0000 7faf47fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf480b1500 0x7faf480b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.607+0000 7faf47fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf48061070 con 0x7faf480a8f60 2022-07-21T01:42:17.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.607+0000 7faf47fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf480b1d20 con 0x7faf48060c90 2022-07-21T01:42:17.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.607+0000 7faf47fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf480b1a80 con 0x7faf480b1500 2022-07-21T01:42:17.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.607+0000 7faf3ffff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7faf480a8f60 0x7faf4813d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59216/0 (socket says 172.21.15.3:59216) 2022-07-21T01:42:17.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.607+0000 7faf3ffff700 1 -- 172.21.15.3:0/3732900456 learned_addr learned my addr 172.21.15.3:0/3732900456 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:17.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.607+0000 7faf477fe700 1 --2- 172.21.15.3:0/3732900456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf480b1500 0x7faf480b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf467fc700 1 -- 172.21.15.3:0/3732900456 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284598034 0 0) 0x7faf48061070 con 0x7faf480a8f60 2022-07-21T01:42:17.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf467fc700 1 -- 172.21.15.3:0/3732900456 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf28003260 con 0x7faf480a8f60 2022-07-21T01:42:17.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf467fc700 1 -- 172.21.15.3:0/3732900456 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524614383 0 0) 0x7faf480b1d20 con 0x7faf48060c90 2022-07-21T01:42:17.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf467fc700 1 -- 172.21.15.3:0/3732900456 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf48061070 con 0x7faf48060c90 2022-07-21T01:42:17.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf477fe700 1 -- 172.21.15.3:0/3732900456 >> v1:172.21.15.143:6789/0 conn(0x7faf48060c90 legacy=0x7faf480ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf477fe700 1 -- 172.21.15.3:0/3732900456 >> v1:172.21.15.3:6789/0 conn(0x7faf480a8f60 legacy=0x7faf4813d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf477fe700 1 -- 172.21.15.3:0/3732900456 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf480b05a0 con 0x7faf480b1500 2022-07-21T01:42:17.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf477fe700 1 --2- 172.21.15.3:0/3732900456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf480b1500 0x7faf480b0ce0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7faf400032f0 tx=0x7faf400091d0).ready entity=mon.2 client_cookie=fe3d34624f8362e5 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf467fc700 1 -- 172.21.15.3:0/3732900456 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf4000b640 con 0x7faf480b1500 2022-07-21T01:42:17.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf467fc700 1 -- 172.21.15.3:0/3732900456 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4000b7a0 con 0x7faf480b1500 2022-07-21T01:42:17.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf467fc700 1 -- 172.21.15.3:0/3732900456 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf4000b910 con 0x7faf480b1500 2022-07-21T01:42:17.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf47fff700 1 -- 172.21.15.3:0/3732900456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf480b1500 msgr2=0x7faf480b0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf47fff700 1 --2- 172.21.15.3:0/3732900456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf480b1500 0x7faf480b0ce0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7faf400032f0 tx=0x7faf400091d0).stop 2022-07-21T01:42:17.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf47fff700 1 -- 172.21.15.3:0/3732900456 shutdown_connections 2022-07-21T01:42:17.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf47fff700 1 --2- 172.21.15.3:0/3732900456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf480b1500 0x7faf480b0ce0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf47fff700 1 -- 172.21.15.3:0/3732900456 >> 172.21.15.3:0/3732900456 conn(0x7faf480b5560 msgr2=0x7faf480b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:17.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf47fff700 1 -- 172.21.15.3:0/3732900456 shutdown_connections 2022-07-21T01:42:17.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.611+0000 7faf47fff700 1 -- 172.21.15.3:0/3732900456 wait complete. 2022-07-21T01:42:17.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 Processor -- start 2022-07-21T01:42:17.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 -- start start 2022-07-21T01:42:17.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf48060c90 0x7faf4813a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf480a8f60 0x7faf4813b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf480b1500 0x7faf4813b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4812de60 con 0x7faf480a8f60 2022-07-21T01:42:17.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4812e370 con 0x7faf480b1500 2022-07-21T01:42:17.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4813c860 con 0x7faf48060c90 2022-07-21T01:42:17.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf3ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf480a8f60 0x7faf4813b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf3ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf480a8f60 0x7faf4813b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50946/0 (socket says 172.21.15.3:50946) 2022-07-21T01:42:17.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf3ffff700 1 -- 172.21.15.3:0/3769789794 learned_addr learned my addr 172.21.15.3:0/3769789794 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:17.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf477fe700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf480b1500 0x7faf4813b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf46ffd700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf48060c90 0x7faf4813a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf3ffff700 1 -- 172.21.15.3:0/3769789794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf48060c90 msgr2=0x7faf4813a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf3ffff700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf48060c90 0x7faf4813a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf3ffff700 1 -- 172.21.15.3:0/3769789794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf480b1500 msgr2=0x7faf4813b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf3ffff700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf480b1500 0x7faf4813b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf3ffff700 1 -- 172.21.15.3:0/3769789794 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf40009470 con 0x7faf480a8f60 2022-07-21T01:42:17.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf3ffff700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf480a8f60 0x7faf4813b990 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7faf30001110 tx=0x7faf30007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf44ff9700 1 -- 172.21.15.3:0/3769789794 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf300070d0 con 0x7faf480a8f60 2022-07-21T01:42:17.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf44ff9700 1 -- 172.21.15.3:0/3769789794 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf30007230 con 0x7faf480a8f60 2022-07-21T01:42:17.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4813fc90 con 0x7faf480a8f60 2022-07-21T01:42:17.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf44ff9700 1 -- 172.21.15.3:0/3769789794 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf30005610 con 0x7faf480a8f60 2022-07-21T01:42:17.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.615+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4813f9f0 con 0x7faf480a8f60 2022-07-21T01:42:17.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.619+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4813c9d0 con 0x7faf480a8f60 2022-07-21T01:42:17.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.619+0000 7faf44ff9700 1 -- 172.21.15.3:0/3769789794 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faf30010070 con 0x7faf480a8f60 2022-07-21T01:42:17.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.619+0000 7faf44ff9700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faf2c053a70 0x7faf2c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.619+0000 7faf44ff9700 1 -- 172.21.15.3:0/3769789794 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7faf3006b520 con 0x7faf480a8f60 2022-07-21T01:42:17.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.619+0000 7faf46ffd700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faf2c053a70 0x7faf2c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.619+0000 7faf46ffd700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faf2c053a70 0x7faf2c055f30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7faf3800d000 tx=0x7faf38007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.627+0000 7faf44ff9700 1 -- 172.21.15.3:0/3769789794 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faf300422e0 con 0x7faf480a8f60 2022-07-21T01:42:17.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.751+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faf480574e0 con 0x7faf480a8f60 2022-07-21T01:42:17.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.751+0000 7faf44ff9700 1 -- 172.21.15.3:0/3769789794 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+10 (secure 0 0 0) 0x7faf300180a0 con 0x7faf480a8f60 2022-07-21T01:42:17.756 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:17.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faf2c053a70 msgr2=0x7faf2c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faf2c053a70 0x7faf2c055f30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7faf3800d000 tx=0x7faf38007a00).stop 2022-07-21T01:42:17.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf480a8f60 msgr2=0x7faf4813b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf480a8f60 0x7faf4813b990 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7faf30001110 tx=0x7faf30007a60).stop 2022-07-21T01:42:17.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 shutdown_connections 2022-07-21T01:42:17.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faf2c053a70 0x7faf2c055f30 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf480b1500 0x7faf4813b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf480a8f60 0x7faf4813b990 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 --2- 172.21.15.3:0/3769789794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf48060c90 0x7faf4813a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 >> 172.21.15.3:0/3769789794 conn(0x7faf480b5560 msgr2=0x7faf480aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 shutdown_connections 2022-07-21T01:42:17.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.755+0000 7faf47fff700 1 -- 172.21.15.3:0/3769789794 wait complete. 2022-07-21T01:42:17.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:42:17.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.867+0000 7facc0d56700 1 Processor -- start 2022-07-21T01:42:17.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.867+0000 7facc0d56700 1 -- start start 2022-07-21T01:42:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 0x7facbc0b3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7facbc0b2100 0x7facbc0add00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc0b2540 con 0x7facbc1417c0 2022-07-21T01:42:17.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc0ae1f0 con 0x7facbc0b2100 2022-07-21T01:42:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc1453f0 con 0x7facbc0ad920 2022-07-21T01:42:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facbb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 0x7facbc0b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facbaffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7facbc0b2100 0x7facbc0add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facbaffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7facbc0b2100 0x7facbc0add00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:56398/0 (socket says 172.21.15.3:56398) 2022-07-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facbaffd700 1 -- 172.21.15.3:0/1292393607 learned_addr learned my addr 172.21.15.3:0/1292393607 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facbaffd700 1 -- 172.21.15.3:0/1292393607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 msgr2=0x7facbc0b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facbaffd700 1 --2- 172.21.15.3:0/1292393607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 0x7facbc0b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facbaffd700 1 -- 172.21.15.3:0/1292393607 >> v1:172.21.15.143:6789/0 conn(0x7facbc1417c0 legacy=0x7facbc129d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facbaffd700 1 -- 172.21.15.3:0/1292393607 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc1445e0 con 0x7facbc0b2100 2022-07-21T01:42:17.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facbaffd700 1 --2- 172.21.15.3:0/1292393607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc0b2100 0x7facbc0add00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7facac002f70 tx=0x7facac006eb0).ready entity=mon.1 client_cookie=a0274799e500f056 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facba7fc700 1 -- 172.21.15.3:0/1292393607 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac00d570 con 0x7facbc0b2100 2022-07-21T01:42:17.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facba7fc700 1 -- 172.21.15.3:0/1292393607 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac000e90 con 0x7facbc0b2100 2022-07-21T01:42:17.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facba7fc700 1 -- 172.21.15.3:0/1292393607 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac00d900 con 0x7facbc0b2100 2022-07-21T01:42:17.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 -- 172.21.15.3:0/1292393607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc0b2100 msgr2=0x7facbc0add00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 --2- 172.21.15.3:0/1292393607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc0b2100 0x7facbc0add00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7facac002f70 tx=0x7facac006eb0).stop 2022-07-21T01:42:17.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 -- 172.21.15.3:0/1292393607 shutdown_connections 2022-07-21T01:42:17.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 --2- 172.21.15.3:0/1292393607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc0b2100 0x7facbc0add00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 --2- 172.21.15.3:0/1292393607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 0x7facbc0b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 -- 172.21.15.3:0/1292393607 >> 172.21.15.3:0/1292393607 conn(0x7facbc0b7760 msgr2=0x7facbc0b7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:17.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 -- 172.21.15.3:0/1292393607 shutdown_connections 2022-07-21T01:42:17.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.871+0000 7facc0d56700 1 -- 172.21.15.3:0/1292393607 wait complete. 2022-07-21T01:42:17.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 Processor -- start 2022-07-21T01:42:17.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 -- start start 2022-07-21T01:42:17.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 0x7facbc150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facbc0b2100 0x7facbc0b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc1417c0 0x7facbc112830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc112440 con 0x7facbc0b2100 2022-07-21T01:42:17.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc1402d0 con 0x7facbc1417c0 2022-07-21T01:42:17.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0b02a0 con 0x7facbc0ad920 2022-07-21T01:42:17.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facbc0b2100 0x7facbc0b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facbc0b2100 0x7facbc0b11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50958/0 (socket says 172.21.15.3:50958) 2022-07-21T01:42:17.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbaffd700 1 -- 172.21.15.3:0/2237698448 learned_addr learned my addr 172.21.15.3:0/2237698448 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:17.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbbfff700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc1417c0 0x7facbc112830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbb7fe700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 0x7facbc150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbaffd700 1 -- 172.21.15.3:0/2237698448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 msgr2=0x7facbc150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbaffd700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 0x7facbc150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbaffd700 1 -- 172.21.15.3:0/2237698448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc1417c0 msgr2=0x7facbc112830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:17.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbaffd700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc1417c0 0x7facbc112830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:17.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbaffd700 1 -- 172.21.15.3:0/2237698448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac00b1e0 con 0x7facbc0b2100 2022-07-21T01:42:17.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbbfff700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc1417c0 0x7facbc112830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:17.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facbaffd700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facbc0b2100 0x7facbc0b11f0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7facbc112390 tx=0x7facac015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facb8ff9700 1 -- 172.21.15.3:0/2237698448 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac0068b0 con 0x7facbc0b2100 2022-07-21T01:42:17.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facb8ff9700 1 -- 172.21.15.3:0/2237698448 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac00de30 con 0x7facbc0b2100 2022-07-21T01:42:17.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc0b0540 con 0x7facbc0b2100 2022-07-21T01:42:17.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facb8ff9700 1 -- 172.21.15.3:0/2237698448 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac011d30 con 0x7facbc0b2100 2022-07-21T01:42:17.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.875+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc0b16e0 con 0x7facbc0b2100 2022-07-21T01:42:17.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.879+0000 7facb8ff9700 1 -- 172.21.15.3:0/2237698448 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7facac025030 con 0x7facbc0b2100 2022-07-21T01:42:17.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.879+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc0574e0 con 0x7facbc0b2100 2022-07-21T01:42:17.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.879+0000 7facb8ff9700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac9c053b00 0x7fac9c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:17.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.879+0000 7facbb7fe700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac9c053b00 0x7fac9c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:17.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.879+0000 7facb8ff9700 1 -- 172.21.15.3:0/2237698448 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7facac06d1f0 con 0x7facbc0b2100 2022-07-21T01:42:17.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.879+0000 7facbb7fe700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac9c053b00 0x7fac9c055fc0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7facb000b000 tx=0x7facb0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:17.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:17.883+0000 7facb8ff9700 1 -- 172.21.15.3:0/2237698448 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7facac043f20 con 0x7facbc0b2100 2022-07-21T01:42:18.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7facbc112ec0 con 0x7facbc0b2100 2022-07-21T01:42:18.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facb8ff9700 1 -- 172.21.15.3:0/2237698448 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+10 (secure 0 0 0) 0x7facac023020 con 0x7facbc0b2100 2022-07-21T01:42:18.018 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:18.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac9c053b00 msgr2=0x7fac9c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:18.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac9c053b00 0x7fac9c055fc0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7facb000b000 tx=0x7facb0006df0).stop 2022-07-21T01:42:18.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facbc0b2100 msgr2=0x7facbc0b11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:18.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facbc0b2100 0x7facbc0b11f0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7facbc112390 tx=0x7facac015000).stop 2022-07-21T01:42:18.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 shutdown_connections 2022-07-21T01:42:18.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac9c053b00 0x7fac9c055fc0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facbc1417c0 0x7facbc112830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facbc0b2100 0x7facbc0b11f0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 --2- 172.21.15.3:0/2237698448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facbc0ad920 0x7facbc150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 >> 172.21.15.3:0/2237698448 conn(0x7facbc0b7760 msgr2=0x7facbc141fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:18.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 shutdown_connections 2022-07-21T01:42:18.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.015+0000 7facc0d56700 1 -- 172.21.15.3:0/2237698448 wait complete. 2022-07-21T01:42:18.031 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:42:18.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:18.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.123+0000 7f8650539700 1 Processor -- start 2022-07-21T01:42:18.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.123+0000 7f8650539700 1 -- start start 2022-07-21T01:42:18.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f864805f9e0 0x7f86480b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:18.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86480ac1f0 0x7f86480b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:18.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86480b0180 con 0x7f864813f7c0 2022-07-21T01:42:18.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86480a9420 con 0x7f864805f9e0 2022-07-21T01:42:18.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86480b03e0 con 0x7f86480ac1f0 2022-07-21T01:42:18.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864dad4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86480ac1f0 0x7f86480b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:18.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864dad4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86480ac1f0 0x7f86480b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:57880/0 (socket says 172.21.15.3:57880) 2022-07-21T01:42:18.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864dad4700 1 -- 172.21.15.3:0/848266190 learned_addr learned my addr 172.21.15.3:0/848266190 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:18.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864e2d5700 1 --2- 172.21.15.3:0/848266190 >> v2:172.21.15.143:3300/0 conn(0x7f864805f9e0 0x7f86480b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:18.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864d2d3700 1 -- 172.21.15.3:0/848266190 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162730146 0 0) 0x7f86480b0180 con 0x7f864813f7c0 2022-07-21T01:42:18.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864d2d3700 1 -- 172.21.15.3:0/848266190 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630003260 con 0x7f864813f7c0 2022-07-21T01:42:18.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864dad4700 1 -- 172.21.15.3:0/848266190 >> v2:172.21.15.143:3300/0 conn(0x7f864805f9e0 msgr2=0x7f86480b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:18.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864dad4700 1 --2- 172.21.15.3:0/848266190 >> v2:172.21.15.143:3300/0 conn(0x7f864805f9e0 0x7f86480b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864dad4700 1 -- 172.21.15.3:0/848266190 >> v1:172.21.15.143:6789/0 conn(0x7f864813f7c0 legacy=0x7f864813fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864dad4700 1 -- 172.21.15.3:0/848266190 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86480b0680 con 0x7f86480ac1f0 2022-07-21T01:42:18.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864dad4700 1 --2- 172.21.15.3:0/848266190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86480ac1f0 0x7f86480b1010 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f86400032f0 tx=0x7f8640009290).ready entity=mon.2 client_cookie=c3ad47073b72763a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864d2d3700 1 -- 172.21.15.3:0/848266190 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f864000b6c0 con 0x7f86480ac1f0 2022-07-21T01:42:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864d2d3700 1 -- 172.21.15.3:0/848266190 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864000b820 con 0x7f86480ac1f0 2022-07-21T01:42:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f864d2d3700 1 -- 172.21.15.3:0/848266190 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f864000b990 con 0x7f86480ac1f0 2022-07-21T01:42:18.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 -- 172.21.15.3:0/848266190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86480ac1f0 msgr2=0x7f86480b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:18.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 --2- 172.21.15.3:0/848266190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86480ac1f0 0x7f86480b1010 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f86400032f0 tx=0x7f8640009290).stop 2022-07-21T01:42:18.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 -- 172.21.15.3:0/848266190 shutdown_connections 2022-07-21T01:42:18.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 --2- 172.21.15.3:0/848266190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86480ac1f0 0x7f86480b1010 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 --2- 172.21.15.3:0/848266190 >> v2:172.21.15.143:3300/0 conn(0x7f864805f9e0 0x7f86480b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 -- 172.21.15.3:0/848266190 >> 172.21.15.3:0/848266190 conn(0x7f864805cbb0 msgr2=0x7f864805cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 -- 172.21.15.3:0/848266190 shutdown_connections 2022-07-21T01:42:18.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.127+0000 7f8650539700 1 -- 172.21.15.3:0/848266190 wait complete. 2022-07-21T01:42:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 Processor -- start 2022-07-21T01:42:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 -- start start 2022-07-21T01:42:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f864805f9e0 0x7f864809f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86480ac1f0 0x7f864809fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f864813f7c0 0x7f8648066fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864809f050 con 0x7f864805f9e0 2022-07-21T01:42:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864809f2f0 con 0x7f86480ac1f0 2022-07-21T01:42:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8648155cc0 con 0x7f864813f7c0 2022-07-21T01:42:18.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864e2d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f864805f9e0 0x7f864809f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864dad4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86480ac1f0 0x7f864809fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864e2d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f864805f9e0 0x7f864809f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50970/0 (socket says 172.21.15.3:50970) 2022-07-21T01:42:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864e2d5700 1 -- 172.21.15.3:0/576537814 learned_addr learned my addr 172.21.15.3:0/576537814 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:18.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864ead6700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f864813f7c0 0x7f8648066fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864dad4700 1 -- 172.21.15.3:0/576537814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f864813f7c0 msgr2=0x7f8648066fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864dad4700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f864813f7c0 0x7f8648066fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864dad4700 1 -- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f864805f9e0 msgr2=0x7f864809f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864dad4700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f864805f9e0 0x7f864809f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864dad4700 1 -- 172.21.15.3:0/576537814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8640009530 con 0x7f86480ac1f0 2022-07-21T01:42:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864e2d5700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f864805f9e0 0x7f864809f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f864dad4700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86480ac1f0 0x7f864809fd70 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f86400032f0 tx=0x7f8640016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f863b7fe700 1 -- 172.21.15.3:0/576537814 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f864000b990 con 0x7f86480ac1f0 2022-07-21T01:42:18.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f863b7fe700 1 -- 172.21.15.3:0/576537814 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8640021670 con 0x7f86480ac1f0 2022-07-21T01:42:18.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8648155f60 con 0x7f86480ac1f0 2022-07-21T01:42:18.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f863b7fe700 1 -- 172.21.15.3:0/576537814 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f864000b6c0 con 0x7f86480ac1f0 2022-07-21T01:42:18.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.131+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864809eb80 con 0x7f86480ac1f0 2022-07-21T01:42:18.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.135+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8648140750 con 0x7f86480ac1f0 2022-07-21T01:42:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.135+0000 7f863b7fe700 1 -- 172.21.15.3:0/576537814 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f864000f070 con 0x7f86480ac1f0 2022-07-21T01:42:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.135+0000 7f863b7fe700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8628053b00 0x7f8628055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.135+0000 7f863b7fe700 1 -- 172.21.15.3:0/576537814 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f864006d170 con 0x7f86480ac1f0 2022-07-21T01:42:18.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.135+0000 7f864e2d5700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8628053b00 0x7f8628055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.135+0000 7f864e2d5700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8628053b00 0x7f8628055fc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f864400b000 tx=0x7f8644006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:18.159 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.143+0000 7f863b7fe700 1 -- 172.21.15.3:0/576537814 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8640043e90 con 0x7f86480ac1f0 2022-07-21T01:42:18.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.307+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86480574e0 con 0x7f86480ac1f0 2022-07-21T01:42:18.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.307+0000 7f863b7fe700 1 -- 172.21.15.3:0/576537814 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f8640015030 con 0x7f86480ac1f0 2022-07-21T01:42:18.312 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:18.313 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":71,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/567682837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":567682837},{"type":"v1","addr":"172.21.15.143:6833","nonce":567682837}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/140242740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140242740},{"type":"v1","addr":"172.21.15.3:6835","nonce":140242740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:07.063040+0000","modified":"2022-07-21T01:42:08.065747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8049":{"gid":8049,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/506442168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":506442168},{"type":"v1","addr":"172.21.15.3:6837","nonce":506442168}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-21T01:42:18.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8628053b00 msgr2=0x7f8628055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:18.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8628053b00 0x7f8628055fc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f864400b000 tx=0x7f8644006df0).stop 2022-07-21T01:42:18.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86480ac1f0 msgr2=0x7f864809fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:18.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86480ac1f0 0x7f864809fd70 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f86400032f0 tx=0x7f8640016000).stop 2022-07-21T01:42:18.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 shutdown_connections 2022-07-21T01:42:18.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f864813f7c0 0x7f8648066fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8628053b00 0x7f8628055fc0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86480ac1f0 0x7f864809fd70 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 --2- 172.21.15.3:0/576537814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f864805f9e0 0x7f864809f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:18.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 >> 172.21.15.3:0/576537814 conn(0x7f864805cbb0 msgr2=0x7f86480aeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:18.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 shutdown_connections 2022-07-21T01:42:18.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:18.311+0000 7f8650539700 1 -- 172.21.15.3:0/576537814 wait complete. 2022-07-21T01:42:18.322 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 71 2022-07-21T01:42:18.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:07.063040+0000', 'modified': '2022-07-21T01:42:08.065747+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8049}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8049': {'gid': 8049, 'name': 'a', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/506442168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 506442168}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 506442168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:42:18.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:42:18.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-21T01:42:18.754 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8049,"want_state":"up:active","state":"up:active","rank_uptime":11.679027777,"mdsmap_epoch":71,"osdmap_epoch":140,"osdmap_epoch_barrier":139,"uptime":16.408676595999999} 2022-07-21T01:42:18.755 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8049, "mdsmap_epoch": 71, "osdmap_epoch": 140, "osdmap_epoch_barrier": 139, "rank_uptime": 11.679027777, "state": "up:active", "uptime": 16.408676596, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:42:18.755 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:42:18.756 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:42:18.756 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:42:18.757 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:18.757 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:18.766 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:42:18.784 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:18.785 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:18.841 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:18.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:18.844 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:42:18.850 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:42:18.851 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:18.894 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:42:18.913 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:18.914 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:18.939 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:18.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:18.942 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:42:18.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:42:18.968+0000 7f2664933080 -1 init, newargv = 0x558cb2524930 newargc=9 2022-07-21T01:42:18.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22961]: starting ceph client 2022-07-21T01:42:18.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22961]: starting fuse 2022-07-21T01:42:20.076 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:42:20.096 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:20.096 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:20.113 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:20.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:20.117 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:42:20.167 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:42:20.167 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:42:20.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:20.168 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:42:20.169 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:42:20.173 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:42:20.174 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:42:20.174 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:42:20.175 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:20.175 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:42:20.176 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:42:20.177 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:42:20.177 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:20.178 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:42:20.178 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:42:20.178 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:42:20.179 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:42:20.179 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:20.180 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:42:20.180 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:42:20.181 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:42:20.181 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:42:20.182 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:42:20.182 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:20.183 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:42:20.183 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:42:20.227 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.22961.asok 2022-07-21T01:42:20.232 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22961.asok 2022-07-21T01:42:20.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:20.235 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22961.asok status 2022-07-21T01:42:20.326 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:42:20.326 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:42:20.326 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:42:20.327 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:42:20.327 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:42:20.327 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:42:20.327 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:42:20.328 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "22961", 2022-07-21T01:42:20.328 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:42:20.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:20.329 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:42:20.329 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:42:20.329 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 8209, 2022-07-21T01:42:20.329 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:42:20.330 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:42:20.330 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:42:20.330 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 8209 2022-07-21T01:42:20.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:20.331 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:42:20.331 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:42:20.331 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:42:20.332 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 1966402305 2022-07-21T01:42:20.332 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:42:20.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:20.333 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:42:20.333 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:42:20.333 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:42:20.333 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 1966402305 2022-07-21T01:42:20.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:20.334 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.8209 172.21.15.174:0/1966402305", 2022-07-21T01:42:20.334 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/1966402305", 2022-07-21T01:42:20.334 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:42:20.335 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 71, 2022-07-21T01:42:20.335 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 140, 2022-07-21T01:42:20.335 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:42:20.335 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:42:20.336 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:42:20.336 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:42:20.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:20.338 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:20.338 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:20.346 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:42:20.347 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:20.348 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:20.349 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:20.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:20.367 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:20.367 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:20.376 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:42:20.377 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:42:20.378 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-21T01:42:20.379 DEBUG:teuthology.orchestra.run.smithi171:> import time 2022-07-21T01:42:20.379 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:42:20.379 DEBUG:teuthology.orchestra.run.smithi171:> with open("/home/ubuntu/cephtest/mnt.0/testfile", '"'"'w'"'"') as f: 2022-07-21T01:42:20.380 DEBUG:teuthology.orchestra.run.smithi171:> f.write('"'"'content'"'"') 2022-07-21T01:42:20.380 DEBUG:teuthology.orchestra.run.smithi171:> f.flush() 2022-07-21T01:42:20.381 DEBUG:teuthology.orchestra.run.smithi171:> f.write('"'"'content2'"'"') 2022-07-21T01:42:20.381 DEBUG:teuthology.orchestra.run.smithi171:> while True: 2022-07-21T01:42:20.382 DEBUG:teuthology.orchestra.run.smithi171:> time.sleep(1) 2022-07-21T01:42:20.382 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:42:20.423 DEBUG:teuthology.orchestra.run.smithi171:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-07-21T01:42:20.445 INFO:teuthology.orchestra.run.smithi171.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/testfile': No such file or directory 2022-07-21T01:42:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-07-21T01:42:21.450 DEBUG:teuthology.orchestra.run.smithi171:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-07-21T01:42:21.468 INFO:teuthology.orchestra.run.smithi171.stdout:/home/ubuntu/cephtest/mnt.0/testfile 2022-07-21T01:42:21.469 DEBUG:tasks.cephfs.mount:File testfile became visible from 0 after 1s 2022-07-21T01:42:21.470 DEBUG:teuthology.orchestra.run.smithi174:> sudo ls /home/ubuntu/cephtest/mnt.1/testfile 2022-07-21T01:42:21.511 INFO:teuthology.orchestra.run.smithi174.stdout:/home/ubuntu/cephtest/mnt.1/testfile 2022-07-21T01:42:21.512 DEBUG:tasks.cephfs.mount:File testfile became visible from 1 after 0s 2022-07-21T01:42:21.513 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-21T01:42:21.513 DEBUG:teuthology.orchestra.run.smithi174:> cd /home/ubuntu/cephtest/mnt.1 && sudo cat testfile 2022-07-21T01:42:21.557 INFO:teuthology.orchestra.run.smithi174.stdout:content 2022-07-21T01:42:21.574 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:42:21.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:21.594 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:21.618 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:21.641 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:21.663 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:42:21.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:21.665 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:21.665 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:21.691 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:42:21.693 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:42:21.693 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:42:21.694 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:42:21.694 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:21.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23330]: fuse finished with error 0 and tester_r 0 2022-07-21T01:42:21.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:21.721 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:21.722 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:21.774 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:42:21.776 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:42:21.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:42:21.893 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:21.902 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:21.902 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:21.903 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:21.911 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:42:21.912 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:21.913 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:42:21.913 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:42:21.914 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:21.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[22961]: fuse finished with error 0 and tester_r 0 2022-07-21T01:42:21.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:21.932 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:21.932 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:21.939 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:42:21.941 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:42:21.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:42:21.967 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:22.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:42:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97cdb88700 1 Processor -- start 2022-07-21T01:42:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97cdb88700 1 -- start start 2022-07-21T01:42:22.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97cdb88700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f97c80b26f0 0x7f97c81405c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97cdb88700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c80b2ad0 con 0x7f97c80a8f60 2022-07-21T01:42:22.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97cdb88700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c80ab390 con 0x7f97c805ffd0 2022-07-21T01:42:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97cdb88700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c8146a80 con 0x7f97c80b26f0 2022-07-21T01:42:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f97c80b26f0 0x7f97c81405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f97c80b26f0 0x7f97c81405c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50978/0 (socket says 172.21.15.3:50978) 2022-07-21T01:42:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c6ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f97c80a8f60 0x7f97c8140ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59262/0 (socket says 172.21.15.3:59262) 2022-07-21T01:42:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c77fe700 1 -- 172.21.15.3:0/3950770056 learned_addr learned my addr 172.21.15.3:0/3950770056 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:22.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c67fc700 1 -- 172.21.15.3:0/3950770056 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046576055 0 0) 0x7f97c80b2ad0 con 0x7f97c80a8f60 2022-07-21T01:42:22.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c67fc700 1 -- 172.21.15.3:0/3950770056 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ac003260 con 0x7f97c80a8f60 2022-07-21T01:42:22.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c67fc700 1 -- 172.21.15.3:0/3950770056 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966532178 0 0) 0x7f97c80ab390 con 0x7f97c805ffd0 2022-07-21T01:42:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c67fc700 1 -- 172.21.15.3:0/3950770056 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c80b2ad0 con 0x7f97c805ffd0 2022-07-21T01:42:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c67fc700 1 -- 172.21.15.3:0/3950770056 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1302206737 0 0) 0x7f97ac003260 con 0x7f97c80a8f60 2022-07-21T01:42:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c77fe700 1 -- 172.21.15.3:0/3950770056 >> v1:172.21.15.143:6789/0 conn(0x7f97c805ffd0 legacy=0x7f97c8143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c77fe700 1 -- 172.21.15.3:0/3950770056 >> v1:172.21.15.3:6789/0 conn(0x7f97c80a8f60 legacy=0x7f97c8140ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c77fe700 1 -- 172.21.15.3:0/3950770056 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c8146d20 con 0x7f97c80b26f0 2022-07-21T01:42:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c77fe700 1 --2- 172.21.15.3:0/3950770056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80b26f0 0x7f97c81405c0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f97b0001110 tx=0x7f97b8009290).ready entity=mon.0 client_cookie=c1e537bb57ca246e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c67fc700 1 -- 172.21.15.3:0/3950770056 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97b0001660 con 0x7f97c80b26f0 2022-07-21T01:42:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c67fc700 1 -- 172.21.15.3:0/3950770056 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b800b580 con 0x7f97c80b26f0 2022-07-21T01:42:22.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.127+0000 7f97c67fc700 1 -- 172.21.15.3:0/3950770056 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97b800b6f0 con 0x7f97c80b26f0 2022-07-21T01:42:22.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.131+0000 7f97cdb88700 1 -- 172.21.15.3:0/3950770056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80b26f0 msgr2=0x7f97c81405c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.131+0000 7f97cdb88700 1 --2- 172.21.15.3:0/3950770056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80b26f0 0x7f97c81405c0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f97b0001110 tx=0x7f97b8009290).stop 2022-07-21T01:42:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.131+0000 7f97cdb88700 1 -- 172.21.15.3:0/3950770056 shutdown_connections 2022-07-21T01:42:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.131+0000 7f97cdb88700 1 --2- 172.21.15.3:0/3950770056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80b26f0 0x7f97c81405c0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.131+0000 7f97cdb88700 1 -- 172.21.15.3:0/3950770056 >> 172.21.15.3:0/3950770056 conn(0x7f97c805cbb0 msgr2=0x7f97c805cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.131+0000 7f97cdb88700 1 -- 172.21.15.3:0/3950770056 shutdown_connections 2022-07-21T01:42:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.131+0000 7f97cdb88700 1 -- 172.21.15.3:0/3950770056 wait complete. 2022-07-21T01:42:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.131+0000 7f97cdb88700 1 Processor -- start 2022-07-21T01:42:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.131+0000 7f97cdb88700 1 -- start start 2022-07-21T01:42:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97cdb88700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c805ffd0 0x7f97c80aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97cdb88700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80a8f60 0x7f97c813e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97cdb88700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c80b26f0 0x7f97c80ad460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97cdb88700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c81532d0 con 0x7f97c80a8f60 2022-07-21T01:42:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97cdb88700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c8153570 con 0x7f97c80b26f0 2022-07-21T01:42:22.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97cdb88700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c80b33e0 con 0x7f97c805ffd0 2022-07-21T01:42:22.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80a8f60 0x7f97c813e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80a8f60 0x7f97c813e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50986/0 (socket says 172.21.15.3:50986) 2022-07-21T01:42:22.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c6ffd700 1 -- 172.21.15.3:0/2694602661 learned_addr learned my addr 172.21.15.3:0/2694602661 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:22.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c77fe700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c805ffd0 0x7f97c80aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c7fff700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c80b26f0 0x7f97c80ad460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c6ffd700 1 -- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c805ffd0 msgr2=0x7f97c80aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c6ffd700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c805ffd0 0x7f97c80aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c6ffd700 1 -- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c80b26f0 msgr2=0x7f97c80ad460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c6ffd700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c80b26f0 0x7f97c80ad460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c6ffd700 1 -- 172.21.15.3:0/2694602661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b8009530 con 0x7f97c80a8f60 2022-07-21T01:42:22.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c77fe700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c805ffd0 0x7f97c80aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:22.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c7fff700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c80b26f0 0x7f97c80ad460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c6ffd700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80a8f60 0x7f97c813e0c0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f97b0001840 tx=0x7f97b0001ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c4ff9700 1 -- 172.21.15.3:0/2694602661 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97b00072e0 con 0x7f97c80a8f60 2022-07-21T01:42:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c4ff9700 1 -- 172.21.15.3:0/2694602661 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b0005550 con 0x7f97c80a8f60 2022-07-21T01:42:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c80b3680 con 0x7f97c80a8f60 2022-07-21T01:42:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97c4ff9700 1 -- 172.21.15.3:0/2694602661 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97b00058b0 con 0x7f97c80a8f60 2022-07-21T01:42:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c80ad950 con 0x7f97c80a8f60 2022-07-21T01:42:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.135+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c80b3810 con 0x7f97c80a8f60 2022-07-21T01:42:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.139+0000 7f97c4ff9700 1 -- 172.21.15.3:0/2694602661 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f97b0010070 con 0x7f97c80a8f60 2022-07-21T01:42:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.139+0000 7f97c4ff9700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97a0053aa0 0x7f97a0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.139+0000 7f97c4ff9700 1 -- 172.21.15.3:0/2694602661 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f97b006ba20 con 0x7f97c80a8f60 2022-07-21T01:42:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.139+0000 7f97c77fe700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97a0053aa0 0x7f97a0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.139+0000 7f97c77fe700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97a0053aa0 0x7f97a0055f60 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f97b800b000 tx=0x7f97b8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.143+0000 7f97c4ff9700 1 -- 172.21.15.3:0/2694602661 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f97b00427e0 con 0x7f97c80a8f60 2022-07-21T01:42:22.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.267+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f97c8146fc0 con 0x7f97c80a8f60 2022-07-21T01:42:22.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.267+0000 7f97c4ff9700 1 -- 172.21.15.3:0/2694602661 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v140) v1 ==== 81+0+34051 (secure 0 0 0) 0x7f97b0018070 con 0x7f97c80a8f60 2022-07-21T01:42:22.272 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:22.272 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:42:22.273 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 140, 2022-07-21T01:42:22.273 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:42:22.273 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:42:22.274 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:42:08.066592+0000", 2022-07-21T01:42:22.274 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:22.274 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:22.274 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:42:22.275 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:42:22.276 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:42:22.276 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:42:22.276 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:42:22.277 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:42:22.277 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:42:22.277 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:22.277 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:42:22.278 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:42:22.278 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:42:22.278 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:42:22.278 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:42:22.279 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 26, 2022-07-21T01:42:22.279 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:42:22.279 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:42:22.279 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:42:22.280 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:42:22.280 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:42:22.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.281 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:42:22.281 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:42:22.281 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:42:22.281 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:22.282 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:22.282 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:22.282 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:22.283 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:22.283 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:22.283 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:22.283 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:22.284 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:42:22.284 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:42:22.284 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:42:22.284 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:42:22.285 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:42:22.285 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:22.285 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:22.286 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:22.286 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:22.287 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:22.287 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:22.287 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:22.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.288 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:42:22.288 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:22.288 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:22.289 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:22.289 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:22.289 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:22.290 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:22.290 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:22.290 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:22.291 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:22.291 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:22.291 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:22.291 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:22.292 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:22.292 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:22.292 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:22.293 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:22.293 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:22.293 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:22.293 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:22.294 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:22.295 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:22.295 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:22.296 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:22.297 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:22.297 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:22.298 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:22.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.299 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:22.299 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:22.300 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:22.300 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:22.301 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:22.301 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:22.301 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:22.302 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:22.302 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:22.303 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:22.303 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:22.304 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:42:22.304 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:42:22.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.305 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:22.305 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:42:22.306 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.308 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:42:22.308 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:42:22.309 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:42:22.309 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:22.310 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:22.310 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:22.311 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:22.311 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:22.312 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:22.312 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:22.313 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:22.313 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:42:22.313 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:42:22.314 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:42:22.314 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:42:22.315 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:42:22.315 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:22.316 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:22.316 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:22.317 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:22.317 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:22.317 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:22.318 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:22.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.319 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:42:22.319 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:22.320 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:22.320 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:22.321 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:22.321 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:22.321 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:22.322 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:22.322 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:22.323 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:22.323 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:22.324 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:22.324 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:22.325 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:22.325 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:22.326 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:22.326 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:22.326 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:22.327 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:22.327 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:22.328 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:22.328 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:22.329 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:22.329 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:22.329 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:22.330 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:22.330 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:22.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.331 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:22.331 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:22.332 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:22.332 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:22.332 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:22.332 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:22.332 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:22.333 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:22.333 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:22.333 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:22.334 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:22.334 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:42:22.334 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:22.334 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:42:22.335 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.336 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 25, 2022-07-21T01:42:22.336 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:42:22.336 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:42:05.096631+0000", 2022-07-21T01:42:22.336 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:22.337 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:22.337 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:22.337 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:22.337 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:22.337 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:22.338 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:22.338 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:22.338 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:42:22.339 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:42:22.339 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:42:22.339 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:42:22.339 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:42:22.340 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:22.340 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:22.340 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:22.340 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:22.341 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:22.341 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:22.341 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:22.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.342 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "139", 2022-07-21T01:42:22.342 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:22.342 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:22.342 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:22.343 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:22.344 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:22.345 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:22.347 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:22.348 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:22.349 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:22.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:22.351 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:22.352 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:22.353 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:22.354 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:22.355 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:22.356 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:42:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:42:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:42:22.357 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:42:22.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.358 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:42:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:42:22.359 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.360 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.361 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 26, 2022-07-21T01:42:22.361 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:42:22.361 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:42:06.060734+0000", 2022-07-21T01:42:22.361 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:22.362 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:22.362 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:22.362 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:22.362 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:22.363 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:22.363 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:22.363 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:22.363 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:42:22.364 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:42:22.364 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:42:22.364 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:42:22.364 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:42:22.365 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:22.365 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:22.365 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:22.365 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:22.366 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:22.366 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:22.366 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:22.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.366 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "140", 2022-07-21T01:42:22.367 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:22.367 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:22.367 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:22.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:22.368 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:22.368 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:22.368 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:22.368 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:22.368 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:22.369 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:22.369 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:22.369 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:22.369 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:22.369 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:22.370 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:22.370 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:22.370 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:22.370 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:22.371 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:22.371 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:22.371 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:22.371 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:22.371 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:22.372 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:22.372 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:22.372 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:22.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.373 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:22.373 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:22.373 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:22.373 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:22.373 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:22.374 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:22.374 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:22.374 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:22.374 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:22.374 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:22.375 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:22.375 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:42:22.375 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:22.375 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:42:22.376 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:42:22.376 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.376 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.376 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.376 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:22.377 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:42:22.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.377 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:22.377 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:42:22.378 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:22.378 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:22.378 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:22.378 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:22.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:22.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:22.379 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:22.379 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 137, 2022-07-21T01:42:22.379 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:22.380 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:22.380 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:22.380 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.381 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.381 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:42:22.381 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:22.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.382 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.382 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:42:22.382 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:22.382 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.383 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.383 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:22.383 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.384 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:42:22.384 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:22.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.385 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.385 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:42:22.385 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:22.386 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.386 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.387 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:22.387 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.387 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.388 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:42:22.388 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:22.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.389 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.389 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:42:22.389 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:22.390 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.390 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.390 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:22.391 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.391 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:42:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.392 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:42:22.393 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:22.393 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.393 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.394 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:42:22.394 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:42:22.394 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:42:22.394 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:42:22.394 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:22.395 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:22.395 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:22.395 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:42:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:22.396 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:22.397 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:22.397 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:22.397 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:22.397 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:22.397 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:22.398 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:22.398 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:22.398 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:22.398 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:42:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:42:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:22.402 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.402 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:42:22.402 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:22.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.403 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:42:22.404 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:22.404 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.404 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.404 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:22.405 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.405 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.405 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:42:22.405 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:42:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:22.408 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.408 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.408 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:42:22.408 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:22.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.409 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:42:22.409 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:22.410 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.410 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.410 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:42:22.411 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:42:22.411 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:42:22.411 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:42:22.411 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:22.411 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:22.412 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:22.412 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.412 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:22.413 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:42:22.413 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:22.413 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:22.413 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:22.414 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:22.414 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:22.414 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:22.414 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:22.414 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:22.415 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:22.415 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:22.415 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:22.415 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.416 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:42:22.416 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:22.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.417 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.417 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:42:22.417 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:22.418 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.418 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.418 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:22.419 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.419 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.419 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:42:22.420 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:22.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.420 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.421 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:42:22.421 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:22.421 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.421 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.422 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:22.422 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.423 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:42:22.423 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:22.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.424 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:42:22.424 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:22.424 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.425 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.425 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:22.425 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.426 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.426 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:42:22.426 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:22.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.427 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.427 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:42:22.427 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:22.427 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:42:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:42:22.428 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:42:22.429 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:42:22.429 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:22.429 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:22.429 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:22.429 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.430 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:22.430 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:42:22.431 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:22.431 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:22.431 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:22.431 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:22.431 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 137, 2022-07-21T01:42:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:22.432 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:22.433 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:22.433 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:42:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.434 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.435 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:42:22.435 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:22.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.435 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.437 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:42:22.437 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:22.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.438 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:42:22.438 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:22.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.438 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.439 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:22.439 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.439 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:42:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:42:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.441 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:42:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:42:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.444 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:42:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:42:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:42:22.445 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:42:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:22.446 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:42:22.447 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:22.448 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:22.449 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.450 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:42:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.451 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:42:22.452 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:22.452 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:42:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:22.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:42:22.455 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:22.456 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.457 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:42:22.457 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:22.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.458 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.458 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:42:22.459 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:22.459 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.459 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.460 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:42:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.461 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.462 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:42:22.462 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:22.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.462 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.463 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:42:22.463 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:42:22.463 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:42:22.464 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:42:22.464 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:22.464 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:22.464 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:22.465 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:42:22.466 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:22.466 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:22.466 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:22.466 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:22.467 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:22.467 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:22.467 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:22.467 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:42:22.468 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:22.468 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:22.468 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:22.468 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:42:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:22.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.470 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.470 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:42:22.470 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:22.471 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.471 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.471 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:22.471 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:42:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:22.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.473 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:42:22.473 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:22.474 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.474 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.474 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:22.474 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:42:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:42:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:42:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:42:22.479 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:22.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.480 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.480 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:42:22.480 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:42:22.481 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:42:22.481 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:42:22.481 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:22.481 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:22.482 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:22.482 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:42:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:22.483 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:22.484 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:22.484 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:22.484 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:22.484 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.486 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:42:22.486 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:22.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.487 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.487 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:42:22.487 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:22.488 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.488 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.488 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:22.488 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:42:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:22.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.490 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:42:22.490 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:22.491 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.491 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.491 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:22.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.492 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:42:22.492 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:22.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.493 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:42:22.493 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:22.494 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.494 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.494 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:22.495 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.495 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.495 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:42:22.495 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:42:22.496 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:22.497 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.497 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.497 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:42:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:42:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:42:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:42:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:22.498 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:22.499 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:22.499 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:42:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:22.500 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:22.501 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:22.502 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:22.502 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:22.502 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:22.502 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:42:22.503 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:22.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.504 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:42:22.504 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:22.505 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.505 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.506 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:42:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:42:22.508 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:22.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.508 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.509 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:22.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.510 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.510 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:42:22.510 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:22.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.511 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.511 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:42:22.511 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:22.512 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.512 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.512 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:22.512 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:22.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.513 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:22.513 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:42:22.513 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:22.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.514 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:22.514 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:42:22.514 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:22.515 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.515 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.515 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:42:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:42:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:42:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:42:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:22.516 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:22.517 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:22.517 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:22.517 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.517 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:22.517 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:42:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:22.518 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:22.519 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:22.519 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:22.519 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:42:22.519 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:22.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.520 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:22.520 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:22.520 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:22.521 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:22.521 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:22.521 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:22.521 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:42:22.521 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:22.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.522 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:22.522 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:22.523 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:42:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:22.524 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:22.525 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:22.525 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:22.525 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:22.525 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:42:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.526 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:22.527 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:22.528 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:42:22.528 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:22.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.529 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:22.529 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:22.529 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:22.529 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:22.529 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:22.530 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:22.530 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:42:22.530 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:22.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.531 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:22.531 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:22.531 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:22.531 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:22.532 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:22.532 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:22.532 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:42:22.532 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:22.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:22.533 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:22.533 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:22.533 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:22.534 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:22.534 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:22.534 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:22.534 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:42:22.534 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:22.535 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.535 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:22.535 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:42:22.535 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:42:22.535 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:42:22.536 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:42:22.536 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:42:22.536 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:42:22.536 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:42:22.537 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:42:22.537 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:22.537 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:42:22.537 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:42:22.537 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:22.538 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:42:22.538 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:22.538 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:22.538 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:42:22.538 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:42:22.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.539 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:42:22.539 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:22.539 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:22.539 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:42:22.540 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:22.540 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:22.540 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:42:22.540 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:42:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:42:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:42:22.541 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:42:22.542 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:42:22.542 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:42:22.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97a0053aa0 msgr2=0x7f97a0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97a0053aa0 0x7f97a0055f60 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f97b800b000 tx=0x7f97b8016000).stop 2022-07-21T01:42:22.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80a8f60 msgr2=0x7f97c813e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80a8f60 0x7f97c813e0c0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f97b0001840 tx=0x7f97b0001ab0).stop 2022-07-21T01:42:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 shutdown_connections 2022-07-21T01:42:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f97a0053aa0 0x7f97a0055f60 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c80b26f0 0x7f97c80ad460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c80a8f60 0x7f97c813e0c0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 --2- 172.21.15.3:0/2694602661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c805ffd0 0x7f97c80aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 >> 172.21.15.3:0/2694602661 conn(0x7f97c805cbb0 msgr2=0x7f97c8143b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:22.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 shutdown_connections 2022-07-21T01:42:22.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.271+0000 7f97cdb88700 1 -- 172.21.15.3:0/2694602661 wait complete. 2022-07-21T01:42:22.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:22.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bec82700 1 Processor -- start 2022-07-21T01:42:22.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bec82700 1 -- start start 2022-07-21T01:42:22.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bec82700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72b80b12e0 0x7f72b80abe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bec82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f72b813fda0 0x7f72b81401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bec82700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b80af3c0 con 0x7f72b80b21a0 2022-07-21T01:42:22.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bec82700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b80ae900 con 0x7f72b80b12e0 2022-07-21T01:42:22.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bec82700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b80b16c0 con 0x7f72b813fda0 2022-07-21T01:42:22.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bdc80700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72b80b12e0 0x7f72b80abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bdc80700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f72b80b12e0 0x7f72b80abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50992/0 (socket says 172.21.15.3:50992) 2022-07-21T01:42:22.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72bdc80700 1 -- 172.21.15.3:0/135024512 learned_addr learned my addr 172.21.15.3:0/135024512 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:22.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.659+0000 7f72be481700 1 --2- 172.21.15.3:0/135024512 >> v2:172.21.15.143:3300/0 conn(0x7f72b813fda0 0x7f72b81401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bdc80700 1 -- 172.21.15.3:0/135024512 >> v2:172.21.15.143:3300/0 conn(0x7f72b813fda0 msgr2=0x7f72b81401f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:22.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bdc80700 1 --2- 172.21.15.3:0/135024512 >> v2:172.21.15.143:3300/0 conn(0x7f72b813fda0 0x7f72b81401f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bdc80700 1 -- 172.21.15.3:0/135024512 >> v1:172.21.15.143:6789/0 conn(0x7f72b80b21a0 legacy=0x7f72b8067660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bdc80700 1 -- 172.21.15.3:0/135024512 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b8143f00 con 0x7f72b80b12e0 2022-07-21T01:42:22.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bdc80700 1 --2- 172.21.15.3:0/135024512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 0x7f72b80abe60 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f72b4002f70 tx=0x7f72b4006eb0).ready entity=mon.0 client_cookie=2e40256231aca711 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:22.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bcc7e700 1 -- 172.21.15.3:0/135024512 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72b400d5c0 con 0x7f72b80b12e0 2022-07-21T01:42:22.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bcc7e700 1 -- 172.21.15.3:0/135024512 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b4000e90 con 0x7f72b80b12e0 2022-07-21T01:42:22.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bcc7e700 1 -- 172.21.15.3:0/135024512 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72b400d950 con 0x7f72b80b12e0 2022-07-21T01:42:22.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bec82700 1 -- 172.21.15.3:0/135024512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 msgr2=0x7f72b80abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bec82700 1 --2- 172.21.15.3:0/135024512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 0x7f72b80abe60 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f72b4002f70 tx=0x7f72b4006eb0).stop 2022-07-21T01:42:22.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bec82700 1 -- 172.21.15.3:0/135024512 shutdown_connections 2022-07-21T01:42:22.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bec82700 1 --2- 172.21.15.3:0/135024512 >> v2:172.21.15.143:3300/0 conn(0x7f72b813fda0 0x7f72b81401f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bec82700 1 --2- 172.21.15.3:0/135024512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 0x7f72b80abe60 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bec82700 1 -- 172.21.15.3:0/135024512 >> 172.21.15.3:0/135024512 conn(0x7f72b805bed0 msgr2=0x7f72b80626f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:22.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bec82700 1 -- 172.21.15.3:0/135024512 shutdown_connections 2022-07-21T01:42:22.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.663+0000 7f72bec82700 1 -- 172.21.15.3:0/135024512 wait complete. 2022-07-21T01:42:22.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bec82700 1 Processor -- start 2022-07-21T01:42:22.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bec82700 1 -- start start 2022-07-21T01:42:22.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bec82700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 0x7f72b813b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bec82700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b80b21a0 0x7f72b813b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bec82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b813fda0 0x7f72b80b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bec82700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80b4600 con 0x7f72b80b12e0 2022-07-21T01:42:22.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bec82700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80b6290 con 0x7f72b80b21a0 2022-07-21T01:42:22.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bec82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b813d520 con 0x7f72b813fda0 2022-07-21T01:42:22.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bdc80700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 0x7f72b813b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bdc80700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 0x7f72b813b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50996/0 (socket says 172.21.15.3:50996) 2022-07-21T01:42:22.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bdc80700 1 -- 172.21.15.3:0/249023329 learned_addr learned my addr 172.21.15.3:0/249023329 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:22.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bd47f700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b80b21a0 0x7f72b813b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72be481700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b813fda0 0x7f72b80b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bdc80700 1 -- 172.21.15.3:0/249023329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b813fda0 msgr2=0x7f72b80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bdc80700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b813fda0 0x7f72b80b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bdc80700 1 -- 172.21.15.3:0/249023329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b80b21a0 msgr2=0x7f72b813b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bdc80700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b80b21a0 0x7f72b813b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bdc80700 1 -- 172.21.15.3:0/249023329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b400b1e0 con 0x7f72b80b12e0 2022-07-21T01:42:22.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.667+0000 7f72bdc80700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 0x7f72b813b4a0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f72b4002f70 tx=0x7f72b4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:22.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72a6ffd700 1 -- 172.21.15.3:0/249023329 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72b4011990 con 0x7f72b80b12e0 2022-07-21T01:42:22.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72a6ffd700 1 -- 172.21.15.3:0/249023329 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b4011af0 con 0x7f72b80b12e0 2022-07-21T01:42:22.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b80b47f0 con 0x7f72b80b12e0 2022-07-21T01:42:22.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72a6ffd700 1 -- 172.21.15.3:0/249023329 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72b4011e50 con 0x7f72b80b12e0 2022-07-21T01:42:22.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b813be80 con 0x7f72b80b12e0 2022-07-21T01:42:22.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72a6ffd700 1 -- 172.21.15.3:0/249023329 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f72b401a040 con 0x7f72b80b12e0 2022-07-21T01:42:22.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b8063380 con 0x7f72b80b12e0 2022-07-21T01:42:22.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72a6ffd700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7298053980 0x7f7298055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72a6ffd700 1 -- 172.21.15.3:0/249023329 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f72b406cf70 con 0x7f72b80b12e0 2022-07-21T01:42:22.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72bd47f700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7298053980 0x7f7298055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.671+0000 7f72bd47f700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7298053980 0x7f7298055e40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f72a800b000 tx=0x7f72a8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:22.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.679+0000 7f72a6ffd700 1 -- 172.21.15.3:0/249023329 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f72b8063380 con 0x7f72b80b12e0 2022-07-21T01:42:22.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.839+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72b8063380 con 0x7f72b80b12e0 2022-07-21T01:42:22.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.843+0000 7f72a6ffd700 1 -- 172.21.15.3:0/249023329 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2483 (secure 0 0 0) 0x7f72b8063380 con 0x7f72b80b12e0 2022-07-21T01:42:22.845 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:22.846 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":71,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/567682837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":567682837},{"type":"v1","addr":"172.21.15.143:6833","nonce":567682837}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/140242740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140242740},{"type":"v1","addr":"172.21.15.3:6835","nonce":140242740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:07.063040+0000","modified":"2022-07-21T01:42:08.065747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8049":{"gid":8049,"name":"a","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/506442168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":506442168},{"type":"v1","addr":"172.21.15.3:6837","nonce":506442168}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-21T01:42:22.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.843+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7298053980 msgr2=0x7f7298055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.843+0000 7f72bec82700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7298053980 0x7f7298055e40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f72a800b000 tx=0x7f72a8009040).stop 2022-07-21T01:42:22.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.843+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 msgr2=0x7f72b813b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.843+0000 7f72bec82700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 0x7f72b813b4a0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f72b4002f70 tx=0x7f72b4015000).stop 2022-07-21T01:42:22.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.847+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 shutdown_connections 2022-07-21T01:42:22.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.847+0000 7f72bec82700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b813fda0 0x7f72b80b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.847+0000 7f72bec82700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7298053980 0x7f7298055e40 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.847+0000 7f72bec82700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b80b21a0 0x7f72b813b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.847+0000 7f72bec82700 1 --2- 172.21.15.3:0/249023329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72b80b12e0 0x7f72b813b4a0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.847+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 >> 172.21.15.3:0/249023329 conn(0x7f72b805bed0 msgr2=0x7f72b813e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:22.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.847+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 shutdown_connections 2022-07-21T01:42:22.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.847+0000 7f72bec82700 1 -- 172.21.15.3:0/249023329 wait complete. 2022-07-21T01:42:22.853 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 71 2022-07-21T01:42:22.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:42:22.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.963+0000 7f5515b5a700 1 Processor -- start 2022-07-21T01:42:22.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.963+0000 7f5515b5a700 1 -- start start 2022-07-21T01:42:22.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.963+0000 7f5515b5a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f551005cbf0 0x7f5510142890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.963+0000 7f5515b5a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5510142d80 0x7f55100b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.963+0000 7f5515b5a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55100603c0 con 0x7f55100b1670 2022-07-21T01:42:22.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.963+0000 7f5515b5a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551005fb50 con 0x7f5510142d80 2022-07-21T01:42:22.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.963+0000 7f5515b5a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551005cfd0 con 0x7f551005cbf0 2022-07-21T01:42:22.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.963+0000 7f550ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5510142d80 0x7f55100b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.963+0000 7f550ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5510142d80 0x7f55100b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51006/0 (socket says 172.21.15.3:51006) 2022-07-21T01:42:22.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515359700 1 --1- 172.21.15.3:0/1888339233 >> v1:172.21.15.143:6789/0 conn(0x7f55100b1670 0x7f55100b6530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:33172/0 (socket says 172.21.15.3:33172) 2022-07-21T01:42:22.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550ffff700 1 -- 172.21.15.3:0/1888339233 learned_addr learned my addr 172.21.15.3:0/1888339233 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:22.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5514b58700 1 --2- 172.21.15.3:0/1888339233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f551005cbf0 0x7f5510142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550ffff700 1 -- 172.21.15.3:0/1888339233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f551005cbf0 msgr2=0x7f5510142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:22.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550ffff700 1 --2- 172.21.15.3:0/1888339233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f551005cbf0 0x7f5510142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550ffff700 1 -- 172.21.15.3:0/1888339233 >> v1:172.21.15.143:6789/0 conn(0x7f55100b1670 legacy=0x7f55100b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550f7fe700 1 -- 172.21.15.3:0/1888339233 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783312451 0 0) 0x7f55100603c0 con 0x7f55100b1670 2022-07-21T01:42:22.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550ffff700 1 -- 172.21.15.3:0/1888339233 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100aecf0 con 0x7f5510142d80 2022-07-21T01:42:22.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550ffff700 1 --2- 172.21.15.3:0/1888339233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5510142d80 0x7f55100b1180 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f54fc002f70 tx=0x7f54fc006eb0).ready entity=mon.0 client_cookie=ac27fbbeb963fe58 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:22.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550f7fe700 1 -- 172.21.15.3:0/1888339233 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54fc00d570 con 0x7f5510142d80 2022-07-21T01:42:22.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550f7fe700 1 -- 172.21.15.3:0/1888339233 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54fc000e90 con 0x7f5510142d80 2022-07-21T01:42:22.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f550f7fe700 1 -- 172.21.15.3:0/1888339233 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54fc00d8c0 con 0x7f5510142d80 2022-07-21T01:42:22.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515b5a700 1 -- 172.21.15.3:0/1888339233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5510142d80 msgr2=0x7f55100b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515b5a700 1 --2- 172.21.15.3:0/1888339233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5510142d80 0x7f55100b1180 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f54fc002f70 tx=0x7f54fc006eb0).stop 2022-07-21T01:42:22.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515b5a700 1 -- 172.21.15.3:0/1888339233 shutdown_connections 2022-07-21T01:42:22.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515b5a700 1 --2- 172.21.15.3:0/1888339233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5510142d80 0x7f55100b1180 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515b5a700 1 --2- 172.21.15.3:0/1888339233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f551005cbf0 0x7f5510142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515b5a700 1 -- 172.21.15.3:0/1888339233 >> 172.21.15.3:0/1888339233 conn(0x7f5510063800 msgr2=0x7f5510063c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:22.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515b5a700 1 -- 172.21.15.3:0/1888339233 shutdown_connections 2022-07-21T01:42:22.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515b5a700 1 -- 172.21.15.3:0/1888339233 wait complete. 2022-07-21T01:42:22.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.967+0000 7f5515b5a700 1 Processor -- start 2022-07-21T01:42:22.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 -- start start 2022-07-21T01:42:22.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f551005cbf0 0x7f55101531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55100b1670 0x7f5510148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5510142d80 0x7f5510140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5510064a10 con 0x7f551005cbf0 2022-07-21T01:42:22.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5510152a90 con 0x7f55100b1670 2022-07-21T01:42:22.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5510152d30 con 0x7f5510142d80 2022-07-21T01:42:22.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5514b58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f551005cbf0 0x7f55101531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5514b58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f551005cbf0 0x7f55101531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51010/0 (socket says 172.21.15.3:51010) 2022-07-21T01:42:22.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5514b58700 1 -- 172.21.15.3:0/769031616 learned_addr learned my addr 172.21.15.3:0/769031616 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:22.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515359700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5510142d80 0x7f5510140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f550ffff700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55100b1670 0x7f5510148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5514b58700 1 -- 172.21.15.3:0/769031616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5510142d80 msgr2=0x7f5510140340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5514b58700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5510142d80 0x7f5510140340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5514b58700 1 -- 172.21.15.3:0/769031616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55100b1670 msgr2=0x7f5510148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5514b58700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55100b1670 0x7f5510148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5514b58700 1 -- 172.21.15.3:0/769031616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc00b1e0 con 0x7f551005cbf0 2022-07-21T01:42:22.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f550ffff700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55100b1670 0x7f5510148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:22.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5514b58700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f551005cbf0 0x7f55101531d0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f550400c000 tx=0x7f5504006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:22.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f550dffb700 1 -- 172.21.15.3:0/769031616 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5504010780 con 0x7f551005cbf0 2022-07-21T01:42:22.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f550dffb700 1 -- 172.21.15.3:0/769031616 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55040108e0 con 0x7f551005cbf0 2022-07-21T01:42:22.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5510148780 con 0x7f551005cbf0 2022-07-21T01:42:22.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f550dffb700 1 -- 172.21.15.3:0/769031616 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5504010c40 con 0x7f551005cbf0 2022-07-21T01:42:22.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55100b5410 con 0x7f551005cbf0 2022-07-21T01:42:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.971+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55100b2270 con 0x7f551005cbf0 2022-07-21T01:42:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.975+0000 7f550dffb700 1 -- 172.21.15.3:0/769031616 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f550401c460 con 0x7f551005cbf0 2022-07-21T01:42:22.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.975+0000 7f550dffb700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54f0053b00 0x7f54f0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.975+0000 7f550dffb700 1 -- 172.21.15.3:0/769031616 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f550406c450 con 0x7f551005cbf0 2022-07-21T01:42:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.975+0000 7f550ffff700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54f0053b00 0x7f54f0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:22.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.975+0000 7f550ffff700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54f0053b00 0x7f54f0055fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f54fc01a000 tx=0x7f54fc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:22.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:22.983+0000 7f550dffb700 1 -- 172.21.15.3:0/769031616 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f55040431b0 con 0x7f551005cbf0 2022-07-21T01:42:23.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.143+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f55100574e0 con 0x7f551005cbf0 2022-07-21T01:42:23.593 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:42:23.591+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:42:23.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.599+0000 7f550dffb700 1 -- 172.21.15.3:0/769031616 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v72) v1 ==== 161+0+0 (secure 0 0 0) 0x7f55040467d0 con 0x7f551005cbf0 2022-07-21T01:42:23.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54f0053b00 msgr2=0x7f54f0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:23.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54f0053b00 0x7f54f0055fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f54fc01a000 tx=0x7f54fc015000).stop 2022-07-21T01:42:23.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f551005cbf0 msgr2=0x7f55101531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:23.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f551005cbf0 0x7f55101531d0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f550400c000 tx=0x7f5504006df0).stop 2022-07-21T01:42:23.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 shutdown_connections 2022-07-21T01:42:23.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5510142d80 0x7f5510140340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f54f0053b00 0x7f54f0055fc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55100b1670 0x7f5510148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 --2- 172.21.15.3:0/769031616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f551005cbf0 0x7f55101531d0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 >> 172.21.15.3:0/769031616 conn(0x7f5510063800 msgr2=0x7f55100ae170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:23.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 shutdown_connections 2022-07-21T01:42:23.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.603+0000 7f5515b5a700 1 -- 172.21.15.3:0/769031616 wait complete. 2022-07-21T01:42:23.610 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:42:23.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:23.634 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:42:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.723+0000 7f34b32ae700 1 Processor -- start 2022-07-21T01:42:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.723+0000 7f34b32ae700 1 -- start start 2022-07-21T01:42:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b32ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac0b68f0 0x7f34ac1430b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b32ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f34ac1435a0 0x7f34ac0aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b32ae700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac146800 con 0x7f34ac0b6510 2022-07-21T01:42:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b32ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac146aa0 con 0x7f34ac1435a0 2022-07-21T01:42:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b32ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac146d40 con 0x7f34ac0b68f0 2022-07-21T01:42:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b22ac700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f34ac0b6510 0x7f34ac05ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59310/0 (socket says 172.21.15.3:59310) 2022-07-21T01:42:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b22ac700 1 -- 172.21.15.3:0/2497147166 learned_addr learned my addr 172.21.15.3:0/2497147166 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b2aad700 1 --2- 172.21.15.3:0/2497147166 >> v2:172.21.15.143:3300/0 conn(0x7f34ac1435a0 0x7f34ac0aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b1aab700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac0b68f0 0x7f34ac1430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616219215 0 0) 0x7f34ac146800 con 0x7f34ac0b6510 2022-07-21T01:42:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3494003260 con 0x7f34ac0b6510 2022-07-21T01:42:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740180195 0 0) 0x7f3494003260 con 0x7f34ac0b6510 2022-07-21T01:42:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac0b68f0 msgr2=0x7f34ac1430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac0b68f0 0x7f34ac1430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 >> v2:172.21.15.143:3300/0 conn(0x7f34ac1435a0 msgr2=0x7f34ac0aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 --2- 172.21.15.3:0/2497147166 >> v2:172.21.15.143:3300/0 conn(0x7f34ac1435a0 0x7f34ac0aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac146fe0 con 0x7f34ac0b6510 2022-07-21T01:42:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f34a8001660 con 0x7f34ac0b6510 2022-07-21T01:42:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 >> v1:172.21.15.3:6789/0 conn(0x7f34ac0b6510 legacy=0x7f34ac05ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3494003750 0x7f3494003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f3494004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac0b68f0 0x7f3494004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac146aa0 con 0x7f3494004500 2022-07-21T01:42:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac146d40 con 0x7f3494003750 2022-07-21T01:42:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b1aab700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f3494004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349400b750 con 0x7f34ac0b68f0 2022-07-21T01:42:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b22ac700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3494003750 0x7f3494003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b2aad700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac0b68f0 0x7f3494004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b1aab700 1 -- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac0b68f0 msgr2=0x7f3494004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b1aab700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac0b68f0 0x7f3494004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b1aab700 1 -- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3494003750 msgr2=0x7f3494003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:23.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b1aab700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3494003750 0x7f3494003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b1aab700 1 -- 172.21.15.3:0/2497147166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3494003260 con 0x7f3494004500 2022-07-21T01:42:23.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.727+0000 7f34b1aab700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f3494004980 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f349c00c000 tx=0x7f349c006df0).ready entity=mon.0 client_cookie=2e4371b662aff502 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f349c00e5f0 con 0x7f3494004500 2022-07-21T01:42:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c00e750 con 0x7f3494004500 2022-07-21T01:42:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b12aa700 1 -- 172.21.15.3:0/2497147166 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f349c00eab0 con 0x7f3494004500 2022-07-21T01:42:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 -- 172.21.15.3:0/2497147166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 msgr2=0x7f3494004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 --2- 172.21.15.3:0/2497147166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f3494004980 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f349c00c000 tx=0x7f349c006df0).stop 2022-07-21T01:42:23.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b2aad700 1 -- 172.21.15.3:0/2497147166 reap_dead start 2022-07-21T01:42:23.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 -- 172.21.15.3:0/2497147166 shutdown_connections 2022-07-21T01:42:23.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 -- 172.21.15.3:0/2497147166 >> 172.21.15.3:0/2497147166 conn(0x7f34ac05c8b0 msgr2=0x7f34ac05cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:23.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 -- 172.21.15.3:0/2497147166 shutdown_connections 2022-07-21T01:42:23.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 -- 172.21.15.3:0/2497147166 wait complete. 2022-07-21T01:42:23.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 Processor -- start 2022-07-21T01:42:23.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 -- start start 2022-07-21T01:42:23.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f34ac0b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:23.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34ac13d700 0x7f34ac156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:23.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac152340 0x7f34ac13c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:23.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac1559b0 con 0x7f3494004500 2022-07-21T01:42:23.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac155c50 con 0x7f34ac13d700 2022-07-21T01:42:23.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b32ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac155ef0 con 0x7f34ac152340 2022-07-21T01:42:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b22ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f34ac0b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b22ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f34ac0b59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51042/0 (socket says 172.21.15.3:51042) 2022-07-21T01:42:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b22ac700 1 -- 172.21.15.3:0/3435467666 learned_addr learned my addr 172.21.15.3:0/3435467666 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:23.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.731+0000 7f34b1aab700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34ac13d700 0x7f34ac156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b2aad700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac152340 0x7f34ac13c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b22ac700 1 -- 172.21.15.3:0/3435467666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac152340 msgr2=0x7f34ac13c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:23.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b22ac700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac152340 0x7f34ac13c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b22ac700 1 -- 172.21.15.3:0/3435467666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34ac13d700 msgr2=0x7f34ac156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:23.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b22ac700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34ac13d700 0x7f34ac156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b22ac700 1 -- 172.21.15.3:0/3435467666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac145fc0 con 0x7f3494004500 2022-07-21T01:42:23.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b1aab700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34ac13d700 0x7f34ac156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:23.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b22ac700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f34ac0b59b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7f34a800d000 tx=0x7f34ac147170).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:23.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f349b7fe700 1 -- 172.21.15.3:0/3435467666 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a8011780 con 0x7f3494004500 2022-07-21T01:42:23.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f349b7fe700 1 -- 172.21.15.3:0/3435467666 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a80118e0 con 0x7f3494004500 2022-07-21T01:42:23.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f349b7fe700 1 -- 172.21.15.3:0/3435467666 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a8011c40 con 0x7f3494004500 2022-07-21T01:42:23.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac13dc10 con 0x7f3494004500 2022-07-21T01:42:23.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac12dd80 con 0x7f3494004500 2022-07-21T01:42:23.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f349b7fe700 1 -- 172.21.15.3:0/3435467666 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f34a801c360 con 0x7f3494004500 2022-07-21T01:42:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac060670 con 0x7f3494004500 2022-07-21T01:42:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f349b7fe700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f348c053b30 0x7f348c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f349b7fe700 1 -- 172.21.15.3:0/3435467666 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f34a806ba90 con 0x7f3494004500 2022-07-21T01:42:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b1aab700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f348c053b30 0x7f348c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:23.768 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.735+0000 7f34b1aab700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f348c053b30 0x7f348c055ff0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f349c01a000 tx=0x7f349c017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:23.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.743+0000 7f349b7fe700 1 -- 172.21.15.3:0/3435467666 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f34a80427a0 con 0x7f3494004500 2022-07-21T01:42:23.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.903+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34ac0574e0 con 0x7f3494004500 2022-07-21T01:42:23.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.903+0000 7f349b7fe700 1 -- 172.21.15.3:0/3435467666 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2129 (secure 0 0 0) 0x7f34a8019030 con 0x7f3494004500 2022-07-21T01:42:23.908 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:23.908 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/567682837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":567682837},{"type":"v1","addr":"172.21.15.143:6833","nonce":567682837}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/140242740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140242740},{"type":"v1","addr":"172.21.15.3:6835","nonce":140242740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:07.063040+0000","modified":"2022-07-21T01:42:23.594054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":141,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-21T01:42:23.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f348c053b30 msgr2=0x7f348c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:23.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f348c053b30 0x7f348c055ff0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f349c01a000 tx=0x7f349c017000).stop 2022-07-21T01:42:23.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 msgr2=0x7f34ac0b59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:23.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f34ac0b59b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7f34a800d000 tx=0x7f34ac147170).stop 2022-07-21T01:42:23.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 shutdown_connections 2022-07-21T01:42:23.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34ac152340 0x7f34ac13c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f348c053b30 0x7f348c055ff0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34ac13d700 0x7f34ac156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 --2- 172.21.15.3:0/3435467666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3494004500 0x7f34ac0b59b0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:23.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 >> 172.21.15.3:0/3435467666 conn(0x7f34ac05c8b0 msgr2=0x7f34ac0b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:23.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 shutdown_connections 2022-07-21T01:42:23.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:23.907+0000 7f34b32ae700 1 -- 172.21.15.3:0/3435467666 wait complete. 2022-07-21T01:42:23.913 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 72 2022-07-21T01:42:23.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:42:24.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8746f74700 1 Processor -- start 2022-07-21T01:42:24.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8746f74700 1 -- start start 2022-07-21T01:42:24.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8746f74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f874014b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8746f74700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f87400603d0 0x7f87400ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8746f74700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874005ff60 con 0x7f874005fb80 2022-07-21T01:42:24.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8746f74700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8740060860 con 0x7f87400603d0 2022-07-21T01:42:24.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8746f74700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874014ac90 con 0x7f87400b16c0 2022-07-21T01:42:24.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8745771700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f874014b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8746773700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f87400603d0 0x7f87400ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8745771700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f874014b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:57968/0 (socket says 172.21.15.3:57968) 2022-07-21T01:42:24.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8745f72700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f874005fb80 0x7f8740147540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:33214/0 (socket says 172.21.15.3:33214) 2022-07-21T01:42:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8745771700 1 -- 172.21.15.3:0/2601623560 learned_addr learned my addr 172.21.15.3:0/2601623560 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8745771700 1 -- 172.21.15.3:0/2601623560 >> v2:172.21.15.143:3300/0 conn(0x7f87400603d0 msgr2=0x7f87400ad8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8745771700 1 --2- 172.21.15.3:0/2601623560 >> v2:172.21.15.143:3300/0 conn(0x7f87400603d0 0x7f87400ad8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8745771700 1 -- 172.21.15.3:0/2601623560 >> v1:172.21.15.143:6789/0 conn(0x7f874005fb80 legacy=0x7f8740147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8745771700 1 -- 172.21.15.3:0/2601623560 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874014af30 con 0x7f87400b16c0 2022-07-21T01:42:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.071+0000 7f8745771700 1 --2- 172.21.15.3:0/2601623560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f874014b670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f87400ad3e0 tx=0x7f873c006eb0).ready entity=mon.2 client_cookie=81518bdc9c26936e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8744f70700 1 -- 172.21.15.3:0/2601623560 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873c00d5c0 con 0x7f87400b16c0 2022-07-21T01:42:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8744f70700 1 -- 172.21.15.3:0/2601623560 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c000e90 con 0x7f87400b16c0 2022-07-21T01:42:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8744f70700 1 -- 172.21.15.3:0/2601623560 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873c00d950 con 0x7f87400b16c0 2022-07-21T01:42:24.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 -- 172.21.15.3:0/2601623560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 msgr2=0x7f874014b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 --2- 172.21.15.3:0/2601623560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f874014b670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f87400ad3e0 tx=0x7f873c006eb0).stop 2022-07-21T01:42:24.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 -- 172.21.15.3:0/2601623560 shutdown_connections 2022-07-21T01:42:24.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 --2- 172.21.15.3:0/2601623560 >> v2:172.21.15.143:3300/0 conn(0x7f87400603d0 0x7f87400ad8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 --2- 172.21.15.3:0/2601623560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f874014b670 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 -- 172.21.15.3:0/2601623560 >> 172.21.15.3:0/2601623560 conn(0x7f87400b54f0 msgr2=0x7f87400b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:24.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 -- 172.21.15.3:0/2601623560 shutdown_connections 2022-07-21T01:42:24.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 -- 172.21.15.3:0/2601623560 wait complete. 2022-07-21T01:42:24.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 Processor -- start 2022-07-21T01:42:24.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 -- start start 2022-07-21T01:42:24.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f874005fb80 0x7f8740149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87400603d0 0x7f8740066580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f8740066a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87400637b0 con 0x7f87400603d0 2022-07-21T01:42:24.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874015a790 con 0x7f874005fb80 2022-07-21T01:42:24.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.075+0000 7f8746f74700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87400ac140 con 0x7f87400b16c0 2022-07-21T01:42:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745771700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87400603d0 0x7f8740066580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745771700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87400603d0 0x7f8740066580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51058/0 (socket says 172.21.15.3:51058) 2022-07-21T01:42:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745771700 1 -- 172.21.15.3:0/2136213886 learned_addr learned my addr 172.21.15.3:0/2136213886 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8746773700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f8740066a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745f72700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f874005fb80 0x7f8740149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745771700 1 -- 172.21.15.3:0/2136213886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 msgr2=0x7f8740066a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745771700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f8740066a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745771700 1 -- 172.21.15.3:0/2136213886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f874005fb80 msgr2=0x7f8740149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745771700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f874005fb80 0x7f8740149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745771700 1 -- 172.21.15.3:0/2136213886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c00b1e0 con 0x7f87400603d0 2022-07-21T01:42:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745f72700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f874005fb80 0x7f8740149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745771700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87400603d0 0x7f8740066580 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7f87400ad3e0 tx=0x7f873c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f872effd700 1 -- 172.21.15.3:0/2136213886 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873c011860 con 0x7f87400603d0 2022-07-21T01:42:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f872effd700 1 -- 172.21.15.3:0/2136213886 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c0119c0 con 0x7f87400603d0 2022-07-21T01:42:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874014a150 con 0x7f87400603d0 2022-07-21T01:42:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f872effd700 1 -- 172.21.15.3:0/2136213886 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873c011d20 con 0x7f87400603d0 2022-07-21T01:42:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8740066f60 con 0x7f87400603d0 2022-07-21T01:42:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87400574e0 con 0x7f87400603d0 2022-07-21T01:42:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f872effd700 1 -- 172.21.15.3:0/2136213886 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f873c00f070 con 0x7f87400603d0 2022-07-21T01:42:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f872effd700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8720053a70 0x7f8720055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f872effd700 1 -- 172.21.15.3:0/2136213886 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f873c06db50 con 0x7f87400603d0 2022-07-21T01:42:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.079+0000 7f8745f72700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8720053a70 0x7f8720055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.083+0000 7f8745f72700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8720053a70 0x7f8720055f30 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f873000b000 tx=0x7f87300070e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.083+0000 7f872effd700 1 -- 172.21.15.3:0/2136213886 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f873c0447b0 con 0x7f87400603d0 2022-07-21T01:42:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.247+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8740069a70 con 0x7f87400603d0 2022-07-21T01:42:24.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.611+0000 7f872effd700 1 -- 172.21.15.3:0/2136213886 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v73) v1 ==== 104+0+0 (secure 0 0 0) 0x7f873c006a40 con 0x7f87400603d0 2022-07-21T01:42:24.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8720053a70 msgr2=0x7f8720055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8720053a70 0x7f8720055f30 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f873000b000 tx=0x7f87300070e0).stop 2022-07-21T01:42:24.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87400603d0 msgr2=0x7f8740066580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87400603d0 0x7f8740066580 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7f87400ad3e0 tx=0x7f873c015000).stop 2022-07-21T01:42:24.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 shutdown_connections 2022-07-21T01:42:24.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87400b16c0 0x7f8740066a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87400603d0 0x7f8740066580 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8720053a70 0x7f8720055f30 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 --2- 172.21.15.3:0/2136213886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f874005fb80 0x7f8740149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 >> 172.21.15.3:0/2136213886 conn(0x7f87400b54f0 msgr2=0x7f874005c870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:24.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 shutdown_connections 2022-07-21T01:42:24.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.615+0000 7f8746f74700 1 -- 172.21.15.3:0/2136213886 wait complete. 2022-07-21T01:42:24.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:42:24.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e9c8f700 1 Processor -- start 2022-07-21T01:42:24.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e9c8f700 1 -- start start 2022-07-21T01:42:24.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e9c8f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b6660 0x7f52e40b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e9c8f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f52e4143de0 0x7f52e4147020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e9c8f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40b6f30 con 0x7f52e40b4590 2022-07-21T01:42:24.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e9c8f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e40b16e0 con 0x7f52e4143de0 2022-07-21T01:42:24.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e9c8f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e41431a0 con 0x7f52e40b6660 2022-07-21T01:42:24.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f52e4143de0 0x7f52e4147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e948e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f52e40b4590 0x7f52e405bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59350/0 (socket says 172.21.15.3:59350) 2022-07-21T01:42:24.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e948e700 1 -- 172.21.15.3:0/4049637265 learned_addr learned my addr 172.21.15.3:0/4049637265 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:24.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e8c8d700 1 --2- 172.21.15.3:0/4049637265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b6660 0x7f52e40b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e3fff700 1 -- 172.21.15.3:0/4049637265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b6660 msgr2=0x7f52e40b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:24.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.747+0000 7f52e3fff700 1 --2- 172.21.15.3:0/4049637265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b6660 0x7f52e40b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e3fff700 1 -- 172.21.15.3:0/4049637265 >> v1:172.21.15.3:6789/0 conn(0x7f52e40b4590 legacy=0x7f52e405bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e3fff700 1 -- 172.21.15.3:0/4049637265 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e4143400 con 0x7f52e4143de0 2022-07-21T01:42:24.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e3fff700 1 --2- 172.21.15.3:0/4049637265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 0x7f52e4147020 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7f52d4002f70 tx=0x7f52d4006df0).ready entity=mon.0 client_cookie=425614e739958a6a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:24.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e37fe700 1 -- 172.21.15.3:0/4049637265 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52d400d570 con 0x7f52e4143de0 2022-07-21T01:42:24.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e37fe700 1 -- 172.21.15.3:0/4049637265 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d4000e90 con 0x7f52e4143de0 2022-07-21T01:42:24.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e37fe700 1 -- 172.21.15.3:0/4049637265 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52d400d8c0 con 0x7f52e4143de0 2022-07-21T01:42:24.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 -- 172.21.15.3:0/4049637265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 msgr2=0x7f52e4147020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 --2- 172.21.15.3:0/4049637265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 0x7f52e4147020 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7f52d4002f70 tx=0x7f52d4006df0).stop 2022-07-21T01:42:24.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 -- 172.21.15.3:0/4049637265 shutdown_connections 2022-07-21T01:42:24.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 --2- 172.21.15.3:0/4049637265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 0x7f52e4147020 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 --2- 172.21.15.3:0/4049637265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b6660 0x7f52e40b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 -- 172.21.15.3:0/4049637265 >> 172.21.15.3:0/4049637265 conn(0x7f52e405f840 msgr2=0x7f52e4061590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:24.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 -- 172.21.15.3:0/4049637265 shutdown_connections 2022-07-21T01:42:24.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 -- 172.21.15.3:0/4049637265 wait complete. 2022-07-21T01:42:24.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 Processor -- start 2022-07-21T01:42:24.769 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.751+0000 7f52e9c8f700 1 -- start start 2022-07-21T01:42:24.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e9c8f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b4590 0x7f52e40b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e9c8f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52e40b6660 0x7f52e4155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e9c8f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 0x7f52e4156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e9c8f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e4145c90 con 0x7f52e4143de0 2022-07-21T01:42:24.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e9c8f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e4145ec0 con 0x7f52e40b6660 2022-07-21T01:42:24.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e9c8f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e4146160 con 0x7f52e40b4590 2022-07-21T01:42:24.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e948e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 0x7f52e4156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e8c8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b4590 0x7f52e40b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52e40b6660 0x7f52e4155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e8c8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b4590 0x7f52e40b2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:57988/0 (socket says 172.21.15.3:57988) 2022-07-21T01:42:24.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e8c8d700 1 -- 172.21.15.3:0/652089861 learned_addr learned my addr 172.21.15.3:0/652089861 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:24.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e948e700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 0x7f52e4156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51076/0 (socket says 172.21.15.3:51076) 2022-07-21T01:42:24.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e8c8d700 1 -- 172.21.15.3:0/652089861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52e40b6660 msgr2=0x7f52e4155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e8c8d700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52e40b6660 0x7f52e4155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e8c8d700 1 -- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 msgr2=0x7f52e4156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:24.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e8c8d700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 0x7f52e4156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:24.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e8c8d700 1 -- 172.21.15.3:0/652089861 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d400b180 con 0x7f52e40b4590 2022-07-21T01:42:24.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e948e700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 0x7f52e4156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:24.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e8c8d700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b4590 0x7f52e40b2e20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f52d80030b0 tx=0x7f52d8008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:24.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e1ffb700 1 -- 172.21.15.3:0/652089861 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52d8011780 con 0x7f52e40b4590 2022-07-21T01:42:24.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e1ffb700 1 -- 172.21.15.3:0/652089861 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d80118e0 con 0x7f52e40b4590 2022-07-21T01:42:24.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e1ffb700 1 -- 172.21.15.3:0/652089861 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52d8011c40 con 0x7f52e40b4590 2022-07-21T01:42:24.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e41558c0 con 0x7f52e40b4590 2022-07-21T01:42:24.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.755+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e40aed90 con 0x7f52e40b4590 2022-07-21T01:42:24.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.759+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e41436a0 con 0x7f52e40b4590 2022-07-21T01:42:24.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.759+0000 7f52e1ffb700 1 -- 172.21.15.3:0/652089861 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f52d8012420 con 0x7f52e40b4590 2022-07-21T01:42:24.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.759+0000 7f52e1ffb700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52c4053aa0 0x7f52c4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:24.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.759+0000 7f52e1ffb700 1 -- 172.21.15.3:0/652089861 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f52d806beb0 con 0x7f52e40b4590 2022-07-21T01:42:24.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.759+0000 7f52e3fff700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52c4053aa0 0x7f52c4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:24.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.763+0000 7f52e3fff700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52c4053aa0 0x7f52c4055f60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f52d401c000 tx=0x7f52d4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:24.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.771+0000 7f52e1ffb700 1 -- 172.21.15.3:0/652089861 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f52d8042b20 con 0x7f52e40b4590 2022-07-21T01:42:24.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:24.899+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52e40574e0 con 0x7f52e40b4590 2022-07-21T01:42:25.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e1ffb700 1 -- 172.21.15.3:0/652089861 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v142) v1 ==== 185+0+0 (secure 0 0 0) 0x7f52d8046140 con 0x7f52e40b4590 2022-07-21T01:42:25.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52c4053aa0 msgr2=0x7f52c4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:25.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52c4053aa0 0x7f52c4055f60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f52d401c000 tx=0x7f52d4015000).stop 2022-07-21T01:42:25.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b4590 msgr2=0x7f52e40b2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:25.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b4590 0x7f52e40b2e20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f52d80030b0 tx=0x7f52d8008040).stop 2022-07-21T01:42:25.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 shutdown_connections 2022-07-21T01:42:25.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52e4143de0 0x7f52e4156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:25.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f52c4053aa0 0x7f52c4055f60 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:25.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52e40b6660 0x7f52e4155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:25.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 --2- 172.21.15.3:0/652089861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40b4590 0x7f52e40b2e20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:25.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 >> 172.21.15.3:0/652089861 conn(0x7f52e405f840 msgr2=0x7f52e4141d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:25.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 shutdown_connections 2022-07-21T01:42:25.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.655+0000 7f52e9c8f700 1 -- 172.21.15.3:0/652089861 wait complete. 2022-07-21T01:42:25.664 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:42:25.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:42:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.775+0000 7f671bfff700 1 Processor -- start 2022-07-21T01:42:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.775+0000 7f671bfff700 1 -- start start 2022-07-21T01:42:25.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f671c144de0 0x7f671c13e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:25.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f671c0b1ee0 0x7f671c141110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c1452c0 con 0x7f671c060290 2022-07-21T01:42:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c060670 con 0x7f671c0b1ee0 2022-07-21T01:42:25.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c0afa80 con 0x7f671c144de0 2022-07-21T01:42:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f671c0b1ee0 0x7f671c141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671b7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f671c0b1ee0 0x7f671c141110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51084/0 (socket says 172.21.15.3:51084) 2022-07-21T01:42:25.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671b7fe700 1 -- 172.21.15.3:0/453999078 learned_addr learned my addr 172.21.15.3:0/453999078 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671a7fc700 1 --2- 172.21.15.3:0/453999078 >> v2:172.21.15.143:3300/0 conn(0x7f671c144de0 0x7f671c13e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671b7fe700 1 -- 172.21.15.3:0/453999078 >> v2:172.21.15.143:3300/0 conn(0x7f671c144de0 msgr2=0x7f671c13e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671b7fe700 1 --2- 172.21.15.3:0/453999078 >> v2:172.21.15.143:3300/0 conn(0x7f671c144de0 0x7f671c13e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671b7fe700 1 -- 172.21.15.3:0/453999078 >> v1:172.21.15.143:6789/0 conn(0x7f671c060290 legacy=0x7f671c13e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671b7fe700 1 -- 172.21.15.3:0/453999078 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c0afd20 con 0x7f671c0b1ee0 2022-07-21T01:42:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671b7fe700 1 --2- 172.21.15.3:0/453999078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c0b1ee0 0x7f671c141110 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f67140032f0 tx=0x7f6714009290).ready entity=mon.0 client_cookie=baad41949ea67d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f6719ffb700 1 -- 172.21.15.3:0/453999078 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f671400b870 con 0x7f671c0b1ee0 2022-07-21T01:42:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f6719ffb700 1 -- 172.21.15.3:0/453999078 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671400b9d0 con 0x7f671c0b1ee0 2022-07-21T01:42:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f6719ffb700 1 -- 172.21.15.3:0/453999078 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f671400bb40 con 0x7f671c0b1ee0 2022-07-21T01:42:25.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 -- 172.21.15.3:0/453999078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c0b1ee0 msgr2=0x7f671c141110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:25.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 --2- 172.21.15.3:0/453999078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c0b1ee0 0x7f671c141110 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f67140032f0 tx=0x7f6714009290).stop 2022-07-21T01:42:25.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 -- 172.21.15.3:0/453999078 shutdown_connections 2022-07-21T01:42:25.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 --2- 172.21.15.3:0/453999078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c0b1ee0 0x7f671c141110 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:25.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 --2- 172.21.15.3:0/453999078 >> v2:172.21.15.143:3300/0 conn(0x7f671c144de0 0x7f671c13e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:25.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 -- 172.21.15.3:0/453999078 >> 172.21.15.3:0/453999078 conn(0x7f671c05d1a0 msgr2=0x7f671c05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:25.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 -- 172.21.15.3:0/453999078 shutdown_connections 2022-07-21T01:42:25.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.779+0000 7f671bfff700 1 -- 172.21.15.3:0/453999078 wait complete. 2022-07-21T01:42:25.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.783+0000 7f671bfff700 1 Processor -- start 2022-07-21T01:42:25.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.783+0000 7f671bfff700 1 -- start start 2022-07-21T01:42:25.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c060290 0x7f671c0b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:25.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0b1ee0 0x7f671c0abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:25.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c144de0 0x7f671c0abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:25.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1544f0 con 0x7f671c060290 2022-07-21T01:42:25.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c153fd0 con 0x7f671c0b1ee0 2022-07-21T01:42:25.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c0663b0 con 0x7f671c144de0 2022-07-21T01:42:25.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c060290 0x7f671c0b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:25.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c060290 0x7f671c0b6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51086/0 (socket says 172.21.15.3:51086) 2022-07-21T01:42:25.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671affd700 1 -- 172.21.15.3:0/2259979150 learned_addr learned my addr 172.21.15.3:0/2259979150 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:25.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671a7fc700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0b1ee0 0x7f671c0abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:25.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671b7fe700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c144de0 0x7f671c0abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:25.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671affd700 1 -- 172.21.15.3:0/2259979150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c144de0 msgr2=0x7f671c0abfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:25.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671affd700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c144de0 0x7f671c0abfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:25.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671affd700 1 -- 172.21.15.3:0/2259979150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0b1ee0 msgr2=0x7f671c0abab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:25.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671affd700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0b1ee0 0x7f671c0abab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:25.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671affd700 1 -- 172.21.15.3:0/2259979150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6714009530 con 0x7f671c060290 2022-07-21T01:42:25.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671affd700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c060290 0x7f671c0b6550 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f671c0b5ca0 tx=0x7f670c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:25.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f670bfff700 1 -- 172.21.15.3:0/2259979150 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f670c012400 con 0x7f671c060290 2022-07-21T01:42:25.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f670bfff700 1 -- 172.21.15.3:0/2259979150 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670c012560 con 0x7f671c060290 2022-07-21T01:42:25.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f670bfff700 1 -- 172.21.15.3:0/2259979150 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f670c0128c0 con 0x7f671c060290 2022-07-21T01:42:25.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c154250 con 0x7f671c060290 2022-07-21T01:42:25.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c09ef30 con 0x7f671c060290 2022-07-21T01:42:25.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f670bfff700 1 -- 172.21.15.3:0/2259979150 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f670c019030 con 0x7f671c060290 2022-07-21T01:42:25.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.787+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c0574e0 con 0x7f671c060290 2022-07-21T01:42:25.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.791+0000 7f670bfff700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66f80539e0 0x7f66f8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:25.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.791+0000 7f670bfff700 1 -- 172.21.15.3:0/2259979150 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f670c06b370 con 0x7f671c060290 2022-07-21T01:42:25.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.791+0000 7f671a7fc700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66f80539e0 0x7f66f8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:25.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.791+0000 7f671a7fc700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66f80539e0 0x7f66f8055ea0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f671000b000 tx=0x7f6710006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:25.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.799+0000 7f670bfff700 1 -- 172.21.15.3:0/2259979150 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f670c0421f0 con 0x7f671c060290 2022-07-21T01:42:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:25.919+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f671c09f0f0 con 0x7f671c060290 2022-07-21T01:42:26.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.663+0000 7f670bfff700 1 -- 172.21.15.3:0/2259979150 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v143) v1 ==== 173+0+0 (secure 0 0 0) 0x7f670c045810 con 0x7f671c060290 2022-07-21T01:42:26.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66f80539e0 msgr2=0x7f66f8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:26.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66f80539e0 0x7f66f8055ea0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f671000b000 tx=0x7f6710006df0).stop 2022-07-21T01:42:26.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c060290 msgr2=0x7f671c0b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:26.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c060290 0x7f671c0b6550 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f671c0b5ca0 tx=0x7f670c007c50).stop 2022-07-21T01:42:26.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 shutdown_connections 2022-07-21T01:42:26.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c144de0 0x7f671c0abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:26.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66f80539e0 0x7f66f8055ea0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:26.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0b1ee0 0x7f671c0abab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:26.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 --2- 172.21.15.3:0/2259979150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f671c060290 0x7f671c0b6550 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:26.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 >> 172.21.15.3:0/2259979150 conn(0x7f671c05d1a0 msgr2=0x7f671c13ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:26.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 shutdown_connections 2022-07-21T01:42:26.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.667+0000 7f671bfff700 1 -- 172.21.15.3:0/2259979150 wait complete. 2022-07-21T01:42:26.677 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:42:26.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:42:26.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:42:26.693 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:42:26.697 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-21T01:42:26.720 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2022-07-21T01:42:26.721 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2022-07-21T01:42:26.721 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:42:26.721 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2022-07-21T01:42:26.721 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.722 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2022-07-21T01:42:26.722 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.722 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2022-07-21T01:42:26.722 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.723 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2022-07-21T01:42:26.723 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2022-07-21T01:42:26.723 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.723 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2022-07-21T01:42:26.724 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2022-07-21T01:42:26.724 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.724 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:42:26.724 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2022-07-21T01:42:26.724 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.725 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:42:26.725 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:42:26.725 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.725 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2022-07-21T01:42:26.726 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.726 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2022-07-21T01:42:26.726 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2022-07-21T01:42:26.726 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:42:26.727 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:42:26.727 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2022-07-21T01:42:26.727 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:42:26.727 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2022-07-21T01:42:26.727 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:42:26.728 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:42:26.728 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.728 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:42:26.728 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.729 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2022-07-21T01:42:26.729 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.729 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2022-07-21T01:42:26.729 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2022-07-21T01:42:26.730 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2022-07-21T01:42:26.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2022-07-21T01:42:26.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:42:26.731 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:42:26.731 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.731 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2022-07-21T01:42:26.731 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2022-07-21T01:42:26.731 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2022-07-21T01:42:26.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2022-07-21T01:42:26.732 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2022-07-21T01:42:26.733 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2022-07-21T01:42:26.733 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.733 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2022-07-21T01:42:26.733 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2022-07-21T01:42:26.733 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2022-07-21T01:42:26.734 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.734 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2022-07-21T01:42:26.734 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2022-07-21T01:42:26.734 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.735 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:42:26.735 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.735 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2022-07-21T01:42:26.735 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.736 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2022-07-21T01:42:26.736 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.736 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:42:26.736 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2022-07-21T01:42:26.736 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.737 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2022-07-21T01:42:26.737 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2022-07-21T01:42:26.737 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2022-07-21T01:42:26.737 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore compression mode = aggressive 2022-07-21T01:42:26.738 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2022-07-21T01:42:26.738 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 20 2022-07-21T01:42:26.738 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 20 2022-07-21T01:42:26.738 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:26.739 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2022-07-21T01:42:26.739 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 10 2022-07-21T01:42:26.739 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:42:26.739 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2022-07-21T01:42:26.739 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:42:26.740 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:42:26.740 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2022-07-21T01:42:26.740 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2022-07-21T01:42:26.740 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.740 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2022-07-21T01:42:26.741 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:26.741 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2022-07-21T01:42:26.741 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2022-07-21T01:42:26.741 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2022-07-21T01:42:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:26.743 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2022-07-21T01:42:26.743 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.743 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2022-07-21T01:42:26.744 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:26.744 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2022-07-21T01:42:26.745 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2022-07-21T01:42:26.745 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2022-07-21T01:42:26.745 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2022-07-21T01:42:26.745 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2022-07-21T01:42:26.746 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:26.746 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2022-07-21T01:42:26.747 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2022-07-21T01:42:26.747 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:26.747 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.748 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:42:26.748 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:42:26.749 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:42:26.749 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.750 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:42:26.750 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:42:26.751 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:42:26.751 INFO:teuthology.orchestra.run.smithi143.stdout: mon op complaint time = 120 2022-07-21T01:42:26.752 INFO:teuthology.orchestra.run.smithi143.stdout: 2022-07-21T01:42:26.752 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2022-07-21T01:42:26.752 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2022-07-21T01:42:26.753 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2022-07-21T01:42:26.753 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2022-07-21T01:42:26.753 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:42:26.753 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:42:26.754 INFO:teuthology.orchestra.run.smithi143.stdout: client mount timeout = 600 2022-07-21T01:42:26.754 INFO:teuthology.orchestra.run.smithi143.stdout: debug client = 20 2022-07-21T01:42:26.754 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:26.755 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 15m 2022-07-21T01:42:26.755 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 15m 2022-07-21T01:42:26.755 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2022-07-21T01:42:26.755 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2022-07-21T01:42:26.756 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2022-07-21T01:42:26.756 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.a] 2022-07-21T01:42:26.756 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.c] 2022-07-21T01:42:26.756 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.b] 2022-07-21T01:42:26.757 INFO:teuthology.orchestra.run.smithi143.stdout:[mds] 2022-07-21T01:42:26.757 INFO:teuthology.orchestra.run.smithi143.stdout: debug mds = 20 2022-07-21T01:42:26.757 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2022-07-21T01:42:26.757 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal frag = True 2022-07-21T01:42:26.758 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal fragment size max = 10000 2022-07-21T01:42:26.758 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal merge size = 5 2022-07-21T01:42:26.758 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal split bits = 3 2022-07-21T01:42:26.759 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal split size = 100 2022-07-21T01:42:26.759 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug frag = True 2022-07-21T01:42:26.759 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug scatterstat = True 2022-07-21T01:42:26.760 INFO:teuthology.orchestra.run.smithi143.stdout: mds op complaint time = 180 2022-07-21T01:42:26.760 INFO:teuthology.orchestra.run.smithi143.stdout: mds verify scatter = True 2022-07-21T01:42:26.760 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2022-07-21T01:42:26.760 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 15m 2022-07-21T01:42:26.761 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 15m 2022-07-21T01:42:26.765 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2022-07-21T01:42:26.765 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2022-07-21T01:42:26.765 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:42:26.766 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = cephx 2022-07-21T01:42:26.766 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.766 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2022-07-21T01:42:26.767 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.767 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2022-07-21T01:42:26.767 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.767 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2022-07-21T01:42:26.768 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2022-07-21T01:42:26.768 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.768 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2022-07-21T01:42:26.769 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2022-07-21T01:42:26.769 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.769 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:42:26.770 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2022-07-21T01:42:26.770 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.770 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:42:26.770 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:42:26.771 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.771 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2022-07-21T01:42:26.771 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.771 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2022-07-21T01:42:26.772 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2022-07-21T01:42:26.772 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:42:26.772 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:42:26.773 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2022-07-21T01:42:26.773 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:42:26.773 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2022-07-21T01:42:26.773 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:42:26.774 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:42:26.774 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.774 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:42:26.774 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.775 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2022-07-21T01:42:26.775 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.775 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2022-07-21T01:42:26.775 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.776 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2022-07-21T01:42:26.776 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2022-07-21T01:42:26.777 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2022-07-21T01:42:26.777 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:42:26.777 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:42:26.778 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.778 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2022-07-21T01:42:26.778 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2022-07-21T01:42:26.778 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.779 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2022-07-21T01:42:26.779 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 600 2022-07-21T01:42:26.779 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.779 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2022-07-21T01:42:26.780 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2022-07-21T01:42:26.780 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.780 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2022-07-21T01:42:26.780 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2022-07-21T01:42:26.781 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2022-07-21T01:42:26.781 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.781 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2022-07-21T01:42:26.782 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2022-07-21T01:42:26.782 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.782 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:42:26.782 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.783 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2022-07-21T01:42:26.783 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.783 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2022-07-21T01:42:26.783 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.784 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:42:26.784 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2022-07-21T01:42:26.784 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.785 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2022-07-21T01:42:26.785 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2022-07-21T01:42:26.786 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore block size = 96636764160 2022-07-21T01:42:26.786 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore compression mode = aggressive 2022-07-21T01:42:26.786 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore fsck on mount = True 2022-07-21T01:42:26.786 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluefs = 20 2022-07-21T01:42:26.787 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluestore = 20 2022-07-21T01:42:26.787 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:26.787 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 20 2022-07-21T01:42:26.788 INFO:teuthology.orchestra.run.smithi174.stdout: debug rocksdb = 10 2022-07-21T01:42:26.788 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:42:26.788 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd full ratio = 0.9 2022-07-21T01:42:26.788 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:42:26.789 INFO:teuthology.orchestra.run.smithi174.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:42:26.789 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = bluestore 2022-07-21T01:42:26.789 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2022-07-21T01:42:26.789 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.790 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2022-07-21T01:42:26.790 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:26.791 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2022-07-21T01:42:26.791 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2022-07-21T01:42:26.791 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2022-07-21T01:42:26.791 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:26.792 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:26.792 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2022-07-21T01:42:26.792 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.792 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2022-07-21T01:42:26.792 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:26.793 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2022-07-21T01:42:26.793 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 20 2022-07-21T01:42:26.793 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2022-07-21T01:42:26.793 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2022-07-21T01:42:26.794 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2022-07-21T01:42:26.794 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:26.794 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2022-07-21T01:42:26.794 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2022-07-21T01:42:26.795 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:26.795 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.795 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:42:26.795 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:42:26.795 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:42:26.796 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.796 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:42:26.796 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:42:26.797 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:42:26.797 INFO:teuthology.orchestra.run.smithi174.stdout: mon op complaint time = 120 2022-07-21T01:42:26.797 INFO:teuthology.orchestra.run.smithi174.stdout: 2022-07-21T01:42:26.798 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2022-07-21T01:42:26.798 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2022-07-21T01:42:26.799 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2022-07-21T01:42:26.799 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2022-07-21T01:42:26.799 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:42:26.800 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:42:26.800 INFO:teuthology.orchestra.run.smithi174.stdout: client mount timeout = 600 2022-07-21T01:42:26.800 INFO:teuthology.orchestra.run.smithi174.stdout: debug client = 20 2022-07-21T01:42:26.800 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:26.801 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 15m 2022-07-21T01:42:26.801 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 15m 2022-07-21T01:42:26.801 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2022-07-21T01:42:26.802 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.b] 2022-07-21T01:42:26.802 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.c] 2022-07-21T01:42:26.802 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.a] 2022-07-21T01:42:26.802 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.c] 2022-07-21T01:42:26.803 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.b] 2022-07-21T01:42:26.803 INFO:teuthology.orchestra.run.smithi174.stdout:[mds] 2022-07-21T01:42:26.804 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds = 20 2022-07-21T01:42:26.804 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2022-07-21T01:42:26.804 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal frag = True 2022-07-21T01:42:26.804 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal fragment size max = 10000 2022-07-21T01:42:26.805 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal merge size = 5 2022-07-21T01:42:26.805 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split bits = 3 2022-07-21T01:42:26.805 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split size = 100 2022-07-21T01:42:26.805 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug frag = True 2022-07-21T01:42:26.806 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug scatterstat = True 2022-07-21T01:42:26.806 INFO:teuthology.orchestra.run.smithi174.stdout: mds op complaint time = 180 2022-07-21T01:42:26.806 INFO:teuthology.orchestra.run.smithi174.stdout: mds verify scatter = True 2022-07-21T01:42:26.807 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2022-07-21T01:42:26.807 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 15m 2022-07-21T01:42:26.807 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 15m 2022-07-21T01:42:26.808 INFO:teuthology.orchestra.run.smithi171.stdout:[global] 2022-07-21T01:42:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: chdir = "" 2022-07-21T01:42:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:42:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: auth supported = cephx 2022-07-21T01:42:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: filestore xattr use omap = true 2022-07-21T01:42:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: mon clock drift allowed = 1.000 2022-07-21T01:42:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: osd crush chooseleaf type = 0 2022-07-21T01:42:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: auth debug = true 2022-07-21T01:42:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on old message = true 2022-07-21T01:42:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on bug = true 2022-07-21T01:42:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:42:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: mon pg warn max object skew = 0 2022-07-21T01:42:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:42:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:42:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default size = 2 2022-07-21T01:42:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow primary affinity = true 2022-07-21T01:42:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow pg remap = true 2022-07-21T01:42:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:42:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:42:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on no sortbitwise = false 2022-07-21T01:42:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:42:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on too few osds = false 2022-07-21T01:42:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:42:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:42:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:42:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: osd default data pool replay window = 5 2022-07-21T01:42:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: mon allow pool delete = true 2022-07-21T01:42:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: mon cluster log file level = debug 2022-07-21T01:42:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: debug asserts on shutdown = true 2022-07-21T01:42:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: mon health detail to clog = false 2022-07-21T01:42:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:42:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:42:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.820 INFO:teuthology.orchestra.run.smithi171.stdout:[osd] 2022-07-21T01:42:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: osd journal size = 100 2022-07-21T01:42:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub load threshold = 5.0 2022-07-21T01:42:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub max interval = 600 2022-07-21T01:42:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: osd recover clone overlap = true 2022-07-21T01:42:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: osd recovery max chunk = 1048576 2022-07-21T01:42:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug shutdown = true 2022-07-21T01:42:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug op order = true 2022-07-21T01:42:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify stray on activate = true 2022-07-21T01:42:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: osd open classes on start = true 2022-07-21T01:42:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug pg log writeout = true 2022-07-21T01:42:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:42:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: osd map max advance = 10 2022-07-21T01:42:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: journal zero on create = true 2022-07-21T01:42:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:42:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: filestore apply finisher threads = 3 2022-07-21T01:42:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: bdev debug aio = true 2022-07-21T01:42:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug misdirected ops = true 2022-07-21T01:42:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore block size = 96636764160 2022-07-21T01:42:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore compression mode = aggressive 2022-07-21T01:42:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore fsck on mount = True 2022-07-21T01:42:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluefs = 20 2022-07-21T01:42:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluestore = 20 2022-07-21T01:42:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: debug osd = 20 2022-07-21T01:42:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: debug rocksdb = 10 2022-07-21T01:42:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:42:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd full ratio = 0.9 2022-07-21T01:42:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:42:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:42:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: osd objectstore = bluestore 2022-07-21T01:42:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: osd op complaint time = 180 2022-07-21T01:42:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.831 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr] 2022-07-21T01:42:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: debug mgr = 20 2022-07-21T01:42:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2022-07-21T01:42:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2022-07-21T01:42:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: mgr/telemetry/nag = false 2022-07-21T01:42:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.833 INFO:teuthology.orchestra.run.smithi171.stdout:[mon] 2022-07-21T01:42:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2022-07-21T01:42:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: debug paxos = 20 2022-07-21T01:42:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2022-07-21T01:42:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: mon data avail warn = 5 2022-07-21T01:42:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: mon mgr mkfs grace = 240 2022-07-21T01:42:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd reporter subtree level = osd 2022-07-21T01:42:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd prime pg temp = true 2022-07-21T01:42:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:42:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:42:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:42:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:42:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:42:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:42:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: mon op complaint time = 120 2022-07-21T01:42:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-07-21T01:42:26.839 INFO:teuthology.orchestra.run.smithi171.stdout:[client] 2022-07-21T01:42:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: rgw cache enabled = true 2022-07-21T01:42:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable ops log = true 2022-07-21T01:42:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable usage log = true 2022-07-21T01:42:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:42:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:42:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: client mount timeout = 600 2022-07-21T01:42:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: debug client = 20 2022-07-21T01:42:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: rados mon op timeout = 15m 2022-07-21T01:42:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: rados osd op timeout = 15m 2022-07-21T01:42:26.841 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.a] 2022-07-21T01:42:26.842 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.b] 2022-07-21T01:42:26.842 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.c] 2022-07-21T01:42:26.842 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.a] 2022-07-21T01:42:26.842 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.c] 2022-07-21T01:42:26.843 INFO:teuthology.orchestra.run.smithi171.stdout:[mds.b] 2022-07-21T01:42:26.843 INFO:teuthology.orchestra.run.smithi171.stdout:[mds] 2022-07-21T01:42:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: debug mds = 20 2022-07-21T01:42:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2022-07-21T01:42:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal frag = True 2022-07-21T01:42:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal fragment size max = 10000 2022-07-21T01:42:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal merge size = 5 2022-07-21T01:42:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal split bits = 3 2022-07-21T01:42:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: mds bal split size = 100 2022-07-21T01:42:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: mds debug frag = True 2022-07-21T01:42:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: mds debug scatterstat = True 2022-07-21T01:42:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: mds op complaint time = 180 2022-07-21T01:42:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: mds verify scatter = True 2022-07-21T01:42:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: osd op complaint time = 180 2022-07-21T01:42:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: rados mon op timeout = 15m 2022-07-21T01:42:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: rados osd op timeout = 15m 2022-07-21T01:42:26.846 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2022-07-21T01:42:26.847 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2022-07-21T01:42:26.847 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-21T01:42:26.847 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2022-07-21T01:42:26.847 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.848 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2022-07-21T01:42:26.848 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.848 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2022-07-21T01:42:26.848 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.849 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2022-07-21T01:42:26.849 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2022-07-21T01:42:26.849 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.849 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2022-07-21T01:42:26.850 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2022-07-21T01:42:26.850 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.850 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-21T01:42:26.850 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2022-07-21T01:42:26.851 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.851 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2022-07-21T01:42:26.851 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-21T01:42:26.851 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.852 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2022-07-21T01:42:26.852 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.852 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2022-07-21T01:42:26.853 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2022-07-21T01:42:26.853 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2022-07-21T01:42:26.853 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2022-07-21T01:42:26.853 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2022-07-21T01:42:26.854 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2022-07-21T01:42:26.854 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2022-07-21T01:42:26.854 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-21T01:42:26.854 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-21T01:42:26.855 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.855 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-21T01:42:26.855 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.855 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2022-07-21T01:42:26.856 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.856 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2022-07-21T01:42:26.856 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.857 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2022-07-21T01:42:26.857 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2022-07-21T01:42:26.857 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2022-07-21T01:42:26.858 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "172.21.15.3,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2022-07-21T01:42:26.858 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 033d20cc-eff0-4d91-a7d1-56acf083b1d2 2022-07-21T01:42:26.858 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.858 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2022-07-21T01:42:26.859 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2022-07-21T01:42:26.859 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.859 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2022-07-21T01:42:26.859 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2022-07-21T01:42:26.860 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.860 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2022-07-21T01:42:26.860 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2022-07-21T01:42:26.860 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.860 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2022-07-21T01:42:26.861 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2022-07-21T01:42:26.861 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2022-07-21T01:42:26.861 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.861 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2022-07-21T01:42:26.862 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2022-07-21T01:42:26.862 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.862 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2022-07-21T01:42:26.862 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.863 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2022-07-21T01:42:26.863 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.863 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2022-07-21T01:42:26.863 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.864 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2022-07-21T01:42:26.864 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2022-07-21T01:42:26.864 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.864 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2022-07-21T01:42:26.864 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2022-07-21T01:42:26.865 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2022-07-21T01:42:26.865 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression mode = aggressive 2022-07-21T01:42:26.865 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2022-07-21T01:42:26.865 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 20 2022-07-21T01:42:26.866 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 20 2022-07-21T01:42:26.866 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:26.866 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2022-07-21T01:42:26.866 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 10 2022-07-21T01:42:26.867 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2022-07-21T01:42:26.867 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2022-07-21T01:42:26.867 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2022-07-21T01:42:26.867 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2022-07-21T01:42:26.868 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2022-07-21T01:42:26.868 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2022-07-21T01:42:26.868 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.868 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2022-07-21T01:42:26.869 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:26.869 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2022-07-21T01:42:26.869 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2022-07-21T01:42:26.869 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2022-07-21T01:42:26.870 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:26.870 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:26.870 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2022-07-21T01:42:26.870 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.870 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2022-07-21T01:42:26.871 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:26.871 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2022-07-21T01:42:26.871 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2022-07-21T01:42:26.871 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2022-07-21T01:42:26.872 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2022-07-21T01:42:26.872 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2022-07-21T01:42:26.872 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2022-07-21T01:42:26.872 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2022-07-21T01:42:26.872 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2022-07-21T01:42:26.873 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2022-07-21T01:42:26.873 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.873 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-21T01:42:26.873 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2022-07-21T01:42:26.874 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2022-07-21T01:42:26.874 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.874 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2022-07-21T01:42:26.874 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-21T01:42:26.875 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-21T01:42:26.875 INFO:teuthology.orchestra.run.smithi003.stdout: mon op complaint time = 120 2022-07-21T01:42:26.875 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:26.875 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2022-07-21T01:42:26.875 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2022-07-21T01:42:26.876 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2022-07-21T01:42:26.876 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2022-07-21T01:42:26.876 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-21T01:42:26.876 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-21T01:42:26.877 INFO:teuthology.orchestra.run.smithi003.stdout: client mount timeout = 600 2022-07-21T01:42:26.877 INFO:teuthology.orchestra.run.smithi003.stdout: debug client = 20 2022-07-21T01:42:26.877 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:26.877 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 15m 2022-07-21T01:42:26.877 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 15m 2022-07-21T01:42:26.878 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2022-07-21T01:42:26.878 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2022-07-21T01:42:26.878 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2022-07-21T01:42:26.878 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a] 2022-07-21T01:42:26.879 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c] 2022-07-21T01:42:26.879 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b] 2022-07-21T01:42:26.879 INFO:teuthology.orchestra.run.smithi003.stdout:[mds] 2022-07-21T01:42:26.879 INFO:teuthology.orchestra.run.smithi003.stdout: debug mds = 20 2022-07-21T01:42:26.879 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2022-07-21T01:42:26.880 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal frag = True 2022-07-21T01:42:26.880 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal fragment size max = 10000 2022-07-21T01:42:26.880 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal merge size = 5 2022-07-21T01:42:26.880 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal split bits = 3 2022-07-21T01:42:26.881 INFO:teuthology.orchestra.run.smithi003.stdout: mds bal split size = 100 2022-07-21T01:42:26.881 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug frag = True 2022-07-21T01:42:26.881 INFO:teuthology.orchestra.run.smithi003.stdout: mds debug scatterstat = True 2022-07-21T01:42:26.881 INFO:teuthology.orchestra.run.smithi003.stdout: mds op complaint time = 180 2022-07-21T01:42:26.881 INFO:teuthology.orchestra.run.smithi003.stdout: mds verify scatter = True 2022-07-21T01:42:26.882 INFO:teuthology.orchestra.run.smithi003.stdout: osd op complaint time = 180 2022-07-21T01:42:26.882 INFO:teuthology.orchestra.run.smithi003.stdout: rados mon op timeout = 15m 2022-07-21T01:42:26.882 INFO:teuthology.orchestra.run.smithi003.stdout: rados osd op timeout = 15m 2022-07-21T01:42:26.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:42:27.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.999+0000 7f3add532700 1 Processor -- start 2022-07-21T01:42:27.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:26.999+0000 7f3add532700 1 -- start start 2022-07-21T01:42:27.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 0x7f3ad80abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ad80b22a0 0x7f3ad80ac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad8140540 con 0x7f3ad80b1a80 2022-07-21T01:42:27.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad80ae590 con 0x7f3ad80b22a0 2022-07-21T01:42:27.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad80aff00 con 0x7f3ad80af480 2022-07-21T01:42:27.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ad80b22a0 0x7f3ad80ac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3adcd31700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3ad80b1a80 0x7f3ad81391f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59376/0 (socket says 172.21.15.3:59376) 2022-07-21T01:42:27.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3ad80b22a0 0x7f3ad80ac000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51094/0 (socket says 172.21.15.3:51094) 2022-07-21T01:42:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3adcd31700 1 -- 172.21.15.3:0/30437201 learned_addr learned my addr 172.21.15.3:0/30437201 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad7fff700 1 --2- 172.21.15.3:0/30437201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 0x7f3ad80abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad77fe700 1 -- 172.21.15.3:0/30437201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 msgr2=0x7f3ad80abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:27.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad77fe700 1 --2- 172.21.15.3:0/30437201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 0x7f3ad80abb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad77fe700 1 -- 172.21.15.3:0/30437201 >> v1:172.21.15.3:6789/0 conn(0x7f3ad80b1a80 legacy=0x7f3ad81391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad77fe700 1 -- 172.21.15.3:0/30437201 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad80b01a0 con 0x7f3ad80b22a0 2022-07-21T01:42:27.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad6ffd700 1 -- 172.21.15.3:0/30437201 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213618125 0 0) 0x7f3ad8140540 con 0x7f3ad80b1a80 2022-07-21T01:42:27.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad7fff700 1 --2- 172.21.15.3:0/30437201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 0x7f3ad80abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:27.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad77fe700 1 --2- 172.21.15.3:0/30437201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b22a0 0x7f3ad80ac000 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f3ac8002f70 tx=0x7f3ac8006d80).ready entity=mon.0 client_cookie=2ab6f050f5db9987 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad6ffd700 1 -- 172.21.15.3:0/30437201 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ac800bb70 con 0x7f3ad80b22a0 2022-07-21T01:42:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad6ffd700 1 -- 172.21.15.3:0/30437201 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac8000e90 con 0x7f3ad80b22a0 2022-07-21T01:42:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3ad6ffd700 1 -- 172.21.15.3:0/30437201 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ac80153f0 con 0x7f3ad80b22a0 2022-07-21T01:42:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 -- 172.21.15.3:0/30437201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b22a0 msgr2=0x7f3ad80ac000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 --2- 172.21.15.3:0/30437201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b22a0 0x7f3ad80ac000 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f3ac8002f70 tx=0x7f3ac8006d80).stop 2022-07-21T01:42:27.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 -- 172.21.15.3:0/30437201 shutdown_connections 2022-07-21T01:42:27.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 --2- 172.21.15.3:0/30437201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b22a0 0x7f3ad80ac000 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 --2- 172.21.15.3:0/30437201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 0x7f3ad80abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 -- 172.21.15.3:0/30437201 >> 172.21.15.3:0/30437201 conn(0x7f3ad80626f0 msgr2=0x7f3ad8062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:27.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 -- 172.21.15.3:0/30437201 shutdown_connections 2022-07-21T01:42:27.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.003+0000 7f3add532700 1 -- 172.21.15.3:0/30437201 wait complete. 2022-07-21T01:42:27.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 Processor -- start 2022-07-21T01:42:27.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 -- start start 2022-07-21T01:42:27.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 0x7f3ad813a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b1a80 0x7f3ad813a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ad80b22a0 0x7f3ad8064e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad813ac30 con 0x7f3ad80b1a80 2022-07-21T01:42:27.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad8064a50 con 0x7f3ad80b22a0 2022-07-21T01:42:27.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad8063e40 con 0x7f3ad80af480 2022-07-21T01:42:27.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b1a80 0x7f3ad813a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b1a80 0x7f3ad813a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51100/0 (socket says 172.21.15.3:51100) 2022-07-21T01:42:27.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad77fe700 1 -- 172.21.15.3:0/1196383218 learned_addr learned my addr 172.21.15.3:0/1196383218 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:27.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad7fff700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 0x7f3ad813a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3adcd31700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ad80b22a0 0x7f3ad8064e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad77fe700 1 -- 172.21.15.3:0/1196383218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 msgr2=0x7f3ad813a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad77fe700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 0x7f3ad813a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad77fe700 1 -- 172.21.15.3:0/1196383218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ad80b22a0 msgr2=0x7f3ad8064e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad77fe700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ad80b22a0 0x7f3ad8064e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad77fe700 1 -- 172.21.15.3:0/1196383218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac800d410 con 0x7f3ad80b1a80 2022-07-21T01:42:27.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad77fe700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b1a80 0x7f3ad813a740 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f3ad8064b90 tx=0x7f3ac8019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:27.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad57fa700 1 -- 172.21.15.3:0/1196383218 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ac80153f0 con 0x7f3ad80b1a80 2022-07-21T01:42:27.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad57fa700 1 -- 172.21.15.3:0/1196383218 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac801e600 con 0x7f3ad80b1a80 2022-07-21T01:42:27.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad8064070 con 0x7f3ad80b1a80 2022-07-21T01:42:27.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3ad57fa700 1 -- 172.21.15.3:0/1196383218 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ac801e960 con 0x7f3ad80b1a80 2022-07-21T01:42:27.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.007+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad8063b90 con 0x7f3ad80b1a80 2022-07-21T01:42:27.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.011+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad80574e0 con 0x7f3ad80b1a80 2022-07-21T01:42:27.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.011+0000 7f3ad57fa700 1 -- 172.21.15.3:0/1196383218 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ac801eac0 con 0x7f3ad80b1a80 2022-07-21T01:42:27.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.011+0000 7f3ad57fa700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ab8053a70 0x7f3ab8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.011+0000 7f3ad7fff700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ab8053a70 0x7f3ab8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.011+0000 7f3ad57fa700 1 -- 172.21.15.3:0/1196383218 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f3ac806f7e0 con 0x7f3ad80b1a80 2022-07-21T01:42:27.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.011+0000 7f3ad7fff700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ab8053a70 0x7f3ab8055f30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f3acc00c000 tx=0x7f3acc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:27.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.019+0000 7f3ad57fa700 1 -- 172.21.15.3:0/1196383218 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ac8046880 con 0x7f3ad80b1a80 2022-07-21T01:42:27.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.143+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3ad80ac590 con 0x7f3ad80b1a80 2022-07-21T01:42:27.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.151+0000 7f3ad57fa700 1 -- 172.21.15.3:0/1196383218 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3ac8049ea0 con 0x7f3ad80b1a80 2022-07-21T01:42:27.160 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ab8053a70 msgr2=0x7f3ab8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ab8053a70 0x7f3ab8055f30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f3acc00c000 tx=0x7f3acc008040).stop 2022-07-21T01:42:27.161 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b1a80 msgr2=0x7f3ad813a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.162 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b1a80 0x7f3ad813a740 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f3ad8064b90 tx=0x7f3ac8019000).stop 2022-07-21T01:42:27.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 shutdown_connections 2022-07-21T01:42:27.163 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3ab8053a70 0x7f3ab8055f30 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ad80b22a0 0x7f3ad8064e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ad80b1a80 0x7f3ad813a740 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.165 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 --2- 172.21.15.3:0/1196383218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ad80af480 0x7f3ad813a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.166 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 >> 172.21.15.3:0/1196383218 conn(0x7f3ad80626f0 msgr2=0x7f3ad80ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:27.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 shutdown_connections 2022-07-21T01:42:27.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.155+0000 7f3add532700 1 -- 172.21.15.3:0/1196383218 wait complete. 2022-07-21T01:42:27.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-07-21T01:42:27.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.283+0000 7f80ac877700 1 Processor -- start 2022-07-21T01:42:27.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.283+0000 7f80ac877700 1 -- start start 2022-07-21T01:42:27.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f80a80b29e0 0x7f80a80b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a80b2dc0 con 0x7f80a80ad0b0 2022-07-21T01:42:27.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a80b17f0 con 0x7f80a813c870 2022-07-21T01:42:27.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a813d3c0 con 0x7f80a80b29e0 2022-07-21T01:42:27.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a77fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f80a80b29e0 0x7f80a80b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a7fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f80a80ad0b0 0x7f80a80ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59390/0 (socket says 172.21.15.3:59390) 2022-07-21T01:42:27.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a7fff700 1 -- 172.21.15.3:0/4133059184 learned_addr learned my addr 172.21.15.3:0/4133059184 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:27.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a67fc700 1 -- 172.21.15.3:0/4133059184 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836843768 0 0) 0x7f80a80b2dc0 con 0x7f80a80ad0b0 2022-07-21T01:42:27.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a67fc700 1 -- 172.21.15.3:0/4133059184 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8094004370 con 0x7f80a80ad0b0 2022-07-21T01:42:27.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a77fe700 1 -- 172.21.15.3:0/4133059184 >> v1:172.21.15.143:6789/0 conn(0x7f80a813c870 legacy=0x7f80a813ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a77fe700 1 -- 172.21.15.3:0/4133059184 >> v1:172.21.15.3:6789/0 conn(0x7f80a80ad0b0 legacy=0x7f80a80ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a77fe700 1 -- 172.21.15.3:0/4133059184 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a80b1a90 con 0x7f80a80b29e0 2022-07-21T01:42:27.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a77fe700 1 --2- 172.21.15.3:0/4133059184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a80b29e0 0x7f80a80b0c30 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f809c002f70 tx=0x7f809c006df0).ready entity=mon.0 client_cookie=2b1bc7812d1f5dce server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:27.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a67fc700 1 -- 172.21.15.3:0/4133059184 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f809c00d610 con 0x7f80a80b29e0 2022-07-21T01:42:27.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a67fc700 1 -- 172.21.15.3:0/4133059184 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c000e90 con 0x7f80a80b29e0 2022-07-21T01:42:27.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80a67fc700 1 -- 172.21.15.3:0/4133059184 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f809c00d960 con 0x7f80a80b29e0 2022-07-21T01:42:27.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 -- 172.21.15.3:0/4133059184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a80b29e0 msgr2=0x7f80a80b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 --2- 172.21.15.3:0/4133059184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a80b29e0 0x7f80a80b0c30 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f809c002f70 tx=0x7f809c006df0).stop 2022-07-21T01:42:27.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 -- 172.21.15.3:0/4133059184 shutdown_connections 2022-07-21T01:42:27.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 --2- 172.21.15.3:0/4133059184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a80b29e0 0x7f80a80b0c30 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 -- 172.21.15.3:0/4133059184 >> 172.21.15.3:0/4133059184 conn(0x7f80a80613c0 msgr2=0x7f80a8062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:27.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 -- 172.21.15.3:0/4133059184 shutdown_connections 2022-07-21T01:42:27.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.287+0000 7f80ac877700 1 -- 172.21.15.3:0/4133059184 wait complete. 2022-07-21T01:42:27.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 Processor -- start 2022-07-21T01:42:27.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 -- start start 2022-07-21T01:42:27.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a80ad0b0 0x7f80a80b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80b29e0 0x7f80a8143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a813c870 0x7f80a8143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a814e490 con 0x7f80a813c870 2022-07-21T01:42:27.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a814e680 con 0x7f80a80ad0b0 2022-07-21T01:42:27.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a814e920 con 0x7f80a80b29e0 2022-07-21T01:42:27.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a813c870 0x7f80a8143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a813c870 0x7f80a8143710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51116/0 (socket says 172.21.15.3:51116) 2022-07-21T01:42:27.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a7fff700 1 -- 172.21.15.3:0/237608668 learned_addr learned my addr 172.21.15.3:0/237608668 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:27.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a77fe700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a80ad0b0 0x7f80a80b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a6ffd700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80b29e0 0x7f80a8143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a7fff700 1 -- 172.21.15.3:0/237608668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80b29e0 msgr2=0x7f80a8143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a7fff700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80b29e0 0x7f80a8143220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a7fff700 1 -- 172.21.15.3:0/237608668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a80ad0b0 msgr2=0x7f80a80b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a7fff700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a80ad0b0 0x7f80a80b5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a7fff700 1 -- 172.21.15.3:0/237608668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c00b180 con 0x7f80a813c870 2022-07-21T01:42:27.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a7fff700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a813c870 0x7f80a8143710 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f80a0001490 tx=0x7f80a0007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:27.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a4ff9700 1 -- 172.21.15.3:0/237608668 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80a0005ba0 con 0x7f80a813c870 2022-07-21T01:42:27.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a814ebc0 con 0x7f80a813c870 2022-07-21T01:42:27.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a4ff9700 1 -- 172.21.15.3:0/237608668 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a0005d00 con 0x7f80a813c870 2022-07-21T01:42:27.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80a4ff9700 1 -- 172.21.15.3:0/237608668 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80a00072c0 con 0x7f80a813c870 2022-07-21T01:42:27.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.291+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a814f300 con 0x7f80a813c870 2022-07-21T01:42:27.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.295+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a80574e0 con 0x7f80a813c870 2022-07-21T01:42:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.295+0000 7f80a4ff9700 1 -- 172.21.15.3:0/237608668 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f80a0010070 con 0x7f80a813c870 2022-07-21T01:42:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.295+0000 7f80a4ff9700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8088053a70 0x7f8088055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.295+0000 7f80a4ff9700 1 -- 172.21.15.3:0/237608668 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f80a006b7e0 con 0x7f80a813c870 2022-07-21T01:42:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.295+0000 7f80a77fe700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8088053a70 0x7f8088055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.295+0000 7f80a77fe700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8088053a70 0x7f8088055f30 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f809c01a000 tx=0x7f809c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:27.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.303+0000 7f80a4ff9700 1 -- 172.21.15.3:0/237608668 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f80a00426f0 con 0x7f80a813c870 2022-07-21T01:42:27.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.471+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7f80a80b0a80 con 0x7f80a813c870 2022-07-21T01:42:27.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.663+0000 7f80a4ff9700 1 -- 172.21.15.3:0/237608668 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v418) v1 ==== 128+0+0 (secure 0 0 0) 0x7f80a0045d10 con 0x7f80a813c870 2022-07-21T01:42:27.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8088053a70 msgr2=0x7f8088055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8088053a70 0x7f8088055f30 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f809c01a000 tx=0x7f809c014000).stop 2022-07-21T01:42:27.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a813c870 msgr2=0x7f80a8143710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a813c870 0x7f80a8143710 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f80a0001490 tx=0x7f80a0007dc0).stop 2022-07-21T01:42:27.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 shutdown_connections 2022-07-21T01:42:27.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a813c870 0x7f80a8143710 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80b29e0 0x7f80a8143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8088053a70 0x7f8088055f30 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 --2- 172.21.15.3:0/237608668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a80ad0b0 0x7f80a80b5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 >> 172.21.15.3:0/237608668 conn(0x7f80a80613c0 msgr2=0x7f80a8140290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:27.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.667+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 shutdown_connections 2022-07-21T01:42:27.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.671+0000 7f80ac877700 1 -- 172.21.15.3:0/237608668 wait complete. 2022-07-21T01:42:27.689 INFO:tasks.cephfs_test_runner:test_getattr_caps (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-21T01:42:27.690 INFO:tasks.cephfs_test_runner:Starting test: test_root_rctime (tasks.cephfs.test_misc.TestMisc) 2022-07-21T01:42:27.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-07-21T01:42:27.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.799+0000 7fd2e94ca700 1 Processor -- start 2022-07-21T01:42:27.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.799+0000 7fd2e94ca700 1 -- start start 2022-07-21T01:42:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd2e413d9b0 0x7fd2e40609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e40b0e60 0x7fd2e4060eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e4066af0 con 0x7fd2e40b6510 2022-07-21T01:42:27.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e4066d90 con 0x7fd2e413d9b0 2022-07-21T01:42:27.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e4067030 con 0x7fd2e40b0e60 2022-07-21T01:42:27.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e3fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd2e40b6510 0x7fd2e413d2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59400/0 (socket says 172.21.15.3:59400) 2022-07-21T01:42:27.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e3fff700 1 -- 172.21.15.3:0/982983819 learned_addr learned my addr 172.21.15.3:0/982983819 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:27.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e37fe700 1 --2- 172.21.15.3:0/982983819 >> v2:172.21.15.3:3300/0 conn(0x7fd2e413d9b0 0x7fd2e40609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e8cc9700 1 --2- 172.21.15.3:0/982983819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e40b0e60 0x7fd2e4060eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e2ffd700 1 -- 172.21.15.3:0/982983819 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622212110 0 0) 0x7fd2e4066af0 con 0x7fd2e40b6510 2022-07-21T01:42:27.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e2ffd700 1 -- 172.21.15.3:0/982983819 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d0003260 con 0x7fd2e40b6510 2022-07-21T01:42:27.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e37fe700 1 -- 172.21.15.3:0/982983819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e40b0e60 msgr2=0x7fd2e4060eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:27.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e37fe700 1 --2- 172.21.15.3:0/982983819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e40b0e60 0x7fd2e4060eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e37fe700 1 -- 172.21.15.3:0/982983819 >> v1:172.21.15.3:6789/0 conn(0x7fd2e40b6510 legacy=0x7fd2e413d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e37fe700 1 -- 172.21.15.3:0/982983819 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e40672d0 con 0x7fd2e413d9b0 2022-07-21T01:42:27.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e37fe700 1 --2- 172.21.15.3:0/982983819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e413d9b0 0x7fd2e40609c0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7fd2d8002f70 tx=0x7fd2d8006df0).ready entity=mon.0 client_cookie=463056d60669ea74 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:27.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e2ffd700 1 -- 172.21.15.3:0/982983819 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2d800d4a0 con 0x7fd2e413d9b0 2022-07-21T01:42:27.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e2ffd700 1 -- 172.21.15.3:0/982983819 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d8000e90 con 0x7fd2e413d9b0 2022-07-21T01:42:27.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e2ffd700 1 -- 172.21.15.3:0/982983819 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2d800d7f0 con 0x7fd2e413d9b0 2022-07-21T01:42:27.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 -- 172.21.15.3:0/982983819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e413d9b0 msgr2=0x7fd2e40609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 --2- 172.21.15.3:0/982983819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e413d9b0 0x7fd2e40609c0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7fd2d8002f70 tx=0x7fd2d8006df0).stop 2022-07-21T01:42:27.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 -- 172.21.15.3:0/982983819 shutdown_connections 2022-07-21T01:42:27.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 --2- 172.21.15.3:0/982983819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e40b0e60 0x7fd2e4060eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 --2- 172.21.15.3:0/982983819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e413d9b0 0x7fd2e40609c0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 -- 172.21.15.3:0/982983819 >> 172.21.15.3:0/982983819 conn(0x7fd2e4063800 msgr2=0x7fd2e4063be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:27.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 -- 172.21.15.3:0/982983819 shutdown_connections 2022-07-21T01:42:27.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.803+0000 7fd2e94ca700 1 -- 172.21.15.3:0/982983819 wait complete. 2022-07-21T01:42:27.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e94ca700 1 Processor -- start 2022-07-21T01:42:27.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e94ca700 1 -- start start 2022-07-21T01:42:27.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e94ca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e40b0e60 0x7fd2e4140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e94ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e40b6510 0x7fd2e40ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e94ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e413d9b0 0x7fd2e40aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e94ca700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e413d570 con 0x7fd2e40b0e60 2022-07-21T01:42:27.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e94ca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e40adb50 con 0x7fd2e40b6510 2022-07-21T01:42:27.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e94ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e40aa170 con 0x7fd2e413d9b0 2022-07-21T01:42:27.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e40b0e60 0x7fd2e4140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e40b0e60 0x7fd2e4140990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51126/0 (socket says 172.21.15.3:51126) 2022-07-21T01:42:27.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e3fff700 1 -- 172.21.15.3:0/2018899024 learned_addr learned my addr 172.21.15.3:0/2018899024 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:27.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e37fe700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e40b6510 0x7fd2e40ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e8cc9700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e413d9b0 0x7fd2e40aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e3fff700 1 -- 172.21.15.3:0/2018899024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e413d9b0 msgr2=0x7fd2e40aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e3fff700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e413d9b0 0x7fd2e40aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e3fff700 1 -- 172.21.15.3:0/2018899024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e40b6510 msgr2=0x7fd2e40ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e3fff700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e40b6510 0x7fd2e40ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e3fff700 1 -- 172.21.15.3:0/2018899024 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d800b180 con 0x7fd2e40b0e60 2022-07-21T01:42:27.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.807+0000 7fd2e8cc9700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e413d9b0 0x7fd2e40aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e3fff700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e40b0e60 0x7fd2e4140990 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7fd2d4001490 tx=0x7fd2d4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e17fa700 1 -- 172.21.15.3:0/2018899024 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2d4005ba0 con 0x7fd2e40b0e60 2022-07-21T01:42:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e17fa700 1 -- 172.21.15.3:0/2018899024 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d4005d00 con 0x7fd2e40b0e60 2022-07-21T01:42:27.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e40ac990 con 0x7fd2e40b0e60 2022-07-21T01:42:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e17fa700 1 -- 172.21.15.3:0/2018899024 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2d40072c0 con 0x7fd2e40b0e60 2022-07-21T01:42:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e40ae020 con 0x7fd2e40b0e60 2022-07-21T01:42:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e40ad2b0 con 0x7fd2e40b0e60 2022-07-21T01:42:27.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e17fa700 1 -- 172.21.15.3:0/2018899024 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd2d4010070 con 0x7fd2e40b0e60 2022-07-21T01:42:27.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e17fa700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c4053b00 0x7fd2c4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:27.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e17fa700 1 -- 172.21.15.3:0/2018899024 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fd2d406b7b0 con 0x7fd2e40b0e60 2022-07-21T01:42:27.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e37fe700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c4053b00 0x7fd2c4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:27.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.811+0000 7fd2e37fe700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c4053b00 0x7fd2c4055fc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fd2d801a000 tx=0x7fd2d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:27.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.819+0000 7fd2e17fa700 1 -- 172.21.15.3:0/2018899024 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd2d4042850 con 0x7fd2e40b0e60 2022-07-21T01:42:27.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:27.991+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7fd2e40574e0 con 0x7fd2e40b0e60 2022-07-21T01:42:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e17fa700 1 -- 172.21.15.3:0/2018899024 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v419) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd2d4045e70 con 0x7fd2e40b0e60 2022-07-21T01:42:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c4053b00 msgr2=0x7fd2c4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c4053b00 0x7fd2c4055fc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fd2d801a000 tx=0x7fd2d8014000).stop 2022-07-21T01:42:28.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e40b0e60 msgr2=0x7fd2e4140990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:28.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e40b0e60 0x7fd2e4140990 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7fd2d4001490 tx=0x7fd2d4007e20).stop 2022-07-21T01:42:28.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 shutdown_connections 2022-07-21T01:42:28.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e413d9b0 0x7fd2e40aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd2c4053b00 0x7fd2c4055fc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e40b6510 0x7fd2e40ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 --2- 172.21.15.3:0/2018899024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2e40b0e60 0x7fd2e4140990 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 >> 172.21.15.3:0/2018899024 conn(0x7fd2e4063800 msgr2=0x7fd2e405fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 shutdown_connections 2022-07-21T01:42:28.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.675+0000 7fd2e94ca700 1 -- 172.21.15.3:0/2018899024 wait complete. 2022-07-21T01:42:28.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:42:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83effff700 1 Processor -- start 2022-07-21T01:42:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83effff700 1 -- start start 2022-07-21T01:42:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83effff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f83f00b2710 0x7f83f014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83effff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f00a7250 con 0x7f83f00a8c50 2022-07-21T01:42:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83effff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f00a7480 con 0x7f83f00b01a0 2022-07-21T01:42:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83effff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f00a7720 con 0x7f83f00b2710 2022-07-21T01:42:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83ef7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f83f00a8c50 0x7f83f0151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59416/0 (socket says 172.21.15.3:59416) 2022-07-21T01:42:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83ef7fe700 1 -- 172.21.15.3:0/508741909 learned_addr learned my addr 172.21.15.3:0/508741909 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83eeffd700 1 --2- 172.21.15.3:0/508741909 >> v2:172.21.15.3:3300/0 conn(0x7f83f00b2710 0x7f83f014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83edffb700 1 -- 172.21.15.3:0/508741909 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687785524 0 0) 0x7f83f00a7250 con 0x7f83f00a8c50 2022-07-21T01:42:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83edffb700 1 -- 172.21.15.3:0/508741909 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83d0003260 con 0x7f83f00a8c50 2022-07-21T01:42:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83edffb700 1 -- 172.21.15.3:0/508741909 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237422142 0 0) 0x7f83f00a7480 con 0x7f83f00b01a0 2022-07-21T01:42:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83eeffd700 1 -- 172.21.15.3:0/508741909 >> v1:172.21.15.143:6789/0 conn(0x7f83f00b01a0 legacy=0x7f83f00a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83eeffd700 1 -- 172.21.15.3:0/508741909 >> v1:172.21.15.3:6789/0 conn(0x7f83f00a8c50 legacy=0x7f83f0151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83eeffd700 1 -- 172.21.15.3:0/508741909 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f00a79c0 con 0x7f83f00b2710 2022-07-21T01:42:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.811+0000 7f83eeffd700 1 --2- 172.21.15.3:0/508741909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 0x7f83f014afd0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f83d8002f70 tx=0x7f83d8006df0).ready entity=mon.0 client_cookie=1ea7794b8f7b5511 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83edffb700 1 -- 172.21.15.3:0/508741909 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83d800d610 con 0x7f83f00b2710 2022-07-21T01:42:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83edffb700 1 -- 172.21.15.3:0/508741909 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8000e90 con 0x7f83f00b2710 2022-07-21T01:42:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83edffb700 1 -- 172.21.15.3:0/508741909 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83d800d960 con 0x7f83f00b2710 2022-07-21T01:42:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 -- 172.21.15.3:0/508741909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 msgr2=0x7f83f014afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 --2- 172.21.15.3:0/508741909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 0x7f83f014afd0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f83d8002f70 tx=0x7f83d8006df0).stop 2022-07-21T01:42:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 -- 172.21.15.3:0/508741909 shutdown_connections 2022-07-21T01:42:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 --2- 172.21.15.3:0/508741909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 0x7f83f014afd0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 -- 172.21.15.3:0/508741909 >> 172.21.15.3:0/508741909 conn(0x7f83f0060310 msgr2=0x7f83f00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 -- 172.21.15.3:0/508741909 shutdown_connections 2022-07-21T01:42:28.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 -- 172.21.15.3:0/508741909 wait complete. 2022-07-21T01:42:28.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 Processor -- start 2022-07-21T01:42:28.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 -- start start 2022-07-21T01:42:28.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83f00a8c50 0x7f83f00636d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83f00b01a0 0x7f83f01465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 0x7f83f0146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f014a650 con 0x7f83f00b2710 2022-07-21T01:42:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f014a150 con 0x7f83f00a8c50 2022-07-21T01:42:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83effff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f014a3b0 con 0x7f83f00b01a0 2022-07-21T01:42:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83ef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 0x7f83f0146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83ef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 0x7f83f0146a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51142/0 (socket says 172.21.15.3:51142) 2022-07-21T01:42:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83ef7fe700 1 -- 172.21.15.3:0/1187237689 learned_addr learned my addr 172.21.15.3:0/1187237689 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83eeffd700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83f00a8c50 0x7f83f00636d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:28.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.815+0000 7f83ee7fc700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83f00b01a0 0x7f83f01465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:28.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83ef7fe700 1 -- 172.21.15.3:0/1187237689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83f00b01a0 msgr2=0x7f83f01465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:28.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83ef7fe700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83f00b01a0 0x7f83f01465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83ef7fe700 1 -- 172.21.15.3:0/1187237689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83f00a8c50 msgr2=0x7f83f00636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:28.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83ef7fe700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83f00a8c50 0x7f83f00636d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83ef7fe700 1 -- 172.21.15.3:0/1187237689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d800b180 con 0x7f83f00b2710 2022-07-21T01:42:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83ef7fe700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 0x7f83f0146a90 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f83e8001490 tx=0x7f83e8007d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83e7fff700 1 -- 172.21.15.3:0/1187237689 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83e8005ba0 con 0x7f83f00b2710 2022-07-21T01:42:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f0148e70 con 0x7f83f00b2710 2022-07-21T01:42:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83e7fff700 1 -- 172.21.15.3:0/1187237689 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e8005d00 con 0x7f83f00b2710 2022-07-21T01:42:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83e7fff700 1 -- 172.21.15.3:0/1187237689 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83e80072c0 con 0x7f83f00b2710 2022-07-21T01:42:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f0063ce0 con 0x7f83f00b2710 2022-07-21T01:42:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f00574e0 con 0x7f83f00b2710 2022-07-21T01:42:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.819+0000 7f83e7fff700 1 -- 172.21.15.3:0/1187237689 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f83e8010070 con 0x7f83f00b2710 2022-07-21T01:42:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.823+0000 7f83e7fff700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f83d4053950 0x7f83d4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.823+0000 7f83e7fff700 1 -- 172.21.15.3:0/1187237689 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f83e806b5f0 con 0x7f83f00b2710 2022-07-21T01:42:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.823+0000 7f83eeffd700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f83d4053950 0x7f83d4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.823+0000 7f83eeffd700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f83d4053950 0x7f83d4055e10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f83d8018000 tx=0x7f83d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:28.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.827+0000 7f83e7fff700 1 -- 172.21.15.3:0/1187237689 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f83e80425f0 con 0x7f83f00b2710 2022-07-21T01:42:28.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.955+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f83f0149000 con 0x7f83f00b2710 2022-07-21T01:42:28.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.959+0000 7f83e7fff700 1 -- 172.21.15.3:0/1187237689 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f83e8018070 con 0x7f83f00b2710 2022-07-21T01:42:28.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f83d4053950 msgr2=0x7f83d4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:28.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f83d4053950 0x7f83d4055e10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f83d8018000 tx=0x7f83d8014000).stop 2022-07-21T01:42:28.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 msgr2=0x7f83f0146a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:28.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 0x7f83f0146a90 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f83e8001490 tx=0x7f83e8007d60).stop 2022-07-21T01:42:28.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 shutdown_connections 2022-07-21T01:42:28.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f83f00b2710 0x7f83f0146a90 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83f00b01a0 0x7f83f01465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f83d4053950 0x7f83d4055e10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 --2- 172.21.15.3:0/1187237689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83f00a8c50 0x7f83f00636d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:28.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 >> 172.21.15.3:0/1187237689 conn(0x7f83f0060310 msgr2=0x7f83f005c200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:28.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 shutdown_connections 2022-07-21T01:42:28.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:28.963+0000 7f83effff700 1 -- 172.21.15.3:0/1187237689 wait complete. 2022-07-21T01:42:28.983 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:42:28.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:29.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:29.023 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:29.044 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:29.067 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:42:29.068 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:29.069 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:29.069 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:29.094 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:42:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:42:29.096 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:42:29.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:29.097 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:29.098 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:29.105 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:42:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:42:29.107 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:29.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:42:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048fd1e700 1 Processor -- start 2022-07-21T01:42:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048fd1e700 1 -- start start 2022-07-21T01:42:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048fd1e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04880b1f40 0x7f04880b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048fd1e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f04880a7250 0x7f04881448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048fd1e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04880b04e0 con 0x7f0488148530 2022-07-21T01:42:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048fd1e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04880a9250 con 0x7f04880a7250 2022-07-21T01:42:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048fd1e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04880b0780 con 0x7f04880b1f40 2022-07-21T01:42:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048d2b9700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0488148530 0x7f04880b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59426/0 (socket says 172.21.15.3:59426) 2022-07-21T01:42:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048d2b9700 1 -- 172.21.15.3:0/1955736708 learned_addr learned my addr 172.21.15.3:0/1955736708 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048daba700 1 --2- 172.21.15.3:0/1955736708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04880b1f40 0x7f04880b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048e2bb700 1 --2- 172.21.15.3:0/1955736708 >> v2:172.21.15.143:3300/0 conn(0x7f04880a7250 0x7f04881448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048cab8700 1 -- 172.21.15.3:0/1955736708 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822014235 0 0) 0x7f04880b04e0 con 0x7f0488148530 2022-07-21T01:42:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048cab8700 1 -- 172.21.15.3:0/1955736708 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0474003260 con 0x7f0488148530 2022-07-21T01:42:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048daba700 1 -- 172.21.15.3:0/1955736708 >> v2:172.21.15.143:3300/0 conn(0x7f04880a7250 msgr2=0x7f04881448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048daba700 1 --2- 172.21.15.3:0/1955736708 >> v2:172.21.15.143:3300/0 conn(0x7f04880a7250 0x7f04881448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048daba700 1 -- 172.21.15.3:0/1955736708 >> v1:172.21.15.3:6789/0 conn(0x7f0488148530 legacy=0x7f04880b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048daba700 1 -- 172.21.15.3:0/1955736708 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04880b0a20 con 0x7f04880b1f40 2022-07-21T01:42:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048daba700 1 --2- 172.21.15.3:0/1955736708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04880b1f40 0x7f04880b1160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0484002f70 tx=0x7f0484006df0).ready entity=mon.2 client_cookie=4f2d515313b9dcc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048cab8700 1 -- 172.21.15.3:0/1955736708 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f048400d4a0 con 0x7f04880b1f40 2022-07-21T01:42:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048cab8700 1 -- 172.21.15.3:0/1955736708 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0484000e90 con 0x7f04880b1f40 2022-07-21T01:42:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.215+0000 7f048cab8700 1 -- 172.21.15.3:0/1955736708 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f048400d7f0 con 0x7f04880b1f40 2022-07-21T01:42:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 -- 172.21.15.3:0/1955736708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04880b1f40 msgr2=0x7f04880b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 --2- 172.21.15.3:0/1955736708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04880b1f40 0x7f04880b1160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0484002f70 tx=0x7f0484006df0).stop 2022-07-21T01:42:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 -- 172.21.15.3:0/1955736708 shutdown_connections 2022-07-21T01:42:29.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 --2- 172.21.15.3:0/1955736708 >> v2:172.21.15.143:3300/0 conn(0x7f04880a7250 0x7f04881448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 --2- 172.21.15.3:0/1955736708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04880b1f40 0x7f04880b1160 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 -- 172.21.15.3:0/1955736708 >> 172.21.15.3:0/1955736708 conn(0x7f048805f840 msgr2=0x7f048805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:29.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 -- 172.21.15.3:0/1955736708 shutdown_connections 2022-07-21T01:42:29.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 -- 172.21.15.3:0/1955736708 wait complete. 2022-07-21T01:42:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 Processor -- start 2022-07-21T01:42:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.219+0000 7f048fd1e700 1 -- start start 2022-07-21T01:42:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048fd1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04880a7250 0x7f0488144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048fd1e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04880b1f40 0x7f0488148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048fd1e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0488148530 0x7f0488149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048fd1e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488156b50 con 0x7f04880a7250 2022-07-21T01:42:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048fd1e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488156df0 con 0x7f04880b1f40 2022-07-21T01:42:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048fd1e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0488157090 con 0x7f0488148530 2022-07-21T01:42:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048daba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04880a7250 0x7f0488144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048daba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04880a7250 0x7f0488144c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51152/0 (socket says 172.21.15.3:51152) 2022-07-21T01:42:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048daba700 1 -- 172.21.15.3:0/1714806725 learned_addr learned my addr 172.21.15.3:0/1714806725 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048d2b9700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04880b1f40 0x7f0488148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048e2bb700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0488148530 0x7f0488149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048daba700 1 -- 172.21.15.3:0/1714806725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0488148530 msgr2=0x7f0488149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048daba700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0488148530 0x7f0488149180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048daba700 1 -- 172.21.15.3:0/1714806725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04880b1f40 msgr2=0x7f0488148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048daba700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04880b1f40 0x7f0488148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048daba700 1 -- 172.21.15.3:0/1714806725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048400b180 con 0x7f04880a7250 2022-07-21T01:42:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048daba700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04880a7250 0x7f0488144c70 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f0484002f70 tx=0x7f0484014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f047effd700 1 -- 172.21.15.3:0/1714806725 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0484016070 con 0x7f04880a7250 2022-07-21T01:42:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f047effd700 1 -- 172.21.15.3:0/1714806725 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0484013800 con 0x7f04880a7250 2022-07-21T01:42:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04880abbe0 con 0x7f04880a7250 2022-07-21T01:42:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f047effd700 1 -- 172.21.15.3:0/1714806725 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0484013b60 con 0x7f04880a7250 2022-07-21T01:42:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04880ac1c0 con 0x7f04880a7250 2022-07-21T01:42:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.223+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04880574e0 con 0x7f04880a7250 2022-07-21T01:42:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.227+0000 7f047effd700 1 -- 172.21.15.3:0/1714806725 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0484025030 con 0x7f04880a7250 2022-07-21T01:42:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.227+0000 7f047effd700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053a70 0x7f0468055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.227+0000 7f047effd700 1 -- 172.21.15.3:0/1714806725 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f048406cd30 con 0x7f04880a7250 2022-07-21T01:42:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.227+0000 7f048d2b9700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053a70 0x7f0468055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.227+0000 7f048d2b9700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053a70 0x7f0468055f30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f048000d000 tx=0x7f0480009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.235+0000 7f047effd700 1 -- 172.21.15.3:0/1714806725 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0484043dd0 con 0x7f04880a7250 2022-07-21T01:42:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f04880447e0 con 0x7f04880a7250 2022-07-21T01:42:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f047effd700 1 -- 172.21.15.3:0/1714806725 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v143) v1 ==== 81+0+29225 (secure 0 0 0) 0x7f0484023070 con 0x7f04880a7250 2022-07-21T01:42:29.367 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:29.367 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:42:29.368 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 143, 2022-07-21T01:42:29.368 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:42:29.368 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:42:29.368 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:42:26.650455+0000", 2022-07-21T01:42:29.369 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:29.369 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:29.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:42:29.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:42:29.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:42:29.370 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:42:29.370 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:42:29.370 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:42:29.370 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:42:29.371 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:29.371 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:42:29.371 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:42:29.371 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:42:29.372 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:42:29.372 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:42:29.372 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 26, 2022-07-21T01:42:29.372 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:42:29.373 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:42:29.373 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:42:29.373 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:42:29.373 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:42:29.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:42:29.374 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:42:29.374 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:42:29.375 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:29.375 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:29.375 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:29.376 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:29.376 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:29.376 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:29.376 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:29.377 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:29.377 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:42:29.377 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:42:29.377 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:42:29.377 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:42:29.378 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:42:29.378 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:29.378 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:29.378 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:29.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:29.379 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:29.379 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:29.379 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:29.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.380 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:42:29.380 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:29.380 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:29.381 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:29.381 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:29.381 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:29.381 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:29.382 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:29.382 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:29.382 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:29.382 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:29.383 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:29.383 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:29.383 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:29.383 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:29.384 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:29.384 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:29.384 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:29.384 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:29.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:29.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:29.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:29.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:29.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:29.386 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:29.386 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:29.386 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:29.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.387 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:29.387 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:29.388 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:29.388 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:29.388 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:29.388 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:29.389 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:29.389 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:29.389 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:29.389 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:29.390 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:29.390 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:42:29.390 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:42:29.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.391 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:29.391 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:42:29.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.392 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:42:29.392 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:42:29.392 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:42:29.393 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:29.393 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:29.393 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:29.393 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:29.394 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:29.394 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:29.394 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:29.394 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:29.395 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:42:29.395 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:42:29.395 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:42:29.395 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:42:29.396 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:42:29.396 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:29.396 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:29.396 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:29.397 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:29.397 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:29.398 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:29.398 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:29.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.399 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:42:29.399 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:29.399 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:29.399 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:29.400 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:29.400 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:29.400 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:29.400 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:29.401 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:29.401 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:29.401 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:29.401 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:29.402 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:29.402 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:29.402 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:29.402 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:29.403 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:29.403 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:29.403 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:29.404 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:29.404 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:29.404 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:29.404 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:29.405 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:29.405 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:29.405 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:29.405 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:29.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.406 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:29.406 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:29.407 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:29.407 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:29.407 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:29.407 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:29.408 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:29.408 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:29.408 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:29.408 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:29.409 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:29.409 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:42:29.409 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:29.409 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:42:29.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.410 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.410 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:29.410 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:42:29.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.411 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:29.411 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:42:29.411 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:29.411 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:29.411 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:29.412 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:29.412 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:29.412 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:29.412 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:29.413 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 137, 2022-07-21T01:42:29.413 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:29.413 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:29.413 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:29.414 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.414 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:42:29.415 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:29.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.415 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.416 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:42:29.416 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:29.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.416 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.417 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:29.417 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.417 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.418 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:42:29.418 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:29.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.418 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.419 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:42:29.419 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:29.419 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.419 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.420 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:29.420 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.421 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:42:29.421 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:29.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.422 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:42:29.422 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:29.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.423 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.423 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:29.423 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.424 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:42:29.424 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:29.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.425 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.425 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:42:29.425 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:29.426 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.426 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.427 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:42:29.427 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:42:29.427 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:42:29.427 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:42:29.427 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:29.428 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:29.428 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:29.428 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.429 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:29.429 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:42:29.429 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:29.429 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:29.430 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:29.430 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:29.430 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:29.431 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:29.431 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:29.431 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:29.431 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:29.431 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:29.432 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:29.432 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.432 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.433 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:42:29.433 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:29.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.434 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.434 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:42:29.434 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:29.434 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.434 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.435 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:29.435 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.436 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:42:29.436 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:29.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.437 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:42:29.437 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:29.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.438 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.438 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:29.439 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.440 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:42:29.440 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:29.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.441 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.441 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:42:29.441 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:29.441 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.442 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.442 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:29.442 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.443 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.443 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:42:29.443 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:29.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.444 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:42:29.445 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:29.445 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.445 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.446 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:42:29.446 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:42:29.446 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:42:29.446 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:42:29.446 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:29.447 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:29.447 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:29.447 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.448 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:29.448 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:42:29.448 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:29.449 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:29.449 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:29.449 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:29.449 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:29.450 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:29.450 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:29.450 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:29.450 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:29.451 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:29.451 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:29.451 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.451 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.452 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:42:29.452 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:29.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.453 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.453 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:42:29.453 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:29.453 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.454 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.454 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:29.454 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.455 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.455 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:42:29.455 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:29.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.456 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:42:29.456 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:29.457 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.457 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.457 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:29.457 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.458 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.458 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:42:29.458 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:29.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.459 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:42:29.460 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:29.460 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.460 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.461 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:29.461 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.461 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.461 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:42:29.462 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:29.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.463 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:42:29.463 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:29.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.464 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:42:29.464 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:42:29.464 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:42:29.464 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:42:29.465 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:29.465 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:29.465 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:29.465 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:29.466 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:42:29.467 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:29.467 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:29.467 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:29.467 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:29.467 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:29.468 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:29.468 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:29.468 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 137, 2022-07-21T01:42:29.468 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:29.469 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:29.469 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:29.469 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.470 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.470 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:42:29.470 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:29.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.481 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.482 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:42:29.482 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:29.482 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.482 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.483 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:29.483 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.484 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.484 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:42:29.484 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:29.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.485 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:42:29.485 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:29.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.486 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.486 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:29.486 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.487 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.487 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:42:29.487 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:29.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.488 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.488 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:42:29.488 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:29.488 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.489 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.489 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:29.489 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.490 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:42:29.490 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:29.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.491 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:42:29.491 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:29.492 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.492 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.492 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:42:29.493 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:42:29.493 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:42:29.493 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:42:29.493 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:29.493 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:29.494 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:29.494 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.495 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:29.495 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:42:29.495 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:29.495 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:29.495 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:29.496 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:29.496 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:29.496 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:29.496 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:29.497 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:29.497 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:29.497 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:29.497 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:29.497 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.498 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.498 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:42:29.498 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:29.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.499 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.499 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:42:29.500 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:29.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.500 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.501 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:29.501 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.501 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.501 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:42:29.502 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:29.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.502 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.503 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:42:29.503 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:29.503 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.503 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.504 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:29.504 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.505 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.505 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:42:29.505 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:29.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.506 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.506 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:42:29.506 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:29.506 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.507 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.507 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:29.507 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.508 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:42:29.508 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:29.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.509 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.509 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:42:29.509 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:29.510 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.510 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.510 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:42:29.511 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:42:29.511 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:42:29.511 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:42:29.511 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:29.511 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:29.512 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:29.512 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.512 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:29.513 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:42:29.513 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:29.513 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:29.513 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:29.514 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:29.514 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:29.514 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:29.514 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:29.515 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 131, 2022-07-21T01:42:29.515 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:29.515 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:29.515 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:29.516 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.516 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.516 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:42:29.516 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:29.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.517 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.517 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:42:29.518 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:29.518 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.518 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.518 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:29.519 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.519 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.519 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:42:29.520 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:29.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.520 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.521 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:42:29.521 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:29.521 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.521 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.522 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:29.522 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.522 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.523 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:42:29.523 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:29.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.524 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.524 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:42:29.524 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:29.524 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.524 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.525 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:29.525 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.525 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.526 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:42:29.526 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:29.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.527 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:42:29.527 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:29.527 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.528 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.528 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:42:29.528 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:42:29.528 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:42:29.529 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:42:29.529 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:29.529 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:29.529 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:29.530 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.530 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:29.531 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:42:29.531 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:29.531 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:29.531 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:29.531 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:29.532 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:29.532 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:29.532 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:29.532 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:29.533 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:29.533 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:29.533 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:29.533 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.534 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:42:29.534 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:29.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.535 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.535 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:42:29.535 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:29.536 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.536 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.536 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:29.536 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.537 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.537 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:42:29.537 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:29.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.538 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:42:29.538 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:29.539 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.539 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.539 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:29.540 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.540 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.540 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:42:29.541 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:29.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.541 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.541 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:42:29.542 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:29.542 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.542 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.543 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:29.543 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.544 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:42:29.544 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:29.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.544 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.545 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:42:29.545 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:29.545 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.545 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.546 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:42:29.546 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:42:29.546 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:42:29.546 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:42:29.547 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:29.547 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:29.547 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:29.547 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.548 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:29.548 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:42:29.548 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:29.549 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:29.549 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:29.549 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:29.549 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:29.549 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:29.550 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:29.550 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 138, 2022-07-21T01:42:29.550 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:29.550 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:29.551 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:29.551 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.551 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.552 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:42:29.552 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:29.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.553 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.553 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:42:29.553 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:29.553 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.553 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.554 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:29.554 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.555 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.555 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:42:29.555 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:29.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.556 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.556 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:42:29.556 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:29.557 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.557 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.557 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:29.558 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.558 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:42:29.558 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:29.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.559 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.559 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:42:29.560 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:29.560 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.560 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.561 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:29.561 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:29.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.561 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:29.562 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:42:29.562 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:29.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.562 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:29.563 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:42:29.563 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:29.563 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.563 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.564 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:42:29.564 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:42:29.564 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:42:29.565 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:42:29.565 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:29.565 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:29.565 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:29.565 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:29.566 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.566 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:29.566 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:42:29.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.567 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:29.567 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:29.567 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:29.567 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:29.568 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:29.568 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:29.568 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:42:29.568 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:29.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.569 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:29.569 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:29.569 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:29.570 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:29.570 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:29.570 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:29.570 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:42:29.571 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:29.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.571 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:29.572 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:29.572 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:29.572 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:29.572 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:29.572 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:29.573 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:42:29.573 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:29.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.574 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:29.574 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:29.574 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:29.574 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:29.575 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:29.575 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:29.575 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:42:29.575 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:29.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.576 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:29.576 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:29.576 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:29.577 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:29.577 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:29.577 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:29.577 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:42:29.578 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:29.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.578 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:29.579 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:29.579 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:29.579 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:29.579 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:29.579 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:29.580 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:42:29.580 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:29.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.581 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:29.581 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:29.581 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:29.581 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:29.582 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:29.582 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:29.582 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:42:29.582 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:29.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:29.583 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:29.583 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:29.583 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:29.584 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:29.584 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:29.584 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:29.584 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:42:29.585 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:29.585 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.585 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:29.585 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:42:29.586 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:42:29.586 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:42:29.586 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:42:29.586 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:42:29.587 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6837/506442168": "2022-07-22T01:42:23.150104+0000", 2022-07-21T01:42:29.587 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6836/506442168": "2022-07-22T01:42:23.150104+0000" 2022-07-21T01:42:29.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.587 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:42:29.587 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:42:29.588 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:42:29.588 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:29.588 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:42:29.588 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:42:29.589 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:29.589 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:42:29.589 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:29.589 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:29.589 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:42:29.590 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:42:29.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.590 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:42:29.590 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:29.591 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:29.591 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:42:29.591 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:29.591 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:29.592 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:42:29.592 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:29.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:29.592 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:42:29.592 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:42:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:42:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:42:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:42:29.593 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:42:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053a70 msgr2=0x7f0468055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053a70 0x7f0468055f30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f048000d000 tx=0x7f0480009040).stop 2022-07-21T01:42:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04880a7250 msgr2=0x7f0488144c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04880a7250 0x7f0488144c70 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f0484002f70 tx=0x7f0484014000).stop 2022-07-21T01:42:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 shutdown_connections 2022-07-21T01:42:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0488148530 0x7f0488149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053a70 0x7f0468055f30 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04880b1f40 0x7f0488148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 --2- 172.21.15.3:0/1714806725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04880a7250 0x7f0488144c70 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 >> 172.21.15.3:0/1714806725 conn(0x7f048805f840 msgr2=0x7f04880b4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.363+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 shutdown_connections 2022-07-21T01:42:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.367+0000 7f048fd1e700 1 -- 172.21.15.3:0/1714806725 wait complete. 2022-07-21T01:42:29.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:29.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.707+0000 7fc0b8d92700 1 Processor -- start 2022-07-21T01:42:29.712 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.707+0000 7fc0b8d92700 1 -- start start 2022-07-21T01:42:29.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc0b40b2930 0x7fc0b4147600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 0x7fc0b414b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b40b2d10 con 0x7fc0b405c620 2022-07-21T01:42:29.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b4145340 con 0x7fc0b40b2930 2022-07-21T01:42:29.715 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b414ac60 con 0x7fc0b40b1ea0 2022-07-21T01:42:29.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc0b40b2930 0x7fc0b4147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b3fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc0b405c620 0x7fc0b40a8d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59444/0 (socket says 172.21.15.3:59444) 2022-07-21T01:42:29.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b3fff700 1 -- 172.21.15.3:0/3797731901 learned_addr learned my addr 172.21.15.3:0/3797731901 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:29.716 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b2ffd700 1 --2- 172.21.15.3:0/3797731901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 0x7fc0b414b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b37fe700 1 -- 172.21.15.3:0/3797731901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 msgr2=0x7fc0b414b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:29.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b37fe700 1 --2- 172.21.15.3:0/3797731901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 0x7fc0b414b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b37fe700 1 -- 172.21.15.3:0/3797731901 >> v1:172.21.15.3:6789/0 conn(0x7fc0b405c620 legacy=0x7fc0b40a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b37fe700 1 -- 172.21.15.3:0/3797731901 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b40b1050 con 0x7fc0b40b2930 2022-07-21T01:42:29.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b37fe700 1 --2- 172.21.15.3:0/3797731901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 0x7fc0b4147600 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7fc0a4002f70 tx=0x7fc0a4006df0).ready entity=mon.0 client_cookie=23806f09fbf2de9b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:29.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b27fc700 1 -- 172.21.15.3:0/3797731901 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0a400d570 con 0x7fc0b40b2930 2022-07-21T01:42:29.718 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b27fc700 1 -- 172.21.15.3:0/3797731901 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a4000e90 con 0x7fc0b40b2930 2022-07-21T01:42:29.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b27fc700 1 -- 172.21.15.3:0/3797731901 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0a400d8c0 con 0x7fc0b40b2930 2022-07-21T01:42:29.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 -- 172.21.15.3:0/3797731901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 msgr2=0x7fc0b4147600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 --2- 172.21.15.3:0/3797731901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 0x7fc0b4147600 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7fc0a4002f70 tx=0x7fc0a4006df0).stop 2022-07-21T01:42:29.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 -- 172.21.15.3:0/3797731901 shutdown_connections 2022-07-21T01:42:29.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 --2- 172.21.15.3:0/3797731901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 0x7fc0b414b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 --2- 172.21.15.3:0/3797731901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 0x7fc0b4147600 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 -- 172.21.15.3:0/3797731901 >> 172.21.15.3:0/3797731901 conn(0x7fc0b40612d0 msgr2=0x7fc0b4062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:29.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 -- 172.21.15.3:0/3797731901 shutdown_connections 2022-07-21T01:42:29.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.711+0000 7fc0b8d92700 1 -- 172.21.15.3:0/3797731901 wait complete. 2022-07-21T01:42:29.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b8d92700 1 Processor -- start 2022-07-21T01:42:29.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b8d92700 1 -- start start 2022-07-21T01:42:29.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b8d92700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b405c620 0x7fc0b4146770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b8d92700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 0x7fc0b40ac150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b8d92700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 0x7fc0b40ac640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b8d92700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41544b0 con 0x7fc0b40b2930 2022-07-21T01:42:29.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b8d92700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4156940 con 0x7fc0b405c620 2022-07-21T01:42:29.726 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b8d92700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4156be0 con 0x7fc0b40b1ea0 2022-07-21T01:42:29.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 0x7fc0b40ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 0x7fc0b40ac640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51168/0 (socket says 172.21.15.3:51168) 2022-07-21T01:42:29.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b3fff700 1 -- 172.21.15.3:0/2859119365 learned_addr learned my addr 172.21.15.3:0/2859119365 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:29.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b37fe700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b405c620 0x7fc0b4146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.715+0000 7fc0b2ffd700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 0x7fc0b40ac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b3fff700 1 -- 172.21.15.3:0/2859119365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 msgr2=0x7fc0b40ac150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b3fff700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 0x7fc0b40ac150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b3fff700 1 -- 172.21.15.3:0/2859119365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b405c620 msgr2=0x7fc0b4146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b3fff700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b405c620 0x7fc0b4146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b3fff700 1 -- 172.21.15.3:0/2859119365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a400b180 con 0x7fc0b40b2930 2022-07-21T01:42:29.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b3fff700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 0x7fc0b40ac640 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7fc0b4069d70 tx=0x7fc0ac00c780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:29.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b0ff9700 1 -- 172.21.15.3:0/2859119365 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0ac005b80 con 0x7fc0b40b2930 2022-07-21T01:42:29.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b0ff9700 1 -- 172.21.15.3:0/2859119365 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ac005ce0 con 0x7fc0b40b2930 2022-07-21T01:42:29.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b4156e80 con 0x7fc0b40b2930 2022-07-21T01:42:29.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b0ff9700 1 -- 172.21.15.3:0/2859119365 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0ac0072c0 con 0x7fc0b40b2930 2022-07-21T01:42:29.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b4144d20 con 0x7fc0b40b2930 2022-07-21T01:42:29.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b4157760 con 0x7fc0b40b2930 2022-07-21T01:42:29.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b0ff9700 1 -- 172.21.15.3:0/2859119365 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc0ac010070 con 0x7fc0b40b2930 2022-07-21T01:42:29.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b0ff9700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc094053a70 0x7fc094055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:29.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b37fe700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc094053a70 0x7fc094055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:29.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.719+0000 7fc0b0ff9700 1 -- 172.21.15.3:0/2859119365 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fc0ac06b580 con 0x7fc0b40b2930 2022-07-21T01:42:29.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.723+0000 7fc0b37fe700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc094053a70 0x7fc094055f30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fc0a401a000 tx=0x7fc0a4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:29.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.727+0000 7fc0b0ff9700 1 -- 172.21.15.3:0/2859119365 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc0ac042620 con 0x7fc0b40b2930 2022-07-21T01:42:29.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0b40574e0 con 0x7fc0b40b2930 2022-07-21T01:42:29.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b0ff9700 1 -- 172.21.15.3:0/2859119365 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+1550 (secure 0 0 0) 0x7fc0ac018020 con 0x7fc0b40b2930 2022-07-21T01:42:29.902 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:29.902 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":73,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/567682837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":567682837},{"type":"v1","addr":"172.21.15.143:6833","nonce":567682837}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/140242740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140242740},{"type":"v1","addr":"172.21.15.3:6835","nonce":140242740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/389188851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":389188851},{"type":"v1","addr":"172.21.15.3:6837","nonce":389188851}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":73}],"filesystems":[]} 2022-07-21T01:42:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc094053a70 msgr2=0x7fc094055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc094053a70 0x7fc094055f30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fc0a401a000 tx=0x7fc0a4014000).stop 2022-07-21T01:42:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 msgr2=0x7fc0b40ac640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 0x7fc0b40ac640 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7fc0b4069d70 tx=0x7fc0ac00c780).stop 2022-07-21T01:42:29.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 shutdown_connections 2022-07-21T01:42:29.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0b40b2930 0x7fc0b40ac640 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b40b1ea0 0x7fc0b40ac150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc094053a70 0x7fc094055f30 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 --2- 172.21.15.3:0/2859119365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b405c620 0x7fc0b4146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:29.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 >> 172.21.15.3:0/2859119365 conn(0x7fc0b40612d0 msgr2=0x7fc0b4143af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:29.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 shutdown_connections 2022-07-21T01:42:29.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:29.899+0000 7fc0b8d92700 1 -- 172.21.15.3:0/2859119365 wait complete. 2022-07-21T01:42:29.908 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 73 2022-07-21T01:42:29.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 Processor -- start 2022-07-21T01:42:30.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 -- start start 2022-07-21T01:42:30.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f59600b28b0 0x7f596013e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59600b1180 0x7f596013e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5960141210 0x7f59601416c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5960140e20 con 0x7f59600b28b0 2022-07-21T01:42:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59600aac20 con 0x7f59600b1180 2022-07-21T01:42:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59600b2c90 con 0x7f5960141210 2022-07-21T01:42:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f59600b28b0 0x7f596013e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f59600b28b0 0x7f596013e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51174/0 (socket says 172.21.15.3:51174) 2022-07-21T01:42:30.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595ffff700 1 -- 172.21.15.3:0/30220121 learned_addr learned my addr 172.21.15.3:0/30220121 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:30.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f59577fe700 1 --2- 172.21.15.3:0/30220121 >> v2:172.21.15.143:3300/0 conn(0x7f59600b1180 0x7f596013e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5964951700 1 --2- 172.21.15.3:0/30220121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5960141210 0x7f59601416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595ffff700 1 -- 172.21.15.3:0/30220121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5960141210 msgr2=0x7f59601416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:30.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595ffff700 1 --2- 172.21.15.3:0/30220121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5960141210 0x7f59601416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595ffff700 1 -- 172.21.15.3:0/30220121 >> v2:172.21.15.143:3300/0 conn(0x7f59600b1180 msgr2=0x7f596013e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:30.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595ffff700 1 --2- 172.21.15.3:0/30220121 >> v2:172.21.15.143:3300/0 conn(0x7f59600b1180 0x7f596013e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595ffff700 1 -- 172.21.15.3:0/30220121 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59600aa440 con 0x7f59600b28b0 2022-07-21T01:42:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595ffff700 1 --2- 172.21.15.3:0/30220121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59600b28b0 0x7f596013e1c0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7f5950002f70 tx=0x7f5950006eb0).ready entity=mon.0 client_cookie=a75d83203b24001d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595f7fe700 1 -- 172.21.15.3:0/30220121 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f595000d570 con 0x7f59600b28b0 2022-07-21T01:42:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595f7fe700 1 -- 172.21.15.3:0/30220121 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5950000e90 con 0x7f59600b28b0 2022-07-21T01:42:30.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f595f7fe700 1 -- 172.21.15.3:0/30220121 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f595000d900 con 0x7f59600b28b0 2022-07-21T01:42:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 -- 172.21.15.3:0/30220121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59600b28b0 msgr2=0x7f596013e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 --2- 172.21.15.3:0/30220121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59600b28b0 0x7f596013e1c0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7f5950002f70 tx=0x7f5950006eb0).stop 2022-07-21T01:42:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 -- 172.21.15.3:0/30220121 shutdown_connections 2022-07-21T01:42:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 --2- 172.21.15.3:0/30220121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5960141210 0x7f59601416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 --2- 172.21.15.3:0/30220121 >> v2:172.21.15.143:3300/0 conn(0x7f59600b1180 0x7f596013e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 --2- 172.21.15.3:0/30220121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59600b28b0 0x7f596013e1c0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 -- 172.21.15.3:0/30220121 >> 172.21.15.3:0/30220121 conn(0x7f5960060310 msgr2=0x7f59600b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:30.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 -- 172.21.15.3:0/30220121 shutdown_connections 2022-07-21T01:42:30.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.019+0000 7f5965152700 1 -- 172.21.15.3:0/30220121 wait complete. 2022-07-21T01:42:30.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5965152700 1 Processor -- start 2022-07-21T01:42:30.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5965152700 1 -- start start 2022-07-21T01:42:30.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5965152700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600b1180 0x7f5960063eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5965152700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59600b28b0 0x7f59600b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5965152700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5960141210 0x7f59600b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5965152700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59601422c0 con 0x7f5960141210 2022-07-21T01:42:30.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5965152700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5960150c20 con 0x7f59600b1180 2022-07-21T01:42:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5965152700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5960150ec0 con 0x7f59600b28b0 2022-07-21T01:42:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5964951700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5960141210 0x7f59600b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5964951700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5960141210 0x7f59600b4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51184/0 (socket says 172.21.15.3:51184) 2022-07-21T01:42:30.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5964951700 1 -- 172.21.15.3:0/3766186494 learned_addr learned my addr 172.21.15.3:0/3766186494 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:30.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f595ffff700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600b1180 0x7f5960063eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f59577fe700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59600b28b0 0x7f59600b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5964951700 1 -- 172.21.15.3:0/3766186494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59600b28b0 msgr2=0x7f59600b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:30.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5964951700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59600b28b0 0x7f59600b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5964951700 1 -- 172.21.15.3:0/3766186494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600b1180 msgr2=0x7f5960063eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:30.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5964951700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600b1180 0x7f5960063eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5964951700 1 -- 172.21.15.3:0/3766186494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595000b1e0 con 0x7f5960141210 2022-07-21T01:42:30.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f5964951700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5960141210 0x7f59600b4460 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f5958006e00 tx=0x7f59580091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.023+0000 7f595dffb700 1 -- 172.21.15.3:0/3766186494 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f595800c6a0 con 0x7f5960141210 2022-07-21T01:42:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f595dffb700 1 -- 172.21.15.3:0/3766186494 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595800c800 con 0x7f5960141210 2022-07-21T01:42:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5960066dc0 con 0x7f5960141210 2022-07-21T01:42:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f595dffb700 1 -- 172.21.15.3:0/3766186494 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f595800cb60 con 0x7f5960141210 2022-07-21T01:42:30.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59600b4af0 con 0x7f5960141210 2022-07-21T01:42:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f595dffb700 1 -- 172.21.15.3:0/3766186494 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5958022070 con 0x7f5960141210 2022-07-21T01:42:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59600574e0 con 0x7f5960141210 2022-07-21T01:42:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f595dffb700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59440539e0 0x7f5944055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f595dffb700 1 -- 172.21.15.3:0/3766186494 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f595806c390 con 0x7f5960141210 2022-07-21T01:42:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f595ffff700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59440539e0 0x7f5944055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.027+0000 7f595ffff700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59440539e0 0x7f5944055ea0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f595001c000 tx=0x7f5950015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:30.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.035+0000 7f595dffb700 1 -- 172.21.15.3:0/3766186494 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5958043430 con 0x7f5960141210 2022-07-21T01:42:30.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.195+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5960069a70 con 0x7f5960141210 2022-07-21T01:42:30.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.195+0000 7f595dffb700 1 -- 172.21.15.3:0/3766186494 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+1550 (secure 0 0 0) 0x7f5958016090 con 0x7f5960141210 2022-07-21T01:42:30.200 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:30.200 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":73,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6926,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.143:6833/567682837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":567682837},{"type":"v1","addr":"172.21.15.143:6833","nonce":567682837}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/140242740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":140242740},{"type":"v1","addr":"172.21.15.3:6835","nonce":140242740}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":69},{"gid":8310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/389188851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":389188851},{"type":"v1","addr":"172.21.15.3:6837","nonce":389188851}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":73}],"filesystems":[]} 2022-07-21T01:42:30.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59440539e0 msgr2=0x7f5944055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:30.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59440539e0 0x7f5944055ea0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f595001c000 tx=0x7f5950015000).stop 2022-07-21T01:42:30.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5960141210 msgr2=0x7f59600b4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:30.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5960141210 0x7f59600b4460 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f5958006e00 tx=0x7f59580091d0).stop 2022-07-21T01:42:30.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 shutdown_connections 2022-07-21T01:42:30.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5960141210 0x7f59600b4460 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59600b28b0 0x7f59600b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f59440539e0 0x7f5944055ea0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 --2- 172.21.15.3:0/3766186494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600b1180 0x7f5960063eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 >> 172.21.15.3:0/3766186494 conn(0x7f5960060310 msgr2=0x7f596013e320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:30.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 shutdown_connections 2022-07-21T01:42:30.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.199+0000 7f5965152700 1 -- 172.21.15.3:0/3766186494 wait complete. 2022-07-21T01:42:30.205 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 73 2022-07-21T01:42:30.211 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:42:30.212 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:42:30.212 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:42:30.213 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:42:30.213 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:42:30.214 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:42:30.214 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:42:30.215 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:42:30.215 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:42:30.216 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:42:30.216 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:42:30.217 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:42:30.230 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:42:30.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:42:30.232 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:42:30.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:42:30.234 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:42:30.235 DEBUG:teuthology.parallel:result is None 2022-07-21T01:42:30.236 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:42:30.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:42:30.237 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:42:30.238 DEBUG:teuthology.parallel:result is None 2022-07-21T01:42:30.240 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:42:30.241 DEBUG:teuthology.parallel:result is None 2022-07-21T01:42:30.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:42:30.298 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:42:30.310 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:42:30.312 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:42:30.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc7fff700 1 Processor -- start 2022-07-21T01:42:30.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc7fff700 1 -- start start 2022-07-21T01:42:30.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8dc80b29e0 0x7f8dc80b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc7fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8dc813c870 0x7f8dc813ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc80ad0b0 0x7f8dc80ad520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc7fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc80b2dc0 con 0x7f8dc80b29e0 2022-07-21T01:42:30.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc7fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc80b17f0 con 0x7f8dc813c870 2022-07-21T01:42:30.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80b1a90 con 0x7f8dc80ad0b0 2022-07-21T01:42:30.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc6ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8dc80b29e0 0x7f8dc80b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc6ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8dc80b29e0 0x7f8dc80b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51212/0 (socket says 172.21.15.3:51212) 2022-07-21T01:42:30.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc6ffd700 1 -- 172.21.15.3:0/1233094807 learned_addr learned my addr 172.21.15.3:0/1233094807 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:30.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc77fe700 1 --2- 172.21.15.3:0/1233094807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc80ad0b0 0x7f8dc80ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc67fc700 1 --2- 172.21.15.3:0/1233094807 >> v2:172.21.15.143:3300/0 conn(0x7f8dc813c870 0x7f8dc813ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc6ffd700 1 -- 172.21.15.3:0/1233094807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc80ad0b0 msgr2=0x7f8dc80ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:30.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc6ffd700 1 --2- 172.21.15.3:0/1233094807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc80ad0b0 0x7f8dc80ad520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc6ffd700 1 -- 172.21.15.3:0/1233094807 >> v2:172.21.15.143:3300/0 conn(0x7f8dc813c870 msgr2=0x7f8dc813ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:30.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc6ffd700 1 --2- 172.21.15.3:0/1233094807 >> v2:172.21.15.143:3300/0 conn(0x7f8dc813c870 0x7f8dc813ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc6ffd700 1 -- 172.21.15.3:0/1233094807 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80b1d30 con 0x7f8dc80b29e0 2022-07-21T01:42:30.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc6ffd700 1 --2- 172.21.15.3:0/1233094807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 0x7f8dc80b0c30 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f8db8002f70 tx=0x7f8db8006eb0).ready entity=mon.0 client_cookie=9407ee88d95d71fd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:30.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc5ffb700 1 -- 172.21.15.3:0/1233094807 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8db800d570 con 0x7f8dc80b29e0 2022-07-21T01:42:30.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc5ffb700 1 -- 172.21.15.3:0/1233094807 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db8000e90 con 0x7f8dc80b29e0 2022-07-21T01:42:30.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.343+0000 7f8dc5ffb700 1 -- 172.21.15.3:0/1233094807 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8db800d900 con 0x7f8dc80b29e0 2022-07-21T01:42:30.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 -- 172.21.15.3:0/1233094807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 msgr2=0x7f8dc80b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:30.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/1233094807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 0x7f8dc80b0c30 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f8db8002f70 tx=0x7f8db8006eb0).stop 2022-07-21T01:42:30.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 -- 172.21.15.3:0/1233094807 shutdown_connections 2022-07-21T01:42:30.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/1233094807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc80ad0b0 0x7f8dc80ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/1233094807 >> v2:172.21.15.143:3300/0 conn(0x7f8dc813c870 0x7f8dc813ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/1233094807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 0x7f8dc80b0c30 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 -- 172.21.15.3:0/1233094807 >> 172.21.15.3:0/1233094807 conn(0x7f8dc80613c0 msgr2=0x7f8dc8062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:30.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 -- 172.21.15.3:0/1233094807 shutdown_connections 2022-07-21T01:42:30.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 -- 172.21.15.3:0/1233094807 wait complete. 2022-07-21T01:42:30.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 Processor -- start 2022-07-21T01:42:30.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 -- start start 2022-07-21T01:42:30.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc80ad0b0 0x7f8dc8060090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 0x7f8dc8135e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc813c870 0x7f8dc8136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8149940 con 0x7f8dc80b29e0 2022-07-21T01:42:30.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8149be0 con 0x7f8dc80ad0b0 2022-07-21T01:42:30.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8145230 con 0x7f8dc813c870 2022-07-21T01:42:30.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc67fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 0x7f8dc8135e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc67fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 0x7f8dc8135e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51220/0 (socket says 172.21.15.3:51220) 2022-07-21T01:42:30.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc67fc700 1 -- 172.21.15.3:0/3545538587 learned_addr learned my addr 172.21.15.3:0/3545538587 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:30.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc6ffd700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc80ad0b0 0x7f8dc8060090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc77fe700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc813c870 0x7f8dc8136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc67fc700 1 -- 172.21.15.3:0/3545538587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc813c870 msgr2=0x7f8dc8136320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:30.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc67fc700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc813c870 0x7f8dc8136320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc67fc700 1 -- 172.21.15.3:0/3545538587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc80ad0b0 msgr2=0x7f8dc8060090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:30.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc67fc700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc80ad0b0 0x7f8dc8060090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:30.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.347+0000 7f8dc67fc700 1 -- 172.21.15.3:0/3545538587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db800b1e0 con 0x7f8dc80b29e0 2022-07-21T01:42:30.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dc67fc700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 0x7f8dc8135e30 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f8db00030b0 tx=0x7f8db0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:30.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dbffff700 1 -- 172.21.15.3:0/3545538587 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8db0011780 con 0x7f8dc80b29e0 2022-07-21T01:42:30.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dbffff700 1 -- 172.21.15.3:0/3545538587 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db00118e0 con 0x7f8dc80b29e0 2022-07-21T01:42:30.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc81454d0 con 0x7f8dc80b29e0 2022-07-21T01:42:30.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dbffff700 1 -- 172.21.15.3:0/3545538587 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8db0011c40 con 0x7f8dc80b29e0 2022-07-21T01:42:30.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc81496e0 con 0x7f8dc80b29e0 2022-07-21T01:42:30.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc81369b0 con 0x7f8dc80b29e0 2022-07-21T01:42:30.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dbffff700 1 -- 172.21.15.3:0/3545538587 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8db0012420 con 0x7f8dc80b29e0 2022-07-21T01:42:30.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dbffff700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8dac053950 0x7f8dac055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:30.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dbffff700 1 -- 172.21.15.3:0/3545538587 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f8db006bbd0 con 0x7f8dc80b29e0 2022-07-21T01:42:30.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dc6ffd700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8dac053950 0x7f8dac055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:30.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.351+0000 7f8dc6ffd700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8dac053950 0x7f8dac055e10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f8db801c000 tx=0x7f8db8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:30.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.355+0000 7f8dbffff700 1 -- 172.21.15.3:0/3545538587 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8db0042c10 con 0x7f8dc80b29e0 2022-07-21T01:42:30.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:30.495+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f8dc80574e0 con 0x7f8dc80b29e0 2022-07-21T01:42:31.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.351+0000 7f8dbffff700 1 -- 172.21.15.3:0/3545538587 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v145) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8db0046230 con 0x7f8dc80b29e0 2022-07-21T01:42:31.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8dac053950 msgr2=0x7f8dac055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8dac053950 0x7f8dac055e10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f8db801c000 tx=0x7f8db8015000).stop 2022-07-21T01:42:31.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 msgr2=0x7f8dc8135e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 0x7f8dc8135e30 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f8db00030b0 tx=0x7f8db0008040).stop 2022-07-21T01:42:31.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 shutdown_connections 2022-07-21T01:42:31.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dc813c870 0x7f8dc8136320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8dc80b29e0 0x7f8dc8135e30 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8dac053950 0x7f8dac055e10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 --2- 172.21.15.3:0/3545538587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc80ad0b0 0x7f8dc8060090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 >> 172.21.15.3:0/3545538587 conn(0x7f8dc80613c0 msgr2=0x7f8dc80ad520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:31.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 shutdown_connections 2022-07-21T01:42:31.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.355+0000 7f8dc7fff700 1 -- 172.21.15.3:0/3545538587 wait complete. 2022-07-21T01:42:31.361 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:42:31.371 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:42:31.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:42:31.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea6396700 1 Processor -- start 2022-07-21T01:42:31.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea6396700 1 -- start start 2022-07-21T01:42:31.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea6396700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbea00614b0 0x7fbea0061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.498 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea6396700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbea00ad750 0x7fbea00b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea6396700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea00aee80 con 0x7fbea01405d0 2022-07-21T01:42:31.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea6396700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea00b70b0 con 0x7fbea00ad750 2022-07-21T01:42:31.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea6396700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea00af0b0 con 0x7fbea00614b0 2022-07-21T01:42:31.500 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea5b95700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbea01405d0 0x7fbea01466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59510/0 (socket says 172.21.15.3:59510) 2022-07-21T01:42:31.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea5b95700 1 -- 172.21.15.3:0/2652658012 learned_addr learned my addr 172.21.15.3:0/2652658012 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:31.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea4b93700 1 --2- 172.21.15.3:0/2652658012 >> v2:172.21.15.143:3300/0 conn(0x7fbea00ad750 0x7fbea00b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea5394700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbea00614b0 0x7fbea0061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.502 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407164521 0 0) 0x7fbea00aee80 con 0x7fbea01405d0 2022-07-21T01:42:31.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe8c003260 con 0x7fbea01405d0 2022-07-21T01:42:31.503 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641819260 0 0) 0x7fbe8c003260 con 0x7fbea01405d0 2022-07-21T01:42:31.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbea00614b0 msgr2=0x7fbea0061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:31.504 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbea00614b0 0x7fbea0061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 >> v2:172.21.15.143:3300/0 conn(0x7fbea00ad750 msgr2=0x7fbea00b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:31.505 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 --2- 172.21.15.3:0/2652658012 >> v2:172.21.15.143:3300/0 conn(0x7fbea00ad750 0x7fbea00b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea00af350 con 0x7fbea01405d0 2022-07-21T01:42:31.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fbe90001660 con 0x7fbea01405d0 2022-07-21T01:42:31.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea4b93700 1 --2- 172.21.15.3:0/2652658012 >> v2:172.21.15.143:3300/0 conn(0x7fbea00ad750 0x7fbea00b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:31.507 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 >> v1:172.21.15.3:6789/0 conn(0x7fbea01405d0 legacy=0x7fbea01466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe8c003750 0x7fbe8c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c004500 0x7fbe8c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8c004e70 0x7fbe8c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea00b70b0 con 0x7fbe8c003750 2022-07-21T01:42:31.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea00af0b0 con 0x7fbe8c004e70 2022-07-21T01:42:31.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c00bc40 con 0x7fbe8c004500 2022-07-21T01:42:31.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea5394700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbea00614b0 0x7fbea0061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:31.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea4b93700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c004500 0x7fbe8c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea5b95700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8c004e70 0x7fbe8c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.487+0000 7fbea5394700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe8c003750 0x7fbe8c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea5394700 1 -- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c004500 msgr2=0x7fbe8c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:31.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea5394700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c004500 0x7fbe8c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea5394700 1 -- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8c004e70 msgr2=0x7fbe8c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:31.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea5394700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8c004e70 0x7fbe8c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea5394700 1 -- 172.21.15.3:0/2652658012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c003260 con 0x7fbe8c003750 2022-07-21T01:42:31.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea5b95700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8c004e70 0x7fbe8c00b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:31.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea5394700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe8c003750 0x7fbe8c003b70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7fbe9400c000 tx=0x7fbe94006df0).ready entity=mon.0 client_cookie=6d04013f0fcd5147 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:31.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe9400e650 con 0x7fbe8c003750 2022-07-21T01:42:31.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9400e7b0 con 0x7fbe8c003750 2022-07-21T01:42:31.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbe9ffff700 1 -- 172.21.15.3:0/2652658012 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe9400eb10 con 0x7fbe8c003750 2022-07-21T01:42:31.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea6396700 1 -- 172.21.15.3:0/2652658012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe8c003750 msgr2=0x7fbe8c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea6396700 1 --2- 172.21.15.3:0/2652658012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe8c003750 0x7fbe8c003b70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7fbe9400c000 tx=0x7fbe94006df0).stop 2022-07-21T01:42:31.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea5b95700 1 -- 172.21.15.3:0/2652658012 reap_dead start 2022-07-21T01:42:31.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea6396700 1 -- 172.21.15.3:0/2652658012 shutdown_connections 2022-07-21T01:42:31.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea6396700 1 -- 172.21.15.3:0/2652658012 >> 172.21.15.3:0/2652658012 conn(0x7fbea0066f70 msgr2=0x7fbea0067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:31.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea6396700 1 -- 172.21.15.3:0/2652658012 shutdown_connections 2022-07-21T01:42:31.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.491+0000 7fbea6396700 1 -- 172.21.15.3:0/2652658012 wait complete. 2022-07-21T01:42:31.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 Processor -- start 2022-07-21T01:42:31.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 -- start start 2022-07-21T01:42:31.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c003750 0x7fbea00b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbea00a1420 0x7fbea00a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbea0155b30 0x7fbea0155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea0153000 con 0x7fbea0155b30 2022-07-21T01:42:31.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea00b3890 con 0x7fbea00a1420 2022-07-21T01:42:31.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea0153230 con 0x7fbe8c003750 2022-07-21T01:42:31.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5b95700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbea0155b30 0x7fbea0155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5b95700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbea0155b30 0x7fbea0155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51242/0 (socket says 172.21.15.3:51242) 2022-07-21T01:42:31.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5b95700 1 -- 172.21.15.3:0/1114195633 learned_addr learned my addr 172.21.15.3:0/1114195633 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:31.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea4b93700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbea00a1420 0x7fbea00a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5394700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c003750 0x7fbea00b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5b95700 1 -- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c003750 msgr2=0x7fbea00b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5b95700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c003750 0x7fbea00b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5b95700 1 -- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbea00a1420 msgr2=0x7fbea00a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5b95700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbea00a1420 0x7fbea00a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5b95700 1 -- 172.21.15.3:0/1114195633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea0146f40 con 0x7fbea0155b30 2022-07-21T01:42:31.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea4b93700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbea00a1420 0x7fbea00a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:31.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5394700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c003750 0x7fbea00b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:31.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea5b95700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbea0155b30 0x7fbea0155fe0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7fbe90000b60 tx=0x7fbea00af4e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:31.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbe9e7fc700 1 -- 172.21.15.3:0/1114195633 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe90011590 con 0x7fbea0155b30 2022-07-21T01:42:31.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbe9e7fc700 1 -- 172.21.15.3:0/1114195633 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe900116f0 con 0x7fbea0155b30 2022-07-21T01:42:31.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea00b2570 con 0x7fbea0155b30 2022-07-21T01:42:31.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbe9e7fc700 1 -- 172.21.15.3:0/1114195633 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe90011a50 con 0x7fbea0155b30 2022-07-21T01:42:31.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.495+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea00b3060 con 0x7fbea0155b30 2022-07-21T01:42:31.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.499+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea00b36d0 con 0x7fbea0155b30 2022-07-21T01:42:31.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.499+0000 7fbe9e7fc700 1 -- 172.21.15.3:0/1114195633 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbe90011bb0 con 0x7fbea0155b30 2022-07-21T01:42:31.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.499+0000 7fbe9e7fc700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe80053950 0x7fbe80055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.499+0000 7fbe9e7fc700 1 -- 172.21.15.3:0/1114195633 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fbe9006bf30 con 0x7fbea0155b30 2022-07-21T01:42:31.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.499+0000 7fbea5394700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe80053950 0x7fbe80055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.499+0000 7fbea5394700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe80053950 0x7fbe80055e10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7fbe9401c000 tx=0x7fbe94019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:31.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.507+0000 7fbe9e7fc700 1 -- 172.21.15.3:0/1114195633 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbe90042f90 con 0x7fbea0155b30 2022-07-21T01:42:31.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.671+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbea00af5f0 con 0x7fbea0155b30 2022-07-21T01:42:31.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.671+0000 7fbe9e7fc700 1 -- 172.21.15.3:0/1114195633 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fbe900465b0 con 0x7fbea0155b30 2022-07-21T01:42:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:31.677 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:42:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:42:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:42:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:42:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:42:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:42:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:42:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:42:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:42:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:42:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:42:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:42:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:42:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:42:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.694 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.694 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.694 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.694 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.695 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:42:31.695 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:42:31.695 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.696 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.696 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.696 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.697 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:42:31.697 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:42:31.698 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.698 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.698 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.698 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.699 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:42:31.699 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:42:31.700 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.700 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.700 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.700 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.701 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:42:31.702 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:42:31.702 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.702 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.703 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.703 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.704 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:42:31.704 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:42:31.705 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.705 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.705 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.705 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.706 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.706 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:42:31.707 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:42:31.707 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.707 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:31.707 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:42:31.707 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:42:31.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.708 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:42:31.709 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:42:31.709 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.709 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:31.709 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:42:31.710 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:42:31.710 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.710 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:42:31.711 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:42:31.711 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.711 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:42:31.711 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:31.712 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:31.712 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.712 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.713 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:42:31.713 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:42:31.713 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.714 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:42:31.714 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:42:31.714 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.714 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:31.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:31.715 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:42:31.715 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:42:31.716 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:31.716 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:42:31.716 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:42:31.717 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:31.717 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.717 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:31.717 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:31.717 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:42:31.719 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe80053950 msgr2=0x7fbe80055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe80053950 0x7fbe80055e10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7fbe9401c000 tx=0x7fbe94019000).stop 2022-07-21T01:42:31.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbea0155b30 msgr2=0x7fbea0155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbea0155b30 0x7fbea0155fe0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7fbe90000b60 tx=0x7fbea00af4e0).stop 2022-07-21T01:42:31.720 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 shutdown_connections 2022-07-21T01:42:31.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbea0155b30 0x7fbea0155fe0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fbe80053950 0x7fbe80055e10 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbea00a1420 0x7fbea00a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 --2- 172.21.15.3:0/1114195633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe8c003750 0x7fbea00b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.721 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 >> 172.21.15.3:0/1114195633 conn(0x7fbea0066f70 msgr2=0x7fbea00b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:31.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 shutdown_connections 2022-07-21T01:42:31.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.675+0000 7fbea6396700 1 -- 172.21.15.3:0/1114195633 wait complete. 2022-07-21T01:42:31.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8329788700 1 Processor -- start 2022-07-21T01:42:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8329788700 1 -- start start 2022-07-21T01:42:31.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8329788700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83240b1f40 0x7f83240b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8329788700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83240b04e0 con 0x7f8324148530 2022-07-21T01:42:31.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8329788700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83240b0780 con 0x7f83240a7250 2022-07-21T01:42:31.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8329788700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83240a9250 con 0x7f83240b1f40 2022-07-21T01:42:31.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f83227fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8324148530 0x7f83240b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59528/0 (socket says 172.21.15.3:59528) 2022-07-21T01:42:31.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f83227fc700 1 -- 172.21.15.3:0/835782993 learned_addr learned my addr 172.21.15.3:0/835782993 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:31.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8322ffd700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83240b1f40 0x7f83240b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042892400 0 0) 0x7f83240b04e0 con 0x7f8324148530 2022-07-21T01:42:31.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8304003260 con 0x7f8324148530 2022-07-21T01:42:31.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402194867 0 0) 0x7f8304003260 con 0x7f8324148530 2022-07-21T01:42:31.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83240b1f40 msgr2=0x7f83240b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:31.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83240b1f40 0x7f83240b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 >> v1:172.21.15.143:6789/0 conn(0x7f83240a7250 legacy=0x7f83241448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83240b0a20 con 0x7f8324148530 2022-07-21T01:42:31.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f831c0019e0 con 0x7f8324148530 2022-07-21T01:42:31.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 >> v1:172.21.15.3:6789/0 conn(0x7f8324148530 legacy=0x7f83240b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 0x7f8304003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8304004500 0x7f8304004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83240b1f40 0x7f8304004f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830400b800 con 0x7f8304004500 2022-07-21T01:42:31.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830400b970 con 0x7f8304003780 2022-07-21T01:42:31.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830400bb30 con 0x7f83240b1f40 2022-07-21T01:42:31.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8322ffd700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 0x7f8304003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f83227fc700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8304004500 0x7f8304004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f83237fe700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83240b1f40 0x7f8304004f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8322ffd700 1 -- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83240b1f40 msgr2=0x7f8304004f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:31.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8322ffd700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83240b1f40 0x7f8304004f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8322ffd700 1 -- 172.21.15.3:0/835782993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8304004500 msgr2=0x7f8304004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:31.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8322ffd700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8304004500 0x7f8304004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8322ffd700 1 -- 172.21.15.3:0/835782993 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8304003260 con 0x7f8304003780 2022-07-21T01:42:31.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f83227fc700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8304004500 0x7f8304004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:31.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.827+0000 7f8322ffd700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 0x7f8304003ba0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f830c00a000 tx=0x7f830c008040).ready entity=mon.1 client_cookie=182f3c23a96e28de server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:31.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f830c013670 con 0x7f8304003780 2022-07-21T01:42:31.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830c0137d0 con 0x7f8304003780 2022-07-21T01:42:31.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8321ffb700 1 -- 172.21.15.3:0/835782993 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f830c013b30 con 0x7f8304003780 2022-07-21T01:42:31.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8329788700 1 -- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 msgr2=0x7f8304003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8329788700 1 --2- 172.21.15.3:0/835782993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 0x7f8304003ba0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f830c00a000 tx=0x7f830c008040).stop 2022-07-21T01:42:31.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f83237fe700 1 -- 172.21.15.3:0/835782993 reap_dead start 2022-07-21T01:42:31.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8329788700 1 -- 172.21.15.3:0/835782993 shutdown_connections 2022-07-21T01:42:31.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8329788700 1 -- 172.21.15.3:0/835782993 >> 172.21.15.3:0/835782993 conn(0x7f832405f840 msgr2=0x7f832405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:31.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8329788700 1 -- 172.21.15.3:0/835782993 shutdown_connections 2022-07-21T01:42:31.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8329788700 1 -- 172.21.15.3:0/835782993 wait complete. 2022-07-21T01:42:31.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8329788700 1 Processor -- start 2022-07-21T01:42:31.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.831+0000 7f8329788700 1 -- start start 2022-07-21T01:42:31.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8329788700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 0x7f83240ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8329788700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8324148c90 0x7f83241439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8329788700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8324143ea0 0x7f8324156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8329788700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832414a5f0 con 0x7f8324143ea0 2022-07-21T01:42:31.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8329788700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832414a890 con 0x7f8304003780 2022-07-21T01:42:31.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8329788700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832414ab30 con 0x7f8324148c90 2022-07-21T01:42:31.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83237fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8324143ea0 0x7f8324156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83237fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8324143ea0 0x7f8324156dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51262/0 (socket says 172.21.15.3:51262) 2022-07-21T01:42:31.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83237fe700 1 -- 172.21.15.3:0/380667822 learned_addr learned my addr 172.21.15.3:0/380667822 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:31.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8322ffd700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 0x7f83240ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83227fc700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8324148c90 0x7f83241439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83237fe700 1 -- 172.21.15.3:0/380667822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8324148c90 msgr2=0x7f83241439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83237fe700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8324148c90 0x7f83241439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83237fe700 1 -- 172.21.15.3:0/380667822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 msgr2=0x7f83240ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83237fe700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 0x7f83240ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83237fe700 1 -- 172.21.15.3:0/380667822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832414add0 con 0x7f8324143ea0 2022-07-21T01:42:31.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f83237fe700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8324143ea0 0x7f8324156dd0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7f8314000e90 tx=0x7f8314009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:31.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f831bfff700 1 -- 172.21.15.3:0/380667822 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f831400edd0 con 0x7f8324143ea0 2022-07-21T01:42:31.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f831bfff700 1 -- 172.21.15.3:0/380667822 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8314014430 con 0x7f8324143ea0 2022-07-21T01:42:31.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832414bc90 con 0x7f8324143ea0 2022-07-21T01:42:31.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f831bfff700 1 -- 172.21.15.3:0/380667822 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8314014790 con 0x7f8324143ea0 2022-07-21T01:42:31.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83240aabe0 con 0x7f8324143ea0 2022-07-21T01:42:31.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832414af60 con 0x7f8324143ea0 2022-07-21T01:42:31.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f831bfff700 1 -- 172.21.15.3:0/380667822 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f83140148f0 con 0x7f8324143ea0 2022-07-21T01:42:31.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.835+0000 7f831bfff700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8308053b00 0x7f8308055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:31.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.839+0000 7f831bfff700 1 -- 172.21.15.3:0/380667822 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f831406b7e0 con 0x7f8324143ea0 2022-07-21T01:42:31.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.839+0000 7f8322ffd700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8308053b00 0x7f8308055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:31.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.839+0000 7f8322ffd700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8308053b00 0x7f8308055fc0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f830c01b000 tx=0x7f830c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:31.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.843+0000 7f831bfff700 1 -- 172.21.15.3:0/380667822 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f83140428e0 con 0x7f8324143ea0 2022-07-21T01:42:31.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.971+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83240574e0 con 0x7f8324143ea0 2022-07-21T01:42:31.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f831bfff700 1 -- 172.21.15.3:0/380667822 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+13424 (secure 0 0 0) 0x7f8314025480 con 0x7f8324143ea0 2022-07-21T01:42:31.978 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:31.978 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":145,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:31.345800+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:31.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8308053b00 msgr2=0x7f8308055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8308053b00 0x7f8308055fc0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f830c01b000 tx=0x7f830c008040).stop 2022-07-21T01:42:31.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8324143ea0 msgr2=0x7f8324156dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:31.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8324143ea0 0x7f8324156dd0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7f8314000e90 tx=0x7f8314009040).stop 2022-07-21T01:42:31.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 shutdown_connections 2022-07-21T01:42:31.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8324143ea0 0x7f8324156dd0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8324148c90 0x7f83241439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8308053b00 0x7f8308055fc0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 --2- 172.21.15.3:0/380667822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8304003780 0x7f83240ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:31.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.975+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 >> 172.21.15.3:0/380667822 conn(0x7f832405f840 msgr2=0x7f83240b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:31.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.979+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 shutdown_connections 2022-07-21T01:42:31.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:31.979+0000 7f8329788700 1 -- 172.21.15.3:0/380667822 wait complete. 2022-07-21T01:42:31.992 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:42:31.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:42:32.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd07c9ae700 1 Processor -- start 2022-07-21T01:42:32.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd07c9ae700 1 -- start start 2022-07-21T01:42:32.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd07c9ae700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd0780b0b20 0x7fd0780afd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd07c9ae700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0780ac520 con 0x7fd0780ae870 2022-07-21T01:42:32.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd07c9ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0780b17a0 con 0x7fd0780b13c0 2022-07-21T01:42:32.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd07c9ae700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0780b0450 con 0x7fd0780b0b20 2022-07-21T01:42:32.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd077fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd0780ae870 0x7fd0780aecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59548/0 (socket says 172.21.15.3:59548) 2022-07-21T01:42:32.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd076ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd0780b0b20 0x7fd0780afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd077fff700 1 -- 172.21.15.3:0/1638988330 learned_addr learned my addr 172.21.15.3:0/1638988330 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:32.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd0767fc700 1 -- 172.21.15.3:0/1638988330 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364987303 0 0) 0x7fd0780ac520 con 0x7fd0780ae870 2022-07-21T01:42:32.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd0767fc700 1 -- 172.21.15.3:0/1638988330 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd064003260 con 0x7fd0780ae870 2022-07-21T01:42:32.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd076ffd700 1 -- 172.21.15.3:0/1638988330 >> v1:172.21.15.143:6789/0 conn(0x7fd0780b13c0 legacy=0x7fd078142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd0767fc700 1 -- 172.21.15.3:0/1638988330 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263139139 0 0) 0x7fd0780b17a0 con 0x7fd0780b13c0 2022-07-21T01:42:32.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd076ffd700 1 -- 172.21.15.3:0/1638988330 >> v1:172.21.15.3:6789/0 conn(0x7fd0780ae870 legacy=0x7fd0780aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd076ffd700 1 -- 172.21.15.3:0/1638988330 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0780b06f0 con 0x7fd0780b0b20 2022-07-21T01:42:32.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd076ffd700 1 --2- 172.21.15.3:0/1638988330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b0b20 0x7fd0780afd80 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7fd06c002f70 tx=0x7fd06c006df0).ready entity=mon.0 client_cookie=34c888e9e0080b15 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd0767fc700 1 -- 172.21.15.3:0/1638988330 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd06c00d610 con 0x7fd0780b0b20 2022-07-21T01:42:32.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd0767fc700 1 -- 172.21.15.3:0/1638988330 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c000e90 con 0x7fd0780b0b20 2022-07-21T01:42:32.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd0767fc700 1 -- 172.21.15.3:0/1638988330 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd06c00d960 con 0x7fd0780b0b20 2022-07-21T01:42:32.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd07c9ae700 1 -- 172.21.15.3:0/1638988330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b0b20 msgr2=0x7fd0780afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.103+0000 7fd07c9ae700 1 --2- 172.21.15.3:0/1638988330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b0b20 0x7fd0780afd80 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7fd06c002f70 tx=0x7fd06c006df0).stop 2022-07-21T01:42:32.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- 172.21.15.3:0/1638988330 shutdown_connections 2022-07-21T01:42:32.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 --2- 172.21.15.3:0/1638988330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b0b20 0x7fd0780afd80 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- 172.21.15.3:0/1638988330 >> 172.21.15.3:0/1638988330 conn(0x7fd0780626f0 msgr2=0x7fd078062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:32.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- 172.21.15.3:0/1638988330 shutdown_connections 2022-07-21T01:42:32.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- 172.21.15.3:0/1638988330 wait complete. 2022-07-21T01:42:32.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 Processor -- start 2022-07-21T01:42:32.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- start start 2022-07-21T01:42:32.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0780ae870 0x7fd07813f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0780b0b20 0x7fd078146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b13c0 0x7fd078146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0781567f0 con 0x7fd0780b13c0 2022-07-21T01:42:32.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078156a90 con 0x7fd0780b0b20 2022-07-21T01:42:32.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07813dd80 con 0x7fd0780ae870 2022-07-21T01:42:32.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd077fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b13c0 0x7fd078146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd077fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b13c0 0x7fd078146fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51276/0 (socket says 172.21.15.3:51276) 2022-07-21T01:42:32.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd077fff700 1 -- 172.21.15.3:0/3548370566 learned_addr learned my addr 172.21.15.3:0/3548370566 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:32.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd076ffd700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0780b0b20 0x7fd078146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd0777fe700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0780ae870 0x7fd07813f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd077fff700 1 -- 172.21.15.3:0/3548370566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0780ae870 msgr2=0x7fd07813f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd077fff700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0780ae870 0x7fd07813f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd077fff700 1 -- 172.21.15.3:0/3548370566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0780b0b20 msgr2=0x7fd078146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd077fff700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0780b0b20 0x7fd078146ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd077fff700 1 -- 172.21.15.3:0/3548370566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c00b180 con 0x7fd0780b13c0 2022-07-21T01:42:32.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd077fff700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b13c0 0x7fd078146fc0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7fd070001490 tx=0x7fd070007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd074ff9700 1 -- 172.21.15.3:0/3548370566 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd070005ba0 con 0x7fd0780b13c0 2022-07-21T01:42:32.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd074ff9700 1 -- 172.21.15.3:0/3548370566 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070005d00 con 0x7fd0780b13c0 2022-07-21T01:42:32.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07813e020 con 0x7fd0780b13c0 2022-07-21T01:42:32.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd074ff9700 1 -- 172.21.15.3:0/3548370566 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0700072c0 con 0x7fd0780b13c0 2022-07-21T01:42:32.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.107+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0781474b0 con 0x7fd0780b13c0 2022-07-21T01:42:32.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.111+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0781565a0 con 0x7fd0780b13c0 2022-07-21T01:42:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.111+0000 7fd074ff9700 1 -- 172.21.15.3:0/3548370566 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd070010070 con 0x7fd0780b13c0 2022-07-21T01:42:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.111+0000 7fd074ff9700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd058053a70 0x7fd058055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.111+0000 7fd074ff9700 1 -- 172.21.15.3:0/3548370566 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fd07006b6e0 con 0x7fd0780b13c0 2022-07-21T01:42:32.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.111+0000 7fd0777fe700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd058053a70 0x7fd058055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.111+0000 7fd0777fe700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd058053a70 0x7fd058055f30 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fd06800d000 tx=0x7fd068007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.119+0000 7fd074ff9700 1 -- 172.21.15.3:0/3548370566 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd0700426d0 con 0x7fd0780b13c0 2022-07-21T01:42:32.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd07813e1b0 con 0x7fd0780b13c0 2022-07-21T01:42:32.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd074ff9700 1 -- 172.21.15.3:0/3548370566 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v145) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd070018030 con 0x7fd0780b13c0 2022-07-21T01:42:32.250 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:42:32.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd058053a70 msgr2=0x7fd058055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd058053a70 0x7fd058055f30 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fd06800d000 tx=0x7fd068007b70).stop 2022-07-21T01:42:32.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b13c0 msgr2=0x7fd078146fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b13c0 0x7fd078146fc0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7fd070001490 tx=0x7fd070007e00).stop 2022-07-21T01:42:32.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 shutdown_connections 2022-07-21T01:42:32.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0780b13c0 0x7fd078146fc0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd058053a70 0x7fd058055f30 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0780b0b20 0x7fd078146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 --2- 172.21.15.3:0/3548370566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0780ae870 0x7fd07813f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 >> 172.21.15.3:0/3548370566 conn(0x7fd0780626f0 msgr2=0x7fd0780aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:32.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.247+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 shutdown_connections 2022-07-21T01:42:32.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.251+0000 7fd07c9ae700 1 -- 172.21.15.3:0/3548370566 wait complete. 2022-07-21T01:42:32.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:42:32.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.363+0000 7f06348f3700 1 Processor -- start 2022-07-21T01:42:32.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.363+0000 7f06348f3700 1 -- start start 2022-07-21T01:42:32.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06300ad750 0x7f0630061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06301405d0 0x7f0630140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06300aee80 con 0x7f06300614b0 2022-07-21T01:42:32.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06300b70b0 con 0x7f06300ad750 2022-07-21T01:42:32.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06300af0b0 con 0x7f06301405d0 2022-07-21T01:42:32.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06300ad750 0x7f0630061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06300ad750 0x7f0630061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51282/0 (socket says 172.21.15.3:51282) 2022-07-21T01:42:32.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062effd700 1 -- 172.21.15.3:0/1797967624 learned_addr learned my addr 172.21.15.3:0/1797967624 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:32.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062ffff700 1 --2- 172.21.15.3:0/1797967624 >> v2:172.21.15.143:3300/0 conn(0x7f06301405d0 0x7f0630140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062effd700 1 -- 172.21.15.3:0/1797967624 >> v2:172.21.15.143:3300/0 conn(0x7f06301405d0 msgr2=0x7f0630140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:32.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062effd700 1 --2- 172.21.15.3:0/1797967624 >> v2:172.21.15.143:3300/0 conn(0x7f06301405d0 0x7f0630140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062effd700 1 -- 172.21.15.3:0/1797967624 >> v1:172.21.15.143:6789/0 conn(0x7f06300614b0 legacy=0x7f06300b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062effd700 1 -- 172.21.15.3:0/1797967624 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06300af350 con 0x7f06300ad750 2022-07-21T01:42:32.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062effd700 1 --2- 172.21.15.3:0/1797967624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300ad750 0x7f0630061890 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f0624002f70 tx=0x7f0624006eb0).ready entity=mon.0 client_cookie=bf7ac9732e97693 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062e7fc700 1 -- 172.21.15.3:0/1797967624 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f062400d5c0 con 0x7f06300ad750 2022-07-21T01:42:32.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062e7fc700 1 -- 172.21.15.3:0/1797967624 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0624000e90 con 0x7f06300ad750 2022-07-21T01:42:32.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f062e7fc700 1 -- 172.21.15.3:0/1797967624 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f062400d950 con 0x7f06300ad750 2022-07-21T01:42:32.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 -- 172.21.15.3:0/1797967624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300ad750 msgr2=0x7f0630061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 --2- 172.21.15.3:0/1797967624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300ad750 0x7f0630061890 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f0624002f70 tx=0x7f0624006eb0).stop 2022-07-21T01:42:32.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 -- 172.21.15.3:0/1797967624 shutdown_connections 2022-07-21T01:42:32.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 --2- 172.21.15.3:0/1797967624 >> v2:172.21.15.143:3300/0 conn(0x7f06301405d0 0x7f0630140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 --2- 172.21.15.3:0/1797967624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300ad750 0x7f0630061890 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 -- 172.21.15.3:0/1797967624 >> 172.21.15.3:0/1797967624 conn(0x7f0630066f70 msgr2=0x7f0630067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:32.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 -- 172.21.15.3:0/1797967624 shutdown_connections 2022-07-21T01:42:32.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.367+0000 7f06348f3700 1 -- 172.21.15.3:0/1797967624 wait complete. 2022-07-21T01:42:32.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 Processor -- start 2022-07-21T01:42:32.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 -- start start 2022-07-21T01:42:32.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300614b0 0x7f0630143450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300ad750 0x7f06300ab640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06301405d0 0x7f06300abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06300ac900 con 0x7f06300614b0 2022-07-21T01:42:32.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06300acdf0 con 0x7f06300ad750 2022-07-21T01:42:32.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06300afc40 con 0x7f06301405d0 2022-07-21T01:42:32.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300614b0 0x7f0630143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300ad750 0x7f06300ab640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300614b0 0x7f0630143450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51286/0 (socket says 172.21.15.3:51286) 2022-07-21T01:42:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06301405d0 0x7f06300abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062f7fe700 1 -- 172.21.15.3:0/60796247 learned_addr learned my addr 172.21.15.3:0/60796247 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062f7fe700 1 -- 172.21.15.3:0/60796247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06301405d0 msgr2=0x7f06300abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062f7fe700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06301405d0 0x7f06300abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062f7fe700 1 -- 172.21.15.3:0/60796247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300ad750 msgr2=0x7f06300ab640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062f7fe700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300ad750 0x7f06300ab640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062f7fe700 1 -- 172.21.15.3:0/60796247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062400b1e0 con 0x7f06300614b0 2022-07-21T01:42:32.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062f7fe700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300614b0 0x7f0630143450 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7f06300ae2b0 tx=0x7f0620007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062cff9700 1 -- 172.21.15.3:0/60796247 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0620012400 con 0x7f06300614b0 2022-07-21T01:42:32.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062cff9700 1 -- 172.21.15.3:0/60796247 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0620012560 con 0x7f06300614b0 2022-07-21T01:42:32.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06300b1ea0 con 0x7f06300614b0 2022-07-21T01:42:32.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f062cff9700 1 -- 172.21.15.3:0/60796247 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06200128c0 con 0x7f06300614b0 2022-07-21T01:42:32.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.371+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06300ac580 con 0x7f06300614b0 2022-07-21T01:42:32.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.375+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06300af5f0 con 0x7f06300614b0 2022-07-21T01:42:32.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.375+0000 7f062cff9700 1 -- 172.21.15.3:0/60796247 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0620019030 con 0x7f06300614b0 2022-07-21T01:42:32.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.375+0000 7f062cff9700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0610053a70 0x7f0610055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.375+0000 7f062cff9700 1 -- 172.21.15.3:0/60796247 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f062006b040 con 0x7f06300614b0 2022-07-21T01:42:32.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.375+0000 7f062effd700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0610053a70 0x7f0610055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.375+0000 7f062effd700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0610053a70 0x7f0610055f30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f062401c000 tx=0x7f0624015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.383+0000 7f062cff9700 1 -- 172.21.15.3:0/60796247 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0620042140 con 0x7f06300614b0 2022-07-21T01:42:32.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.503+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f06300574e0 con 0x7f06300614b0 2022-07-21T01:42:32.506 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.503+0000 7f062cff9700 1 -- 172.21.15.3:0/60796247 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v145) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0620045760 con 0x7f06300614b0 2022-07-21T01:42:32.507 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:32.508 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.503+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0610053a70 msgr2=0x7f0610055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.503+0000 7f06348f3700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0610053a70 0x7f0610055f30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f062401c000 tx=0x7f0624015000).stop 2022-07-21T01:42:32.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.503+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300614b0 msgr2=0x7f0630143450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.503+0000 7f06348f3700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300614b0 0x7f0630143450 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7f06300ae2b0 tx=0x7f0620007c50).stop 2022-07-21T01:42:32.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.507+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 shutdown_connections 2022-07-21T01:42:32.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.507+0000 7f06348f3700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06301405d0 0x7f06300abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.507+0000 7f06348f3700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0610053a70 0x7f0610055f30 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.507+0000 7f06348f3700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300ad750 0x7f06300ab640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.507+0000 7f06348f3700 1 --2- 172.21.15.3:0/60796247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06300614b0 0x7f0630143450 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.507+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 >> 172.21.15.3:0/60796247 conn(0x7f0630066f70 msgr2=0x7f0630144cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:32.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.507+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 shutdown_connections 2022-07-21T01:42:32.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.507+0000 7f06348f3700 1 -- 172.21.15.3:0/60796247 wait complete. 2022-07-21T01:42:32.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:42:32.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:42:32.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:42:32.548 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:42:32.550 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:42:32.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.615+0000 7f8a304a8700 1 Processor -- start 2022-07-21T01:42:32.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.615+0000 7f8a304a8700 1 -- start start 2022-07-21T01:42:32.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a280b1f40 0x7f8a280b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8a28148530 0x7f8a280b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a280b04e0 con 0x7f8a280a7250 2022-07-21T01:42:32.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a280a9250 con 0x7f8a28148530 2022-07-21T01:42:32.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280b0780 con 0x7f8a280b1f40 2022-07-21T01:42:32.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2ea45700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8a280a7250 0x7f8a281448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59578/0 (socket says 172.21.15.3:59578) 2022-07-21T01:42:32.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2ea45700 1 -- 172.21.15.3:0/1347750403 learned_addr learned my addr 172.21.15.3:0/1347750403 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:32.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2e244700 1 --2- 172.21.15.3:0/1347750403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a280b1f40 0x7f8a280b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2da43700 1 --2- 172.21.15.3:0/1347750403 >> v2:172.21.15.143:3300/0 conn(0x7f8a28148530 0x7f8a280b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2d242700 1 -- 172.21.15.3:0/1347750403 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122187432 0 0) 0x7f8a280b04e0 con 0x7f8a280a7250 2022-07-21T01:42:32.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2d242700 1 -- 172.21.15.3:0/1347750403 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a14003260 con 0x7f8a280a7250 2022-07-21T01:42:32.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2e244700 1 -- 172.21.15.3:0/1347750403 >> v2:172.21.15.143:3300/0 conn(0x7f8a28148530 msgr2=0x7f8a280b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:32.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2e244700 1 --2- 172.21.15.3:0/1347750403 >> v2:172.21.15.143:3300/0 conn(0x7f8a28148530 0x7f8a280b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2e244700 1 -- 172.21.15.3:0/1347750403 >> v1:172.21.15.3:6789/0 conn(0x7f8a280a7250 legacy=0x7f8a281448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2e244700 1 -- 172.21.15.3:0/1347750403 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a280b0a20 con 0x7f8a280b1f40 2022-07-21T01:42:32.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2e244700 1 --2- 172.21.15.3:0/1347750403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a280b1f40 0x7f8a280b1160 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8a200032f0 tx=0x7f8a200091d0).ready entity=mon.2 client_cookie=467d3a5c75118324 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2d242700 1 -- 172.21.15.3:0/1347750403 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a2000b600 con 0x7f8a280b1f40 2022-07-21T01:42:32.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2d242700 1 -- 172.21.15.3:0/1347750403 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2000b760 con 0x7f8a280b1f40 2022-07-21T01:42:32.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a2d242700 1 -- 172.21.15.3:0/1347750403 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a2000b8d0 con 0x7f8a280b1f40 2022-07-21T01:42:32.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 -- 172.21.15.3:0/1347750403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a280b1f40 msgr2=0x7f8a280b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 --2- 172.21.15.3:0/1347750403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a280b1f40 0x7f8a280b1160 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8a200032f0 tx=0x7f8a200091d0).stop 2022-07-21T01:42:32.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 -- 172.21.15.3:0/1347750403 shutdown_connections 2022-07-21T01:42:32.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 --2- 172.21.15.3:0/1347750403 >> v2:172.21.15.143:3300/0 conn(0x7f8a28148530 0x7f8a280b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 --2- 172.21.15.3:0/1347750403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a280b1f40 0x7f8a280b1160 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 -- 172.21.15.3:0/1347750403 >> 172.21.15.3:0/1347750403 conn(0x7f8a2805f840 msgr2=0x7f8a2805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:32.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 -- 172.21.15.3:0/1347750403 shutdown_connections 2022-07-21T01:42:32.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.619+0000 7f8a304a8700 1 -- 172.21.15.3:0/1347750403 wait complete. 2022-07-21T01:42:32.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a304a8700 1 Processor -- start 2022-07-21T01:42:32.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a304a8700 1 -- start start 2022-07-21T01:42:32.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a304a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a280a7250 0x7f8a280ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a304a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a280b1f40 0x7f8a28148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a304a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a28148530 0x7f8a28149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a304a8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2814b490 con 0x7f8a280a7250 2022-07-21T01:42:32.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a304a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2814b900 con 0x7f8a280b1f40 2022-07-21T01:42:32.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a304a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2814b1f0 con 0x7f8a28148530 2022-07-21T01:42:32.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a2e244700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a280a7250 0x7f8a280ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a2e244700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a280a7250 0x7f8a280ae4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51300/0 (socket says 172.21.15.3:51300) 2022-07-21T01:42:32.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.623+0000 7f8a2e244700 1 -- 172.21.15.3:0/494962580 learned_addr learned my addr 172.21.15.3:0/494962580 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:32.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a2da43700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a280b1f40 0x7f8a28148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a2ea45700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a28148530 0x7f8a28149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a2e244700 1 -- 172.21.15.3:0/494962580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a28148530 msgr2=0x7f8a28149380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a2e244700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a28148530 0x7f8a28149380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a2e244700 1 -- 172.21.15.3:0/494962580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a280b1f40 msgr2=0x7f8a28148e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a2e244700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a280b1f40 0x7f8a28148e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a2e244700 1 -- 172.21.15.3:0/494962580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a20009470 con 0x7f8a280a7250 2022-07-21T01:42:32.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a2e244700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a280a7250 0x7f8a280ae4f0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f8a200032f0 tx=0x7f8a20016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a277fe700 1 -- 172.21.15.3:0/494962580 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a2000b8d0 con 0x7f8a280a7250 2022-07-21T01:42:32.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a277fe700 1 -- 172.21.15.3:0/494962580 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a20011e80 con 0x7f8a280a7250 2022-07-21T01:42:32.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2814bb60 con 0x7f8a280a7250 2022-07-21T01:42:32.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a277fe700 1 -- 172.21.15.3:0/494962580 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a2000b600 con 0x7f8a280a7250 2022-07-21T01:42:32.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a28147320 con 0x7f8a280a7250 2022-07-21T01:42:32.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a280b0cc0 con 0x7f8a280a7250 2022-07-21T01:42:32.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.627+0000 7f8a277fe700 1 -- 172.21.15.3:0/494962580 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8a20026030 con 0x7f8a280a7250 2022-07-21T01:42:32.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.631+0000 7f8a277fe700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.631+0000 7f8a277fe700 1 -- 172.21.15.3:0/494962580 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f8a2006c7e0 con 0x7f8a280a7250 2022-07-21T01:42:32.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.631+0000 7f8a2da43700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.631+0000 7f8a2da43700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f8a1c00b000 tx=0x7f8a1c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.635+0000 7f8a277fe700 1 -- 172.21.15.3:0/494962580 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a200437d0 con 0x7f8a280a7250 2022-07-21T01:42:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.767+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8a280574e0 con 0x7f8a280a7250 2022-07-21T01:42:32.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.767+0000 7f8a277fe700 1 -- 172.21.15.3:0/494962580 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v145) v1 ==== 84+0+143 (secure 0 0 0) 0x7f8a20015020 con 0x7f8a280a7250 2022-07-21T01:42:32.772 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:32.772 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:42:32.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:32.772 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:42:32.773 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:42:32.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:32.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:32.773 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:42:32.774 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:42:32.774 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:32.774 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:42:32.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 msgr2=0x7f8a08055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f8a1c00b000 tx=0x7f8a1c006df0).stop 2022-07-21T01:42:32.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a280a7250 msgr2=0x7f8a280ae4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a280a7250 0x7f8a280ae4f0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f8a200032f0 tx=0x7f8a20016000).stop 2022-07-21T01:42:32.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 shutdown_connections 2022-07-21T01:42:32.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a28148530 0x7f8a28149380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8a08053a70 0x7f8a08055f30 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a280b1f40 0x7f8a28148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 --2- 172.21.15.3:0/494962580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a280a7250 0x7f8a280ae4f0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 >> 172.21.15.3:0/494962580 conn(0x7f8a2805f840 msgr2=0x7f8a280b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:32.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 shutdown_connections 2022-07-21T01:42:32.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.771+0000 7f8a304a8700 1 -- 172.21.15.3:0/494962580 wait complete. 2022-07-21T01:42:32.786 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:42:32.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:42:32.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9382230700 1 Processor -- start 2022-07-21T01:42:32.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9382230700 1 -- start start 2022-07-21T01:42:32.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9382230700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f937c05fd90 0x7f937c0b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9382230700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c0b66f0 0x7f937c0ba130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9382230700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c1440f0 con 0x7f937c0b5b20 2022-07-21T01:42:32.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9382230700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937c060170 con 0x7f937c05fd90 2022-07-21T01:42:32.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9382230700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937c143be0 con 0x7f937c0b66f0 2022-07-21T01:42:32.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937bfff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f937c0b5b20 0x7f937c0b4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59592/0 (socket says 172.21.15.3:59592) 2022-07-21T01:42:32.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937bfff700 1 -- 172.21.15.3:0/4086942165 learned_addr learned my addr 172.21.15.3:0/4086942165 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:32.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937b7fe700 1 --2- 172.21.15.3:0/4086942165 >> v2:172.21.15.143:3300/0 conn(0x7f937c05fd90 0x7f937c0b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9372ffd700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c0b66f0 0x7f937c0ba130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327491761 0 0) 0x7f937c1440f0 con 0x7f937c0b5b20 2022-07-21T01:42:32.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935c003260 con 0x7f937c0b5b20 2022-07-21T01:42:32.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1387271227 0 0) 0x7f935c003260 con 0x7f937c0b5b20 2022-07-21T01:42:32.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c0b66f0 msgr2=0x7f937c0ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:32.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c0b66f0 0x7f937c0ba130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 >> v2:172.21.15.143:3300/0 conn(0x7f937c05fd90 msgr2=0x7f937c0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:32.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 --2- 172.21.15.3:0/4086942165 >> v2:172.21.15.143:3300/0 conn(0x7f937c05fd90 0x7f937c0b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c143e50 con 0x7f937c0b5b20 2022-07-21T01:42:32.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f93740019e0 con 0x7f937c0b5b20 2022-07-21T01:42:32.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937b7fe700 1 --2- 172.21.15.3:0/4086942165 >> v2:172.21.15.143:3300/0 conn(0x7f937c05fd90 0x7f937c0b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:32.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 >> v1:172.21.15.3:6789/0 conn(0x7f937c0b5b20 legacy=0x7f937c0b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f935c003750 0x7f935c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004500 0x7f935c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c0b66f0 0x7f935c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c060170 con 0x7f935c003750 2022-07-21T01:42:32.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c143be0 con 0x7f935c004500 2022-07-21T01:42:32.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c00b750 con 0x7f937c0b66f0 2022-07-21T01:42:32.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937b7fe700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f935c003750 0x7f935c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937bfff700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c0b66f0 0x7f935c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9372ffd700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004500 0x7f935c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937b7fe700 1 -- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c0b66f0 msgr2=0x7f935c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:32.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937b7fe700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c0b66f0 0x7f935c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937b7fe700 1 -- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004500 msgr2=0x7f935c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:32.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937b7fe700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004500 0x7f935c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937b7fe700 1 -- 172.21.15.3:0/4086942165 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c003260 con 0x7f935c003750 2022-07-21T01:42:32.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937bfff700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c0b66f0 0x7f935c004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:32.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f9372ffd700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004500 0x7f935c004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:32.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937b7fe700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f935c003750 0x7f935c003b70 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7f936c00c000 tx=0x7f936c006df0).ready entity=mon.0 client_cookie=2feda6fc7727774 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.891+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f936c00e830 con 0x7f935c003750 2022-07-21T01:42:32.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c00e990 con 0x7f935c003750 2022-07-21T01:42:32.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937affd700 1 -- 172.21.15.3:0/4086942165 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f936c00ecf0 con 0x7f935c003750 2022-07-21T01:42:32.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 -- 172.21.15.3:0/4086942165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f935c003750 msgr2=0x7f935c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 --2- 172.21.15.3:0/4086942165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f935c003750 0x7f935c003b70 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7f936c00c000 tx=0x7f936c006df0).stop 2022-07-21T01:42:32.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937bfff700 1 -- 172.21.15.3:0/4086942165 reap_dead start 2022-07-21T01:42:32.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 -- 172.21.15.3:0/4086942165 shutdown_connections 2022-07-21T01:42:32.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 -- 172.21.15.3:0/4086942165 >> 172.21.15.3:0/4086942165 conn(0x7f937c062530 msgr2=0x7f937c066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 -- 172.21.15.3:0/4086942165 shutdown_connections 2022-07-21T01:42:32.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 -- 172.21.15.3:0/4086942165 wait complete. 2022-07-21T01:42:32.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 Processor -- start 2022-07-21T01:42:32.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 -- start start 2022-07-21T01:42:32.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c003750 0x7f937c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f937c0b0280 0x7f937c156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c156550 0x7f937c152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c152780 con 0x7f937c0b0280 2022-07-21T01:42:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c0b2c50 con 0x7f935c003750 2022-07-21T01:42:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9382230700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937c0b2e80 con 0x7f937c156550 2022-07-21T01:42:32.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9372ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f937c0b0280 0x7f937c156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9372ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f937c0b0280 0x7f937c156060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51320/0 (socket says 172.21.15.3:51320) 2022-07-21T01:42:32.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9372ffd700 1 -- 172.21.15.3:0/92796671 learned_addr learned my addr 172.21.15.3:0/92796671 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:32.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937b7fe700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c003750 0x7f937c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937bfff700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c156550 0x7f937c152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937b7fe700 1 -- 172.21.15.3:0/92796671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c156550 msgr2=0x7f937c152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937b7fe700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c156550 0x7f937c152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937b7fe700 1 -- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f937c0b0280 msgr2=0x7f937c156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:32.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937b7fe700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f937c0b0280 0x7f937c156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:32.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937b7fe700 1 -- 172.21.15.3:0/92796671 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c0b3120 con 0x7f935c003750 2022-07-21T01:42:32.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f9372ffd700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f937c0b0280 0x7f937c156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:32.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.895+0000 7f937b7fe700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c003750 0x7f937c0afcb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f937c0b0750 tx=0x7f936c019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f93797fa700 1 -- 172.21.15.3:0/92796671 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f936c01c800 con 0x7f935c003750 2022-07-21T01:42:32.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f93797fa700 1 -- 172.21.15.3:0/92796671 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c000e90 con 0x7f935c003750 2022-07-21T01:42:32.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937c0af650 con 0x7f935c003750 2022-07-21T01:42:32.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f93797fa700 1 -- 172.21.15.3:0/92796671 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f936c01cb50 con 0x7f935c003750 2022-07-21T01:42:32.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937c0aea20 con 0x7f935c003750 2022-07-21T01:42:32.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937c155d80 con 0x7f935c003750 2022-07-21T01:42:32.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f93797fa700 1 -- 172.21.15.3:0/92796671 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f936c01ccb0 con 0x7f935c003750 2022-07-21T01:42:32.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f93797fa700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93600539e0 0x7f9360055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:32.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f93797fa700 1 -- 172.21.15.3:0/92796671 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f936c06d070 con 0x7f935c003750 2022-07-21T01:42:32.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f9372ffd700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93600539e0 0x7f9360055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:32.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.899+0000 7f9372ffd700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93600539e0 0x7f9360055ea0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f936400b000 tx=0x7f9364008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:32.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:32.907+0000 7f93797fa700 1 -- 172.21.15.3:0/92796671 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f936c0440d0 con 0x7f935c003750 2022-07-21T01:42:33.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.035+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f937c0b99f0 con 0x7f935c003750 2022-07-21T01:42:33.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.715+0000 7f93797fa700 1 -- 172.21.15.3:0/92796671 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v146) v1 ==== 133+0+0 (secure 0 0 0) 0x7f936c0476f0 con 0x7f935c003750 2022-07-21T01:42:33.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93600539e0 msgr2=0x7f9360055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:33.722 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93600539e0 0x7f9360055ea0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f936400b000 tx=0x7f9364008040).stop 2022-07-21T01:42:33.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c003750 msgr2=0x7f937c0afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:33.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c003750 0x7f937c0afcb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f937c0b0750 tx=0x7f936c019000).stop 2022-07-21T01:42:33.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 shutdown_connections 2022-07-21T01:42:33.723 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937c156550 0x7f937c152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:33.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f937c0b0280 0x7f937c156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:33.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f93600539e0 0x7f9360055ea0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:33.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 --2- 172.21.15.3:0/92796671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c003750 0x7f937c0afcb0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:33.724 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 >> 172.21.15.3:0/92796671 conn(0x7f937c062530 msgr2=0x7f937c0b7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:33.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 shutdown_connections 2022-07-21T01:42:33.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.719+0000 7f9382230700 1 -- 172.21.15.3:0/92796671 wait complete. 2022-07-21T01:42:33.725 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:42:33.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:42:33.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.843+0000 7f43e359e700 1 Processor -- start 2022-07-21T01:42:33.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.843+0000 7f43e359e700 1 -- start start 2022-07-21T01:42:33.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.843+0000 7f43e359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f43e40b2aa0 0x7f43e40b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:33.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.843+0000 7f43e359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e4140c10 0x7f43e40b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:33.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.843+0000 7f43e359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e41467f0 con 0x7f43e40b6510 2022-07-21T01:42:33.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.843+0000 7f43e359e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e4062140 con 0x7f43e40b2aa0 2022-07-21T01:42:33.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.843+0000 7f43e359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e40656b0 con 0x7f43e4140c10 2022-07-21T01:42:33.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.843+0000 7f43e1d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f43e40b2aa0 0x7f43e40b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:33.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e1d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f43e40b2aa0 0x7f43e40b0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51330/0 (socket says 172.21.15.3:51330) 2022-07-21T01:42:33.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e1d9b700 1 -- 172.21.15.3:0/2140325116 learned_addr learned my addr 172.21.15.3:0/2140325116 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:33.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e2d9d700 1 --2- 172.21.15.3:0/2140325116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e4140c10 0x7f43e40b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:33.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e1d9b700 1 -- 172.21.15.3:0/2140325116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e4140c10 msgr2=0x7f43e40b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:33.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e1d9b700 1 --2- 172.21.15.3:0/2140325116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e4140c10 0x7f43e40b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:33.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e1d9b700 1 -- 172.21.15.3:0/2140325116 >> v1:172.21.15.143:6789/0 conn(0x7f43e40b6510 legacy=0x7f43e4140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:33.854 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e1d9b700 1 -- 172.21.15.3:0/2140325116 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e41476a0 con 0x7f43e40b2aa0 2022-07-21T01:42:33.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e2d9d700 1 --2- 172.21.15.3:0/2140325116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e4140c10 0x7f43e40b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:33.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e1d9b700 1 --2- 172.21.15.3:0/2140325116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b2aa0 0x7f43e40b0e60 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f43d4002f70 tx=0x7f43d4006eb0).ready entity=mon.0 client_cookie=adc056a725f340b0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:33.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e159a700 1 -- 172.21.15.3:0/2140325116 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43d400d5c0 con 0x7f43e40b2aa0 2022-07-21T01:42:33.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e159a700 1 -- 172.21.15.3:0/2140325116 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d4000e90 con 0x7f43e40b2aa0 2022-07-21T01:42:33.855 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e159a700 1 -- 172.21.15.3:0/2140325116 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43d400d950 con 0x7f43e40b2aa0 2022-07-21T01:42:33.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e359e700 1 -- 172.21.15.3:0/2140325116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b2aa0 msgr2=0x7f43e40b0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:33.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e359e700 1 --2- 172.21.15.3:0/2140325116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b2aa0 0x7f43e40b0e60 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f43d4002f70 tx=0x7f43d4006eb0).stop 2022-07-21T01:42:33.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e359e700 1 -- 172.21.15.3:0/2140325116 shutdown_connections 2022-07-21T01:42:33.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e359e700 1 --2- 172.21.15.3:0/2140325116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e4140c10 0x7f43e40b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:33.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e359e700 1 --2- 172.21.15.3:0/2140325116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b2aa0 0x7f43e40b0e60 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:33.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e359e700 1 -- 172.21.15.3:0/2140325116 >> 172.21.15.3:0/2140325116 conn(0x7f43e4063800 msgr2=0x7f43e4063be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:33.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e359e700 1 -- 172.21.15.3:0/2140325116 shutdown_connections 2022-07-21T01:42:33.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.847+0000 7f43e359e700 1 -- 172.21.15.3:0/2140325116 wait complete. 2022-07-21T01:42:33.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 Processor -- start 2022-07-21T01:42:33.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 -- start start 2022-07-21T01:42:33.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e40b2aa0 0x7f43e41527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:33.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b6510 0x7f43e40668f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:33.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43e4140c10 0x7f43e4156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:33.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e4155e10 con 0x7f43e40b6510 2022-07-21T01:42:33.862 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e41443c0 con 0x7f43e4140c10 2022-07-21T01:42:33.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e4156070 con 0x7f43e40b2aa0 2022-07-21T01:42:33.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b6510 0x7f43e40668f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:33.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b6510 0x7f43e40668f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51336/0 (socket says 172.21.15.3:51336) 2022-07-21T01:42:33.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e1d9b700 1 -- 172.21.15.3:0/3410342795 learned_addr learned my addr 172.21.15.3:0/3410342795 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:33.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e259c700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e40b2aa0 0x7f43e41527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:33.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e2d9d700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43e4140c10 0x7f43e4156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:33.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e1d9b700 1 -- 172.21.15.3:0/3410342795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e40b2aa0 msgr2=0x7f43e41527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:33.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e1d9b700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e40b2aa0 0x7f43e41527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:33.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e1d9b700 1 -- 172.21.15.3:0/3410342795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43e4140c10 msgr2=0x7f43e4156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:33.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e1d9b700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43e4140c10 0x7f43e4156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:33.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e1d9b700 1 -- 172.21.15.3:0/3410342795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d400b1e0 con 0x7f43e40b6510 2022-07-21T01:42:33.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e259c700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e40b2aa0 0x7f43e41527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:33.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e1d9b700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b6510 0x7f43e40668f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7f43d4002f70 tx=0x7f43d4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:33.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43d37fe700 1 -- 172.21.15.3:0/3410342795 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43d40117e0 con 0x7f43e40b6510 2022-07-21T01:42:33.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43d37fe700 1 -- 172.21.15.3:0/3410342795 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d4011940 con 0x7f43e40b6510 2022-07-21T01:42:33.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e4144660 con 0x7f43e40b6510 2022-07-21T01:42:33.866 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43d37fe700 1 -- 172.21.15.3:0/3410342795 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43d4011ca0 con 0x7f43e40b6510 2022-07-21T01:42:33.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.851+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e4143360 con 0x7f43e40b6510 2022-07-21T01:42:33.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.855+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e4146a20 con 0x7f43e40b6510 2022-07-21T01:42:33.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.855+0000 7f43d37fe700 1 -- 172.21.15.3:0/3410342795 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f43d4025030 con 0x7f43e40b6510 2022-07-21T01:42:33.867 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.855+0000 7f43d37fe700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43c0053a70 0x7f43c0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:33.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.855+0000 7f43d37fe700 1 -- 172.21.15.3:0/3410342795 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5954+0+0 (secure 0 0 0) 0x7f43d406d2a0 con 0x7f43e40b6510 2022-07-21T01:42:33.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.855+0000 7f43e259c700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43c0053a70 0x7f43c0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:33.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.855+0000 7f43e259c700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43c0053a70 0x7f43c0055f30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f43d8008000 tx=0x7f43d8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:33.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.867+0000 7f43d37fe700 1 -- 172.21.15.3:0/3410342795 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43d40440a0 con 0x7f43e40b6510 2022-07-21T01:42:33.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:33.991+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f43e40574e0 con 0x7f43e40b6510 2022-07-21T01:42:34.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.331+0000 7f43d37fe700 1 -- 172.21.15.3:0/3410342795 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v147) v1 ==== 125+0+0 (secure 0 0 0) 0x7f43d4023020 con 0x7f43e40b6510 2022-07-21T01:42:34.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43c0053a70 msgr2=0x7f43c0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43c0053a70 0x7f43c0055f30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f43d8008000 tx=0x7f43d8009040).stop 2022-07-21T01:42:34.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b6510 msgr2=0x7f43e40668f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b6510 0x7f43e40668f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7f43d4002f70 tx=0x7f43d4015000).stop 2022-07-21T01:42:34.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 shutdown_connections 2022-07-21T01:42:34.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f43c0053a70 0x7f43c0055f30 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43e4140c10 0x7f43e4156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43e40b6510 0x7f43e40668f0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 --2- 172.21.15.3:0/3410342795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43e40b2aa0 0x7f43e41527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.341 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 >> 172.21.15.3:0/3410342795 conn(0x7f43e4063800 msgr2=0x7f43e41404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:34.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.335+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 shutdown_connections 2022-07-21T01:42:34.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.339+0000 7f43e359e700 1 -- 172.21.15.3:0/3410342795 wait complete. 2022-07-21T01:42:34.344 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:42:34.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:42:34.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e305f700 1 Processor -- start 2022-07-21T01:42:34.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e305f700 1 -- start start 2022-07-21T01:42:34.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e305f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa3dc0af450 0x7fa3dc0af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e305f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa3dc0ad4b0 0x7fa3dc1432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e305f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc0aff30 con 0x7fa3dc1437d0 2022-07-21T01:42:34.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e305f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3dc05c7a0 con 0x7fa3dc0af450 2022-07-21T01:42:34.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e305f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3dc140990 con 0x7fa3dc0ad4b0 2022-07-21T01:42:34.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e205d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa3dc0af450 0x7fa3dc0af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e285e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa3dc1437d0 0x7fa3dc0a8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59626/0 (socket says 172.21.15.3:59626) 2022-07-21T01:42:34.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e205d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa3dc0af450 0x7fa3dc0af830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51342/0 (socket says 172.21.15.3:51342) 2022-07-21T01:42:34.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.459+0000 7fa3e285e700 1 -- 172.21.15.3:0/3350394851 learned_addr learned my addr 172.21.15.3:0/3350394851 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:34.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e185c700 1 --2- 172.21.15.3:0/3350394851 >> v2:172.21.15.143:3300/0 conn(0x7fa3dc0ad4b0 0x7fa3dc1432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e205d700 1 -- 172.21.15.3:0/3350394851 >> v2:172.21.15.143:3300/0 conn(0x7fa3dc0ad4b0 msgr2=0x7fa3dc1432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:34.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e205d700 1 --2- 172.21.15.3:0/3350394851 >> v2:172.21.15.143:3300/0 conn(0x7fa3dc0ad4b0 0x7fa3dc1432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e205d700 1 -- 172.21.15.3:0/3350394851 >> v1:172.21.15.3:6789/0 conn(0x7fa3dc1437d0 legacy=0x7fa3dc0a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e205d700 1 -- 172.21.15.3:0/3350394851 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc0a9550 con 0x7fa3dc0af450 2022-07-21T01:42:34.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e105b700 1 -- 172.21.15.3:0/3350394851 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591777704 0 0) 0x7fa3dc0aff30 con 0x7fa3dc1437d0 2022-07-21T01:42:34.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e205d700 1 --2- 172.21.15.3:0/3350394851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 0x7fa3dc0af830 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7fa3dc0ad900 tx=0x7fa3d8006e80).ready entity=mon.0 client_cookie=4ee8d23672dffecb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:34.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e105b700 1 -- 172.21.15.3:0/3350394851 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3d800ba30 con 0x7fa3dc0af450 2022-07-21T01:42:34.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e105b700 1 -- 172.21.15.3:0/3350394851 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d8000e90 con 0x7fa3dc0af450 2022-07-21T01:42:34.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e105b700 1 -- 172.21.15.3:0/3350394851 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3d800bd80 con 0x7fa3dc0af450 2022-07-21T01:42:34.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 -- 172.21.15.3:0/3350394851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 msgr2=0x7fa3dc0af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 --2- 172.21.15.3:0/3350394851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 0x7fa3dc0af830 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7fa3dc0ad900 tx=0x7fa3d8006e80).stop 2022-07-21T01:42:34.475 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 -- 172.21.15.3:0/3350394851 shutdown_connections 2022-07-21T01:42:34.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 --2- 172.21.15.3:0/3350394851 >> v2:172.21.15.143:3300/0 conn(0x7fa3dc0ad4b0 0x7fa3dc1432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 --2- 172.21.15.3:0/3350394851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 0x7fa3dc0af830 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 -- 172.21.15.3:0/3350394851 >> 172.21.15.3:0/3350394851 conn(0x7fa3dc0612d0 msgr2=0x7fa3dc062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:34.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 -- 172.21.15.3:0/3350394851 shutdown_connections 2022-07-21T01:42:34.476 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 -- 172.21.15.3:0/3350394851 wait complete. 2022-07-21T01:42:34.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 Processor -- start 2022-07-21T01:42:34.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 -- start start 2022-07-21T01:42:34.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3dc0ad4b0 0x7fa3dc0b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 0x7fa3dc13cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3dc1437d0 0x7fa3dc0b6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc1381e0 con 0x7fa3dc0af450 2022-07-21T01:42:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0b3ca0 con 0x7fa3dc1437d0 2022-07-21T01:42:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e305f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0a99d0 con 0x7fa3dc0ad4b0 2022-07-21T01:42:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e185c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 0x7fa3dc13cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e185c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 0x7fa3dc13cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51350/0 (socket says 172.21.15.3:51350) 2022-07-21T01:42:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e185c700 1 -- 172.21.15.3:0/1248113401 learned_addr learned my addr 172.21.15.3:0/1248113401 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e205d700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3dc0ad4b0 0x7fa3dc0b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.463+0000 7fa3e285e700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3dc1437d0 0x7fa3dc0b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3e185c700 1 -- 172.21.15.3:0/1248113401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3dc0ad4b0 msgr2=0x7fa3dc0b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3e185c700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3dc0ad4b0 0x7fa3dc0b4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3e185c700 1 -- 172.21.15.3:0/1248113401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3dc1437d0 msgr2=0x7fa3dc0b6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3e185c700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3dc1437d0 0x7fa3dc0b6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3e185c700 1 -- 172.21.15.3:0/1248113401 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d800d410 con 0x7fa3dc0af450 2022-07-21T01:42:34.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3e185c700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 0x7fa3dc13cc40 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7fa3dc0b42f0 tx=0x7fa3cc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3cb7fe700 1 -- 172.21.15.3:0/1248113401 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3cc011780 con 0x7fa3dc0af450 2022-07-21T01:42:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3cb7fe700 1 -- 172.21.15.3:0/1248113401 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc0118e0 con 0x7fa3dc0af450 2022-07-21T01:42:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc0b6780 con 0x7fa3dc0af450 2022-07-21T01:42:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3cb7fe700 1 -- 172.21.15.3:0/1248113401 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3cc011c40 con 0x7fa3dc0af450 2022-07-21T01:42:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc0a9c70 con 0x7fa3dc0af450 2022-07-21T01:42:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc0b49b0 con 0x7fa3dc0af450 2022-07-21T01:42:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3cb7fe700 1 -- 172.21.15.3:0/1248113401 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa3cc01c460 con 0x7fa3dc0af450 2022-07-21T01:42:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3cb7fe700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3bc053b00 0x7fa3bc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.467+0000 7fa3cb7fe700 1 -- 172.21.15.3:0/1248113401 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6304+0+0 (secure 0 0 0) 0x7fa3cc06cce0 con 0x7fa3dc0af450 2022-07-21T01:42:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.471+0000 7fa3e205d700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3bc053b00 0x7fa3bc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.471+0000 7fa3e205d700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3bc053b00 0x7fa3bc055fc0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fa3d801c000 tx=0x7fa3d8006e80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.475+0000 7fa3cb7fe700 1 -- 172.21.15.3:0/1248113401 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa3cc043a40 con 0x7fa3dc0af450 2022-07-21T01:42:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.635+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fa3dc0574e0 con 0x7fa3dc0af450 2022-07-21T01:42:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.651+0000 7fa3cb7fe700 1 -- 172.21.15.3:0/1248113401 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 27 and data pool 28 v75) v1 ==== 189+0+0 (secure 0 0 0) 0x7fa3cc047060 con 0x7fa3dc0af450 2022-07-21T01:42:34.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3bc053b00 msgr2=0x7fa3bc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3bc053b00 0x7fa3bc055fc0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fa3d801c000 tx=0x7fa3d8006e80).stop 2022-07-21T01:42:34.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 msgr2=0x7fa3dc13cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 0x7fa3dc13cc40 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7fa3dc0b42f0 tx=0x7fa3cc008040).stop 2022-07-21T01:42:34.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 shutdown_connections 2022-07-21T01:42:34.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa3bc053b00 0x7fa3bc055fc0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3dc1437d0 0x7fa3dc0b6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3dc0af450 0x7fa3dc13cc40 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 --2- 172.21.15.3:0/1248113401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3dc0ad4b0 0x7fa3dc0b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 >> 172.21.15.3:0/1248113401 conn(0x7fa3dc0612d0 msgr2=0x7fa3dc0a8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:34.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 shutdown_connections 2022-07-21T01:42:34.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.655+0000 7fa3e305f700 1 -- 172.21.15.3:0/1248113401 wait complete. 2022-07-21T01:42:34.664 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 27 and data pool 28 2022-07-21T01:42:34.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:34.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7359e700 1 Processor -- start 2022-07-21T01:42:34.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7359e700 1 -- start start 2022-07-21T01:42:34.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9e740af120 0x7f9e740b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7359e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e7405f840 0x7f9e7405fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7359e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e740b0c40 con 0x7f9e7405f840 2022-07-21T01:42:34.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e740b0ee0 con 0x7f9e740af120 2022-07-21T01:42:34.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740ae9e0 con 0x7f9e740ad670 2022-07-21T01:42:34.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e72d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e7405f840 0x7f9e7405fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e72d9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9e7405f840 0x7f9e7405fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51364/0 (socket says 172.21.15.3:51364) 2022-07-21T01:42:34.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e72d9d700 1 -- 172.21.15.3:0/4239388217 learned_addr learned my addr 172.21.15.3:0/4239388217 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:34.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7259c700 1 --2- 172.21.15.3:0/4239388217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e71d9b700 1 --2- 172.21.15.3:0/4239388217 >> v2:172.21.15.143:3300/0 conn(0x7f9e740af120 0x7f9e740b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e72d9d700 1 -- 172.21.15.3:0/4239388217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 msgr2=0x7f9e740b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:34.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e72d9d700 1 --2- 172.21.15.3:0/4239388217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e72d9d700 1 -- 172.21.15.3:0/4239388217 >> v2:172.21.15.143:3300/0 conn(0x7f9e740af120 msgr2=0x7f9e740b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:34.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e72d9d700 1 --2- 172.21.15.3:0/4239388217 >> v2:172.21.15.143:3300/0 conn(0x7f9e740af120 0x7f9e740b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e72d9d700 1 -- 172.21.15.3:0/4239388217 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740aec80 con 0x7f9e7405f840 2022-07-21T01:42:34.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e71d9b700 1 --2- 172.21.15.3:0/4239388217 >> v2:172.21.15.143:3300/0 conn(0x7f9e740af120 0x7f9e740b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:34.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.767+0000 7f9e7259c700 1 --2- 172.21.15.3:0/4239388217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:34.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e72d9d700 1 --2- 172.21.15.3:0/4239388217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 0x7f9e7405fcd0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f9e6c002f70 tx=0x7f9e6c006f40).ready entity=mon.0 client_cookie=c779b0106e48b35f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:34.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7159a700 1 -- 172.21.15.3:0/4239388217 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e6c00d9a0 con 0x7f9e7405f840 2022-07-21T01:42:34.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7159a700 1 -- 172.21.15.3:0/4239388217 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c000e90 con 0x7f9e7405f840 2022-07-21T01:42:34.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7159a700 1 -- 172.21.15.3:0/4239388217 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e6c00dd30 con 0x7f9e7405f840 2022-07-21T01:42:34.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 -- 172.21.15.3:0/4239388217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 msgr2=0x7f9e7405fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 --2- 172.21.15.3:0/4239388217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 0x7f9e7405fcd0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f9e6c002f70 tx=0x7f9e6c006f40).stop 2022-07-21T01:42:34.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 -- 172.21.15.3:0/4239388217 shutdown_connections 2022-07-21T01:42:34.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 --2- 172.21.15.3:0/4239388217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 0x7f9e7405fcd0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 --2- 172.21.15.3:0/4239388217 >> v2:172.21.15.143:3300/0 conn(0x7f9e740af120 0x7f9e740b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 --2- 172.21.15.3:0/4239388217 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 -- 172.21.15.3:0/4239388217 >> 172.21.15.3:0/4239388217 conn(0x7f9e74061360 msgr2=0x7f9e74062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:34.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 -- 172.21.15.3:0/4239388217 shutdown_connections 2022-07-21T01:42:34.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 -- 172.21.15.3:0/4239388217 wait complete. 2022-07-21T01:42:34.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 Processor -- start 2022-07-21T01:42:34.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 -- start start 2022-07-21T01:42:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 0x7f9e74142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e740af120 0x7f9e740ac290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e74065e30 con 0x7f9e7405f840 2022-07-21T01:42:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740aa5d0 con 0x7f9e740af120 2022-07-21T01:42:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740aa870 con 0x7f9e740ad670 2022-07-21T01:42:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 0x7f9e74142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 0x7f9e74142730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51366/0 (socket says 172.21.15.3:51366) 2022-07-21T01:42:34.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.771+0000 7f9e7259c700 1 -- 172.21.15.3:0/2897592114 learned_addr learned my addr 172.21.15.3:0/2897592114 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:34.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e71d9b700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e72d9d700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e740af120 0x7f9e740ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e71d9b700 1 -- 172.21.15.3:0/2897592114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e740af120 msgr2=0x7f9e740ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e71d9b700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e740af120 0x7f9e740ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e71d9b700 1 -- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 msgr2=0x7f9e74142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e71d9b700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 0x7f9e74142730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e71d9b700 1 -- 172.21.15.3:0/2897592114 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64009470 con 0x7f9e740ad670 2022-07-21T01:42:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e7259c700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 0x7f9e74142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e71d9b700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740abda0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f9e5c00c000 tx=0x7f9e5c006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e6b7fe700 1 -- 172.21.15.3:0/2897592114 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e5c010780 con 0x7f9e740ad670 2022-07-21T01:42:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e6b7fe700 1 -- 172.21.15.3:0/2897592114 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c0108e0 con 0x7f9e740ad670 2022-07-21T01:42:34.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c00b180 con 0x7f9e740ad670 2022-07-21T01:42:34.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e6b7fe700 1 -- 172.21.15.3:0/2897592114 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e5c010c40 con 0x7f9e740ad670 2022-07-21T01:42:34.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c00b460 con 0x7f9e740ad670 2022-07-21T01:42:34.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e740abb40 con 0x7f9e740ad670 2022-07-21T01:42:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e6b7fe700 1 -- 172.21.15.3:0/2897592114 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9e5c01c460 con 0x7f9e740ad670 2022-07-21T01:42:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e6b7fe700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e58053980 0x7f9e58055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e6b7fe700 1 -- 172.21.15.3:0/2897592114 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9e5c06c5a0 con 0x7f9e740ad670 2022-07-21T01:42:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.775+0000 7f9e7259c700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e58053980 0x7f9e58055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.779+0000 7f9e7259c700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e58053980 0x7f9e58055e40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f9e6400b000 tx=0x7f9e640091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.787+0000 7f9e6b7fe700 1 -- 172.21.15.3:0/2897592114 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9e5c043170 con 0x7f9e740ad670 2022-07-21T01:42:34.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.911+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e740b34e0 con 0x7f9e740ad670 2022-07-21T01:42:34.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.915+0000 7f9e6b7fe700 1 -- 172.21.15.3:0/2897592114 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16298 (secure 0 0 0) 0x7f9e5c046790 con 0x7f9e740ad670 2022-07-21T01:42:34.918 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:34.918 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":149,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:34.642051+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:33.092672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:34.046506+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:34.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e58053980 msgr2=0x7f9e58055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e58053980 0x7f9e58055e40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f9e6400b000 tx=0x7f9e640091d0).stop 2022-07-21T01:42:34.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 msgr2=0x7f9e740abda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:34.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740abda0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f9e5c00c000 tx=0x7f9e5c006df0).stop 2022-07-21T01:42:34.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 shutdown_connections 2022-07-21T01:42:34.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9e58053980 0x7f9e58055e40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e740af120 0x7f9e740ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e740ad670 0x7f9e740abda0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 --2- 172.21.15.3:0/2897592114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e7405f840 0x7f9e74142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:34.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 >> 172.21.15.3:0/2897592114 conn(0x7f9e74061360 msgr2=0x7f9e7405be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:34.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 shutdown_connections 2022-07-21T01:42:34.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:34.919+0000 7f9e7359e700 1 -- 172.21.15.3:0/2897592114 wait complete. 2022-07-21T01:42:34.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:35.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.031+0000 7f1f9bb94700 1 Processor -- start 2022-07-21T01:42:35.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.031+0000 7f1f9bb94700 1 -- start start 2022-07-21T01:42:35.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1f940b1f40 0x7f1f940b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940a7250 0x7f1f941448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f940b04e0 con 0x7f1f94148530 2022-07-21T01:42:35.041 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f940a9250 con 0x7f1f940b1f40 2022-07-21T01:42:35.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940b0780 con 0x7f1f940a7250 2022-07-21T01:42:35.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9912f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1f94148530 0x7f1f940b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59654/0 (socket says 172.21.15.3:59654) 2022-07-21T01:42:35.042 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9912f700 1 -- 172.21.15.3:0/3912353837 learned_addr learned my addr 172.21.15.3:0/3912353837 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:35.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f99930700 1 --2- 172.21.15.3:0/3912353837 >> v2:172.21.15.3:3300/0 conn(0x7f1f940b1f40 0x7f1f940b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9a131700 1 --2- 172.21.15.3:0/3912353837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940a7250 0x7f1f941448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9892e700 1 -- 172.21.15.3:0/3912353837 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349450404 0 0) 0x7f1f940b04e0 con 0x7f1f94148530 2022-07-21T01:42:35.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9892e700 1 -- 172.21.15.3:0/3912353837 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f7c003260 con 0x7f1f94148530 2022-07-21T01:42:35.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f99930700 1 -- 172.21.15.3:0/3912353837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940a7250 msgr2=0x7f1f941448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:35.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f99930700 1 --2- 172.21.15.3:0/3912353837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940a7250 0x7f1f941448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f99930700 1 -- 172.21.15.3:0/3912353837 >> v1:172.21.15.3:6789/0 conn(0x7f1f94148530 legacy=0x7f1f940b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f99930700 1 -- 172.21.15.3:0/3912353837 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f940b0a20 con 0x7f1f940b1f40 2022-07-21T01:42:35.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9a131700 1 --2- 172.21.15.3:0/3912353837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940a7250 0x7f1f941448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:35.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f99930700 1 --2- 172.21.15.3:0/3912353837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940b1f40 0x7f1f940b1160 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7f1f941478c0 tx=0x7f1f84006df0).ready entity=mon.0 client_cookie=2d5d77b3f147f1d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:35.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9892e700 1 -- 172.21.15.3:0/3912353837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f8400d570 con 0x7f1f940b1f40 2022-07-21T01:42:35.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9892e700 1 -- 172.21.15.3:0/3912353837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f84000e90 con 0x7f1f940b1f40 2022-07-21T01:42:35.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9892e700 1 -- 172.21.15.3:0/3912353837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f8400d8c0 con 0x7f1f940b1f40 2022-07-21T01:42:35.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 -- 172.21.15.3:0/3912353837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940b1f40 msgr2=0x7f1f940b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.046 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 --2- 172.21.15.3:0/3912353837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940b1f40 0x7f1f940b1160 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7f1f941478c0 tx=0x7f1f84006df0).stop 2022-07-21T01:42:35.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 -- 172.21.15.3:0/3912353837 shutdown_connections 2022-07-21T01:42:35.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 --2- 172.21.15.3:0/3912353837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940a7250 0x7f1f941448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 --2- 172.21.15.3:0/3912353837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940b1f40 0x7f1f940b1160 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.047 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 -- 172.21.15.3:0/3912353837 >> 172.21.15.3:0/3912353837 conn(0x7f1f9405f840 msgr2=0x7f1f9405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:35.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 -- 172.21.15.3:0/3912353837 shutdown_connections 2022-07-21T01:42:35.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.035+0000 7f1f9bb94700 1 -- 172.21.15.3:0/3912353837 wait complete. 2022-07-21T01:42:35.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 Processor -- start 2022-07-21T01:42:35.048 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 -- start start 2022-07-21T01:42:35.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940a7250 0x7f1f940a9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940b1f40 0x7f1f94143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f94148530 0x7f1f94144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940a9f40 con 0x7f1f940a7250 2022-07-21T01:42:35.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f941444f0 con 0x7f1f94148530 2022-07-21T01:42:35.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f940aafd0 con 0x7f1f940b1f40 2022-07-21T01:42:35.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f99930700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940a7250 0x7f1f940a9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f99930700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940a7250 0x7f1f940a9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51380/0 (socket says 172.21.15.3:51380) 2022-07-21T01:42:35.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f99930700 1 -- 172.21.15.3:0/2695633792 learned_addr learned my addr 172.21.15.3:0/2695633792 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:35.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9912f700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940b1f40 0x7f1f94143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9a131700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f94148530 0x7f1f94144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f99930700 1 -- 172.21.15.3:0/2695633792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940b1f40 msgr2=0x7f1f94143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f99930700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940b1f40 0x7f1f94143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f99930700 1 -- 172.21.15.3:0/2695633792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f94148530 msgr2=0x7f1f94144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f99930700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f94148530 0x7f1f94144000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f99930700 1 -- 172.21.15.3:0/2695633792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8400b180 con 0x7f1f940a7250 2022-07-21T01:42:35.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9a131700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f94148530 0x7f1f94144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:35.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f99930700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940a7250 0x7f1f940a9910 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f1f941478c0 tx=0x7f1f84014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:35.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f82ffd700 1 -- 172.21.15.3:0/2695633792 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f84006b20 con 0x7f1f940a7250 2022-07-21T01:42:35.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f82ffd700 1 -- 172.21.15.3:0/2695633792 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f84013e90 con 0x7f1f940a7250 2022-07-21T01:42:35.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f940ab270 con 0x7f1f940a7250 2022-07-21T01:42:35.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f82ffd700 1 -- 172.21.15.3:0/2695633792 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f8401e5f0 con 0x7f1f940a7250 2022-07-21T01:42:35.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.039+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f940ab7b0 con 0x7f1f940a7250 2022-07-21T01:42:35.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.043+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f940574e0 con 0x7f1f940a7250 2022-07-21T01:42:35.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.043+0000 7f1f82ffd700 1 -- 172.21.15.3:0/2695633792 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1f84025030 con 0x7f1f940a7250 2022-07-21T01:42:35.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.043+0000 7f1f82ffd700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1f740539e0 0x7f1f74055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.043+0000 7f1f82ffd700 1 -- 172.21.15.3:0/2695633792 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f1f8406ccc0 con 0x7f1f940a7250 2022-07-21T01:42:35.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.043+0000 7f1f9912f700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1f740539e0 0x7f1f74055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.043+0000 7f1f9912f700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1f740539e0 0x7f1f74055ea0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f1f9000d000 tx=0x7f1f90009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:35.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.051+0000 7f1f82ffd700 1 -- 172.21.15.3:0/2695633792 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f84073080 con 0x7f1f940a7250 2022-07-21T01:42:35.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.175+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f9414a6c0 con 0x7f1f940a7250 2022-07-21T01:42:35.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f82ffd700 1 -- 172.21.15.3:0/2695633792 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16298 (secure 0 0 0) 0x7f1f84023070 con 0x7f1f940a7250 2022-07-21T01:42:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:35.182 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":149,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:34.642051+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:33.092672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:34.046506+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:35.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1f740539e0 msgr2=0x7f1f74055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1f740539e0 0x7f1f74055ea0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f1f9000d000 tx=0x7f1f90009040).stop 2022-07-21T01:42:35.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940a7250 msgr2=0x7f1f940a9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940a7250 0x7f1f940a9910 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f1f941478c0 tx=0x7f1f84014000).stop 2022-07-21T01:42:35.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 shutdown_connections 2022-07-21T01:42:35.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1f740539e0 0x7f1f74055ea0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f94148530 0x7f1f94144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f940b1f40 0x7f1f94143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 --2- 172.21.15.3:0/2695633792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f940a7250 0x7f1f940a9910 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 >> 172.21.15.3:0/2695633792 conn(0x7f1f9405f840 msgr2=0x7f1f940a79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:35.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 shutdown_connections 2022-07-21T01:42:35.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.179+0000 7f1f9bb94700 1 -- 172.21.15.3:0/2695633792 wait complete. 2022-07-21T01:42:35.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:42:35.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.311+0000 7fdbeb57f700 1 Processor -- start 2022-07-21T01:42:35.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 -- start start 2022-07-21T01:42:35.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdbe40aebd0 0x7fdbe4143940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40ad190 0x7fdbe4143e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdbe40b0610 0x7fdbe4146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe40adde0 con 0x7fdbe40aebd0 2022-07-21T01:42:35.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe40b0b20 con 0x7fdbe40b0610 2022-07-21T01:42:35.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe4146890 con 0x7fdbe40ad190 2022-07-21T01:42:35.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbea57d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdbe40aebd0 0x7fdbe4143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbea57d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdbe40aebd0 0x7fdbe4143940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51388/0 (socket says 172.21.15.3:51388) 2022-07-21T01:42:35.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbea57d700 1 -- 172.21.15.3:0/1821045970 learned_addr learned my addr 172.21.15.3:0/1821045970 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:35.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe9d7c700 1 --2- 172.21.15.3:0/1821045970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40ad190 0x7fdbe4143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbead7e700 1 --2- 172.21.15.3:0/1821045970 >> v2:172.21.15.143:3300/0 conn(0x7fdbe40b0610 0x7fdbe4146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe9d7c700 1 -- 172.21.15.3:0/1821045970 >> v2:172.21.15.143:3300/0 conn(0x7fdbe40b0610 msgr2=0x7fdbe4146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:35.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe9d7c700 1 --2- 172.21.15.3:0/1821045970 >> v2:172.21.15.143:3300/0 conn(0x7fdbe40b0610 0x7fdbe4146cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe9d7c700 1 -- 172.21.15.3:0/1821045970 >> v2:172.21.15.3:3300/0 conn(0x7fdbe40aebd0 msgr2=0x7fdbe4143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:35.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe9d7c700 1 --2- 172.21.15.3:0/1821045970 >> v2:172.21.15.3:3300/0 conn(0x7fdbe40aebd0 0x7fdbe4143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe9d7c700 1 -- 172.21.15.3:0/1821045970 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe40aefb0 con 0x7fdbe40ad190 2022-07-21T01:42:35.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbea57d700 1 --2- 172.21.15.3:0/1821045970 >> v2:172.21.15.3:3300/0 conn(0x7fdbe40aebd0 0x7fdbe4143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:35.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe9d7c700 1 --2- 172.21.15.3:0/1821045970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40ad190 0x7fdbe4143e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fdbd8002f70 tx=0x7fdbd8006df0).ready entity=mon.2 client_cookie=d969d89c8274223d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:35.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe957b700 1 -- 172.21.15.3:0/1821045970 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbd800d4a0 con 0x7fdbe40ad190 2022-07-21T01:42:35.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe957b700 1 -- 172.21.15.3:0/1821045970 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd8000e90 con 0x7fdbe40ad190 2022-07-21T01:42:35.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbe957b700 1 -- 172.21.15.3:0/1821045970 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbd800d7f0 con 0x7fdbe40ad190 2022-07-21T01:42:35.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 -- 172.21.15.3:0/1821045970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40ad190 msgr2=0x7fdbe4143e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/1821045970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40ad190 0x7fdbe4143e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fdbd8002f70 tx=0x7fdbd8006df0).stop 2022-07-21T01:42:35.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 -- 172.21.15.3:0/1821045970 shutdown_connections 2022-07-21T01:42:35.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/1821045970 >> v2:172.21.15.143:3300/0 conn(0x7fdbe40b0610 0x7fdbe4146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/1821045970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40ad190 0x7fdbe4143e30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/1821045970 >> v2:172.21.15.3:3300/0 conn(0x7fdbe40aebd0 0x7fdbe4143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.315+0000 7fdbeb57f700 1 -- 172.21.15.3:0/1821045970 >> 172.21.15.3:0/1821045970 conn(0x7fdbe40613c0 msgr2=0x7fdbe4062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:35.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 -- 172.21.15.3:0/1821045970 shutdown_connections 2022-07-21T01:42:35.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 -- 172.21.15.3:0/1821045970 wait complete. 2022-07-21T01:42:35.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 Processor -- start 2022-07-21T01:42:35.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 -- start start 2022-07-21T01:42:35.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe40ad190 0x7fdbe40b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe40aebd0 0x7fdbe40afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40b0610 0x7fdbe4067290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe4066bc0 con 0x7fdbe40aebd0 2022-07-21T01:42:35.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe40b6010 con 0x7fdbe40ad190 2022-07-21T01:42:35.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe4067780 con 0x7fdbe40b0610 2022-07-21T01:42:35.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbe9d7c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe40aebd0 0x7fdbe40afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbe9d7c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe40aebd0 0x7fdbe40afad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51394/0 (socket says 172.21.15.3:51394) 2022-07-21T01:42:35.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbe9d7c700 1 -- 172.21.15.3:0/2969124150 learned_addr learned my addr 172.21.15.3:0/2969124150 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:35.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbea57d700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe40ad190 0x7fdbe40b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbead7e700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40b0610 0x7fdbe4067290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbe9d7c700 1 -- 172.21.15.3:0/2969124150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40b0610 msgr2=0x7fdbe4067290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.359 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbe9d7c700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40b0610 0x7fdbe4067290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbe9d7c700 1 -- 172.21.15.3:0/2969124150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe40ad190 msgr2=0x7fdbe40b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbe9d7c700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe40ad190 0x7fdbe40b57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbe9d7c700 1 -- 172.21.15.3:0/2969124150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe000b1e0 con 0x7fdbe40aebd0 2022-07-21T01:42:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbe9d7c700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe40aebd0 0x7fdbe40afad0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0x7fdbe4067910 tx=0x7fdbd8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:35.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbd77fe700 1 -- 172.21.15.3:0/2969124150 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbd8016070 con 0x7fdbe40aebd0 2022-07-21T01:42:35.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd800b180 con 0x7fdbe40aebd0 2022-07-21T01:42:35.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe4066df0 con 0x7fdbe40aebd0 2022-07-21T01:42:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbd77fe700 1 -- 172.21.15.3:0/2969124150 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd800de50 con 0x7fdbe40aebd0 2022-07-21T01:42:35.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.319+0000 7fdbd77fe700 1 -- 172.21.15.3:0/2969124150 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbd8013e50 con 0x7fdbe40aebd0 2022-07-21T01:42:35.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.323+0000 7fdbd77fe700 1 -- 172.21.15.3:0/2969124150 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdbd800f030 con 0x7fdbe40aebd0 2022-07-21T01:42:35.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.323+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe40b0d50 con 0x7fdbe40aebd0 2022-07-21T01:42:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.323+0000 7fdbd77fe700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbc40539e0 0x7fdbc4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.323+0000 7fdbd77fe700 1 -- 172.21.15.3:0/2969124150 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdbd806ce30 con 0x7fdbe40aebd0 2022-07-21T01:42:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.323+0000 7fdbea57d700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbc40539e0 0x7fdbc4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.323+0000 7fdbea57d700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbc40539e0 0x7fdbc4055ea0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7fdbe000b000 tx=0x7fdbe0006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:35.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.331+0000 7fdbd77fe700 1 -- 172.21.15.3:0/2969124150 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdbd8043a60 con 0x7fdbe40aebd0 2022-07-21T01:42:35.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.491+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdbe40574e0 con 0x7fdbe40aebd0 2022-07-21T01:42:35.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbd77fe700 1 -- 172.21.15.3:0/2969124150 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdbd8023020 con 0x7fdbe40aebd0 2022-07-21T01:42:35.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbc40539e0 msgr2=0x7fdbc4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbc40539e0 0x7fdbc4055ea0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7fdbe000b000 tx=0x7fdbe0006eb0).stop 2022-07-21T01:42:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe40aebd0 msgr2=0x7fdbe40afad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe40aebd0 0x7fdbe40afad0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0x7fdbe4067910 tx=0x7fdbd8014000).stop 2022-07-21T01:42:35.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 shutdown_connections 2022-07-21T01:42:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40b0610 0x7fdbe4067290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe40aebd0 0x7fdbe40afad0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbc40539e0 0x7fdbc4055ea0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.663+0000 7fdbeb57f700 1 --2- 172.21.15.3:0/2969124150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe40ad190 0x7fdbe40b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.667+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 >> 172.21.15.3:0/2969124150 conn(0x7fdbe40613c0 msgr2=0x7fdbe4145140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.667+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 shutdown_connections 2022-07-21T01:42:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.667+0000 7fdbeb57f700 1 -- 172.21.15.3:0/2969124150 wait complete. 2022-07-21T01:42:35.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:35.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9ac31700 1 Processor -- start 2022-07-21T01:42:35.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9ac31700 1 -- start start 2022-07-21T01:42:35.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9ac31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940ad4b0 0x7f7d940af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9ac31700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7d940ad9e0 0x7f7d940a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9ac31700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d940aff30 con 0x7f7d940af450 2022-07-21T01:42:35.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9ac31700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9405c7a0 con 0x7f7d940ad9e0 2022-07-21T01:42:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9ac31700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d94140990 con 0x7f7d940ad4b0 2022-07-21T01:42:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9942e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940ad4b0 0x7f7d940af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9a430700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7d940ad9e0 0x7f7d940a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9942e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940ad4b0 0x7f7d940af830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:58318/0 (socket says 172.21.15.3:58318) 2022-07-21T01:42:35.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9942e700 1 -- 172.21.15.3:0/379543814 learned_addr learned my addr 172.21.15.3:0/379543814 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:35.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9942e700 1 -- 172.21.15.3:0/379543814 >> v2:172.21.15.143:3300/0 conn(0x7f7d940ad9e0 msgr2=0x7f7d940a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:35.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9942e700 1 --2- 172.21.15.3:0/379543814 >> v2:172.21.15.143:3300/0 conn(0x7f7d940ad9e0 0x7f7d940a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9942e700 1 -- 172.21.15.3:0/379543814 >> v1:172.21.15.143:6789/0 conn(0x7f7d940af450 legacy=0x7f7d941432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9942e700 1 -- 172.21.15.3:0/379543814 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d941439f0 con 0x7f7d940ad4b0 2022-07-21T01:42:35.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.779+0000 7f7d9942e700 1 --2- 172.21.15.3:0/379543814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940ad4b0 0x7f7d940af830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f7d900032f0 tx=0x7f7d90009290).ready entity=mon.2 client_cookie=1a8e05fbda1e0bbc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:35.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d98c2d700 1 -- 172.21.15.3:0/379543814 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d9000b8a0 con 0x7f7d940ad4b0 2022-07-21T01:42:35.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d98c2d700 1 -- 172.21.15.3:0/379543814 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9000ba00 con 0x7f7d940ad4b0 2022-07-21T01:42:35.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d98c2d700 1 -- 172.21.15.3:0/379543814 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d9000bb70 con 0x7f7d940ad4b0 2022-07-21T01:42:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 -- 172.21.15.3:0/379543814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940ad4b0 msgr2=0x7f7d940af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 --2- 172.21.15.3:0/379543814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940ad4b0 0x7f7d940af830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f7d900032f0 tx=0x7f7d90009290).stop 2022-07-21T01:42:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 -- 172.21.15.3:0/379543814 shutdown_connections 2022-07-21T01:42:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 --2- 172.21.15.3:0/379543814 >> v2:172.21.15.143:3300/0 conn(0x7f7d940ad9e0 0x7f7d940a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 --2- 172.21.15.3:0/379543814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940ad4b0 0x7f7d940af830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 -- 172.21.15.3:0/379543814 >> 172.21.15.3:0/379543814 conn(0x7f7d940612d0 msgr2=0x7f7d94062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 -- 172.21.15.3:0/379543814 shutdown_connections 2022-07-21T01:42:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 -- 172.21.15.3:0/379543814 wait complete. 2022-07-21T01:42:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 Processor -- start 2022-07-21T01:42:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 -- start start 2022-07-21T01:42:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d940ad4b0 0x7f7d9413c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d940ad9e0 0x7f7d940647f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940af450 0x7f7d94064ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940b6160 con 0x7f7d940ad9e0 2022-07-21T01:42:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940b65e0 con 0x7f7d940ad4b0 2022-07-21T01:42:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9ac31700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d94143700 con 0x7f7d940af450 2022-07-21T01:42:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.783+0000 7f7d9942e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d940ad9e0 0x7f7d940647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9942e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d940ad9e0 0x7f7d940647f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51408/0 (socket says 172.21.15.3:51408) 2022-07-21T01:42:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9942e700 1 -- 172.21.15.3:0/421172498 learned_addr learned my addr 172.21.15.3:0/421172498 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d99c2f700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d940ad4b0 0x7f7d9413c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9a430700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940af450 0x7f7d94064ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9942e700 1 -- 172.21.15.3:0/421172498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940af450 msgr2=0x7f7d94064ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9942e700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940af450 0x7f7d94064ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9942e700 1 -- 172.21.15.3:0/421172498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d940ad4b0 msgr2=0x7f7d9413c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9942e700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d940ad4b0 0x7f7d9413c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9942e700 1 -- 172.21.15.3:0/421172498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d90009530 con 0x7f7d940ad9e0 2022-07-21T01:42:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9a430700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940af450 0x7f7d94064ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9942e700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d940ad9e0 0x7f7d940647f0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7f7d900032f0 tx=0x7f7d90016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d86ffd700 1 -- 172.21.15.3:0/421172498 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d9000b8a0 con 0x7f7d940ad9e0 2022-07-21T01:42:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d86ffd700 1 -- 172.21.15.3:0/421172498 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d90021440 con 0x7f7d940ad9e0 2022-07-21T01:42:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d94154240 con 0x7f7d940ad9e0 2022-07-21T01:42:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d86ffd700 1 -- 172.21.15.3:0/421172498 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d900217a0 con 0x7f7d940ad9e0 2022-07-21T01:42:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d941502e0 con 0x7f7d940ad9e0 2022-07-21T01:42:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d940574e0 con 0x7f7d940ad9e0 2022-07-21T01:42:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d86ffd700 1 -- 172.21.15.3:0/421172498 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7d90029070 con 0x7f7d940ad9e0 2022-07-21T01:42:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d86ffd700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d74053b00 0x7f7d74055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d86ffd700 1 -- 172.21.15.3:0/421172498 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f7d9006d210 con 0x7f7d940ad9e0 2022-07-21T01:42:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.787+0000 7f7d99c2f700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d74053b00 0x7f7d74055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.791+0000 7f7d99c2f700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d74053b00 0x7f7d74055fc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f7d8800d000 tx=0x7f7d88009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.791+0000 7f7d86ffd700 1 -- 172.21.15.3:0/421172498 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7d90043f30 con 0x7f7d940ad9e0 2022-07-21T01:42:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.955+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d94061b20 con 0x7f7d940ad9e0 2022-07-21T01:42:35.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.955+0000 7f7d86ffd700 1 -- 172.21.15.3:0/421172498 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2492 (secure 0 0 0) 0x7f7d90015070 con 0x7f7d940ad9e0 2022-07-21T01:42:35.959 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:35.959 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/1563084587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1563084587},{"type":"v1","addr":"172.21.15.3:6835","nonce":1563084587}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1403026914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1403026914},{"type":"v1","addr":"172.21.15.3:6837","nonce":1403026914}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:34.642400+0000","modified":"2022-07-21T01:42:35.646272+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8409":{"gid":8409,"name":"b","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/1158093851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1158093851},{"type":"v1","addr":"172.21.15.143:6833","nonce":1158093851}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-21T01:42:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.955+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d74053b00 msgr2=0x7f7d74055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.955+0000 7f7d9ac31700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d74053b00 0x7f7d74055fc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f7d8800d000 tx=0x7f7d88009040).stop 2022-07-21T01:42:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d940ad9e0 msgr2=0x7f7d940647f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d940ad9e0 0x7f7d940647f0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7f7d900032f0 tx=0x7f7d90016000).stop 2022-07-21T01:42:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 shutdown_connections 2022-07-21T01:42:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d940af450 0x7f7d94064ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d940ad9e0 0x7f7d940647f0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7d74053b00 0x7f7d74055fc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 --2- 172.21.15.3:0/421172498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d940ad4b0 0x7f7d9413c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 >> 172.21.15.3:0/421172498 conn(0x7f7d940612d0 msgr2=0x7f7d940a8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:35.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 shutdown_connections 2022-07-21T01:42:35.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:35.959+0000 7f7d9ac31700 1 -- 172.21.15.3:0/421172498 wait complete. 2022-07-21T01:42:35.968 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 76 2022-07-21T01:42:35.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.075+0000 7f27cd2cc700 1 Processor -- start 2022-07-21T01:42:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.075+0000 7f27cd2cc700 1 -- start start 2022-07-21T01:42:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f27c80b2710 0x7f27c814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b01a0 0x7f27c814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c80a7250 con 0x7f27c80afd60 2022-07-21T01:42:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c80a7480 con 0x7f27c80b2710 2022-07-21T01:42:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c80a7720 con 0x7f27c80b01a0 2022-07-21T01:42:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27ccacb700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f27c80afd60 0x7f27c80a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59698/0 (socket says 172.21.15.3:59698) 2022-07-21T01:42:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27ccacb700 1 -- 172.21.15.3:0/1322228966 learned_addr learned my addr 172.21.15.3:0/1322228966 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c7fff700 1 --2- 172.21.15.3:0/1322228966 >> v2:172.21.15.3:3300/0 conn(0x7f27c80b2710 0x7f27c814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c77fe700 1 --2- 172.21.15.3:0/1322228966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b01a0 0x7f27c814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c6ffd700 1 -- 172.21.15.3:0/1322228966 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388496108 0 0) 0x7f27c80a7250 con 0x7f27c80afd60 2022-07-21T01:42:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c6ffd700 1 -- 172.21.15.3:0/1322228966 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b4003260 con 0x7f27c80afd60 2022-07-21T01:42:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c7fff700 1 -- 172.21.15.3:0/1322228966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b01a0 msgr2=0x7f27c814b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c7fff700 1 --2- 172.21.15.3:0/1322228966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b01a0 0x7f27c814b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c7fff700 1 -- 172.21.15.3:0/1322228966 >> v1:172.21.15.3:6789/0 conn(0x7f27c80afd60 legacy=0x7f27c80a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c7fff700 1 -- 172.21.15.3:0/1322228966 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c80a79c0 con 0x7f27c80b2710 2022-07-21T01:42:36.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c77fe700 1 --2- 172.21.15.3:0/1322228966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b01a0 0x7f27c814b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:36.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c7fff700 1 --2- 172.21.15.3:0/1322228966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80b2710 0x7f27c814afd0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f27bc002f70 tx=0x7f27bc006df0).ready entity=mon.0 client_cookie=3c7af393e62fe61e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:36.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c6ffd700 1 -- 172.21.15.3:0/1322228966 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27bc00d570 con 0x7f27c80b2710 2022-07-21T01:42:36.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c6ffd700 1 -- 172.21.15.3:0/1322228966 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc000e90 con 0x7f27c80b2710 2022-07-21T01:42:36.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27c6ffd700 1 -- 172.21.15.3:0/1322228966 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27bc00d8c0 con 0x7f27c80b2710 2022-07-21T01:42:36.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 -- 172.21.15.3:0/1322228966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80b2710 msgr2=0x7f27c814afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 --2- 172.21.15.3:0/1322228966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80b2710 0x7f27c814afd0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f27bc002f70 tx=0x7f27bc006df0).stop 2022-07-21T01:42:36.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 -- 172.21.15.3:0/1322228966 shutdown_connections 2022-07-21T01:42:36.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 --2- 172.21.15.3:0/1322228966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b01a0 0x7f27c814b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 --2- 172.21.15.3:0/1322228966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80b2710 0x7f27c814afd0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 -- 172.21.15.3:0/1322228966 >> 172.21.15.3:0/1322228966 conn(0x7f27c8060310 msgr2=0x7f27c80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:36.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 -- 172.21.15.3:0/1322228966 shutdown_connections 2022-07-21T01:42:36.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 -- 172.21.15.3:0/1322228966 wait complete. 2022-07-21T01:42:36.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 Processor -- start 2022-07-21T01:42:36.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.079+0000 7f27cd2cc700 1 -- start start 2022-07-21T01:42:36.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27cd2cc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80afd60 0x7f27c80b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27cd2cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c80b01a0 0x7f27c814cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27cd2cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b2710 0x7f27c8156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27cd2cc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c80b5da0 con 0x7f27c80afd60 2022-07-21T01:42:36.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27cd2cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c8144640 con 0x7f27c80b01a0 2022-07-21T01:42:36.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27cd2cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c80b4450 con 0x7f27c80b2710 2022-07-21T01:42:36.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80afd60 0x7f27c80b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80afd60 0x7f27c80b3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51422/0 (socket says 172.21.15.3:51422) 2022-07-21T01:42:36.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c7fff700 1 -- 172.21.15.3:0/674272105 learned_addr learned my addr 172.21.15.3:0/674272105 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:36.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c77fe700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c80b01a0 0x7f27c814cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27ccacb700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b2710 0x7f27c8156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c7fff700 1 -- 172.21.15.3:0/674272105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b2710 msgr2=0x7f27c8156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c7fff700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b2710 0x7f27c8156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c7fff700 1 -- 172.21.15.3:0/674272105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c80b01a0 msgr2=0x7f27c814cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c7fff700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c80b01a0 0x7f27c814cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c7fff700 1 -- 172.21.15.3:0/674272105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc00b180 con 0x7f27c80afd60 2022-07-21T01:42:36.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27ccacb700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b2710 0x7f27c8156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:36.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c7fff700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80afd60 0x7f27c80b3570 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f27bc002f70 tx=0x7f27bc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:36.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c57fa700 1 -- 172.21.15.3:0/674272105 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27bc006b20 con 0x7f27c80afd60 2022-07-21T01:42:36.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c57fa700 1 -- 172.21.15.3:0/674272105 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc013e90 con 0x7f27c80afd60 2022-07-21T01:42:36.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c80b5f90 con 0x7f27c80afd60 2022-07-21T01:42:36.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27c57fa700 1 -- 172.21.15.3:0/674272105 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27bc00dea0 con 0x7f27c80afd60 2022-07-21T01:42:36.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.083+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c80b48e0 con 0x7f27c80afd60 2022-07-21T01:42:36.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.087+0000 7f27c57fa700 1 -- 172.21.15.3:0/674272105 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f27bc025030 con 0x7f27c80afd60 2022-07-21T01:42:36.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.087+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c80574e0 con 0x7f27c80afd60 2022-07-21T01:42:36.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.087+0000 7f27c57fa700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27a8053b00 0x7f27a8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.087+0000 7f27c57fa700 1 -- 172.21.15.3:0/674272105 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f27bc06cd50 con 0x7f27c80afd60 2022-07-21T01:42:36.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.087+0000 7f27c77fe700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27a8053b00 0x7f27a8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.087+0000 7f27c77fe700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27a8053b00 0x7f27a8055fc0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f27b800c000 tx=0x7f27b8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:36.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.095+0000 7f27c57fa700 1 -- 172.21.15.3:0/674272105 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f27bc043b10 con 0x7f27c80afd60 2022-07-21T01:42:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.219+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27c8069a70 con 0x7f27c80afd60 2022-07-21T01:42:36.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27c57fa700 1 -- 172.21.15.3:0/674272105 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f27bc023070 con 0x7f27c80afd60 2022-07-21T01:42:36.227 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:36.227 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":150,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:35.645281+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:33.092672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:34.046506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:36.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27a8053b00 msgr2=0x7f27a8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27a8053b00 0x7f27a8055fc0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f27b800c000 tx=0x7f27b8008040).stop 2022-07-21T01:42:36.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80afd60 msgr2=0x7f27c80b3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80afd60 0x7f27c80b3570 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f27bc002f70 tx=0x7f27bc014000).stop 2022-07-21T01:42:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 shutdown_connections 2022-07-21T01:42:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27c80b2710 0x7f27c8156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f27a8053b00 0x7f27a8055fc0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c80b01a0 0x7f27c814cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 --2- 172.21.15.3:0/674272105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27c80afd60 0x7f27c80b3570 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 >> 172.21.15.3:0/674272105 conn(0x7f27c8060310 msgr2=0x7f27c8148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.223+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 shutdown_connections 2022-07-21T01:42:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.227+0000 7f27cd2cc700 1 -- 172.21.15.3:0/674272105 wait complete. 2022-07-21T01:42:36.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:42:36.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd294f700 1 Processor -- start 2022-07-21T01:42:36.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd294f700 1 -- start start 2022-07-21T01:42:36.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd294f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0614b0 0x7f9bcc061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd294f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9bcc0ad750 0x7f9bcc0b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd294f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0aee80 con 0x7f9bcc1405d0 2022-07-21T01:42:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd294f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bcc0b70b0 con 0x7f9bcc0ad750 2022-07-21T01:42:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd294f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0af0b0 con 0x7f9bcc0614b0 2022-07-21T01:42:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd114c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9bcc0ad750 0x7f9bcc0b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd194d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0614b0 0x7f9bcc061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd194d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0614b0 0x7f9bcc061890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:58346/0 (socket says 172.21.15.3:58346) 2022-07-21T01:42:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd114c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9bcc0ad750 0x7f9bcc0b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:56880/0 (socket says 172.21.15.3:56880) 2022-07-21T01:42:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd194d700 1 -- 172.21.15.3:0/738808068 learned_addr learned my addr 172.21.15.3:0/738808068 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd114c700 1 -- 172.21.15.3:0/738808068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0614b0 msgr2=0x7f9bcc061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd114c700 1 --2- 172.21.15.3:0/738808068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0614b0 0x7f9bcc061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd114c700 1 -- 172.21.15.3:0/738808068 >> v1:172.21.15.143:6789/0 conn(0x7f9bcc1405d0 legacy=0x7f9bcc1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd114c700 1 -- 172.21.15.3:0/738808068 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0af350 con 0x7f9bcc0ad750 2022-07-21T01:42:36.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd194d700 1 --2- 172.21.15.3:0/738808068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0614b0 0x7f9bcc061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:36.358 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd114c700 1 --2- 172.21.15.3:0/738808068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bcc0ad750 0x7f9bcc0b9ff0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f9bc0002f70 tx=0x7f9bc0006df0).ready entity=mon.1 client_cookie=5dce5183315b4155 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:36.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd094b700 1 -- 172.21.15.3:0/738808068 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bc000d570 con 0x7f9bcc0ad750 2022-07-21T01:42:36.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd094b700 1 -- 172.21.15.3:0/738808068 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc0000e90 con 0x7f9bcc0ad750 2022-07-21T01:42:36.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.351+0000 7f9bd094b700 1 -- 172.21.15.3:0/738808068 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bc000d8c0 con 0x7f9bcc0ad750 2022-07-21T01:42:36.362 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 -- 172.21.15.3:0/738808068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bcc0ad750 msgr2=0x7f9bcc0b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 --2- 172.21.15.3:0/738808068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bcc0ad750 0x7f9bcc0b9ff0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f9bc0002f70 tx=0x7f9bc0006df0).stop 2022-07-21T01:42:36.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 -- 172.21.15.3:0/738808068 shutdown_connections 2022-07-21T01:42:36.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 --2- 172.21.15.3:0/738808068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bcc0ad750 0x7f9bcc0b9ff0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 --2- 172.21.15.3:0/738808068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0614b0 0x7f9bcc061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 -- 172.21.15.3:0/738808068 >> 172.21.15.3:0/738808068 conn(0x7f9bcc066f70 msgr2=0x7f9bcc067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:36.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 -- 172.21.15.3:0/738808068 shutdown_connections 2022-07-21T01:42:36.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 -- 172.21.15.3:0/738808068 wait complete. 2022-07-21T01:42:36.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 Processor -- start 2022-07-21T01:42:36.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 -- start start 2022-07-21T01:42:36.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bcc0614b0 0x7f9bcc152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0ad750 0x7f9bcc152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bcc1405d0 0x7f9bcc0abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc153070 con 0x7f9bcc0614b0 2022-07-21T01:42:36.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0aa960 con 0x7f9bcc1405d0 2022-07-21T01:42:36.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd294f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0aaee0 con 0x7f9bcc0ad750 2022-07-21T01:42:36.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd194d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bcc0614b0 0x7f9bcc152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd194d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bcc0614b0 0x7f9bcc152690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51438/0 (socket says 172.21.15.3:51438) 2022-07-21T01:42:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd194d700 1 -- 172.21.15.3:0/3137435214 learned_addr learned my addr 172.21.15.3:0/3137435214 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd114c700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0ad750 0x7f9bcc152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd214e700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bcc1405d0 0x7f9bcc0abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd194d700 1 -- 172.21.15.3:0/3137435214 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0ad750 msgr2=0x7f9bcc152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd194d700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0ad750 0x7f9bcc152b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd194d700 1 -- 172.21.15.3:0/3137435214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bcc1405d0 msgr2=0x7f9bcc0abce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd194d700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bcc1405d0 0x7f9bcc0abce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd194d700 1 -- 172.21.15.3:0/3137435214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc8009530 con 0x7f9bcc0614b0 2022-07-21T01:42:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd114c700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0ad750 0x7f9bcc152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.355+0000 7f9bd194d700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bcc0614b0 0x7f9bcc152690 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f9bc8006b60 tx=0x7f9bc8009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bbeffd700 1 -- 172.21.15.3:0/3137435214 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bc800c850 con 0x7f9bcc0614b0 2022-07-21T01:42:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bbeffd700 1 -- 172.21.15.3:0/3137435214 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc800c9b0 con 0x7f9bcc0614b0 2022-07-21T01:42:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bbeffd700 1 -- 172.21.15.3:0/3137435214 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bc800cd10 con 0x7f9bcc0614b0 2022-07-21T01:42:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc000b180 con 0x7f9bcc0614b0 2022-07-21T01:42:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc14a740 con 0x7f9bcc0614b0 2022-07-21T01:42:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc0aab20 con 0x7f9bcc0614b0 2022-07-21T01:42:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bbeffd700 1 -- 172.21.15.3:0/3137435214 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9bc8022070 con 0x7f9bcc0614b0 2022-07-21T01:42:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bbeffd700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9bac053b30 0x7f9bac055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bd114c700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9bac053b30 0x7f9bac055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.359+0000 7f9bbeffd700 1 -- 172.21.15.3:0/3137435214 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9bc806c780 con 0x7f9bcc0614b0 2022-07-21T01:42:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.363+0000 7f9bd114c700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9bac053b30 0x7f9bac055ff0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f9bc001a000 tx=0x7f9bc0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.367+0000 7f9bbeffd700 1 -- 172.21.15.3:0/3137435214 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9bc8043540 con 0x7f9bcc0614b0 2022-07-21T01:42:36.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.531+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9bcc0ab0a0 con 0x7f9bcc0614b0 2022-07-21T01:42:36.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bbeffd700 1 -- 172.21.15.3:0/3137435214 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v36) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9bc8046b60 con 0x7f9bcc0614b0 2022-07-21T01:42:36.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9bac053b30 msgr2=0x7f9bac055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9bac053b30 0x7f9bac055ff0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f9bc001a000 tx=0x7f9bc0014000).stop 2022-07-21T01:42:36.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bcc0614b0 msgr2=0x7f9bcc152690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bcc0614b0 0x7f9bcc152690 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f9bc8006b60 tx=0x7f9bc8009290).stop 2022-07-21T01:42:36.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 shutdown_connections 2022-07-21T01:42:36.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9bac053b30 0x7f9bac055ff0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bcc1405d0 0x7f9bcc0abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bcc0ad750 0x7f9bcc152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 --2- 172.21.15.3:0/3137435214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9bcc0614b0 0x7f9bcc152690 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.539+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 >> 172.21.15.3:0/3137435214 conn(0x7f9bcc066f70 msgr2=0x7f9bcc067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:36.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.543+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 shutdown_connections 2022-07-21T01:42:36.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.543+0000 7f9bd294f700 1 -- 172.21.15.3:0/3137435214 wait complete. 2022-07-21T01:42:36.551 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:42:36.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:42:36.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b79df3700 1 Processor -- start 2022-07-21T01:42:36.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b79df3700 1 -- start start 2022-07-21T01:42:36.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b79df3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9b740b2710 0x7f9b7414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b79df3700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9b740a8c50 0x7f9b7414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b79df3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b740a7250 con 0x7f9b740b01a0 2022-07-21T01:42:36.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b79df3700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b740a7480 con 0x7f9b740a8c50 2022-07-21T01:42:36.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b79df3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b740a7720 con 0x7f9b740b2710 2022-07-21T01:42:36.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b795f2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9b740a8c50 0x7f9b7414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b795f2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9b740a8c50 0x7f9b7414b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51448/0 (socket says 172.21.15.3:51448) 2022-07-21T01:42:36.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b795f2700 1 -- 172.21.15.3:0/1711488291 learned_addr learned my addr 172.21.15.3:0/1711488291 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:36.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b78df1700 1 --2- 172.21.15.3:0/1711488291 >> v2:172.21.15.143:3300/0 conn(0x7f9b740b2710 0x7f9b7414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b795f2700 1 -- 172.21.15.3:0/1711488291 >> v2:172.21.15.143:3300/0 conn(0x7f9b740b2710 msgr2=0x7f9b7414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:36.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b795f2700 1 --2- 172.21.15.3:0/1711488291 >> v2:172.21.15.143:3300/0 conn(0x7f9b740b2710 0x7f9b7414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/1711488291 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482531873 0 0) 0x7f9b740a7250 con 0x7f9b740b01a0 2022-07-21T01:42:36.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b795f2700 1 -- 172.21.15.3:0/1711488291 >> v1:172.21.15.143:6789/0 conn(0x7f9b740b01a0 legacy=0x7f9b740a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b795f2700 1 -- 172.21.15.3:0/1711488291 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b740a79c0 con 0x7f9b740a8c50 2022-07-21T01:42:36.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b795f2700 1 --2- 172.21.15.3:0/1711488291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 0x7f9b7414b4c0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f9b6c0032f0 tx=0x7f9b6c009290).ready entity=mon.0 client_cookie=f3613c0c9bd98ce5 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:36.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/1711488291 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b6c00b870 con 0x7f9b740a8c50 2022-07-21T01:42:36.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/1711488291 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c00b9d0 con 0x7f9b740a8c50 2022-07-21T01:42:36.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/1711488291 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b6c00bb40 con 0x7f9b740a8c50 2022-07-21T01:42:36.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b79df3700 1 -- 172.21.15.3:0/1711488291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 msgr2=0x7f9b7414b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.675+0000 7f9b79df3700 1 --2- 172.21.15.3:0/1711488291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 0x7f9b7414b4c0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f9b6c0032f0 tx=0x7f9b6c009290).stop 2022-07-21T01:42:36.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 -- 172.21.15.3:0/1711488291 shutdown_connections 2022-07-21T01:42:36.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 --2- 172.21.15.3:0/1711488291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 0x7f9b7414b4c0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 --2- 172.21.15.3:0/1711488291 >> v2:172.21.15.143:3300/0 conn(0x7f9b740b2710 0x7f9b7414afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 -- 172.21.15.3:0/1711488291 >> 172.21.15.3:0/1711488291 conn(0x7f9b74060310 msgr2=0x7f9b740b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:36.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 -- 172.21.15.3:0/1711488291 shutdown_connections 2022-07-21T01:42:36.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 -- 172.21.15.3:0/1711488291 wait complete. 2022-07-21T01:42:36.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 Processor -- start 2022-07-21T01:42:36.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 -- start start 2022-07-21T01:42:36.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 0x7f9b740ab510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b740b01a0 0x7f9b740b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b740b2710 0x7f9b740aa750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b740abfb0 con 0x7f9b740a8c50 2022-07-21T01:42:36.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b74157200 con 0x7f9b740b01a0 2022-07-21T01:42:36.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.679+0000 7f9b79df3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b740ac1e0 con 0x7f9b740b2710 2022-07-21T01:42:36.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b78df1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 0x7f9b740ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b78df1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 0x7f9b740ab510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51450/0 (socket says 172.21.15.3:51450) 2022-07-21T01:42:36.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b78df1700 1 -- 172.21.15.3:0/2239548731 learned_addr learned my addr 172.21.15.3:0/2239548731 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:36.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b6bfff700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b740b01a0 0x7f9b740b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b795f2700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b740b2710 0x7f9b740aa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b78df1700 1 -- 172.21.15.3:0/2239548731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b740b2710 msgr2=0x7f9b740aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b78df1700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b740b2710 0x7f9b740aa750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b78df1700 1 -- 172.21.15.3:0/2239548731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b740b01a0 msgr2=0x7f9b740b4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b78df1700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b740b01a0 0x7f9b740b4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b78df1700 1 -- 172.21.15.3:0/2239548731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c009530 con 0x7f9b740a8c50 2022-07-21T01:42:36.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b795f2700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b740b2710 0x7f9b740aa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:36.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b78df1700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 0x7f9b740ab510 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7f9b7000c000 tx=0x7f9b70006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:36.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b69ffb700 1 -- 172.21.15.3:0/2239548731 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b70010780 con 0x7f9b740a8c50 2022-07-21T01:42:36.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b69ffb700 1 -- 172.21.15.3:0/2239548731 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b700108e0 con 0x7f9b740a8c50 2022-07-21T01:42:36.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b740ac480 con 0x7f9b740a8c50 2022-07-21T01:42:36.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b69ffb700 1 -- 172.21.15.3:0/2239548731 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b70010c40 con 0x7f9b740a8c50 2022-07-21T01:42:36.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b740aac40 con 0x7f9b740a8c50 2022-07-21T01:42:36.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b740574e0 con 0x7f9b740a8c50 2022-07-21T01:42:36.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.683+0000 7f9b69ffb700 1 -- 172.21.15.3:0/2239548731 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9b7001c460 con 0x7f9b740a8c50 2022-07-21T01:42:36.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.687+0000 7f9b69ffb700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b54053aa0 0x7f9b54055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.687+0000 7f9b69ffb700 1 -- 172.21.15.3:0/2239548731 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9b7006c370 con 0x7f9b740a8c50 2022-07-21T01:42:36.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.687+0000 7f9b6bfff700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b54053aa0 0x7f9b54055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.687+0000 7f9b6bfff700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b54053aa0 0x7f9b54055f60 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f9b6400d000 tx=0x7f9b64007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:36.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.691+0000 7f9b69ffb700 1 -- 172.21.15.3:0/2239548731 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9b700430d0 con 0x7f9b740a8c50 2022-07-21T01:42:36.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.859+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9b740ab270 con 0x7f9b740a8c50 2022-07-21T01:42:36.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.867+0000 7f9b69ffb700 1 -- 172.21.15.3:0/2239548731 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v37) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9b700466f0 con 0x7f9b740a8c50 2022-07-21T01:42:36.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b54053aa0 msgr2=0x7f9b54055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b54053aa0 0x7f9b54055f60 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f9b6400d000 tx=0x7f9b64007a00).stop 2022-07-21T01:42:36.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 msgr2=0x7f9b740ab510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 0x7f9b740ab510 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7f9b7000c000 tx=0x7f9b70006df0).stop 2022-07-21T01:42:36.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 shutdown_connections 2022-07-21T01:42:36.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b740b2710 0x7f9b740aa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b54053aa0 0x7f9b54055f60 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b740b01a0 0x7f9b740b4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 --2- 172.21.15.3:0/2239548731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b740a8c50 0x7f9b740ab510 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:36.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 >> 172.21.15.3:0/2239548731 conn(0x7f9b74060310 msgr2=0x7f9b74148850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:36.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 shutdown_connections 2022-07-21T01:42:36.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.871+0000 7f9b79df3700 1 -- 172.21.15.3:0/2239548731 wait complete. 2022-07-21T01:42:36.881 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:42:36.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:36.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34433bd700 1 Processor -- start 2022-07-21T01:42:36.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34433bd700 1 -- start start 2022-07-21T01:42:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34433bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c0614b0 0x7f343c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34433bd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c0aee80 con 0x7f343c1405d0 2022-07-21T01:42:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34433bd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c0b70b0 con 0x7f343c0ad750 2022-07-21T01:42:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34433bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0af0b0 con 0x7f343c0614b0 2022-07-21T01:42:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f3442bbc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f343c1405d0 0x7f343c1466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59742/0 (socket says 172.21.15.3:59742) 2022-07-21T01:42:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f3442bbc700 1 -- 172.21.15.3:0/3033884049 learned_addr learned my addr 172.21.15.3:0/3033884049 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34423bb700 1 --2- 172.21.15.3:0/3033884049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c0614b0 0x7f343c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34413b9700 1 -- 172.21.15.3:0/3033884049 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633046518 0 0) 0x7f343c0aee80 con 0x7f343c1405d0 2022-07-21T01:42:36.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34413b9700 1 -- 172.21.15.3:0/3033884049 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3424003260 con 0x7f343c1405d0 2022-07-21T01:42:36.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34413b9700 1 -- 172.21.15.3:0/3033884049 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872083865 0 0) 0x7f343c0b70b0 con 0x7f343c0ad750 2022-07-21T01:42:36.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34413b9700 1 -- 172.21.15.3:0/3033884049 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f343c0aee80 con 0x7f343c0ad750 2022-07-21T01:42:36.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34423bb700 1 -- 172.21.15.3:0/3033884049 >> v1:172.21.15.143:6789/0 conn(0x7f343c0ad750 legacy=0x7f343c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34413b9700 1 -- 172.21.15.3:0/3033884049 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3699482974 0 0) 0x7f3424003260 con 0x7f343c1405d0 2022-07-21T01:42:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34423bb700 1 -- 172.21.15.3:0/3033884049 >> v1:172.21.15.3:6789/0 conn(0x7f343c1405d0 legacy=0x7f343c1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34423bb700 1 -- 172.21.15.3:0/3033884049 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0af350 con 0x7f343c0614b0 2022-07-21T01:42:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.991+0000 7f34423bb700 1 --2- 172.21.15.3:0/3033884049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c0614b0 0x7f343c061890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f3434001110 tx=0x7f342c006df0).ready entity=mon.2 client_cookie=d0f2b4319860d1a0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34413b9700 1 -- 172.21.15.3:0/3033884049 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3434001660 con 0x7f343c0614b0 2022-07-21T01:42:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34413b9700 1 -- 172.21.15.3:0/3033884049 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c000e90 con 0x7f343c0614b0 2022-07-21T01:42:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34413b9700 1 -- 172.21.15.3:0/3033884049 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f342c012460 con 0x7f343c0614b0 2022-07-21T01:42:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 -- 172.21.15.3:0/3033884049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c0614b0 msgr2=0x7f343c061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 --2- 172.21.15.3:0/3033884049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c0614b0 0x7f343c061890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f3434001110 tx=0x7f342c006df0).stop 2022-07-21T01:42:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 -- 172.21.15.3:0/3033884049 shutdown_connections 2022-07-21T01:42:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 --2- 172.21.15.3:0/3033884049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c0614b0 0x7f343c061890 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 -- 172.21.15.3:0/3033884049 >> 172.21.15.3:0/3033884049 conn(0x7f343c066f70 msgr2=0x7f343c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 -- 172.21.15.3:0/3033884049 shutdown_connections 2022-07-21T01:42:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 -- 172.21.15.3:0/3033884049 wait complete. 2022-07-21T01:42:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 Processor -- start 2022-07-21T01:42:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 -- start start 2022-07-21T01:42:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f343c0614b0 0x7f343c0b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0ad750 0x7f343c1425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c1405d0 0x7f343c0b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0ac5f0 con 0x7f343c0614b0 2022-07-21T01:42:37.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0acac0 con 0x7f343c0ad750 2022-07-21T01:42:37.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34433bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0ac820 con 0x7f343c1405d0 2022-07-21T01:42:37.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34423bb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f343c0614b0 0x7f343c0b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34423bb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f343c0614b0 0x7f343c0b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51464/0 (socket says 172.21.15.3:51464) 2022-07-21T01:42:37.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f34423bb700 1 -- 172.21.15.3:0/2052639852 learned_addr learned my addr 172.21.15.3:0/2052639852 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:37.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f3442bbc700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c1405d0 0x7f343c0b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f3441bba700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0ad750 0x7f343c1425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f3441bba700 1 -- 172.21.15.3:0/2052639852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c1405d0 msgr2=0x7f343c0b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f3441bba700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c1405d0 0x7f343c0b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f3441bba700 1 -- 172.21.15.3:0/2052639852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f343c0614b0 msgr2=0x7f343c0b0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.995+0000 7f3441bba700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f343c0614b0 0x7f343c0b0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f3441bba700 1 -- 172.21.15.3:0/2052639852 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c00b180 con 0x7f343c0ad750 2022-07-21T01:42:37.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f3441bba700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0ad750 0x7f343c1425d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f343c0b2b50 tx=0x7f3438007da0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:37.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f342b7fe700 1 -- 172.21.15.3:0/2052639852 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3438002310 con 0x7f343c0ad750 2022-07-21T01:42:37.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f342b7fe700 1 -- 172.21.15.3:0/2052639852 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3438002470 con 0x7f343c0ad750 2022-07-21T01:42:37.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c0abbe0 con 0x7f343c0ad750 2022-07-21T01:42:37.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f342b7fe700 1 -- 172.21.15.3:0/2052639852 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3438001840 con 0x7f343c0ad750 2022-07-21T01:42:37.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c0b3dd0 con 0x7f343c0ad750 2022-07-21T01:42:37.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c0af5f0 con 0x7f343c0ad750 2022-07-21T01:42:37.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f342b7fe700 1 -- 172.21.15.3:0/2052639852 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3438010030 con 0x7f343c0ad750 2022-07-21T01:42:37.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f342b7fe700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f341c053b00 0x7f341c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:36.999+0000 7f342b7fe700 1 -- 172.21.15.3:0/2052639852 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f343806c730 con 0x7f343c0ad750 2022-07-21T01:42:37.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.003+0000 7f34423bb700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f341c053b00 0x7f341c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.003+0000 7f34423bb700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f341c053b00 0x7f341c055fc0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f342c01a000 tx=0x7f342c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:37.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.007+0000 7f342b7fe700 1 -- 172.21.15.3:0/2052639852 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f34380434f0 con 0x7f343c0ad750 2022-07-21T01:42:37.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.171+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f343c0574e0 con 0x7f343c0ad750 2022-07-21T01:42:37.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.171+0000 7f342b7fe700 1 -- 172.21.15.3:0/2052639852 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2492 (secure 0 0 0) 0x7f3438046b10 con 0x7f343c0ad750 2022-07-21T01:42:37.176 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:37.176 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/1563084587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1563084587},{"type":"v1","addr":"172.21.15.3:6835","nonce":1563084587}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1403026914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1403026914},{"type":"v1","addr":"172.21.15.3:6837","nonce":1403026914}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:34.642400+0000","modified":"2022-07-21T01:42:35.646272+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8409":{"gid":8409,"name":"b","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/1158093851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1158093851},{"type":"v1","addr":"172.21.15.143:6833","nonce":1158093851}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-21T01:42:37.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f341c053b00 msgr2=0x7f341c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f341c053b00 0x7f341c055fc0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f342c01a000 tx=0x7f342c014000).stop 2022-07-21T01:42:37.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0ad750 msgr2=0x7f343c1425d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0ad750 0x7f343c1425d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f343c0b2b50 tx=0x7f3438007da0).stop 2022-07-21T01:42:37.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 shutdown_connections 2022-07-21T01:42:37.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c1405d0 0x7f343c0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f341c053b00 0x7f341c055fc0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0ad750 0x7f343c1425d0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 --2- 172.21.15.3:0/2052639852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f343c0614b0 0x7f343c0b0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 >> 172.21.15.3:0/2052639852 conn(0x7f343c066f70 msgr2=0x7f343c13fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:37.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 shutdown_connections 2022-07-21T01:42:37.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.175+0000 7f34433bd700 1 -- 172.21.15.3:0/2052639852 wait complete. 2022-07-21T01:42:37.181 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 76 2022-07-21T01:42:37.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:34.642400+0000', 'modified': '2022-07-21T01:42:35.646272+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8409}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8409': {'gid': 8409, 'name': 'b', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6833/1158093851', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 1158093851}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 1158093851}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:42:37.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:42:37.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-21T01:42:37.611 INFO:teuthology.orchestra.run.smithi143.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8409,"want_state":"up:active","state":"up:active","rank_uptime":2.9572537539999999,"mdsmap_epoch":76,"osdmap_epoch":150,"osdmap_epoch_barrier":150,"uptime":7.2895024289999997} 2022-07-21T01:42:37.612 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8409, "mdsmap_epoch": 76, "osdmap_epoch": 150, "osdmap_epoch_barrier": 150, "rank_uptime": 2.957253754, "state": "up:active", "uptime": 7.289502429, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:42:37.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:37.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 Processor -- start 2022-07-21T01:42:37.725 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 -- start start 2022-07-21T01:42:37.727 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2530148530 0x7f25300b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 0x7f25301448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25300b04e0 con 0x7f25300b1f40 2022-07-21T01:42:37.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25300a9250 con 0x7f2530148530 2022-07-21T01:42:37.728 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25300b0780 con 0x7f25300a7250 2022-07-21T01:42:37.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2530148530 0x7f25300b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2530148530 0x7f25300b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51474/0 (socket says 172.21.15.3:51474) 2022-07-21T01:42:37.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252effd700 1 -- 172.21.15.3:0/191879394 learned_addr learned my addr 172.21.15.3:0/191879394 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:37.729 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252ffff700 1 --2- 172.21.15.3:0/191879394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 0x7f25301448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252effd700 1 -- 172.21.15.3:0/191879394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 msgr2=0x7f25301448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:37.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252effd700 1 --2- 172.21.15.3:0/191879394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 0x7f25301448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252effd700 1 -- 172.21.15.3:0/191879394 >> v1:172.21.15.143:6789/0 conn(0x7f25300b1f40 legacy=0x7f25300b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252effd700 1 -- 172.21.15.3:0/191879394 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25300b0a20 con 0x7f2530148530 2022-07-21T01:42:37.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252ffff700 1 --2- 172.21.15.3:0/191879394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 0x7f25301448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:37.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252effd700 1 --2- 172.21.15.3:0/191879394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2530148530 0x7f25300b1160 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7f25301478c0 tx=0x7f2528009290).ready entity=mon.0 client_cookie=f208322f281058fc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:37.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252e7fc700 1 -- 172.21.15.3:0/191879394 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f252800b860 con 0x7f2530148530 2022-07-21T01:42:37.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252e7fc700 1 -- 172.21.15.3:0/191879394 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252800b9c0 con 0x7f2530148530 2022-07-21T01:42:37.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f252e7fc700 1 -- 172.21.15.3:0/191879394 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f252800bb30 con 0x7f2530148530 2022-07-21T01:42:37.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 -- 172.21.15.3:0/191879394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2530148530 msgr2=0x7f25300b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 --2- 172.21.15.3:0/191879394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2530148530 0x7f25300b1160 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7f25301478c0 tx=0x7f2528009290).stop 2022-07-21T01:42:37.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 -- 172.21.15.3:0/191879394 shutdown_connections 2022-07-21T01:42:37.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 --2- 172.21.15.3:0/191879394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 0x7f25301448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 --2- 172.21.15.3:0/191879394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2530148530 0x7f25300b1160 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 -- 172.21.15.3:0/191879394 >> 172.21.15.3:0/191879394 conn(0x7f253005f840 msgr2=0x7f253005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:37.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 -- 172.21.15.3:0/191879394 shutdown_connections 2022-07-21T01:42:37.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.723+0000 7f253605e700 1 -- 172.21.15.3:0/191879394 wait complete. 2022-07-21T01:42:37.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f253605e700 1 Processor -- start 2022-07-21T01:42:37.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f253605e700 1 -- start start 2022-07-21T01:42:37.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f253605e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 0x7f2530149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f253605e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25300b1f40 0x7f2530143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f253605e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2530148530 0x7f2530144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f253605e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25301444f0 con 0x7f25300b1f40 2022-07-21T01:42:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f253605e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253014b0b0 con 0x7f2530148530 2022-07-21T01:42:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f253605e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253014b350 con 0x7f25300a7250 2022-07-21T01:42:37.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25300b1f40 0x7f2530143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25300b1f40 0x7f2530143b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51480/0 (socket says 172.21.15.3:51480) 2022-07-21T01:42:37.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252effd700 1 -- 172.21.15.3:0/1701506684 learned_addr learned my addr 172.21.15.3:0/1701506684 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:37.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252f7fe700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 0x7f2530149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252ffff700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2530148530 0x7f2530144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252effd700 1 -- 172.21.15.3:0/1701506684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 msgr2=0x7f2530149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252effd700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 0x7f2530149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252effd700 1 -- 172.21.15.3:0/1701506684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2530148530 msgr2=0x7f2530144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252effd700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2530148530 0x7f2530144000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252effd700 1 -- 172.21.15.3:0/1701506684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2528009530 con 0x7f25300b1f40 2022-07-21T01:42:37.740 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.727+0000 7f252effd700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25300b1f40 0x7f2530143b10 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f25301478c0 tx=0x7f2528016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:37.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.731+0000 7f252cff9700 1 -- 172.21.15.3:0/1701506684 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f252800b860 con 0x7f25300b1f40 2022-07-21T01:42:37.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f252cff9700 1 -- 172.21.15.3:0/1701506684 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25280032d0 con 0x7f25300b1f40 2022-07-21T01:42:37.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253014b5f0 con 0x7f25300b1f40 2022-07-21T01:42:37.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f252cff9700 1 -- 172.21.15.3:0/1701506684 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2528020430 con 0x7f25300b1f40 2022-07-21T01:42:37.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25300ab6e0 con 0x7f25300b1f40 2022-07-21T01:42:37.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25300574e0 con 0x7f25300b1f40 2022-07-21T01:42:37.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f252cff9700 1 -- 172.21.15.3:0/1701506684 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2528026020 con 0x7f25300b1f40 2022-07-21T01:42:37.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f252cff9700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2508053a70 0x7f2508055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:37.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f252cff9700 1 -- 172.21.15.3:0/1701506684 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f252806cea0 con 0x7f25300b1f40 2022-07-21T01:42:37.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f252f7fe700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2508053a70 0x7f2508055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:37.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.735+0000 7f252f7fe700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2508053a70 0x7f2508055f30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f251800d000 tx=0x7f2518007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:37.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.739+0000 7f252cff9700 1 -- 172.21.15.3:0/1701506684 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2528043bc0 con 0x7f25300b1f40 2022-07-21T01:42:37.869 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.867+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25300447e0 con 0x7f25300b1f40 2022-07-21T01:42:37.871 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.867+0000 7f252cff9700 1 -- 172.21.15.3:0/1701506684 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f2528015020 con 0x7f25300b1f40 2022-07-21T01:42:37.872 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:37.872 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":150,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:35.645281+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:33.092672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:34.046506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2508053a70 msgr2=0x7f2508055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2508053a70 0x7f2508055f30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f251800d000 tx=0x7f2518007b90).stop 2022-07-21T01:42:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25300b1f40 msgr2=0x7f2530143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25300b1f40 0x7f2530143b10 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f25301478c0 tx=0x7f2528016000).stop 2022-07-21T01:42:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 shutdown_connections 2022-07-21T01:42:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2508053a70 0x7f2508055f30 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2530148530 0x7f2530144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25300b1f40 0x7f2530143b10 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 --2- 172.21.15.3:0/1701506684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25300a7250 0x7f2530149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:37.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 >> 172.21.15.3:0/1701506684 conn(0x7f253005f840 msgr2=0x7f2530144bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:37.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 shutdown_connections 2022-07-21T01:42:37.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.871+0000 7f253605e700 1 -- 172.21.15.3:0/1701506684 wait complete. 2022-07-21T01:42:37.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:42:33.092672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:42:34.046506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:42:37.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:42:38.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8999ba700 1 Processor -- start 2022-07-21T01:42:38.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8999ba700 1 -- start start 2022-07-21T01:42:38.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8999ba700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe8940b2340 0x7fe89405f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8999ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 0x7fe89412c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8999ba700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8940b3dd0 con 0x7fe894060eb0 2022-07-21T01:42:38.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8999ba700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe894063f80 con 0x7fe8940b2340 2022-07-21T01:42:38.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8999ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940b2720 con 0x7fe894143e20 2022-07-21T01:42:38.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8989b8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe8940b2340 0x7fe89405f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8989b8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe8940b2340 0x7fe89405f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51486/0 (socket says 172.21.15.3:51486) 2022-07-21T01:42:38.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8989b8700 1 -- 172.21.15.3:0/3010782184 learned_addr learned my addr 172.21.15.3:0/3010782184 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:38.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe893fff700 1 --1- 172.21.15.3:0/3010782184 >> v1:172.21.15.3:6789/0 conn(0x7fe894060eb0 0x7fe8940b36c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59768/0 (socket says 172.21.15.3:59768) 2022-07-21T01:42:38.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8991b9700 1 --2- 172.21.15.3:0/3010782184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 0x7fe89412c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8989b8700 1 -- 172.21.15.3:0/3010782184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 msgr2=0x7fe89412c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:38.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8989b8700 1 --2- 172.21.15.3:0/3010782184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 0x7fe89412c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8989b8700 1 -- 172.21.15.3:0/3010782184 >> v1:172.21.15.3:6789/0 conn(0x7fe894060eb0 legacy=0x7fe8940b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8989b8700 1 -- 172.21.15.3:0/3010782184 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8940b2fc0 con 0x7fe8940b2340 2022-07-21T01:42:38.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8937fe700 1 -- 172.21.15.3:0/3010782184 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694649786 0 0) 0x7fe8940b3dd0 con 0x7fe894060eb0 2022-07-21T01:42:38.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8989b8700 1 --2- 172.21.15.3:0/3010782184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 0x7fe89405f840 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7fe888002f70 tx=0x7fe888006eb0).ready entity=mon.0 client_cookie=2185bb691f92a0e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8937fe700 1 -- 172.21.15.3:0/3010782184 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe88800d5c0 con 0x7fe8940b2340 2022-07-21T01:42:38.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8937fe700 1 -- 172.21.15.3:0/3010782184 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe888000e90 con 0x7fe8940b2340 2022-07-21T01:42:38.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:37.999+0000 7fe8937fe700 1 -- 172.21.15.3:0/3010782184 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe88800d910 con 0x7fe8940b2340 2022-07-21T01:42:38.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 -- 172.21.15.3:0/3010782184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 msgr2=0x7fe89405f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 --2- 172.21.15.3:0/3010782184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 0x7fe89405f840 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7fe888002f70 tx=0x7fe888006eb0).stop 2022-07-21T01:42:38.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 -- 172.21.15.3:0/3010782184 shutdown_connections 2022-07-21T01:42:38.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 --2- 172.21.15.3:0/3010782184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 0x7fe89412c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 --2- 172.21.15.3:0/3010782184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 0x7fe89405f840 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 -- 172.21.15.3:0/3010782184 >> 172.21.15.3:0/3010782184 conn(0x7fe89405c620 msgr2=0x7fe89405ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:38.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 -- 172.21.15.3:0/3010782184 shutdown_connections 2022-07-21T01:42:38.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 -- 172.21.15.3:0/3010782184 wait complete. 2022-07-21T01:42:38.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 Processor -- start 2022-07-21T01:42:38.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.003+0000 7fe8999ba700 1 -- start start 2022-07-21T01:42:38.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8999ba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894060eb0 0x7fe8940b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8999ba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 0x7fe894128840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8999ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 0x7fe8941459e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8999ba700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8941509b0 con 0x7fe8940b2340 2022-07-21T01:42:38.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8999ba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe894145ed0 con 0x7fe894060eb0 2022-07-21T01:42:38.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8999ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe894146150 con 0x7fe894143e20 2022-07-21T01:42:38.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe893fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 0x7fe894128840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe893fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 0x7fe894128840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51494/0 (socket says 172.21.15.3:51494) 2022-07-21T01:42:38.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe893fff700 1 -- 172.21.15.3:0/1908527289 learned_addr learned my addr 172.21.15.3:0/1908527289 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:38.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8991b9700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 0x7fe8941459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8989b8700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894060eb0 0x7fe8940b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe893fff700 1 -- 172.21.15.3:0/1908527289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 msgr2=0x7fe8941459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe893fff700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 0x7fe8941459e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe893fff700 1 -- 172.21.15.3:0/1908527289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894060eb0 msgr2=0x7fe8940b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe893fff700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894060eb0 0x7fe8940b0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe893fff700 1 -- 172.21.15.3:0/1908527289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88800b1e0 con 0x7fe8940b2340 2022-07-21T01:42:38.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe893fff700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 0x7fe894128840 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7fe880000ed0 tx=0x7fe880007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.035 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe891ffb700 1 -- 172.21.15.3:0/1908527289 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8800070d0 con 0x7fe8940b2340 2022-07-21T01:42:38.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe891ffb700 1 -- 172.21.15.3:0/1908527289 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe880007230 con 0x7fe8940b2340 2022-07-21T01:42:38.036 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8941563a0 con 0x7fe8940b2340 2022-07-21T01:42:38.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe891ffb700 1 -- 172.21.15.3:0/1908527289 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe880005740 con 0x7fe8940b2340 2022-07-21T01:42:38.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe894129c60 con 0x7fe8940b2340 2022-07-21T01:42:38.037 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe891ffb700 1 -- 172.21.15.3:0/1908527289 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe88001a060 con 0x7fe8940b2340 2022-07-21T01:42:38.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.007+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe894146350 con 0x7fe8940b2340 2022-07-21T01:42:38.038 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.011+0000 7fe891ffb700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe874053a70 0x7fe874055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.011+0000 7fe891ffb700 1 -- 172.21.15.3:0/1908527289 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe88006b310 con 0x7fe8940b2340 2022-07-21T01:42:38.039 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.011+0000 7fe8989b8700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe874053a70 0x7fe874055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.011+0000 7fe8989b8700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe874053a70 0x7fe874055f30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fe88801c000 tx=0x7fe888015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.040 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.015+0000 7fe891ffb700 1 -- 172.21.15.3:0/1908527289 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe8800420d0 con 0x7fe8940b2340 2022-07-21T01:42:38.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.143+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe894061d60 con 0x7fe8940b2340 2022-07-21T01:42:38.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.143+0000 7fe891ffb700 1 -- 172.21.15.3:0/1908527289 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v150) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe880018020 con 0x7fe8940b2340 2022-07-21T01:42:38.147 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:42:38.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe874053a70 msgr2=0x7fe874055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe874053a70 0x7fe874055f30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fe88801c000 tx=0x7fe888015000).stop 2022-07-21T01:42:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 msgr2=0x7fe894128840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 0x7fe894128840 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7fe880000ed0 tx=0x7fe880007ac0).stop 2022-07-21T01:42:38.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 shutdown_connections 2022-07-21T01:42:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe894143e20 0x7fe8941459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8940b2340 0x7fe894128840 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe874053a70 0x7fe874055f30 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 --2- 172.21.15.3:0/1908527289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894060eb0 0x7fe8940b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 >> 172.21.15.3:0/1908527289 conn(0x7fe89405c620 msgr2=0x7fe8941292c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 shutdown_connections 2022-07-21T01:42:38.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.147+0000 7fe8999ba700 1 -- 172.21.15.3:0/1908527289 wait complete. 2022-07-21T01:42:38.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:42:38.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.271+0000 7fc687c47700 1 Processor -- start 2022-07-21T01:42:38.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.271+0000 7fc687c47700 1 -- start start 2022-07-21T01:42:38.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc687c47700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc6800b4820 0x7fc680140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc687c47700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc680143cf0 con 0x7fc680146ae0 2022-07-21T01:42:38.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc687c47700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc680143f90 con 0x7fc6800b4310 2022-07-21T01:42:38.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc687c47700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6801476a0 con 0x7fc6800b4820 2022-07-21T01:42:38.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6859e3700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc680146ae0 0x7fc6801403c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59782/0 (socket says 172.21.15.3:59782) 2022-07-21T01:42:38.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6859e3700 1 -- 172.21.15.3:0/1825982341 learned_addr learned my addr 172.21.15.3:0/1825982341 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:38.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6861e4700 1 --2- 172.21.15.3:0/1825982341 >> v2:172.21.15.143:3300/0 conn(0x7fc6800b4820 0x7fc680140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6861e4700 1 -- 172.21.15.3:0/1825982341 >> v1:172.21.15.143:6789/0 conn(0x7fc6800b4310 legacy=0x7fc6800b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6861e4700 1 -- 172.21.15.3:0/1825982341 >> v1:172.21.15.3:6789/0 conn(0x7fc680146ae0 legacy=0x7fc6801403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6861e4700 1 -- 172.21.15.3:0/1825982341 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6800b0e20 con 0x7fc6800b4820 2022-07-21T01:42:38.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6849e1700 1 -- 172.21.15.3:0/1825982341 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460659280 0 0) 0x7fc680143cf0 con 0x7fc680146ae0 2022-07-21T01:42:38.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6861e4700 1 --2- 172.21.15.3:0/1825982341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6800b4820 0x7fc680140ad0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fc674001110 tx=0x7fc6780091d0).ready entity=mon.1 client_cookie=74145ebcfa5fd8f2 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6849e1700 1 -- 172.21.15.3:0/1825982341 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc67800b600 con 0x7fc6800b4820 2022-07-21T01:42:38.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6849e1700 1 -- 172.21.15.3:0/1825982341 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67800b760 con 0x7fc6800b4820 2022-07-21T01:42:38.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.275+0000 7fc6849e1700 1 -- 172.21.15.3:0/1825982341 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc67800b8d0 con 0x7fc6800b4820 2022-07-21T01:42:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 -- 172.21.15.3:0/1825982341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6800b4820 msgr2=0x7fc680140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 --2- 172.21.15.3:0/1825982341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6800b4820 0x7fc680140ad0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fc674001110 tx=0x7fc6780091d0).stop 2022-07-21T01:42:38.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 -- 172.21.15.3:0/1825982341 shutdown_connections 2022-07-21T01:42:38.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 --2- 172.21.15.3:0/1825982341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6800b4820 0x7fc680140ad0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 -- 172.21.15.3:0/1825982341 >> 172.21.15.3:0/1825982341 conn(0x7fc68005c620 msgr2=0x7fc68005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:38.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 -- 172.21.15.3:0/1825982341 shutdown_connections 2022-07-21T01:42:38.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 -- 172.21.15.3:0/1825982341 wait complete. 2022-07-21T01:42:38.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 Processor -- start 2022-07-21T01:42:38.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 -- start start 2022-07-21T01:42:38.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6800b4310 0x7fc6800b6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6800b4820 0x7fc68013a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc680146ae0 0x7fc68013a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800b8110 con 0x7fc6800b4310 2022-07-21T01:42:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800b83b0 con 0x7fc6800b4820 2022-07-21T01:42:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc687c47700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800b8650 con 0x7fc680146ae0 2022-07-21T01:42:38.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.279+0000 7fc6859e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6800b4310 0x7fc6800b6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6859e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6800b4310 0x7fc6800b6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51506/0 (socket says 172.21.15.3:51506) 2022-07-21T01:42:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6859e3700 1 -- 172.21.15.3:0/1392868361 learned_addr learned my addr 172.21.15.3:0/1392868361 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6861e4700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc680146ae0 0x7fc68013a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6851e2700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6800b4820 0x7fc68013a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6859e3700 1 -- 172.21.15.3:0/1392868361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc680146ae0 msgr2=0x7fc68013a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6859e3700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc680146ae0 0x7fc68013a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6859e3700 1 -- 172.21.15.3:0/1392868361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6800b4820 msgr2=0x7fc68013a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6859e3700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6800b4820 0x7fc68013a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6859e3700 1 -- 172.21.15.3:0/1392868361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc678009470 con 0x7fc6800b4310 2022-07-21T01:42:38.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc6859e3700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6800b4310 0x7fc6800b6840 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fc67c000bc0 tx=0x7fc67c007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc672ffd700 1 -- 172.21.15.3:0/1392868361 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc67c0070d0 con 0x7fc6800b4310 2022-07-21T01:42:38.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc672ffd700 1 -- 172.21.15.3:0/1392868361 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67c007230 con 0x7fc6800b4310 2022-07-21T01:42:38.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6800b88f0 con 0x7fc6800b4310 2022-07-21T01:42:38.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc672ffd700 1 -- 172.21.15.3:0/1392868361 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc67c005610 con 0x7fc6800b4310 2022-07-21T01:42:38.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68013ab10 con 0x7fc6800b4310 2022-07-21T01:42:38.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.283+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6800b8a80 con 0x7fc6800b4310 2022-07-21T01:42:38.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.287+0000 7fc672ffd700 1 -- 172.21.15.3:0/1392868361 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc67c01a060 con 0x7fc6800b4310 2022-07-21T01:42:38.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.287+0000 7fc672ffd700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc660053aa0 0x7fc660055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.287+0000 7fc672ffd700 1 -- 172.21.15.3:0/1392868361 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc67c06b360 con 0x7fc6800b4310 2022-07-21T01:42:38.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.287+0000 7fc6851e2700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc660053aa0 0x7fc660055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.287+0000 7fc6851e2700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc660053aa0 0x7fc660055f60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fc67400d000 tx=0x7fc674007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.295+0000 7fc672ffd700 1 -- 172.21.15.3:0/1392868361 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc67c042120 con 0x7fc6800b4310 2022-07-21T01:42:38.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc6800574e0 con 0x7fc6800b4310 2022-07-21T01:42:38.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc672ffd700 1 -- 172.21.15.3:0/1392868361 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v150) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc67c018020 con 0x7fc6800b4310 2022-07-21T01:42:38.426 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:38.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc660053aa0 msgr2=0x7fc660055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc660053aa0 0x7fc660055f60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fc67400d000 tx=0x7fc674007b70).stop 2022-07-21T01:42:38.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6800b4310 msgr2=0x7fc6800b6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6800b4310 0x7fc6800b6840 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fc67c000bc0 tx=0x7fc67c007b20).stop 2022-07-21T01:42:38.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 shutdown_connections 2022-07-21T01:42:38.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc680146ae0 0x7fc68013a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc660053aa0 0x7fc660055f60 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6800b4820 0x7fc68013a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 --2- 172.21.15.3:0/1392868361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6800b4310 0x7fc6800b6840 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 >> 172.21.15.3:0/1392868361 conn(0x7fc68005c620 msgr2=0x7fc68013b350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:38.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 shutdown_connections 2022-07-21T01:42:38.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.423+0000 7fc687c47700 1 -- 172.21.15.3:0/1392868361 wait complete. 2022-07-21T01:42:38.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:42:38.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad3fff700 1 Processor -- start 2022-07-21T01:42:38.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad3fff700 1 -- start start 2022-07-21T01:42:38.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40af480 0x7faad40abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faad40b22a0 0x7faad40ac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad3fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad4140540 con 0x7faad40b1a80 2022-07-21T01:42:38.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad3fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad40ae590 con 0x7faad40b22a0 2022-07-21T01:42:38.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad3fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad40aff00 con 0x7faad40af480 2022-07-21T01:42:38.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad2ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40af480 0x7faad40abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad2ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40af480 0x7faad40abb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:58430/0 (socket says 172.21.15.3:58430) 2022-07-21T01:42:38.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad27fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faad40b22a0 0x7faad40ac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad2ffd700 1 -- 172.21.15.3:0/2867676402 learned_addr learned my addr 172.21.15.3:0/2867676402 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:38.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad1ffb700 1 -- 172.21.15.3:0/2867676402 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271232102 0 0) 0x7faad4140540 con 0x7faad40b1a80 2022-07-21T01:42:38.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad1ffb700 1 -- 172.21.15.3:0/2867676402 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab8003260 con 0x7faad40b1a80 2022-07-21T01:42:38.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad2ffd700 1 -- 172.21.15.3:0/2867676402 >> v2:172.21.15.143:3300/0 conn(0x7faad40b22a0 msgr2=0x7faad40ac000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:38.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad2ffd700 1 --2- 172.21.15.3:0/2867676402 >> v2:172.21.15.143:3300/0 conn(0x7faad40b22a0 0x7faad40ac000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad2ffd700 1 -- 172.21.15.3:0/2867676402 >> v1:172.21.15.143:6789/0 conn(0x7faad40b1a80 legacy=0x7faad41391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad2ffd700 1 -- 172.21.15.3:0/2867676402 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad40b01a0 con 0x7faad40af480 2022-07-21T01:42:38.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad2ffd700 1 --2- 172.21.15.3:0/2867676402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40af480 0x7faad40abb10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7faac4002f70 tx=0x7faac4006eb0).ready entity=mon.2 client_cookie=fd5ae35ebea4f81 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad1ffb700 1 -- 172.21.15.3:0/2867676402 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faac4014070 con 0x7faad40af480 2022-07-21T01:42:38.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad1ffb700 1 -- 172.21.15.3:0/2867676402 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac4000e90 con 0x7faad40af480 2022-07-21T01:42:38.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.551+0000 7faad1ffb700 1 -- 172.21.15.3:0/2867676402 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faac400eaf0 con 0x7faad40af480 2022-07-21T01:42:38.564 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 -- 172.21.15.3:0/2867676402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40af480 msgr2=0x7faad40abb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 --2- 172.21.15.3:0/2867676402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40af480 0x7faad40abb10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7faac4002f70 tx=0x7faac4006eb0).stop 2022-07-21T01:42:38.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 -- 172.21.15.3:0/2867676402 shutdown_connections 2022-07-21T01:42:38.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 --2- 172.21.15.3:0/2867676402 >> v2:172.21.15.143:3300/0 conn(0x7faad40b22a0 0x7faad40ac000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 --2- 172.21.15.3:0/2867676402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40af480 0x7faad40abb10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 -- 172.21.15.3:0/2867676402 >> 172.21.15.3:0/2867676402 conn(0x7faad40626f0 msgr2=0x7faad4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:38.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 -- 172.21.15.3:0/2867676402 shutdown_connections 2022-07-21T01:42:38.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 -- 172.21.15.3:0/2867676402 wait complete. 2022-07-21T01:42:38.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 Processor -- start 2022-07-21T01:42:38.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 -- start start 2022-07-21T01:42:38.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faad40af480 0x7faad4147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40b1a80 0x7faad40b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad40b22a0 0x7faad4154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad40b59e0 con 0x7faad40af480 2022-07-21T01:42:38.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad4063550 con 0x7faad40b22a0 2022-07-21T01:42:38.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad3fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad4151840 con 0x7faad40b1a80 2022-07-21T01:42:38.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faad40af480 0x7faad4147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faad40af480 0x7faad4147650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51520/0 (socket says 172.21.15.3:51520) 2022-07-21T01:42:38.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad2ffd700 1 -- 172.21.15.3:0/275415661 learned_addr learned my addr 172.21.15.3:0/275415661 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:38.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad27fc700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40b1a80 0x7faad40b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad37fe700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad40b22a0 0x7faad4154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad2ffd700 1 -- 172.21.15.3:0/275415661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40b1a80 msgr2=0x7faad40b54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad2ffd700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40b1a80 0x7faad40b54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad2ffd700 1 -- 172.21.15.3:0/275415661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad40b22a0 msgr2=0x7faad4154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad2ffd700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad40b22a0 0x7faad4154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad2ffd700 1 -- 172.21.15.3:0/275415661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac400b1e0 con 0x7faad40af480 2022-07-21T01:42:38.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.555+0000 7faad37fe700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad40b22a0 0x7faad4154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:38.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faad2ffd700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faad40af480 0x7faad4147650 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7faac4002f70 tx=0x7faac4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faac3fff700 1 -- 172.21.15.3:0/275415661 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faac4010e30 con 0x7faad40af480 2022-07-21T01:42:38.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faac3fff700 1 -- 172.21.15.3:0/275415661 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac401c440 con 0x7faad40af480 2022-07-21T01:42:38.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad4063970 con 0x7faad40af480 2022-07-21T01:42:38.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faac3fff700 1 -- 172.21.15.3:0/275415661 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faac401c7a0 con 0x7faad40af480 2022-07-21T01:42:38.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad413a290 con 0x7faad40af480 2022-07-21T01:42:38.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad413d510 con 0x7faad40af480 2022-07-21T01:42:38.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faac3fff700 1 -- 172.21.15.3:0/275415661 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faac400e040 con 0x7faad40af480 2022-07-21T01:42:38.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faac3fff700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faab00539e0 0x7faab0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faad27fc700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faab00539e0 0x7faab0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faac3fff700 1 -- 172.21.15.3:0/275415661 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7faac406dc30 con 0x7faad40af480 2022-07-21T01:42:38.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.559+0000 7faad27fc700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faab00539e0 0x7faab0055ea0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7faac800b000 tx=0x7faac8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.567+0000 7faac3fff700 1 -- 172.21.15.3:0/275415661 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faad413d510 con 0x7faad40af480 2022-07-21T01:42:38.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.695+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faad413d510 con 0x7faad40af480 2022-07-21T01:42:38.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.695+0000 7faac3fff700 1 -- 172.21.15.3:0/275415661 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v150) v1 ==== 104+0+10 (secure 0 0 0) 0x7faad413d510 con 0x7faad40af480 2022-07-21T01:42:38.701 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:38.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faab00539e0 msgr2=0x7faab0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faab00539e0 0x7faab0055ea0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7faac800b000 tx=0x7faac8006df0).stop 2022-07-21T01:42:38.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faad40af480 msgr2=0x7faad4147650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faad40af480 0x7faad4147650 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7faac4002f70 tx=0x7faac4016000).stop 2022-07-21T01:42:38.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 shutdown_connections 2022-07-21T01:42:38.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7faab00539e0 0x7faab0055ea0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad40b22a0 0x7faad4154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40b1a80 0x7faad40b54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 --2- 172.21.15.3:0/275415661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faad40af480 0x7faad4147650 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 >> 172.21.15.3:0/275415661 conn(0x7faad40626f0 msgr2=0x7faad413dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:38.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 shutdown_connections 2022-07-21T01:42:38.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.699+0000 7faad3fff700 1 -- 172.21.15.3:0/275415661 wait complete. 2022-07-21T01:42:38.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:42:38.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.823+0000 7fa3aa9d8700 1 Processor -- start 2022-07-21T01:42:38.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.823+0000 7fa3aa9d8700 1 -- start start 2022-07-21T01:42:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a40af550 0x7fa3a40b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa3a405c6f0 0x7fa3a40a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a40aa630 con 0x7fa3a4060330 2022-07-21T01:42:38.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a40b15e0 con 0x7fa3a405c6f0 2022-07-21T01:42:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a40af930 con 0x7fa3a40af550 2022-07-21T01:42:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa1d7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa3a405c6f0 0x7fa3a40a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa1d7700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fa3a405c6f0 0x7fa3a40a8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51532/0 (socket says 172.21.15.3:51532) 2022-07-21T01:42:38.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa1d7700 1 -- 172.21.15.3:0/1454940852 learned_addr learned my addr 172.21.15.3:0/1454940852 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3a91d5700 1 --1- 172.21.15.3:0/1454940852 >> v1:172.21.15.3:6789/0 conn(0x7fa3a4060330 0x7fa3a41445e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59810/0 (socket says 172.21.15.3:59810) 2022-07-21T01:42:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3a99d6700 1 --2- 172.21.15.3:0/1454940852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a40af550 0x7fa3a40b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa1d7700 1 -- 172.21.15.3:0/1454940852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a40af550 msgr2=0x7fa3a40b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:38.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa1d7700 1 --2- 172.21.15.3:0/1454940852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a40af550 0x7fa3a40b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa1d7700 1 -- 172.21.15.3:0/1454940852 >> v1:172.21.15.3:6789/0 conn(0x7fa3a4060330 legacy=0x7fa3a41445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa1d7700 1 -- 172.21.15.3:0/1454940852 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a405fd10 con 0x7fa3a405c6f0 2022-07-21T01:42:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa1d7700 1 --2- 172.21.15.3:0/1454940852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a405c6f0 0x7fa3a40a8d80 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7fa39c0032f0 tx=0x7fa39c009290).ready entity=mon.0 client_cookie=da5663f652237934 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3a89d4700 1 -- 172.21.15.3:0/1454940852 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa39c00b8a0 con 0x7fa3a405c6f0 2022-07-21T01:42:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3a89d4700 1 -- 172.21.15.3:0/1454940852 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39c00ba00 con 0x7fa3a405c6f0 2022-07-21T01:42:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3a89d4700 1 -- 172.21.15.3:0/1454940852 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa39c00bb70 con 0x7fa3a405c6f0 2022-07-21T01:42:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/1454940852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a405c6f0 msgr2=0x7fa3a40a8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 --2- 172.21.15.3:0/1454940852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a405c6f0 0x7fa3a40a8d80 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7fa39c0032f0 tx=0x7fa39c009290).stop 2022-07-21T01:42:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/1454940852 shutdown_connections 2022-07-21T01:42:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 --2- 172.21.15.3:0/1454940852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a405c6f0 0x7fa3a40a8d80 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 --2- 172.21.15.3:0/1454940852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a40af550 0x7fa3a40b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/1454940852 >> 172.21.15.3:0/1454940852 conn(0x7fa3a40612d0 msgr2=0x7fa3a4062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/1454940852 shutdown_connections 2022-07-21T01:42:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.827+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/1454940852 wait complete. 2022-07-21T01:42:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa9d8700 1 Processor -- start 2022-07-21T01:42:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa9d8700 1 -- start start 2022-07-21T01:42:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa9d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3a405c6f0 0x7fa3a4143000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa9d8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a4060330 0x7fa3a4144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa9d8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a40af550 0x7fa3a413dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa9d8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a4154150 con 0x7fa3a40af550 2022-07-21T01:42:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa9d8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a4154620 con 0x7fa3a405c6f0 2022-07-21T01:42:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa9d8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a41452f0 con 0x7fa3a4060330 2022-07-21T01:42:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa1d7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a40af550 0x7fa3a413dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa1d7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a40af550 0x7fa3a413dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51536/0 (socket says 172.21.15.3:51536) 2022-07-21T01:42:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa1d7700 1 -- 172.21.15.3:0/917634731 learned_addr learned my addr 172.21.15.3:0/917634731 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3a99d6700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3a405c6f0 0x7fa3a4143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3a91d5700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a4060330 0x7fa3a4144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa1d7700 1 -- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a4060330 msgr2=0x7fa3a4144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa1d7700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a4060330 0x7fa3a4144520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa1d7700 1 -- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3a405c6f0 msgr2=0x7fa3a4143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa1d7700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3a405c6f0 0x7fa3a4143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa1d7700 1 -- 172.21.15.3:0/917634731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c009530 con 0x7fa3a40af550 2022-07-21T01:42:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3a99d6700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3a405c6f0 0x7fa3a4143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3a91d5700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a4060330 0x7fa3a4144520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa1d7700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a40af550 0x7fa3a413dd70 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7fa39c0032f0 tx=0x7fa39c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa392ffd700 1 -- 172.21.15.3:0/917634731 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa39c0203f0 con 0x7fa3a40af550 2022-07-21T01:42:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa392ffd700 1 -- 172.21.15.3:0/917634731 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39c020550 con 0x7fa3a40af550 2022-07-21T01:42:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.831+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a405f840 con 0x7fa3a40af550 2022-07-21T01:42:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.835+0000 7fa392ffd700 1 -- 172.21.15.3:0/917634731 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa39c0208b0 con 0x7fa3a40af550 2022-07-21T01:42:38.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.835+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a41438f0 con 0x7fa3a40af550 2022-07-21T01:42:38.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.835+0000 7fa392ffd700 1 -- 172.21.15.3:0/917634731 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa39c00f070 con 0x7fa3a40af550 2022-07-21T01:42:38.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.835+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a4143ab0 con 0x7fa3a40af550 2022-07-21T01:42:38.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.835+0000 7fa392ffd700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa384053b30 0x7fa384055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:38.857 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.835+0000 7fa392ffd700 1 -- 172.21.15.3:0/917634731 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fa39c06da50 con 0x7fa3a40af550 2022-07-21T01:42:38.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.835+0000 7fa3a99d6700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa384053b30 0x7fa384055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:38.858 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.835+0000 7fa3a99d6700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa384053b30 0x7fa384055ff0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fa3a000c000 tx=0x7fa3a0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:38.859 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.843+0000 7fa392ffd700 1 -- 172.21.15.3:0/917634731 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa39c044700 con 0x7fa3a40af550 2022-07-21T01:42:38.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3a405fa70 con 0x7fa3a40af550 2022-07-21T01:42:38.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa392ffd700 1 -- 172.21.15.3:0/917634731 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v150) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa39c047d20 con 0x7fa3a40af550 2022-07-21T01:42:38.974 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:38.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa384053b30 msgr2=0x7fa384055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa384053b30 0x7fa384055ff0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fa3a000c000 tx=0x7fa3a0008040).stop 2022-07-21T01:42:38.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a40af550 msgr2=0x7fa3a413dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:38.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a40af550 0x7fa3a413dd70 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7fa39c0032f0 tx=0x7fa39c016000).stop 2022-07-21T01:42:38.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 shutdown_connections 2022-07-21T01:42:38.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3a40af550 0x7fa3a413dd70 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3a4060330 0x7fa3a4144520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa384053b30 0x7fa384055ff0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 --2- 172.21.15.3:0/917634731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3a405c6f0 0x7fa3a4143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:38.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 >> 172.21.15.3:0/917634731 conn(0x7fa3a40612d0 msgr2=0x7fa3a40a9260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:38.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.971+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 shutdown_connections 2022-07-21T01:42:38.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:38.975+0000 7fa3aa9d8700 1 -- 172.21.15.3:0/917634731 wait complete. 2022-07-21T01:42:38.992 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:42:38.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:39.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 Processor -- start 2022-07-21T01:42:39.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 -- start start 2022-07-21T01:42:39.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c0b1f40 0x7fe59c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:39.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe59c148530 0x7fe59c0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:39.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59c0b04e0 con 0x7fe59c0a7250 2022-07-21T01:42:39.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59c0a9250 con 0x7fe59c148530 2022-07-21T01:42:39.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0b0780 con 0x7fe59c0b1f40 2022-07-21T01:42:39.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe593fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe59c148530 0x7fe59c0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:39.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe593fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe59c148530 0x7fe59c0b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:56994/0 (socket says 172.21.15.3:56994) 2022-07-21T01:42:39.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe59affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c0b1f40 0x7fe59c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:39.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe593fff700 1 -- 172.21.15.3:0/4121872608 learned_addr learned my addr 172.21.15.3:0/4121872608 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:39.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe593fff700 1 -- 172.21.15.3:0/4121872608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c0b1f40 msgr2=0x7fe59c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:39.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe593fff700 1 --2- 172.21.15.3:0/4121872608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c0b1f40 0x7fe59c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:39.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe593fff700 1 -- 172.21.15.3:0/4121872608 >> v1:172.21.15.143:6789/0 conn(0x7fe59c0a7250 legacy=0x7fe59c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:39.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe593fff700 1 -- 172.21.15.3:0/4121872608 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c0b0a20 con 0x7fe59c148530 2022-07-21T01:42:39.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe593fff700 1 --2- 172.21.15.3:0/4121872608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59c148530 0x7fe59c0b2320 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7fe584002f70 tx=0x7fe584006eb0).ready entity=mon.1 client_cookie=4c9852e916768784 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:39.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe59a7fc700 1 -- 172.21.15.3:0/4121872608 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe58400d570 con 0x7fe59c148530 2022-07-21T01:42:39.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe59a7fc700 1 -- 172.21.15.3:0/4121872608 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe584000e90 con 0x7fe59c148530 2022-07-21T01:42:39.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe59a7fc700 1 -- 172.21.15.3:0/4121872608 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe58400d900 con 0x7fe59c148530 2022-07-21T01:42:39.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 -- 172.21.15.3:0/4121872608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59c148530 msgr2=0x7fe59c0b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:39.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 --2- 172.21.15.3:0/4121872608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59c148530 0x7fe59c0b2320 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7fe584002f70 tx=0x7fe584006eb0).stop 2022-07-21T01:42:39.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 -- 172.21.15.3:0/4121872608 shutdown_connections 2022-07-21T01:42:39.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 --2- 172.21.15.3:0/4121872608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59c148530 0x7fe59c0b2320 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:39.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 --2- 172.21.15.3:0/4121872608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c0b1f40 0x7fe59c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:39.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 -- 172.21.15.3:0/4121872608 >> 172.21.15.3:0/4121872608 conn(0x7fe59c05f840 msgr2=0x7fe59c05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:39.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 -- 172.21.15.3:0/4121872608 shutdown_connections 2022-07-21T01:42:39.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.095+0000 7fe5a16a1700 1 -- 172.21.15.3:0/4121872608 wait complete. 2022-07-21T01:42:39.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe5a16a1700 1 Processor -- start 2022-07-21T01:42:39.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe5a16a1700 1 -- start start 2022-07-21T01:42:39.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe5a16a1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe59c0a7250 0x7fe59c14b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:39.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe5a16a1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59c0b1f40 0x7fe59c149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:39.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe5a16a1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c148530 0x7fe59c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:39.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe5a16a1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c156e20 con 0x7fe59c0a7250 2022-07-21T01:42:39.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe5a16a1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c14b250 con 0x7fe59c0b1f40 2022-07-21T01:42:39.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe5a16a1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c14a380 con 0x7fe59c148530 2022-07-21T01:42:39.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe59c0a7250 0x7fe59c14b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:39.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe59c0a7250 0x7fe59c14b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51550/0 (socket says 172.21.15.3:51550) 2022-07-21T01:42:39.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59affd700 1 -- 172.21.15.3:0/3753599404 learned_addr learned my addr 172.21.15.3:0/3753599404 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:39.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe593fff700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59c0b1f40 0x7fe59c149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:39.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59b7fe700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c148530 0x7fe59c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:39.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59affd700 1 -- 172.21.15.3:0/3753599404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c148530 msgr2=0x7fe59c0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:39.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59affd700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c148530 0x7fe59c0ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:39.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59affd700 1 -- 172.21.15.3:0/3753599404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59c0b1f40 msgr2=0x7fe59c149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:39.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59affd700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59c0b1f40 0x7fe59c149e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:39.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59affd700 1 -- 172.21.15.3:0/3753599404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58400b1e0 con 0x7fe59c0a7250 2022-07-21T01:42:39.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe59affd700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe59c0a7250 0x7fe59c14b760 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7fe58c00c000 tx=0x7fe58c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:39.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe598ff9700 1 -- 172.21.15.3:0/3753599404 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe58c010780 con 0x7fe59c0a7250 2022-07-21T01:42:39.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe598ff9700 1 -- 172.21.15.3:0/3753599404 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58c0108e0 con 0x7fe59c0a7250 2022-07-21T01:42:39.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe598ff9700 1 -- 172.21.15.3:0/3753599404 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe58c010c40 con 0x7fe59c0a7250 2022-07-21T01:42:39.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.099+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59c0a96e0 con 0x7fe59c0a7250 2022-07-21T01:42:39.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.103+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59c0a9c20 con 0x7fe59c0a7250 2022-07-21T01:42:39.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.103+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59c14b010 con 0x7fe59c0a7250 2022-07-21T01:42:39.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.103+0000 7fe598ff9700 1 -- 172.21.15.3:0/3753599404 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe58c01c460 con 0x7fe59c0a7250 2022-07-21T01:42:39.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.103+0000 7fe598ff9700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe580053b00 0x7fe580055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:39.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.103+0000 7fe598ff9700 1 -- 172.21.15.3:0/3753599404 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe58c06c3a0 con 0x7fe59c0a7250 2022-07-21T01:42:39.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.103+0000 7fe593fff700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe580053b00 0x7fe580055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:39.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.103+0000 7fe593fff700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe580053b00 0x7fe580055fc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fe58401c000 tx=0x7fe584015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:39.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.111+0000 7fe598ff9700 1 -- 172.21.15.3:0/3753599404 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe59c14b010 con 0x7fe59c0a7250 2022-07-21T01:42:39.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.279+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe59c14b010 con 0x7fe59c0a7250 2022-07-21T01:42:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.279+0000 7fe598ff9700 1 -- 172.21.15.3:0/3753599404 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2492 (secure 0 0 0) 0x7fe59c14b010 con 0x7fe59c0a7250 2022-07-21T01:42:39.283 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:39.284 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/1563084587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1563084587},{"type":"v1","addr":"172.21.15.3:6835","nonce":1563084587}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1403026914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1403026914},{"type":"v1","addr":"172.21.15.3:6837","nonce":1403026914}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:34.642400+0000","modified":"2022-07-21T01:42:35.646272+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8409":{"gid":8409,"name":"b","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/1158093851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1158093851},{"type":"v1","addr":"172.21.15.143:6833","nonce":1158093851}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-21T01:42:39.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe580053b00 msgr2=0x7fe580055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe580053b00 0x7fe580055fc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fe58401c000 tx=0x7fe584015000).stop 2022-07-21T01:42:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe59c0a7250 msgr2=0x7fe59c14b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe59c0a7250 0x7fe59c14b760 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7fe58c00c000 tx=0x7fe58c006df0).stop 2022-07-21T01:42:39.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 shutdown_connections 2022-07-21T01:42:39.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59c148530 0x7fe59c0ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:39.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe580053b00 0x7fe580055fc0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:39.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59c0b1f40 0x7fe59c149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 --2- 172.21.15.3:0/3753599404 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe59c0a7250 0x7fe59c14b760 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 >> 172.21.15.3:0/3753599404 conn(0x7fe59c05f840 msgr2=0x7fe59c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 shutdown_connections 2022-07-21T01:42:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:39.283+0000 7fe5a16a1700 1 -- 172.21.15.3:0/3753599404 wait complete. 2022-07-21T01:42:39.289 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 76 2022-07-21T01:42:39.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:34.642400+0000', 'modified': '2022-07-21T01:42:35.646272+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8409}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8409': {'gid': 8409, 'name': 'b', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6833/1158093851', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 1158093851}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 1158093851}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:42:39.606 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:42:39.608 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-21T01:42:39.726 INFO:teuthology.orchestra.run.smithi143.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8409,"want_state":"up:active","state":"up:active","rank_uptime":5.0712924140000002,"mdsmap_epoch":76,"osdmap_epoch":150,"osdmap_epoch_barrier":150,"uptime":9.4035420159999994} 2022-07-21T01:42:39.727 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8409, "mdsmap_epoch": 76, "osdmap_epoch": 150, "osdmap_epoch_barrier": 150, "rank_uptime": 5.071292414, "state": "up:active", "uptime": 9.403542016, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:42:39.727 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:42:39.728 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:42:39.729 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:42:39.729 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:39.730 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:39.743 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:42:39.808 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:39.809 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:39.825 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:39.828 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:39.828 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:42:39.874 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:42:39.875 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:39.877 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:42:39.895 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:39.896 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:39.905 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:39.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:39.908 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:42:39.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23442]: starting ceph client2022-07-21T01:42:39.947+0000 7f0c95581080 -1 init, newargv = 0x55613a6c67b0 newargc=9 2022-07-21T01:42:39.952 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr: 2022-07-21T01:42:39.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23442]: starting fuse 2022-07-21T01:42:41.055 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:42:41.074 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:41.075 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:41.090 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:41.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:41.092 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:42:41.142 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:42:41.143 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:42:41.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:41.144 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:42:41.144 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:42:41.145 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:42:41.145 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:42:41.146 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:42:41.146 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:42:41.147 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:42:41.147 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:42:41.148 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:42:41.148 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:42:41.148 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:42:41.149 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:42:41.149 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:42:41.150 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:42:41.150 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:42:41.150 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:42:41.151 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:42:41.151 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:42:41.152 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:42:41.152 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:42:41.153 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:42:41.153 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:42:41.153 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:42:41.198 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.23442.asok 2022-07-21T01:42:41.203 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23442.asok 2022-07-21T01:42:41.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:41.204 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23442.asok status 2022-07-21T01:42:41.298 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:42:41.298 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:42:41.298 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:42:41.299 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:42:41.299 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:42:41.299 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:42:41.299 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:42:41.300 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "23442", 2022-07-21T01:42:41.300 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:42:41.300 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:42:41.300 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:42:41.301 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:42:41.301 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 8464, 2022-07-21T01:42:41.301 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:42:41.301 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:42:41.302 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:42:41.302 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 8464 2022-07-21T01:42:41.302 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:42:41.302 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:42:41.303 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:42:41.303 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:42:41.303 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3118881381 2022-07-21T01:42:41.303 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:42:41.304 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:42:41.304 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:42:41.304 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:42:41.304 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:42:41.305 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 3118881381 2022-07-21T01:42:41.305 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:42:41.305 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.8464 172.21.15.171:0/3118881381", 2022-07-21T01:42:41.306 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/3118881381", 2022-07-21T01:42:41.306 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:42:41.306 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 76, 2022-07-21T01:42:41.307 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 150, 2022-07-21T01:42:41.307 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:42:41.307 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:42:41.307 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:42:41.308 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:42:41.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:41.309 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:41.310 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:41.317 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:42:41.318 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:42:41.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:41.319 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:41.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:41.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a68e0d700 1 Processor -- start 2022-07-21T01:42:41.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a68e0d700 1 -- start start 2022-07-21T01:42:41.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a68e0d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3a64067180 0x7f3a64143a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a68e0d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a64067560 con 0x7f3a640b0870 2022-07-21T01:42:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a68e0d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640b0160 con 0x7f3a640ad1c0 2022-07-21T01:42:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a68e0d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a640b0400 con 0x7f3a64067180 2022-07-21T01:42:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a62d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3a640b0870 0x7f3a640b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59840/0 (socket says 172.21.15.3:59840) 2022-07-21T01:42:41.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a62d9d700 1 -- 172.21.15.3:0/3263876566 learned_addr learned my addr 172.21.15.3:0/3263876566 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:41.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a6259c700 1 --2- 172.21.15.3:0/3263876566 >> v2:172.21.15.3:3300/0 conn(0x7f3a64067180 0x7f3a64143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a6159a700 1 -- 172.21.15.3:0/3263876566 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126633897 0 0) 0x7f3a64067560 con 0x7f3a640b0870 2022-07-21T01:42:41.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a6159a700 1 -- 172.21.15.3:0/3263876566 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a48003260 con 0x7f3a640b0870 2022-07-21T01:42:41.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a6159a700 1 -- 172.21.15.3:0/3263876566 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082953739 0 0) 0x7f3a640b0160 con 0x7f3a640ad1c0 2022-07-21T01:42:41.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a6159a700 1 -- 172.21.15.3:0/3263876566 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a64067560 con 0x7f3a640ad1c0 2022-07-21T01:42:41.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.439+0000 7f3a6259c700 1 -- 172.21.15.3:0/3263876566 >> v1:172.21.15.143:6789/0 conn(0x7f3a640ad1c0 legacy=0x7f3a64143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a6259c700 1 -- 172.21.15.3:0/3263876566 >> v1:172.21.15.3:6789/0 conn(0x7f3a640b0870 legacy=0x7f3a640b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a6159a700 1 -- 172.21.15.3:0/3263876566 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074548660 0 0) 0x7f3a48003260 con 0x7f3a640b0870 2022-07-21T01:42:41.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a6259c700 1 -- 172.21.15.3:0/3263876566 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a640af530 con 0x7f3a64067180 2022-07-21T01:42:41.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a6259c700 1 --2- 172.21.15.3:0/3263876566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a64067180 0x7f3a64143a60 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f3a54001820 tx=0x7f3a58006df0).ready entity=mon.0 client_cookie=f91be098c60108e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a6159a700 1 -- 172.21.15.3:0/3263876566 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a54001680 con 0x7f3a64067180 2022-07-21T01:42:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a6159a700 1 -- 172.21.15.3:0/3263876566 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a58000e90 con 0x7f3a64067180 2022-07-21T01:42:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a6159a700 1 -- 172.21.15.3:0/3263876566 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a58012460 con 0x7f3a64067180 2022-07-21T01:42:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a68e0d700 1 -- 172.21.15.3:0/3263876566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a64067180 msgr2=0x7f3a64143a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a68e0d700 1 --2- 172.21.15.3:0/3263876566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a64067180 0x7f3a64143a60 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f3a54001820 tx=0x7f3a58006df0).stop 2022-07-21T01:42:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a68e0d700 1 -- 172.21.15.3:0/3263876566 shutdown_connections 2022-07-21T01:42:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a68e0d700 1 --2- 172.21.15.3:0/3263876566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a64067180 0x7f3a64143a60 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a68e0d700 1 -- 172.21.15.3:0/3263876566 >> 172.21.15.3:0/3263876566 conn(0x7f3a6405f840 msgr2=0x7f3a6405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a68e0d700 1 -- 172.21.15.3:0/3263876566 shutdown_connections 2022-07-21T01:42:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a68e0d700 1 -- 172.21.15.3:0/3263876566 wait complete. 2022-07-21T01:42:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a68e0d700 1 Processor -- start 2022-07-21T01:42:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.443+0000 7f3a68e0d700 1 -- start start 2022-07-21T01:42:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a68e0d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a64067180 0x7f3a640a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a68e0d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a640b0870 0x7f3a64152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a68e0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a64152fa0 0x7f3a6414fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a68e0d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a640a4a30 con 0x7f3a64152fa0 2022-07-21T01:42:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a68e0d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a640a2b20 con 0x7f3a64067180 2022-07-21T01:42:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a68e0d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a640a2dc0 con 0x7f3a640b0870 2022-07-21T01:42:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a62d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a64152fa0 0x7f3a6414fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a61d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a640b0870 0x7f3a64152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a6259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a64067180 0x7f3a640a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a61d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a640b0870 0x7f3a64152ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:58478/0 (socket says 172.21.15.3:58478) 2022-07-21T01:42:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a61d9b700 1 -- 172.21.15.3:0/4129973549 learned_addr learned my addr 172.21.15.3:0/4129973549 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a61d9b700 1 -- 172.21.15.3:0/4129973549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a64067180 msgr2=0x7f3a640a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a61d9b700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a64067180 0x7f3a640a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a61d9b700 1 -- 172.21.15.3:0/4129973549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a64152fa0 msgr2=0x7f3a6414fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a61d9b700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a64152fa0 0x7f3a6414fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a61d9b700 1 -- 172.21.15.3:0/4129973549 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5800b180 con 0x7f3a640b0870 2022-07-21T01:42:41.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a61d9b700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a640b0870 0x7f3a64152ab0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f3a5c001490 tx=0x7f3a5c007e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:41.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a537fe700 1 -- 172.21.15.3:0/4129973549 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a5c002310 con 0x7f3a640b0870 2022-07-21T01:42:41.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a537fe700 1 -- 172.21.15.3:0/4129973549 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c002470 con 0x7f3a640b0870 2022-07-21T01:42:41.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a640a3060 con 0x7f3a640b0870 2022-07-21T01:42:41.470 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a537fe700 1 -- 172.21.15.3:0/4129973549 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a5c001840 con 0x7f3a640b0870 2022-07-21T01:42:41.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.447+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a640a44b0 con 0x7f3a640b0870 2022-07-21T01:42:41.471 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.451+0000 7f3a537fe700 1 -- 172.21.15.3:0/4129973549 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3a5c019020 con 0x7f3a640b0870 2022-07-21T01:42:41.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.451+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a640574e0 con 0x7f3a640b0870 2022-07-21T01:42:41.472 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.451+0000 7f3a537fe700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a40053a70 0x7f3a40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.451+0000 7f3a537fe700 1 -- 172.21.15.3:0/4129973549 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3a5c06be50 con 0x7f3a640b0870 2022-07-21T01:42:41.473 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.451+0000 7f3a6259c700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a40053a70 0x7f3a40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.451+0000 7f3a6259c700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a40053a70 0x7f3a40055f30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f3a5801a000 tx=0x7f3a58014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:41.474 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.459+0000 7f3a537fe700 1 -- 172.21.15.3:0/4129973549 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3a5c042bb0 con 0x7f3a640b0870 2022-07-21T01:42:41.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.587+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a640447e0 con 0x7f3a640b0870 2022-07-21T01:42:41.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.595+0000 7f3a537fe700 1 -- 172.21.15.3:0/4129973549 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f3a5c0461d0 con 0x7f3a640b0870 2022-07-21T01:42:41.597 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:41.597 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":150,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:35.645281+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:33.092672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:34.046506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:41.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a40053a70 msgr2=0x7f3a40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a40053a70 0x7f3a40055f30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f3a5801a000 tx=0x7f3a58014000).stop 2022-07-21T01:42:41.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a640b0870 msgr2=0x7f3a64152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a640b0870 0x7f3a64152ab0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f3a5c001490 tx=0x7f3a5c007e30).stop 2022-07-21T01:42:41.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 shutdown_connections 2022-07-21T01:42:41.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a64152fa0 0x7f3a6414fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a640b0870 0x7f3a64152ab0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3a40053a70 0x7f3a40055f30 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 --2- 172.21.15.3:0/4129973549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a64067180 0x7f3a640a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 >> 172.21.15.3:0/4129973549 conn(0x7f3a6405f840 msgr2=0x7f3a64141ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:41.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 shutdown_connections 2022-07-21T01:42:41.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.599+0000 7f3a68e0d700 1 -- 172.21.15.3:0/4129973549 wait complete. 2022-07-21T01:42:41.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:42:33.092672+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:42:34.046506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:42:41.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:42:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f6621534700 1 Processor -- start 2022-07-21T01:42:41.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f6621534700 1 -- start start 2022-07-21T01:42:41.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f6621534700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f661c0aebd0 0x7f661c143940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f6621534700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 0x7f661c143e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f6621534700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0b0b20 con 0x7f661c0ad190 2022-07-21T01:42:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f6621534700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c0adde0 con 0x7f661c0aebd0 2022-07-21T01:42:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f6621534700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0aefb0 con 0x7f661c0b0610 2022-07-21T01:42:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f661bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f661c0aebd0 0x7f661c143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f661bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f661c0aebd0 0x7f661c143940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51570/0 (socket says 172.21.15.3:51570) 2022-07-21T01:42:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.727+0000 7f661bfff700 1 -- 172.21.15.3:0/2902237230 learned_addr learned my addr 172.21.15.3:0/2902237230 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6620d33700 1 --2- 172.21.15.3:0/2902237230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 0x7f661c143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f661bfff700 1 -- 172.21.15.3:0/2902237230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 msgr2=0x7f661c143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f661bfff700 1 --2- 172.21.15.3:0/2902237230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 0x7f661c143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f661bfff700 1 -- 172.21.15.3:0/2902237230 >> v1:172.21.15.143:6789/0 conn(0x7f661c0ad190 legacy=0x7f661c146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f661bfff700 1 -- 172.21.15.3:0/2902237230 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c146890 con 0x7f661c0aebd0 2022-07-21T01:42:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f661bfff700 1 --2- 172.21.15.3:0/2902237230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0aebd0 0x7f661c143940 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f660c002f70 tx=0x7f660c006eb0).ready entity=mon.0 client_cookie=c5097d584fe59e09 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f661affd700 1 -- 172.21.15.3:0/2902237230 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c00d5c0 con 0x7f661c0aebd0 2022-07-21T01:42:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f661affd700 1 -- 172.21.15.3:0/2902237230 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c000e90 con 0x7f661c0aebd0 2022-07-21T01:42:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f661affd700 1 -- 172.21.15.3:0/2902237230 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c00d950 con 0x7f661c0aebd0 2022-07-21T01:42:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 -- 172.21.15.3:0/2902237230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0aebd0 msgr2=0x7f661c143940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 --2- 172.21.15.3:0/2902237230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0aebd0 0x7f661c143940 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f660c002f70 tx=0x7f660c006eb0).stop 2022-07-21T01:42:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 -- 172.21.15.3:0/2902237230 shutdown_connections 2022-07-21T01:42:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 --2- 172.21.15.3:0/2902237230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 0x7f661c143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 --2- 172.21.15.3:0/2902237230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0aebd0 0x7f661c143940 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 -- 172.21.15.3:0/2902237230 >> 172.21.15.3:0/2902237230 conn(0x7f661c0613c0 msgr2=0x7f661c062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 -- 172.21.15.3:0/2902237230 shutdown_connections 2022-07-21T01:42:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 -- 172.21.15.3:0/2902237230 wait complete. 2022-07-21T01:42:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 Processor -- start 2022-07-21T01:42:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.731+0000 7f6621534700 1 -- start start 2022-07-21T01:42:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6621534700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0ad190 0x7f661c0b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6621534700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c0aebd0 0x7f661c0637b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6621534700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 0x7f661c063ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6621534700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c13ebd0 con 0x7f661c0ad190 2022-07-21T01:42:41.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6621534700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c064190 con 0x7f661c0aebd0 2022-07-21T01:42:41.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6621534700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0b4480 con 0x7f661c0b0610 2022-07-21T01:42:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0ad190 0x7f661c0b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0ad190 0x7f661c0b50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51576/0 (socket says 172.21.15.3:51576) 2022-07-21T01:42:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661bfff700 1 -- 172.21.15.3:0/4137107750 learned_addr learned my addr 172.21.15.3:0/4137107750 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661b7fe700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c0aebd0 0x7f661c0637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6620d33700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 0x7f661c063ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661bfff700 1 -- 172.21.15.3:0/4137107750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 msgr2=0x7f661c063ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661bfff700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 0x7f661c063ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661bfff700 1 -- 172.21.15.3:0/4137107750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c0aebd0 msgr2=0x7f661c0637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661bfff700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c0aebd0 0x7f661c0637b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661bfff700 1 -- 172.21.15.3:0/4137107750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c00b1e0 con 0x7f661c0ad190 2022-07-21T01:42:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f661bfff700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0ad190 0x7f661c0b50d0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f661c065cc0 tx=0x7f660c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f66197fa700 1 -- 172.21.15.3:0/4137107750 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c0116e0 con 0x7f661c0ad190 2022-07-21T01:42:41.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f66197fa700 1 -- 172.21.15.3:0/4137107750 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c011840 con 0x7f661c0ad190 2022-07-21T01:42:41.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f66197fa700 1 -- 172.21.15.3:0/4137107750 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c011ba0 con 0x7f661c0ad190 2022-07-21T01:42:41.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c0b4720 con 0x7f661c0ad190 2022-07-21T01:42:41.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c13edc0 con 0x7f661c0ad190 2022-07-21T01:42:41.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c0b0d50 con 0x7f661c0ad190 2022-07-21T01:42:41.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.735+0000 7f66197fa700 1 -- 172.21.15.3:0/4137107750 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f660c025030 con 0x7f661c0ad190 2022-07-21T01:42:41.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.739+0000 7f66197fa700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6600053a10 0x7f6600055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:41.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.739+0000 7f66197fa700 1 -- 172.21.15.3:0/4137107750 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f660c06d210 con 0x7f661c0ad190 2022-07-21T01:42:41.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.739+0000 7f661b7fe700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6600053a10 0x7f6600055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:41.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.739+0000 7f661b7fe700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6600053a10 0x7f6600055ed0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f660400d000 tx=0x7f6604007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:41.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.747+0000 7f66197fa700 1 -- 172.21.15.3:0/4137107750 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f660c043fd0 con 0x7f661c0ad190 2022-07-21T01:42:41.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f661c0574e0 con 0x7f661c0ad190 2022-07-21T01:42:41.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f66197fa700 1 -- 172.21.15.3:0/4137107750 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v150) v1 ==== 110+0+10 (secure 0 0 0) 0x7f660c0230b0 con 0x7f661c0ad190 2022-07-21T01:42:41.879 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:42:41.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6600053a10 msgr2=0x7f6600055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6600053a10 0x7f6600055ed0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f660400d000 tx=0x7f6604007b90).stop 2022-07-21T01:42:41.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0ad190 msgr2=0x7f661c0b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:41.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0ad190 0x7f661c0b50d0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f661c065cc0 tx=0x7f660c015000).stop 2022-07-21T01:42:41.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 shutdown_connections 2022-07-21T01:42:41.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f661c0b0610 0x7f661c063ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f6600053a10 0x7f6600055ed0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c0aebd0 0x7f661c0637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 --2- 172.21.15.3:0/4137107750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f661c0ad190 0x7f661c0b50d0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:41.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 >> 172.21.15.3:0/4137107750 conn(0x7f661c0613c0 msgr2=0x7f661c13bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:41.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.875+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 shutdown_connections 2022-07-21T01:42:41.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.879+0000 7f6621534700 1 -- 172.21.15.3:0/4137107750 wait complete. 2022-07-21T01:42:41.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:42:42.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.999+0000 7f624f59e700 1 Processor -- start 2022-07-21T01:42:42.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.999+0000 7f624f59e700 1 -- start start 2022-07-21T01:42:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.999+0000 7f624f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f62500b6420 0x7f6250061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.999+0000 7f624f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b41f0 0x7f6250061af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.999+0000 7f624f59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62500b60e0 con 0x7f6250061fe0 2022-07-21T01:42:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.999+0000 7f624f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62500b2100 con 0x7f62500b6420 2022-07-21T01:42:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:41.999+0000 7f624f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62500b23a0 con 0x7f62500b41f0 2022-07-21T01:42:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624ed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6250061fe0 0x7f62501465c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59868/0 (socket says 172.21.15.3:59868) 2022-07-21T01:42:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624ed9d700 1 -- 172.21.15.3:0/1197791455 learned_addr learned my addr 172.21.15.3:0/1197791455 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624e59c700 1 --2- 172.21.15.3:0/1197791455 >> v2:172.21.15.143:3300/0 conn(0x7f62500b6420 0x7f6250061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624dd9b700 1 --2- 172.21.15.3:0/1197791455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b41f0 0x7f6250061af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624d59a700 1 -- 172.21.15.3:0/1197791455 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461571061 0 0) 0x7f6248001490 con 0x7f6250061fe0 2022-07-21T01:42:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624e59c700 1 -- 172.21.15.3:0/1197791455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b41f0 msgr2=0x7f6250061af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624e59c700 1 --2- 172.21.15.3:0/1197791455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b41f0 0x7f6250061af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624e59c700 1 -- 172.21.15.3:0/1197791455 >> v1:172.21.15.3:6789/0 conn(0x7f6250061fe0 legacy=0x7f62501465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624e59c700 1 -- 172.21.15.3:0/1197791455 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62500b2640 con 0x7f62500b6420 2022-07-21T01:42:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624e59c700 1 --2- 172.21.15.3:0/1197791455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62500b6420 0x7f6250061600 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f6240002f70 tx=0x7f6240006df0).ready entity=mon.1 client_cookie=8fd7b8d677f39098 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624d59a700 1 -- 172.21.15.3:0/1197791455 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f624000d570 con 0x7f62500b6420 2022-07-21T01:42:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624d59a700 1 -- 172.21.15.3:0/1197791455 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6240000e90 con 0x7f62500b6420 2022-07-21T01:42:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624d59a700 1 -- 172.21.15.3:0/1197791455 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f624000d8c0 con 0x7f62500b6420 2022-07-21T01:42:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 -- 172.21.15.3:0/1197791455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62500b6420 msgr2=0x7f6250061600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 --2- 172.21.15.3:0/1197791455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62500b6420 0x7f6250061600 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f6240002f70 tx=0x7f6240006df0).stop 2022-07-21T01:42:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 -- 172.21.15.3:0/1197791455 shutdown_connections 2022-07-21T01:42:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 --2- 172.21.15.3:0/1197791455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b41f0 0x7f6250061af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 --2- 172.21.15.3:0/1197791455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62500b6420 0x7f6250061600 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 -- 172.21.15.3:0/1197791455 >> 172.21.15.3:0/1197791455 conn(0x7f625005c620 msgr2=0x7f625005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 -- 172.21.15.3:0/1197791455 shutdown_connections 2022-07-21T01:42:42.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 -- 172.21.15.3:0/1197791455 wait complete. 2022-07-21T01:42:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 Processor -- start 2022-07-21T01:42:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.003+0000 7f624f59e700 1 -- start start 2022-07-21T01:42:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6250061fe0 0x7f62500b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624f59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62500b41f0 0x7f6250141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b6420 0x7f6250141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624f59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6250153830 con 0x7f62500b41f0 2022-07-21T01:42:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6250143ae0 con 0x7f6250061fe0 2022-07-21T01:42:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6250143d80 con 0x7f62500b6420 2022-07-21T01:42:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62500b41f0 0x7f6250141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62500b41f0 0x7f6250141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51592/0 (socket says 172.21.15.3:51592) 2022-07-21T01:42:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624dd9b700 1 -- 172.21.15.3:0/3124295491 learned_addr learned my addr 172.21.15.3:0/3124295491 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624e59c700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6250061fe0 0x7f62500b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624ed9d700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b6420 0x7f6250141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624dd9b700 1 -- 172.21.15.3:0/3124295491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b6420 msgr2=0x7f6250141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624dd9b700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b6420 0x7f6250141780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624dd9b700 1 -- 172.21.15.3:0/3124295491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6250061fe0 msgr2=0x7f62500b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624dd9b700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6250061fe0 0x7f62500b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624dd9b700 1 -- 172.21.15.3:0/3124295491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624000b180 con 0x7f62500b41f0 2022-07-21T01:42:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624dd9b700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62500b41f0 0x7f6250141290 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7f624400c000 tx=0x7f6244006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f623f7fe700 1 -- 172.21.15.3:0/3124295491 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6244010590 con 0x7f62500b41f0 2022-07-21T01:42:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f623f7fe700 1 -- 172.21.15.3:0/3124295491 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62440106f0 con 0x7f62500b41f0 2022-07-21T01:42:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6250144020 con 0x7f62500b41f0 2022-07-21T01:42:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f623f7fe700 1 -- 172.21.15.3:0/3124295491 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6244010a50 con 0x7f62500b41f0 2022-07-21T01:42:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6250141c70 con 0x7f62500b41f0 2022-07-21T01:42:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62500afef0 con 0x7f62500b41f0 2022-07-21T01:42:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.007+0000 7f623f7fe700 1 -- 172.21.15.3:0/3124295491 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6244010bb0 con 0x7f62500b41f0 2022-07-21T01:42:42.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.011+0000 7f623f7fe700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f622c053a70 0x7f622c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.011+0000 7f623f7fe700 1 -- 172.21.15.3:0/3124295491 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f624406c090 con 0x7f62500b41f0 2022-07-21T01:42:42.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.011+0000 7f624e59c700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f622c053a70 0x7f622c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.011+0000 7f624e59c700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f622c053a70 0x7f622c055f30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f624001a000 tx=0x7f6240014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.015+0000 7f623f7fe700 1 -- 172.21.15.3:0/3124295491 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6244042d40 con 0x7f62500b41f0 2022-07-21T01:42:42.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.139+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f62500b0c40 con 0x7f62500b41f0 2022-07-21T01:42:42.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.139+0000 7f623f7fe700 1 -- 172.21.15.3:0/3124295491 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v150) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6244046360 con 0x7f62500b41f0 2022-07-21T01:42:42.145 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:42.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f622c053a70 msgr2=0x7f622c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f622c053a70 0x7f622c055f30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f624001a000 tx=0x7f6240014000).stop 2022-07-21T01:42:42.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62500b41f0 msgr2=0x7f6250141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62500b41f0 0x7f6250141290 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7f624400c000 tx=0x7f6244006df0).stop 2022-07-21T01:42:42.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 shutdown_connections 2022-07-21T01:42:42.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62500b6420 0x7f6250141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62500b41f0 0x7f6250141290 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f622c053a70 0x7f622c055f30 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 --2- 172.21.15.3:0/3124295491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6250061fe0 0x7f62500b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.152 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 >> 172.21.15.3:0/3124295491 conn(0x7f625005c620 msgr2=0x7f62500b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:42.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 shutdown_connections 2022-07-21T01:42:42.153 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.143+0000 7f624f59e700 1 -- 172.21.15.3:0/3124295491 wait complete. 2022-07-21T01:42:42.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:42:42.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc70630b700 1 Processor -- start 2022-07-21T01:42:42.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc70630b700 1 -- start start 2022-07-21T01:42:42.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc70630b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc700063a60 0x7fc700143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc70630b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700061ab0 0x7fc700140b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc70630b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc700063e40 con 0x7fc7000b1460 2022-07-21T01:42:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc70630b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70013ff40 con 0x7fc700063a60 2022-07-21T01:42:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc70630b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc700062180 con 0x7fc700061ab0 2022-07-21T01:42:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc705309700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc700063a60 0x7fc700143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc705309700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc700063a60 0x7fc700143ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51598/0 (socket says 172.21.15.3:51598) 2022-07-21T01:42:42.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.271+0000 7fc705309700 1 -- 172.21.15.3:0/1951349542 learned_addr learned my addr 172.21.15.3:0/1951349542 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:42.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc704b08700 1 --2- 172.21.15.3:0/1951349542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700061ab0 0x7fc700140b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc705309700 1 -- 172.21.15.3:0/1951349542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700061ab0 msgr2=0x7fc700140b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:42.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc705309700 1 --2- 172.21.15.3:0/1951349542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700061ab0 0x7fc700140b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc705309700 1 -- 172.21.15.3:0/1951349542 >> v1:172.21.15.143:6789/0 conn(0x7fc7000b1460 legacy=0x7fc700128e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc705309700 1 -- 172.21.15.3:0/1951349542 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc700140130 con 0x7fc700063a60 2022-07-21T01:42:42.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc705309700 1 --2- 172.21.15.3:0/1951349542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc700063a60 0x7fc700143ee0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7fc6fc002f70 tx=0x7fc6fc006eb0).ready entity=mon.0 client_cookie=c6c58ec10d2b632b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc6ebfff700 1 -- 172.21.15.3:0/1951349542 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6fc00d570 con 0x7fc700063a60 2022-07-21T01:42:42.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc6ebfff700 1 -- 172.21.15.3:0/1951349542 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc000e90 con 0x7fc700063a60 2022-07-21T01:42:42.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc6ebfff700 1 -- 172.21.15.3:0/1951349542 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6fc00d900 con 0x7fc700063a60 2022-07-21T01:42:42.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc70630b700 1 -- 172.21.15.3:0/1951349542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc700063a60 msgr2=0x7fc700143ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc70630b700 1 --2- 172.21.15.3:0/1951349542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc700063a60 0x7fc700143ee0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7fc6fc002f70 tx=0x7fc6fc006eb0).stop 2022-07-21T01:42:42.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc70630b700 1 -- 172.21.15.3:0/1951349542 shutdown_connections 2022-07-21T01:42:42.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc70630b700 1 --2- 172.21.15.3:0/1951349542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700061ab0 0x7fc700140b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc70630b700 1 --2- 172.21.15.3:0/1951349542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc700063a60 0x7fc700143ee0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc70630b700 1 -- 172.21.15.3:0/1951349542 >> 172.21.15.3:0/1951349542 conn(0x7fc70005c620 msgr2=0x7fc70005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:42.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc70630b700 1 -- 172.21.15.3:0/1951349542 shutdown_connections 2022-07-21T01:42:42.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc70630b700 1 -- 172.21.15.3:0/1951349542 wait complete. 2022-07-21T01:42:42.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.275+0000 7fc70630b700 1 Processor -- start 2022-07-21T01:42:42.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc70630b700 1 -- start start 2022-07-21T01:42:42.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc70630b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc700061ab0 0x7fc70013f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc70630b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700063a60 0x7fc700152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc70630b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000b1460 0x7fc7001532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc70630b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7001537e0 con 0x7fc7000b1460 2022-07-21T01:42:42.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc70630b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc700064d60 con 0x7fc700061ab0 2022-07-21T01:42:42.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc70630b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc700065000 con 0x7fc700063a60 2022-07-21T01:42:42.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705b0a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000b1460 0x7fc7001532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705b0a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000b1460 0x7fc7001532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51608/0 (socket says 172.21.15.3:51608) 2022-07-21T01:42:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705309700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc700061ab0 0x7fc70013f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc704b08700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700063a60 0x7fc700152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705b0a700 1 -- 172.21.15.3:0/3907488259 learned_addr learned my addr 172.21.15.3:0/3907488259 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:42.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705b0a700 1 -- 172.21.15.3:0/3907488259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700063a60 msgr2=0x7fc700152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705b0a700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700063a60 0x7fc700152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705b0a700 1 -- 172.21.15.3:0/3907488259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc700061ab0 msgr2=0x7fc70013f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705b0a700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc700061ab0 0x7fc70013f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705b0a700 1 -- 172.21.15.3:0/3907488259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc00b1e0 con 0x7fc7000b1460 2022-07-21T01:42:42.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc705b0a700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000b1460 0x7fc7001532f0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7fc7000b6450 tx=0x7fc6f8007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc6f2ffd700 1 -- 172.21.15.3:0/3907488259 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6f8006aa0 con 0x7fc7000b1460 2022-07-21T01:42:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc6f2ffd700 1 -- 172.21.15.3:0/3907488259 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f8006c00 con 0x7fc7000b1460 2022-07-21T01:42:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc700152960 con 0x7fc7000b1460 2022-07-21T01:42:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc6f2ffd700 1 -- 172.21.15.3:0/3907488259 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6f8007030 con 0x7fc7000b1460 2022-07-21T01:42:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.279+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70013eb10 con 0x7fc7000b1460 2022-07-21T01:42:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.283+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7000574e0 con 0x7fc7000b1460 2022-07-21T01:42:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.283+0000 7fc6f2ffd700 1 -- 172.21.15.3:0/3907488259 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc6f801b060 con 0x7fc7000b1460 2022-07-21T01:42:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.283+0000 7fc6f2ffd700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6dc053a70 0x7fc6dc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.283+0000 7fc6f2ffd700 1 -- 172.21.15.3:0/3907488259 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fc6f806bc30 con 0x7fc7000b1460 2022-07-21T01:42:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.283+0000 7fc705309700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6dc053a70 0x7fc6dc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.283+0000 7fc705309700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6dc053a70 0x7fc6dc055f30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fc6fc01c000 tx=0x7fc6fc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.291+0000 7fc6f2ffd700 1 -- 172.21.15.3:0/3907488259 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc6f80429f0 con 0x7fc7000b1460 2022-07-21T01:42:42.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.419+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc700152af0 con 0x7fc7000b1460 2022-07-21T01:42:42.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc6f2ffd700 1 -- 172.21.15.3:0/3907488259 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v150) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc6f8019070 con 0x7fc7000b1460 2022-07-21T01:42:42.427 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:42.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6dc053a70 msgr2=0x7fc6dc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6dc053a70 0x7fc6dc055f30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fc6fc01c000 tx=0x7fc6fc015000).stop 2022-07-21T01:42:42.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000b1460 msgr2=0x7fc7001532f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000b1460 0x7fc7001532f0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7fc7000b6450 tx=0x7fc6f8007f70).stop 2022-07-21T01:42:42.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 shutdown_connections 2022-07-21T01:42:42.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7000b1460 0x7fc7001532f0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc700063a60 0x7fc700152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc6dc053a70 0x7fc6dc055f30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 --2- 172.21.15.3:0/3907488259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc700061ab0 0x7fc70013f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 >> 172.21.15.3:0/3907488259 conn(0x7fc70005c620 msgr2=0x7fc700142d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:42.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 shutdown_connections 2022-07-21T01:42:42.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.423+0000 7fc70630b700 1 -- 172.21.15.3:0/3907488259 wait complete. 2022-07-21T01:42:42.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:42:42.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.547+0000 7f5f34b3b700 1 Processor -- start 2022-07-21T01:42:42.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.547+0000 7f5f34b3b700 1 -- start start 2022-07-21T01:42:42.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.547+0000 7f5f34b3b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 0x7f5f30061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.547+0000 7f5f34b3b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5f300ad750 0x7f5f300b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.547+0000 7f5f34b3b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f300aee80 con 0x7f5f301405d0 2022-07-21T01:42:42.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.547+0000 7f5f34b3b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f300b70b0 con 0x7f5f300ad750 2022-07-21T01:42:42.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.547+0000 7f5f34b3b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300af0b0 con 0x7f5f300614b0 2022-07-21T01:42:42.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5f300ad750 0x7f5f300b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5f300ad750 0x7f5f300b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51614/0 (socket says 172.21.15.3:51614) 2022-07-21T01:42:42.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2effd700 1 -- 172.21.15.3:0/2296190954 learned_addr learned my addr 172.21.15.3:0/2296190954 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:42.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2f7fe700 1 --2- 172.21.15.3:0/2296190954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 0x7f5f30061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2effd700 1 -- 172.21.15.3:0/2296190954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 msgr2=0x7f5f30061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:42.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2effd700 1 --2- 172.21.15.3:0/2296190954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 0x7f5f30061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2e7fc700 1 -- 172.21.15.3:0/2296190954 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073821077 0 0) 0x7f5f300aee80 con 0x7f5f301405d0 2022-07-21T01:42:42.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2effd700 1 -- 172.21.15.3:0/2296190954 >> v1:172.21.15.143:6789/0 conn(0x7f5f301405d0 legacy=0x7f5f301466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2effd700 1 -- 172.21.15.3:0/2296190954 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f300af350 con 0x7f5f300ad750 2022-07-21T01:42:42.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2f7fe700 1 --2- 172.21.15.3:0/2296190954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 0x7f5f30061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:42.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2effd700 1 --2- 172.21.15.3:0/2296190954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 0x7f5f300b9ff0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f5f20002f70 tx=0x7f5f20006eb0).ready entity=mon.0 client_cookie=af5850c960e137cc server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2e7fc700 1 -- 172.21.15.3:0/2296190954 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2000d570 con 0x7f5f300ad750 2022-07-21T01:42:42.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2e7fc700 1 -- 172.21.15.3:0/2296190954 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f20000e90 con 0x7f5f300ad750 2022-07-21T01:42:42.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f2e7fc700 1 -- 172.21.15.3:0/2296190954 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2000d8c0 con 0x7f5f300ad750 2022-07-21T01:42:42.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2296190954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 msgr2=0x7f5f300b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 --2- 172.21.15.3:0/2296190954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 0x7f5f300b9ff0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f5f20002f70 tx=0x7f5f20006eb0).stop 2022-07-21T01:42:42.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2296190954 shutdown_connections 2022-07-21T01:42:42.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 --2- 172.21.15.3:0/2296190954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 0x7f5f300b9ff0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 --2- 172.21.15.3:0/2296190954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 0x7f5f30061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2296190954 >> 172.21.15.3:0/2296190954 conn(0x7f5f30066f70 msgr2=0x7f5f30067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:42.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2296190954 shutdown_connections 2022-07-21T01:42:42.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2296190954 wait complete. 2022-07-21T01:42:42.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 Processor -- start 2022-07-21T01:42:42.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.551+0000 7f5f34b3b700 1 -- start start 2022-07-21T01:42:42.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f34b3b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 0x7f5f300b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f34b3b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 0x7f5f300b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f34b3b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f301405d0 0x7f5f3009ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f34b3b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300afe70 con 0x7f5f300ad750 2022-07-21T01:42:42.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f34b3b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300b0f90 con 0x7f5f301405d0 2022-07-21T01:42:42.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f34b3b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300b1390 con 0x7f5f300614b0 2022-07-21T01:42:42.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 0x7f5f300b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 0x7f5f300b25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51618/0 (socket says 172.21.15.3:51618) 2022-07-21T01:42:42.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f301405d0 0x7f5f3009ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2effd700 1 -- 172.21.15.3:0/2011741021 learned_addr learned my addr 172.21.15.3:0/2011741021 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:42.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2f7fe700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 0x7f5f300b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2effd700 1 -- 172.21.15.3:0/2011741021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 msgr2=0x7f5f300b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2effd700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 0x7f5f300b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2effd700 1 -- 172.21.15.3:0/2011741021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f301405d0 msgr2=0x7f5f3009ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2effd700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f301405d0 0x7f5f3009ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2effd700 1 -- 172.21.15.3:0/2011741021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2400b180 con 0x7f5f300ad750 2022-07-21T01:42:42.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2effd700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 0x7f5f300b25c0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f5f300b3b80 tx=0x7f5f20015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2cff9700 1 -- 172.21.15.3:0/2011741021 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f200068b0 con 0x7f5f300ad750 2022-07-21T01:42:42.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2cff9700 1 -- 172.21.15.3:0/2011741021 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2000d8f0 con 0x7f5f300ad750 2022-07-21T01:42:42.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2000b1e0 con 0x7f5f300ad750 2022-07-21T01:42:42.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2cff9700 1 -- 172.21.15.3:0/2011741021 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f20011d30 con 0x7f5f300ad750 2022-07-21T01:42:42.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f301568c0 con 0x7f5f300ad750 2022-07-21T01:42:42.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.555+0000 7f5f2cff9700 1 -- 172.21.15.3:0/2011741021 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f20025030 con 0x7f5f300ad750 2022-07-21T01:42:42.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.559+0000 7f5f2cff9700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f10053950 0x7f5f10055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.559+0000 7f5f2cff9700 1 -- 172.21.15.3:0/2011741021 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f5f2006d0d0 con 0x7f5f300ad750 2022-07-21T01:42:42.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.559+0000 7f5f2f7fe700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f10053950 0x7f5f10055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.559+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f300af5f0 con 0x7f5f300ad750 2022-07-21T01:42:42.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.559+0000 7f5f2f7fe700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f10053950 0x7f5f10055e10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f5f2400b000 tx=0x7f5f24006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.563+0000 7f5f2cff9700 1 -- 172.21.15.3:0/2011741021 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f20023020 con 0x7f5f300ad750 2022-07-21T01:42:42.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.687+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5f300574e0 con 0x7f5f300ad750 2022-07-21T01:42:42.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.687+0000 7f5f2cff9700 1 -- 172.21.15.3:0/2011741021 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v150) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5f20043e90 con 0x7f5f300ad750 2022-07-21T01:42:42.692 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:42.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f10053950 msgr2=0x7f5f10055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f10053950 0x7f5f10055e10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f5f2400b000 tx=0x7f5f24006df0).stop 2022-07-21T01:42:42.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 msgr2=0x7f5f300b25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 0x7f5f300b25c0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f5f300b3b80 tx=0x7f5f20015000).stop 2022-07-21T01:42:42.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 shutdown_connections 2022-07-21T01:42:42.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5f10053950 0x7f5f10055e10 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f301405d0 0x7f5f3009ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f300ad750 0x7f5f300b25c0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 --2- 172.21.15.3:0/2011741021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f300614b0 0x7f5f300b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 >> 172.21.15.3:0/2011741021 conn(0x7f5f30066f70 msgr2=0x7f5f30146910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:42.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 shutdown_connections 2022-07-21T01:42:42.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.691+0000 7f5f34b3b700 1 -- 172.21.15.3:0/2011741021 wait complete. 2022-07-21T01:42:42.706 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:42:42.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.807+0000 7f9b7b59e700 1 Processor -- start 2022-07-21T01:42:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.807+0000 7f9b7b59e700 1 -- start start 2022-07-21T01:42:42.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 0x7f9b7c14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9b7c0b01a0 0x7f9b7c14b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c0a7250 con 0x7f9b7c0afd60 2022-07-21T01:42:42.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b7c0a7480 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0a7720 con 0x7f9b7c0b2710 2022-07-21T01:42:42.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b79d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9b7c0b01a0 0x7f9b7c14b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b79d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f9b7c0b01a0 0x7f9b7c14b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51626/0 (socket says 172.21.15.3:51626) 2022-07-21T01:42:42.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b79d9b700 1 -- 172.21.15.3:0/3691693595 learned_addr learned my addr 172.21.15.3:0/3691693595 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:42.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7a59c700 1 --2- 172.21.15.3:0/3691693595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 0x7f9b7c14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b79d9b700 1 -- 172.21.15.3:0/3691693595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 msgr2=0x7f9b7c14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:42.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b79d9b700 1 --2- 172.21.15.3:0/3691693595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 0x7f9b7c14afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b79d9b700 1 -- 172.21.15.3:0/3691693595 >> v1:172.21.15.143:6789/0 conn(0x7f9b7c0afd60 legacy=0x7f9b7c0a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b79d9b700 1 -- 172.21.15.3:0/3691693595 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c0a79c0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7a59c700 1 --2- 172.21.15.3:0/3691693595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 0x7f9b7c14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:42.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b79d9b700 1 --2- 172.21.15.3:0/3691693595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 0x7f9b7c14b4c0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f9b6c002f70 tx=0x7f9b6c006eb0).ready entity=mon.0 client_cookie=bd6f7489cadd1d65 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.824 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7959a700 1 -- 172.21.15.3:0/3691693595 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b6c00d5c0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7959a700 1 -- 172.21.15.3:0/3691693595 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c000e90 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.825 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7959a700 1 -- 172.21.15.3:0/3691693595 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b6c00d950 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 -- 172.21.15.3:0/3691693595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 msgr2=0x7f9b7c14b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 --2- 172.21.15.3:0/3691693595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 0x7f9b7c14b4c0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f9b6c002f70 tx=0x7f9b6c006eb0).stop 2022-07-21T01:42:42.826 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 -- 172.21.15.3:0/3691693595 shutdown_connections 2022-07-21T01:42:42.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 --2- 172.21.15.3:0/3691693595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 0x7f9b7c14b4c0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 --2- 172.21.15.3:0/3691693595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 0x7f9b7c14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 -- 172.21.15.3:0/3691693595 >> 172.21.15.3:0/3691693595 conn(0x7f9b7c060310 msgr2=0x7f9b7c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:42.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 -- 172.21.15.3:0/3691693595 shutdown_connections 2022-07-21T01:42:42.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 -- 172.21.15.3:0/3691693595 wait complete. 2022-07-21T01:42:42.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 Processor -- start 2022-07-21T01:42:42.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.811+0000 7f9b7b59e700 1 -- start start 2022-07-21T01:42:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b7c0afd60 0x7f9b7c0ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 0x7f9b7c0b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 0x7f9b7c1414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0aacb0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0b3390 con 0x7f9b7c0afd60 2022-07-21T01:42:42.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0b3630 con 0x7f9b7c0b2710 2022-07-21T01:42:42.835 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b79d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 0x7f9b7c0b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b79d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 0x7f9b7c0b3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51636/0 (socket says 172.21.15.3:51636) 2022-07-21T01:42:42.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b79d9b700 1 -- 172.21.15.3:0/13052944 learned_addr learned my addr 172.21.15.3:0/13052944 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:42.836 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7a59c700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b7c0afd60 0x7f9b7c0ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7ad9d700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 0x7f9b7c1414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.837 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b79d9b700 1 -- 172.21.15.3:0/13052944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 msgr2=0x7f9b7c1414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b79d9b700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 0x7f9b7c1414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b79d9b700 1 -- 172.21.15.3:0/13052944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b7c0afd60 msgr2=0x7f9b7c0ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.838 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b79d9b700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b7c0afd60 0x7f9b7c0ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b79d9b700 1 -- 172.21.15.3:0/13052944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c00b1e0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b79d9b700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 0x7f9b7c0b3eb0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f9b6c002f70 tx=0x7f9b6c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/13052944 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b6c0117e0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/13052944 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c011940 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/13052944 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b6c011ca0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7c0b38d0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7c0b43a0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.815+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7c0574e0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.819+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/13052944 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9b6c025030 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.819+0000 7f9b6b7fe700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b58053b00 0x7f9b58055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:42.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.819+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/13052944 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f9b6c06d3a0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.819+0000 7f9b7a59c700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b58053b00 0x7f9b58055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:42.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.819+0000 7f9b7a59c700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b58053b00 0x7f9b58055fc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f9b7000c000 tx=0x7f9b70008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:42.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.827+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/13052944 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9b6c043fd0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b7c0aadf0 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b6b7fe700 1 -- 172.21.15.3:0/13052944 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2492 (secure 0 0 0) 0x7f9b6c023090 con 0x7f9b7c0b01a0 2022-07-21T01:42:42.994 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:42.994 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/1563084587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1563084587},{"type":"v1","addr":"172.21.15.3:6835","nonce":1563084587}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1403026914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1403026914},{"type":"v1","addr":"172.21.15.3:6837","nonce":1403026914}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:34.642400+0000","modified":"2022-07-21T01:42:35.646272+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8409":{"gid":8409,"name":"b","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/1158093851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1158093851},{"type":"v1","addr":"172.21.15.143:6833","nonce":1158093851}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-21T01:42:42.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b58053b00 msgr2=0x7f9b58055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b58053b00 0x7f9b58055fc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f9b7000c000 tx=0x7f9b70008040).stop 2022-07-21T01:42:42.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 msgr2=0x7f9b7c0b3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 0x7f9b7c0b3eb0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f9b6c002f70 tx=0x7f9b6c015000).stop 2022-07-21T01:42:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 shutdown_connections 2022-07-21T01:42:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b7c0b2710 0x7f9b7c1414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b7c0b01a0 0x7f9b7c0b3eb0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f9b58053b00 0x7f9b58055fc0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 --2- 172.21.15.3:0/13052944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b7c0afd60 0x7f9b7c0ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:42.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 >> 172.21.15.3:0/13052944 conn(0x7f9b7c060310 msgr2=0x7f9b7c146150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:42.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 shutdown_connections 2022-07-21T01:42:42.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:42.991+0000 7f9b7b59e700 1 -- 172.21.15.3:0/13052944 wait complete. 2022-07-21T01:42:42.999 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 76 2022-07-21T01:42:43.310 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:34.642400+0000', 'modified': '2022-07-21T01:42:35.646272+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8409}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8409': {'gid': 8409, 'name': 'b', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6833/1158093851', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 1158093851}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 1158093851}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:42:43.311 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:42:43.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-21T01:42:43.425 INFO:teuthology.orchestra.run.smithi143.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8409,"want_state":"up:active","state":"up:active","rank_uptime":8.7716173469999994,"mdsmap_epoch":76,"osdmap_epoch":150,"osdmap_epoch_barrier":150,"uptime":13.103856834} 2022-07-21T01:42:43.426 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8409, "mdsmap_epoch": 76, "osdmap_epoch": 150, "osdmap_epoch_barrier": 150, "rank_uptime": 8.771617347, "state": "up:active", "uptime": 13.103856834, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:42:43.427 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:42:43.427 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:42:43.427 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:42:43.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:43.428 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:43.437 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:42:43.497 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:43.498 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:43.510 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:43.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:43.514 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:42:43.520 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:42:43.521 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:43.566 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:42:43.584 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:43.586 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:43.597 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:43.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:43.600 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:42:43.642 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:42:43.639+0000 7fa51b048080 -1 init, newargv = 0x55a6efb36aa0 newargc=9ceph-fuse[23060]: starting ceph client 2022-07-21T01:42:43.643 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr: 2022-07-21T01:42:43.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[23060]: starting fuse 2022-07-21T01:42:44.747 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:42:44.767 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:42:44.769 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:42:44.785 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:42:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:42:44.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:44.788 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:42:44.834 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:42:44.835 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:42:44.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:44.836 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:42:44.836 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:42:44.836 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:42:44.837 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:42:44.837 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:42:44.846 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:44.846 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:42:44.847 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:42:44.847 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:42:44.848 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:44.848 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:42:44.848 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:42:44.849 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:42:44.849 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:42:44.852 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:44.853 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:42:44.853 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:42:44.854 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:42:44.854 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:42:44.855 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:42:44.856 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:42:44.856 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:42:44.857 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:42:44.902 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.23060.asok 2022-07-21T01:42:44.907 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.23060.asok 2022-07-21T01:42:44.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:44.908 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.23060.asok status 2022-07-21T01:42:44.998 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:42:44.998 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:42:44.999 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:42:44.999 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:42:44.999 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:42:44.999 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:42:45.000 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:42:45.000 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "23060", 2022-07-21T01:42:45.000 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:42:45.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:45.001 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:42:45.001 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:42:45.001 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 8601, 2022-07-21T01:42:45.001 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:42:45.002 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:42:45.002 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:42:45.002 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 8601 2022-07-21T01:42:45.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:45.003 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:42:45.003 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:42:45.003 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:42:45.004 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 1310010324 2022-07-21T01:42:45.004 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:42:45.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:45.004 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:42:45.005 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:42:45.005 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:42:45.005 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 1310010324 2022-07-21T01:42:45.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:42:45.005 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.8601 172.21.15.174:0/1310010324", 2022-07-21T01:42:45.006 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/1310010324", 2022-07-21T01:42:45.006 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:42:45.006 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 76, 2022-07-21T01:42:45.007 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 150, 2022-07-21T01:42:45.007 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:42:45.007 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:42:45.008 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:42:45.008 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:42:45.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:45.010 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:45.010 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:45.017 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:42:45.019 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:45.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:45.020 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:45.037 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-21T01:42:45.038 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.rctime . 2022-07-21T01:42:45.071 INFO:teuthology.orchestra.run.smithi171.stdout:1658367765.033796310 2022-07-21T01:42:45.072 INFO:tasks.cephfs.test_misc:rctime = 1658367765.033796310 2022-07-21T01:42:45.073 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:42:45.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:45.095 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:45.118 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:45.139 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:45.162 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:42:45.162 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:45.163 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:45.163 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:45.172 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:42:45.173 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:42:45.174 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi171.front.sepia.ceph.com... 2022-07-21T01:42:45.174 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:42:45.175 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:45.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23442]: fuse finished with error 0 and tester_r 0 2022-07-21T01:42:45.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:45.198 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:45.199 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:45.242 INFO:teuthology.orchestra.run.smithi171.stdout:ext2/ext3 2022-07-21T01:42:45.244 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:42:45.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:42:45.354 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:45.363 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:45.364 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:45.365 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:45.374 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:42:45.375 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:45.376 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:42:45.376 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:42:45.377 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:45.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[23060]: fuse finished with error 0 and tester_r 0 2022-07-21T01:42:45.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:45.395 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:45.396 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:45.442 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:42:45.444 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:42:45.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:42:45.446 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:45.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:42:45.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.563+0000 7f3386772700 1 Processor -- start 2022-07-21T01:42:45.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.563+0000 7f3386772700 1 -- start start 2022-07-21T01:42:45.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f33800b1f40 0x7f33800b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:45.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33800b04e0 con 0x7f33800a7250 2022-07-21T01:42:45.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33800b0780 con 0x7f3380148530 2022-07-21T01:42:45.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33800a9250 con 0x7f33800b1f40 2022-07-21T01:42:45.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3384d0f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f33800a7250 0x7f33801448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59924/0 (socket says 172.21.15.3:59924) 2022-07-21T01:42:45.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3384d0f700 1 -- 172.21.15.3:0/3483340342 learned_addr learned my addr 172.21.15.3:0/3483340342 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:45.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337ffff700 1 --2- 172.21.15.3:0/3483340342 >> v2:172.21.15.143:3300/0 conn(0x7f33800b1f40 0x7f33800b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:45.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337effd700 1 -- 172.21.15.3:0/3483340342 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138309435 0 0) 0x7f33800b04e0 con 0x7f33800a7250 2022-07-21T01:42:45.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337effd700 1 -- 172.21.15.3:0/3483340342 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f336c003260 con 0x7f33800a7250 2022-07-21T01:42:45.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337effd700 1 -- 172.21.15.3:0/3483340342 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966602230 0 0) 0x7f33800b0780 con 0x7f3380148530 2022-07-21T01:42:45.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337effd700 1 -- 172.21.15.3:0/3483340342 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33800b04e0 con 0x7f3380148530 2022-07-21T01:42:45.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337effd700 1 -- 172.21.15.3:0/3483340342 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1488658208 0 0) 0x7f336c003260 con 0x7f33800a7250 2022-07-21T01:42:45.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337ffff700 1 -- 172.21.15.3:0/3483340342 >> v1:172.21.15.143:6789/0 conn(0x7f3380148530 legacy=0x7f33800b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:45.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337ffff700 1 -- 172.21.15.3:0/3483340342 >> v1:172.21.15.3:6789/0 conn(0x7f33800a7250 legacy=0x7f33801448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:45.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337ffff700 1 -- 172.21.15.3:0/3483340342 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33800b0a20 con 0x7f33800b1f40 2022-07-21T01:42:45.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337ffff700 1 --2- 172.21.15.3:0/3483340342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800b1f40 0x7f33800b1160 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f3378001b80 tx=0x7f3374006df0).ready entity=mon.1 client_cookie=63b9b7995ca40bc3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:45.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337effd700 1 -- 172.21.15.3:0/3483340342 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33780019e0 con 0x7f33800b1f40 2022-07-21T01:42:45.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337effd700 1 -- 172.21.15.3:0/3483340342 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3374000e90 con 0x7f33800b1f40 2022-07-21T01:42:45.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f337effd700 1 -- 172.21.15.3:0/3483340342 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3374012460 con 0x7f33800b1f40 2022-07-21T01:42:45.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 -- 172.21.15.3:0/3483340342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800b1f40 msgr2=0x7f33800b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:45.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 --2- 172.21.15.3:0/3483340342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800b1f40 0x7f33800b1160 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f3378001b80 tx=0x7f3374006df0).stop 2022-07-21T01:42:45.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 -- 172.21.15.3:0/3483340342 shutdown_connections 2022-07-21T01:42:45.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 --2- 172.21.15.3:0/3483340342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800b1f40 0x7f33800b1160 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:45.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 -- 172.21.15.3:0/3483340342 >> 172.21.15.3:0/3483340342 conn(0x7f338005f840 msgr2=0x7f338005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:45.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 -- 172.21.15.3:0/3483340342 shutdown_connections 2022-07-21T01:42:45.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.567+0000 7f3386772700 1 -- 172.21.15.3:0/3483340342 wait complete. 2022-07-21T01:42:45.588 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 Processor -- start 2022-07-21T01:42:45.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 -- start start 2022-07-21T01:42:45.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800a7250 0x7f33800a9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:45.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33800b1f40 0x7f3380149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:45.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3380148530 0x7f33800abe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:45.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338014a740 con 0x7f33800b1f40 2022-07-21T01:42:45.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338014b1f0 con 0x7f33800a7250 2022-07-21T01:42:45.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338014b490 con 0x7f3380148530 2022-07-21T01:42:45.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33800b1f40 0x7f3380149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:45.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800a7250 0x7f33800a9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:45.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3384d0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3380148530 0x7f33800abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:45.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800a7250 0x7f33800a9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:57094/0 (socket says 172.21.15.3:57094) 2022-07-21T01:42:45.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3384d0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3380148530 0x7f33800abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:58564/0 (socket says 172.21.15.3:58564) 2022-07-21T01:42:45.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337ffff700 1 -- 172.21.15.3:0/1928945626 learned_addr learned my addr 172.21.15.3:0/1928945626 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:45.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337ffff700 1 -- 172.21.15.3:0/1928945626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3380148530 msgr2=0x7f33800abe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:45.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337ffff700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3380148530 0x7f33800abe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:45.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337ffff700 1 -- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33800b1f40 msgr2=0x7f3380149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:45.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337ffff700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33800b1f40 0x7f3380149dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:45.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337ffff700 1 -- 172.21.15.3:0/1928945626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337400b180 con 0x7f33800a7250 2022-07-21T01:42:45.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337f7fe700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33800b1f40 0x7f3380149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:45.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337ffff700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800a7250 0x7f33800a9910 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f3378001b80 tx=0x7f3374014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:45.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337d7fa700 1 -- 172.21.15.3:0/1928945626 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f337400d920 con 0x7f33800a7250 2022-07-21T01:42:45.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337d7fa700 1 -- 172.21.15.3:0/1928945626 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337400da80 con 0x7f33800a7250 2022-07-21T01:42:45.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338014b730 con 0x7f33800a7250 2022-07-21T01:42:45.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f337d7fa700 1 -- 172.21.15.3:0/1928945626 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f337400dde0 con 0x7f33800a7250 2022-07-21T01:42:45.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.571+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33800abb00 con 0x7f33800a7250 2022-07-21T01:42:45.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.575+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338014b8c0 con 0x7f33800a7250 2022-07-21T01:42:45.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.575+0000 7f337d7fa700 1 -- 172.21.15.3:0/1928945626 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3374025030 con 0x7f33800a7250 2022-07-21T01:42:45.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.575+0000 7f337d7fa700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3360053a10 0x7f3360055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:45.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.575+0000 7f337d7fa700 1 -- 172.21.15.3:0/1928945626 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f337406cba0 con 0x7f33800a7250 2022-07-21T01:42:45.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.575+0000 7f337f7fe700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3360053a10 0x7f3360055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:45.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.575+0000 7f337f7fe700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3360053a10 0x7f3360055ed0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f337000d000 tx=0x7f3370007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:45.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.583+0000 7f337d7fa700 1 -- 172.21.15.3:0/1928945626 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f33740438e0 con 0x7f33800a7250 2022-07-21T01:42:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.711+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f33800574e0 con 0x7f33800a7250 2022-07-21T01:42:45.717 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f337d7fa700 1 -- 172.21.15.3:0/1928945626 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v150) v1 ==== 81+0+34051 (secure 0 0 0) 0x7f3374023070 con 0x7f33800a7250 2022-07-21T01:42:45.719 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:45.719 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:42:45.720 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 150, 2022-07-21T01:42:45.720 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:42:45.721 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:42:45.721 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:42:35.645281+0000", 2022-07-21T01:42:45.722 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:45.722 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:45.723 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:42:45.723 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:42:45.724 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:42:45.724 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:42:45.725 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:42:45.725 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:42:45.726 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:42:45.726 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:45.726 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:42:45.727 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:42:45.727 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:42:45.727 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:42:45.727 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:42:45.728 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 28, 2022-07-21T01:42:45.728 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:42:45.728 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:42:45.728 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:42:45.729 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:42:45.729 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:42:45.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.729 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:42:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:42:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:42:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:45.731 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:42:45.732 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:42:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:42:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:42:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:42:45.733 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:45.734 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.735 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:42:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:45.736 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:45.737 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:45.738 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:45.739 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:45.740 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:45.741 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:45.743 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:45.744 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:45.745 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:42:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:42:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:45.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:42:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.747 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:42:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:42:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:42:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:45.748 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:45.749 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:45.750 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:45.750 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:42:45.750 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:42:45.751 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:42:45.751 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:42:45.751 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:42:45.751 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:45.752 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:45.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:45.752 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:45.752 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:45.752 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:45.753 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:45.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.753 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:42:45.753 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:45.754 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:45.754 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:45.754 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:45.754 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:45.754 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:45.755 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:45.755 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:45.755 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:45.755 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:45.756 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:45.756 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:45.756 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:45.756 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:45.757 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:45.757 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:45.757 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:45.757 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:45.757 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:45.758 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:45.758 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:45.758 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:45.758 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:45.759 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:45.759 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:45.759 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:45.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.759 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:45.760 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:45.760 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:45.760 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:45.760 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:45.761 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:45.761 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:45.761 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:45.761 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:45.761 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:45.762 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:45.762 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:42:45.763 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:45.763 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:42:45.763 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 27, 2022-07-21T01:42:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:42:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:42:33.092672+0000", 2022-07-21T01:42:45.764 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:45.765 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:45.765 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:45.765 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:45.765 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:45.766 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:45.766 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:45.766 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:45.766 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:42:45.767 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:42:45.767 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:42:45.767 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:42:45.767 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:42:45.767 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:45.768 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:45.768 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:45.768 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:45.768 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:45.769 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:45.769 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:45.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.769 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "149", 2022-07-21T01:42:45.770 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:45.770 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:45.770 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:45.770 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:45.770 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:45.771 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:45.771 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:45.771 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:45.771 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:45.772 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:45.772 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:45.772 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:45.772 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:45.772 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:45.773 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:45.773 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:45.773 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:45.773 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:45.774 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:45.774 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:45.774 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:45.774 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:45.775 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:45.775 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:45.775 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:45.775 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:45.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.776 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:45.776 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:45.776 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:45.776 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:45.777 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:45.777 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:45.777 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:45.777 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:45.778 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:45.778 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:45.778 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:45.778 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:42:45.778 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:42:45.779 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:42:45.779 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:42:45.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.779 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:45.780 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:42:45.780 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:42:45.780 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.780 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.781 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 28, 2022-07-21T01:42:45.781 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:42:45.782 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:42:34.046506+0000", 2022-07-21T01:42:45.782 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:45.782 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:45.782 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:45.782 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:45.783 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:45.783 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:45.783 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:45.783 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:45.784 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:42:45.784 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:42:45.784 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:42:45.784 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:42:45.785 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:42:45.785 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:45.785 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:45.785 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:45.785 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:45.786 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:45.786 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:45.786 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:45.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.787 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "150", 2022-07-21T01:42:45.787 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:45.787 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:45.787 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:45.788 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:45.788 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:45.788 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:45.788 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:45.789 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:45.789 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:45.789 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:45.789 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:45.789 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:45.790 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:45.790 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:45.790 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:45.790 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:45.791 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:45.791 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:45.791 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:45.791 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:45.792 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:45.792 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:45.792 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:45.792 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:45.793 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:45.793 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:45.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.793 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:45.793 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:45.794 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:45.794 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:45.794 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:45.794 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:45.795 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:45.795 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:45.795 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:45.795 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:45.795 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:45.796 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:42:45.796 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:45.796 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:42:45.796 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:42:45.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.797 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:45.797 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:42:45.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.798 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:45.798 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:42:45.798 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:45.799 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:45.799 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:45.799 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:45.799 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:45.800 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:45.800 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:45.800 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:45.800 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:45.800 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:45.801 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:45.801 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.801 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.802 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:42:45.802 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:45.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.803 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.803 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:42:45.803 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:45.803 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.803 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.804 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:45.804 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.805 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.805 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:42:45.805 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:45.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.806 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.806 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:42:45.806 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:45.806 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.807 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.807 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:45.807 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.808 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.808 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:42:45.808 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:45.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.809 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.809 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:42:45.809 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:45.809 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.810 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.810 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:45.810 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.811 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.811 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:42:45.811 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:45.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.812 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.812 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:42:45.812 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:45.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.813 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.813 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:42:45.814 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:42:45.814 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:42:45.814 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:42:45.814 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:45.814 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:45.815 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:45.815 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.816 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:45.816 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:42:45.816 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:45.816 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:45.817 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:45.817 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:45.817 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:45.817 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:45.817 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:45.818 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:45.818 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:45.818 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:45.818 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:45.819 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.819 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.819 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:42:45.820 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:45.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.820 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.820 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:42:45.821 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:45.821 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.821 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.822 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:45.822 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.822 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:42:45.823 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:45.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.823 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.824 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:42:45.824 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:45.824 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.824 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.825 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:45.825 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.825 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.826 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:42:45.826 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:45.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.827 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:42:45.827 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:45.827 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.827 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.828 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:45.828 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.829 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.829 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:42:45.829 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:45.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.830 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.830 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:42:45.830 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:45.830 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.831 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.831 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:42:45.831 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:42:45.831 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:42:45.832 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:42:45.832 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:45.832 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:45.832 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:45.833 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.833 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:45.833 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:42:45.834 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:45.834 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:45.834 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:45.834 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:45.835 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:45.835 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:45.835 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:45.835 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 146, 2022-07-21T01:42:45.835 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:45.836 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:45.836 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:45.836 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.837 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.837 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:42:45.837 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:45.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.838 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.838 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:42:45.838 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:45.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.839 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.839 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:45.839 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.840 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.840 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:42:45.840 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:45.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.841 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.841 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:42:45.841 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:45.842 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.842 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.842 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:45.842 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.843 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.843 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:42:45.843 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:42:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:45.845 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.845 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.845 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:45.846 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.846 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.846 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:42:45.847 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:45.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.847 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.847 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:42:45.848 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:45.848 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.848 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.849 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:42:45.849 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:42:45.849 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:42:45.849 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:42:45.849 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:45.850 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:45.850 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:45.850 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.851 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:45.851 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:42:45.851 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:45.852 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:45.852 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:45.852 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:45.852 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:45.852 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:45.853 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:45.853 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 137, 2022-07-21T01:42:45.853 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:45.853 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:45.853 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:45.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.854 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:42:45.855 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:45.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.856 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:42:45.856 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:45.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.856 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.857 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:45.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.858 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:42:45.858 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:45.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.858 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.859 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:42:45.859 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:45.859 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.859 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.860 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:45.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.860 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:42:45.861 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:45.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.862 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.862 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:42:45.862 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:45.862 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.863 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.863 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:45.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.864 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.864 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:42:45.864 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:45.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.865 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.865 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:42:45.865 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:45.865 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.866 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.866 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:42:45.866 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:42:45.867 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:42:45.867 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:42:45.867 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:45.867 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:45.867 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:45.868 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.868 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:45.869 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:42:45.869 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:45.869 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:45.869 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:45.869 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:45.870 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:45.870 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:45.870 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:45.870 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 146, 2022-07-21T01:42:45.871 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:45.871 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:45.871 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:45.871 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.872 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.872 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:42:45.872 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:45.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.873 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.873 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:42:45.873 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:45.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.874 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.874 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:45.875 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.875 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.875 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:42:45.875 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:45.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.876 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.876 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:42:45.877 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:45.877 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.877 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.877 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:45.878 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.878 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.878 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:42:45.879 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:45.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.879 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.880 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:42:45.880 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:45.880 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.880 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.881 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:45.881 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.881 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.882 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:42:45.882 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:45.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.883 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.883 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:42:45.883 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:45.883 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.884 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.884 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:42:45.884 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:42:45.884 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:42:45.885 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:42:45.885 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:45.885 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:45.885 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:45.886 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.886 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:45.887 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:42:45.887 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:45.887 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:45.887 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:45.887 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:45.888 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:45.888 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:45.888 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:45.888 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:45.889 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:45.889 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:45.889 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:45.889 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.890 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.890 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:42:45.890 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:45.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.891 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.891 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:42:45.891 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:45.892 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.892 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.892 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:45.892 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.893 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.893 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:42:45.893 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:45.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.894 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.894 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:42:45.894 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:45.895 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.895 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.895 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:42:45.896 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.897 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:42:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.899 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:42:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.900 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:42:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:42:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:42:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:42:45.902 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:42:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:42:45.904 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:45.905 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:45.906 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.907 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:42:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.908 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:42:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.909 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.910 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:42:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:42:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.912 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:42:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.915 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.915 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:42:45.915 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:45.915 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.916 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.916 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:45.916 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.917 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.917 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:42:45.917 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:45.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.918 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.918 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:42:45.918 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:45.919 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.919 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.919 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:42:45.919 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:42:45.920 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:42:45.920 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:42:45.920 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:45.920 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:45.921 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:45.921 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.921 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:45.922 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:42:45.922 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:45.922 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:45.922 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:45.923 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:45.923 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:45.923 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:45.923 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:45.924 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:45.924 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:45.924 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:45.924 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:45.924 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.925 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.925 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:42:45.925 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:45.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.926 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.926 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:42:45.926 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:45.927 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.927 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.927 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:45.928 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.928 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.928 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:42:45.928 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:45.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.929 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.929 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:42:45.930 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:45.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.930 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.931 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:45.931 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.931 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.931 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:42:45.932 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:45.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.932 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.933 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:42:45.933 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:45.933 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.933 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.934 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:45.934 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:45.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.934 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:45.935 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:42:45.935 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:45.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.936 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:45.936 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:42:45.936 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:45.936 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.936 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.937 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:42:45.937 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:42:45.937 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:42:45.938 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:42:45.938 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:45.938 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:45.938 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:45.939 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:45.939 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.939 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:45.939 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:42:45.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.940 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:45.940 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:45.940 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:45.940 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:45.941 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:45.941 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:45.941 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:42:45.941 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:45.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.942 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:45.942 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:45.942 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:45.943 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:45.943 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:45.943 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:45.943 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:42:45.944 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:45.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.944 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:45.944 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:45.945 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:45.945 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:45.945 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:45.945 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:45.946 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:42:45.946 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:45.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.946 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:45.947 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:45.947 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:45.947 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:45.947 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:45.948 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:45.948 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:42:45.948 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:45.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.949 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:45.949 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:45.949 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:45.949 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:45.950 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:45.950 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:45.950 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:42:45.950 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:45.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.951 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:45.951 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:45.951 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:45.952 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:45.952 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:45.952 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:45.952 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:42:45.953 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:45.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.953 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:45.953 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:45.954 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:45.954 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:45.954 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:45.954 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:45.955 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:42:45.955 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:45.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:45.956 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:45.956 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:45.956 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:45.956 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:45.956 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:45.957 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:45.957 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:42:45.957 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:45.957 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.958 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:45.958 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:42:45.958 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:42:45.958 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:42:45.958 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:42:45.959 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:42:45.959 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:42:45.959 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:42:45.959 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:42:45.960 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:45.960 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:42:45.960 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:42:45.960 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:45.960 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:42:45.961 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:45.961 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:45.961 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:42:45.962 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:42:45.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.962 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:42:45.962 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:45.963 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:45.963 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:42:45.963 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:45.963 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:45.964 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:42:45.964 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:45.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:45.964 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:42:45.964 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:42:45.965 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:42:45.965 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:42:45.965 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:42:45.965 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:42:45.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3360053a10 msgr2=0x7f3360055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:45.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3360053a10 0x7f3360055ed0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f337000d000 tx=0x7f3370007a20).stop 2022-07-21T01:42:45.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800a7250 msgr2=0x7f33800a9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:45.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800a7250 0x7f33800a9910 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f3378001b80 tx=0x7f3374014000).stop 2022-07-21T01:42:45.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 shutdown_connections 2022-07-21T01:42:45.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3380148530 0x7f33800abe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:45.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33800b1f40 0x7f3380149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:45.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3360053a10 0x7f3360055ed0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:45.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 --2- 172.21.15.3:0/1928945626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33800a7250 0x7f33800a9910 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:45.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.715+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 >> 172.21.15.3:0/1928945626 conn(0x7f338005f840 msgr2=0x7f338014e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:45.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.719+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 shutdown_connections 2022-07-21T01:42:45.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:45.719+0000 7f3386772700 1 -- 172.21.15.3:0/1928945626 wait complete. 2022-07-21T01:42:45.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:46.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.079+0000 7f3b4c919700 1 Processor -- start 2022-07-21T01:42:46.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.079+0000 7f3b4c919700 1 -- start start 2022-07-21T01:42:46.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.079+0000 7f3b4c919700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b480b2710 0x7f3b4814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.079+0000 7f3b4c919700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3b480b01a0 0x7f3b4814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.091 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.079+0000 7f3b4c919700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b480a7250 con 0x7f3b480afd60 2022-07-21T01:42:46.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.079+0000 7f3b4c919700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b480a7480 con 0x7f3b480b01a0 2022-07-21T01:42:46.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.079+0000 7f3b4c919700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480a7720 con 0x7f3b480b2710 2022-07-21T01:42:46.092 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4759e700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3b480afd60 0x7f3b480a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59938/0 (socket says 172.21.15.3:59938) 2022-07-21T01:42:46.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4759e700 1 -- 172.21.15.3:0/1509842653 learned_addr learned my addr 172.21.15.3:0/1509842653 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:46.093 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b46d9d700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b480b2710 0x7f3b4814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4659c700 1 --2- 172.21.15.3:0/1509842653 >> v2:172.21.15.143:3300/0 conn(0x7f3b480b01a0 0x7f3b4814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508281186 0 0) 0x7f3b480a7250 con 0x7f3b480afd60 2022-07-21T01:42:46.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b28003260 con 0x7f3b480afd60 2022-07-21T01:42:46.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2077267278 0 0) 0x7f3b28003260 con 0x7f3b480afd60 2022-07-21T01:42:46.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b480b2710 msgr2=0x7f3b4814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:46.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b480b2710 0x7f3b4814afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 >> v2:172.21.15.143:3300/0 conn(0x7f3b480b01a0 msgr2=0x7f3b4814b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:46.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 --2- 172.21.15.3:0/1509842653 >> v2:172.21.15.143:3300/0 conn(0x7f3b480b01a0 0x7f3b4814b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b480a79c0 con 0x7f3b480afd60 2022-07-21T01:42:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f3b38001660 con 0x7f3b480afd60 2022-07-21T01:42:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 >> v1:172.21.15.3:6789/0 conn(0x7f3b480afd60 legacy=0x7f3b480a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28003750 0x7f3b28003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28004500 0x7f3b28004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b480b2710 0x7f3b28004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480a7480 con 0x7f3b28003750 2022-07-21T01:42:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480a7720 con 0x7f3b28004500 2022-07-21T01:42:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2800b750 con 0x7f3b480b2710 2022-07-21T01:42:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b46d9d700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28003750 0x7f3b28003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4659c700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28004500 0x7f3b28004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4759e700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b480b2710 0x7f3b28004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b46d9d700 1 -- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b480b2710 msgr2=0x7f3b28004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b46d9d700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b480b2710 0x7f3b28004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b46d9d700 1 -- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28004500 msgr2=0x7f3b28004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b46d9d700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28004500 0x7f3b28004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b46d9d700 1 -- 172.21.15.3:0/1509842653 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28003260 con 0x7f3b28003750 2022-07-21T01:42:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b46d9d700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28003750 0x7f3b28003b70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f3b4000c000 tx=0x7f3b400091d0).ready entity=mon.0 client_cookie=1b5fddb9f82ca37f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b4000e7e0 con 0x7f3b28003750 2022-07-21T01:42:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4000e940 con 0x7f3b28003750 2022-07-21T01:42:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b45d9b700 1 -- 172.21.15.3:0/1509842653 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b4000eca0 con 0x7f3b28003750 2022-07-21T01:42:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4c919700 1 -- 172.21.15.3:0/1509842653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28003750 msgr2=0x7f3b28003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4c919700 1 --2- 172.21.15.3:0/1509842653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28003750 0x7f3b28003b70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f3b4000c000 tx=0x7f3b400091d0).stop 2022-07-21T01:42:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4759e700 1 -- 172.21.15.3:0/1509842653 reap_dead start 2022-07-21T01:42:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4c919700 1 -- 172.21.15.3:0/1509842653 shutdown_connections 2022-07-21T01:42:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4c919700 1 -- 172.21.15.3:0/1509842653 >> 172.21.15.3:0/1509842653 conn(0x7f3b48060310 msgr2=0x7f3b480b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4c919700 1 -- 172.21.15.3:0/1509842653 shutdown_connections 2022-07-21T01:42:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.083+0000 7f3b4c919700 1 -- 172.21.15.3:0/1509842653 wait complete. 2022-07-21T01:42:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.087+0000 7f3b4c919700 1 Processor -- start 2022-07-21T01:42:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.087+0000 7f3b4c919700 1 -- start start 2022-07-21T01:42:46.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.087+0000 7f3b4c919700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28003750 0x7f3b480ae490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.087+0000 7f3b4c919700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b4815a0c0 0x7f3b4815a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.087+0000 7f3b4c919700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b4815aa30 0x7f3b48156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.087+0000 7f3b4c919700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b48156820 con 0x7f3b4815a0c0 2022-07-21T01:42:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.087+0000 7f3b4c919700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480ab630 con 0x7f3b28003750 2022-07-21T01:42:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.087+0000 7f3b4c919700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480ab0f0 con 0x7f3b4815aa30 2022-07-21T01:42:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b4815a0c0 0x7f3b4815a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b4815a0c0 0x7f3b4815a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51668/0 (socket says 172.21.15.3:51668) 2022-07-21T01:42:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b4815aa30 0x7f3b48156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b46d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28003750 0x7f3b480ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4659c700 1 -- 172.21.15.3:0/3662306251 learned_addr learned my addr 172.21.15.3:0/3662306251 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4659c700 1 -- 172.21.15.3:0/3662306251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b4815aa30 msgr2=0x7f3b48156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4659c700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b4815aa30 0x7f3b48156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4659c700 1 -- 172.21.15.3:0/3662306251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28003750 msgr2=0x7f3b480ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4659c700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28003750 0x7f3b480ae490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4659c700 1 -- 172.21.15.3:0/3662306251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b480ab390 con 0x7f3b4815a0c0 2022-07-21T01:42:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4759e700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b4815aa30 0x7f3b48156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b46d9d700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28003750 0x7f3b480ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:46.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4659c700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b4815a0c0 0x7f3b4815a540 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f3b480ab7c0 tx=0x7f3b30008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:46.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b3ffff700 1 -- 172.21.15.3:0/3662306251 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b30012610 con 0x7f3b4815a0c0 2022-07-21T01:42:46.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b3ffff700 1 -- 172.21.15.3:0/3662306251 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b30012770 con 0x7f3b4815a0c0 2022-07-21T01:42:46.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b481543d0 con 0x7f3b4815a0c0 2022-07-21T01:42:46.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b3ffff700 1 -- 172.21.15.3:0/3662306251 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b30012ad0 con 0x7f3b4815a0c0 2022-07-21T01:42:46.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b480aa3b0 con 0x7f3b4815a0c0 2022-07-21T01:42:46.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b480574e0 con 0x7f3b4815a0c0 2022-07-21T01:42:46.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.091+0000 7f3b3ffff700 1 -- 172.21.15.3:0/3662306251 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3b30012c30 con 0x7f3b4815a0c0 2022-07-21T01:42:46.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.095+0000 7f3b3ffff700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b2c0539e0 0x7f3b2c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.095+0000 7f3b3ffff700 1 -- 172.21.15.3:0/3662306251 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3b3006c7c0 con 0x7f3b4815a0c0 2022-07-21T01:42:46.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.095+0000 7f3b46d9d700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b2c0539e0 0x7f3b2c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.095+0000 7f3b46d9d700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b2c0539e0 0x7f3b2c055ea0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f3b4001c000 tx=0x7f3b4001a000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:46.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.099+0000 7f3b3ffff700 1 -- 172.21.15.3:0/3662306251 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3b30043580 con 0x7f3b4815a0c0 2022-07-21T01:42:46.263 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.259+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b4805d410 con 0x7f3b4815a0c0 2022-07-21T01:42:46.264 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.259+0000 7f3b3ffff700 1 -- 172.21.15.3:0/3662306251 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2492 (secure 0 0 0) 0x7f3b30046ba0 con 0x7f3b4815a0c0 2022-07-21T01:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:46.265 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/1563084587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1563084587},{"type":"v1","addr":"172.21.15.3:6835","nonce":1563084587}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1403026914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1403026914},{"type":"v1","addr":"172.21.15.3:6837","nonce":1403026914}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:34.642400+0000","modified":"2022-07-21T01:42:35.646272+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8409},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8409":{"gid":8409,"name":"b","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6833/1158093851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":1158093851},{"type":"v1","addr":"172.21.15.143:6833","nonce":1158093851}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-21T01:42:46.268 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b2c0539e0 msgr2=0x7f3b2c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b2c0539e0 0x7f3b2c055ea0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f3b4001c000 tx=0x7f3b4001a000).stop 2022-07-21T01:42:46.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b4815a0c0 msgr2=0x7f3b4815a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.269 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b4815a0c0 0x7f3b4815a540 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f3b480ab7c0 tx=0x7f3b30008040).stop 2022-07-21T01:42:46.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 shutdown_connections 2022-07-21T01:42:46.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b4815aa30 0x7f3b48156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b4815a0c0 0x7f3b4815a540 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3b2c0539e0 0x7f3b2c055ea0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 --2- 172.21.15.3:0/3662306251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b28003750 0x7f3b480ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 >> 172.21.15.3:0/3662306251 conn(0x7f3b48060310 msgr2=0x7f3b48148bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 shutdown_connections 2022-07-21T01:42:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.263+0000 7f3b4c919700 1 -- 172.21.15.3:0/3662306251 wait complete. 2022-07-21T01:42:46.273 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 76 2022-07-21T01:42:46.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:42:46.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.379+0000 7fd1d3fff700 1 Processor -- start 2022-07-21T01:42:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.379+0000 7fd1d3fff700 1 -- start start 2022-07-21T01:42:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1d40b07a0 0x7fd1d40af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd1d4147730 0x7fd1d40acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d4147b80 con 0x7fd1d40b5630 2022-07-21T01:42:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d4145470 con 0x7fd1d40b07a0 2022-07-21T01:42:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d40ae180 con 0x7fd1d4147730 2022-07-21T01:42:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d27fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1d40b07a0 0x7fd1d40af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d2ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd1d40b5630 0x7fd1d4067730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59954/0 (socket says 172.21.15.3:59954) 2022-07-21T01:42:46.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d27fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd1d40b07a0 0x7fd1d40af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51674/0 (socket says 172.21.15.3:51674) 2022-07-21T01:42:46.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d27fc700 1 -- 172.21.15.3:0/4220752380 learned_addr learned my addr 172.21.15.3:0/4220752380 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:46.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d37fe700 1 --2- 172.21.15.3:0/4220752380 >> v2:172.21.15.143:3300/0 conn(0x7fd1d4147730 0x7fd1d40acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d27fc700 1 -- 172.21.15.3:0/4220752380 >> v2:172.21.15.143:3300/0 conn(0x7fd1d4147730 msgr2=0x7fd1d40acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d27fc700 1 --2- 172.21.15.3:0/4220752380 >> v2:172.21.15.143:3300/0 conn(0x7fd1d4147730 0x7fd1d40acdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d27fc700 1 -- 172.21.15.3:0/4220752380 >> v1:172.21.15.3:6789/0 conn(0x7fd1d40b5630 legacy=0x7fd1d4067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d27fc700 1 -- 172.21.15.3:0/4220752380 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d40b5a10 con 0x7fd1d40b07a0 2022-07-21T01:42:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d27fc700 1 --2- 172.21.15.3:0/4220752380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b07a0 0x7fd1d40af6e0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7fd1c40032f0 tx=0x7fd1c4009290).ready entity=mon.0 client_cookie=5b42b20482612702 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d1ffb700 1 -- 172.21.15.3:0/4220752380 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1c400b870 con 0x7fd1d40b07a0 2022-07-21T01:42:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d1ffb700 1 -- 172.21.15.3:0/4220752380 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c400b9d0 con 0x7fd1d40b07a0 2022-07-21T01:42:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d1ffb700 1 -- 172.21.15.3:0/4220752380 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1c400bb40 con 0x7fd1d40b07a0 2022-07-21T01:42:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4220752380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b07a0 msgr2=0x7fd1d40af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 --2- 172.21.15.3:0/4220752380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b07a0 0x7fd1d40af6e0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7fd1c40032f0 tx=0x7fd1c4009290).stop 2022-07-21T01:42:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4220752380 shutdown_connections 2022-07-21T01:42:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 --2- 172.21.15.3:0/4220752380 >> v2:172.21.15.143:3300/0 conn(0x7fd1d4147730 0x7fd1d40acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 --2- 172.21.15.3:0/4220752380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b07a0 0x7fd1d40af6e0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4220752380 >> 172.21.15.3:0/4220752380 conn(0x7fd1d40626f0 msgr2=0x7fd1d4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4220752380 shutdown_connections 2022-07-21T01:42:46.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4220752380 wait complete. 2022-07-21T01:42:46.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 Processor -- start 2022-07-21T01:42:46.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.383+0000 7fd1d3fff700 1 -- start start 2022-07-21T01:42:46.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1d40b07a0 0x7fd1d4149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b5630 0x7fd1d41576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1d4147730 0x7fd1d4063e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d3fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d4065e00 con 0x7fd1d40b5630 2022-07-21T01:42:46.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d3fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d4063780 con 0x7fd1d40b07a0 2022-07-21T01:42:46.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d3fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d40b5120 con 0x7fd1d4147730 2022-07-21T01:42:46.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b5630 0x7fd1d41576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d27fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b5630 0x7fd1d41576f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51682/0 (socket says 172.21.15.3:51682) 2022-07-21T01:42:46.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d27fc700 1 -- 172.21.15.3:0/4236839281 learned_addr learned my addr 172.21.15.3:0/4236839281 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:46.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d2ffd700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1d40b07a0 0x7fd1d4149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d37fe700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1d4147730 0x7fd1d4063e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d27fc700 1 -- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1d4147730 msgr2=0x7fd1d4063e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d27fc700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1d4147730 0x7fd1d4063e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d27fc700 1 -- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1d40b07a0 msgr2=0x7fd1d4149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d27fc700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1d40b07a0 0x7fd1d4149c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d27fc700 1 -- 172.21.15.3:0/4236839281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c4009530 con 0x7fd1d40b5630 2022-07-21T01:42:46.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d2ffd700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1d40b07a0 0x7fd1d4149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:46.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d37fe700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1d4147730 0x7fd1d4063e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:46.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d27fc700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b5630 0x7fd1d41576f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7fd1d40b52b0 tx=0x7fd1c4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:46.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1c3fff700 1 -- 172.21.15.3:0/4236839281 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1c400bb40 con 0x7fd1d40b5630 2022-07-21T01:42:46.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1c3fff700 1 -- 172.21.15.3:0/4236839281 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c401f3f0 con 0x7fd1d40b5630 2022-07-21T01:42:46.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d40b4e80 con 0x7fd1d40b5630 2022-07-21T01:42:46.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1c3fff700 1 -- 172.21.15.3:0/4236839281 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1c401f750 con 0x7fd1d40b5630 2022-07-21T01:42:46.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d4157be0 con 0x7fd1d40b5630 2022-07-21T01:42:46.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.387+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d40b4c40 con 0x7fd1d40b5630 2022-07-21T01:42:46.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.391+0000 7fd1c3fff700 1 -- 172.21.15.3:0/4236839281 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd1c4026030 con 0x7fd1d40b5630 2022-07-21T01:42:46.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.391+0000 7fd1c3fff700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd1b0053b00 0x7fd1b0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.391+0000 7fd1c3fff700 1 -- 172.21.15.3:0/4236839281 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd1c406d080 con 0x7fd1d40b5630 2022-07-21T01:42:46.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.391+0000 7fd1d2ffd700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd1b0053b00 0x7fd1b0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.391+0000 7fd1d2ffd700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd1b0053b00 0x7fd1b0055fc0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7fd1c800b000 tx=0x7fd1c80070e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:46.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.395+0000 7fd1c3fff700 1 -- 172.21.15.3:0/4236839281 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd1c4047760 con 0x7fd1d40b5630 2022-07-21T01:42:46.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.555+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd1d40574e0 con 0x7fd1d40b5630 2022-07-21T01:42:46.777 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:42:46.771+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:42:46.792 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.787+0000 7fd1c3fff700 1 -- 172.21.15.3:0/4236839281 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v77) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd1c4024070 con 0x7fd1d40b5630 2022-07-21T01:42:46.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd1b0053b00 msgr2=0x7fd1b0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd1b0053b00 0x7fd1b0055fc0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7fd1c800b000 tx=0x7fd1c80070e0).stop 2022-07-21T01:42:46.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b5630 msgr2=0x7fd1d41576f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b5630 0x7fd1d41576f0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7fd1d40b52b0 tx=0x7fd1c4016000).stop 2022-07-21T01:42:46.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 shutdown_connections 2022-07-21T01:42:46.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1d4147730 0x7fd1d4063e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1d40b5630 0x7fd1d41576f0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd1b0053b00 0x7fd1b0055fc0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 --2- 172.21.15.3:0/4236839281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1d40b07a0 0x7fd1d4149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 >> 172.21.15.3:0/4236839281 conn(0x7fd1d40626f0 msgr2=0x7fd1d4067860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:46.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 shutdown_connections 2022-07-21T01:42:46.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.791+0000 7fd1d3fff700 1 -- 172.21.15.3:0/4236839281 wait complete. 2022-07-21T01:42:46.800 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:42:46.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:46.818 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:42:46.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188ffff700 1 Processor -- start 2022-07-21T01:42:46.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188ffff700 1 -- start start 2022-07-21T01:42:46.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18900b28b0 0x7f189013e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188ffff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f18900b1180 0x7f189013e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188ffff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18900aac20 con 0x7f1890141210 2022-07-21T01:42:46.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188ffff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1890141db0 con 0x7f18900b1180 2022-07-21T01:42:46.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1890140e20 con 0x7f18900b28b0 2022-07-21T01:42:46.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188e7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f18900b1180 0x7f189013e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188e7fc700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f18900b1180 0x7f189013e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51688/0 (socket says 172.21.15.3:51688) 2022-07-21T01:42:46.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188e7fc700 1 -- 172.21.15.3:0/2696578762 learned_addr learned my addr 172.21.15.3:0/2696578762 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:46.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.919+0000 7f188effd700 1 --2- 172.21.15.3:0/2696578762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18900b28b0 0x7f189013e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188e7fc700 1 -- 172.21.15.3:0/2696578762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18900b28b0 msgr2=0x7f189013e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:46.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188e7fc700 1 --2- 172.21.15.3:0/2696578762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18900b28b0 0x7f189013e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188e7fc700 1 -- 172.21.15.3:0/2696578762 >> v1:172.21.15.143:6789/0 conn(0x7f1890141210 legacy=0x7f18901416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188e7fc700 1 -- 172.21.15.3:0/2696578762 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18900b2c90 con 0x7f18900b1180 2022-07-21T01:42:46.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188e7fc700 1 --2- 172.21.15.3:0/2696578762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b1180 0x7f189013e6b0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7f18800032f0 tx=0x7f1880009290).ready entity=mon.0 client_cookie=1b818a933d5be701 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:46.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188dffb700 1 -- 172.21.15.3:0/2696578762 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f188000b8a0 con 0x7f18900b1180 2022-07-21T01:42:46.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188dffb700 1 -- 172.21.15.3:0/2696578762 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188000ba00 con 0x7f18900b1180 2022-07-21T01:42:46.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188dffb700 1 -- 172.21.15.3:0/2696578762 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f188000bb70 con 0x7f18900b1180 2022-07-21T01:42:46.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 -- 172.21.15.3:0/2696578762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b1180 msgr2=0x7f189013e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 --2- 172.21.15.3:0/2696578762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b1180 0x7f189013e6b0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7f18800032f0 tx=0x7f1880009290).stop 2022-07-21T01:42:46.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 -- 172.21.15.3:0/2696578762 shutdown_connections 2022-07-21T01:42:46.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 --2- 172.21.15.3:0/2696578762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b1180 0x7f189013e6b0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 --2- 172.21.15.3:0/2696578762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18900b28b0 0x7f189013e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 -- 172.21.15.3:0/2696578762 >> 172.21.15.3:0/2696578762 conn(0x7f1890060310 msgr2=0x7f18900b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:46.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 -- 172.21.15.3:0/2696578762 shutdown_connections 2022-07-21T01:42:46.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 -- 172.21.15.3:0/2696578762 wait complete. 2022-07-21T01:42:46.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 Processor -- start 2022-07-21T01:42:46.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.923+0000 7f188ffff700 1 -- start start 2022-07-21T01:42:46.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18900b1180 0x7f1890144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b28b0 0x7f1890143dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1890141210 0x7f18901442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188ffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1890142310 con 0x7f18900b28b0 2022-07-21T01:42:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18900a87b0 con 0x7f18900b1180 2022-07-21T01:42:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18900a8510 con 0x7f1890141210 2022-07-21T01:42:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b28b0 0x7f1890143dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b28b0 0x7f1890143dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51696/0 (socket says 172.21.15.3:51696) 2022-07-21T01:42:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188e7fc700 1 -- 172.21.15.3:0/3809814615 learned_addr learned my addr 172.21.15.3:0/3809814615 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188f7fe700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1890141210 0x7f18901442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188effd700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18900b1180 0x7f1890144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188e7fc700 1 -- 172.21.15.3:0/3809814615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1890141210 msgr2=0x7f18901442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188e7fc700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1890141210 0x7f18901442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188e7fc700 1 -- 172.21.15.3:0/3809814615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18900b1180 msgr2=0x7f1890144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:46.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188e7fc700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18900b1180 0x7f1890144e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:46.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188e7fc700 1 -- 172.21.15.3:0/3809814615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1880009530 con 0x7f18900b28b0 2022-07-21T01:42:46.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188f7fe700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1890141210 0x7f18901442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:46.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188e7fc700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b28b0 0x7f1890143dd0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f18800032f0 tx=0x7f1880016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:46.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f1887fff700 1 -- 172.21.15.3:0/3809814615 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1880021630 con 0x7f18900b28b0 2022-07-21T01:42:46.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f1887fff700 1 -- 172.21.15.3:0/3809814615 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1880021790 con 0x7f18900b28b0 2022-07-21T01:42:46.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1890142570 con 0x7f18900b28b0 2022-07-21T01:42:46.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f1887fff700 1 -- 172.21.15.3:0/3809814615 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1880021af0 con 0x7f18900b28b0 2022-07-21T01:42:46.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1890142a10 con 0x7f18900b28b0 2022-07-21T01:42:46.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.927+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1890142700 con 0x7f18900b28b0 2022-07-21T01:42:46.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.931+0000 7f1887fff700 1 -- 172.21.15.3:0/3809814615 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1880027020 con 0x7f18900b28b0 2022-07-21T01:42:46.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.931+0000 7f1887fff700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f18740539e0 0x7f1874055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:46.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.931+0000 7f1887fff700 1 -- 172.21.15.3:0/3809814615 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f188006de50 con 0x7f18900b28b0 2022-07-21T01:42:46.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.931+0000 7f188effd700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f18740539e0 0x7f1874055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:46.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.931+0000 7f188effd700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f18740539e0 0x7f1874055ea0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f187800c000 tx=0x7f1878008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:46.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:46.939+0000 7f1887fff700 1 -- 172.21.15.3:0/3809814615 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1880044bc0 con 0x7f18900b28b0 2022-07-21T01:42:47.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.107+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18900574e0 con 0x7f18900b28b0 2022-07-21T01:42:47.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.107+0000 7f1887fff700 1 -- 172.21.15.3:0/3809814615 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2129 (secure 0 0 0) 0x7f1880015030 con 0x7f18900b28b0 2022-07-21T01:42:47.112 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:47.112 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/1563084587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1563084587},{"type":"v1","addr":"172.21.15.3:6835","nonce":1563084587}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1403026914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1403026914},{"type":"v1","addr":"172.21.15.3:6837","nonce":1403026914}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:34.642400+0000","modified":"2022-07-21T01:42:46.776986+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":151,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-21T01:42:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f18740539e0 msgr2=0x7f1874055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f18740539e0 0x7f1874055ea0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f187800c000 tx=0x7f1878008040).stop 2022-07-21T01:42:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b28b0 msgr2=0x7f1890143dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b28b0 0x7f1890143dd0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f18800032f0 tx=0x7f1880016000).stop 2022-07-21T01:42:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 shutdown_connections 2022-07-21T01:42:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1890141210 0x7f18901442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18900b28b0 0x7f1890143dd0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f18740539e0 0x7f1874055ea0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 --2- 172.21.15.3:0/3809814615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18900b1180 0x7f1890144e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 >> 172.21.15.3:0/3809814615 conn(0x7f1890060310 msgr2=0x7f18901404c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 shutdown_connections 2022-07-21T01:42:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.111+0000 7f188ffff700 1 -- 172.21.15.3:0/3809814615 wait complete. 2022-07-21T01:42:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 77 2022-07-21T01:42:47.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:42:47.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.231+0000 7f88ef59e700 1 Processor -- start 2022-07-21T01:42:47.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.231+0000 7f88ef59e700 1 -- start start 2022-07-21T01:42:47.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.231+0000 7f88ef59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f88f0061d10 0x7f88f0144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.231+0000 7f88ef59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88f00b41f0 0x7f88f0144b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.231+0000 7f88ef59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f00b2100 con 0x7f88f00b65c0 2022-07-21T01:42:47.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.231+0000 7f88ef59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f00b1630 con 0x7f88f0061d10 2022-07-21T01:42:47.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.231+0000 7f88ef59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f00b69a0 con 0x7f88f00b41f0 2022-07-21T01:42:47.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.231+0000 7f88edd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f88f0061d10 0x7f88f0144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ee59c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f88f00b65c0 0x7f88f0061600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59982/0 (socket says 172.21.15.3:59982) 2022-07-21T01:42:47.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ee59c700 1 -- 172.21.15.3:0/2487127456 learned_addr learned my addr 172.21.15.3:0/2487127456 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:47.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88eed9d700 1 --2- 172.21.15.3:0/2487127456 >> v2:172.21.15.143:3300/0 conn(0x7f88f00b41f0 0x7f88f0144b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ed59a700 1 -- 172.21.15.3:0/2487127456 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702099980 0 0) 0x7f88f00b2100 con 0x7f88f00b65c0 2022-07-21T01:42:47.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ed59a700 1 -- 172.21.15.3:0/2487127456 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d4003260 con 0x7f88f00b65c0 2022-07-21T01:42:47.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88edd9b700 1 -- 172.21.15.3:0/2487127456 >> v2:172.21.15.143:3300/0 conn(0x7f88f00b41f0 msgr2=0x7f88f0144b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:47.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88edd9b700 1 --2- 172.21.15.3:0/2487127456 >> v2:172.21.15.143:3300/0 conn(0x7f88f00b41f0 0x7f88f0144b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88edd9b700 1 -- 172.21.15.3:0/2487127456 >> v1:172.21.15.3:6789/0 conn(0x7f88f00b65c0 legacy=0x7f88f0061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88edd9b700 1 -- 172.21.15.3:0/2487127456 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f01482a0 con 0x7f88f0061d10 2022-07-21T01:42:47.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88edd9b700 1 --2- 172.21.15.3:0/2487127456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f0061d10 0x7f88f0144630 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f88d8002f70 tx=0x7f88d8006df0).ready entity=mon.0 client_cookie=4cf40a199fb74d67 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:47.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ed59a700 1 -- 172.21.15.3:0/2487127456 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d800d4f0 con 0x7f88f0061d10 2022-07-21T01:42:47.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ed59a700 1 -- 172.21.15.3:0/2487127456 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d8000e90 con 0x7f88f0061d10 2022-07-21T01:42:47.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ed59a700 1 -- 172.21.15.3:0/2487127456 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d800d840 con 0x7f88f0061d10 2022-07-21T01:42:47.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 -- 172.21.15.3:0/2487127456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f0061d10 msgr2=0x7f88f0144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 --2- 172.21.15.3:0/2487127456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f0061d10 0x7f88f0144630 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f88d8002f70 tx=0x7f88d8006df0).stop 2022-07-21T01:42:47.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 -- 172.21.15.3:0/2487127456 shutdown_connections 2022-07-21T01:42:47.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 --2- 172.21.15.3:0/2487127456 >> v2:172.21.15.143:3300/0 conn(0x7f88f00b41f0 0x7f88f0144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 --2- 172.21.15.3:0/2487127456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f0061d10 0x7f88f0144630 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 -- 172.21.15.3:0/2487127456 >> 172.21.15.3:0/2487127456 conn(0x7f88f005c620 msgr2=0x7f88f005ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:47.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 -- 172.21.15.3:0/2487127456 shutdown_connections 2022-07-21T01:42:47.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 -- 172.21.15.3:0/2487127456 wait complete. 2022-07-21T01:42:47.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 Processor -- start 2022-07-21T01:42:47.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 -- start start 2022-07-21T01:42:47.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88f0061d10 0x7f88f00b8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f00b41f0 0x7f88f014e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88f00b65c0 0x7f88f014e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00aef30 con 0x7f88f00b41f0 2022-07-21T01:42:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00af1d0 con 0x7f88f00b65c0 2022-07-21T01:42:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ef59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00af470 con 0x7f88f0061d10 2022-07-21T01:42:47.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88edd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f00b41f0 0x7f88f014e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88edd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f00b41f0 0x7f88f014e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51710/0 (socket says 172.21.15.3:51710) 2022-07-21T01:42:47.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88edd9b700 1 -- 172.21.15.3:0/2619463364 learned_addr learned my addr 172.21.15.3:0/2619463364 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:47.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88eed9d700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88f00b65c0 0x7f88f014e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.235+0000 7f88ee59c700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88f0061d10 0x7f88f00b8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88edd9b700 1 -- 172.21.15.3:0/2619463364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88f0061d10 msgr2=0x7f88f00b8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88edd9b700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88f0061d10 0x7f88f00b8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88edd9b700 1 -- 172.21.15.3:0/2619463364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88f00b65c0 msgr2=0x7f88f014e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88edd9b700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88f00b65c0 0x7f88f014e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88edd9b700 1 -- 172.21.15.3:0/2619463364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d800b180 con 0x7f88f00b41f0 2022-07-21T01:42:47.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88edd9b700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f00b41f0 0x7f88f014e2a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f88d8002f70 tx=0x7f88d8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:47.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88e77fe700 1 -- 172.21.15.3:0/2619463364 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d8013de0 con 0x7f88f00b41f0 2022-07-21T01:42:47.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88e77fe700 1 -- 172.21.15.3:0/2619463364 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d800de60 con 0x7f88f00b41f0 2022-07-21T01:42:47.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f014ec80 con 0x7f88f00b41f0 2022-07-21T01:42:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88e77fe700 1 -- 172.21.15.3:0/2619463364 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d801e5f0 con 0x7f88f00b41f0 2022-07-21T01:42:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f00b0200 con 0x7f88f00b41f0 2022-07-21T01:42:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f00af630 con 0x7f88f00b41f0 2022-07-21T01:42:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88e77fe700 1 -- 172.21.15.3:0/2619463364 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f88d8025030 con 0x7f88f00b41f0 2022-07-21T01:42:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88e77fe700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053950 0x7f88c8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88e77fe700 1 -- 172.21.15.3:0/2619463364 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f88d806cc80 con 0x7f88f00b41f0 2022-07-21T01:42:47.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.239+0000 7f88ee59c700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053950 0x7f88c8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.243+0000 7f88ee59c700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053950 0x7f88c8055e10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f88e800d000 tx=0x7f88e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:47.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.247+0000 7f88e77fe700 1 -- 172.21.15.3:0/2619463364 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f88d8072020 con 0x7f88f00b41f0 2022-07-21T01:42:47.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.411+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f88f00b2330 con 0x7f88f00b41f0 2022-07-21T01:42:47.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.791+0000 7f88e77fe700 1 -- 172.21.15.3:0/2619463364 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v78) v1 ==== 104+0+0 (secure 0 0 0) 0x7f88d80230a0 con 0x7f88f00b41f0 2022-07-21T01:42:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053950 msgr2=0x7f88c8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053950 0x7f88c8055e10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f88e800d000 tx=0x7f88e8009040).stop 2022-07-21T01:42:47.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f00b41f0 msgr2=0x7f88f014e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f00b41f0 0x7f88f014e2a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f88d8002f70 tx=0x7f88d8014000).stop 2022-07-21T01:42:47.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 shutdown_connections 2022-07-21T01:42:47.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c8053950 0x7f88c8055e10 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88f00b65c0 0x7f88f014e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f00b41f0 0x7f88f014e2a0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 --2- 172.21.15.3:0/2619463364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88f0061d10 0x7f88f00b8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 >> 172.21.15.3:0/2619463364 conn(0x7f88f005c620 msgr2=0x7f88f00b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 shutdown_connections 2022-07-21T01:42:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.795+0000 7f88ef59e700 1 -- 172.21.15.3:0/2619463364 wait complete. 2022-07-21T01:42:47.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:42:47.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd1a1161700 1 Processor -- start 2022-07-21T01:42:47.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd1a1161700 1 -- start start 2022-07-21T01:42:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd1a1161700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd19c0b28b0 0x7fd19c13e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd1a1161700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd19c0b1180 0x7fd19c13e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd1a1161700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 0x7fd19c1416c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd1a1161700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd19c140e20 con 0x7fd19c0b28b0 2022-07-21T01:42:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd1a1161700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd19c0aac20 con 0x7fd19c0b1180 2022-07-21T01:42:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd1a1161700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c0b2c90 con 0x7fd19c141210 2022-07-21T01:42:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd19bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd19c0b28b0 0x7fd19c13e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd19bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fd19c0b28b0 0x7fd19c13e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51716/0 (socket says 172.21.15.3:51716) 2022-07-21T01:42:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.923+0000 7fd19bfff700 1 -- 172.21.15.3:0/1174593052 learned_addr learned my addr 172.21.15.3:0/1174593052 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a0960700 1 --2- 172.21.15.3:0/1174593052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 0x7fd19c1416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd193fff700 1 --2- 172.21.15.3:0/1174593052 >> v2:172.21.15.143:3300/0 conn(0x7fd19c0b1180 0x7fd19c13e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd19bfff700 1 -- 172.21.15.3:0/1174593052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 msgr2=0x7fd19c1416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd19bfff700 1 --2- 172.21.15.3:0/1174593052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 0x7fd19c1416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd19bfff700 1 -- 172.21.15.3:0/1174593052 >> v2:172.21.15.143:3300/0 conn(0x7fd19c0b1180 msgr2=0x7fd19c13e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd19bfff700 1 --2- 172.21.15.3:0/1174593052 >> v2:172.21.15.143:3300/0 conn(0x7fd19c0b1180 0x7fd19c13e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd19bfff700 1 -- 172.21.15.3:0/1174593052 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c0aa440 con 0x7fd19c0b28b0 2022-07-21T01:42:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd193fff700 1 --2- 172.21.15.3:0/1174593052 >> v2:172.21.15.143:3300/0 conn(0x7fd19c0b1180 0x7fd19c13e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd19bfff700 1 --2- 172.21.15.3:0/1174593052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b28b0 0x7fd19c13e1c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7fd184002f70 tx=0x7fd184006eb0).ready entity=mon.0 client_cookie=8b6ebe51f9695445 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:47.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd19b7fe700 1 -- 172.21.15.3:0/1174593052 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18400d570 con 0x7fd19c0b28b0 2022-07-21T01:42:47.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd19b7fe700 1 -- 172.21.15.3:0/1174593052 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd184000e90 con 0x7fd19c0b28b0 2022-07-21T01:42:47.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd19b7fe700 1 -- 172.21.15.3:0/1174593052 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18400d900 con 0x7fd19c0b28b0 2022-07-21T01:42:47.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 -- 172.21.15.3:0/1174593052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b28b0 msgr2=0x7fd19c13e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 --2- 172.21.15.3:0/1174593052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b28b0 0x7fd19c13e1c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7fd184002f70 tx=0x7fd184006eb0).stop 2022-07-21T01:42:47.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 -- 172.21.15.3:0/1174593052 shutdown_connections 2022-07-21T01:42:47.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 --2- 172.21.15.3:0/1174593052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 0x7fd19c1416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 --2- 172.21.15.3:0/1174593052 >> v2:172.21.15.143:3300/0 conn(0x7fd19c0b1180 0x7fd19c13e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 --2- 172.21.15.3:0/1174593052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b28b0 0x7fd19c13e1c0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 -- 172.21.15.3:0/1174593052 >> 172.21.15.3:0/1174593052 conn(0x7fd19c060310 msgr2=0x7fd19c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 -- 172.21.15.3:0/1174593052 shutdown_connections 2022-07-21T01:42:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 -- 172.21.15.3:0/1174593052 wait complete. 2022-07-21T01:42:47.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 Processor -- start 2022-07-21T01:42:47.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.927+0000 7fd1a1161700 1 -- start start 2022-07-21T01:42:47.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a1161700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b1180 0x7fd19c0b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.943 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a1161700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd19c0b28b0 0x7fd19c142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a1161700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 0x7fd19c0ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a1161700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c1431e0 con 0x7fd19c0b1180 2022-07-21T01:42:47.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a1161700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c0ae7d0 con 0x7fd19c0b28b0 2022-07-21T01:42:47.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a1161700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c143c90 con 0x7fd19c141210 2022-07-21T01:42:47.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd19bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b1180 0x7fd19c0b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.946 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd19bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b1180 0x7fd19c0b4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51722/0 (socket says 172.21.15.3:51722) 2022-07-21T01:42:47.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd19bfff700 1 -- 172.21.15.3:0/2133088580 learned_addr learned my addr 172.21.15.3:0/2133088580 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:47.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd193fff700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd19c0b28b0 0x7fd19c142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a0960700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 0x7fd19c0ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd19bfff700 1 -- 172.21.15.3:0/2133088580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 msgr2=0x7fd19c0ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd19bfff700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 0x7fd19c0ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd19bfff700 1 -- 172.21.15.3:0/2133088580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd19c0b28b0 msgr2=0x7fd19c142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:47.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd19bfff700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd19c0b28b0 0x7fd19c142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:47.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd19bfff700 1 -- 172.21.15.3:0/2133088580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c00b180 con 0x7fd19c0b1180 2022-07-21T01:42:47.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd19bfff700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b1180 0x7fd19c0b4530 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7fd184002f70 tx=0x7fd184015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:47.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd199ffb700 1 -- 172.21.15.3:0/2133088580 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd184011760 con 0x7fd19c0b1180 2022-07-21T01:42:47.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd199ffb700 1 -- 172.21.15.3:0/2133088580 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1840118c0 con 0x7fd19c0b1180 2022-07-21T01:42:47.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18400b1e0 con 0x7fd19c0b1180 2022-07-21T01:42:47.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd199ffb700 1 -- 172.21.15.3:0/2133088580 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd184011c20 con 0x7fd19c0b1180 2022-07-21T01:42:47.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19c0b34b0 con 0x7fd19c0b1180 2022-07-21T01:42:47.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.931+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19c0574e0 con 0x7fd19c0b1180 2022-07-21T01:42:47.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.935+0000 7fd199ffb700 1 -- 172.21.15.3:0/2133088580 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd184025030 con 0x7fd19c0b1180 2022-07-21T01:42:47.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.935+0000 7fd199ffb700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd174053a10 0x7fd174055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:47.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.935+0000 7fd199ffb700 1 -- 172.21.15.3:0/2133088580 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fd18406d1e0 con 0x7fd19c0b1180 2022-07-21T01:42:47.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.935+0000 7fd193fff700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd174053a10 0x7fd174055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:47.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.935+0000 7fd193fff700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd174053a10 0x7fd174055ed0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fd18c00b000 tx=0x7fd18c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:47.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:47.939+0000 7fd199ffb700 1 -- 172.21.15.3:0/2133088580 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd184043e40 con 0x7fd19c0b1180 2022-07-21T01:42:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.067+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd19c069a70 con 0x7fd19c0b1180 2022-07-21T01:42:48.856 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.855+0000 7fd199ffb700 1 -- 172.21.15.3:0/2133088580 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v152) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd184023020 con 0x7fd19c0b1180 2022-07-21T01:42:48.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd174053a10 msgr2=0x7fd174055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:48.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd174053a10 0x7fd174055ed0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fd18c00b000 tx=0x7fd18c006df0).stop 2022-07-21T01:42:48.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b1180 msgr2=0x7fd19c0b4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:48.863 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b1180 0x7fd19c0b4530 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7fd184002f70 tx=0x7fd184015000).stop 2022-07-21T01:42:48.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 shutdown_connections 2022-07-21T01:42:48.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19c141210 0x7fd19c0ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:48.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd174053a10 0x7fd174055ed0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:48.864 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd19c0b28b0 0x7fd19c142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:48.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 --2- 172.21.15.3:0/2133088580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c0b1180 0x7fd19c0b4530 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:48.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 >> 172.21.15.3:0/2133088580 conn(0x7fd19c060310 msgr2=0x7fd19c13e320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:48.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 shutdown_connections 2022-07-21T01:42:48.865 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.859+0000 7fd1a1161700 1 -- 172.21.15.3:0/2133088580 wait complete. 2022-07-21T01:42:48.866 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:42:48.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:42:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.987+0000 7f0e2bfff700 1 Processor -- start 2022-07-21T01:42:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.987+0000 7f0e2bfff700 1 -- start start 2022-07-21T01:42:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0e2c0614b0 0x7f0e2c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c0aee80 con 0x7f0e2c0ad750 2022-07-21T01:42:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c0b70b0 con 0x7f0e2c1405d0 2022-07-21T01:42:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2c0af0b0 con 0x7f0e2c0614b0 2022-07-21T01:42:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2affd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0e2c0614b0 0x7f0e2c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2a7fc700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0e2c0ad750 0x7f0e2c0b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60012/0 (socket says 172.21.15.3:60012) 2022-07-21T01:42:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2a7fc700 1 -- 172.21.15.3:0/1583221448 learned_addr learned my addr 172.21.15.3:0/1583221448 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2affd700 1 -- 172.21.15.3:0/1583221448 >> v1:172.21.15.143:6789/0 conn(0x7f0e2c1405d0 legacy=0x7f0e2c1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2affd700 1 -- 172.21.15.3:0/1583221448 >> v1:172.21.15.3:6789/0 conn(0x7f0e2c0ad750 legacy=0x7f0e2c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e29ffb700 1 -- 172.21.15.3:0/1583221448 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923760265 0 0) 0x7f0e2c0aee80 con 0x7f0e2c0ad750 2022-07-21T01:42:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2affd700 1 -- 172.21.15.3:0/1583221448 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c0af350 con 0x7f0e2c0614b0 2022-07-21T01:42:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2affd700 1 --2- 172.21.15.3:0/1583221448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 0x7f0e2c061890 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f0e2c0aed30 tx=0x7f0e1c006df0).ready entity=mon.0 client_cookie=8095cf5bd46c1263 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e29ffb700 1 -- 172.21.15.3:0/1583221448 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e1c00d610 con 0x7f0e2c0614b0 2022-07-21T01:42:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e29ffb700 1 -- 172.21.15.3:0/1583221448 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1c000e90 con 0x7f0e2c0614b0 2022-07-21T01:42:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e29ffb700 1 -- 172.21.15.3:0/1583221448 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e1c00d960 con 0x7f0e2c0614b0 2022-07-21T01:42:49.002 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1583221448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 msgr2=0x7f0e2c061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:49.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 --2- 172.21.15.3:0/1583221448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 0x7f0e2c061890 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f0e2c0aed30 tx=0x7f0e1c006df0).stop 2022-07-21T01:42:49.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1583221448 shutdown_connections 2022-07-21T01:42:49.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 --2- 172.21.15.3:0/1583221448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 0x7f0e2c061890 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:49.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1583221448 >> 172.21.15.3:0/1583221448 conn(0x7f0e2c066f70 msgr2=0x7f0e2c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:49.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1583221448 shutdown_connections 2022-07-21T01:42:49.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.991+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1583221448 wait complete. 2022-07-21T01:42:49.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2bfff700 1 Processor -- start 2022-07-21T01:42:49.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2bfff700 1 -- start start 2022-07-21T01:42:49.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 0x7f0e2c148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:49.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c0ad750 0x7f0e2c156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:49.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e2c1405d0 0x7f0e2c1425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:49.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c155cf0 con 0x7f0e2c0614b0 2022-07-21T01:42:49.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c156b50 con 0x7f0e2c0ad750 2022-07-21T01:42:49.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c155f20 con 0x7f0e2c1405d0 2022-07-21T01:42:49.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 0x7f0e2c148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:49.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 0x7f0e2c148270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51736/0 (socket says 172.21.15.3:51736) 2022-07-21T01:42:49.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2affd700 1 -- 172.21.15.3:0/1296893300 learned_addr learned my addr 172.21.15.3:0/1296893300 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:49.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2a7fc700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c0ad750 0x7f0e2c156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:49.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2b7fe700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e2c1405d0 0x7f0e2c1425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2affd700 1 -- 172.21.15.3:0/1296893300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e2c1405d0 msgr2=0x7f0e2c1425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2affd700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e2c1405d0 0x7f0e2c1425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2affd700 1 -- 172.21.15.3:0/1296893300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c0ad750 msgr2=0x7f0e2c156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2affd700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c0ad750 0x7f0e2c156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2affd700 1 -- 172.21.15.3:0/1296893300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c00b180 con 0x7f0e2c0614b0 2022-07-21T01:42:49.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2affd700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 0x7f0e2c148270 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7f0e2c0aed30 tx=0x7f0e1c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:49.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e1bfff700 1 -- 172.21.15.3:0/1296893300 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e1c00de90 con 0x7f0e2c0614b0 2022-07-21T01:42:49.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e1bfff700 1 -- 172.21.15.3:0/1296893300 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1c002f70 con 0x7f0e2c0614b0 2022-07-21T01:42:49.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c1561c0 con 0x7f0e2c0614b0 2022-07-21T01:42:49.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.995+0000 7f0e1bfff700 1 -- 172.21.15.3:0/1296893300 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e1c011d70 con 0x7f0e2c0614b0 2022-07-21T01:42:49.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.999+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c1526a0 con 0x7f0e2c0614b0 2022-07-21T01:42:49.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.999+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c156350 con 0x7f0e2c0614b0 2022-07-21T01:42:49.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.999+0000 7f0e1bfff700 1 -- 172.21.15.3:0/1296893300 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e1c025030 con 0x7f0e2c0614b0 2022-07-21T01:42:49.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.999+0000 7f0e1bfff700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e08053a10 0x7f0e08055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:49.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.999+0000 7f0e1bfff700 1 -- 172.21.15.3:0/1296893300 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f0e1c06d360 con 0x7f0e2c0614b0 2022-07-21T01:42:49.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.999+0000 7f0e2a7fc700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e08053a10 0x7f0e08055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:49.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:48.999+0000 7f0e2a7fc700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e08053a10 0x7f0e08055ed0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f0e2000d000 tx=0x7f0e200011a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:49.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.007+0000 7f0e1bfff700 1 -- 172.21.15.3:0/1296893300 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e1c0441e0 con 0x7f0e2c0614b0 2022-07-21T01:42:49.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.139+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0e2c0af5f0 con 0x7f0e2c0614b0 2022-07-21T01:42:49.868 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.863+0000 7f0e1bfff700 1 -- 172.21.15.3:0/1296893300 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v153) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0e1c023020 con 0x7f0e2c0614b0 2022-07-21T01:42:49.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.867+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e08053a10 msgr2=0x7f0e08055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:49.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.867+0000 7f0e2bfff700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e08053a10 0x7f0e08055ed0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f0e2000d000 tx=0x7f0e200011a0).stop 2022-07-21T01:42:49.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.867+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 msgr2=0x7f0e2c148270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:49.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.867+0000 7f0e2bfff700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 0x7f0e2c148270 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7f0e2c0aed30 tx=0x7f0e1c015000).stop 2022-07-21T01:42:49.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.871+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 shutdown_connections 2022-07-21T01:42:49.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.871+0000 7f0e2bfff700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e2c1405d0 0x7f0e2c1425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:49.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.871+0000 7f0e2bfff700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0e08053a10 0x7f0e08055ed0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:49.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.871+0000 7f0e2bfff700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c0ad750 0x7f0e2c156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:49.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.871+0000 7f0e2bfff700 1 --2- 172.21.15.3:0/1296893300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e2c0614b0 0x7f0e2c148270 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:49.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.871+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 >> 172.21.15.3:0/1296893300 conn(0x7f0e2c066f70 msgr2=0x7f0e2c1447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:49.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.871+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 shutdown_connections 2022-07-21T01:42:49.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.871+0000 7f0e2bfff700 1 -- 172.21.15.3:0/1296893300 wait complete. 2022-07-21T01:42:49.880 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:42:49.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-21T01:42:49.992 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fcbc29700 1 Processor -- start 2022-07-21T01:42:49.993 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fcbc29700 1 -- start start 2022-07-21T01:42:49.995 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fcbc29700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2fc4061eb0 0x7f2fc40b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:49.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fcbc29700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 0x7f2fc40af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:49.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fcbc29700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc4062290 con 0x7f2fc40afdb0 2022-07-21T01:42:49.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fcbc29700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc40b22e0 con 0x7f2fc4061eb0 2022-07-21T01:42:49.996 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fcbc29700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40b2500 con 0x7f2fc40b2a40 2022-07-21T01:42:49.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc99c5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2fc4061eb0 0x7f2fc40b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:49.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc99c5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2fc4061eb0 0x7f2fc40b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51744/0 (socket says 172.21.15.3:51744) 2022-07-21T01:42:49.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc99c5700 1 -- 172.21.15.3:0/1414231412 learned_addr learned my addr 172.21.15.3:0/1414231412 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:49.997 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fca1c6700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2fc40afdb0 0x7f2fc4143620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60028/0 (socket says 172.21.15.3:60028) 2022-07-21T01:42:49.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc91c4700 1 --2- 172.21.15.3:0/1414231412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 0x7f2fc40af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:49.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc99c5700 1 -- 172.21.15.3:0/1414231412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 msgr2=0x7f2fc40af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:49.998 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc89c3700 1 -- 172.21.15.3:0/1414231412 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615854008 0 0) 0x7f2fc4062290 con 0x7f2fc40afdb0 2022-07-21T01:42:49.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc99c5700 1 --2- 172.21.15.3:0/1414231412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 0x7f2fc40af8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:49.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc99c5700 1 -- 172.21.15.3:0/1414231412 >> v1:172.21.15.3:6789/0 conn(0x7f2fc40afdb0 legacy=0x7f2fc4143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:49.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc99c5700 1 -- 172.21.15.3:0/1414231412 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40b27a0 con 0x7f2fc4061eb0 2022-07-21T01:42:49.999 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc99c5700 1 --2- 172.21.15.3:0/1414231412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 0x7f2fc40b1df0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f2fc0002f70 tx=0x7f2fc0006f40).ready entity=mon.0 client_cookie=56e3d25ab280bf08 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:50.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc89c3700 1 -- 172.21.15.3:0/1414231412 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fc000da80 con 0x7f2fc4061eb0 2022-07-21T01:42:50.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc89c3700 1 -- 172.21.15.3:0/1414231412 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc0000e90 con 0x7f2fc4061eb0 2022-07-21T01:42:50.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.991+0000 7f2fc89c3700 1 -- 172.21.15.3:0/1414231412 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fc000ddd0 con 0x7f2fc4061eb0 2022-07-21T01:42:50.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 -- 172.21.15.3:0/1414231412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 msgr2=0x7f2fc40b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 --2- 172.21.15.3:0/1414231412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 0x7f2fc40b1df0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f2fc0002f70 tx=0x7f2fc0006f40).stop 2022-07-21T01:42:50.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 -- 172.21.15.3:0/1414231412 shutdown_connections 2022-07-21T01:42:50.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 --2- 172.21.15.3:0/1414231412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 0x7f2fc40af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 --2- 172.21.15.3:0/1414231412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 0x7f2fc40b1df0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 -- 172.21.15.3:0/1414231412 >> 172.21.15.3:0/1414231412 conn(0x7f2fc4063b40 msgr2=0x7f2fc4063f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:50.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 -- 172.21.15.3:0/1414231412 shutdown_connections 2022-07-21T01:42:50.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 -- 172.21.15.3:0/1414231412 wait complete. 2022-07-21T01:42:50.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 Processor -- start 2022-07-21T01:42:50.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 -- start start 2022-07-21T01:42:50.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 0x7f2fc4060b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc40afdb0 0x7f2fc413b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 0x7f2fc413b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc4064b60 con 0x7f2fc4061eb0 2022-07-21T01:42:50.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc4064db0 con 0x7f2fc40afdb0 2022-07-21T01:42:50.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.995+0000 7f2fcbc29700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc4065050 con 0x7f2fc40b2a40 2022-07-21T01:42:50.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc99c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 0x7f2fc4060b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc99c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 0x7f2fc4060b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51750/0 (socket says 172.21.15.3:51750) 2022-07-21T01:42:50.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc99c5700 1 -- 172.21.15.3:0/2505133230 learned_addr learned my addr 172.21.15.3:0/2505133230 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:50.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fca1c6700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 0x7f2fc413b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc91c4700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc40afdb0 0x7f2fc413b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc99c5700 1 -- 172.21.15.3:0/2505133230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 msgr2=0x7f2fc413b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc99c5700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 0x7f2fc413b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc99c5700 1 -- 172.21.15.3:0/2505133230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc40afdb0 msgr2=0x7f2fc413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc99c5700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc40afdb0 0x7f2fc413b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc99c5700 1 -- 172.21.15.3:0/2505133230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc000b460 con 0x7f2fc4061eb0 2022-07-21T01:42:50.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fc99c5700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 0x7f2fc4060b10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f2fc0002f70 tx=0x7f2fc41400d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:50.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fb2ffd700 1 -- 172.21.15.3:0/2505133230 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fc001d070 con 0x7f2fc4061eb0 2022-07-21T01:42:50.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc40652f0 con 0x7f2fc4061eb0 2022-07-21T01:42:50.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fb2ffd700 1 -- 172.21.15.3:0/2505133230 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc00068b0 con 0x7f2fc4061eb0 2022-07-21T01:42:50.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fb2ffd700 1 -- 172.21.15.3:0/2505133230 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fc0018800 con 0x7f2fc4061eb0 2022-07-21T01:42:50.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:49.999+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc4145b90 con 0x7f2fc4061eb0 2022-07-21T01:42:50.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.003+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc4065480 con 0x7f2fc4061eb0 2022-07-21T01:42:50.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.003+0000 7f2fb2ffd700 1 -- 172.21.15.3:0/2505133230 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2fc0018960 con 0x7f2fc4061eb0 2022-07-21T01:42:50.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.003+0000 7f2fb2ffd700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2fa4053aa0 0x7f2fa4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.003+0000 7f2fb2ffd700 1 -- 172.21.15.3:0/2505133230 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f2fc006cef0 con 0x7f2fc4061eb0 2022-07-21T01:42:50.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.003+0000 7f2fc91c4700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2fa4053aa0 0x7f2fa4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.003+0000 7f2fc91c4700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2fa4053aa0 0x7f2fa4055f60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f2fb400b000 tx=0x7f2fb4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:50.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.007+0000 7f2fb2ffd700 1 -- 172.21.15.3:0/2505133230 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2fc0043ef0 con 0x7f2fc4061eb0 2022-07-21T01:42:50.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.139+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2fc40af180 con 0x7f2fc4061eb0 2022-07-21T01:42:50.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.147+0000 7f2fb2ffd700 1 -- 172.21.15.3:0/2505133230 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2fc0047510 con 0x7f2fc4061eb0 2022-07-21T01:42:50.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2fa4053aa0 msgr2=0x7f2fa4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2fa4053aa0 0x7f2fa4055f60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f2fb400b000 tx=0x7f2fb4006df0).stop 2022-07-21T01:42:50.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 msgr2=0x7f2fc4060b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 0x7f2fc4060b10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f2fc0002f70 tx=0x7f2fc41400d0).stop 2022-07-21T01:42:50.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 shutdown_connections 2022-07-21T01:42:50.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc40b2a40 0x7f2fc413b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2fa4053aa0 0x7f2fa4055f60 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc40afdb0 0x7f2fc413b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 --2- 172.21.15.3:0/2505133230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2fc4061eb0 0x7f2fc4060b10 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 >> 172.21.15.3:0/2505133230 conn(0x7f2fc4063b40 msgr2=0x7f2fc413ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:50.157 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 shutdown_connections 2022-07-21T01:42:50.158 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.151+0000 7f2fcbc29700 1 -- 172.21.15.3:0/2505133230 wait complete. 2022-07-21T01:42:50.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-07-21T01:42:50.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd381c82700 1 Processor -- start 2022-07-21T01:42:50.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd381c82700 1 -- start start 2022-07-21T01:42:50.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd381c82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0619b0 0x7fd37c0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd381c82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd37c0b6ee0 0x7fd37c143890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd381c82700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c061d90 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd381c82700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd37c142f60 con 0x7fd37c0b6ee0 2022-07-21T01:42:50.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd381c82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0ba5a0 con 0x7fd37c0619b0 2022-07-21T01:42:50.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd380c80700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0619b0 0x7fd37c0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd373fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd37c0b6ee0 0x7fd37c143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd373fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd37c0b6ee0 0x7fd37c143890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:57208/0 (socket says 172.21.15.3:57208) 2022-07-21T01:42:50.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd373fff700 1 -- 172.21.15.3:0/2243397014 learned_addr learned my addr 172.21.15.3:0/2243397014 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:50.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd373fff700 1 -- 172.21.15.3:0/2243397014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0619b0 msgr2=0x7fd37c0b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:50.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd373fff700 1 --2- 172.21.15.3:0/2243397014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0619b0 0x7fd37c0b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd373fff700 1 -- 172.21.15.3:0/2243397014 >> v1:172.21.15.143:6789/0 conn(0x7fd37c0b5ed0 legacy=0x7fd37c0b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd373fff700 1 -- 172.21.15.3:0/2243397014 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c143150 con 0x7fd37c0b6ee0 2022-07-21T01:42:50.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd373fff700 1 --2- 172.21.15.3:0/2243397014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd37c0b6ee0 0x7fd37c143890 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7fd36c002f70 tx=0x7fd36c006eb0).ready entity=mon.1 client_cookie=72c62b121554672d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:50.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd3737fe700 1 -- 172.21.15.3:0/2243397014 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd36c00d570 con 0x7fd37c0b6ee0 2022-07-21T01:42:50.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd3737fe700 1 -- 172.21.15.3:0/2243397014 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c000e90 con 0x7fd37c0b6ee0 2022-07-21T01:42:50.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.271+0000 7fd3737fe700 1 -- 172.21.15.3:0/2243397014 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd36c00d900 con 0x7fd37c0b6ee0 2022-07-21T01:42:50.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 -- 172.21.15.3:0/2243397014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd37c0b6ee0 msgr2=0x7fd37c143890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 --2- 172.21.15.3:0/2243397014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd37c0b6ee0 0x7fd37c143890 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7fd36c002f70 tx=0x7fd36c006eb0).stop 2022-07-21T01:42:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 -- 172.21.15.3:0/2243397014 shutdown_connections 2022-07-21T01:42:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 --2- 172.21.15.3:0/2243397014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd37c0b6ee0 0x7fd37c143890 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 --2- 172.21.15.3:0/2243397014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0619b0 0x7fd37c0b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 -- 172.21.15.3:0/2243397014 >> 172.21.15.3:0/2243397014 conn(0x7fd37c06af80 msgr2=0x7fd37c05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:50.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 -- 172.21.15.3:0/2243397014 shutdown_connections 2022-07-21T01:42:50.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 -- 172.21.15.3:0/2243397014 wait complete. 2022-07-21T01:42:50.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 Processor -- start 2022-07-21T01:42:50.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 -- start start 2022-07-21T01:42:50.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd37c0619b0 0x7fd37c13dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd37c0b5ed0 0x7fd37c13bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0b6ee0 0x7fd37c13c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c147210 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c1532a0 con 0x7fd37c0619b0 2022-07-21T01:42:50.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381c82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c152f80 con 0x7fd37c0b6ee0 2022-07-21T01:42:50.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd373fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd37c0b5ed0 0x7fd37c13bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd373fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd37c0b5ed0 0x7fd37c13bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51766/0 (socket says 172.21.15.3:51766) 2022-07-21T01:42:50.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd373fff700 1 -- 172.21.15.3:0/2381018615 learned_addr learned my addr 172.21.15.3:0/2381018615 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:50.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd380c80700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd37c0619b0 0x7fd37c13dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.275+0000 7fd381481700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0b6ee0 0x7fd37c13c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd373fff700 1 -- 172.21.15.3:0/2381018615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0b6ee0 msgr2=0x7fd37c13c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd373fff700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0b6ee0 0x7fd37c13c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd373fff700 1 -- 172.21.15.3:0/2381018615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd37c0619b0 msgr2=0x7fd37c13dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd373fff700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd37c0619b0 0x7fd37c13dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd373fff700 1 -- 172.21.15.3:0/2381018615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c00b1e0 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd373fff700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd37c0b5ed0 0x7fd37c13bcc0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fd36c002f70 tx=0x7fd36c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:50.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd371ffb700 1 -- 172.21.15.3:0/2381018615 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd36c0117e0 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd371ffb700 1 -- 172.21.15.3:0/2381018615 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c011940 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37c146e90 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd371ffb700 1 -- 172.21.15.3:0/2381018615 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd36c011ca0 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37c13c840 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37c144220 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd371ffb700 1 -- 172.21.15.3:0/2381018615 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd36c025030 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd371ffb700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd35c053a70 0x7fd35c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:50.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd371ffb700 1 -- 172.21.15.3:0/2381018615 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fd36c06cce0 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.279+0000 7fd380c80700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd35c053a70 0x7fd35c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:50.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.283+0000 7fd380c80700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd35c053a70 0x7fd35c055f30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fd37800b000 tx=0x7fd378006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:50.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.287+0000 7fd371ffb700 1 -- 172.21.15.3:0/2381018615 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd37c144220 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.459+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7fd37c13c840 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.870 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.867+0000 7fd371ffb700 1 -- 172.21.15.3:0/2381018615 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v440) v1 ==== 127+0+0 (secure 0 0 0) 0x7fd37c13c840 con 0x7fd37c0b5ed0 2022-07-21T01:42:50.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd35c053a70 msgr2=0x7fd35c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd35c053a70 0x7fd35c055f30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fd37800b000 tx=0x7fd378006df0).stop 2022-07-21T01:42:50.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd37c0b5ed0 msgr2=0x7fd37c13bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:50.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd37c0b5ed0 0x7fd37c13bcc0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fd36c002f70 tx=0x7fd36c015000).stop 2022-07-21T01:42:50.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 shutdown_connections 2022-07-21T01:42:50.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd37c0b6ee0 0x7fd37c13c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd37c0b5ed0 0x7fd37c13bcc0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd35c053a70 0x7fd35c055f30 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 --2- 172.21.15.3:0/2381018615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd37c0619b0 0x7fd37c13dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:50.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 >> 172.21.15.3:0/2381018615 conn(0x7fd37c06af80 msgr2=0x7fd37c141520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:50.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 shutdown_connections 2022-07-21T01:42:50.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.871+0000 7fd381c82700 1 -- 172.21.15.3:0/2381018615 wait complete. 2022-07-21T01:42:50.889 INFO:tasks.cephfs_test_runner:test_root_rctime (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-21T01:42:50.890 INFO:tasks.cephfs_test_runner:Starting test: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-21T01:42:50.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs' 2022-07-21T01:42:51.000 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 Processor -- start 2022-07-21T01:42:51.001 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 -- start start 2022-07-21T01:42:51.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5b413c870 0x7fb5b413ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:51.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5b40ad0b0 0x7fb5b40ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:51.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b40b2dc0 con 0x7fb5b40b29e0 2022-07-21T01:42:51.003 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b40b1340 con 0x7fb5b413c870 2022-07-21T01:42:51.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b40b17f0 con 0x7fb5b40ad0b0 2022-07-21T01:42:51.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5b413c870 0x7fb5b413ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:51.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b37fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb5b413c870 0x7fb5b413ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51774/0 (socket says 172.21.15.3:51774) 2022-07-21T01:42:51.004 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b37fe700 1 -- 172.21.15.3:0/1324969809 learned_addr learned my addr 172.21.15.3:0/1324969809 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:51.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b8a97700 1 --2- 172.21.15.3:0/1324969809 >> v2:172.21.15.143:3300/0 conn(0x7fb5b40ad0b0 0x7fb5b40ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:51.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b2ffd700 1 -- 172.21.15.3:0/1324969809 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236627543 0 0) 0x7fb5b40b2dc0 con 0x7fb5b40b29e0 2022-07-21T01:42:51.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b37fe700 1 -- 172.21.15.3:0/1324969809 >> v2:172.21.15.143:3300/0 conn(0x7fb5b40ad0b0 msgr2=0x7fb5b40ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:51.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b37fe700 1 --2- 172.21.15.3:0/1324969809 >> v2:172.21.15.143:3300/0 conn(0x7fb5b40ad0b0 0x7fb5b40ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:51.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b37fe700 1 -- 172.21.15.3:0/1324969809 >> v1:172.21.15.3:6789/0 conn(0x7fb5b40b29e0 legacy=0x7fb5b40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:51.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b37fe700 1 -- 172.21.15.3:0/1324969809 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b40b1a90 con 0x7fb5b413c870 2022-07-21T01:42:51.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b37fe700 1 --2- 172.21.15.3:0/1324969809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b413c870 0x7fb5b413ccb0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7fb5a8002f70 tx=0x7fb5a8006eb0).ready entity=mon.0 client_cookie=5d53041c751bf11 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:51.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b2ffd700 1 -- 172.21.15.3:0/1324969809 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a800d5c0 con 0x7fb5b413c870 2022-07-21T01:42:51.006 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b2ffd700 1 -- 172.21.15.3:0/1324969809 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a8000e90 con 0x7fb5b413c870 2022-07-21T01:42:51.007 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b2ffd700 1 -- 172.21.15.3:0/1324969809 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a800d910 con 0x7fb5b413c870 2022-07-21T01:42:51.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 -- 172.21.15.3:0/1324969809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b413c870 msgr2=0x7fb5b413ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:51.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 --2- 172.21.15.3:0/1324969809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b413c870 0x7fb5b413ccb0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7fb5a8002f70 tx=0x7fb5a8006eb0).stop 2022-07-21T01:42:51.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 -- 172.21.15.3:0/1324969809 shutdown_connections 2022-07-21T01:42:51.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 --2- 172.21.15.3:0/1324969809 >> v2:172.21.15.143:3300/0 conn(0x7fb5b40ad0b0 0x7fb5b40ad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:51.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 --2- 172.21.15.3:0/1324969809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b413c870 0x7fb5b413ccb0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:51.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:50.999+0000 7fb5b9298700 1 -- 172.21.15.3:0/1324969809 >> 172.21.15.3:0/1324969809 conn(0x7fb5b40613c0 msgr2=0x7fb5b4062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:51.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 -- 172.21.15.3:0/1324969809 shutdown_connections 2022-07-21T01:42:51.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 -- 172.21.15.3:0/1324969809 wait complete. 2022-07-21T01:42:51.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 Processor -- start 2022-07-21T01:42:51.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 -- start start 2022-07-21T01:42:51.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b40ad0b0 0x7fb5b413ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:51.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b40b29e0 0x7fb5b4064780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:51.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b413c870 0x7fb5b413e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:51.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b413fa90 con 0x7fb5b40b29e0 2022-07-21T01:42:51.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b413fd30 con 0x7fb5b413c870 2022-07-21T01:42:51.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b9298700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b413dac0 con 0x7fb5b40ad0b0 2022-07-21T01:42:51.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b40b29e0 0x7fb5b4064780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:51.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.003+0000 7fb5b37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b40b29e0 0x7fb5b4064780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51780/0 (socket says 172.21.15.3:51780) 2022-07-21T01:42:51.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b37fe700 1 -- 172.21.15.3:0/2870580441 learned_addr learned my addr 172.21.15.3:0/2870580441 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:51.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b3fff700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b40ad0b0 0x7fb5b413ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:51.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b8a97700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b413c870 0x7fb5b413e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:51.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b37fe700 1 -- 172.21.15.3:0/2870580441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b40ad0b0 msgr2=0x7fb5b413ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:51.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b37fe700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b40ad0b0 0x7fb5b413ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:51.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b37fe700 1 -- 172.21.15.3:0/2870580441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b413c870 msgr2=0x7fb5b413e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:51.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b37fe700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b413c870 0x7fb5b413e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:51.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b37fe700 1 -- 172.21.15.3:0/2870580441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a800b1e0 con 0x7fb5b40b29e0 2022-07-21T01:42:51.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b37fe700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b40b29e0 0x7fb5b4064780 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7fb5a8002f70 tx=0x7fb5a8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:51.016 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b17fa700 1 -- 172.21.15.3:0/2870580441 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a800d960 con 0x7fb5b40b29e0 2022-07-21T01:42:51.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b17fa700 1 -- 172.21.15.3:0/2870580441 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a8011c30 con 0x7fb5b40b29e0 2022-07-21T01:42:51.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b41429a0 con 0x7fb5b40b29e0 2022-07-21T01:42:51.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b17fa700 1 -- 172.21.15.3:0/2870580441 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a801e460 con 0x7fb5b40b29e0 2022-07-21T01:42:51.017 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b4142ea0 con 0x7fb5b40b29e0 2022-07-21T01:42:51.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b413fef0 con 0x7fb5b40b29e0 2022-07-21T01:42:51.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.007+0000 7fb5b17fa700 1 -- 172.21.15.3:0/2870580441 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb5a8025030 con 0x7fb5b40b29e0 2022-07-21T01:42:51.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.011+0000 7fb5b17fa700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb594053950 0x7fb594055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:51.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.011+0000 7fb5b17fa700 1 -- 172.21.15.3:0/2870580441 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fb5a806ce90 con 0x7fb5b40b29e0 2022-07-21T01:42:51.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.011+0000 7fb5b3fff700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb594053950 0x7fb594055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:51.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.011+0000 7fb5b3fff700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb594053950 0x7fb594055e10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fb5a400d000 tx=0x7fb5a4007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:51.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.015+0000 7fb5b17fa700 1 -- 172.21.15.3:0/2870580441 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5a8043f30 con 0x7fb5b40b29e0 2022-07-21T01:42:51.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.183+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]} v 0) v1 -- 0x7fb5b40b1c70 con 0x7fb5b40b29e0 2022-07-21T01:42:51.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.867+0000 7fb5b17fa700 1 -- 172.21.15.3:0/2870580441 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]}]=0 v441) v1 ==== 139+0+0 (secure 0 0 0) 0x7fb5a8023020 con 0x7fb5b40b29e0 2022-07-21T01:42:51.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb594053950 msgr2=0x7fb594055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:51.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb594053950 0x7fb594055e10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fb5a400d000 tx=0x7fb5a4007da0).stop 2022-07-21T01:42:51.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b40b29e0 msgr2=0x7fb5b4064780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:51.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b40b29e0 0x7fb5b4064780 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7fb5a8002f70 tx=0x7fb5a8015000).stop 2022-07-21T01:42:51.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 shutdown_connections 2022-07-21T01:42:51.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb594053950 0x7fb594055e10 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:51.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5b413c870 0x7fb5b413e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:51.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b40b29e0 0x7fb5b4064780 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:51.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 --2- 172.21.15.3:0/2870580441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5b40ad0b0 0x7fb5b413ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:51.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 >> 172.21.15.3:0/2870580441 conn(0x7fb5b40613c0 msgr2=0x7fb5b40b6440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:51.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 shutdown_connections 2022-07-21T01:42:51.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:51.875+0000 7fb5b9298700 1 -- 172.21.15.3:0/2870580441 wait complete. 2022-07-21T01:42:51.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-21T01:42:52.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.003+0000 7f210be90700 1 Processor -- start 2022-07-21T01:42:52.008 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.003+0000 7f210be90700 1 -- start start 2022-07-21T01:42:52.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210be90700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21040aa050 0x7f21040abea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210be90700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2104144eb0 con 0x7f21040b1110 2022-07-21T01:42:52.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210be90700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2104145110 con 0x7f2104144ad0 2022-07-21T01:42:52.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210be90700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21040ac390 con 0x7f21040aa050 2022-07-21T01:42:52.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210a42d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21040aa050 0x7f21040abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210a42d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f21040aa050 0x7f21040abea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51790/0 (socket says 172.21.15.3:51790) 2022-07-21T01:42:52.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210942b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f21040b1110 0x7f210413e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60068/0 (socket says 172.21.15.3:60068) 2022-07-21T01:42:52.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210a42d700 1 -- 172.21.15.3:0/2510466734 learned_addr learned my addr 172.21.15.3:0/2510466734 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:52.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210a42d700 1 -- 172.21.15.3:0/2510466734 >> v1:172.21.15.143:6789/0 conn(0x7f2104144ad0 legacy=0x7f21040b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210a42d700 1 -- 172.21.15.3:0/2510466734 >> v1:172.21.15.3:6789/0 conn(0x7f21040b1110 legacy=0x7f210413e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.013 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210a42d700 1 -- 172.21.15.3:0/2510466734 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21041453b0 con 0x7f21040aa050 2022-07-21T01:42:52.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210a42d700 1 --2- 172.21.15.3:0/2510466734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 0x7f21040abea0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7f20fc0032f0 tx=0x7f20fc009290).ready entity=mon.0 client_cookie=6ecd8cfb369233c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:52.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f2108c2a700 1 -- 172.21.15.3:0/2510466734 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20fc00b8a0 con 0x7f21040aa050 2022-07-21T01:42:52.014 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f2108c2a700 1 -- 172.21.15.3:0/2510466734 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc00ba00 con 0x7f21040aa050 2022-07-21T01:42:52.015 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f2108c2a700 1 -- 172.21.15.3:0/2510466734 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20fc00bb70 con 0x7f21040aa050 2022-07-21T01:42:52.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210be90700 1 -- 172.21.15.3:0/2510466734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 msgr2=0x7f21040abea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.019 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.007+0000 7f210be90700 1 --2- 172.21.15.3:0/2510466734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 0x7f21040abea0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7f20fc0032f0 tx=0x7f20fc009290).stop 2022-07-21T01:42:52.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 -- 172.21.15.3:0/2510466734 shutdown_connections 2022-07-21T01:42:52.020 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 --2- 172.21.15.3:0/2510466734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 0x7f21040abea0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 -- 172.21.15.3:0/2510466734 >> 172.21.15.3:0/2510466734 conn(0x7f21040b2b00 msgr2=0x7f21040b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:52.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 -- 172.21.15.3:0/2510466734 shutdown_connections 2022-07-21T01:42:52.021 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 -- 172.21.15.3:0/2510466734 wait complete. 2022-07-21T01:42:52.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 Processor -- start 2022-07-21T01:42:52.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 -- start start 2022-07-21T01:42:52.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 0x7f2104064420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21040b1110 0x7f21041435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2104144ad0 0x7f21041546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.023 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2104154bb0 con 0x7f21040aa050 2022-07-21T01:42:52.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2104150830 con 0x7f21040b1110 2022-07-21T01:42:52.024 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210be90700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2104063740 con 0x7f2104144ad0 2022-07-21T01:42:52.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f2109c2c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 0x7f2104064420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f2109c2c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 0x7f2104064420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51792/0 (socket says 172.21.15.3:51792) 2022-07-21T01:42:52.025 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f2109c2c700 1 -- 172.21.15.3:0/3968398515 learned_addr learned my addr 172.21.15.3:0/3968398515 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:52.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.011+0000 7f210942b700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21040b1110 0x7f21041435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f210a42d700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2104144ad0 0x7f21041546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f2109c2c700 1 -- 172.21.15.3:0/3968398515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2104144ad0 msgr2=0x7f21041546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f2109c2c700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2104144ad0 0x7f21041546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f2109c2c700 1 -- 172.21.15.3:0/3968398515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21040b1110 msgr2=0x7f21041435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f2109c2c700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21040b1110 0x7f21041435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f2109c2c700 1 -- 172.21.15.3:0/3968398515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc009530 con 0x7f21040aa050 2022-07-21T01:42:52.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f2109c2c700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 0x7f2104064420 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f2104061c10 tx=0x7f2100007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:52.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f20f2ffd700 1 -- 172.21.15.3:0/3968398515 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2100007e70 con 0x7f21040aa050 2022-07-21T01:42:52.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f20f2ffd700 1 -- 172.21.15.3:0/3968398515 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2100007030 con 0x7f21040aa050 2022-07-21T01:42:52.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2104150a60 con 0x7f21040aa050 2022-07-21T01:42:52.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f20f2ffd700 1 -- 172.21.15.3:0/3968398515 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2100007390 con 0x7f21040aa050 2022-07-21T01:42:52.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21040639e0 con 0x7f21040aa050 2022-07-21T01:42:52.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2104145650 con 0x7f21040aa050 2022-07-21T01:42:52.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.015+0000 7f20f2ffd700 1 -- 172.21.15.3:0/3968398515 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2100010030 con 0x7f21040aa050 2022-07-21T01:42:52.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.019+0000 7f20f2ffd700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20e4053980 0x7f20e4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.019+0000 7f20f2ffd700 1 -- 172.21.15.3:0/3968398515 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f210006b810 con 0x7f21040aa050 2022-07-21T01:42:52.033 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.019+0000 7f210942b700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20e4053980 0x7f20e4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.019+0000 7f210942b700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20e4053980 0x7f20e4055e40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f20f400d000 tx=0x7f20f4007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:52.034 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.023+0000 7f20f2ffd700 1 -- 172.21.15.3:0/3968398515 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2100042810 con 0x7f21040aa050 2022-07-21T01:42:52.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.151+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2104143aa0 con 0x7f21040aa050 2022-07-21T01:42:52.164 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.159+0000 7f20f2ffd700 1 -- 172.21.15.3:0/3968398515 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2100045e30 con 0x7f21040aa050 2022-07-21T01:42:52.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20e4053980 msgr2=0x7f20e4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20e4053980 0x7f20e4055e40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f20f400d000 tx=0x7f20f4007b80).stop 2022-07-21T01:42:52.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 msgr2=0x7f2104064420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 0x7f2104064420 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f2104061c10 tx=0x7f2100007c00).stop 2022-07-21T01:42:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 shutdown_connections 2022-07-21T01:42:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2104144ad0 0x7f21041546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20e4053980 0x7f20e4055e40 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21040b1110 0x7f21041435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 --2- 172.21.15.3:0/3968398515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21040aa050 0x7f2104064420 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 >> 172.21.15.3:0/3968398515 conn(0x7f21040b2b00 msgr2=0x7f210413d900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:52.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 shutdown_connections 2022-07-21T01:42:52.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.163+0000 7f210be90700 1 -- 172.21.15.3:0/3968398515 wait complete. 2022-07-21T01:42:52.179 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:42:52.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:52.198 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:52.221 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:52.243 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:42:52.265 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:42:52.266 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:52.267 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:52.267 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:42:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:42:52.281 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-21T01:42:52.282 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-21T01:42:52.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:42:52.284 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:42:52.285 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:42:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:42:52.295 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-21T01:42:52.295 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-21T01:42:52.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:42:52.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754f89f700 1 Processor -- start 2022-07-21T01:42:52.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754f89f700 1 -- start start 2022-07-21T01:42:52.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754f89f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b6710 0x7f75480b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754f89f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f75480b4550 0x7f75480b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754f89f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7548144240 con 0x7f7548061ec0 2022-07-21T01:42:52.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754f89f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75480b6fe0 con 0x7f75480b4550 2022-07-21T01:42:52.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754f89f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7548143d00 con 0x7f75480b6710 2022-07-21T01:42:52.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754de3c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7548061ec0 0x7f7548146850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60084/0 (socket says 172.21.15.3:60084) 2022-07-21T01:42:52.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754de3c700 1 -- 172.21.15.3:0/76318520 learned_addr learned my addr 172.21.15.3:0/76318520 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:52.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754ce3a700 1 --2- 172.21.15.3:0/76318520 >> v2:172.21.15.143:3300/0 conn(0x7f75480b4550 0x7f75480b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754d63b700 1 --2- 172.21.15.3:0/76318520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b6710 0x7f75480b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f7533fff700 1 -- 172.21.15.3:0/76318520 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699349804 0 0) 0x7f7548144240 con 0x7f7548061ec0 2022-07-21T01:42:52.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f7533fff700 1 -- 172.21.15.3:0/76318520 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c003260 con 0x7f7548061ec0 2022-07-21T01:42:52.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754ce3a700 1 -- 172.21.15.3:0/76318520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b6710 msgr2=0x7f75480b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:52.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754ce3a700 1 --2- 172.21.15.3:0/76318520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b6710 0x7f75480b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754ce3a700 1 -- 172.21.15.3:0/76318520 >> v1:172.21.15.3:6789/0 conn(0x7f7548061ec0 legacy=0x7f7548146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f7533fff700 1 -- 172.21.15.3:0/76318520 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 284985625 0 0) 0x7f752c003260 con 0x7f7548061ec0 2022-07-21T01:42:52.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754ce3a700 1 -- 172.21.15.3:0/76318520 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7548143fa0 con 0x7f75480b4550 2022-07-21T01:42:52.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754d63b700 1 --2- 172.21.15.3:0/76318520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b6710 0x7f75480b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:52.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f754ce3a700 1 --2- 172.21.15.3:0/76318520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75480b4550 0x7f75480b6af0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f7540001b80 tx=0x7f753c006df0).ready entity=mon.1 client_cookie=64bd7e0f964a0e8f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:52.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f7533fff700 1 -- 172.21.15.3:0/76318520 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75400019e0 con 0x7f75480b4550 2022-07-21T01:42:52.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f7533fff700 1 -- 172.21.15.3:0/76318520 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c000e90 con 0x7f75480b4550 2022-07-21T01:42:52.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.403+0000 7f7533fff700 1 -- 172.21.15.3:0/76318520 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f753c012460 con 0x7f75480b4550 2022-07-21T01:42:52.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 -- 172.21.15.3:0/76318520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75480b4550 msgr2=0x7f75480b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 --2- 172.21.15.3:0/76318520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75480b4550 0x7f75480b6af0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f7540001b80 tx=0x7f753c006df0).stop 2022-07-21T01:42:52.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 -- 172.21.15.3:0/76318520 shutdown_connections 2022-07-21T01:42:52.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 --2- 172.21.15.3:0/76318520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75480b4550 0x7f75480b6af0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 --2- 172.21.15.3:0/76318520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b6710 0x7f75480b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 -- 172.21.15.3:0/76318520 >> 172.21.15.3:0/76318520 conn(0x7f754805c620 msgr2=0x7f754805ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:52.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 -- 172.21.15.3:0/76318520 shutdown_connections 2022-07-21T01:42:52.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 -- 172.21.15.3:0/76318520 wait complete. 2022-07-21T01:42:52.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 Processor -- start 2022-07-21T01:42:52.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 -- start start 2022-07-21T01:42:52.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7548061ec0 0x7f75481449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b4550 0x7f7548140440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75480b6710 0x7f7548140930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754812b8f0 con 0x7f75480b6710 2022-07-21T01:42:52.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754812b000 con 0x7f7548061ec0 2022-07-21T01:42:52.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754f89f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7548126ed0 con 0x7f75480b4550 2022-07-21T01:42:52.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754de3c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75480b6710 0x7f7548140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754de3c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75480b6710 0x7f7548140930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51810/0 (socket says 172.21.15.3:51810) 2022-07-21T01:42:52.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754de3c700 1 -- 172.21.15.3:0/577415950 learned_addr learned my addr 172.21.15.3:0/577415950 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:52.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754ce3a700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b4550 0x7f7548140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.407+0000 7f754d63b700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7548061ec0 0x7f75481449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f754de3c700 1 -- 172.21.15.3:0/577415950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b4550 msgr2=0x7f7548140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f754de3c700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b4550 0x7f7548140440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f754de3c700 1 -- 172.21.15.3:0/577415950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7548061ec0 msgr2=0x7f75481449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f754de3c700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7548061ec0 0x7f75481449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f754de3c700 1 -- 172.21.15.3:0/577415950 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754400b180 con 0x7f75480b6710 2022-07-21T01:42:52.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f754de3c700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75480b6710 0x7f7548140930 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f754812b2c0 tx=0x7f7540002160).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:52.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f753affd700 1 -- 172.21.15.3:0/577415950 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7540004ab0 con 0x7f75480b6710 2022-07-21T01:42:52.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f753affd700 1 -- 172.21.15.3:0/577415950 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7540004c10 con 0x7f75480b6710 2022-07-21T01:42:52.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753c00b180 con 0x7f75480b6710 2022-07-21T01:42:52.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f753affd700 1 -- 172.21.15.3:0/577415950 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7540007170 con 0x7f75480b6710 2022-07-21T01:42:52.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754812b3a0 con 0x7f75480b6710 2022-07-21T01:42:52.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754812b560 con 0x7f75480b6710 2022-07-21T01:42:52.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f753affd700 1 -- 172.21.15.3:0/577415950 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f754001b070 con 0x7f75480b6710 2022-07-21T01:42:52.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f753affd700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7524053b30 0x7f7524055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.411+0000 7f753affd700 1 -- 172.21.15.3:0/577415950 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f754006b5a0 con 0x7f75480b6710 2022-07-21T01:42:52.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.415+0000 7f754d63b700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7524053b30 0x7f7524055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.415+0000 7f754d63b700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7524053b30 0x7f7524055ff0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f754400b000 tx=0x7f7544006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:52.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.419+0000 7f753affd700 1 -- 172.21.15.3:0/577415950 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7540042640 con 0x7f75480b6710 2022-07-21T01:42:52.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.551+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f7548061780 con 0x7f75480b6710 2022-07-21T01:42:52.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.551+0000 7f753affd700 1 -- 172.21.15.3:0/577415950 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v153) v1 ==== 81+0+29231 (secure 0 0 0) 0x7f7540045c60 con 0x7f75480b6710 2022-07-21T01:42:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:52.557 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:42:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 153, 2022-07-21T01:42:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:42:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:42:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:42:49.852415+0000", 2022-07-21T01:42:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:42:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:42:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:42:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:42:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:42:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:42:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:42:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:42:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:42:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:42:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:42:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:42:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:42:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 28, 2022-07-21T01:42:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:42:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:42:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:42:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:42:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:42:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:42:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:42:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:42:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:42:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:42:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:42:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:42:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:42:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:42:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:42:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:42:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:42:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:42:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:42:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:42:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:42:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:42:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:42:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:42:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:42:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:42:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:42:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:42:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:42:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:42:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:42:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:42:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:42:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:42:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:42:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:42:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:42:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:42:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:42:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:42:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:42:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:42:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:42:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:42:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:42:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:42:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:42:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:42:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:42:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:42:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:42:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:42:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:42:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:42:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:42:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:42:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:42:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:42:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:42:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:42:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:42:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:42:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:42:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:42:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:42:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:42:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:42:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:42:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:42:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:42:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:42:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:42:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:42:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:42:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:42:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:42:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:42:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:42:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:42:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:42:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:42:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:42:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:42:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:42:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:42:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:42:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.619 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:42:52.619 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:52.619 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.619 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.620 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:52.620 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.620 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.621 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:42:52.621 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:52.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.622 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.622 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:42:52.622 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:52.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.623 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.623 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:52.623 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.624 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.624 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:42:52.624 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:52.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.625 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.625 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:42:52.625 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:42:52.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.626 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.626 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:42:52.627 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:42:52.627 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:42:52.627 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:42:52.627 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:52.628 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:52.628 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:52.628 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.629 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:52.629 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:42:52.629 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:52.630 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:52.630 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:52.630 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:52.630 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:52.630 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:52.631 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:52.631 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:52.631 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:52.631 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:52.632 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:52.632 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.632 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:42:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:42:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:42:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.638 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.638 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:52.638 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.639 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.639 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:42:52.639 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:52.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.640 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:42:52.640 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:52.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.641 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.641 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:52.642 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.642 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.642 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:42:52.643 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:52.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.643 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.644 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:42:52.644 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:42:52.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.644 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.645 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:42:52.645 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:42:52.645 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:42:52.645 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:42:52.646 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:52.646 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:52.646 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:52.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.647 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:52.647 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:42:52.648 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:52.648 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:52.648 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:52.648 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:52.649 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:52.649 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:52.649 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:52.649 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 146, 2022-07-21T01:42:52.649 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:42:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:42:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:52.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.653 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:52.654 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.654 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:42:52.655 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:52.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.655 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.656 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:42:52.656 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:52.656 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.656 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.657 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:52.657 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.657 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.658 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:42:52.658 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:52.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.659 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.659 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:42:52.659 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:52.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.660 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.660 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:52.661 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.661 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:42:52.662 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:52.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.662 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.663 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:42:52.663 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:42:52.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.663 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.664 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:42:52.664 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:42:52.664 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:42:52.664 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:42:52.665 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:52.665 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:52.665 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:52.665 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.666 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:52.666 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:42:52.667 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:52.667 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:52.667 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:52.667 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:42:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 137, 2022-07-21T01:42:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:52.669 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:52.669 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:52.669 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.670 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.670 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:42:52.670 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:52.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.671 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.671 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:42:52.671 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:52.671 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.672 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.672 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:52.672 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.673 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:42:52.673 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:52.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.674 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:42:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:52.676 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.676 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.676 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:42:52.677 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:52.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.677 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.678 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:42:52.678 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:52.678 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.678 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.679 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:52.679 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.680 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.680 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:42:52.680 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:52.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.681 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.681 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:42:52.681 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:42:52.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.682 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.682 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:42:52.683 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:42:52.683 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:42:52.683 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:42:52.683 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:52.684 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:52.684 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:52.684 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.685 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:52.685 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:42:52.686 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:52.686 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:52.686 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:52.687 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:52.687 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:52.687 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:52.687 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:52.688 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 146, 2022-07-21T01:42:52.688 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:52.688 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:52.688 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:52.689 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.689 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.689 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:42:52.689 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:52.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.690 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.690 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:42:52.691 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:52.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.691 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.692 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:52.692 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.693 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:42:52.693 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:52.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.694 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.694 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:42:52.694 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:52.694 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.695 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.695 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:52.696 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.697 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:42:52.697 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:52.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.698 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.698 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:42:52.698 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:52.699 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.699 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.699 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:52.700 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.700 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.700 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:42:52.701 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:52.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.702 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.702 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:42:52.703 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:42:52.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.703 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.704 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:42:52.704 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:42:52.704 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:42:52.704 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:42:52.704 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:52.705 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:52.705 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:52.705 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.706 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:52.706 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:42:52.706 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:52.707 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:52.707 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:52.707 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:52.707 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:52.707 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:52.708 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:52.708 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:52.708 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:52.708 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:52.709 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:52.709 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.709 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.710 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:42:52.710 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:52.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.711 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.711 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:42:52.711 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:52.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.711 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.712 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:52.712 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.713 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.713 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:42:52.713 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:52.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.714 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.714 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:42:52.715 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:52.715 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.715 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.715 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:52.716 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.716 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.716 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:42:52.717 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:52.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.717 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.718 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:42:52.718 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:52.719 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.719 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.720 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:52.720 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.721 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.721 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:42:52.721 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:52.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.723 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:42:52.723 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:42:52.724 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.724 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.725 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:42:52.725 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:42:52.725 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:42:52.726 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:42:52.726 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:52.726 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:52.727 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:52.727 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.728 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:52.728 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:42:52.728 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:52.729 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:52.729 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:52.729 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:52.730 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:52.730 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:52.730 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:52.731 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:52.731 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:52.731 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:52.732 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:52.732 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.733 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.733 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:42:52.733 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:52.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.734 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.735 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:42:52.735 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:52.735 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.736 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.736 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:52.737 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.737 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.738 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:42:52.738 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:52.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.739 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.739 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:42:52.740 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:52.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.740 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.741 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:52.741 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.742 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.742 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:42:52.743 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:52.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.744 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.744 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:42:52.745 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:52.745 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.745 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.746 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:52.747 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.747 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.747 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:42:52.748 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:52.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.749 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.749 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:42:52.749 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:42:52.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.750 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.750 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:42:52.751 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:42:52.751 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:42:52.752 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:42:52.752 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:52.752 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:52.753 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:52.753 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.754 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:52.754 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:42:52.755 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:42:52.755 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:42:52.755 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:42:52.755 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:42:52.756 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:42:52.756 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:42:52.756 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:42:52.756 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 147, 2022-07-21T01:42:52.757 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:42:52.757 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:42:52.757 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:42:52.758 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.758 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.759 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:42:52.759 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:52.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.760 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:42:52.761 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:52.761 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.761 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.762 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:42:52.762 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.763 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.763 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:42:52.763 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:52.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.764 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.764 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:42:52.764 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:52.765 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.765 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.766 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:42:52.766 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.766 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.767 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:42:52.767 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:52.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.768 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:42:52.769 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:52.769 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.769 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.770 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:42:52.770 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:42:52.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.771 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:42:52.771 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:42:52.772 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:52.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.773 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:42:52.773 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:42:52.773 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:42:52.774 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.774 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.775 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:42:52.775 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:42:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:42:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:42:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:42:52.777 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:42:52.777 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:42:52.777 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:52.778 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.778 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:52.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:42:52.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:42:52.779 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:52.780 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:52.780 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:52.781 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:52.781 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:52.781 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:42:52.782 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:52.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.783 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:42:52.783 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:52.783 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:52.784 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:52.784 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:52.785 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:52.785 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:42:52.786 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:52.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.787 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:42:52.788 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:52.788 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:52.789 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:52.789 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:52.790 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:52.790 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:42:52.791 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:52.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:42:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:52.794 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:52.794 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:52.800 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:52.800 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:52.801 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:42:52.801 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:52.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.803 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:42:52.803 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:52.804 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:52.804 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:52.805 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:52.805 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:52.805 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:42:52.807 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:52.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.808 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:42:52.808 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:52.809 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:52.809 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:52.809 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:52.809 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:52.810 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:42:52.810 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:52.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.811 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:42:52.811 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:52.811 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:52.812 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:52.812 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:52.812 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:52.813 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:42:52.813 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:52.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:52.814 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:42:52.814 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:42:52.814 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:42:52.814 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:42:52.815 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:42:52.815 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:42:52.815 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:42:52.815 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:42:52.815 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.816 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:42:52.816 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:42:52.816 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:42:52.817 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:42:52.817 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:42:52.817 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": { 2022-07-21T01:42:52.818 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.143:6833/1158093851": "2022-07-22T01:42:46.559120+0000", 2022-07-21T01:42:52.818 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.143:6832/1158093851": "2022-07-22T01:42:46.559120+0000" 2022-07-21T01:42:52.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.819 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:42:52.819 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:42:52.819 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:42:52.820 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:52.820 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:42:52.820 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:42:52.820 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:52.821 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:42:52.821 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:52.821 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:52.821 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:42:52.822 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:42:52.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.822 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:42:52.822 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:42:52.823 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:42:52.823 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:42:52.823 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:42:52.824 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:42:52.824 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:42:52.824 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:52.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:52.825 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:42:52.825 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:42:52.825 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:42:52.825 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:42:52.826 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:42:52.826 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:42:52.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7524053b30 msgr2=0x7f7524055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7524053b30 0x7f7524055ff0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f754400b000 tx=0x7f7544006df0).stop 2022-07-21T01:42:52.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75480b6710 msgr2=0x7f7548140930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75480b6710 0x7f7548140930 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f754812b2c0 tx=0x7f7540002160).stop 2022-07-21T01:42:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 shutdown_connections 2022-07-21T01:42:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75480b6710 0x7f7548140930 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75480b4550 0x7f7548140440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f7524053b30 0x7f7524055ff0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 --2- 172.21.15.3:0/577415950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7548061ec0 0x7f75481449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 >> 172.21.15.3:0/577415950 conn(0x7f754805c620 msgr2=0x7f75481489b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 shutdown_connections 2022-07-21T01:42:52.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.555+0000 7f754f89f700 1 -- 172.21.15.3:0/577415950 wait complete. 2022-07-21T01:42:52.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:52.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94f0cf8700 1 Processor -- start 2022-07-21T01:42:52.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94f0cf8700 1 -- start start 2022-07-21T01:42:52.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94f0cf8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f94ec0b1160 0x7f94ec1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94f0cf8700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec0b04e0 con 0x7f94ec148530 2022-07-21T01:42:52.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94f0cf8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec0b0780 con 0x7f94ec0b1f40 2022-07-21T01:42:52.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94f0cf8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ec0a9250 con 0x7f94ec0b1160 2022-07-21T01:42:52.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e9d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f94ec148530 0x7f94ec0a7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60094/0 (socket says 172.21.15.3:60094) 2022-07-21T01:42:52.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e9d9b700 1 -- 172.21.15.3:0/584639430 learned_addr learned my addr 172.21.15.3:0/584639430 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:52.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ead9d700 1 --2- 172.21.15.3:0/584639430 >> v2:172.21.15.3:3300/0 conn(0x7f94ec0b1160 0x7f94ec1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876139684 0 0) 0x7f94ec0b04e0 con 0x7f94ec148530 2022-07-21T01:42:52.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d0003260 con 0x7f94ec148530 2022-07-21T01:42:52.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3528425675 0 0) 0x7f94d0003260 con 0x7f94ec148530 2022-07-21T01:42:52.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 >> v2:172.21.15.3:3300/0 conn(0x7f94ec0b1160 msgr2=0x7f94ec1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:52.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 --2- 172.21.15.3:0/584639430 >> v2:172.21.15.3:3300/0 conn(0x7f94ec0b1160 0x7f94ec1448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 >> v1:172.21.15.143:6789/0 conn(0x7f94ec0b1f40 legacy=0x7f94ec0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec0b0a20 con 0x7f94ec148530 2022-07-21T01:42:52.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f94e40019e0 con 0x7f94ec148530 2022-07-21T01:42:52.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 >> v1:172.21.15.3:6789/0 conn(0x7f94ec148530 legacy=0x7f94ec0a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94d0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94d0004500 0x7f94d0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d0004e70 0x7f94d000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d000bcf0 con 0x7f94d0003750 2022-07-21T01:42:52.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d000be60 con 0x7f94d0004500 2022-07-21T01:42:52.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d000c020 con 0x7f94d0004e70 2022-07-21T01:42:52.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ea59c700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94d0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94e9d9b700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94d0004500 0x7f94d0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ead9d700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d0004e70 0x7f94d000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ea59c700 1 -- 172.21.15.3:0/584639430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d0004e70 msgr2=0x7f94d000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:52.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ea59c700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d0004e70 0x7f94d000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ea59c700 1 -- 172.21.15.3:0/584639430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94d0004500 msgr2=0x7f94d0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:52.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ea59c700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94d0004500 0x7f94d0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ea59c700 1 -- 172.21.15.3:0/584639430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d0003260 con 0x7f94d0003750 2022-07-21T01:42:52.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ead9d700 1 -- 172.21.15.3:0/584639430 reap_dead start 2022-07-21T01:42:52.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.951+0000 7f94ea59c700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94d0003b70 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0x7f94ec0b3300 tx=0x7f94d4007a00).ready entity=mon.0 client_cookie=e8db8e3f29451c33 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:52.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d4007d10 con 0x7f94d0003750 2022-07-21T01:42:52.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d4007030 con 0x7f94d0003750 2022-07-21T01:42:52.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94e959a700 1 -- 172.21.15.3:0/584639430 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d4007390 con 0x7f94d0003750 2022-07-21T01:42:52.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94f0cf8700 1 -- 172.21.15.3:0/584639430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 msgr2=0x7f94d0003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94f0cf8700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94d0003b70 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0x7f94ec0b3300 tx=0x7f94d4007a00).stop 2022-07-21T01:42:52.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94f0cf8700 1 -- 172.21.15.3:0/584639430 shutdown_connections 2022-07-21T01:42:52.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94f0cf8700 1 --2- 172.21.15.3:0/584639430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94d0003b70 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94f0cf8700 1 -- 172.21.15.3:0/584639430 >> 172.21.15.3:0/584639430 conn(0x7f94ec05f840 msgr2=0x7f94ec05fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:52.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94f0cf8700 1 -- 172.21.15.3:0/584639430 shutdown_connections 2022-07-21T01:42:52.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.955+0000 7f94f0cf8700 1 -- 172.21.15.3:0/584639430 wait complete. 2022-07-21T01:42:52.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 Processor -- start 2022-07-21T01:42:52.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 -- start start 2022-07-21T01:42:52.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94ec0abe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94ec14b890 0x7f94ec0ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94ec14a7b0 0x7f94ec14ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec0ab4b0 con 0x7f94d0003750 2022-07-21T01:42:52.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec0ab750 con 0x7f94ec14a7b0 2022-07-21T01:42:52.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec0ab9f0 con 0x7f94ec14b890 2022-07-21T01:42:52.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ea59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94ec0abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ea59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94ec0abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51828/0 (socket says 172.21.15.3:51828) 2022-07-21T01:42:52.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ea59c700 1 -- 172.21.15.3:0/632640243 learned_addr learned my addr 172.21.15.3:0/632640243 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:52.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94e9d9b700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94ec14b890 0x7f94ec0ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ead9d700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94ec14a7b0 0x7f94ec14ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ea59c700 1 -- 172.21.15.3:0/632640243 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94ec14b890 msgr2=0x7f94ec0ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ea59c700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94ec14b890 0x7f94ec0ae300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ea59c700 1 -- 172.21.15.3:0/632640243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94ec14a7b0 msgr2=0x7f94ec14ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:52.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ea59c700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94ec14a7b0 0x7f94ec14ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:52.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ea59c700 1 -- 172.21.15.3:0/632640243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec0ab130 con 0x7f94d0003750 2022-07-21T01:42:52.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94ea59c700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94ec0abe10 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f94ec0b3300 tx=0x7f94d4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:52.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94e37fe700 1 -- 172.21.15.3:0/632640243 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d4020450 con 0x7f94d0003750 2022-07-21T01:42:52.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94e37fe700 1 -- 172.21.15.3:0/632640243 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d40205b0 con 0x7f94d0003750 2022-07-21T01:42:52.984 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ec0aea00 con 0x7f94d0003750 2022-07-21T01:42:52.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94e37fe700 1 -- 172.21.15.3:0/632640243 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d4020910 con 0x7f94d0003750 2022-07-21T01:42:52.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.959+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ec143d40 con 0x7f94d0003750 2022-07-21T01:42:52.985 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.963+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ec0abbb0 con 0x7f94d0003750 2022-07-21T01:42:52.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.963+0000 7f94e37fe700 1 -- 172.21.15.3:0/632640243 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f94d401d070 con 0x7f94d0003750 2022-07-21T01:42:52.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.963+0000 7f94e37fe700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f94c4053a10 0x7f94c4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:52.986 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.963+0000 7f94e37fe700 1 -- 172.21.15.3:0/632640243 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f94d406cb60 con 0x7f94d0003750 2022-07-21T01:42:52.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.963+0000 7f94e9d9b700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f94c4053a10 0x7f94c4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:52.987 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.963+0000 7f94e9d9b700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f94c4053a10 0x7f94c4055ed0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f94e4005000 tx=0x7f94ec0b0910).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:52.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:52.967+0000 7f94e37fe700 1 -- 172.21.15.3:0/632640243 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f94d4043ba0 con 0x7f94d0003750 2022-07-21T01:42:53.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94ec0b0cc0 con 0x7f94d0003750 2022-07-21T01:42:53.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94e37fe700 1 -- 172.21.15.3:0/632640243 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+1559 (secure 0 0 0) 0x7f94d40471c0 con 0x7f94d0003750 2022-07-21T01:42:53.138 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:53.139 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":78,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2873064851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2873064851},{"type":"v1","addr":"172.21.15.143:6833","nonce":2873064851}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":78},{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/1563084587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1563084587},{"type":"v1","addr":"172.21.15.3:6835","nonce":1563084587}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1403026914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1403026914},{"type":"v1","addr":"172.21.15.3:6837","nonce":1403026914}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[]} 2022-07-21T01:42:53.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f94c4053a10 msgr2=0x7f94c4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f94c4053a10 0x7f94c4055ed0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f94e4005000 tx=0x7f94ec0b0910).stop 2022-07-21T01:42:53.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 msgr2=0x7f94ec0abe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94ec0abe10 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f94ec0b3300 tx=0x7f94d4015000).stop 2022-07-21T01:42:53.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 shutdown_connections 2022-07-21T01:42:53.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f94c4053a10 0x7f94c4055ed0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94ec14a7b0 0x7f94ec14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94ec14b890 0x7f94ec0ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 --2- 172.21.15.3:0/632640243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d0003750 0x7f94ec0abe10 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 >> 172.21.15.3:0/632640243 conn(0x7f94ec05f840 msgr2=0x7f94ec144890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:53.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 shutdown_connections 2022-07-21T01:42:53.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.135+0000 7f94f0cf8700 1 -- 172.21.15.3:0/632640243 wait complete. 2022-07-21T01:42:53.145 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 78 2022-07-21T01:42:53.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfbcdff700 1 Processor -- start 2022-07-21T01:42:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfbcdff700 1 -- start start 2022-07-21T01:42:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfbcdff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 0x7fcfb80afd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfbcdff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcfb80b1920 0x7fcfb80b0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfbcdff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb80b0740 con 0x7fcfb80b2710 2022-07-21T01:42:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfbcdff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb80adf10 con 0x7fcfb80b1920 2022-07-21T01:42:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfbcdff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb80ae1b0 con 0x7fcfb80b13f0 2022-07-21T01:42:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfb7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcfb80b1920 0x7fcfb80b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfb7fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcfb80b1920 0x7fcfb80b0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51836/0 (socket says 172.21.15.3:51836) 2022-07-21T01:42:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.263+0000 7fcfb7fff700 1 -- 172.21.15.3:0/563279321 learned_addr learned my addr 172.21.15.3:0/563279321 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb6ffd700 1 --2- 172.21.15.3:0/563279321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 0x7fcfb80afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb7fff700 1 -- 172.21.15.3:0/563279321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 msgr2=0x7fcfb80afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb7fff700 1 --2- 172.21.15.3:0/563279321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 0x7fcfb80afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb7fff700 1 -- 172.21.15.3:0/563279321 >> v1:172.21.15.143:6789/0 conn(0x7fcfb80b2710 legacy=0x7fcfb80abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb7fff700 1 -- 172.21.15.3:0/563279321 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb80a9520 con 0x7fcfb80b1920 2022-07-21T01:42:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb7fff700 1 --2- 172.21.15.3:0/563279321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b1920 0x7fcfb80b0250 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7fcfb8067bc0 tx=0x7fcfb000b530).ready entity=mon.0 client_cookie=41aa9e254a40730 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:53.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb67fc700 1 -- 172.21.15.3:0/563279321 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfb000dde0 con 0x7fcfb80b1920 2022-07-21T01:42:53.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb67fc700 1 -- 172.21.15.3:0/563279321 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb0010470 con 0x7fcfb80b1920 2022-07-21T01:42:53.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb67fc700 1 -- 172.21.15.3:0/563279321 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfb00107d0 con 0x7fcfb80b1920 2022-07-21T01:42:53.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 -- 172.21.15.3:0/563279321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b1920 msgr2=0x7fcfb80b0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 --2- 172.21.15.3:0/563279321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b1920 0x7fcfb80b0250 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7fcfb8067bc0 tx=0x7fcfb000b530).stop 2022-07-21T01:42:53.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 -- 172.21.15.3:0/563279321 shutdown_connections 2022-07-21T01:42:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 --2- 172.21.15.3:0/563279321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b1920 0x7fcfb80b0250 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 --2- 172.21.15.3:0/563279321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 0x7fcfb80afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 -- 172.21.15.3:0/563279321 >> 172.21.15.3:0/563279321 conn(0x7fcfb8060310 msgr2=0x7fcfb80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:53.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 -- 172.21.15.3:0/563279321 shutdown_connections 2022-07-21T01:42:53.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 -- 172.21.15.3:0/563279321 wait complete. 2022-07-21T01:42:53.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 Processor -- start 2022-07-21T01:42:53.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 -- start start 2022-07-21T01:42:53.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 0x7fcfb813afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb80b1920 0x7fcfb813b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b2710 0x7fcfb813cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb813c330 con 0x7fcfb80b2710 2022-07-21T01:42:53.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb813c590 con 0x7fcfb80b1920 2022-07-21T01:42:53.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfbcdff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb813c830 con 0x7fcfb80b13f0 2022-07-21T01:42:53.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b2710 0x7fcfb813cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b2710 0x7fcfb813cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51842/0 (socket says 172.21.15.3:51842) 2022-07-21T01:42:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb7fff700 1 -- 172.21.15.3:0/63472415 learned_addr learned my addr 172.21.15.3:0/63472415 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb6ffd700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb80b1920 0x7fcfb813b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.267+0000 7fcfb77fe700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 0x7fcfb813afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb7fff700 1 -- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 msgr2=0x7fcfb813afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb7fff700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 0x7fcfb813afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.289 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb7fff700 1 -- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb80b1920 msgr2=0x7fcfb813b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb7fff700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb80b1920 0x7fcfb813b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb7fff700 1 -- 172.21.15.3:0/63472415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb000b7d0 con 0x7fcfb80b2710 2022-07-21T01:42:53.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb77fe700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 0x7fcfb813afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:53.290 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb6ffd700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb80b1920 0x7fcfb813b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:53.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb7fff700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b2710 0x7fcfb813cf70 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7fcfb8067bc0 tx=0x7fcfb000b530).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:53.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb4ff9700 1 -- 172.21.15.3:0/63472415 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfb00032f0 con 0x7fcfb80b2710 2022-07-21T01:42:53.291 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb4ff9700 1 -- 172.21.15.3:0/63472415 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb0018610 con 0x7fcfb80b2710 2022-07-21T01:42:53.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb4ff9700 1 -- 172.21.15.3:0/63472415 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfb0018970 con 0x7fcfb80b2710 2022-07-21T01:42:53.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb813cad0 con 0x7fcfb80b2710 2022-07-21T01:42:53.292 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb81514e0 con 0x7fcfb80b2710 2022-07-21T01:42:53.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb4ff9700 1 -- 172.21.15.3:0/63472415 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcfb002b070 con 0x7fcfb80b2710 2022-07-21T01:42:53.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb80574e0 con 0x7fcfb80b2710 2022-07-21T01:42:53.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb4ff9700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf980539e0 0x7fcf98055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb4ff9700 1 -- 172.21.15.3:0/63472415 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fcfb006fa50 con 0x7fcfb80b2710 2022-07-21T01:42:53.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb77fe700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf980539e0 0x7fcf98055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.271+0000 7fcfb77fe700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf980539e0 0x7fcf98055ea0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7fcfa800b000 tx=0x7fcfa8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:53.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.275+0000 7fcfb4ff9700 1 -- 172.21.15.3:0/63472415 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcfb0046b30 con 0x7fcfb80b2710 2022-07-21T01:42:53.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.435+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfb813cc60 con 0x7fcfb80b2710 2022-07-21T01:42:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfb4ff9700 1 -- 172.21.15.3:0/63472415 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+1559 (secure 0 0 0) 0x7fcfb004a110 con 0x7fcfb80b2710 2022-07-21T01:42:53.441 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:53.442 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":78,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2873064851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2873064851},{"type":"v1","addr":"172.21.15.143:6833","nonce":2873064851}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":78},{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6835/1563084587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":1563084587},{"type":"v1","addr":"172.21.15.3:6835","nonce":1563084587}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8406,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.3:6837/1403026914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1403026914},{"type":"v1","addr":"172.21.15.3:6837","nonce":1403026914}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[]} 2022-07-21T01:42:53.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf980539e0 msgr2=0x7fcf98055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf980539e0 0x7fcf98055ea0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7fcfa800b000 tx=0x7fcfa8009040).stop 2022-07-21T01:42:53.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b2710 msgr2=0x7fcfb813cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b2710 0x7fcfb813cf70 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7fcfb8067bc0 tx=0x7fcfb000b530).stop 2022-07-21T01:42:53.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 shutdown_connections 2022-07-21T01:42:53.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfb80b2710 0x7fcfb813cf70 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf980539e0 0x7fcf98055ea0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb80b1920 0x7fcfb813b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 --2- 172.21.15.3:0/63472415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfb80b13f0 0x7fcfb813afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 >> 172.21.15.3:0/63472415 conn(0x7fcfb8060310 msgr2=0x7fcfb813fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:53.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 shutdown_connections 2022-07-21T01:42:53.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.439+0000 7fcfbcdff700 1 -- 172.21.15.3:0/63472415 wait complete. 2022-07-21T01:42:53.448 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 78 2022-07-21T01:42:53.455 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-21T01:42:53.456 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-21T01:42:53.456 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T01:42:53.457 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:42:53.457 INFO:tasks.ceph.mds.c:Restarting daemon 2022-07-21T01:42:53.458 INFO:tasks.ceph.mds.c:Stopping old one... 2022-07-21T01:42:53.458 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T01:42:53.459 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:42:53.459 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-21T01:42:53.460 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-21T01:42:53.460 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T01:42:53.460 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T01:42:53.471 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T01:42:53.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-21T01:42:53.475 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T01:42:53.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-07-21T01:42:53.477 INFO:tasks.ceph.mds.a:Started 2022-07-21T01:42:53.477 DEBUG:teuthology.parallel:result is None 2022-07-21T01:42:53.479 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T01:42:53.479 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-21T01:42:53.483 INFO:tasks.ceph.mds.b:Started 2022-07-21T01:42:53.483 DEBUG:teuthology.parallel:result is None 2022-07-21T01:42:53.524 INFO:tasks.ceph.mds.c:Started 2022-07-21T01:42:53.525 DEBUG:teuthology.parallel:result is None 2022-07-21T01:42:53.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-21T01:42:53.549 INFO:tasks.ceph.mds.a.smithi003.stdout:starting mds.a at 2022-07-21T01:42:53.560 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2022-07-21T01:42:53.597 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:42:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 Processor -- start 2022-07-21T01:42:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 -- start start 2022-07-21T01:42:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3c740619b0 0x7f3c740b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 0x7f3c740b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c74061d90 con 0x7f3c740b6ee0 2022-07-21T01:42:53.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c74143fa0 con 0x7f3c740619b0 2022-07-21T01:42:53.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c74142f60 con 0x7f3c740b5ed0 2022-07-21T01:42:53.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c73fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f3c740619b0 0x7f3c740b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c737fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3c740b6ee0 0x7f3c74143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60152/0 (socket says 172.21.15.3:60152) 2022-07-21T01:42:53.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c737fe700 1 -- 172.21.15.3:0/2838994736 learned_addr learned my addr 172.21.15.3:0/2838994736 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:53.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c789b1700 1 --2- 172.21.15.3:0/2838994736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 0x7f3c740b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c72ffd700 1 -- 172.21.15.3:0/2838994736 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330054285 0 0) 0x7f3c74061d90 con 0x7f3c740b6ee0 2022-07-21T01:42:53.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c72ffd700 1 -- 172.21.15.3:0/2838994736 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c54003260 con 0x7f3c740b6ee0 2022-07-21T01:42:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c73fff700 1 -- 172.21.15.3:0/2838994736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 msgr2=0x7f3c740b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c73fff700 1 --2- 172.21.15.3:0/2838994736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 0x7f3c740b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c73fff700 1 -- 172.21.15.3:0/2838994736 >> v1:172.21.15.3:6789/0 conn(0x7f3c740b6ee0 legacy=0x7f3c74143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c73fff700 1 -- 172.21.15.3:0/2838994736 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c740ba5a0 con 0x7f3c740619b0 2022-07-21T01:42:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c789b1700 1 --2- 172.21.15.3:0/2838994736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 0x7f3c740b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c73fff700 1 --2- 172.21.15.3:0/2838994736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740619b0 0x7f3c740b48a0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f3c740b6880 tx=0x7f3c640091d0).ready entity=mon.0 client_cookie=1fcda642ed86baa4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c72ffd700 1 -- 172.21.15.3:0/2838994736 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6400b680 con 0x7f3c740619b0 2022-07-21T01:42:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c72ffd700 1 -- 172.21.15.3:0/2838994736 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6400b7e0 con 0x7f3c740619b0 2022-07-21T01:42:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c72ffd700 1 -- 172.21.15.3:0/2838994736 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6400b950 con 0x7f3c740619b0 2022-07-21T01:42:53.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 -- 172.21.15.3:0/2838994736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740619b0 msgr2=0x7f3c740b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 --2- 172.21.15.3:0/2838994736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740619b0 0x7f3c740b48a0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f3c740b6880 tx=0x7f3c640091d0).stop 2022-07-21T01:42:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 -- 172.21.15.3:0/2838994736 shutdown_connections 2022-07-21T01:42:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 --2- 172.21.15.3:0/2838994736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 0x7f3c740b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 --2- 172.21.15.3:0/2838994736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740619b0 0x7f3c740b48a0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 -- 172.21.15.3:0/2838994736 >> 172.21.15.3:0/2838994736 conn(0x7f3c7406af80 msgr2=0x7f3c7405fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:53.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 -- 172.21.15.3:0/2838994736 shutdown_connections 2022-07-21T01:42:53.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.635+0000 7f3c791b2700 1 -- 172.21.15.3:0/2838994736 wait complete. 2022-07-21T01:42:53.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 Processor -- start 2022-07-21T01:42:53.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 -- start start 2022-07-21T01:42:53.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c740619b0 0x7f3c740b9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 0x7f3c741494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740b6ee0 0x7f3c74155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7413c020 con 0x7f3c740b6ee0 2022-07-21T01:42:53.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c74144460 con 0x7f3c740619b0 2022-07-21T01:42:53.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c740ade20 con 0x7f3c740b5ed0 2022-07-21T01:42:53.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c789b1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740b6ee0 0x7f3c74155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c789b1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740b6ee0 0x7f3c74155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51880/0 (socket says 172.21.15.3:51880) 2022-07-21T01:42:53.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c789b1700 1 -- 172.21.15.3:0/2443878129 learned_addr learned my addr 172.21.15.3:0/2443878129 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:53.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c737fe700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 0x7f3c741494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c73fff700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c740619b0 0x7f3c740b9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c789b1700 1 -- 172.21.15.3:0/2443878129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 msgr2=0x7f3c741494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c789b1700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 0x7f3c741494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c789b1700 1 -- 172.21.15.3:0/2443878129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c740619b0 msgr2=0x7f3c740b9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:53.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c789b1700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c740619b0 0x7f3c740b9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:53.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c789b1700 1 -- 172.21.15.3:0/2443878129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c64009470 con 0x7f3c740b6ee0 2022-07-21T01:42:53.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c73fff700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c740619b0 0x7f3c740b9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:53.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c789b1700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740b6ee0 0x7f3c74155fe0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0x7f3c6c006940 tx=0x7f3c6c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:53.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c717fa700 1 -- 172.21.15.3:0/2443878129 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6c011780 con 0x7f3c740b6ee0 2022-07-21T01:42:53.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c717fa700 1 -- 172.21.15.3:0/2443878129 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c0118e0 con 0x7f3c740b6ee0 2022-07-21T01:42:53.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7414a170 con 0x7f3c740b6ee0 2022-07-21T01:42:53.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c717fa700 1 -- 172.21.15.3:0/2443878129 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6c011c40 con 0x7f3c740b6ee0 2022-07-21T01:42:53.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.639+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c741488e0 con 0x7f3c740b6ee0 2022-07-21T01:42:53.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.643+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c740574e0 con 0x7f3c740b6ee0 2022-07-21T01:42:53.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.643+0000 7f3c717fa700 1 -- 172.21.15.3:0/2443878129 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3c6c012420 con 0x7f3c740b6ee0 2022-07-21T01:42:53.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.643+0000 7f3c717fa700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c58053a70 0x7f3c58055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:53.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.643+0000 7f3c717fa700 1 -- 172.21.15.3:0/2443878129 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f3c6c06bb90 con 0x7f3c740b6ee0 2022-07-21T01:42:53.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.647+0000 7f3c73fff700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c58053a70 0x7f3c58055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:53.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.647+0000 7f3c73fff700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c58053a70 0x7f3c58055f30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f3c64019000 tx=0x7f3c64016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:53.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.651+0000 7f3c717fa700 1 -- 172.21.15.3:0/2443878129 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3c6c042bd0 con 0x7f3c740b6ee0 2022-07-21T01:42:53.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:53.791+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f3c740b4390 con 0x7f3c740b6ee0 2022-07-21T01:42:54.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.623+0000 7f3c717fa700 1 -- 172.21.15.3:0/2443878129 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v157) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3c6c0461f0 con 0x7f3c740b6ee0 2022-07-21T01:42:54.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c58053a70 msgr2=0x7f3c58055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:54.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c58053a70 0x7f3c58055f30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f3c64019000 tx=0x7f3c64016000).stop 2022-07-21T01:42:54.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740b6ee0 msgr2=0x7f3c74155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:54.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740b6ee0 0x7f3c74155fe0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0x7f3c6c006940 tx=0x7f3c6c008040).stop 2022-07-21T01:42:54.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 shutdown_connections 2022-07-21T01:42:54.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c740b6ee0 0x7f3c74155fe0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c740b5ed0 0x7f3c741494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c58053a70 0x7f3c58055f30 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 --2- 172.21.15.3:0/2443878129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c740619b0 0x7f3c740b9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 >> 172.21.15.3:0/2443878129 conn(0x7f3c7406af80 msgr2=0x7f3c74142090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:54.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 shutdown_connections 2022-07-21T01:42:54.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.627+0000 7f3c791b2700 1 -- 172.21.15.3:0/2443878129 wait complete. 2022-07-21T01:42:54.635 INFO:teuthology.orchestra.run.smithi003.stderr: removed all blacklist entries 2022-07-21T01:42:54.647 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2022-07-21T01:42:54.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-21T01:42:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbf59e700 1 Processor -- start 2022-07-21T01:42:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbf59e700 1 -- start start 2022-07-21T01:42:54.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbf59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b2710 0x7fdcc014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:54.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbf59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcc00b01a0 0x7fdcc014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbf59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdcc00afd60 0x7fdcc00a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbf59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc00a7250 con 0x7fdcc00b01a0 2022-07-21T01:42:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbf59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc00a7480 con 0x7fdcc00afd60 2022-07-21T01:42:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbf59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00a7720 con 0x7fdcc00b2710 2022-07-21T01:42:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbdd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcc00b01a0 0x7fdcc014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbdd9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdcc00b01a0 0x7fdcc014b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51888/0 (socket says 172.21.15.3:51888) 2022-07-21T01:42:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.751+0000 7fdcbdd9b700 1 -- 172.21.15.3:0/3924841648 learned_addr learned my addr 172.21.15.3:0/3924841648 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbed9d700 1 --2- 172.21.15.3:0/3924841648 >> v2:172.21.15.143:3300/0 conn(0x7fdcc00afd60 0x7fdcc00a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:54.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbe59c700 1 --2- 172.21.15.3:0/3924841648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b2710 0x7fdcc014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:54.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbdd9b700 1 -- 172.21.15.3:0/3924841648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b2710 msgr2=0x7fdcc014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:54.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbdd9b700 1 --2- 172.21.15.3:0/3924841648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b2710 0x7fdcc014afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbdd9b700 1 -- 172.21.15.3:0/3924841648 >> v2:172.21.15.143:3300/0 conn(0x7fdcc00afd60 msgr2=0x7fdcc00a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:54.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbdd9b700 1 --2- 172.21.15.3:0/3924841648 >> v2:172.21.15.143:3300/0 conn(0x7fdcc00afd60 0x7fdcc00a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbdd9b700 1 -- 172.21.15.3:0/3924841648 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00a79c0 con 0x7fdcc00b01a0 2022-07-21T01:42:54.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbed9d700 1 --2- 172.21.15.3:0/3924841648 >> v2:172.21.15.143:3300/0 conn(0x7fdcc00afd60 0x7fdcc00a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:54.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbdd9b700 1 --2- 172.21.15.3:0/3924841648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00b01a0 0x7fdcc014b4c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7fdcb0002f70 tx=0x7fdcb0006eb0).ready entity=mon.0 client_cookie=6a112e0b7388e6de server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:54.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbd59a700 1 -- 172.21.15.3:0/3924841648 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb000d570 con 0x7fdcc00b01a0 2022-07-21T01:42:54.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbd59a700 1 -- 172.21.15.3:0/3924841648 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb0000e90 con 0x7fdcc00b01a0 2022-07-21T01:42:54.764 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbd59a700 1 -- 172.21.15.3:0/3924841648 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb000d900 con 0x7fdcc00b01a0 2022-07-21T01:42:54.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbf59e700 1 -- 172.21.15.3:0/3924841648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00b01a0 msgr2=0x7fdcc014b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:54.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/3924841648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00b01a0 0x7fdcc014b4c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7fdcb0002f70 tx=0x7fdcb0006eb0).stop 2022-07-21T01:42:54.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbf59e700 1 -- 172.21.15.3:0/3924841648 shutdown_connections 2022-07-21T01:42:54.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/3924841648 >> v2:172.21.15.143:3300/0 conn(0x7fdcc00afd60 0x7fdcc00a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/3924841648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00b01a0 0x7fdcc014b4c0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.766 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/3924841648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b2710 0x7fdcc014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbf59e700 1 -- 172.21.15.3:0/3924841648 >> 172.21.15.3:0/3924841648 conn(0x7fdcc0060310 msgr2=0x7fdcc00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:54.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbf59e700 1 -- 172.21.15.3:0/3924841648 shutdown_connections 2022-07-21T01:42:54.767 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.755+0000 7fdcbf59e700 1 -- 172.21.15.3:0/3924841648 wait complete. 2022-07-21T01:42:54.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbf59e700 1 Processor -- start 2022-07-21T01:42:54.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbf59e700 1 -- start start 2022-07-21T01:42:54.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbf59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00afd60 0x7fdcc00af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:54.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbf59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b01a0 0x7fdcc00b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:54.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbf59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcc00b2710 0x7fdcc00b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:54.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbf59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc014d490 con 0x7fdcc00afd60 2022-07-21T01:42:54.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbf59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00b34e0 con 0x7fdcc00b2710 2022-07-21T01:42:54.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbf59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00b3990 con 0x7fdcc00b01a0 2022-07-21T01:42:54.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00afd60 0x7fdcc00af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00afd60 0x7fdcc00af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51894/0 (socket says 172.21.15.3:51894) 2022-07-21T01:42:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbe59c700 1 -- 172.21.15.3:0/1481477349 learned_addr learned my addr 172.21.15.3:0/1481477349 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbdd9b700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b01a0 0x7fdcc00b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.759+0000 7fdcbed9d700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcc00b2710 0x7fdcc00b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbe59c700 1 -- 172.21.15.3:0/1481477349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b01a0 msgr2=0x7fdcc00b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbe59c700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b01a0 0x7fdcc00b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbe59c700 1 -- 172.21.15.3:0/1481477349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcc00b2710 msgr2=0x7fdcc00b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbe59c700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcc00b2710 0x7fdcc00b4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbe59c700 1 -- 172.21.15.3:0/1481477349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb8009470 con 0x7fdcc00afd60 2022-07-21T01:42:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbe59c700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00afd60 0x7fdcc00af3d0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7fdcb400c000 tx=0x7fdcb4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcaf7fe700 1 -- 172.21.15.3:0/1481477349 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb4010780 con 0x7fdcc00afd60 2022-07-21T01:42:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcaf7fe700 1 -- 172.21.15.3:0/1481477349 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb40108e0 con 0x7fdcc00afd60 2022-07-21T01:42:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb000b1e0 con 0x7fdcc00afd60 2022-07-21T01:42:54.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcaf7fe700 1 -- 172.21.15.3:0/1481477349 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb4010c40 con 0x7fdcc00afd60 2022-07-21T01:42:54.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc00b3ed0 con 0x7fdcc00afd60 2022-07-21T01:42:54.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc00b46e0 con 0x7fdcc00afd60 2022-07-21T01:42:54.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcaf7fe700 1 -- 172.21.15.3:0/1481477349 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdcb401c460 con 0x7fdcc00afd60 2022-07-21T01:42:54.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcaf7fe700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc9c0539e0 0x7fdc9c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:54.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcaf7fe700 1 -- 172.21.15.3:0/1481477349 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fdcb406c1d0 con 0x7fdcc00afd60 2022-07-21T01:42:54.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.763+0000 7fdcbdd9b700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc9c0539e0 0x7fdc9c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:54.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.767+0000 7fdcbdd9b700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc9c0539e0 0x7fdc9c055ea0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7fdcb001c000 tx=0x7fdcb0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.771+0000 7fdcaf7fe700 1 -- 172.21.15.3:0/1481477349 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdcb40431d0 con 0x7fdcc00afd60 2022-07-21T01:42:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.935+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fdcc00574e0 con 0x7fdcc00afd60 2022-07-21T01:42:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcaf7fe700 1 -- 172.21.15.3:0/1481477349 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v37) v1 ==== 80+0+4255 (secure 0 0 0) 0x7fdcb40467f0 con 0x7fdcc00afd60 2022-07-21T01:42:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:54.942 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:42:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_dump": [ 2022-07-21T01:42:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.a", 2022-07-21T01:42:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiYMnrCxAAm0sHEigbn3GONJmNYCDuSw==", 2022-07-21T01:42:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.b", 2022-07-21T01:42:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiTR1rFRAAjzRE3jyYG75c1g/977NjYQ==", 2022-07-21T01:42:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mds.c", 2022-07-21T01:42:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhixNb6ERAAFKxC1lefVR9Clbo6Q2E+Fg==", 2022-07-21T01:42:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.0", 2022-07-21T01:42:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCzrNhiSbfFBRAAumEed1kom1xlSC/VmSoYyQ==", 2022-07-21T01:42:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.1", 2022-07-21T01:42:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC1rNhi1QsOJRAA8V41CGSEclSdcJcfa69U9g==", 2022-07-21T01:42:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.2", 2022-07-21T01:42:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC4rNhi4LI/CxAAXlk2ao2fHykyAWXatjdWbg==", 2022-07-21T01:42:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.3", 2022-07-21T01:42:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC6rNhi+cV1LRAA8oNRlb3RoJ0ZdxuYu7f0ag==", 2022-07-21T01:42:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.4", 2022-07-21T01:42:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQC/rNhiqsVXMxAAgXQi5gCX0WihGYZfVBRn0w==", 2022-07-21T01:42:54.960 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.960 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.960 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.5", 2022-07-21T01:42:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDCrNhijz5XFxAA8KgBRyvyyc/6bOap3CZgaw==", 2022-07-21T01:42:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.6", 2022-07-21T01:42:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDErNhiFIZOOBAANyb8j2vlUo5HKMW+Rlrggg==", 2022-07-21T01:42:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "osd.7", 2022-07-21T01:42:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQDHrNhiXeWJGxAAomJQTsWhUn8FwwKFq7Hq8A==", 2022-07-21T01:42:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.0", 2022-07-21T01:42:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiCbb3GBAAIFURLj3UOrino0vkEbby9Q==", 2022-07-21T01:42:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:42:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:42:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.1", 2022-07-21T01:42:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhiKJj7HBAAf7sUPuATYpDcTeEDHsoeKw==", 2022-07-21T01:42:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow", 2022-07-21T01:42:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow r", 2022-07-21T01:42:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-21T01:42:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "client.admin", 2022-07-21T01:42:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCvrNhiRpD8LRAAjat8fzHfeCEk5AcHHvhBWA==", 2022-07-21T01:42:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:42:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": "allow *", 2022-07-21T01:42:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow *", 2022-07-21T01:42:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.x", 2022-07-21T01:42:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhijk7XCBAAqjhsk3tjUHx4r0IIKyeBvw==", 2022-07-21T01:42:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:42:54.979 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:42:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.980 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "entity": "mgr.y", 2022-07-21T01:42:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "AQCwrNhi/4CYBRAA7E3fSJzZNFKUI4lr/9NAQw==", 2022-07-21T01:42:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": { 2022-07-21T01:42:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": "allow *", 2022-07-21T01:42:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": "allow profile mgr", 2022-07-21T01:42:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": "allow *" 2022-07-21T01:42:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:42:54.985 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:42:54.988 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc9c0539e0 msgr2=0x7fdc9c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:54.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc9c0539e0 0x7fdc9c055ea0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7fdcb001c000 tx=0x7fdcb0015000).stop 2022-07-21T01:42:54.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00afd60 msgr2=0x7fdcc00af3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:54.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00afd60 0x7fdcc00af3d0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7fdcb400c000 tx=0x7fdcb4006df0).stop 2022-07-21T01:42:54.989 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 shutdown_connections 2022-07-21T01:42:54.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdc9c0539e0 0x7fdc9c055ea0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcc00b2710 0x7fdcc00b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcc00b01a0 0x7fdcc00b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.990 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 --2- 172.21.15.3:0/1481477349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcc00afd60 0x7fdcc00af3d0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:54.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 >> 172.21.15.3:0/1481477349 conn(0x7fdcc0060310 msgr2=0x7fdcc005c890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:54.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 shutdown_connections 2022-07-21T01:42:54.991 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:54.939+0000 7fdcbf59e700 1 -- 172.21.15.3:0/1481477349 wait complete. 2022-07-21T01:42:54.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:55.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 Processor -- start 2022-07-21T01:42:55.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 -- start start 2022-07-21T01:42:55.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa5e40b1f40 0x7fa5e40b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e40a7250 0x7fa5e41448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e40b04e0 con 0x7fa5e4148530 2022-07-21T01:42:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e40a9250 con 0x7fa5e40b1f40 2022-07-21T01:42:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e40b0780 con 0x7fa5e40a7250 2022-07-21T01:42:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e37fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa5e4148530 0x7fa5e40b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60182/0 (socket says 172.21.15.3:60182) 2022-07-21T01:42:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e37fe700 1 -- 172.21.15.3:0/313749735 learned_addr learned my addr 172.21.15.3:0/313749735 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:55.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e3fff700 1 --2- 172.21.15.3:0/313749735 >> v2:172.21.15.143:3300/0 conn(0x7fa5e40b1f40 0x7fa5e40b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e8bb2700 1 --2- 172.21.15.3:0/313749735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e40a7250 0x7fa5e41448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e2ffd700 1 -- 172.21.15.3:0/313749735 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541272919 0 0) 0x7fa5e40b04e0 con 0x7fa5e4148530 2022-07-21T01:42:55.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e2ffd700 1 -- 172.21.15.3:0/313749735 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d0003260 con 0x7fa5e4148530 2022-07-21T01:42:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e3fff700 1 -- 172.21.15.3:0/313749735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e40a7250 msgr2=0x7fa5e41448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e3fff700 1 --2- 172.21.15.3:0/313749735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e40a7250 0x7fa5e41448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e3fff700 1 -- 172.21.15.3:0/313749735 >> v1:172.21.15.3:6789/0 conn(0x7fa5e4148530 legacy=0x7fa5e40b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e3fff700 1 -- 172.21.15.3:0/313749735 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e40b0a20 con 0x7fa5e40b1f40 2022-07-21T01:42:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e3fff700 1 --2- 172.21.15.3:0/313749735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40b1f40 0x7fa5e40b1160 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fa5e41478c0 tx=0x7fa5d40091d0).ready entity=mon.1 client_cookie=543055e211ecd648 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e2ffd700 1 -- 172.21.15.3:0/313749735 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d400b600 con 0x7fa5e40b1f40 2022-07-21T01:42:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e2ffd700 1 -- 172.21.15.3:0/313749735 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d400b760 con 0x7fa5e40b1f40 2022-07-21T01:42:55.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5e2ffd700 1 -- 172.21.15.3:0/313749735 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d400b8d0 con 0x7fa5e40b1f40 2022-07-21T01:42:55.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 -- 172.21.15.3:0/313749735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40b1f40 msgr2=0x7fa5e40b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 --2- 172.21.15.3:0/313749735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40b1f40 0x7fa5e40b1160 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fa5e41478c0 tx=0x7fa5d40091d0).stop 2022-07-21T01:42:55.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 -- 172.21.15.3:0/313749735 shutdown_connections 2022-07-21T01:42:55.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 --2- 172.21.15.3:0/313749735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e40a7250 0x7fa5e41448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 --2- 172.21.15.3:0/313749735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40b1f40 0x7fa5e40b1160 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.095+0000 7fa5ea615700 1 -- 172.21.15.3:0/313749735 >> 172.21.15.3:0/313749735 conn(0x7fa5e405f840 msgr2=0x7fa5e405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:55.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.099+0000 7fa5ea615700 1 -- 172.21.15.3:0/313749735 shutdown_connections 2022-07-21T01:42:55.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.099+0000 7fa5ea615700 1 -- 172.21.15.3:0/313749735 wait complete. 2022-07-21T01:42:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.099+0000 7fa5ea615700 1 Processor -- start 2022-07-21T01:42:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.099+0000 7fa5ea615700 1 -- start start 2022-07-21T01:42:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5ea615700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40a7250 0x7fa5e40ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5ea615700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e40b1f40 0x7fa5e4156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5ea615700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e4148530 0x7fa5e4157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5ea615700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e40abf10 con 0x7fa5e40b1f40 2022-07-21T01:42:55.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5ea615700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e40ac470 con 0x7fa5e40a7250 2022-07-21T01:42:55.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5ea615700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e40ac710 con 0x7fa5e4148530 2022-07-21T01:42:55.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e40b1f40 0x7fa5e4156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40a7250 0x7fa5e40ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40a7250 0x7fa5e40ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:57356/0 (socket says 172.21.15.3:57356) 2022-07-21T01:42:55.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e3fff700 1 -- 172.21.15.3:0/3967853945 learned_addr learned my addr 172.21.15.3:0/3967853945 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:55.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e8bb2700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e4148530 0x7fa5e4157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e3fff700 1 -- 172.21.15.3:0/3967853945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e4148530 msgr2=0x7fa5e4157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e3fff700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e4148530 0x7fa5e4157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e3fff700 1 -- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e40b1f40 msgr2=0x7fa5e4156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e3fff700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e40b1f40 0x7fa5e4156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e3fff700 1 -- 172.21.15.3:0/3967853945 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d4009470 con 0x7fa5e40a7250 2022-07-21T01:42:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e37fe700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e40b1f40 0x7fa5e4156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e3fff700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40a7250 0x7fa5e40ae300 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7fa5e41478c0 tx=0x7fa5d4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e17fa700 1 -- 172.21.15.3:0/3967853945 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d400b8d0 con 0x7fa5e40a7250 2022-07-21T01:42:55.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e17fa700 1 -- 172.21.15.3:0/3967853945 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d4021630 con 0x7fa5e40a7250 2022-07-21T01:42:55.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e40ac1b0 con 0x7fa5e40a7250 2022-07-21T01:42:55.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5e17fa700 1 -- 172.21.15.3:0/3967853945 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d400b600 con 0x7fa5e40a7250 2022-07-21T01:42:55.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e40af530 con 0x7fa5e40a7250 2022-07-21T01:42:55.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.103+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e40574e0 con 0x7fa5e40a7250 2022-07-21T01:42:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.107+0000 7fa5e17fa700 1 -- 172.21.15.3:0/3967853945 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa5d4027020 con 0x7fa5e40a7250 2022-07-21T01:42:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.107+0000 7fa5e17fa700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa5c4053a10 0x7fa5c4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.107+0000 7fa5e17fa700 1 -- 172.21.15.3:0/3967853945 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fa5d406d1b0 con 0x7fa5e40a7250 2022-07-21T01:42:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.107+0000 7fa5e37fe700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa5c4053a10 0x7fa5c4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.107+0000 7fa5e37fe700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa5c4053a10 0x7fa5c4055ed0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7fa5d800d000 tx=0x7fa5d8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.115+0000 7fa5e17fa700 1 -- 172.21.15.3:0/3967853945 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa5d40442b0 con 0x7fa5e40a7250 2022-07-21T01:42:55.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.243+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5e40acbb0 con 0x7fa5e40a7250 2022-07-21T01:42:55.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.247+0000 7fa5e17fa700 1 -- 172.21.15.3:0/3967853945 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+13424 (secure 0 0 0) 0x7fa5d4015090 con 0x7fa5e40a7250 2022-07-21T01:42:55.250 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:55.252 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":157,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:54.614285+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa5c4053a10 msgr2=0x7fa5c4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa5c4053a10 0x7fa5c4055ed0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7fa5d800d000 tx=0x7fa5d8009040).stop 2022-07-21T01:42:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40a7250 msgr2=0x7fa5e40ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40a7250 0x7fa5e40ae300 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7fa5e41478c0 tx=0x7fa5d4016000).stop 2022-07-21T01:42:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 shutdown_connections 2022-07-21T01:42:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5e4148530 0x7fa5e4157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e40b1f40 0x7fa5e4156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa5c4053a10 0x7fa5c4055ed0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 --2- 172.21.15.3:0/3967853945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5e40a7250 0x7fa5e40ae300 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 >> 172.21.15.3:0/3967853945 conn(0x7fa5e405f840 msgr2=0x7fa5e4144890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:55.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 shutdown_connections 2022-07-21T01:42:55.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.251+0000 7fa5ea615700 1 -- 172.21.15.3:0/3967853945 wait complete. 2022-07-21T01:42:55.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-21T01:42:55.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:42:55.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.371+0000 7f211c933700 1 Processor -- start 2022-07-21T01:42:55.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.371+0000 7f211c933700 1 -- start start 2022-07-21T01:42:55.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.371+0000 7f211c933700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 0x7f2118062050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.371+0000 7f211c933700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f21180b1940 0x7f211813e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.371+0000 7f211c933700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21180b10d0 con 0x7f211813e5b0 2022-07-21T01:42:55.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.371+0000 7f211c933700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2118063e40 con 0x7f21180b1940 2022-07-21T01:42:55.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.371+0000 7f211c933700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2118060200 con 0x7f2118063a60 2022-07-21T01:42:55.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f2116ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f21180b1940 0x7f211813e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f21177fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 0x7f2118062050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f2117fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f211813e5b0 0x7f21181414c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:34084/0 (socket says 172.21.15.3:34084) 2022-07-21T01:42:55.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f2117fff700 1 -- 172.21.15.3:0/437876977 learned_addr learned my addr 172.21.15.3:0/437876977 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:55.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f2116ffd700 1 --2- 172.21.15.3:0/437876977 >> v2:172.21.15.143:3300/0 conn(0x7f21180b1940 0x7f211813e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:57364/0 (socket says 172.21.15.3:57364) 2022-07-21T01:42:55.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f2116ffd700 1 -- 172.21.15.3:0/437876977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 msgr2=0x7f2118062050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:55.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f2116ffd700 1 --2- 172.21.15.3:0/437876977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 0x7f2118062050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f2116ffd700 1 -- 172.21.15.3:0/437876977 >> v1:172.21.15.143:6789/0 conn(0x7f211813e5b0 legacy=0x7f21181414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f2116ffd700 1 -- 172.21.15.3:0/437876977 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21180b1300 con 0x7f21180b1940 2022-07-21T01:42:55.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f2116ffd700 1 --2- 172.21.15.3:0/437876977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21180b1940 0x7f211813e0c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f2100002f70 tx=0x7f2100006df0).ready entity=mon.1 client_cookie=d3d1279041b7cb08 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f21167fc700 1 -- 172.21.15.3:0/437876977 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f210000d570 con 0x7f21180b1940 2022-07-21T01:42:55.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f21167fc700 1 -- 172.21.15.3:0/437876977 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2100000e90 con 0x7f21180b1940 2022-07-21T01:42:55.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f21167fc700 1 -- 172.21.15.3:0/437876977 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f210000d8c0 con 0x7f21180b1940 2022-07-21T01:42:55.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f211c933700 1 -- 172.21.15.3:0/437876977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21180b1940 msgr2=0x7f211813e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f211c933700 1 --2- 172.21.15.3:0/437876977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21180b1940 0x7f211813e0c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f2100002f70 tx=0x7f2100006df0).stop 2022-07-21T01:42:55.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f211c933700 1 -- 172.21.15.3:0/437876977 shutdown_connections 2022-07-21T01:42:55.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f211c933700 1 --2- 172.21.15.3:0/437876977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21180b1940 0x7f211813e0c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f211c933700 1 --2- 172.21.15.3:0/437876977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 0x7f2118062050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f211c933700 1 -- 172.21.15.3:0/437876977 >> 172.21.15.3:0/437876977 conn(0x7f211805c620 msgr2=0x7f211805ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:55.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f211c933700 1 -- 172.21.15.3:0/437876977 shutdown_connections 2022-07-21T01:42:55.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.375+0000 7f211c933700 1 -- 172.21.15.3:0/437876977 wait complete. 2022-07-21T01:42:55.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 Processor -- start 2022-07-21T01:42:55.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 -- start start 2022-07-21T01:42:55.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 0x7f21180b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21180b1940 0x7f211813f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f211813e5b0 0x7f211813ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2118066a20 con 0x7f21180b1940 2022-07-21T01:42:55.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211813ebc0 con 0x7f211813e5b0 2022-07-21T01:42:55.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2118066780 con 0x7f2118063a60 2022-07-21T01:42:55.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2116ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21180b1940 0x7f211813f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2116ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21180b1940 0x7f211813f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51924/0 (socket says 172.21.15.3:51924) 2022-07-21T01:42:55.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2116ffd700 1 -- 172.21.15.3:0/3242368617 learned_addr learned my addr 172.21.15.3:0/3242368617 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:55.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2117fff700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f211813e5b0 0x7f211813ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f21177fe700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 0x7f21180b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2116ffd700 1 -- 172.21.15.3:0/3242368617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 msgr2=0x7f21180b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2116ffd700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 0x7f21180b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2116ffd700 1 -- 172.21.15.3:0/3242368617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f211813e5b0 msgr2=0x7f211813ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2116ffd700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f211813e5b0 0x7f211813ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2116ffd700 1 -- 172.21.15.3:0/3242368617 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210000b180 con 0x7f21180b1940 2022-07-21T01:42:55.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f21177fe700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 0x7f21180b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:55.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2116ffd700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21180b1940 0x7f211813f2c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f2100002f70 tx=0x7f2100014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2114ff9700 1 -- 172.21.15.3:0/3242368617 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f210000dea0 con 0x7f21180b1940 2022-07-21T01:42:55.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2114ff9700 1 -- 172.21.15.3:0/3242368617 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2100013e90 con 0x7f21180b1940 2022-07-21T01:42:55.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211813ee20 con 0x7f21180b1940 2022-07-21T01:42:55.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f2114ff9700 1 -- 172.21.15.3:0/3242368617 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f210001e5f0 con 0x7f21180b1940 2022-07-21T01:42:55.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.379+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21180644e0 con 0x7f21180b1940 2022-07-21T01:42:55.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.383+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21180b6220 con 0x7f21180b1940 2022-07-21T01:42:55.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.383+0000 7f2114ff9700 1 -- 172.21.15.3:0/3242368617 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2100025030 con 0x7f21180b1940 2022-07-21T01:42:55.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.383+0000 7f2114ff9700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20fc053a10 0x7f20fc055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.383+0000 7f2114ff9700 1 -- 172.21.15.3:0/3242368617 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f210006c8e0 con 0x7f21180b1940 2022-07-21T01:42:55.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.383+0000 7f21177fe700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20fc053a10 0x7f20fc055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.383+0000 7f21177fe700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20fc053a10 0x7f20fc055ed0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f210800b000 tx=0x7f2108008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.391+0000 7f2114ff9700 1 -- 172.21.15.3:0/3242368617 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f21000439e0 con 0x7f21180b1940 2022-07-21T01:42:55.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.515+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f211813efb0 con 0x7f21180b1940 2022-07-21T01:42:55.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f2114ff9700 1 -- 172.21.15.3:0/3242368617 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v157) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2100023070 con 0x7f21180b1940 2022-07-21T01:42:55.520 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:42:55.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20fc053a10 msgr2=0x7f20fc055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20fc053a10 0x7f20fc055ed0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f210800b000 tx=0x7f2108008040).stop 2022-07-21T01:42:55.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21180b1940 msgr2=0x7f211813f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21180b1940 0x7f211813f2c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f2100002f70 tx=0x7f2100014000).stop 2022-07-21T01:42:55.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 shutdown_connections 2022-07-21T01:42:55.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f20fc053a10 0x7f20fc055ed0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f211813e5b0 0x7f211813ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21180b1940 0x7f211813f2c0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 --2- 172.21.15.3:0/3242368617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2118063a60 0x7f21180b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 >> 172.21.15.3:0/3242368617 conn(0x7f211805c620 msgr2=0x7f21181460a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:55.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 shutdown_connections 2022-07-21T01:42:55.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.519+0000 7f211c933700 1 -- 172.21.15.3:0/3242368617 wait complete. 2022-07-21T01:42:55.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:42:55.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f759e700 1 Processor -- start 2022-07-21T01:42:55.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f759e700 1 -- start start 2022-07-21T01:42:55.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06f80b3190 0x7f06f80b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f759e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06f80b1a80 0x7f06f80656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80b3570 con 0x7f06f80a9da0 2022-07-21T01:42:55.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f759e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80b2980 con 0x7f06f80b1a80 2022-07-21T01:42:55.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80b0e00 con 0x7f06f80b3190 2022-07-21T01:42:55.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f5d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06f80b1a80 0x7f06f80656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f5d9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f06f80b1a80 0x7f06f80656a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51932/0 (socket says 172.21.15.3:51932) 2022-07-21T01:42:55.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f5d9b700 1 -- 172.21.15.3:0/2944298891 learned_addr learned my addr 172.21.15.3:0/2944298891 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:55.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f659c700 1 --2- 172.21.15.3:0/2944298891 >> v2:172.21.15.143:3300/0 conn(0x7f06f80b3190 0x7f06f80b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f5d9b700 1 -- 172.21.15.3:0/2944298891 >> v2:172.21.15.143:3300/0 conn(0x7f06f80b3190 msgr2=0x7f06f80b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:55.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f5d9b700 1 --2- 172.21.15.3:0/2944298891 >> v2:172.21.15.143:3300/0 conn(0x7f06f80b3190 0x7f06f80b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f5d9b700 1 -- 172.21.15.3:0/2944298891 >> v1:172.21.15.143:6789/0 conn(0x7f06f80a9da0 legacy=0x7f06f8147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f5d9b700 1 -- 172.21.15.3:0/2944298891 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80b10a0 con 0x7f06f80b1a80 2022-07-21T01:42:55.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f5d9b700 1 --2- 172.21.15.3:0/2944298891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80b1a80 0x7f06f80656a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7f06ec002f70 tx=0x7f06ec006eb0).ready entity=mon.0 client_cookie=aace818b8bfe9138 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f559a700 1 -- 172.21.15.3:0/2944298891 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06ec00d610 con 0x7f06f80b1a80 2022-07-21T01:42:55.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f559a700 1 -- 172.21.15.3:0/2944298891 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec000e90 con 0x7f06f80b1a80 2022-07-21T01:42:55.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.635+0000 7f06f559a700 1 -- 172.21.15.3:0/2944298891 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06ec00d9a0 con 0x7f06f80b1a80 2022-07-21T01:42:55.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 -- 172.21.15.3:0/2944298891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80b1a80 msgr2=0x7f06f80656a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 --2- 172.21.15.3:0/2944298891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80b1a80 0x7f06f80656a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7f06ec002f70 tx=0x7f06ec006eb0).stop 2022-07-21T01:42:55.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 -- 172.21.15.3:0/2944298891 shutdown_connections 2022-07-21T01:42:55.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 --2- 172.21.15.3:0/2944298891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80b1a80 0x7f06f80656a0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 --2- 172.21.15.3:0/2944298891 >> v2:172.21.15.143:3300/0 conn(0x7f06f80b3190 0x7f06f80b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 -- 172.21.15.3:0/2944298891 >> 172.21.15.3:0/2944298891 conn(0x7f06f8063870 msgr2=0x7f06f8063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:55.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 -- 172.21.15.3:0/2944298891 shutdown_connections 2022-07-21T01:42:55.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 -- 172.21.15.3:0/2944298891 wait complete. 2022-07-21T01:42:55.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 Processor -- start 2022-07-21T01:42:55.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 -- start start 2022-07-21T01:42:55.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80a9da0 0x7f06f814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f80b1a80 0x7f06f80ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f80b3190 0x7f06f80abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8149d90 con 0x7f06f80a9da0 2022-07-21T01:42:55.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80aafb0 con 0x7f06f80b1a80 2022-07-21T01:42:55.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8145a90 con 0x7f06f80b3190 2022-07-21T01:42:55.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80a9da0 0x7f06f814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80a9da0 0x7f06f814a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51936/0 (socket says 172.21.15.3:51936) 2022-07-21T01:42:55.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.639+0000 7f06f659c700 1 -- 172.21.15.3:0/3996199657 learned_addr learned my addr 172.21.15.3:0/3996199657 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:55.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f5d9b700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f80b1a80 0x7f06f80ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f6d9d700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f80b3190 0x7f06f80abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f659c700 1 -- 172.21.15.3:0/3996199657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f80b3190 msgr2=0x7f06f80abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f659c700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f80b3190 0x7f06f80abb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f659c700 1 -- 172.21.15.3:0/3996199657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f80b1a80 msgr2=0x7f06f80ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f659c700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f80b1a80 0x7f06f80ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f659c700 1 -- 172.21.15.3:0/3996199657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec00b1e0 con 0x7f06f80a9da0 2022-07-21T01:42:55.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f659c700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80a9da0 0x7f06f814a5f0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f06e80030b0 tx=0x7f06e8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06e77fe700 1 -- 172.21.15.3:0/3996199657 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06e8011780 con 0x7f06f80a9da0 2022-07-21T01:42:55.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06e77fe700 1 -- 172.21.15.3:0/3996199657 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e80118e0 con 0x7f06f80a9da0 2022-07-21T01:42:55.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f8145d30 con 0x7f06f80a9da0 2022-07-21T01:42:55.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06e77fe700 1 -- 172.21.15.3:0/3996199657 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06e8011c40 con 0x7f06f80a9da0 2022-07-21T01:42:55.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f81465d0 con 0x7f06f80a9da0 2022-07-21T01:42:55.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f80ac220 con 0x7f06f80a9da0 2022-07-21T01:42:55.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06e77fe700 1 -- 172.21.15.3:0/3996199657 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f06e801c460 con 0x7f06f80a9da0 2022-07-21T01:42:55.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06e77fe700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06d40539e0 0x7f06d4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06e77fe700 1 -- 172.21.15.3:0/3996199657 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5600+0+0 (secure 0 0 0) 0x7f06e806bef0 con 0x7f06f80a9da0 2022-07-21T01:42:55.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.643+0000 7f06f5d9b700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06d40539e0 0x7f06d4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.647+0000 7f06f5d9b700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06d40539e0 0x7f06d4055ea0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f06ec01c000 tx=0x7f06ec015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.651+0000 7f06e77fe700 1 -- 172.21.15.3:0/3996199657 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f06e8042f90 con 0x7f06f80a9da0 2022-07-21T01:42:55.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.779+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f06f80574e0 con 0x7f06f80a9da0 2022-07-21T01:42:55.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06e77fe700 1 -- 172.21.15.3:0/3996199657 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v157) v1 ==== 92+0+10 (secure 0 0 0) 0x7f06e80465b0 con 0x7f06f80a9da0 2022-07-21T01:42:55.785 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:42:55.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06d40539e0 msgr2=0x7f06d4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06d40539e0 0x7f06d4055ea0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f06ec01c000 tx=0x7f06ec015000).stop 2022-07-21T01:42:55.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80a9da0 msgr2=0x7f06f814a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80a9da0 0x7f06f814a5f0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f06e80030b0 tx=0x7f06e8008040).stop 2022-07-21T01:42:55.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 shutdown_connections 2022-07-21T01:42:55.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f80b3190 0x7f06f80abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f06d40539e0 0x7f06d4055ea0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f80b1a80 0x7f06f80ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 --2- 172.21.15.3:0/3996199657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80a9da0 0x7f06f814a5f0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 >> 172.21.15.3:0/3996199657 conn(0x7f06f8063870 msgr2=0x7f06f80b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:55.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 shutdown_connections 2022-07-21T01:42:55.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.783+0000 7f06f759e700 1 -- 172.21.15.3:0/3996199657 wait complete. 2022-07-21T01:42:55.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-21T01:42:55.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facbb20e700 1 Processor -- start 2022-07-21T01:42:55.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facbb20e700 1 -- start start 2022-07-21T01:42:55.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facbb20e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7facb40b1f40 0x7facb40b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facbb20e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40a7250 0x7facb41448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facbb20e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb40b04e0 con 0x7facb4148530 2022-07-21T01:42:55.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facbb20e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb40a9250 con 0x7facb40b1f40 2022-07-21T01:42:55.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facbb20e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb40b0780 con 0x7facb40a7250 2022-07-21T01:42:55.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb8faa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7facb40b1f40 0x7facb40b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb97ab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40a7250 0x7facb41448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb8faa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7facb40b1f40 0x7facb40b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.3:57392/0 (socket says 172.21.15.3:57392) 2022-07-21T01:42:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb8faa700 1 -- 172.21.15.3:0/2809163498 learned_addr learned my addr 172.21.15.3:0/2809163498 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb3fff700 1 --1- 172.21.15.3:0/2809163498 >> v1:172.21.15.143:6789/0 conn(0x7facb4148530 0x7facb40b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.3:34110/0 (socket says 172.21.15.3:34110) 2022-07-21T01:42:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb8faa700 1 -- 172.21.15.3:0/2809163498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40a7250 msgr2=0x7facb41448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb8faa700 1 --2- 172.21.15.3:0/2809163498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40a7250 0x7facb41448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb8faa700 1 -- 172.21.15.3:0/2809163498 >> v1:172.21.15.143:6789/0 conn(0x7facb4148530 legacy=0x7facb40b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb8faa700 1 -- 172.21.15.3:0/2809163498 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb40b0a20 con 0x7facb40b1f40 2022-07-21T01:42:55.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.895+0000 7facb8faa700 1 --2- 172.21.15.3:0/2809163498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facb40b1f40 0x7facb40b1160 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7facb41478c0 tx=0x7facac009290).ready entity=mon.1 client_cookie=4c69a366a528041b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb37fe700 1 -- 172.21.15.3:0/2809163498 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac00b860 con 0x7facb40b1f40 2022-07-21T01:42:55.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb37fe700 1 -- 172.21.15.3:0/2809163498 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac00b9c0 con 0x7facb40b1f40 2022-07-21T01:42:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb37fe700 1 -- 172.21.15.3:0/2809163498 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac00bb30 con 0x7facb40b1f40 2022-07-21T01:42:55.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 -- 172.21.15.3:0/2809163498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facb40b1f40 msgr2=0x7facb40b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 --2- 172.21.15.3:0/2809163498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facb40b1f40 0x7facb40b1160 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7facb41478c0 tx=0x7facac009290).stop 2022-07-21T01:42:55.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 -- 172.21.15.3:0/2809163498 shutdown_connections 2022-07-21T01:42:55.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 --2- 172.21.15.3:0/2809163498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40a7250 0x7facb41448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 --2- 172.21.15.3:0/2809163498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facb40b1f40 0x7facb40b1160 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 -- 172.21.15.3:0/2809163498 >> 172.21.15.3:0/2809163498 conn(0x7facb405f840 msgr2=0x7facb405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:55.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 -- 172.21.15.3:0/2809163498 shutdown_connections 2022-07-21T01:42:55.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 -- 172.21.15.3:0/2809163498 wait complete. 2022-07-21T01:42:55.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 Processor -- start 2022-07-21T01:42:55.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 -- start start 2022-07-21T01:42:55.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facb40a7250 0x7facb414a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40b1f40 0x7facb4143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facb4148530 0x7facb4144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb40aafa0 con 0x7facb40a7250 2022-07-21T01:42:55.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb4149780 con 0x7facb4148530 2022-07-21T01:42:55.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facbb20e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb4144510 con 0x7facb40b1f40 2022-07-21T01:42:55.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb8faa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facb40a7250 0x7facb414a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb8faa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facb40a7250 0x7facb414a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51950/0 (socket says 172.21.15.3:51950) 2022-07-21T01:42:55.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb8faa700 1 -- 172.21.15.3:0/3207494326 learned_addr learned my addr 172.21.15.3:0/3207494326 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:55.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb97ab700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facb4148530 0x7facb4144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb3fff700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40b1f40 0x7facb4143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb8faa700 1 -- 172.21.15.3:0/3207494326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40b1f40 msgr2=0x7facb4143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb8faa700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40b1f40 0x7facb4143b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb8faa700 1 -- 172.21.15.3:0/3207494326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facb4148530 msgr2=0x7facb4144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:55.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb8faa700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facb4148530 0x7facb4144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:55.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.899+0000 7facb8faa700 1 -- 172.21.15.3:0/3207494326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac009530 con 0x7facb40a7250 2022-07-21T01:42:55.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb3fff700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40b1f40 0x7facb4143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:55.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb8faa700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facb40a7250 0x7facb414a300 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7facb41478c0 tx=0x7facac016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb1ffb700 1 -- 172.21.15.3:0/3207494326 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac021630 con 0x7facb40a7250 2022-07-21T01:42:55.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb1ffb700 1 -- 172.21.15.3:0/3207494326 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac021790 con 0x7facb40a7250 2022-07-21T01:42:55.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb40aac60 con 0x7facb40a7250 2022-07-21T01:42:55.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb1ffb700 1 -- 172.21.15.3:0/3207494326 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac021af0 con 0x7facb40a7250 2022-07-21T01:42:55.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb414aad0 con 0x7facb40a7250 2022-07-21T01:42:55.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb40574e0 con 0x7facb40a7250 2022-07-21T01:42:55.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb1ffb700 1 -- 172.21.15.3:0/3207494326 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7facac027020 con 0x7facb40a7250 2022-07-21T01:42:55.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb1ffb700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac98053a70 0x7fac98055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:55.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb1ffb700 1 -- 172.21.15.3:0/3207494326 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5600+0+0 (secure 0 0 0) 0x7facac06d6b0 con 0x7facb40a7250 2022-07-21T01:42:55.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb3fff700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac98053a70 0x7fac98055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:55.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.903+0000 7facb3fff700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac98053a70 0x7fac98055f30 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fac9c00b000 tx=0x7fac9c0070e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:55.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:55.915+0000 7facb1ffb700 1 -- 172.21.15.3:0/3207494326 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7facac0447c0 con 0x7facb40a7250 2022-07-21T01:42:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.039+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7facb40447e0 con 0x7facb40a7250 2022-07-21T01:42:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.039+0000 7facb1ffb700 1 -- 172.21.15.3:0/3207494326 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v157) v1 ==== 84+0+143 (secure 0 0 0) 0x7facac015030 con 0x7facb40a7250 2022-07-21T01:42:56.044 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:56.045 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2022-07-21T01:42:56.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:56.045 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2022-07-21T01:42:56.045 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "device_health_metrics" 2022-07-21T01:42:56.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:42:56.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:42:56.046 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 2, 2022-07-21T01:42:56.046 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": "rbd" 2022-07-21T01:42:56.047 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:42:56.047 INFO:teuthology.orchestra.run.smithi003.stdout:] 2022-07-21T01:42:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac98053a70 msgr2=0x7fac98055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac98053a70 0x7fac98055f30 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fac9c00b000 tx=0x7fac9c0070e0).stop 2022-07-21T01:42:56.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facb40a7250 msgr2=0x7facb414a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:56.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facb40a7250 0x7facb414a300 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7facb41478c0 tx=0x7facac016000).stop 2022-07-21T01:42:56.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 shutdown_connections 2022-07-21T01:42:56.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fac98053a70 0x7fac98055f30 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facb4148530 0x7facb4144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7facb40b1f40 0x7facb4143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 --2- 172.21.15.3:0/3207494326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7facb40a7250 0x7facb414a300 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 >> 172.21.15.3:0/3207494326 conn(0x7facb405f840 msgr2=0x7facb40b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:56.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 shutdown_connections 2022-07-21T01:42:56.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.043+0000 7facbb20e700 1 -- 172.21.15.3:0/3207494326 wait complete. 2022-07-21T01:42:56.057 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-21T01:42:56.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-21T01:42:56.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.167+0000 7fe730ddb700 1 Processor -- start 2022-07-21T01:42:56.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.167+0000 7fe730ddb700 1 -- start start 2022-07-21T01:42:56.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe72c05fd90 0x7fe72c0b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:56.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c1440f0 con 0x7fe72c0b5b20 2022-07-21T01:42:56.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c060170 con 0x7fe72c0b66f0 2022-07-21T01:42:56.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72c0ba840 con 0x7fe72c05fd90 2022-07-21T01:42:56.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72a59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe72c05fd90 0x7fe72c0b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:56.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72ad9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe72c0b5b20 0x7fe72c0b4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60242/0 (socket says 172.21.15.3:60242) 2022-07-21T01:42:56.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72ad9d700 1 -- 172.21.15.3:0/4121655569 learned_addr learned my addr 172.21.15.3:0/4121655569 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:56.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72a59c700 1 -- 172.21.15.3:0/4121655569 >> v1:172.21.15.143:6789/0 conn(0x7fe72c0b66f0 legacy=0x7fe72c0ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:56.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72a59c700 1 -- 172.21.15.3:0/4121655569 >> v1:172.21.15.3:6789/0 conn(0x7fe72c0b5b20 legacy=0x7fe72c0b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:56.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72a59c700 1 -- 172.21.15.3:0/4121655569 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c143be0 con 0x7fe72c05fd90 2022-07-21T01:42:56.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72a59c700 1 --2- 172.21.15.3:0/4121655569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c05fd90 0x7fe72c0b4340 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7fe720002f70 tx=0x7fe720006df0).ready entity=mon.0 client_cookie=ca2854c1930ae449 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:56.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72959a700 1 -- 172.21.15.3:0/4121655569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe72000d610 con 0x7fe72c05fd90 2022-07-21T01:42:56.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72959a700 1 -- 172.21.15.3:0/4121655569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe720000e90 con 0x7fe72c05fd90 2022-07-21T01:42:56.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe72959a700 1 -- 172.21.15.3:0/4121655569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe72000d960 con 0x7fe72c05fd90 2022-07-21T01:42:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 -- 172.21.15.3:0/4121655569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c05fd90 msgr2=0x7fe72c0b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 --2- 172.21.15.3:0/4121655569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c05fd90 0x7fe72c0b4340 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7fe720002f70 tx=0x7fe720006df0).stop 2022-07-21T01:42:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 -- 172.21.15.3:0/4121655569 shutdown_connections 2022-07-21T01:42:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 --2- 172.21.15.3:0/4121655569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c05fd90 0x7fe72c0b4340 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 -- 172.21.15.3:0/4121655569 >> 172.21.15.3:0/4121655569 conn(0x7fe72c062530 msgr2=0x7fe72c066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:56.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 -- 172.21.15.3:0/4121655569 shutdown_connections 2022-07-21T01:42:56.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.171+0000 7fe730ddb700 1 -- 172.21.15.3:0/4121655569 wait complete. 2022-07-21T01:42:56.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe730ddb700 1 Processor -- start 2022-07-21T01:42:56.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe730ddb700 1 -- start start 2022-07-21T01:42:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe730ddb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe72c05fd90 0x7fe72c0ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe730ddb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe72c0b5b20 0x7fe72c13fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe730ddb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c0b66f0 0x7fe72c155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe730ddb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0b3b00 con 0x7fe72c0b66f0 2022-07-21T01:42:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe730ddb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c153460 con 0x7fe72c0b5b20 2022-07-21T01:42:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe730ddb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0add70 con 0x7fe72c05fd90 2022-07-21T01:42:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe72ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c0b66f0 0x7fe72c155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe72ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c0b66f0 0x7fe72c155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51968/0 (socket says 172.21.15.3:51968) 2022-07-21T01:42:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe729d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe72c0b5b20 0x7fe72c13fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe72ad9d700 1 -- 172.21.15.3:0/4036280268 learned_addr learned my addr 172.21.15.3:0/4036280268 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe72a59c700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe72c05fd90 0x7fe72c0ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe729d9b700 1 -- 172.21.15.3:0/4036280268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe72c05fd90 msgr2=0x7fe72c0ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe729d9b700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe72c05fd90 0x7fe72c0ae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe729d9b700 1 -- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c0b66f0 msgr2=0x7fe72c155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:56.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe729d9b700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c0b66f0 0x7fe72c155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe729d9b700 1 -- 172.21.15.3:0/4036280268 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72000b180 con 0x7fe72c0b5b20 2022-07-21T01:42:56.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe72ad9d700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c0b66f0 0x7fe72c155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:56.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe729d9b700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe72c0b5b20 0x7fe72c13fac0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7fe72c0afe00 tx=0x7fe71c007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:56.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe71b7fe700 1 -- 172.21.15.3:0/4036280268 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe71c00ccc0 con 0x7fe72c0b5b20 2022-07-21T01:42:56.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe71b7fe700 1 -- 172.21.15.3:0/4036280268 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c00ce20 con 0x7fe72c0b5b20 2022-07-21T01:42:56.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.175+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c0b0620 con 0x7fe72c0b5b20 2022-07-21T01:42:56.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.179+0000 7fe71b7fe700 1 -- 172.21.15.3:0/4036280268 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe71c0125f0 con 0x7fe72c0b5b20 2022-07-21T01:42:56.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.179+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c13d8b0 con 0x7fe72c0b5b20 2022-07-21T01:42:56.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.179+0000 7fe71b7fe700 1 -- 172.21.15.3:0/4036280268 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe71c019030 con 0x7fe72c0b5b20 2022-07-21T01:42:56.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.179+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c0aeb30 con 0x7fe72c0b5b20 2022-07-21T01:42:56.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.179+0000 7fe71b7fe700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe708053b00 0x7fe708055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:56.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.179+0000 7fe71b7fe700 1 -- 172.21.15.3:0/4036280268 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5600+0+0 (secure 0 0 0) 0x7fe71c06aec0 con 0x7fe72c0b5b20 2022-07-21T01:42:56.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.179+0000 7fe72a59c700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe708053b00 0x7fe708055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:56.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.179+0000 7fe72a59c700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe708053b00 0x7fe708055fc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fe72001c000 tx=0x7fe720015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:56.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.187+0000 7fe71b7fe700 1 -- 172.21.15.3:0/4036280268 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe71c041f60 con 0x7fe72c0b5b20 2022-07-21T01:42:56.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.315+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fe72c143e50 con 0x7fe72c0b5b20 2022-07-21T01:42:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.907+0000 7fe71b7fe700 1 -- 172.21.15.3:0/4036280268 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v158) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe71c045580 con 0x7fe72c0b5b20 2022-07-21T01:42:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe708053b00 msgr2=0x7fe708055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:56.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe708053b00 0x7fe708055fc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fe72001c000 tx=0x7fe720015000).stop 2022-07-21T01:42:56.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe72c0b5b20 msgr2=0x7fe72c13fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:56.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe72c0b5b20 0x7fe72c13fac0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7fe72c0afe00 tx=0x7fe71c007b90).stop 2022-07-21T01:42:56.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 shutdown_connections 2022-07-21T01:42:56.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c0b66f0 0x7fe72c155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe708053b00 0x7fe708055fc0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe72c0b5b20 0x7fe72c13fac0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 --2- 172.21.15.3:0/4036280268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe72c05fd90 0x7fe72c0ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:56.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 >> 172.21.15.3:0/4036280268 conn(0x7fe72c062530 msgr2=0x7fe72c1427a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:56.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 shutdown_connections 2022-07-21T01:42:56.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:56.911+0000 7fe730ddb700 1 -- 172.21.15.3:0/4036280268 wait complete. 2022-07-21T01:42:56.918 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2022-07-21T01:42:56.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-21T01:42:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe8b72d700 1 Processor -- start 2022-07-21T01:42:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe8b72d700 1 -- start start 2022-07-21T01:42:57.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe8b72d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efe840a9da0 0x7efe840b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:57.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe8b72d700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe84143ab0 con 0x7efe840b38a0 2022-07-21T01:42:57.049 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe8b72d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe84143d50 con 0x7efe840b0740 2022-07-21T01:42:57.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe8b72d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe84143ff0 con 0x7efe840a9da0 2022-07-21T01:42:57.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe88cc8700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efe840b38a0 0x7efe840b1240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60254/0 (socket says 172.21.15.3:60254) 2022-07-21T01:42:57.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe88cc8700 1 -- 172.21.15.3:0/674857614 learned_addr learned my addr 172.21.15.3:0/674857614 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:57.050 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe89cca700 1 --2- 172.21.15.3:0/674857614 >> v2:172.21.15.143:3300/0 conn(0x7efe840a9da0 0x7efe840b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:57.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe83fff700 1 -- 172.21.15.3:0/674857614 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325987849 0 0) 0x7efe84143ab0 con 0x7efe840b38a0 2022-07-21T01:42:57.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe83fff700 1 -- 172.21.15.3:0/674857614 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe70003260 con 0x7efe840b38a0 2022-07-21T01:42:57.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe83fff700 1 -- 172.21.15.3:0/674857614 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047909526 0 0) 0x7efe84143d50 con 0x7efe840b0740 2022-07-21T01:42:57.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe83fff700 1 -- 172.21.15.3:0/674857614 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe84143ab0 con 0x7efe840b0740 2022-07-21T01:42:57.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe89cca700 1 -- 172.21.15.3:0/674857614 >> v1:172.21.15.143:6789/0 conn(0x7efe840b0740 legacy=0x7efe8413d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:57.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe89cca700 1 -- 172.21.15.3:0/674857614 >> v1:172.21.15.3:6789/0 conn(0x7efe840b38a0 legacy=0x7efe840b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:57.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe89cca700 1 -- 172.21.15.3:0/674857614 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84144290 con 0x7efe840a9da0 2022-07-21T01:42:57.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe89cca700 1 --2- 172.21.15.3:0/674857614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe840a9da0 0x7efe840b0b20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7efe7c0032f0 tx=0x7efe7c0091d0).ready entity=mon.1 client_cookie=195176b00148ba7f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:57.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe83fff700 1 -- 172.21.15.3:0/674857614 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe7c00f8a0 con 0x7efe840a9da0 2022-07-21T01:42:57.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe83fff700 1 -- 172.21.15.3:0/674857614 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7c00fa00 con 0x7efe840a9da0 2022-07-21T01:42:57.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.043+0000 7efe83fff700 1 -- 172.21.15.3:0/674857614 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe7c00fd60 con 0x7efe840a9da0 2022-07-21T01:42:57.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 -- 172.21.15.3:0/674857614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe840a9da0 msgr2=0x7efe840b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:57.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 --2- 172.21.15.3:0/674857614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe840a9da0 0x7efe840b0b20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7efe7c0032f0 tx=0x7efe7c0091d0).stop 2022-07-21T01:42:57.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 -- 172.21.15.3:0/674857614 shutdown_connections 2022-07-21T01:42:57.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 --2- 172.21.15.3:0/674857614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe840a9da0 0x7efe840b0b20 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:57.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 -- 172.21.15.3:0/674857614 >> 172.21.15.3:0/674857614 conn(0x7efe84060950 msgr2=0x7efe84060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:57.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 -- 172.21.15.3:0/674857614 shutdown_connections 2022-07-21T01:42:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 -- 172.21.15.3:0/674857614 wait complete. 2022-07-21T01:42:57.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 Processor -- start 2022-07-21T01:42:57.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 -- start start 2022-07-21T01:42:57.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.047+0000 7efe8b72d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe840a9da0 0x7efe8413ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:57.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe8b72d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe840b0740 0x7efe84065e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:57.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe8b72d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe840b38a0 0x7efe840662f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:57.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe8b72d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe84064680 con 0x7efe840b38a0 2022-07-21T01:42:57.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe8b72d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe84064920 con 0x7efe840b0740 2022-07-21T01:42:57.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe8b72d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe840b5ee0 con 0x7efe840a9da0 2022-07-21T01:42:57.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe89cca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe840b38a0 0x7efe840662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:57.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe89cca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe840b38a0 0x7efe840662f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51982/0 (socket says 172.21.15.3:51982) 2022-07-21T01:42:57.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe89cca700 1 -- 172.21.15.3:0/3136925213 learned_addr learned my addr 172.21.15.3:0/3136925213 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe88cc8700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe840b0740 0x7efe84065e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe894c9700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe840a9da0 0x7efe8413ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe894c9700 1 -- 172.21.15.3:0/3136925213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe840b0740 msgr2=0x7efe84065e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe894c9700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe840b0740 0x7efe84065e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe894c9700 1 -- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe840b38a0 msgr2=0x7efe840662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe894c9700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe840b38a0 0x7efe840662f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe894c9700 1 -- 172.21.15.3:0/3136925213 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c009470 con 0x7efe840a9da0 2022-07-21T01:42:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe89cca700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe840b38a0 0x7efe840662f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:42:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe894c9700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe840a9da0 0x7efe8413ed50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7efe78001110 tx=0x7efe78007a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe827fc700 1 -- 172.21.15.3:0/3136925213 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe78012400 con 0x7efe840a9da0 2022-07-21T01:42:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe827fc700 1 -- 172.21.15.3:0/3136925213 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe78012560 con 0x7efe840a9da0 2022-07-21T01:42:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe840b6180 con 0x7efe840a9da0 2022-07-21T01:42:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe827fc700 1 -- 172.21.15.3:0/3136925213 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe780128c0 con 0x7efe840a9da0 2022-07-21T01:42:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe840667e0 con 0x7efe840a9da0 2022-07-21T01:42:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe840b6310 con 0x7efe840a9da0 2022-07-21T01:42:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.051+0000 7efe827fc700 1 -- 172.21.15.3:0/3136925213 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efe78019030 con 0x7efe840a9da0 2022-07-21T01:42:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.055+0000 7efe827fc700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efe64053aa0 0x7efe64055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.055+0000 7efe827fc700 1 -- 172.21.15.3:0/3136925213 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 5954+0+0 (secure 0 0 0) 0x7efe7806b150 con 0x7efe840a9da0 2022-07-21T01:42:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.055+0000 7efe88cc8700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efe64053aa0 0x7efe64055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.055+0000 7efe88cc8700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efe64053aa0 0x7efe64055f60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7efe7400d000 tx=0x7efe74009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:57.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.063+0000 7efe827fc700 1 -- 172.21.15.3:0/3136925213 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efe780420e0 con 0x7efe840a9da0 2022-07-21T01:42:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.195+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7efe8413ddf0 con 0x7efe840a9da0 2022-07-21T01:42:57.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.919+0000 7efe827fc700 1 -- 172.21.15.3:0/3136925213 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v159) v1 ==== 125+0+0 (secure 0 0 0) 0x7efe78045700 con 0x7efe840a9da0 2022-07-21T01:42:57.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.923+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efe64053aa0 msgr2=0x7efe64055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:57.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.923+0000 7efe8b72d700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efe64053aa0 0x7efe64055f60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7efe7400d000 tx=0x7efe74009040).stop 2022-07-21T01:42:57.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.923+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe840a9da0 msgr2=0x7efe8413ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:57.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.923+0000 7efe8b72d700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe840a9da0 0x7efe8413ed50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7efe78001110 tx=0x7efe78007a20).stop 2022-07-21T01:42:57.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.927+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 shutdown_connections 2022-07-21T01:42:57.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.927+0000 7efe8b72d700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe840b38a0 0x7efe840662f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:57.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.927+0000 7efe8b72d700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7efe64053aa0 0x7efe64055f60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:57.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.927+0000 7efe8b72d700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe840b0740 0x7efe84065e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:57.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.927+0000 7efe8b72d700 1 --2- 172.21.15.3:0/3136925213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe840a9da0 0x7efe8413ed50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:57.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.927+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 >> 172.21.15.3:0/3136925213 conn(0x7efe84060950 msgr2=0x7efe84140080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:57.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.927+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 shutdown_connections 2022-07-21T01:42:57.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:57.927+0000 7efe8b72d700 1 -- 172.21.15.3:0/3136925213 wait complete. 2022-07-21T01:42:57.933 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2022-07-21T01:42:57.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-21T01:42:58.053 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7472953700 1 Processor -- start 2022-07-21T01:42:58.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7472953700 1 -- start start 2022-07-21T01:42:58.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7472953700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f746c0b1180 0x7f746c141920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7472953700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f746c0aac20 con 0x7f746c0b28b0 2022-07-21T01:42:58.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7472953700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f746c0b2c90 con 0x7f746c13e8d0 2022-07-21T01:42:58.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7472953700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f746c140e20 con 0x7f746c0b1180 2022-07-21T01:42:58.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7472152700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f746c0b1180 0x7f746c141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7471951700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f746c0b28b0 0x7f746c13e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60266/0 (socket says 172.21.15.3:60266) 2022-07-21T01:42:58.199 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7471951700 1 -- 172.21.15.3:0/2385782568 learned_addr learned my addr 172.21.15.3:0/2385782568 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:58.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f747094f700 1 -- 172.21.15.3:0/2385782568 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869087296 0 0) 0x7f7468001110 con 0x7f746c0b28b0 2022-07-21T01:42:58.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f747094f700 1 -- 172.21.15.3:0/2385782568 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7454003260 con 0x7f746c0b28b0 2022-07-21T01:42:58.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f747094f700 1 -- 172.21.15.3:0/2385782568 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728241142 0 0) 0x7f746c0b2c90 con 0x7f746c13e8d0 2022-07-21T01:42:58.200 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.051+0000 7f7472152700 1 -- 172.21.15.3:0/2385782568 >> v1:172.21.15.143:6789/0 conn(0x7f746c13e8d0 legacy=0x7f746c141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472152700 1 -- 172.21.15.3:0/2385782568 >> v1:172.21.15.3:6789/0 conn(0x7f746c0b28b0 legacy=0x7f746c13e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472152700 1 -- 172.21.15.3:0/2385782568 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c0aa440 con 0x7f746c0b1180 2022-07-21T01:42:58.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472152700 1 --2- 172.21.15.3:0/2385782568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 0x7f746c141920 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f7468001990 tx=0x7f74640091d0).ready entity=mon.0 client_cookie=9cf4b284961815fd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.201 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f747094f700 1 -- 172.21.15.3:0/2385782568 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74680017f0 con 0x7f746c0b1180 2022-07-21T01:42:58.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f747094f700 1 -- 172.21.15.3:0/2385782568 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746400b580 con 0x7f746c0b1180 2022-07-21T01:42:58.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f747094f700 1 -- 172.21.15.3:0/2385782568 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f746400b6f0 con 0x7f746c0b1180 2022-07-21T01:42:58.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472953700 1 -- 172.21.15.3:0/2385782568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 msgr2=0x7f746c141920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472953700 1 --2- 172.21.15.3:0/2385782568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 0x7f746c141920 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f7468001990 tx=0x7f74640091d0).stop 2022-07-21T01:42:58.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472953700 1 -- 172.21.15.3:0/2385782568 shutdown_connections 2022-07-21T01:42:58.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472953700 1 --2- 172.21.15.3:0/2385782568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 0x7f746c141920 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472953700 1 -- 172.21.15.3:0/2385782568 >> 172.21.15.3:0/2385782568 conn(0x7f746c060310 msgr2=0x7f746c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:58.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472953700 1 -- 172.21.15.3:0/2385782568 shutdown_connections 2022-07-21T01:42:58.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472953700 1 -- 172.21.15.3:0/2385782568 wait complete. 2022-07-21T01:42:58.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472953700 1 Processor -- start 2022-07-21T01:42:58.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.055+0000 7f7472953700 1 -- start start 2022-07-21T01:42:58.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472953700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 0x7f746c1407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472953700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f746c0b28b0 0x7f746c0ab2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472953700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c0ab7c0 0x7f746c130700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472953700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746c136250 con 0x7f746c0b1180 2022-07-21T01:42:58.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472953700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746c1375d0 con 0x7f746c0b28b0 2022-07-21T01:42:58.205 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472953700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746c146950 con 0x7f746c0ab7c0 2022-07-21T01:42:58.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471951700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 0x7f746c1407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471951700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 0x7f746c1407f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51992/0 (socket says 172.21.15.3:51992) 2022-07-21T01:42:58.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471951700 1 -- 172.21.15.3:0/1485406955 learned_addr learned my addr 172.21.15.3:0/1485406955 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:58.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471150700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f746c0b28b0 0x7f746c0ab2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.206 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472152700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c0ab7c0 0x7f746c130700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471951700 1 -- 172.21.15.3:0/1485406955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c0ab7c0 msgr2=0x7f746c130700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471951700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c0ab7c0 0x7f746c130700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471951700 1 -- 172.21.15.3:0/1485406955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f746c0b28b0 msgr2=0x7f746c0ab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471951700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f746c0b28b0 0x7f746c0ab2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471951700 1 -- 172.21.15.3:0/1485406955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7464009470 con 0x7f746c0b1180 2022-07-21T01:42:58.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472152700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c0ab7c0 0x7f746c130700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:58.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7471951700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 0x7f746c1407f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f74680019d0 tx=0x7f7468001ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f745affd700 1 -- 172.21.15.3:0/1485406955 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7468007170 con 0x7f746c0b1180 2022-07-21T01:42:58.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f745affd700 1 -- 172.21.15.3:0/1485406955 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74680072d0 con 0x7f746c0b1180 2022-07-21T01:42:58.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746c146bf0 con 0x7f746c0b1180 2022-07-21T01:42:58.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f745affd700 1 -- 172.21.15.3:0/1485406955 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7468004920 con 0x7f746c0b1180 2022-07-21T01:42:58.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746c134fe0 con 0x7f746c0b1180 2022-07-21T01:42:58.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.059+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746c137770 con 0x7f746c0b1180 2022-07-21T01:42:58.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.063+0000 7f745affd700 1 -- 172.21.15.3:0/1485406955 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f746801b070 con 0x7f746c0b1180 2022-07-21T01:42:58.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.063+0000 7f745affd700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f744c053aa0 0x7f744c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.063+0000 7f7471150700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f744c053aa0 0x7f744c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.063+0000 7f745affd700 1 -- 172.21.15.3:0/1485406955 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6304+0+0 (secure 0 0 0) 0x7f746806b630 con 0x7f746c0b1180 2022-07-21T01:42:58.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.063+0000 7f7471150700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f744c053aa0 0x7f744c055f60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f745c00d000 tx=0x7f745c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.071+0000 7f745affd700 1 -- 172.21.15.3:0/1485406955 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f746c137770 con 0x7f746c0b1180 2022-07-21T01:42:58.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.231+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f746c1467b0 con 0x7f746c0b1180 2022-07-21T01:42:58.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.247+0000 7f745affd700 1 -- 172.21.15.3:0/1485406955 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 29 and data pool 30 v80) v1 ==== 189+0+0 (secure 0 0 0) 0x7f746c1467b0 con 0x7f746c0b1180 2022-07-21T01:42:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.251+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f744c053aa0 msgr2=0x7f744c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.251+0000 7f7472953700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f744c053aa0 0x7f744c055f60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f745c00d000 tx=0x7f745c007b90).stop 2022-07-21T01:42:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.251+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 msgr2=0x7f746c1407f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.251+0000 7f7472953700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 0x7f746c1407f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f74680019d0 tx=0x7f7468001ab0).stop 2022-07-21T01:42:58.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.255+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 shutdown_connections 2022-07-21T01:42:58.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.255+0000 7f7472953700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c0ab7c0 0x7f746c130700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.255+0000 7f7472953700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f744c053aa0 0x7f744c055f60 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.255+0000 7f7472953700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f746c0b28b0 0x7f746c0ab2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.255+0000 7f7472953700 1 --2- 172.21.15.3:0/1485406955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f746c0b1180 0x7f746c1407f0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.255+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 >> 172.21.15.3:0/1485406955 conn(0x7f746c060310 msgr2=0x7f746c0afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.255+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 shutdown_connections 2022-07-21T01:42:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.255+0000 7f7472953700 1 -- 172.21.15.3:0/1485406955 wait complete. 2022-07-21T01:42:58.261 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 29 and data pool 30 2022-07-21T01:42:58.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.359+0000 7fb3c9eab700 1 Processor -- start 2022-07-21T01:42:58.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 -- start start 2022-07-21T01:42:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 0x7fb3c40b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb3c4148530 0x7fb3c40b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c40b04e0 con 0x7fb3c40a7250 2022-07-21T01:42:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c40a9250 con 0x7fb3c4148530 2022-07-21T01:42:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c40b0780 con 0x7fb3c40b1f40 2022-07-21T01:42:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3bbfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fb3c4148530 0x7fb3c40b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c3fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fb3c40a7250 0x7fb3c41448b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60288/0 (socket says 172.21.15.3:60288) 2022-07-21T01:42:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c3fff700 1 -- 172.21.15.3:0/4167541354 learned_addr learned my addr 172.21.15.3:0/4167541354 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c37fe700 1 --2- 172.21.15.3:0/4167541354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 0x7fb3c40b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3bbfff700 1 -- 172.21.15.3:0/4167541354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 msgr2=0x7fb3c40b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3bbfff700 1 --2- 172.21.15.3:0/4167541354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 0x7fb3c40b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3bbfff700 1 -- 172.21.15.3:0/4167541354 >> v1:172.21.15.3:6789/0 conn(0x7fb3c40a7250 legacy=0x7fb3c41448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c2ffd700 1 -- 172.21.15.3:0/4167541354 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291720298 0 0) 0x7fb3c40b04e0 con 0x7fb3c40a7250 2022-07-21T01:42:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3bbfff700 1 -- 172.21.15.3:0/4167541354 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c40b0a20 con 0x7fb3c4148530 2022-07-21T01:42:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3bbfff700 1 --2- 172.21.15.3:0/4167541354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c4148530 0x7fb3c40b2320 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7fb3ac002f70 tx=0x7fb3ac006df0).ready entity=mon.0 client_cookie=7f3d6bc3b5e83d6b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c2ffd700 1 -- 172.21.15.3:0/4167541354 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3ac00d570 con 0x7fb3c4148530 2022-07-21T01:42:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c2ffd700 1 -- 172.21.15.3:0/4167541354 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac000e90 con 0x7fb3c4148530 2022-07-21T01:42:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c2ffd700 1 -- 172.21.15.3:0/4167541354 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3ac00d8c0 con 0x7fb3c4148530 2022-07-21T01:42:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 -- 172.21.15.3:0/4167541354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c4148530 msgr2=0x7fb3c40b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 --2- 172.21.15.3:0/4167541354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c4148530 0x7fb3c40b2320 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7fb3ac002f70 tx=0x7fb3ac006df0).stop 2022-07-21T01:42:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 -- 172.21.15.3:0/4167541354 shutdown_connections 2022-07-21T01:42:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 --2- 172.21.15.3:0/4167541354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c4148530 0x7fb3c40b2320 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 --2- 172.21.15.3:0/4167541354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 0x7fb3c40b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 -- 172.21.15.3:0/4167541354 >> 172.21.15.3:0/4167541354 conn(0x7fb3c405f840 msgr2=0x7fb3c405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 -- 172.21.15.3:0/4167541354 shutdown_connections 2022-07-21T01:42:58.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.363+0000 7fb3c9eab700 1 -- 172.21.15.3:0/4167541354 wait complete. 2022-07-21T01:42:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c9eab700 1 Processor -- start 2022-07-21T01:42:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c9eab700 1 -- start start 2022-07-21T01:42:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c9eab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40a7250 0x7fb3c40ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c9eab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 0x7fb3c4148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c9eab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c4148530 0x7fb3c4157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c9eab700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4156b40 con 0x7fb3c40a7250 2022-07-21T01:42:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c9eab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c414b120 con 0x7fb3c4148530 2022-07-21T01:42:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c9eab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4156d30 con 0x7fb3c40b1f40 2022-07-21T01:42:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40a7250 0x7fb3c40ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40a7250 0x7fb3c40ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52012/0 (socket says 172.21.15.3:52012) 2022-07-21T01:42:58.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c37fe700 1 -- 172.21.15.3:0/608766657 learned_addr learned my addr 172.21.15.3:0/608766657 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:58.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3bbfff700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 0x7fb3c4148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c3fff700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c4148530 0x7fb3c4157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c37fe700 1 -- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 msgr2=0x7fb3c4148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c37fe700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 0x7fb3c4148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c37fe700 1 -- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c4148530 msgr2=0x7fb3c4157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c37fe700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c4148530 0x7fb3c4157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.367+0000 7fb3c37fe700 1 -- 172.21.15.3:0/608766657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac00b180 con 0x7fb3c40a7250 2022-07-21T01:42:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3bbfff700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 0x7fb3c4148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3c3fff700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c4148530 0x7fb3c4157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3c37fe700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40a7250 0x7fb3c40ae300 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7fb3c40aa7f0 tx=0x7fb3b4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3c17fa700 1 -- 172.21.15.3:0/608766657 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3b4011780 con 0x7fb3c40a7250 2022-07-21T01:42:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3c17fa700 1 -- 172.21.15.3:0/608766657 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b40118e0 con 0x7fb3c40a7250 2022-07-21T01:42:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c414ae80 con 0x7fb3c40a7250 2022-07-21T01:42:58.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3c17fa700 1 -- 172.21.15.3:0/608766657 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3b4011c40 con 0x7fb3c40a7250 2022-07-21T01:42:58.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c40ae930 con 0x7fb3c40a7250 2022-07-21T01:42:58.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c414a470 con 0x7fb3c40a7250 2022-07-21T01:42:58.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.371+0000 7fb3c17fa700 1 -- 172.21.15.3:0/608766657 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb3b4012420 con 0x7fb3c40a7250 2022-07-21T01:42:58.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.375+0000 7fb3c17fa700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb39c053a70 0x7fb39c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.375+0000 7fb3c17fa700 1 -- 172.21.15.3:0/608766657 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fb3b406bf50 con 0x7fb3c40a7250 2022-07-21T01:42:58.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.375+0000 7fb3bbfff700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb39c053a70 0x7fb39c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.375+0000 7fb3bbfff700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb39c053a70 0x7fb39c055f30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fb3ac01a000 tx=0x7fb3ac015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.379+0000 7fb3c17fa700 1 -- 172.21.15.3:0/608766657 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3b4042cb0 con 0x7fb3c40a7250 2022-07-21T01:42:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.507+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3c4157ac0 con 0x7fb3c40a7250 2022-07-21T01:42:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.507+0000 7fb3c17fa700 1 -- 172.21.15.3:0/608766657 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16298 (secure 0 0 0) 0x7fb3b40462d0 con 0x7fb3c40a7250 2022-07-21T01:42:58.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:58.511 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":160,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:58.238173+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:56.372231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:57.252984+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb39c053a70 msgr2=0x7fb39c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb39c053a70 0x7fb39c055f30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fb3ac01a000 tx=0x7fb3ac015000).stop 2022-07-21T01:42:58.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40a7250 msgr2=0x7fb3c40ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40a7250 0x7fb3c40ae300 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7fb3c40aa7f0 tx=0x7fb3b4008040).stop 2022-07-21T01:42:58.517 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 shutdown_connections 2022-07-21T01:42:58.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fb39c053a70 0x7fb39c055f30 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.518 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3c4148530 0x7fb3c4157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3c40b1f40 0x7fb3c4148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.519 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 --2- 172.21.15.3:0/608766657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3c40a7250 0x7fb3c40ae300 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 >> 172.21.15.3:0/608766657 conn(0x7fb3c405f840 msgr2=0x7fb3c4147120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:58.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 shutdown_connections 2022-07-21T01:42:58.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.511+0000 7fb3c9eab700 1 -- 172.21.15.3:0/608766657 wait complete. 2022-07-21T01:42:58.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:58.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 Processor -- start 2022-07-21T01:42:58.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 -- start start 2022-07-21T01:42:58.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3eb80b2710 0x7f3eb814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b01a0 0x7f3eb814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb80a7250 con 0x7f3eb80afd60 2022-07-21T01:42:58.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb80a7480 con 0x7f3eb80b2710 2022-07-21T01:42:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80a7720 con 0x7f3eb80b01a0 2022-07-21T01:42:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebd791700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3eb80afd60 0x7f3eb80a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60304/0 (socket says 172.21.15.3:60304) 2022-07-21T01:42:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebd791700 1 -- 172.21.15.3:0/2426015329 learned_addr learned my addr 172.21.15.3:0/2426015329 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb7fff700 1 --2- 172.21.15.3:0/2426015329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b01a0 0x7f3eb814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebcf90700 1 --2- 172.21.15.3:0/2426015329 >> v2:172.21.15.143:3300/0 conn(0x7f3eb80b2710 0x7f3eb814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb77fe700 1 -- 172.21.15.3:0/2426015329 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80295475 0 0) 0x7f3eb80a7250 con 0x7f3eb80afd60 2022-07-21T01:42:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb77fe700 1 -- 172.21.15.3:0/2426015329 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea4003260 con 0x7f3eb80afd60 2022-07-21T01:42:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb7fff700 1 -- 172.21.15.3:0/2426015329 >> v2:172.21.15.143:3300/0 conn(0x7f3eb80b2710 msgr2=0x7f3eb814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb7fff700 1 --2- 172.21.15.3:0/2426015329 >> v2:172.21.15.143:3300/0 conn(0x7f3eb80b2710 0x7f3eb814afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb7fff700 1 -- 172.21.15.3:0/2426015329 >> v1:172.21.15.3:6789/0 conn(0x7f3eb80afd60 legacy=0x7f3eb80a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb7fff700 1 -- 172.21.15.3:0/2426015329 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80a79c0 con 0x7f3eb80b01a0 2022-07-21T01:42:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb7fff700 1 --2- 172.21.15.3:0/2426015329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b01a0 0x7f3eb814b4c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f3ea80032f0 tx=0x7f3ea80091d0).ready entity=mon.2 client_cookie=5765c3e056a7e0a4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb77fe700 1 -- 172.21.15.3:0/2426015329 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea800b600 con 0x7f3eb80b01a0 2022-07-21T01:42:58.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb77fe700 1 -- 172.21.15.3:0/2426015329 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea800b760 con 0x7f3eb80b01a0 2022-07-21T01:42:58.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3eb77fe700 1 -- 172.21.15.3:0/2426015329 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea800b8d0 con 0x7f3eb80b01a0 2022-07-21T01:42:58.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2426015329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b01a0 msgr2=0x7f3eb814b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 --2- 172.21.15.3:0/2426015329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b01a0 0x7f3eb814b4c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f3ea80032f0 tx=0x7f3ea80091d0).stop 2022-07-21T01:42:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2426015329 shutdown_connections 2022-07-21T01:42:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 --2- 172.21.15.3:0/2426015329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b01a0 0x7f3eb814b4c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 --2- 172.21.15.3:0/2426015329 >> v2:172.21.15.143:3300/0 conn(0x7f3eb80b2710 0x7f3eb814afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2426015329 >> 172.21.15.3:0/2426015329 conn(0x7f3eb8060310 msgr2=0x7f3eb80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:58.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2426015329 shutdown_connections 2022-07-21T01:42:58.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.635+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2426015329 wait complete. 2022-07-21T01:42:58.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 Processor -- start 2022-07-21T01:42:58.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 -- start start 2022-07-21T01:42:58.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb80afd60 0x7f3eb80af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3eb80b01a0 0x7f3eb80b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b2710 0x7f3eb80b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b2fa0 con 0x7f3eb80b01a0 2022-07-21T01:42:58.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b31d0 con 0x7f3eb80afd60 2022-07-21T01:42:58.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b3470 con 0x7f3eb80b2710 2022-07-21T01:42:58.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3eb80b01a0 0x7f3eb80b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3eb80b01a0 0x7f3eb80b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52030/0 (socket says 172.21.15.3:52030) 2022-07-21T01:42:58.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb7fff700 1 -- 172.21.15.3:0/2115388870 learned_addr learned my addr 172.21.15.3:0/2115388870 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:58.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebcf90700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb80afd60 0x7f3eb80af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebd791700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b2710 0x7f3eb80b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb7fff700 1 -- 172.21.15.3:0/2115388870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b2710 msgr2=0x7f3eb80b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb7fff700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b2710 0x7f3eb80b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb7fff700 1 -- 172.21.15.3:0/2115388870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb80afd60 msgr2=0x7f3eb80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb7fff700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb80afd60 0x7f3eb80af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb7fff700 1 -- 172.21.15.3:0/2115388870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8009470 con 0x7f3eb80b01a0 2022-07-21T01:42:58.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb7fff700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3eb80b01a0 0x7f3eb80b4d00 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0x7f3ea80032f0 tx=0x7f3ea8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb5ffb700 1 -- 172.21.15.3:0/2115388870 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea8011b60 con 0x7f3eb80b01a0 2022-07-21T01:42:58.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb5ffb700 1 -- 172.21.15.3:0/2115388870 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea8011cc0 con 0x7f3eb80b01a0 2022-07-21T01:42:58.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb80b3710 con 0x7f3eb80b01a0 2022-07-21T01:42:58.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3eb5ffb700 1 -- 172.21.15.3:0/2115388870 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea801f5e0 con 0x7f3eb80b01a0 2022-07-21T01:42:58.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.639+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb80aa5f0 con 0x7f3eb80b01a0 2022-07-21T01:42:58.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.643+0000 7f3eb5ffb700 1 -- 172.21.15.3:0/2115388870 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ea8026030 con 0x7f3eb80b01a0 2022-07-21T01:42:58.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.643+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb80574e0 con 0x7f3eb80b01a0 2022-07-21T01:42:58.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.643+0000 7f3eb5ffb700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e98053a10 0x7f3e98055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.643+0000 7f3ebcf90700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e98053a10 0x7f3e98055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.643+0000 7f3eb5ffb700 1 -- 172.21.15.3:0/2115388870 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3ea806cc00 con 0x7f3eb80b01a0 2022-07-21T01:42:58.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.643+0000 7f3ebcf90700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e98053a10 0x7f3e98055ed0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f3eac00b000 tx=0x7f3eac006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.647+0000 7f3eb5ffb700 1 -- 172.21.15.3:0/2115388870 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ea8043ed0 con 0x7f3eb80b01a0 2022-07-21T01:42:58.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3eb80b5880 con 0x7f3eb80b01a0 2022-07-21T01:42:58.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3eb5ffb700 1 -- 172.21.15.3:0/2115388870 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16298 (secure 0 0 0) 0x7f3ea8024020 con 0x7f3eb80b01a0 2022-07-21T01:42:58.778 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:58.778 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":160,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:58.238173+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:56.372231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:57.252984+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:58.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e98053a10 msgr2=0x7f3e98055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e98053a10 0x7f3e98055ed0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f3eac00b000 tx=0x7f3eac006df0).stop 2022-07-21T01:42:58.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3eb80b01a0 msgr2=0x7f3eb80b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3eb80b01a0 0x7f3eb80b4d00 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0x7f3ea80032f0 tx=0x7f3ea8016000).stop 2022-07-21T01:42:58.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 shutdown_connections 2022-07-21T01:42:58.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3eb80b2710 0x7f3eb80b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3eb80b01a0 0x7f3eb80b4d00 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e98053a10 0x7f3e98055ed0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 --2- 172.21.15.3:0/2115388870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3eb80afd60 0x7f3eb80af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 >> 172.21.15.3:0/2115388870 conn(0x7f3eb8060310 msgr2=0x7f3eb81498d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:58.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 shutdown_connections 2022-07-21T01:42:58.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.775+0000 7f3ebdf92700 1 -- 172.21.15.3:0/2115388870 wait complete. 2022-07-21T01:42:58.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-21T01:42:58.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35d09fe700 1 Processor -- start 2022-07-21T01:42:58.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35d09fe700 1 -- start start 2022-07-21T01:42:58.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35d09fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f35cc0619b0 0x7f35cc0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35d09fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0b5ed0 0x7f35cc0b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35d09fe700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc061d90 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35d09fe700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35cc143fa0 con 0x7f35cc0619b0 2022-07-21T01:42:58.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35d09fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc142f60 con 0x7f35cc0b5ed0 2022-07-21T01:42:58.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35c3fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f35cc0b6ee0 0x7f35cc143890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60314/0 (socket says 172.21.15.3:60314) 2022-07-21T01:42:58.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35c3fff700 1 -- 172.21.15.3:0/1026838263 learned_addr learned my addr 172.21.15.3:0/1026838263 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:58.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35cbfff700 1 --2- 172.21.15.3:0/1026838263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0b5ed0 0x7f35cc0b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35cb7fe700 1 --2- 172.21.15.3:0/1026838263 >> v2:172.21.15.143:3300/0 conn(0x7f35cc0619b0 0x7f35cc0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35caffd700 1 -- 172.21.15.3:0/1026838263 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256187760 0 0) 0x7f35cc061d90 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35caffd700 1 -- 172.21.15.3:0/1026838263 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ac003260 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35cbfff700 1 -- 172.21.15.3:0/1026838263 >> v2:172.21.15.143:3300/0 conn(0x7f35cc0619b0 msgr2=0x7f35cc0b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:58.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35cbfff700 1 --2- 172.21.15.3:0/1026838263 >> v2:172.21.15.143:3300/0 conn(0x7f35cc0619b0 0x7f35cc0b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35cbfff700 1 -- 172.21.15.3:0/1026838263 >> v1:172.21.15.3:6789/0 conn(0x7f35cc0b6ee0 legacy=0x7f35cc143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35cbfff700 1 -- 172.21.15.3:0/1026838263 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc0ba5a0 con 0x7f35cc0b5ed0 2022-07-21T01:42:58.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35cbfff700 1 --2- 172.21.15.3:0/1026838263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0b5ed0 0x7f35cc0b5190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f35c40032f0 tx=0x7f35c40091d0).ready entity=mon.2 client_cookie=8202a5b46f9152ee server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35caffd700 1 -- 172.21.15.3:0/1026838263 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35c400b600 con 0x7f35cc0b5ed0 2022-07-21T01:42:58.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35caffd700 1 -- 172.21.15.3:0/1026838263 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c400b760 con 0x7f35cc0b5ed0 2022-07-21T01:42:58.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.899+0000 7f35caffd700 1 -- 172.21.15.3:0/1026838263 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35c400b8d0 con 0x7f35cc0b5ed0 2022-07-21T01:42:58.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 -- 172.21.15.3:0/1026838263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0b5ed0 msgr2=0x7f35cc0b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 --2- 172.21.15.3:0/1026838263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0b5ed0 0x7f35cc0b5190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f35c40032f0 tx=0x7f35c40091d0).stop 2022-07-21T01:42:58.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 -- 172.21.15.3:0/1026838263 shutdown_connections 2022-07-21T01:42:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 --2- 172.21.15.3:0/1026838263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0b5ed0 0x7f35cc0b5190 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 --2- 172.21.15.3:0/1026838263 >> v2:172.21.15.143:3300/0 conn(0x7f35cc0619b0 0x7f35cc0b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 -- 172.21.15.3:0/1026838263 >> 172.21.15.3:0/1026838263 conn(0x7f35cc06af80 msgr2=0x7f35cc05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 -- 172.21.15.3:0/1026838263 shutdown_connections 2022-07-21T01:42:58.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 -- 172.21.15.3:0/1026838263 wait complete. 2022-07-21T01:42:58.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 Processor -- start 2022-07-21T01:42:58.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 -- start start 2022-07-21T01:42:58.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0619b0 0x7f35cc0b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0b5ed0 0x7f35cc147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35cc0b6ee0 0x7f35cc148090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0af640 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0aece0 con 0x7f35cc0b5ed0 2022-07-21T01:42:58.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35d09fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc148580 con 0x7f35cc0619b0 2022-07-21T01:42:58.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35cbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35cc0b6ee0 0x7f35cc148090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35cbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35cc0b6ee0 0x7f35cc148090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52044/0 (socket says 172.21.15.3:52044) 2022-07-21T01:42:58.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35cbfff700 1 -- 172.21.15.3:0/1623187053 learned_addr learned my addr 172.21.15.3:0/1623187053 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:58.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35c3fff700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0b5ed0 0x7f35cc147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.903+0000 7f35cb7fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0619b0 0x7f35cc0b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35cbfff700 1 -- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0619b0 msgr2=0x7f35cc0b8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35cbfff700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0619b0 0x7f35cc0b8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35cbfff700 1 -- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0b5ed0 msgr2=0x7f35cc147ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:58.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35cbfff700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0b5ed0 0x7f35cc147ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:58.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35cbfff700 1 -- 172.21.15.3:0/1623187053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c4009470 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35cb7fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0619b0 0x7f35cc0b8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T01:42:58.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35c3fff700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0b5ed0 0x7f35cc147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:58.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35cbfff700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35cc0b6ee0 0x7f35cc148090 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f35c40032f0 tx=0x7f35c4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35c97fa700 1 -- 172.21.15.3:0/1623187053 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35c400b8d0 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35c97fa700 1 -- 172.21.15.3:0/1623187053 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c40107a0 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35cc0af870 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35c97fa700 1 -- 172.21.15.3:0/1623187053 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35c400b600 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35cc0afe90 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35cc148a30 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35c97fa700 1 -- 172.21.15.3:0/1623187053 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f35c4027020 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35c97fa700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b00 0x7f35b0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:58.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35c97fa700 1 -- 172.21.15.3:0/1623187053 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f35c406d6d0 con 0x7f35cc0b6ee0 2022-07-21T01:42:58.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35cb7fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b00 0x7f35b0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:58.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.907+0000 7f35cb7fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b00 0x7f35b0055fc0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f35bc00b000 tx=0x7f35bc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:58.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:58.911+0000 7f35c97fa700 1 -- 172.21.15.3:0/1623187053 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35c4044490 con 0x7f35cc0b6ee0 2022-07-21T01:42:59.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.075+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f35cc0b0050 con 0x7f35cc0b6ee0 2022-07-21T01:42:59.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.243+0000 7f35c97fa700 1 -- 172.21.15.3:0/1623187053 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v81) v1 ==== 118+0+0 (secure 0 0 0) 0x7f35c40150b0 con 0x7f35cc0b6ee0 2022-07-21T01:42:59.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.247+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b00 msgr2=0x7f35b0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.247+0000 7f35d09fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b00 0x7f35b0055fc0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f35bc00b000 tx=0x7f35bc008040).stop 2022-07-21T01:42:59.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.247+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35cc0b6ee0 msgr2=0x7f35cc148090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.247+0000 7f35d09fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35cc0b6ee0 0x7f35cc148090 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f35c40032f0 tx=0x7f35c4016000).stop 2022-07-21T01:42:59.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.251+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 shutdown_connections 2022-07-21T01:42:59.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.251+0000 7f35d09fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35cc0b6ee0 0x7f35cc148090 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.251+0000 7f35d09fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f35b0053b00 0x7f35b0055fc0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.251+0000 7f35d09fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0b5ed0 0x7f35cc147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.251+0000 7f35d09fe700 1 --2- 172.21.15.3:0/1623187053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc0619b0 0x7f35cc0b8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.251+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 >> 172.21.15.3:0/1623187053 conn(0x7f35cc06af80 msgr2=0x7f35cc141100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:59.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.251+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 shutdown_connections 2022-07-21T01:42:59.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.251+0000 7f35d09fe700 1 -- 172.21.15.3:0/1623187053 wait complete. 2022-07-21T01:42:59.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:42:59.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde4736b700 1 Processor -- start 2022-07-21T01:42:59.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde4736b700 1 -- start start 2022-07-21T01:42:59.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde4736b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 0x7fde40140500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde4736b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fde400b2aa0 0x7fde401409f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde4736b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde401467f0 con 0x7fde40061480 2022-07-21T01:42:59.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde4736b700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde40062140 con 0x7fde400b2aa0 2022-07-21T01:42:59.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde4736b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde400656b0 con 0x7fde400b6510 2022-07-21T01:42:59.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde45b68700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fde400b2aa0 0x7fde401409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde46b6a700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fde40061480 0x7fde400b0e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60332/0 (socket says 172.21.15.3:60332) 2022-07-21T01:42:59.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde46b6a700 1 -- 172.21.15.3:0/3405406196 learned_addr learned my addr 172.21.15.3:0/3405406196 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:59.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde46369700 1 --2- 172.21.15.3:0/3405406196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 0x7fde40140500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde45b68700 1 -- 172.21.15.3:0/3405406196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 msgr2=0x7fde40140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:59.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde45b68700 1 --2- 172.21.15.3:0/3405406196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 0x7fde40140500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde45b68700 1 -- 172.21.15.3:0/3405406196 >> v1:172.21.15.3:6789/0 conn(0x7fde40061480 legacy=0x7fde400b0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde45b68700 1 -- 172.21.15.3:0/3405406196 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde401476a0 con 0x7fde400b2aa0 2022-07-21T01:42:59.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.375+0000 7fde45b68700 1 --2- 172.21.15.3:0/3405406196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 0x7fde401409f0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7fde3c002f70 tx=0x7fde3c006df0).ready entity=mon.0 client_cookie=31e09b3d5a5e0ebd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:59.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde45367700 1 -- 172.21.15.3:0/3405406196 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde3c00d570 con 0x7fde400b2aa0 2022-07-21T01:42:59.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde45367700 1 -- 172.21.15.3:0/3405406196 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c000e90 con 0x7fde400b2aa0 2022-07-21T01:42:59.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde45367700 1 -- 172.21.15.3:0/3405406196 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde3c00d8c0 con 0x7fde400b2aa0 2022-07-21T01:42:59.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde4736b700 1 -- 172.21.15.3:0/3405406196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 msgr2=0x7fde401409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde4736b700 1 --2- 172.21.15.3:0/3405406196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 0x7fde401409f0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7fde3c002f70 tx=0x7fde3c006df0).stop 2022-07-21T01:42:59.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde4736b700 1 -- 172.21.15.3:0/3405406196 shutdown_connections 2022-07-21T01:42:59.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde4736b700 1 --2- 172.21.15.3:0/3405406196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 0x7fde401409f0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde4736b700 1 --2- 172.21.15.3:0/3405406196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 0x7fde40140500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde4736b700 1 -- 172.21.15.3:0/3405406196 >> 172.21.15.3:0/3405406196 conn(0x7fde40063800 msgr2=0x7fde40063be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:59.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde4736b700 1 -- 172.21.15.3:0/3405406196 shutdown_connections 2022-07-21T01:42:59.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.379+0000 7fde4736b700 1 -- 172.21.15.3:0/3405406196 wait complete. 2022-07-21T01:42:59.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde4736b700 1 Processor -- start 2022-07-21T01:42:59.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde4736b700 1 -- start start 2022-07-21T01:42:59.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde4736b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde40061480 0x7fde401458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde4736b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 0x7fde4013f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde4736b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 0x7fde4013fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde4736b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde40143a20 con 0x7fde400b2aa0 2022-07-21T01:42:59.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde4736b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde400afd20 con 0x7fde40061480 2022-07-21T01:42:59.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde4736b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde40142d40 con 0x7fde400b6510 2022-07-21T01:42:59.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde45b68700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 0x7fde4013f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde45b68700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 0x7fde4013f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52056/0 (socket says 172.21.15.3:52056) 2022-07-21T01:42:59.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde45b68700 1 -- 172.21.15.3:0/1908468497 learned_addr learned my addr 172.21.15.3:0/1908468497 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:59.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde46369700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde40061480 0x7fde401458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde46b6a700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 0x7fde4013fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde45b68700 1 -- 172.21.15.3:0/1908468497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 msgr2=0x7fde4013fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde45b68700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 0x7fde4013fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde45b68700 1 -- 172.21.15.3:0/1908468497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde40061480 msgr2=0x7fde401458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde45b68700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde40061480 0x7fde401458d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde45b68700 1 -- 172.21.15.3:0/1908468497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c00b180 con 0x7fde400b2aa0 2022-07-21T01:42:59.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.383+0000 7fde45b68700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 0x7fde4013f620 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fde3c002f70 tx=0x7fde3c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:59.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde2f7fe700 1 -- 172.21.15.3:0/1908468497 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde3c00bed0 con 0x7fde400b2aa0 2022-07-21T01:42:59.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde2f7fe700 1 -- 172.21.15.3:0/1908468497 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c00de30 con 0x7fde400b2aa0 2022-07-21T01:42:59.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde40143c70 con 0x7fde400b2aa0 2022-07-21T01:42:59.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde2f7fe700 1 -- 172.21.15.3:0/1908468497 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde3c011d30 con 0x7fde400b2aa0 2022-07-21T01:42:59.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde40140000 con 0x7fde400b2aa0 2022-07-21T01:42:59.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde400b0940 con 0x7fde400b2aa0 2022-07-21T01:42:59.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde2f7fe700 1 -- 172.21.15.3:0/1908468497 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fde3c025030 con 0x7fde400b2aa0 2022-07-21T01:42:59.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde2f7fe700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde20053a70 0x7fde20055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde2f7fe700 1 -- 172.21.15.3:0/1908468497 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fde3c06d160 con 0x7fde400b2aa0 2022-07-21T01:42:59.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde46369700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde20053a70 0x7fde20055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.387+0000 7fde46369700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde20053a70 0x7fde20055f30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7fde3000c000 tx=0x7fde30008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:59.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.395+0000 7fde2f7fe700 1 -- 172.21.15.3:0/1908468497 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fde3c043f20 con 0x7fde400b2aa0 2022-07-21T01:42:59.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.559+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde40146a20 con 0x7fde400b2aa0 2022-07-21T01:42:59.563 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.559+0000 7fde2f7fe700 1 -- 172.21.15.3:0/1908468497 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2489 (secure 0 0 0) 0x7fde3c023090 con 0x7fde400b2aa0 2022-07-21T01:42:59.563 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:59.563 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":81,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/781340149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":781340149},{"type":"v1","addr":"172.21.15.143:6833","nonce":781340149}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79},{"gid":8712,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3829565046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3829565046},{"type":"v1","addr":"172.21.15.3:6835","nonce":3829565046}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:58.238508+0000","modified":"2022-07-21T01:42:59.241585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8727},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8727":{"gid":8727,"name":"c","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1787668325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1787668325},{"type":"v1","addr":"172.21.15.3:6837","nonce":1787668325}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-21T01:42:59.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde20053a70 msgr2=0x7fde20055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde20053a70 0x7fde20055f30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7fde3000c000 tx=0x7fde30008040).stop 2022-07-21T01:42:59.565 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 msgr2=0x7fde4013f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 0x7fde4013f620 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fde3c002f70 tx=0x7fde3c015000).stop 2022-07-21T01:42:59.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 shutdown_connections 2022-07-21T01:42:59.566 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde400b6510 0x7fde4013fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde400b2aa0 0x7fde4013f620 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fde20053a70 0x7fde20055f30 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 --2- 172.21.15.3:0/1908468497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde40061480 0x7fde401458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 >> 172.21.15.3:0/1908468497 conn(0x7fde40063800 msgr2=0x7fde400b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:59.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 shutdown_connections 2022-07-21T01:42:59.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.563+0000 7fde4736b700 1 -- 172.21.15.3:0/1908468497 wait complete. 2022-07-21T01:42:59.568 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 81 2022-07-21T01:42:59.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:42:59.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 Processor -- start 2022-07-21T01:42:59.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 -- start start 2022-07-21T01:42:59.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f54640b0270 0x7f5464143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 0x7f54641434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5464067a10 con 0x7f54640b2710 2022-07-21T01:42:59.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54640ad740 con 0x7f54640b0270 2022-07-21T01:42:59.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5464067cb0 con 0x7f54640ac880 2022-07-21T01:42:59.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5462ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f54640b2710 0x7f54641401e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60342/0 (socket says 172.21.15.3:60342) 2022-07-21T01:42:59.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5462ffd700 1 -- 172.21.15.3:0/1312164770 learned_addr learned my addr 172.21.15.3:0/1312164770 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:42:59.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54637fe700 1 --2- 172.21.15.3:0/1312164770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 0x7f54641434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f545a7fc700 1 --2- 172.21.15.3:0/1312164770 >> v2:172.21.15.143:3300/0 conn(0x7f54640b0270 0x7f5464143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54627fc700 1 -- 172.21.15.3:0/1312164770 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838948771 0 0) 0x7f5464067a10 con 0x7f54640b2710 2022-07-21T01:42:59.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54627fc700 1 -- 172.21.15.3:0/1312164770 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5444003260 con 0x7f54640b2710 2022-07-21T01:42:59.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54637fe700 1 -- 172.21.15.3:0/1312164770 >> v2:172.21.15.143:3300/0 conn(0x7f54640b0270 msgr2=0x7f5464143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:59.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54637fe700 1 --2- 172.21.15.3:0/1312164770 >> v2:172.21.15.143:3300/0 conn(0x7f54640b0270 0x7f5464143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54627fc700 1 -- 172.21.15.3:0/1312164770 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 14198201 0 0) 0x7f5444003260 con 0x7f54640b2710 2022-07-21T01:42:59.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54637fe700 1 -- 172.21.15.3:0/1312164770 >> v1:172.21.15.3:6789/0 conn(0x7f54640b2710 legacy=0x7f54641401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54637fe700 1 -- 172.21.15.3:0/1312164770 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464147160 con 0x7f54640ac880 2022-07-21T01:42:59.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54637fe700 1 --2- 172.21.15.3:0/1312164770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 0x7f54641434f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5454001800 tx=0x7f545c0091d0).ready entity=mon.2 client_cookie=3cbe3caee80bd931 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:59.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54627fc700 1 -- 172.21.15.3:0/1312164770 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5454001660 con 0x7f54640ac880 2022-07-21T01:42:59.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54627fc700 1 -- 172.21.15.3:0/1312164770 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c00f510 con 0x7f54640ac880 2022-07-21T01:42:59.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f54627fc700 1 -- 172.21.15.3:0/1312164770 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f545c00f680 con 0x7f54640ac880 2022-07-21T01:42:59.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 -- 172.21.15.3:0/1312164770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 msgr2=0x7f54641434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 --2- 172.21.15.3:0/1312164770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 0x7f54641434f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5454001800 tx=0x7f545c0091d0).stop 2022-07-21T01:42:59.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 -- 172.21.15.3:0/1312164770 shutdown_connections 2022-07-21T01:42:59.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 --2- 172.21.15.3:0/1312164770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 0x7f54641434f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 --2- 172.21.15.3:0/1312164770 >> v2:172.21.15.143:3300/0 conn(0x7f54640b0270 0x7f5464143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 -- 172.21.15.3:0/1312164770 >> 172.21.15.3:0/1312164770 conn(0x7f5464060310 msgr2=0x7f54640b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:59.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 -- 172.21.15.3:0/1312164770 shutdown_connections 2022-07-21T01:42:59.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.679+0000 7f5463fff700 1 -- 172.21.15.3:0/1312164770 wait complete. 2022-07-21T01:42:59.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5463fff700 1 Processor -- start 2022-07-21T01:42:59.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5463fff700 1 -- start start 2022-07-21T01:42:59.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5463fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 0x7f54640ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5463fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54640b0270 0x7f5464152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5463fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54640b2710 0x7f54641564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5463fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464142e50 con 0x7f54640b2710 2022-07-21T01:42:59.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5463fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464155dd0 con 0x7f54640b0270 2022-07-21T01:42:59.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5463fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5464143080 con 0x7f54640ac880 2022-07-21T01:42:59.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f54637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54640b2710 0x7f54641564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f54637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54640b2710 0x7f54641564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52070/0 (socket says 172.21.15.3:52070) 2022-07-21T01:42:59.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f54637fe700 1 -- 172.21.15.3:0/2642150410 learned_addr learned my addr 172.21.15.3:0/2642150410 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:59.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5462ffd700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 0x7f54640ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f545a7fc700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54640b0270 0x7f5464152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f54637fe700 1 -- 172.21.15.3:0/2642150410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 msgr2=0x7f54640ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f54637fe700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 0x7f54640ab9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f54637fe700 1 -- 172.21.15.3:0/2642150410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54640b0270 msgr2=0x7f5464152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f54637fe700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54640b0270 0x7f5464152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f54637fe700 1 -- 172.21.15.3:0/2642150410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545c009470 con 0x7f54640b2710 2022-07-21T01:42:59.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f54637fe700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54640b2710 0x7f54641564d0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f5454001800 tx=0x7f545c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:59.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5460ff9700 1 -- 172.21.15.3:0/2642150410 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f545c0032d0 con 0x7f54640b2710 2022-07-21T01:42:59.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5460ff9700 1 -- 172.21.15.3:0/2642150410 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c011c30 con 0x7f54640b2710 2022-07-21T01:42:59.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5464156030 con 0x7f54640b2710 2022-07-21T01:42:59.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.683+0000 7f5460ff9700 1 -- 172.21.15.3:0/2642150410 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f545c021440 con 0x7f54640b2710 2022-07-21T01:42:59.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.687+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5464144090 con 0x7f54640b2710 2022-07-21T01:42:59.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.687+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54641561c0 con 0x7f54640b2710 2022-07-21T01:42:59.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.687+0000 7f5460ff9700 1 -- 172.21.15.3:0/2642150410 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f545c027020 con 0x7f54640b2710 2022-07-21T01:42:59.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.687+0000 7f5460ff9700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5448053950 0x7f5448055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.687+0000 7f5460ff9700 1 -- 172.21.15.3:0/2642150410 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f545c06d070 con 0x7f54640b2710 2022-07-21T01:42:59.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.687+0000 7f5462ffd700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5448053950 0x7f5448055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.687+0000 7f5462ffd700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5448053950 0x7f5448055e10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f545400d000 tx=0x7f54540019e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:59.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.691+0000 7f5460ff9700 1 -- 172.21.15.3:0/2642150410 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f545c043e70 con 0x7f54640b2710 2022-07-21T01:42:59.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.819+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54640574e0 con 0x7f54640b2710 2022-07-21T01:42:59.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.819+0000 7f5460ff9700 1 -- 172.21.15.3:0/2642150410 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f545c015020 con 0x7f54640b2710 2022-07-21T01:42:59.823 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:42:59.824 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":161,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:59.242307+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:56.372231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:57.252984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:42:59.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5448053950 msgr2=0x7f5448055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5448053950 0x7f5448055e10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f545400d000 tx=0x7f54540019e0).stop 2022-07-21T01:42:59.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54640b2710 msgr2=0x7f54641564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54640b2710 0x7f54641564d0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f5454001800 tx=0x7f545c016000).stop 2022-07-21T01:42:59.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 shutdown_connections 2022-07-21T01:42:59.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54640b2710 0x7f54641564d0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5448053950 0x7f5448055e10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54640b0270 0x7f5464152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 --2- 172.21.15.3:0/2642150410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54640ac880 0x7f54640ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 >> 172.21.15.3:0/2642150410 conn(0x7f5464060310 msgr2=0x7f546413fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:59.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 shutdown_connections 2022-07-21T01:42:59.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.823+0000 7f5463fff700 1 -- 172.21.15.3:0/2642150410 wait complete. 2022-07-21T01:42:59.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:42:59.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.943+0000 7f851d191700 1 Processor -- start 2022-07-21T01:42:59.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.943+0000 7f851d191700 1 -- start start 2022-07-21T01:42:59.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f85180b1f40 0x7f85180b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8518148530 0x7f85180b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85180b04e0 con 0x7f85180a7250 2022-07-21T01:42:59.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85180a9250 con 0x7f8518148530 2022-07-21T01:42:59.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85180b0780 con 0x7f85180b1f40 2022-07-21T01:42:59.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851659c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8518148530 0x7f85180b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851659c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8518148530 0x7f85180b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52076/0 (socket says 172.21.15.3:52076) 2022-07-21T01:42:59.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851659c700 1 -- 172.21.15.3:0/398576570 learned_addr learned my addr 172.21.15.3:0/398576570 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:59.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f8516d9d700 1 --2- 172.21.15.3:0/398576570 >> v2:172.21.15.143:3300/0 conn(0x7f85180b1f40 0x7f85180b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851659c700 1 -- 172.21.15.3:0/398576570 >> v2:172.21.15.143:3300/0 conn(0x7f85180b1f40 msgr2=0x7f85180b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:42:59.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851659c700 1 --2- 172.21.15.3:0/398576570 >> v2:172.21.15.143:3300/0 conn(0x7f85180b1f40 0x7f85180b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851659c700 1 -- 172.21.15.3:0/398576570 >> v1:172.21.15.3:6789/0 conn(0x7f85180a7250 legacy=0x7f85181448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851659c700 1 -- 172.21.15.3:0/398576570 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85180b0a20 con 0x7f8518148530 2022-07-21T01:42:59.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f8516d9d700 1 --2- 172.21.15.3:0/398576570 >> v2:172.21.15.143:3300/0 conn(0x7f85180b1f40 0x7f85180b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:42:59.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851659c700 1 --2- 172.21.15.3:0/398576570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8518148530 0x7f85180b2320 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f85181478c0 tx=0x7f850c006eb0).ready entity=mon.0 client_cookie=b9fadee0a2945c00 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:59.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f8515d9b700 1 -- 172.21.15.3:0/398576570 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850c00d610 con 0x7f8518148530 2022-07-21T01:42:59.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f8515d9b700 1 -- 172.21.15.3:0/398576570 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c000e90 con 0x7f8518148530 2022-07-21T01:42:59.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f8515d9b700 1 -- 172.21.15.3:0/398576570 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850c00d960 con 0x7f8518148530 2022-07-21T01:42:59.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 -- 172.21.15.3:0/398576570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8518148530 msgr2=0x7f85180b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 --2- 172.21.15.3:0/398576570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8518148530 0x7f85180b2320 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f85181478c0 tx=0x7f850c006eb0).stop 2022-07-21T01:42:59.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 -- 172.21.15.3:0/398576570 shutdown_connections 2022-07-21T01:42:59.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 --2- 172.21.15.3:0/398576570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8518148530 0x7f85180b2320 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 --2- 172.21.15.3:0/398576570 >> v2:172.21.15.143:3300/0 conn(0x7f85180b1f40 0x7f85180b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 -- 172.21.15.3:0/398576570 >> 172.21.15.3:0/398576570 conn(0x7f851805f840 msgr2=0x7f851805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:42:59.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 -- 172.21.15.3:0/398576570 shutdown_connections 2022-07-21T01:42:59.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 -- 172.21.15.3:0/398576570 wait complete. 2022-07-21T01:42:59.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 Processor -- start 2022-07-21T01:42:59.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.947+0000 7f851d191700 1 -- start start 2022-07-21T01:42:59.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851d191700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85180a7250 0x7f851814a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851d191700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85180b1f40 0x7f85180ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851d191700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8518148530 0x7f85180ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851d191700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85180afc90 con 0x7f85180b1f40 2022-07-21T01:42:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851d191700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85180a9c40 con 0x7f85180a7250 2022-07-21T01:42:59.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851d191700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85180af750 con 0x7f8518148530 2022-07-21T01:42:59.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85180b1f40 0x7f85180ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85180b1f40 0x7f85180ae1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52082/0 (socket says 172.21.15.3:52082) 2022-07-21T01:42:59.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f8516d9d700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85180a7250 0x7f851814a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851659c700 1 -- 172.21.15.3:0/3803992474 learned_addr learned my addr 172.21.15.3:0/3803992474 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:42:59.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851759e700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8518148530 0x7f85180ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851659c700 1 -- 172.21.15.3:0/3803992474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8518148530 msgr2=0x7f85180ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851659c700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8518148530 0x7f85180ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851659c700 1 -- 172.21.15.3:0/3803992474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85180a7250 msgr2=0x7f851814a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:42:59.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851659c700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85180a7250 0x7f851814a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:42:59.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851659c700 1 -- 172.21.15.3:0/3803992474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c00b1e0 con 0x7f85180b1f40 2022-07-21T01:42:59.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851659c700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85180b1f40 0x7f85180ae1a0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f85181478c0 tx=0x7f850c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f8507fff700 1 -- 172.21.15.3:0/3803992474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850c0068b0 con 0x7f85180b1f40 2022-07-21T01:42:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f8507fff700 1 -- 172.21.15.3:0/3803992474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c002f50 con 0x7f85180b1f40 2022-07-21T01:42:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85180af9f0 con 0x7f85180b1f40 2022-07-21T01:42:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f8507fff700 1 -- 172.21.15.3:0/3803992474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850c011b80 con 0x7f85180b1f40 2022-07-21T01:42:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.951+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85180abcb0 con 0x7f85180b1f40 2022-07-21T01:42:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.955+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85180adee0 con 0x7f85180b1f40 2022-07-21T01:42:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.955+0000 7f8507fff700 1 -- 172.21.15.3:0/3803992474 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f850c025030 con 0x7f85180b1f40 2022-07-21T01:42:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.955+0000 7f8507fff700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84f4053a70 0x7f84f4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:42:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.955+0000 7f8516d9d700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84f4053a70 0x7f84f4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:42:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.955+0000 7f8507fff700 1 -- 172.21.15.3:0/3803992474 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f850c06d070 con 0x7f85180b1f40 2022-07-21T01:42:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.955+0000 7f8516d9d700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84f4053a70 0x7f84f4055f30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f851000d000 tx=0x7f8510008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:42:59.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:42:59.963+0000 7f8507fff700 1 -- 172.21.15.3:0/3803992474 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f85180adee0 con 0x7f85180b1f40 2022-07-21T01:43:00.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.131+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f85180adee0 con 0x7f85180b1f40 2022-07-21T01:43:00.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f8507fff700 1 -- 172.21.15.3:0/3803992474 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v38) v1 ==== 184+0+0 (secure 0 0 0) 0x7f85180adee0 con 0x7f85180b1f40 2022-07-21T01:43:00.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84f4053a70 msgr2=0x7f84f4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84f4053a70 0x7f84f4055f30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f851000d000 tx=0x7f8510008040).stop 2022-07-21T01:43:00.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85180b1f40 msgr2=0x7f85180ae1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85180b1f40 0x7f85180ae1a0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f85181478c0 tx=0x7f850c015000).stop 2022-07-21T01:43:00.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 shutdown_connections 2022-07-21T01:43:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8518148530 0x7f85180ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85180b1f40 0x7f85180ae1a0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f84f4053a70 0x7f84f4055f30 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 --2- 172.21.15.3:0/3803992474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85180a7250 0x7f851814a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.139+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 >> 172.21.15.3:0/3803992474 conn(0x7f851805f840 msgr2=0x7f85180b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:00.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.143+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 shutdown_connections 2022-07-21T01:43:00.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.143+0000 7f851d191700 1 -- 172.21.15.3:0/3803992474 wait complete. 2022-07-21T01:43:00.147 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.0 2022-07-21T01:43:00.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-21T01:43:00.270 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39c14be700 1 Processor -- start 2022-07-21T01:43:00.271 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39c14be700 1 -- start start 2022-07-21T01:43:00.276 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39c14be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f39bc0614b0 0x7f39bc061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39c14be700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f39bc0ad750 0x7f39bc0b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39c14be700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39bc0aee80 con 0x7f39bc1405d0 2022-07-21T01:43:00.277 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39c14be700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39bc0b70b0 con 0x7f39bc0ad750 2022-07-21T01:43:00.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39c14be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39bc0af0b0 con 0x7f39bc0614b0 2022-07-21T01:43:00.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39bb7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f39bc0ad750 0x7f39bc0b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39bb7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f39bc0ad750 0x7f39bc0b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52088/0 (socket says 172.21.15.3:52088) 2022-07-21T01:43:00.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39bb7fe700 1 -- 172.21.15.3:0/431843000 learned_addr learned my addr 172.21.15.3:0/431843000 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:00.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.267+0000 7f39bbfff700 1 --2- 172.21.15.3:0/431843000 >> v2:172.21.15.143:3300/0 conn(0x7f39bc0614b0 0x7f39bc061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39baffd700 1 -- 172.21.15.3:0/431843000 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953803177 0 0) 0x7f39bc0aee80 con 0x7f39bc1405d0 2022-07-21T01:43:00.280 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39baffd700 1 -- 172.21.15.3:0/431843000 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f399c003260 con 0x7f39bc1405d0 2022-07-21T01:43:00.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39bb7fe700 1 -- 172.21.15.3:0/431843000 >> v2:172.21.15.143:3300/0 conn(0x7f39bc0614b0 msgr2=0x7f39bc061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:00.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39bb7fe700 1 --2- 172.21.15.3:0/431843000 >> v2:172.21.15.143:3300/0 conn(0x7f39bc0614b0 0x7f39bc061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.281 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39bb7fe700 1 -- 172.21.15.3:0/431843000 >> v1:172.21.15.143:6789/0 conn(0x7f39bc1405d0 legacy=0x7f39bc1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39bb7fe700 1 -- 172.21.15.3:0/431843000 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc0af350 con 0x7f39bc0ad750 2022-07-21T01:43:00.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39bb7fe700 1 --2- 172.21.15.3:0/431843000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 0x7f39bc0b9ff0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7f39bc0aed30 tx=0x7f39a4006eb0).ready entity=mon.0 client_cookie=5e7145dcc37fb05b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:00.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39baffd700 1 -- 172.21.15.3:0/431843000 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39a400e6b0 con 0x7f39bc0ad750 2022-07-21T01:43:00.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39baffd700 1 -- 172.21.15.3:0/431843000 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a4000e90 con 0x7f39bc0ad750 2022-07-21T01:43:00.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39baffd700 1 -- 172.21.15.3:0/431843000 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39a400ea40 con 0x7f39bc0ad750 2022-07-21T01:43:00.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39c14be700 1 -- 172.21.15.3:0/431843000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 msgr2=0x7f39bc0b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39c14be700 1 --2- 172.21.15.3:0/431843000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 0x7f39bc0b9ff0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7f39bc0aed30 tx=0x7f39a4006eb0).stop 2022-07-21T01:43:00.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39c14be700 1 -- 172.21.15.3:0/431843000 shutdown_connections 2022-07-21T01:43:00.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39c14be700 1 --2- 172.21.15.3:0/431843000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 0x7f39bc0b9ff0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39c14be700 1 --2- 172.21.15.3:0/431843000 >> v2:172.21.15.143:3300/0 conn(0x7f39bc0614b0 0x7f39bc061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39c14be700 1 -- 172.21.15.3:0/431843000 >> 172.21.15.3:0/431843000 conn(0x7f39bc066f70 msgr2=0x7f39bc067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:00.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39c14be700 1 -- 172.21.15.3:0/431843000 shutdown_connections 2022-07-21T01:43:00.287 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.271+0000 7f39c14be700 1 -- 172.21.15.3:0/431843000 wait complete. 2022-07-21T01:43:00.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c14be700 1 Processor -- start 2022-07-21T01:43:00.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c14be700 1 -- start start 2022-07-21T01:43:00.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c14be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39bc0614b0 0x7f39bc0b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c14be700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 0x7f39bc14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.294 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c14be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39bc1405d0 0x7f39bc152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c14be700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc0b0930 con 0x7f39bc0ad750 2022-07-21T01:43:00.295 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c14be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc143920 con 0x7f39bc1405d0 2022-07-21T01:43:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c14be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc143b80 con 0x7f39bc0614b0 2022-07-21T01:43:00.296 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 0x7f39bc14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 0x7f39bc14a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52096/0 (socket says 172.21.15.3:52096) 2022-07-21T01:43:00.297 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bb7fe700 1 -- 172.21.15.3:0/756390640 learned_addr learned my addr 172.21.15.3:0/756390640 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bbfff700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39bc0614b0 0x7f39bc0b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c0cbd700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39bc1405d0 0x7f39bc152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bb7fe700 1 -- 172.21.15.3:0/756390640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39bc0614b0 msgr2=0x7f39bc0b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bb7fe700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39bc0614b0 0x7f39bc0b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bb7fe700 1 -- 172.21.15.3:0/756390640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39bc1405d0 msgr2=0x7f39bc152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bb7fe700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39bc1405d0 0x7f39bc152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bb7fe700 1 -- 172.21.15.3:0/756390640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a400b1e0 con 0x7f39bc0ad750 2022-07-21T01:43:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39bb7fe700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 0x7f39bc14a6d0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f39bc0aed30 tx=0x7f39a4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:00.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39b97fa700 1 -- 172.21.15.3:0/756390640 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39a400ea30 con 0x7f39bc0ad750 2022-07-21T01:43:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39b97fa700 1 -- 172.21.15.3:0/756390640 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a4002f50 con 0x7f39bc0ad750 2022-07-21T01:43:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.275+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39bc143e20 con 0x7f39bc0ad750 2022-07-21T01:43:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.279+0000 7f39b97fa700 1 -- 172.21.15.3:0/756390640 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39a4011c30 con 0x7f39bc0ad750 2022-07-21T01:43:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.279+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39bc0b37e0 con 0x7f39bc0ad750 2022-07-21T01:43:00.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.279+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc0af5f0 con 0x7f39bc0ad750 2022-07-21T01:43:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.279+0000 7f39b97fa700 1 -- 172.21.15.3:0/756390640 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f39a401a040 con 0x7f39bc0ad750 2022-07-21T01:43:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.279+0000 7f39b97fa700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f39a00539e0 0x7f39a0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.279+0000 7f39bbfff700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f39a00539e0 0x7f39a0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.279+0000 7f39b97fa700 1 -- 172.21.15.3:0/756390640 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f39a406cf70 con 0x7f39bc0ad750 2022-07-21T01:43:00.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.279+0000 7f39bbfff700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f39a00539e0 0x7f39a0055ea0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f39ac00b000 tx=0x7f39ac006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:00.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.283+0000 7f39b97fa700 1 -- 172.21.15.3:0/756390640 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f39a4043d10 con 0x7f39bc0ad750 2022-07-21T01:43:00.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.451+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f39bc143fb0 con 0x7f39bc0ad750 2022-07-21T01:43:00.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39b97fa700 1 -- 172.21.15.3:0/756390640 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v39) v1 ==== 184+0+0 (secure 0 0 0) 0x7f39a40472b0 con 0x7f39bc0ad750 2022-07-21T01:43:00.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f39a00539e0 msgr2=0x7f39a0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f39a00539e0 0x7f39a0055ea0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f39ac00b000 tx=0x7f39ac006df0).stop 2022-07-21T01:43:00.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 msgr2=0x7f39bc14a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 0x7f39bc14a6d0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f39bc0aed30 tx=0x7f39a4015000).stop 2022-07-21T01:43:00.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 shutdown_connections 2022-07-21T01:43:00.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f39a00539e0 0x7f39a0055ea0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.467 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39bc1405d0 0x7f39bc152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39bc0ad750 0x7f39bc14a6d0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 --2- 172.21.15.3:0/756390640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39bc0614b0 0x7f39bc0b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.468 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 >> 172.21.15.3:0/756390640 conn(0x7f39bc066f70 msgr2=0x7f39bc0acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:00.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 shutdown_connections 2022-07-21T01:43:00.469 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.459+0000 7f39c14be700 1 -- 172.21.15.3:0/756390640 wait complete. 2022-07-21T01:43:00.470 INFO:teuthology.orchestra.run.smithi003.stderr:updated caps for client.1 2022-07-21T01:43:00.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:43:00.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c764cd700 1 Processor -- start 2022-07-21T01:43:00.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c764cd700 1 -- start start 2022-07-21T01:43:00.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c764cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700b1f40 0x7f3c700b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c764cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c700a7250 0x7f3c701448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c764cd700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c700b04e0 con 0x7f3c70148530 2022-07-21T01:43:00.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c764cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c700a9250 con 0x7f3c700a7250 2022-07-21T01:43:00.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c764cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700b0780 con 0x7f3c700b1f40 2022-07-21T01:43:00.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c67fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f3c70148530 0x7f3c700b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60382/0 (socket says 172.21.15.3:60382) 2022-07-21T01:43:00.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c67fff700 1 -- 172.21.15.3:0/87871664 learned_addr learned my addr 172.21.15.3:0/87871664 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:00.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6ffff700 1 --2- 172.21.15.3:0/87871664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700b1f40 0x7f3c700b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c74a6a700 1 --2- 172.21.15.3:0/87871664 >> v2:172.21.15.143:3300/0 conn(0x7f3c700a7250 0x7f3c701448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6f7fe700 1 -- 172.21.15.3:0/87871664 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171070738 0 0) 0x7f3c700b04e0 con 0x7f3c70148530 2022-07-21T01:43:00.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6f7fe700 1 -- 172.21.15.3:0/87871664 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c50003260 con 0x7f3c70148530 2022-07-21T01:43:00.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6ffff700 1 -- 172.21.15.3:0/87871664 >> v2:172.21.15.143:3300/0 conn(0x7f3c700a7250 msgr2=0x7f3c701448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:00.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6ffff700 1 --2- 172.21.15.3:0/87871664 >> v2:172.21.15.143:3300/0 conn(0x7f3c700a7250 0x7f3c701448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6ffff700 1 -- 172.21.15.3:0/87871664 >> v1:172.21.15.3:6789/0 conn(0x7f3c70148530 legacy=0x7f3c700b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6ffff700 1 -- 172.21.15.3:0/87871664 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c700b0a20 con 0x7f3c700b1f40 2022-07-21T01:43:00.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6ffff700 1 --2- 172.21.15.3:0/87871664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700b1f40 0x7f3c700b1160 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f3c60002f70 tx=0x7f3c60006df0).ready entity=mon.2 client_cookie=6953885330e99546 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:00.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6f7fe700 1 -- 172.21.15.3:0/87871664 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6000d4a0 con 0x7f3c700b1f40 2022-07-21T01:43:00.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6f7fe700 1 -- 172.21.15.3:0/87871664 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c60000e90 con 0x7f3c700b1f40 2022-07-21T01:43:00.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c6f7fe700 1 -- 172.21.15.3:0/87871664 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6000d7f0 con 0x7f3c700b1f40 2022-07-21T01:43:00.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c764cd700 1 -- 172.21.15.3:0/87871664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700b1f40 msgr2=0x7f3c700b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.579+0000 7f3c764cd700 1 --2- 172.21.15.3:0/87871664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700b1f40 0x7f3c700b1160 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f3c60002f70 tx=0x7f3c60006df0).stop 2022-07-21T01:43:00.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 -- 172.21.15.3:0/87871664 shutdown_connections 2022-07-21T01:43:00.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 --2- 172.21.15.3:0/87871664 >> v2:172.21.15.143:3300/0 conn(0x7f3c700a7250 0x7f3c701448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 --2- 172.21.15.3:0/87871664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700b1f40 0x7f3c700b1160 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 -- 172.21.15.3:0/87871664 >> 172.21.15.3:0/87871664 conn(0x7f3c7005f840 msgr2=0x7f3c7005fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:00.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 -- 172.21.15.3:0/87871664 shutdown_connections 2022-07-21T01:43:00.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 -- 172.21.15.3:0/87871664 wait complete. 2022-07-21T01:43:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 Processor -- start 2022-07-21T01:43:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 -- start start 2022-07-21T01:43:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700a7250 0x7f3c700ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c700b1f40 0x7f3c700abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c70148530 0x7f3c7014b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7014b0e0 con 0x7f3c700b1f40 2022-07-21T01:43:00.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700a96e0 con 0x7f3c70148530 2022-07-21T01:43:00.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c764cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700a9980 con 0x7f3c700a7250 2022-07-21T01:43:00.597 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c67fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c700b1f40 0x7f3c700abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c6ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700a7250 0x7f3c700ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c6ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700a7250 0x7f3c700ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:59024/0 (socket says 172.21.15.3:59024) 2022-07-21T01:43:00.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c6ffff700 1 -- 172.21.15.3:0/452060551 learned_addr learned my addr 172.21.15.3:0/452060551 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:00.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c74a6a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c70148530 0x7f3c7014b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.598 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c67fff700 1 -- 172.21.15.3:0/452060551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700a7250 msgr2=0x7f3c700ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c67fff700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700a7250 0x7f3c700ae300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c67fff700 1 -- 172.21.15.3:0/452060551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c70148530 msgr2=0x7f3c7014b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c67fff700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c70148530 0x7f3c7014b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c67fff700 1 -- 172.21.15.3:0/452060551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6000b180 con 0x7f3c700b1f40 2022-07-21T01:43:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.583+0000 7f3c67fff700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c700b1f40 0x7f3c700abe80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f3c58001110 tx=0x7f3c58007a20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c6dffb700 1 -- 172.21.15.3:0/452060551 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c5800cd20 con 0x7f3c700b1f40 2022-07-21T01:43:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c6dffb700 1 -- 172.21.15.3:0/452060551 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5800ce80 con 0x7f3c700b1f40 2022-07-21T01:43:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c700a9c20 con 0x7f3c700b1f40 2022-07-21T01:43:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c6dffb700 1 -- 172.21.15.3:0/452060551 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c580072c0 con 0x7f3c700b1f40 2022-07-21T01:43:00.601 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c70149dc0 con 0x7f3c700b1f40 2022-07-21T01:43:00.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c700574e0 con 0x7f3c700b1f40 2022-07-21T01:43:00.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c6dffb700 1 -- 172.21.15.3:0/452060551 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3c58010070 con 0x7f3c700b1f40 2022-07-21T01:43:00.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c6dffb700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c54053a70 0x7f3c54055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:00.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c6dffb700 1 -- 172.21.15.3:0/452060551 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3c5806b4f0 con 0x7f3c700b1f40 2022-07-21T01:43:00.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.587+0000 7f3c6ffff700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c54053a70 0x7f3c54055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:00.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.591+0000 7f3c6ffff700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c54053a70 0x7f3c54055f30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f3c6001a000 tx=0x7f3c60014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:00.603 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.595+0000 7f3c6dffb700 1 -- 172.21.15.3:0/452060551 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3c58042140 con 0x7f3c700b1f40 2022-07-21T01:43:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.755+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c700447e0 con 0x7f3c700b1f40 2022-07-21T01:43:00.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.755+0000 7f3c6dffb700 1 -- 172.21.15.3:0/452060551 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2489 (secure 0 0 0) 0x7f3c58045760 con 0x7f3c700b1f40 2022-07-21T01:43:00.759 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:43:00.759 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":81,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/781340149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":781340149},{"type":"v1","addr":"172.21.15.143:6833","nonce":781340149}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79},{"gid":8712,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3829565046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3829565046},{"type":"v1","addr":"172.21.15.3:6835","nonce":3829565046}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:58.238508+0000","modified":"2022-07-21T01:42:59.241585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8727},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8727":{"gid":8727,"name":"c","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1787668325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1787668325},{"type":"v1","addr":"172.21.15.3:6837","nonce":1787668325}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-21T01:43:00.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.755+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c54053a70 msgr2=0x7f3c54055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.755+0000 7f3c764cd700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c54053a70 0x7f3c54055f30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f3c6001a000 tx=0x7f3c60014000).stop 2022-07-21T01:43:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.755+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c700b1f40 msgr2=0x7f3c700abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.755+0000 7f3c764cd700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c700b1f40 0x7f3c700abe80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f3c58001110 tx=0x7f3c58007a20).stop 2022-07-21T01:43:00.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.759+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 shutdown_connections 2022-07-21T01:43:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.759+0000 7f3c764cd700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3c54053a70 0x7f3c54055f30 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.759+0000 7f3c764cd700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c70148530 0x7f3c7014b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.759+0000 7f3c764cd700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c700b1f40 0x7f3c700abe80 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.759+0000 7f3c764cd700 1 --2- 172.21.15.3:0/452060551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c700a7250 0x7f3c700ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.759+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 >> 172.21.15.3:0/452060551 conn(0x7f3c7005f840 msgr2=0x7f3c700a76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.759+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 shutdown_connections 2022-07-21T01:43:00.763 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:00.759+0000 7f3c764cd700 1 -- 172.21.15.3:0/452060551 wait complete. 2022-07-21T01:43:00.764 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 81 2022-07-21T01:43:01.074 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:58.238508+0000', 'modified': '2022-07-21T01:42:59.241585+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8727}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8727': {'gid': 8727, 'name': 'c', 'rank': 0, 'incarnation': 80, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/1787668325', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 1787668325}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 1787668325}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:43:01.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:43:01.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:43:01.197 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8727,"want_state":"up:active","state":"up:active","rank_uptime":2.9457466480000001,"mdsmap_epoch":81,"osdmap_epoch":161,"osdmap_epoch_barrier":160,"uptime":7.5870086150000002} 2022-07-21T01:43:01.198 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8727, "mdsmap_epoch": 81, "osdmap_epoch": 161, "osdmap_epoch_barrier": 160, "rank_uptime": 2.945746648, "state": "up:active", "uptime": 7.587008615, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:43:01.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:43:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.303+0000 7f5188a0a700 1 Processor -- start 2022-07-21T01:43:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.303+0000 7f5188a0a700 1 -- start start 2022-07-21T01:43:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840ad4b0 0x7f51840af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51840aff30 con 0x7f51840af450 2022-07-21T01:43:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518405c7a0 con 0x7f51840ad9e0 2022-07-21T01:43:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5184140990 con 0x7f51840ad4b0 2022-07-21T01:43:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5182d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f51840af450 0x7f51841432e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60396/0 (socket says 172.21.15.3:60396) 2022-07-21T01:43:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5182d9d700 1 -- 172.21.15.3:0/1241449639 learned_addr learned my addr 172.21.15.3:0/1241449639 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f517a59c700 1 --2- 172.21.15.3:0/1241449639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840ad4b0 0x7f51840af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f518259c700 1 -- 172.21.15.3:0/1241449639 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831334034 0 0) 0x7f51840aff30 con 0x7f51840af450 2022-07-21T01:43:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f518259c700 1 -- 172.21.15.3:0/1241449639 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5164003260 con 0x7f51840af450 2022-07-21T01:43:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f518259c700 1 -- 172.21.15.3:0/1241449639 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432994318 0 0) 0x7f518405c7a0 con 0x7f51840ad9e0 2022-07-21T01:43:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f518259c700 1 -- 172.21.15.3:0/1241449639 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51840aff30 con 0x7f51840ad9e0 2022-07-21T01:43:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f517a59c700 1 -- 172.21.15.3:0/1241449639 >> v1:172.21.15.143:6789/0 conn(0x7f51840ad9e0 legacy=0x7f51840a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f517a59c700 1 -- 172.21.15.3:0/1241449639 >> v1:172.21.15.3:6789/0 conn(0x7f51840af450 legacy=0x7f51841432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f517a59c700 1 -- 172.21.15.3:0/1241449639 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51841439f0 con 0x7f51840ad4b0 2022-07-21T01:43:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f517a59c700 1 --2- 172.21.15.3:0/1241449639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840ad4b0 0x7f51840af830 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f516c002f70 tx=0x7f516c006df0).ready entity=mon.2 client_cookie=1bbfac0e9a73c774 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f518259c700 1 -- 172.21.15.3:0/1241449639 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516c00d5c0 con 0x7f51840ad4b0 2022-07-21T01:43:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f518259c700 1 -- 172.21.15.3:0/1241449639 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c000e90 con 0x7f51840ad4b0 2022-07-21T01:43:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f518259c700 1 -- 172.21.15.3:0/1241449639 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516c00d910 con 0x7f51840ad4b0 2022-07-21T01:43:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 -- 172.21.15.3:0/1241449639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840ad4b0 msgr2=0x7f51840af830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 --2- 172.21.15.3:0/1241449639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840ad4b0 0x7f51840af830 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f516c002f70 tx=0x7f516c006df0).stop 2022-07-21T01:43:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 -- 172.21.15.3:0/1241449639 shutdown_connections 2022-07-21T01:43:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 --2- 172.21.15.3:0/1241449639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840ad4b0 0x7f51840af830 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 -- 172.21.15.3:0/1241449639 >> 172.21.15.3:0/1241449639 conn(0x7f51840612d0 msgr2=0x7f5184062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 -- 172.21.15.3:0/1241449639 shutdown_connections 2022-07-21T01:43:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.307+0000 7f5188a0a700 1 -- 172.21.15.3:0/1241449639 wait complete. 2022-07-21T01:43:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 Processor -- start 2022-07-21T01:43:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 -- start start 2022-07-21T01:43:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51840ad4b0 0x7f51840aa240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51840ad9e0 0x7f51840abee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840af450 0x7f51840ac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51840b6530 con 0x7f51840ad9e0 2022-07-21T01:43:01.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51840b5e90 con 0x7f51840ad4b0 2022-07-21T01:43:01.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51840b4a20 con 0x7f51840af450 2022-07-21T01:43:01.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f517a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51840ad9e0 0x7f51840abee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5182d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51840ad4b0 0x7f51840aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f517a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51840ad9e0 0x7f51840abee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52124/0 (socket says 172.21.15.3:52124) 2022-07-21T01:43:01.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f517a59c700 1 -- 172.21.15.3:0/772581622 learned_addr learned my addr 172.21.15.3:0/772581622 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:01.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f518359e700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840af450 0x7f51840ac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f517a59c700 1 -- 172.21.15.3:0/772581622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840af450 msgr2=0x7f51840ac3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f517a59c700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840af450 0x7f51840ac3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f517a59c700 1 -- 172.21.15.3:0/772581622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51840ad4b0 msgr2=0x7f51840aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f517a59c700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51840ad4b0 0x7f51840aa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f517a59c700 1 -- 172.21.15.3:0/772581622 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c00b180 con 0x7f51840ad9e0 2022-07-21T01:43:01.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f517a59c700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51840ad9e0 0x7f51840abee0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f51840b5fd0 tx=0x7f516c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:01.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5180d99700 1 -- 172.21.15.3:0/772581622 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516c013da0 con 0x7f51840ad9e0 2022-07-21T01:43:01.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5180d99700 1 -- 172.21.15.3:0/772581622 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c01e400 con 0x7f51840ad9e0 2022-07-21T01:43:01.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5180d99700 1 -- 172.21.15.3:0/772581622 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516c01e760 con 0x7f51840ad9e0 2022-07-21T01:43:01.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51840b4cc0 con 0x7f51840ad9e0 2022-07-21T01:43:01.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.311+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51840ac8c0 con 0x7f51840ad9e0 2022-07-21T01:43:01.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.315+0000 7f5180d99700 1 -- 172.21.15.3:0/772581622 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f516c01c070 con 0x7f51840ad9e0 2022-07-21T01:43:01.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.315+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51840574e0 con 0x7f51840ad9e0 2022-07-21T01:43:01.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.315+0000 7f5180d99700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5168053b00 0x7f5168055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.330 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.315+0000 7f5180d99700 1 -- 172.21.15.3:0/772581622 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f516c06cae0 con 0x7f51840ad9e0 2022-07-21T01:43:01.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.315+0000 7f5182d9d700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5168053b00 0x7f5168055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.315+0000 7f5182d9d700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5168053b00 0x7f5168055fc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f517400d000 tx=0x7f5174009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:01.331 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.327+0000 7f5180d99700 1 -- 172.21.15.3:0/772581622 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f516c043790 con 0x7f51840ad9e0 2022-07-21T01:43:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.455+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5184061b20 con 0x7f51840ad9e0 2022-07-21T01:43:01.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.455+0000 7f5180d99700 1 -- 172.21.15.3:0/772581622 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f516c046db0 con 0x7f51840ad9e0 2022-07-21T01:43:01.459 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:43:01.459 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":161,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:59.242307+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:56.372231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:57.252984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:43:01.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5168053b00 msgr2=0x7f5168055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5168053b00 0x7f5168055fc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f517400d000 tx=0x7f5174009040).stop 2022-07-21T01:43:01.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51840ad9e0 msgr2=0x7f51840abee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51840ad9e0 0x7f51840abee0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f51840b5fd0 tx=0x7f516c014000).stop 2022-07-21T01:43:01.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 shutdown_connections 2022-07-21T01:43:01.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51840af450 0x7f51840ac3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51840ad9e0 0x7f51840abee0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5168053b00 0x7f5168055fc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 --2- 172.21.15.3:0/772581622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51840ad4b0 0x7f51840aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 >> 172.21.15.3:0/772581622 conn(0x7f51840612d0 msgr2=0x7f518413cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:01.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 shutdown_connections 2022-07-21T01:43:01.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.459+0000 7f5188a0a700 1 -- 172.21.15.3:0/772581622 wait complete. 2022-07-21T01:43:01.474 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:42:56.372231+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:42:57.252984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:43:01.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:43:01.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.587+0000 7fd39ef72700 1 Processor -- start 2022-07-21T01:43:01.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.587+0000 7fd39ef72700 1 -- start start 2022-07-21T01:43:01.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.587+0000 7fd39ef72700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3980ad0b0 0x7fd3980ad520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.587+0000 7fd39ef72700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3980b2dc0 con 0x7fd3980b29e0 2022-07-21T01:43:01.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.587+0000 7fd39ef72700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3980b17f0 con 0x7fd39813c870 2022-07-21T01:43:01.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.587+0000 7fd39ef72700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980b1340 con 0x7fd3980ad0b0 2022-07-21T01:43:01.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.587+0000 7fd39df70700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fd3980b29e0 0x7fd3980b0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60410/0 (socket says 172.21.15.3:60410) 2022-07-21T01:43:01.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.587+0000 7fd39df70700 1 -- 172.21.15.3:0/1552281519 learned_addr learned my addr 172.21.15.3:0/1552281519 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:01.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39e771700 1 --2- 172.21.15.3:0/1552281519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3980ad0b0 0x7fd3980ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39cf6e700 1 -- 172.21.15.3:0/1552281519 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259371482 0 0) 0x7fd3980b2dc0 con 0x7fd3980b29e0 2022-07-21T01:43:01.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39cf6e700 1 -- 172.21.15.3:0/1552281519 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd384003260 con 0x7fd3980b29e0 2022-07-21T01:43:01.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39cf6e700 1 -- 172.21.15.3:0/1552281519 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998681424 0 0) 0x7fd3980b17f0 con 0x7fd39813c870 2022-07-21T01:43:01.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39cf6e700 1 -- 172.21.15.3:0/1552281519 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3980b2dc0 con 0x7fd39813c870 2022-07-21T01:43:01.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39e771700 1 -- 172.21.15.3:0/1552281519 >> v1:172.21.15.143:6789/0 conn(0x7fd39813c870 legacy=0x7fd39813ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39e771700 1 -- 172.21.15.3:0/1552281519 >> v1:172.21.15.3:6789/0 conn(0x7fd3980b29e0 legacy=0x7fd3980b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39cf6e700 1 -- 172.21.15.3:0/1552281519 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3757254330 0 0) 0x7fd384003260 con 0x7fd3980b29e0 2022-07-21T01:43:01.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39e771700 1 -- 172.21.15.3:0/1552281519 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980b1a90 con 0x7fd3980ad0b0 2022-07-21T01:43:01.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39e771700 1 --2- 172.21.15.3:0/1552281519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3980ad0b0 0x7fd3980ad520 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fd388001110 tx=0x7fd3900091d0).ready entity=mon.2 client_cookie=ef081f1940468f8c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:01.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39cf6e700 1 -- 172.21.15.3:0/1552281519 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd388001680 con 0x7fd3980ad0b0 2022-07-21T01:43:01.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39cf6e700 1 -- 172.21.15.3:0/1552281519 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39000f510 con 0x7fd3980ad0b0 2022-07-21T01:43:01.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39cf6e700 1 -- 172.21.15.3:0/1552281519 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd39000f680 con 0x7fd3980ad0b0 2022-07-21T01:43:01.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39ef72700 1 -- 172.21.15.3:0/1552281519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3980ad0b0 msgr2=0x7fd3980ad520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39ef72700 1 --2- 172.21.15.3:0/1552281519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3980ad0b0 0x7fd3980ad520 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fd388001110 tx=0x7fd3900091d0).stop 2022-07-21T01:43:01.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39ef72700 1 -- 172.21.15.3:0/1552281519 shutdown_connections 2022-07-21T01:43:01.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39ef72700 1 --2- 172.21.15.3:0/1552281519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3980ad0b0 0x7fd3980ad520 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39ef72700 1 -- 172.21.15.3:0/1552281519 >> 172.21.15.3:0/1552281519 conn(0x7fd3980613c0 msgr2=0x7fd398062f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:01.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39ef72700 1 -- 172.21.15.3:0/1552281519 shutdown_connections 2022-07-21T01:43:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.591+0000 7fd39ef72700 1 -- 172.21.15.3:0/1552281519 wait complete. 2022-07-21T01:43:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39ef72700 1 Processor -- start 2022-07-21T01:43:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39ef72700 1 -- start start 2022-07-21T01:43:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39ef72700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3980ad0b0 0x7fd398143680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39ef72700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3980b29e0 0x7fd398153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39ef72700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd39813c870 0x7fd398067710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39ef72700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980af100 con 0x7fd3980ad0b0 2022-07-21T01:43:01.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39ef72700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd398153d30 con 0x7fd3980b29e0 2022-07-21T01:43:01.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39ef72700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980af330 con 0x7fd39813c870 2022-07-21T01:43:01.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39df70700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3980ad0b0 0x7fd398143680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39df70700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3980ad0b0 0x7fd398143680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52136/0 (socket says 172.21.15.3:52136) 2022-07-21T01:43:01.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39df70700 1 -- 172.21.15.3:0/1075766255 learned_addr learned my addr 172.21.15.3:0/1075766255 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:01.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39e771700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd39813c870 0x7fd398067710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39d76f700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3980b29e0 0x7fd398153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39df70700 1 -- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd39813c870 msgr2=0x7fd398067710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39df70700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd39813c870 0x7fd398067710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39df70700 1 -- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3980b29e0 msgr2=0x7fd398153840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39df70700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3980b29e0 0x7fd398153840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39df70700 1 -- 172.21.15.3:0/1075766255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd390009470 con 0x7fd3980ad0b0 2022-07-21T01:43:01.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39e771700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd39813c870 0x7fd398067710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:43:01.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39d76f700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3980b29e0 0x7fd398153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:43:01.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.595+0000 7fd39df70700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3980ad0b0 0x7fd398143680 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7fd388001860 tx=0x7fd388001ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:01.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd38effd700 1 -- 172.21.15.3:0/1075766255 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd388007170 con 0x7fd3980ad0b0 2022-07-21T01:43:01.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd38effd700 1 -- 172.21.15.3:0/1075766255 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3880072d0 con 0x7fd3980ad0b0 2022-07-21T01:43:01.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3980af5d0 con 0x7fd3980ad0b0 2022-07-21T01:43:01.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd38effd700 1 -- 172.21.15.3:0/1075766255 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd388004920 con 0x7fd3980ad0b0 2022-07-21T01:43:01.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd398067270 con 0x7fd3980ad0b0 2022-07-21T01:43:01.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd398067430 con 0x7fd3980ad0b0 2022-07-21T01:43:01.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd38effd700 1 -- 172.21.15.3:0/1075766255 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd38801b070 con 0x7fd3980ad0b0 2022-07-21T01:43:01.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd38effd700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd378053b00 0x7fd378055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd38effd700 1 -- 172.21.15.3:0/1075766255 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fd38806b9c0 con 0x7fd3980ad0b0 2022-07-21T01:43:01.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd39d76f700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd378053b00 0x7fd378055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.599+0000 7fd39d76f700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd378053b00 0x7fd378055fc0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fd39400b000 tx=0x7fd394009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:01.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.603+0000 7fd38effd700 1 -- 172.21.15.3:0/1075766255 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd388042680 con 0x7fd3980ad0b0 2022-07-21T01:43:01.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd3980574e0 con 0x7fd3980ad0b0 2022-07-21T01:43:01.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd38effd700 1 -- 172.21.15.3:0/1075766255 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v161) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd388045ca0 con 0x7fd3980ad0b0 2022-07-21T01:43:01.734 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:43:01.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd378053b00 msgr2=0x7fd378055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd378053b00 0x7fd378055fc0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fd39400b000 tx=0x7fd394009040).stop 2022-07-21T01:43:01.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3980ad0b0 msgr2=0x7fd398143680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3980ad0b0 0x7fd398143680 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7fd388001860 tx=0x7fd388001ad0).stop 2022-07-21T01:43:01.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 shutdown_connections 2022-07-21T01:43:01.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd39813c870 0x7fd398067710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fd378053b00 0x7fd378055fc0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3980b29e0 0x7fd398153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 --2- 172.21.15.3:0/1075766255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3980ad0b0 0x7fd398143680 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 >> 172.21.15.3:0/1075766255 conn(0x7fd3980613c0 msgr2=0x7fd39813bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 shutdown_connections 2022-07-21T01:43:01.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.731+0000 7fd39ef72700 1 -- 172.21.15.3:0/1075766255 wait complete. 2022-07-21T01:43:01.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:43:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.855+0000 7fdb67052700 1 Processor -- start 2022-07-21T01:43:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.855+0000 7fdb67052700 1 -- start start 2022-07-21T01:43:01.874 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb600a9da0 0x7fdb60060950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdb601402d0 0x7fdb60140760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb60067030 con 0x7fdb600b31d0 2022-07-21T01:43:01.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb60064a70 con 0x7fdb601402d0 2022-07-21T01:43:01.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb60061f60 con 0x7fdb600a9da0 2022-07-21T01:43:01.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb66851700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdb601402d0 0x7fdb60140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb66851700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fdb601402d0 0x7fdb60140760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52148/0 (socket says 172.21.15.3:52148) 2022-07-21T01:43:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb66851700 1 -- 172.21.15.3:0/1861684860 learned_addr learned my addr 172.21.15.3:0/1861684860 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb66050700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdb600b31d0 0x7fdb60066020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60424/0 (socket says 172.21.15.3:60424) 2022-07-21T01:43:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb6584f700 1 --2- 172.21.15.3:0/1861684860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb600a9da0 0x7fdb60060950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb66851700 1 -- 172.21.15.3:0/1861684860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb600a9da0 msgr2=0x7fdb60060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:01.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb66851700 1 --2- 172.21.15.3:0/1861684860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb600a9da0 0x7fdb60060950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb66851700 1 -- 172.21.15.3:0/1861684860 >> v1:172.21.15.3:6789/0 conn(0x7fdb600b31d0 legacy=0x7fdb60066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb66851700 1 -- 172.21.15.3:0/1861684860 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb600b35b0 con 0x7fdb601402d0 2022-07-21T01:43:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb66851700 1 --2- 172.21.15.3:0/1861684860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb601402d0 0x7fdb60140760 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7fdb580032f0 tx=0x7fdb58009290).ready entity=mon.0 client_cookie=42e5ae5f5ea04fad server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb6504e700 1 -- 172.21.15.3:0/1861684860 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5800b870 con 0x7fdb601402d0 2022-07-21T01:43:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb6504e700 1 -- 172.21.15.3:0/1861684860 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5800b9d0 con 0x7fdb601402d0 2022-07-21T01:43:01.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb6504e700 1 -- 172.21.15.3:0/1861684860 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5800bb80 con 0x7fdb601402d0 2022-07-21T01:43:01.883 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 -- 172.21.15.3:0/1861684860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb601402d0 msgr2=0x7fdb60140760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 --2- 172.21.15.3:0/1861684860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb601402d0 0x7fdb60140760 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7fdb580032f0 tx=0x7fdb58009290).stop 2022-07-21T01:43:01.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 -- 172.21.15.3:0/1861684860 shutdown_connections 2022-07-21T01:43:01.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 --2- 172.21.15.3:0/1861684860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb601402d0 0x7fdb60140760 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 --2- 172.21.15.3:0/1861684860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb600a9da0 0x7fdb60060950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 -- 172.21.15.3:0/1861684860 >> 172.21.15.3:0/1861684860 conn(0x7fdb60063870 msgr2=0x7fdb60063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:01.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 -- 172.21.15.3:0/1861684860 shutdown_connections 2022-07-21T01:43:01.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.859+0000 7fdb67052700 1 -- 172.21.15.3:0/1861684860 wait complete. 2022-07-21T01:43:01.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 Processor -- start 2022-07-21T01:43:01.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 -- start start 2022-07-21T01:43:01.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb600a9da0 0x7fdb6014a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb600b31d0 0x7fdb600ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb601402d0 0x7fdb60146760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600a8f60 con 0x7fdb600a9da0 2022-07-21T01:43:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600b04d0 con 0x7fdb600b31d0 2022-07-21T01:43:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600b0230 con 0x7fdb601402d0 2022-07-21T01:43:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66050700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb600a9da0 0x7fdb6014a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66050700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb600a9da0 0x7fdb6014a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52150/0 (socket says 172.21.15.3:52150) 2022-07-21T01:43:01.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66050700 1 -- 172.21.15.3:0/1525379121 learned_addr learned my addr 172.21.15.3:0/1525379121 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:01.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66851700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb601402d0 0x7fdb60146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb6584f700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb600b31d0 0x7fdb600ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66050700 1 -- 172.21.15.3:0/1525379121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb601402d0 msgr2=0x7fdb60146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66050700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb601402d0 0x7fdb60146760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66050700 1 -- 172.21.15.3:0/1525379121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb600b31d0 msgr2=0x7fdb600ae4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:01.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66050700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb600b31d0 0x7fdb600ae4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:01.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66050700 1 -- 172.21.15.3:0/1525379121 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb58009530 con 0x7fdb600a9da0 2022-07-21T01:43:01.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb6584f700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb600b31d0 0x7fdb600ae4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:43:01.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb66050700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb600a9da0 0x7fdb6014a300 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7fdb60061c40 tx=0x7fdb5c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:01.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb537fe700 1 -- 172.21.15.3:0/1525379121 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5c0060d0 con 0x7fdb600a9da0 2022-07-21T01:43:01.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb537fe700 1 -- 172.21.15.3:0/1525379121 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5c006230 con 0x7fdb600a9da0 2022-07-21T01:43:01.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb600af190 con 0x7fdb600a9da0 2022-07-21T01:43:01.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb537fe700 1 -- 172.21.15.3:0/1525379121 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5c0046f0 con 0x7fdb600a9da0 2022-07-21T01:43:01.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.863+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6014a7f0 con 0x7fdb600a9da0 2022-07-21T01:43:01.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.867+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb60144e80 con 0x7fdb600a9da0 2022-07-21T01:43:01.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.867+0000 7fdb537fe700 1 -- 172.21.15.3:0/1525379121 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdb5c01a070 con 0x7fdb600a9da0 2022-07-21T01:43:01.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.867+0000 7fdb537fe700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb40053a10 0x7fdb40055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:01.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.867+0000 7fdb537fe700 1 -- 172.21.15.3:0/1525379121 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fdb5c06b390 con 0x7fdb600a9da0 2022-07-21T01:43:01.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.867+0000 7fdb6584f700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb40053a10 0x7fdb40055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:01.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.867+0000 7fdb6584f700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb40053a10 0x7fdb40055ed0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fdb5400c000 tx=0x7fdb54008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:01.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:01.875+0000 7fdb537fe700 1 -- 172.21.15.3:0/1525379121 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb5c042050 con 0x7fdb600a9da0 2022-07-21T01:43:02.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.003+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fdb600a90a0 con 0x7fdb600a9da0 2022-07-21T01:43:02.005 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.003+0000 7fdb537fe700 1 -- 172.21.15.3:0/1525379121 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v161) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdb600a90a0 con 0x7fdb600a9da0 2022-07-21T01:43:02.006 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:43:02.009 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb40053a10 msgr2=0x7fdb40055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb40053a10 0x7fdb40055ed0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fdb5400c000 tx=0x7fdb54008040).stop 2022-07-21T01:43:02.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb600a9da0 msgr2=0x7fdb6014a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb600a9da0 0x7fdb6014a300 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7fdb60061c40 tx=0x7fdb5c007c50).stop 2022-07-21T01:43:02.010 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 shutdown_connections 2022-07-21T01:43:02.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb601402d0 0x7fdb60146760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdb40053a10 0x7fdb40055ed0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb600b31d0 0x7fdb600ae4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 --2- 172.21.15.3:0/1525379121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb600a9da0 0x7fdb6014a300 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.011 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 >> 172.21.15.3:0/1525379121 conn(0x7fdb60063870 msgr2=0x7fdb60063c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:02.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 shutdown_connections 2022-07-21T01:43:02.012 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.007+0000 7fdb67052700 1 -- 172.21.15.3:0/1525379121 wait complete. 2022-07-21T01:43:02.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:43:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e8b17c700 1 Processor -- start 2022-07-21T01:43:02.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e8b17c700 1 -- start start 2022-07-21T01:43:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e8b17c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e8414ad10 0x7f3e8405f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e8b17c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3e840b00a0 0x7f3e8414b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e8b17c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e840614f0 con 0x7f3e840acb40 2022-07-21T01:43:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e8b17c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8414b770 con 0x7f3e840b00a0 2022-07-21T01:43:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e8b17c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8414ba10 con 0x7f3e8414ad10 2022-07-21T01:43:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e88f18700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e8414ad10 0x7f3e8405f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e88f18700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e8414ad10 0x7f3e8405f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:59074/0 (socket says 172.21.15.3:59074) 2022-07-21T01:43:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e88f18700 1 -- 172.21.15.3:0/2073150156 learned_addr learned my addr 172.21.15.3:0/2073150156 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e83fff700 1 --2- 172.21.15.3:0/2073150156 >> v2:172.21.15.143:3300/0 conn(0x7f3e840b00a0 0x7f3e8414b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e88f18700 1 -- 172.21.15.3:0/2073150156 >> v2:172.21.15.143:3300/0 conn(0x7f3e840b00a0 msgr2=0x7f3e8414b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e88f18700 1 --2- 172.21.15.3:0/2073150156 >> v2:172.21.15.143:3300/0 conn(0x7f3e840b00a0 0x7f3e8414b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e88f18700 1 -- 172.21.15.3:0/2073150156 >> v1:172.21.15.143:6789/0 conn(0x7f3e840acb40 legacy=0x7f3e840af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e88f18700 1 -- 172.21.15.3:0/2073150156 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e840aeeb0 con 0x7f3e8414ad10 2022-07-21T01:43:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e88f18700 1 --2- 172.21.15.3:0/2073150156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e8414ad10 0x7f3e8405f840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3e740032f0 tx=0x7f3e74009290).ready entity=mon.2 client_cookie=13c5b3ae621891e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e837fe700 1 -- 172.21.15.3:0/2073150156 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e7400b8a0 con 0x7f3e8414ad10 2022-07-21T01:43:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e837fe700 1 -- 172.21.15.3:0/2073150156 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7400ba00 con 0x7f3e8414ad10 2022-07-21T01:43:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e837fe700 1 -- 172.21.15.3:0/2073150156 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e7400bb70 con 0x7f3e8414ad10 2022-07-21T01:43:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e8b17c700 1 -- 172.21.15.3:0/2073150156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e8414ad10 msgr2=0x7f3e8405f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.127+0000 7f3e8b17c700 1 --2- 172.21.15.3:0/2073150156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e8414ad10 0x7f3e8405f840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3e740032f0 tx=0x7f3e74009290).stop 2022-07-21T01:43:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 -- 172.21.15.3:0/2073150156 shutdown_connections 2022-07-21T01:43:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 --2- 172.21.15.3:0/2073150156 >> v2:172.21.15.143:3300/0 conn(0x7f3e840b00a0 0x7f3e8414b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 --2- 172.21.15.3:0/2073150156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e8414ad10 0x7f3e8405f840 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 -- 172.21.15.3:0/2073150156 >> 172.21.15.3:0/2073150156 conn(0x7f3e840b5700 msgr2=0x7f3e840b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:02.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 -- 172.21.15.3:0/2073150156 shutdown_connections 2022-07-21T01:43:02.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 -- 172.21.15.3:0/2073150156 wait complete. 2022-07-21T01:43:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 Processor -- start 2022-07-21T01:43:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 -- start start 2022-07-21T01:43:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e840acb40 0x7f3e84156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e840b00a0 0x7f3e84148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e8414ad10 0x7f3e8415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84147170 con 0x7f3e8414ad10 2022-07-21T01:43:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8415a290 con 0x7f3e840b00a0 2022-07-21T01:43:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e8b17c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8415a530 con 0x7f3e840acb40 2022-07-21T01:43:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e88f18700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e840acb40 0x7f3e84156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e89719700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e8414ad10 0x7f3e8415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e83fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e840b00a0 0x7f3e84148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e88f18700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e840acb40 0x7f3e84156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:59080/0 (socket says 172.21.15.3:59080) 2022-07-21T01:43:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e88f18700 1 -- 172.21.15.3:0/679136188 learned_addr learned my addr 172.21.15.3:0/679136188 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e89719700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e8414ad10 0x7f3e8415a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52168/0 (socket says 172.21.15.3:52168) 2022-07-21T01:43:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e83fff700 1 -- 172.21.15.3:0/679136188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e840acb40 msgr2=0x7f3e84156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e83fff700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e840acb40 0x7f3e84156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e83fff700 1 -- 172.21.15.3:0/679136188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e8414ad10 msgr2=0x7f3e8415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e83fff700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e8414ad10 0x7f3e8415a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.131+0000 7f3e83fff700 1 -- 172.21.15.3:0/679136188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74009530 con 0x7f3e840b00a0 2022-07-21T01:43:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.135+0000 7f3e83fff700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e840b00a0 0x7f3e84148ce0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f3e6c00c000 tx=0x7f3e6c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.135+0000 7f3e81ffb700 1 -- 172.21.15.3:0/679136188 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e6c011750 con 0x7f3e840b00a0 2022-07-21T01:43:02.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.135+0000 7f3e81ffb700 1 -- 172.21.15.3:0/679136188 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6c0118b0 con 0x7f3e840b00a0 2022-07-21T01:43:02.147 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.135+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e84157030 con 0x7f3e840b00a0 2022-07-21T01:43:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.135+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8415aec0 con 0x7f3e840b00a0 2022-07-21T01:43:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.135+0000 7f3e81ffb700 1 -- 172.21.15.3:0/679136188 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e6c011c10 con 0x7f3e840b00a0 2022-07-21T01:43:02.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.135+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e84147330 con 0x7f3e840b00a0 2022-07-21T01:43:02.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.135+0000 7f3e81ffb700 1 -- 172.21.15.3:0/679136188 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3e6c011d70 con 0x7f3e840b00a0 2022-07-21T01:43:02.149 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.139+0000 7f3e81ffb700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e68053a70 0x7f3e68055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.139+0000 7f3e81ffb700 1 -- 172.21.15.3:0/679136188 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f3e6c06bff0 con 0x7f3e840b00a0 2022-07-21T01:43:02.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.139+0000 7f3e88f18700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e68053a70 0x7f3e68055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.150 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.139+0000 7f3e88f18700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e68053a70 0x7f3e68055f30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f3e7401b000 tx=0x7f3e74016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:02.151 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.147+0000 7f3e81ffb700 1 -- 172.21.15.3:0/679136188 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3e6c042d50 con 0x7f3e840b00a0 2022-07-21T01:43:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.275+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e840afa60 con 0x7f3e840b00a0 2022-07-21T01:43:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.275+0000 7f3e81ffb700 1 -- 172.21.15.3:0/679136188 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v161) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3e6c046370 con 0x7f3e840b00a0 2022-07-21T01:43:02.279 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:43:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e68053a70 msgr2=0x7f3e68055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e68053a70 0x7f3e68055f30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f3e7401b000 tx=0x7f3e74016000).stop 2022-07-21T01:43:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e840b00a0 msgr2=0x7f3e84148ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e840b00a0 0x7f3e84148ce0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f3e6c00c000 tx=0x7f3e6c006df0).stop 2022-07-21T01:43:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 shutdown_connections 2022-07-21T01:43:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e8414ad10 0x7f3e8415a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3e68053a70 0x7f3e68055f30 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e840b00a0 0x7f3e84148ce0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 --2- 172.21.15.3:0/679136188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e840acb40 0x7f3e84156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 >> 172.21.15.3:0/679136188 conn(0x7f3e840b5700 msgr2=0x7f3e8405be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 shutdown_connections 2022-07-21T01:43:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.279+0000 7f3e8b17c700 1 -- 172.21.15.3:0/679136188 wait complete. 2022-07-21T01:43:02.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:43:02.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.399+0000 7f048c9fe700 1 Processor -- start 2022-07-21T01:43:02.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.399+0000 7f048c9fe700 1 -- start start 2022-07-21T01:43:02.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0480040050 0x7f0480040db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048003f950 0x7f048003c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048003f250 con 0x7f0480042560 2022-07-21T01:43:02.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f048003e690 con 0x7f0480040050 2022-07-21T01:43:02.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0480041b40 con 0x7f048003f950 2022-07-21T01:43:02.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04877fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f0480040050 0x7f0480040db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f0486ffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0480042560 0x7f04800d3f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60452/0 (socket says 172.21.15.3:60452) 2022-07-21T01:43:02.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f0486ffd700 1 -- 172.21.15.3:0/451185311 learned_addr learned my addr 172.21.15.3:0/451185311 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:02.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f0487fff700 1 --2- 172.21.15.3:0/451185311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048003f950 0x7f048003c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04867fc700 1 -- 172.21.15.3:0/451185311 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698619669 0 0) 0x7f048003f250 con 0x7f0480042560 2022-07-21T01:43:02.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04867fc700 1 -- 172.21.15.3:0/451185311 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046c003260 con 0x7f0480042560 2022-07-21T01:43:02.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04877fe700 1 -- 172.21.15.3:0/451185311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048003f950 msgr2=0x7f048003c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:02.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04877fe700 1 --2- 172.21.15.3:0/451185311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048003f950 0x7f048003c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04877fe700 1 -- 172.21.15.3:0/451185311 >> v1:172.21.15.3:6789/0 conn(0x7f0480042560 legacy=0x7f04800d3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04877fe700 1 -- 172.21.15.3:0/451185311 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0480041de0 con 0x7f0480040050 2022-07-21T01:43:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04877fe700 1 --2- 172.21.15.3:0/451185311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 0x7f0480040db0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f047c002f70 tx=0x7f047c006df0).ready entity=mon.0 client_cookie=c4067e227c6f7709 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04867fc700 1 -- 172.21.15.3:0/451185311 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f047c00d4a0 con 0x7f0480040050 2022-07-21T01:43:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04867fc700 1 -- 172.21.15.3:0/451185311 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c000e90 con 0x7f0480040050 2022-07-21T01:43:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f04867fc700 1 -- 172.21.15.3:0/451185311 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f047c00d7f0 con 0x7f0480040050 2022-07-21T01:43:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 -- 172.21.15.3:0/451185311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 msgr2=0x7f0480040db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 --2- 172.21.15.3:0/451185311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 0x7f0480040db0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f047c002f70 tx=0x7f047c006df0).stop 2022-07-21T01:43:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 -- 172.21.15.3:0/451185311 shutdown_connections 2022-07-21T01:43:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 --2- 172.21.15.3:0/451185311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048003f950 0x7f048003c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 --2- 172.21.15.3:0/451185311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 0x7f0480040db0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 -- 172.21.15.3:0/451185311 >> 172.21.15.3:0/451185311 conn(0x7f04800462c0 msgr2=0x7f04800466d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 -- 172.21.15.3:0/451185311 shutdown_connections 2022-07-21T01:43:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.403+0000 7f048c9fe700 1 -- 172.21.15.3:0/451185311 wait complete. 2022-07-21T01:43:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f048c9fe700 1 Processor -- start 2022-07-21T01:43:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f048c9fe700 1 -- start start 2022-07-21T01:43:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f048c9fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048003f950 0x7f0480043ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f048c9fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 0x7f04800d1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f048c9fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0480042560 0x7f04800e6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f048c9fe700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04800a5730 con 0x7f0480040050 2022-07-21T01:43:02.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f048c9fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04800a59d0 con 0x7f048003f950 2022-07-21T01:43:02.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f048c9fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04800a5c70 con 0x7f0480042560 2022-07-21T01:43:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0486ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 0x7f04800d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0486ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 0x7f04800d1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52178/0 (socket says 172.21.15.3:52178) 2022-07-21T01:43:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0486ffd700 1 -- 172.21.15.3:0/1571459918 learned_addr learned my addr 172.21.15.3:0/1571459918 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0487fff700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0480042560 0x7f04800e6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f04877fe700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048003f950 0x7f0480043ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0486ffd700 1 -- 172.21.15.3:0/1571459918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0480042560 msgr2=0x7f04800e6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0486ffd700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0480042560 0x7f04800e6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0486ffd700 1 -- 172.21.15.3:0/1571459918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048003f950 msgr2=0x7f0480043ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0486ffd700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048003f950 0x7f0480043ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0486ffd700 1 -- 172.21.15.3:0/1571459918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c00b180 con 0x7f0480040050 2022-07-21T01:43:02.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0486ffd700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 0x7f04800d1240 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7f0474001110 tx=0x7f0474007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:02.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.407+0000 7f0484ff9700 1 -- 172.21.15.3:0/1571459918 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04740070d0 con 0x7f0480040050 2022-07-21T01:43:02.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f0484ff9700 1 -- 172.21.15.3:0/1571459918 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0474007230 con 0x7f0480040050 2022-07-21T01:43:02.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04800e6a20 con 0x7f0480040050 2022-07-21T01:43:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f0484ff9700 1 -- 172.21.15.3:0/1571459918 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0474005610 con 0x7f0480040050 2022-07-21T01:43:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048003b5a0 con 0x7f0480040050 2022-07-21T01:43:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04800e6bb0 con 0x7f0480040050 2022-07-21T01:43:02.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f0484ff9700 1 -- 172.21.15.3:0/1571459918 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f047401a070 con 0x7f0480040050 2022-07-21T01:43:02.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f0484ff9700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053b30 0x7f0468055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f0484ff9700 1 -- 172.21.15.3:0/1571459918 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f047406afb0 con 0x7f0480040050 2022-07-21T01:43:02.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f04877fe700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053b30 0x7f0468055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.411+0000 7f04877fe700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053b30 0x7f0468055ff0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f047c01a000 tx=0x7f047c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:02.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.415+0000 7f0484ff9700 1 -- 172.21.15.3:0/1571459918 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0474041d70 con 0x7f0480040050 2022-07-21T01:43:02.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.539+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f048003f4b0 con 0x7f0480040050 2022-07-21T01:43:02.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.539+0000 7f0484ff9700 1 -- 172.21.15.3:0/1571459918 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v161) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0474045390 con 0x7f0480040050 2022-07-21T01:43:02.544 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:43:02.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053b30 msgr2=0x7f0468055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053b30 0x7f0468055ff0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f047c01a000 tx=0x7f047c014000).stop 2022-07-21T01:43:02.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 msgr2=0x7f04800d1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 0x7f04800d1240 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7f0474001110 tx=0x7f0474007a60).stop 2022-07-21T01:43:02.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 shutdown_connections 2022-07-21T01:43:02.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0480042560 0x7f04800e6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0480040050 0x7f04800d1240 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f0468053b30 0x7f0468055ff0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 --2- 172.21.15.3:0/1571459918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048003f950 0x7f0480043ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 >> 172.21.15.3:0/1571459918 conn(0x7f04800462c0 msgr2=0x7f04800d5840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:02.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 shutdown_connections 2022-07-21T01:43:02.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.543+0000 7f048c9fe700 1 -- 172.21.15.3:0/1571459918 wait complete. 2022-07-21T01:43:02.556 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:43:02.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:43:02.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:43:02.608 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:43:02.610 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:43:02.613 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:43:02.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.663+0000 7f66fe8fb700 1 Processor -- start 2022-07-21T01:43:02.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.663+0000 7f66fe8fb700 1 -- start start 2022-07-21T01:43:02.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.663+0000 7f66fe8fb700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f66f813a230 0x7f66f80acd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.663+0000 7f66fe8fb700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f80ad200 con 0x7f66f80b2350 2022-07-21T01:43:02.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.663+0000 7f66fe8fb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f8060ed0 con 0x7f66f8139e50 2022-07-21T01:43:02.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.663+0000 7f66fe8fb700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f80b28c0 con 0x7f66f813a230 2022-07-21T01:43:02.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.663+0000 7f66f77fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f66f80b2350 0x7f66f80b6750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60468/0 (socket says 172.21.15.3:60468) 2022-07-21T01:43:02.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.663+0000 7f66fce98700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f66f813a230 0x7f66f80acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.663+0000 7f66f77fe700 1 -- 172.21.15.3:0/2927654561 learned_addr learned my addr 172.21.15.3:0/2927654561 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:02.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fce98700 1 -- 172.21.15.3:0/2927654561 >> v1:172.21.15.143:6789/0 conn(0x7f66f8139e50 legacy=0x7f66f8143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fce98700 1 -- 172.21.15.3:0/2927654561 >> v1:172.21.15.3:6789/0 conn(0x7f66f80b2350 legacy=0x7f66f80b6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66f6ffd700 1 -- 172.21.15.3:0/2927654561 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017640043 0 0) 0x7f66ec001110 con 0x7f66f80b2350 2022-07-21T01:43:02.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fce98700 1 -- 172.21.15.3:0/2927654561 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f80b6e60 con 0x7f66f813a230 2022-07-21T01:43:02.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fce98700 1 --2- 172.21.15.3:0/2927654561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 0x7f66f80acd10 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f66f0002f70 tx=0x7f66f0006df0).ready entity=mon.0 client_cookie=d5a0061d236606ff server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:02.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66f6ffd700 1 -- 172.21.15.3:0/2927654561 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66f000d610 con 0x7f66f813a230 2022-07-21T01:43:02.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66f6ffd700 1 -- 172.21.15.3:0/2927654561 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f0000e90 con 0x7f66f813a230 2022-07-21T01:43:02.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66f6ffd700 1 -- 172.21.15.3:0/2927654561 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66f000d960 con 0x7f66f813a230 2022-07-21T01:43:02.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fe8fb700 1 -- 172.21.15.3:0/2927654561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 msgr2=0x7f66f80acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fe8fb700 1 --2- 172.21.15.3:0/2927654561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 0x7f66f80acd10 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f66f0002f70 tx=0x7f66f0006df0).stop 2022-07-21T01:43:02.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fe8fb700 1 -- 172.21.15.3:0/2927654561 shutdown_connections 2022-07-21T01:43:02.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fe8fb700 1 --2- 172.21.15.3:0/2927654561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 0x7f66f80acd10 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fe8fb700 1 -- 172.21.15.3:0/2927654561 >> 172.21.15.3:0/2927654561 conn(0x7f66f80b3610 msgr2=0x7f66f80b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:02.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fe8fb700 1 -- 172.21.15.3:0/2927654561 shutdown_connections 2022-07-21T01:43:02.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fe8fb700 1 -- 172.21.15.3:0/2927654561 wait complete. 2022-07-21T01:43:02.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fe8fb700 1 Processor -- start 2022-07-21T01:43:02.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.667+0000 7f66fe8fb700 1 -- start start 2022-07-21T01:43:02.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fe8fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f80b2350 0x7f66f8143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fe8fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66f8139e50 0x7f66f8150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fe8fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 0x7f66f8150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fe8fb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f8154a80 con 0x7f66f813a230 2022-07-21T01:43:02.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fe8fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f813b1b0 con 0x7f66f80b2350 2022-07-21T01:43:02.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fe8fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f813a720 con 0x7f66f8139e50 2022-07-21T01:43:02.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fce98700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 0x7f66f8150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fce98700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 0x7f66f8150f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52196/0 (socket says 172.21.15.3:52196) 2022-07-21T01:43:02.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fce98700 1 -- 172.21.15.3:0/1659789458 learned_addr learned my addr 172.21.15.3:0/1659789458 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:02.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66f7fff700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f80b2350 0x7f66f8143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66f77fe700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66f8139e50 0x7f66f8150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fce98700 1 -- 172.21.15.3:0/1659789458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66f8139e50 msgr2=0x7f66f8150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fce98700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66f8139e50 0x7f66f8150aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fce98700 1 -- 172.21.15.3:0/1659789458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f80b2350 msgr2=0x7f66f8143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fce98700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f80b2350 0x7f66f8143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fce98700 1 -- 172.21.15.3:0/1659789458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f000b180 con 0x7f66f813a230 2022-07-21T01:43:02.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66f77fe700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66f8139e50 0x7f66f8150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:43:02.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fce98700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 0x7f66f8150f90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f66f0002f70 tx=0x7f66f0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:02.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66f57fa700 1 -- 172.21.15.3:0/1659789458 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66f000d990 con 0x7f66f813a230 2022-07-21T01:43:02.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66f57fa700 1 -- 172.21.15.3:0/1659789458 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f0011c30 con 0x7f66f813a230 2022-07-21T01:43:02.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f813a9c0 con 0x7f66f813a230 2022-07-21T01:43:02.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66f57fa700 1 -- 172.21.15.3:0/1659789458 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66f001e460 con 0x7f66f813a230 2022-07-21T01:43:02.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f8151480 con 0x7f66f813a230 2022-07-21T01:43:02.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.671+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f8150800 con 0x7f66f813a230 2022-07-21T01:43:02.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.675+0000 7f66f57fa700 1 -- 172.21.15.3:0/1659789458 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f66f0025030 con 0x7f66f813a230 2022-07-21T01:43:02.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.675+0000 7f66f57fa700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66d8053950 0x7f66d8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:02.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.675+0000 7f66f57fa700 1 -- 172.21.15.3:0/1659789458 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f66f006d3e0 con 0x7f66f813a230 2022-07-21T01:43:02.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.675+0000 7f66f7fff700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66d8053950 0x7f66d8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:02.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.675+0000 7f66f7fff700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66d8053950 0x7f66d8055e10 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f66e800d000 tx=0x7f66e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:02.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.679+0000 7f66f57fa700 1 -- 172.21.15.3:0/1659789458 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f66f8150800 con 0x7f66f813a230 2022-07-21T01:43:02.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.843+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66f8150800 con 0x7f66f813a230 2022-07-21T01:43:02.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.843+0000 7f66f57fa700 1 -- 172.21.15.3:0/1659789458 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2489 (secure 0 0 0) 0x7f66f8150800 con 0x7f66f813a230 2022-07-21T01:43:02.847 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:43:02.848 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":81,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/781340149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":781340149},{"type":"v1","addr":"172.21.15.143:6833","nonce":781340149}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79},{"gid":8712,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3829565046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3829565046},{"type":"v1","addr":"172.21.15.3:6835","nonce":3829565046}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:58.238508+0000","modified":"2022-07-21T01:42:59.241585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8727},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8727":{"gid":8727,"name":"c","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1787668325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1787668325},{"type":"v1","addr":"172.21.15.3:6837","nonce":1787668325}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-21T01:43:02.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66d8053950 msgr2=0x7f66d8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66d8053950 0x7f66d8055e10 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f66e800d000 tx=0x7f66e8009040).stop 2022-07-21T01:43:02.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 msgr2=0x7f66f8150f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:02.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 0x7f66f8150f90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f66f0002f70 tx=0x7f66f0015000).stop 2022-07-21T01:43:02.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 shutdown_connections 2022-07-21T01:43:02.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66f813a230 0x7f66f8150f90 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66f8139e50 0x7f66f8150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f66d8053950 0x7f66d8055e10 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 --2- 172.21.15.3:0/1659789458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f80b2350 0x7f66f8143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:02.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 >> 172.21.15.3:0/1659789458 conn(0x7f66f80b3610 msgr2=0x7f66f80af460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:02.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 shutdown_connections 2022-07-21T01:43:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:02.847+0000 7f66fe8fb700 1 -- 172.21.15.3:0/1659789458 wait complete. 2022-07-21T01:43:02.852 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 81 2022-07-21T01:43:03.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:58.238508+0000', 'modified': '2022-07-21T01:42:59.241585+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8727}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8727': {'gid': 8727, 'name': 'c', 'rank': 0, 'incarnation': 80, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/1787668325', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 1787668325}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 1787668325}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:43:03.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:43:03.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:43:03.284 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8727,"want_state":"up:active","state":"up:active","rank_uptime":5.0335580770000004,"mdsmap_epoch":81,"osdmap_epoch":161,"osdmap_epoch_barrier":160,"uptime":9.6748171870000004} 2022-07-21T01:43:03.284 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8727, "mdsmap_epoch": 81, "osdmap_epoch": 161, "osdmap_epoch_barrier": 160, "rank_uptime": 5.033558077, "state": "up:active", "uptime": 9.674817187, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:43:03.285 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:43:03.285 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-21T01:43:03.286 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi171.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-21T01:43:03.286 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:03.287 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:43:03.295 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:43:03.353 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:43:03.354 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:43:03.365 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:43:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:43:03.368 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:03.369 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:43:03.376 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:43:03.377 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:43:03.422 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:43:03.439 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:43:03.440 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:43:03.454 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:43:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:43:03.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:03.457 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:43:03.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:2022-07-21T01:43:03.490+0000 7f81eadb9080 -1 init, newargv = 0x55a16e421a10 newargc=9ceph-fuse[23531 2022-07-21T01:43:03.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:]: starting ceph client 2022-07-21T01:43:03.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi171.stderr:ceph-fuse[23531]: starting fuse 2022-07-21T01:43:04.605 DEBUG:teuthology.orchestra.run.smithi171:> sudo modprobe fuse 2022-07-21T01:43:04.624 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:43:04.625 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:43:04.683 INFO:teuthology.orchestra.run.smithi171.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:43:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:43:04.686 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:04.687 DEBUG:teuthology.orchestra.run.smithi171:> ls /sys/fs/fuse/connections 2022-07-21T01:43:04.694 INFO:teuthology.orchestra.run.smithi171.stdout:52 2022-07-21T01:43:04.694 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:43:04.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:04.695 DEBUG:teuthology.orchestra.run.smithi171:> sudo python3 -c ' 2022-07-21T01:43:04.696 DEBUG:teuthology.orchestra.run.smithi171:> import glob 2022-07-21T01:43:04.696 DEBUG:teuthology.orchestra.run.smithi171:> import re 2022-07-21T01:43:04.697 DEBUG:teuthology.orchestra.run.smithi171:> import os 2022-07-21T01:43:04.697 DEBUG:teuthology.orchestra.run.smithi171:> import subprocess 2022-07-21T01:43:04.698 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:43:04.698 DEBUG:teuthology.orchestra.run.smithi171:> def find_socket(client_name): 2022-07-21T01:43:04.698 DEBUG:teuthology.orchestra.run.smithi171:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-21T01:43:04.699 DEBUG:teuthology.orchestra.run.smithi171:> files = glob.glob(asok_path) 2022-07-21T01:43:04.699 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:43:04.700 DEBUG:teuthology.orchestra.run.smithi171:> # Given a non-glob path, it better be there 2022-07-21T01:43:04.700 DEBUG:teuthology.orchestra.run.smithi171:> if "*" not in asok_path: 2022-07-21T01:43:04.701 DEBUG:teuthology.orchestra.run.smithi171:> assert(len(files) == 1) 2022-07-21T01:43:04.701 DEBUG:teuthology.orchestra.run.smithi171:> return files[0] 2022-07-21T01:43:04.701 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:43:04.702 DEBUG:teuthology.orchestra.run.smithi171:> for f in files: 2022-07-21T01:43:04.702 DEBUG:teuthology.orchestra.run.smithi171:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:43:04.703 DEBUG:teuthology.orchestra.run.smithi171:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:43:04.703 DEBUG:teuthology.orchestra.run.smithi171:> return f 2022-07-21T01:43:04.704 DEBUG:teuthology.orchestra.run.smithi171:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:43:04.704 DEBUG:teuthology.orchestra.run.smithi171:> 2022-07-21T01:43:04.704 DEBUG:teuthology.orchestra.run.smithi171:> print(find_socket("client.0")) 2022-07-21T01:43:04.705 DEBUG:teuthology.orchestra.run.smithi171:> ' 2022-07-21T01:43:04.749 INFO:teuthology.orchestra.run.smithi171.stdout:/var/run/ceph/ceph-client.0.23531.asok 2022-07-21T01:43:04.754 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23531.asok 2022-07-21T01:43:04.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:04.755 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23531.asok status 2022-07-21T01:43:04.892 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2022-07-21T01:43:04.893 INFO:teuthology.orchestra.run.smithi171.stdout: "metadata": { 2022-07-21T01:43:04.893 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:43:04.894 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:43:04.894 INFO:teuthology.orchestra.run.smithi171.stdout: "entity_id": "0", 2022-07-21T01:43:04.894 INFO:teuthology.orchestra.run.smithi171.stdout: "hostname": "smithi171", 2022-07-21T01:43:04.895 INFO:teuthology.orchestra.run.smithi171.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-21T01:43:04.895 INFO:teuthology.orchestra.run.smithi171.stdout: "pid": "23531", 2022-07-21T01:43:04.895 INFO:teuthology.orchestra.run.smithi171.stdout: "root": "/" 2022-07-21T01:43:04.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:43:04.896 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_count": 0, 2022-07-21T01:43:04.897 INFO:teuthology.orchestra.run.smithi171.stdout: "dentry_pinned_count": 0, 2022-07-21T01:43:04.897 INFO:teuthology.orchestra.run.smithi171.stdout: "id": 8767, 2022-07-21T01:43:04.897 INFO:teuthology.orchestra.run.smithi171.stdout: "inst": { 2022-07-21T01:43:04.898 INFO:teuthology.orchestra.run.smithi171.stdout: "name": { 2022-07-21T01:43:04.898 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "client", 2022-07-21T01:43:04.898 INFO:teuthology.orchestra.run.smithi171.stdout: "num": 8767 2022-07-21T01:43:04.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:43:04.899 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:43:04.899 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:43:04.899 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:43:04.899 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2974429840 2022-07-21T01:43:04.900 INFO:teuthology.orchestra.run.smithi171.stdout: } 2022-07-21T01:43:04.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:43:04.900 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": { 2022-07-21T01:43:04.901 INFO:teuthology.orchestra.run.smithi171.stdout: "type": "v1", 2022-07-21T01:43:04.901 INFO:teuthology.orchestra.run.smithi171.stdout: "addr": "172.21.15.171:0", 2022-07-21T01:43:04.902 INFO:teuthology.orchestra.run.smithi171.stdout: "nonce": 2974429840 2022-07-21T01:43:04.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2022-07-21T01:43:04.903 INFO:teuthology.orchestra.run.smithi171.stdout: "inst_str": "client.8767 172.21.15.171:0/2974429840", 2022-07-21T01:43:04.903 INFO:teuthology.orchestra.run.smithi171.stdout: "addr_str": "172.21.15.171:0/2974429840", 2022-07-21T01:43:04.904 INFO:teuthology.orchestra.run.smithi171.stdout: "inode_count": 1, 2022-07-21T01:43:04.904 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_epoch": 81, 2022-07-21T01:43:04.905 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch": 161, 2022-07-21T01:43:04.905 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:43:04.905 INFO:teuthology.orchestra.run.smithi171.stdout: "blacklisted": false, 2022-07-21T01:43:04.906 INFO:teuthology.orchestra.run.smithi171.stdout: "fs_name": "cephfs" 2022-07-21T01:43:04.906 INFO:teuthology.orchestra.run.smithi171.stdout:} 2022-07-21T01:43:04.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:04.909 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:43:04.910 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:43:04.918 INFO:teuthology.orchestra.run.smithi171.stdout:fuseblk 2022-07-21T01:43:04.919 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-21T01:43:04.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:04.920 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:43:04.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T01:43:05.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.047+0000 7fce022ec700 1 Processor -- start 2022-07-21T01:43:05.051 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.047+0000 7fce022ec700 1 -- start start 2022-07-21T01:43:05.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.047+0000 7fce022ec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcdfc0b65c0 0x7fcdfc061600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.047+0000 7fce022ec700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc0b2100 con 0x7fcdfc0b41f0 2022-07-21T01:43:05.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.047+0000 7fce022ec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc0b69a0 con 0x7fcdfc147d80 2022-07-21T01:43:05.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.047+0000 7fce022ec700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdfc0b1630 con 0x7fcdfc0b65c0 2022-07-21T01:43:05.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.047+0000 7fce00ae9700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fcdfc0b41f0 0x7fcdfc144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60480/0 (socket says 172.21.15.3:60480) 2022-07-21T01:43:05.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.047+0000 7fce00ae9700 1 -- 172.21.15.3:0/1288764519 learned_addr learned my addr 172.21.15.3:0/1288764519 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:05.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.047+0000 7fce012ea700 1 --2- 172.21.15.3:0/1288764519 >> v2:172.21.15.143:3300/0 conn(0x7fcdfc0b65c0 0x7fcdfc061600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234969244 0 0) 0x7fcdfc0b2100 con 0x7fcdfc0b41f0 2022-07-21T01:43:05.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde0003260 con 0x7fcdfc0b41f0 2022-07-21T01:43:05.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289545172 0 0) 0x7fcdfc0b69a0 con 0x7fcdfc147d80 2022-07-21T01:43:05.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdfc0b2100 con 0x7fcdfc147d80 2022-07-21T01:43:05.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2159167840 0 0) 0x7fcde0003260 con 0x7fcdfc0b41f0 2022-07-21T01:43:05.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 >> v2:172.21.15.143:3300/0 conn(0x7fcdfc0b65c0 msgr2=0x7fcdfc061600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:05.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 --2- 172.21.15.3:0/1288764519 >> v2:172.21.15.143:3300/0 conn(0x7fcdfc0b65c0 0x7fcdfc061600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 >> v1:172.21.15.143:6789/0 conn(0x7fcdfc147d80 legacy=0x7fcdfc151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc1482a0 con 0x7fcdfc0b41f0 2022-07-21T01:43:05.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fcdf0001660 con 0x7fcdfc0b41f0 2022-07-21T01:43:05.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 >> v1:172.21.15.3:6789/0 conn(0x7fcdfc0b41f0 legacy=0x7fcdfc144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcde00038e0 0x7fcde0003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcde0004500 0x7fcde0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcde000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0b1630 con 0x7fcde0004500 2022-07-21T01:43:05.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde000bcf0 con 0x7fcde0004e70 2022-07-21T01:43:05.060 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde000beb0 con 0x7fcde00038e0 2022-07-21T01:43:05.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce01aeb700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcde000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce00ae9700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcde0004500 0x7fcde0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce012ea700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcde00038e0 0x7fcde0003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.061 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce01aeb700 1 -- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcde00038e0 msgr2=0x7fcde0003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:05.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce01aeb700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcde00038e0 0x7fcde0003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce01aeb700 1 -- 172.21.15.3:0/1288764519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcde0004500 msgr2=0x7fcde0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:05.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce01aeb700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcde0004500 0x7fcde0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.062 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce01aeb700 1 -- 172.21.15.3:0/1288764519 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde0003260 con 0x7fcde0004e70 2022-07-21T01:43:05.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce00ae9700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcde0004500 0x7fcde0004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:43:05.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce01aeb700 1 -- 172.21.15.3:0/1288764519 reap_dead start 2022-07-21T01:43:05.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce01aeb700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcde000b750 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fcde00054d0 tx=0x7fcdf4007d80).ready entity=mon.1 client_cookie=b0050d17f72bd2ae server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.063 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf4006890 con 0x7fcde0004e70 2022-07-21T01:43:05.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf40069f0 con 0x7fcde0004e70 2022-07-21T01:43:05.064 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fcdeffff700 1 -- 172.21.15.3:0/1288764519 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf4006d50 con 0x7fcde0004e70 2022-07-21T01:43:05.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce022ec700 1 -- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 msgr2=0x7fcde000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce022ec700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcde000b750 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fcde00054d0 tx=0x7fcdf4007d80).stop 2022-07-21T01:43:05.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce022ec700 1 -- 172.21.15.3:0/1288764519 shutdown_connections 2022-07-21T01:43:05.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce022ec700 1 --2- 172.21.15.3:0/1288764519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcde000b750 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce022ec700 1 -- 172.21.15.3:0/1288764519 >> 172.21.15.3:0/1288764519 conn(0x7fcdfc05c620 msgr2=0x7fcdfc05ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:05.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce022ec700 1 -- 172.21.15.3:0/1288764519 shutdown_connections 2022-07-21T01:43:05.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.051+0000 7fce022ec700 1 -- 172.21.15.3:0/1288764519 wait complete. 2022-07-21T01:43:05.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.055+0000 7fce022ec700 1 Processor -- start 2022-07-21T01:43:05.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.055+0000 7fce022ec700 1 -- start start 2022-07-21T01:43:05.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce022ec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcdfc0ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce022ec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdfc0addb0 0x7fcdfc1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.071 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce022ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdfc0b9000 0x7fcdfc156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce022ec700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc14a8f0 con 0x7fcdfc0addb0 2022-07-21T01:43:05.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce022ec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc14adf0 con 0x7fcde0004e70 2022-07-21T01:43:05.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce022ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0ae250 con 0x7fcdfc0b9000 2022-07-21T01:43:05.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce00ae9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdfc0addb0 0x7fcdfc1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce00ae9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdfc0addb0 0x7fcdfc1566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52214/0 (socket says 172.21.15.3:52214) 2022-07-21T01:43:05.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce00ae9700 1 -- 172.21.15.3:0/2793583051 learned_addr learned my addr 172.21.15.3:0/2793583051 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:05.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce012ea700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcdfc0ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce01aeb700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdfc0b9000 0x7fcdfc156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce012ea700 1 -- 172.21.15.3:0/2793583051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdfc0b9000 msgr2=0x7fcdfc156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce012ea700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdfc0b9000 0x7fcdfc156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce012ea700 1 -- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdfc0addb0 msgr2=0x7fcdfc1566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce012ea700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdfc0addb0 0x7fcdfc1566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce012ea700 1 -- 172.21.15.3:0/2793583051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc14a1b0 con 0x7fcde0004e70 2022-07-21T01:43:05.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce00ae9700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdfc0addb0 0x7fcdfc1566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:43:05.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce012ea700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcdfc0ae960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fcdf800b000 tx=0x7fcdf8006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fcdee7fc700 1 -- 172.21.15.3:0/2793583051 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf8013680 con 0x7fcde0004e70 2022-07-21T01:43:05.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fcdee7fc700 1 -- 172.21.15.3:0/2793583051 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf80137e0 con 0x7fcde0004e70 2022-07-21T01:43:05.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdfc1570a0 con 0x7fcde0004e70 2022-07-21T01:43:05.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fcdee7fc700 1 -- 172.21.15.3:0/2793583051 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf8013b40 con 0x7fcde0004e70 2022-07-21T01:43:05.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdfc0661a0 con 0x7fcde0004e70 2022-07-21T01:43:05.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdfc0ae410 con 0x7fcde0004e70 2022-07-21T01:43:05.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.059+0000 7fcdee7fc700 1 -- 172.21.15.3:0/2793583051 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcdf8013ca0 con 0x7fcde0004e70 2022-07-21T01:43:05.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.063+0000 7fcdee7fc700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcde4053aa0 0x7fcde4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.063+0000 7fce00ae9700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcde4053aa0 0x7fcde4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.063+0000 7fcdee7fc700 1 -- 172.21.15.3:0/2793583051 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcdf806c810 con 0x7fcde0004e70 2022-07-21T01:43:05.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.063+0000 7fce00ae9700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcde4053aa0 0x7fcde4055f60 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fcdf0005000 tx=0x7fcdfc148430).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.071+0000 7fcdee7fc700 1 -- 172.21.15.3:0/2793583051 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcdf8043500 con 0x7fcde0004e70 2022-07-21T01:43:05.202 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.199+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdfc0b2330 con 0x7fcde0004e70 2022-07-21T01:43:05.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.199+0000 7fcdee7fc700 1 -- 172.21.15.3:0/2793583051 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16285 (secure 0 0 0) 0x7fcdf8018020 con 0x7fcde0004e70 2022-07-21T01:43:05.204 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:43:05.205 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":161,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:42:59.242307+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-21T01:42:56.372231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-21T01:42:57.252984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T01:43:05.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.203+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcde4053aa0 msgr2=0x7fcde4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.203+0000 7fce022ec700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcde4053aa0 0x7fcde4055f60 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fcdf0005000 tx=0x7fcdfc148430).stop 2022-07-21T01:43:05.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.203+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 msgr2=0x7fcdfc0ae960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.203+0000 7fce022ec700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcdfc0ae960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fcdf800b000 tx=0x7fcdf8006df0).stop 2022-07-21T01:43:05.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.203+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 shutdown_connections 2022-07-21T01:43:05.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.203+0000 7fce022ec700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdfc0b9000 0x7fcdfc156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.203+0000 7fce022ec700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdfc0addb0 0x7fcdfc1566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.203+0000 7fce022ec700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcde4053aa0 0x7fcde4055f60 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.203+0000 7fce022ec700 1 --2- 172.21.15.3:0/2793583051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde0004e70 0x7fcdfc0ae960 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.207+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 >> 172.21.15.3:0/2793583051 conn(0x7fcdfc05c620 msgr2=0x7fcdfc144a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:05.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.207+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 shutdown_connections 2022-07-21T01:43:05.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.207+0000 7fce022ec700 1 -- 172.21.15.3:0/2793583051 wait complete. 2022-07-21T01:43:05.223 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-21T01:33:09.606566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-21T01:33:18.012897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-21T01:42:56.372231+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-07-21T01:42:57.252984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-21T01:43:05.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-21T01:43:05.334 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.331+0000 7f22a1bce700 1 Processor -- start 2022-07-21T01:43:05.335 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.331+0000 7f22a1bce700 1 -- start start 2022-07-21T01:43:05.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.331+0000 7f22a1bce700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c0b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.331+0000 7f22a1bce700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c1440f0 con 0x7f229c0b66f0 2022-07-21T01:43:05.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.331+0000 7f22a1bce700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c060170 con 0x7f229c0b5b20 2022-07-21T01:43:05.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.331+0000 7f22a1bce700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c0ba840 con 0x7f229c05fd90 2022-07-21T01:43:05.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.331+0000 7f229affd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f229c0b66f0 0x7f229c0ba130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60500/0 (socket says 172.21.15.3:60500) 2022-07-21T01:43:05.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.331+0000 7f229affd700 1 -- 172.21.15.3:0/370932592 learned_addr learned my addr 172.21.15.3:0/370932592 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:05.338 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229b7fe700 1 --2- 172.21.15.3:0/370932592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c0b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229a7fc700 1 -- 172.21.15.3:0/370932592 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564156838 0 0) 0x7f229c1440f0 con 0x7f229c0b66f0 2022-07-21T01:43:05.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229a7fc700 1 -- 172.21.15.3:0/370932592 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f227c003260 con 0x7f229c0b66f0 2022-07-21T01:43:05.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229a7fc700 1 -- 172.21.15.3:0/370932592 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572143862 0 0) 0x7f229c060170 con 0x7f229c0b5b20 2022-07-21T01:43:05.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229a7fc700 1 -- 172.21.15.3:0/370932592 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f229c1440f0 con 0x7f229c0b5b20 2022-07-21T01:43:05.339 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229b7fe700 1 -- 172.21.15.3:0/370932592 >> v1:172.21.15.143:6789/0 conn(0x7f229c0b5b20 legacy=0x7f229c0b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229a7fc700 1 -- 172.21.15.3:0/370932592 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3584902080 0 0) 0x7f227c003260 con 0x7f229c0b66f0 2022-07-21T01:43:05.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229b7fe700 1 -- 172.21.15.3:0/370932592 >> v1:172.21.15.3:6789/0 conn(0x7f229c0b66f0 legacy=0x7f229c0ba130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229b7fe700 1 -- 172.21.15.3:0/370932592 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c143be0 con 0x7f229c05fd90 2022-07-21T01:43:05.340 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229b7fe700 1 --2- 172.21.15.3:0/370932592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c0b4340 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f228c001110 tx=0x7f2284006df0).ready entity=mon.2 client_cookie=f3e7dea6a7f95bf3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229a7fc700 1 -- 172.21.15.3:0/370932592 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f228c001660 con 0x7f229c05fd90 2022-07-21T01:43:05.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229a7fc700 1 -- 172.21.15.3:0/370932592 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2284000e90 con 0x7f229c05fd90 2022-07-21T01:43:05.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229a7fc700 1 -- 172.21.15.3:0/370932592 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2284012460 con 0x7f229c05fd90 2022-07-21T01:43:05.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 -- 172.21.15.3:0/370932592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 msgr2=0x7f229c0b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 --2- 172.21.15.3:0/370932592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c0b4340 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f228c001110 tx=0x7f2284006df0).stop 2022-07-21T01:43:05.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 -- 172.21.15.3:0/370932592 shutdown_connections 2022-07-21T01:43:05.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 --2- 172.21.15.3:0/370932592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c0b4340 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 -- 172.21.15.3:0/370932592 >> 172.21.15.3:0/370932592 conn(0x7f229c062530 msgr2=0x7f229c066f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:05.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 -- 172.21.15.3:0/370932592 shutdown_connections 2022-07-21T01:43:05.346 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 -- 172.21.15.3:0/370932592 wait complete. 2022-07-21T01:43:05.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 Processor -- start 2022-07-21T01:43:05.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 -- start start 2022-07-21T01:43:05.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c13d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.347 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f229c0b5b20 0x7f229c0b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f229c0b66f0 0x7f229c14f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c13e790 con 0x7f229c0b66f0 2022-07-21T01:43:05.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c152950 con 0x7f229c0b5b20 2022-07-21T01:43:05.348 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f22a1bce700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c0ad930 con 0x7f229c05fd90 2022-07-21T01:43:05.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.335+0000 7f229bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f229c0b66f0 0x7f229c14f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f229c0b66f0 0x7f229c14f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52230/0 (socket says 172.21.15.3:52230) 2022-07-21T01:43:05.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229bfff700 1 -- 172.21.15.3:0/1912014019 learned_addr learned my addr 172.21.15.3:0/1912014019 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:05.349 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229b7fe700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c13d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229affd700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f229c0b5b20 0x7f229c0b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229b7fe700 1 -- 172.21.15.3:0/1912014019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f229c0b5b20 msgr2=0x7f229c0b2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229b7fe700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f229c0b5b20 0x7f229c0b2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.350 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229b7fe700 1 -- 172.21.15.3:0/1912014019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f229c0b66f0 msgr2=0x7f229c14f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229b7fe700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f229c0b66f0 0x7f229c14f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229b7fe700 1 -- 172.21.15.3:0/1912014019 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228400b180 con 0x7f229c05fd90 2022-07-21T01:43:05.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229b7fe700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c13d8b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f229c142ed0 tx=0x7f2284014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.351 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f2298ff9700 1 -- 172.21.15.3:0/1912014019 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f228400d8a0 con 0x7f229c05fd90 2022-07-21T01:43:05.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f2298ff9700 1 -- 172.21.15.3:0/1912014019 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228400da00 con 0x7f229c05fd90 2022-07-21T01:43:05.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229c1526b0 con 0x7f229c05fd90 2022-07-21T01:43:05.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f2298ff9700 1 -- 172.21.15.3:0/1912014019 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f228400dd60 con 0x7f229c05fd90 2022-07-21T01:43:05.352 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229c153440 con 0x7f229c05fd90 2022-07-21T01:43:05.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229c143e50 con 0x7f229c05fd90 2022-07-21T01:43:05.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f2298ff9700 1 -- 172.21.15.3:0/1912014019 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f228401c070 con 0x7f229c05fd90 2022-07-21T01:43:05.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f2298ff9700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2280053b30 0x7f2280055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.353 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f2298ff9700 1 -- 172.21.15.3:0/1912014019 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f228406c740 con 0x7f229c05fd90 2022-07-21T01:43:05.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.339+0000 7f229affd700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2280053b30 0x7f2280055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.354 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.343+0000 7f229affd700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2280053b30 0x7f2280055ff0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f228c00d000 tx=0x7f228c000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.355 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.351+0000 7f2298ff9700 1 -- 172.21.15.3:0/1912014019 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2284069e90 con 0x7f229c05fd90 2022-07-21T01:43:05.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.475+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f229c0574e0 con 0x7f229c05fd90 2022-07-21T01:43:05.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.475+0000 7f2298ff9700 1 -- 172.21.15.3:0/1912014019 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v161) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2284023020 con 0x7f229c05fd90 2022-07-21T01:43:05.480 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2022-07-21T01:43:05.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2280053b30 msgr2=0x7f2280055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2280053b30 0x7f2280055ff0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f228c00d000 tx=0x7f228c000bc0).stop 2022-07-21T01:43:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 msgr2=0x7f229c13d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c13d8b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f229c142ed0 tx=0x7f2284014000).stop 2022-07-21T01:43:05.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 shutdown_connections 2022-07-21T01:43:05.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f229c0b66f0 0x7f229c14f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2280053b30 0x7f2280055ff0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f229c0b5b20 0x7f229c0b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 --2- 172.21.15.3:0/1912014019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f229c05fd90 0x7f229c13d8b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 >> 172.21.15.3:0/1912014019 conn(0x7f229c062530 msgr2=0x7f229c12c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:05.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 shutdown_connections 2022-07-21T01:43:05.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.479+0000 7f22a1bce700 1 -- 172.21.15.3:0/1912014019 wait complete. 2022-07-21T01:43:05.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-21T01:43:05.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.599+0000 7f10fb59e700 1 Processor -- start 2022-07-21T01:43:05.602 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.599+0000 7f10fb59e700 1 -- start start 2022-07-21T01:43:05.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.599+0000 7f10fb59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f10fc0614b0 0x7f10fc061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.599+0000 7f10fb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc1405d0 0x7f10fc140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.599+0000 7f10fb59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10fc0aee80 con 0x7f10fc0ad750 2022-07-21T01:43:05.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.599+0000 7f10fb59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10fc0b70b0 con 0x7f10fc0614b0 2022-07-21T01:43:05.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.599+0000 7f10fb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc0af0b0 con 0x7f10fc1405d0 2022-07-21T01:43:05.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.599+0000 7f10f9d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f10fc0ad750 0x7f10fc0b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60514/0 (socket says 172.21.15.3:60514) 2022-07-21T01:43:05.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.599+0000 7f10f9d9b700 1 -- 172.21.15.3:0/226448788 learned_addr learned my addr 172.21.15.3:0/226448788 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:05.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fad9d700 1 --2- 172.21.15.3:0/226448788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc1405d0 0x7f10fc140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fa59c700 1 --2- 172.21.15.3:0/226448788 >> v2:172.21.15.143:3300/0 conn(0x7f10fc0614b0 0x7f10fc061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10f959a700 1 -- 172.21.15.3:0/226448788 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862694841 0 0) 0x7f10fc0aee80 con 0x7f10fc0ad750 2022-07-21T01:43:05.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10f959a700 1 -- 172.21.15.3:0/226448788 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10dc003260 con 0x7f10fc0ad750 2022-07-21T01:43:05.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fad9d700 1 -- 172.21.15.3:0/226448788 >> v2:172.21.15.143:3300/0 conn(0x7f10fc0614b0 msgr2=0x7f10fc061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:05.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fad9d700 1 --2- 172.21.15.3:0/226448788 >> v2:172.21.15.143:3300/0 conn(0x7f10fc0614b0 0x7f10fc061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fad9d700 1 -- 172.21.15.3:0/226448788 >> v1:172.21.15.3:6789/0 conn(0x7f10fc0ad750 legacy=0x7f10fc0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fad9d700 1 -- 172.21.15.3:0/226448788 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc0af350 con 0x7f10fc1405d0 2022-07-21T01:43:05.610 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fad9d700 1 --2- 172.21.15.3:0/226448788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc1405d0 0x7f10fc140a60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f10fc0aed30 tx=0x7f10f40091d0).ready entity=mon.2 client_cookie=fa2ba87927d8b25f server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10f959a700 1 -- 172.21.15.3:0/226448788 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10f400b600 con 0x7f10fc1405d0 2022-07-21T01:43:05.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10f959a700 1 -- 172.21.15.3:0/226448788 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f400b760 con 0x7f10fc1405d0 2022-07-21T01:43:05.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10f959a700 1 -- 172.21.15.3:0/226448788 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10f400b8d0 con 0x7f10fc1405d0 2022-07-21T01:43:05.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fb59e700 1 -- 172.21.15.3:0/226448788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc1405d0 msgr2=0x7f10fc140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fb59e700 1 --2- 172.21.15.3:0/226448788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc1405d0 0x7f10fc140a60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f10fc0aed30 tx=0x7f10f40091d0).stop 2022-07-21T01:43:05.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fb59e700 1 -- 172.21.15.3:0/226448788 shutdown_connections 2022-07-21T01:43:05.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fb59e700 1 --2- 172.21.15.3:0/226448788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc1405d0 0x7f10fc140a60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fb59e700 1 --2- 172.21.15.3:0/226448788 >> v2:172.21.15.143:3300/0 conn(0x7f10fc0614b0 0x7f10fc061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fb59e700 1 -- 172.21.15.3:0/226448788 >> 172.21.15.3:0/226448788 conn(0x7f10fc066f70 msgr2=0x7f10fc067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:05.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fb59e700 1 -- 172.21.15.3:0/226448788 shutdown_connections 2022-07-21T01:43:05.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.603+0000 7f10fb59e700 1 -- 172.21.15.3:0/226448788 wait complete. 2022-07-21T01:43:05.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fb59e700 1 Processor -- start 2022-07-21T01:43:05.616 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fb59e700 1 -- start start 2022-07-21T01:43:05.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fb59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10fc0614b0 0x7f10fc1421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc0ad750 0x7f10fc152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fb59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10fc1405d0 0x7f10fc152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fb59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc0b0380 con 0x7f10fc1405d0 2022-07-21T01:43:05.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fb59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc0b3e70 con 0x7f10fc0614b0 2022-07-21T01:43:05.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc0b1120 con 0x7f10fc0ad750 2022-07-21T01:43:05.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10fc1405d0 0x7f10fc152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.621 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10fc1405d0 0x7f10fc152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52244/0 (socket says 172.21.15.3:52244) 2022-07-21T01:43:05.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fad9d700 1 -- 172.21.15.3:0/3576261941 learned_addr learned my addr 172.21.15.3:0/3576261941 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:05.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10fa59c700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10fc0614b0 0x7f10fc1421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.607+0000 7f10f9d9b700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc0ad750 0x7f10fc152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10fad9d700 1 -- 172.21.15.3:0/3576261941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc0ad750 msgr2=0x7f10fc152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10fad9d700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc0ad750 0x7f10fc152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10fad9d700 1 -- 172.21.15.3:0/3576261941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10fc0614b0 msgr2=0x7f10fc1421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10fad9d700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10fc0614b0 0x7f10fc1421a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10fad9d700 1 -- 172.21.15.3:0/3576261941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4009470 con 0x7f10fc1405d0 2022-07-21T01:43:05.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10fad9d700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10fc1405d0 0x7f10fc152f30 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7f10fc0aed30 tx=0x7f10f4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10f37fe700 1 -- 172.21.15.3:0/3576261941 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10f400b8d0 con 0x7f10fc1405d0 2022-07-21T01:43:05.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10f37fe700 1 -- 172.21.15.3:0/3576261941 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f40032f0 con 0x7f10fc1405d0 2022-07-21T01:43:05.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc0b13c0 con 0x7f10fc1405d0 2022-07-21T01:43:05.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10f37fe700 1 -- 172.21.15.3:0/3576261941 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10f400b600 con 0x7f10fc1405d0 2022-07-21T01:43:05.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc1525a0 con 0x7f10fc1405d0 2022-07-21T01:43:05.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc0b4040 con 0x7f10fc1405d0 2022-07-21T01:43:05.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.611+0000 7f10f37fe700 1 -- 172.21.15.3:0/3576261941 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f10f4026030 con 0x7f10fc1405d0 2022-07-21T01:43:05.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.615+0000 7f10f37fe700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10e0053980 0x7f10e0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.615+0000 7f10f37fe700 1 -- 172.21.15.3:0/3576261941 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f10f406cbd0 con 0x7f10fc1405d0 2022-07-21T01:43:05.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.615+0000 7f10fa59c700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10e0053980 0x7f10e0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.615+0000 7f10fa59c700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10e0053980 0x7f10e0055e40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f10ec00b000 tx=0x7f10ec006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.619+0000 7f10f37fe700 1 -- 172.21.15.3:0/3576261941 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f10f4043800 con 0x7f10fc1405d0 2022-07-21T01:43:05.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.743+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f10fc155b80 con 0x7f10fc1405d0 2022-07-21T01:43:05.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.743+0000 7f10f37fe700 1 -- 172.21.15.3:0/3576261941 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v161) v1 ==== 92+0+10 (secure 0 0 0) 0x7f10f40240a0 con 0x7f10fc1405d0 2022-07-21T01:43:05.748 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:43:05.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10e0053980 msgr2=0x7f10e0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10e0053980 0x7f10e0055e40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f10ec00b000 tx=0x7f10ec006df0).stop 2022-07-21T01:43:05.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10fc1405d0 msgr2=0x7f10fc152f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10fc1405d0 0x7f10fc152f30 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7f10fc0aed30 tx=0x7f10f4016000).stop 2022-07-21T01:43:05.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 shutdown_connections 2022-07-21T01:43:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10fc1405d0 0x7f10fc152f30 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10fc0ad750 0x7f10fc152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f10e0053980 0x7f10e0055e40 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 --2- 172.21.15.3:0/3576261941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10fc0614b0 0x7f10fc1421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 >> 172.21.15.3:0/3576261941 conn(0x7f10fc066f70 msgr2=0x7f10fc0b8060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 shutdown_connections 2022-07-21T01:43:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.747+0000 7f10fb59e700 1 -- 172.21.15.3:0/3576261941 wait complete. 2022-07-21T01:43:05.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-21T01:43:05.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.867+0000 7f15b45e4700 1 Processor -- start 2022-07-21T01:43:05.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.867+0000 7f15b45e4700 1 -- start start 2022-07-21T01:43:05.875 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f15ac0b00a0 0x7f15ac05f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ac0614f0 con 0x7f15ac14ad10 2022-07-21T01:43:05.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ac14b770 con 0x7f15ac0acb40 2022-07-21T01:43:05.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ac14ba10 con 0x7f15ac0b00a0 2022-07-21T01:43:05.876 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b2380700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f15ac14ad10 0x7f15ac0af350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60528/0 (socket says 172.21.15.3:60528) 2022-07-21T01:43:05.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b2380700 1 -- 172.21.15.3:0/2416856305 learned_addr learned my addr 172.21.15.3:0/2416856305 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:05.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b1b7f700 1 --2- 172.21.15.3:0/2416856305 >> v2:172.21.15.143:3300/0 conn(0x7f15ac0b00a0 0x7f15ac05f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b137e700 1 -- 172.21.15.3:0/2416856305 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008252099 0 0) 0x7f15ac0614f0 con 0x7f15ac14ad10 2022-07-21T01:43:05.877 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b137e700 1 -- 172.21.15.3:0/2416856305 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1594003260 con 0x7f15ac14ad10 2022-07-21T01:43:05.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b1b7f700 1 -- 172.21.15.3:0/2416856305 >> v1:172.21.15.143:6789/0 conn(0x7f15ac0acb40 legacy=0x7f15ac0b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b1b7f700 1 -- 172.21.15.3:0/2416856305 >> v1:172.21.15.3:6789/0 conn(0x7f15ac14ad10 legacy=0x7f15ac0af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b1b7f700 1 -- 172.21.15.3:0/2416856305 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac0aeeb0 con 0x7f15ac0b00a0 2022-07-21T01:43:05.878 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b1b7f700 1 --2- 172.21.15.3:0/2416856305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15ac0b00a0 0x7f15ac05f840 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f15a80032f0 tx=0x7f15a80091d0).ready entity=mon.1 client_cookie=a370844d523c5a18 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b137e700 1 -- 172.21.15.3:0/2416856305 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15a800b6c0 con 0x7f15ac0b00a0 2022-07-21T01:43:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b137e700 1 -- 172.21.15.3:0/2416856305 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a800b820 con 0x7f15ac0b00a0 2022-07-21T01:43:05.879 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b137e700 1 -- 172.21.15.3:0/2416856305 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15a800b990 con 0x7f15ac0b00a0 2022-07-21T01:43:05.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 -- 172.21.15.3:0/2416856305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15ac0b00a0 msgr2=0x7f15ac05f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 --2- 172.21.15.3:0/2416856305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15ac0b00a0 0x7f15ac05f840 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f15a80032f0 tx=0x7f15a80091d0).stop 2022-07-21T01:43:05.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 -- 172.21.15.3:0/2416856305 shutdown_connections 2022-07-21T01:43:05.880 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 --2- 172.21.15.3:0/2416856305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15ac0b00a0 0x7f15ac05f840 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 -- 172.21.15.3:0/2416856305 >> 172.21.15.3:0/2416856305 conn(0x7f15ac0b5700 msgr2=0x7f15ac0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:05.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 -- 172.21.15.3:0/2416856305 shutdown_connections 2022-07-21T01:43:05.881 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 -- 172.21.15.3:0/2416856305 wait complete. 2022-07-21T01:43:05.884 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 Processor -- start 2022-07-21T01:43:05.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.871+0000 7f15b45e4700 1 -- start start 2022-07-21T01:43:05.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b45e4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15ac0acb40 0x7f15ac148090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b45e4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15ac0b00a0 0x7f15ac13b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.885 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b45e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15ac14ad10 0x7f15ac13be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b45e4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0b50f0 con 0x7f15ac0acb40 2022-07-21T01:43:05.886 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b45e4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0b44f0 con 0x7f15ac14ad10 2022-07-21T01:43:05.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b45e4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0b4790 con 0x7f15ac0b00a0 2022-07-21T01:43:05.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2380700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15ac0acb40 0x7f15ac148090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2380700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15ac0acb40 0x7f15ac148090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52254/0 (socket says 172.21.15.3:52254) 2022-07-21T01:43:05.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2380700 1 -- 172.21.15.3:0/3804925553 learned_addr learned my addr 172.21.15.3:0/3804925553 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:05.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b1b7f700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15ac0b00a0 0x7f15ac13b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2b81700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15ac14ad10 0x7f15ac13be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2380700 1 -- 172.21.15.3:0/3804925553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15ac0b00a0 msgr2=0x7f15ac13b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2380700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15ac0b00a0 0x7f15ac13b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2380700 1 -- 172.21.15.3:0/3804925553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15ac14ad10 msgr2=0x7f15ac13be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:05.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2380700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15ac14ad10 0x7f15ac13be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:05.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2380700 1 -- 172.21.15.3:0/3804925553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a8009470 con 0x7f15ac0acb40 2022-07-21T01:43:05.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b2380700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15ac0acb40 0x7f15ac148090 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f159c001110 tx=0x7f159c007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f159b7fe700 1 -- 172.21.15.3:0/3804925553 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f159c0070d0 con 0x7f15ac0acb40 2022-07-21T01:43:05.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f159b7fe700 1 -- 172.21.15.3:0/3804925553 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c007230 con 0x7f15ac0acb40 2022-07-21T01:43:05.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac067780 con 0x7f15ac0acb40 2022-07-21T01:43:05.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f159b7fe700 1 -- 172.21.15.3:0/3804925553 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f159c005610 con 0x7f15ac0acb40 2022-07-21T01:43:05.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac0b3490 con 0x7f15ac0acb40 2022-07-21T01:43:05.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.875+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac141e40 con 0x7f15ac0acb40 2022-07-21T01:43:05.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.879+0000 7f159b7fe700 1 -- 172.21.15.3:0/3804925553 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f159c010070 con 0x7f15ac0acb40 2022-07-21T01:43:05.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.879+0000 7f159b7fe700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f158c053a70 0x7f158c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:05.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.879+0000 7f159b7fe700 1 -- 172.21.15.3:0/3804925553 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f159c06b760 con 0x7f15ac0acb40 2022-07-21T01:43:05.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.879+0000 7f15b1b7f700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f158c053a70 0x7f158c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:05.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.879+0000 7f15b1b7f700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f158c053a70 0x7f158c055f30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f15a801b000 tx=0x7f15a8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:05.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:05.887+0000 7f159b7fe700 1 -- 172.21.15.3:0/3804925553 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f159c0423b0 con 0x7f15ac0acb40 2022-07-21T01:43:06.018 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.015+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f15ac0b52f0 con 0x7f15ac0acb40 2022-07-21T01:43:06.022 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.015+0000 7f159b7fe700 1 -- 172.21.15.3:0/3804925553 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v161) v1 ==== 104+0+10 (secure 0 0 0) 0x7f159c018070 con 0x7f15ac0acb40 2022-07-21T01:43:06.023 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:43:06.026 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f158c053a70 msgr2=0x7f158c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f158c053a70 0x7f158c055f30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f15a801b000 tx=0x7f15a8016000).stop 2022-07-21T01:43:06.027 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15ac0acb40 msgr2=0x7f15ac148090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15ac0acb40 0x7f15ac148090 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f159c001110 tx=0x7f159c007ad0).stop 2022-07-21T01:43:06.028 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 shutdown_connections 2022-07-21T01:43:06.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f158c053a70 0x7f158c055f30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.029 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15ac14ad10 0x7f15ac13be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15ac0b00a0 0x7f15ac13b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.030 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 --2- 172.21.15.3:0/3804925553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15ac0acb40 0x7f15ac148090 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 >> 172.21.15.3:0/3804925553 conn(0x7f15ac0b5700 msgr2=0x7f15ac13c750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:06.031 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 shutdown_connections 2022-07-21T01:43:06.032 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.019+0000 7f15b45e4700 1 -- 172.21.15.3:0/3804925553 wait complete. 2022-07-21T01:43:06.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-21T01:43:06.155 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.151+0000 7fcf4602a700 1 Processor -- start 2022-07-21T01:43:06.156 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.151+0000 7fcf4602a700 1 -- start start 2022-07-21T01:43:06.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcf400b31d0 0x7fcf40066020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.167 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf40140780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.168 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf40067030 con 0x7fcf400a9da0 2022-07-21T01:43:06.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf40064a70 con 0x7fcf400b31d0 2022-07-21T01:43:06.169 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf40061f60 con 0x7fcf401402d0 2022-07-21T01:43:06.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf45829700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf40140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.170 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf45829700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf40140780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:59180/0 (socket says 172.21.15.3:59180) 2022-07-21T01:43:06.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf45829700 1 -- 172.21.15.3:0/3061378055 learned_addr learned my addr 172.21.15.3:0/3061378055 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:06.171 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf45028700 1 --2- 172.21.15.3:0/3061378055 >> v2:172.21.15.143:3300/0 conn(0x7fcf400b31d0 0x7fcf40066020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf45829700 1 -- 172.21.15.3:0/3061378055 >> v2:172.21.15.143:3300/0 conn(0x7fcf400b31d0 msgr2=0x7fcf40066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:06.172 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf45829700 1 --2- 172.21.15.3:0/3061378055 >> v2:172.21.15.143:3300/0 conn(0x7fcf400b31d0 0x7fcf40066020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf45829700 1 -- 172.21.15.3:0/3061378055 >> v1:172.21.15.143:6789/0 conn(0x7fcf400a9da0 legacy=0x7fcf40060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf45829700 1 -- 172.21.15.3:0/3061378055 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf400b35b0 con 0x7fcf401402d0 2022-07-21T01:43:06.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf45829700 1 --2- 172.21.15.3:0/3061378055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf40140780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fcf380032f0 tx=0x7fcf3800b530).ready entity=mon.2 client_cookie=fca8feaa032e7071 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:06.174 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf2ffff700 1 -- 172.21.15.3:0/3061378055 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf3800de20 con 0x7fcf401402d0 2022-07-21T01:43:06.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf2ffff700 1 -- 172.21.15.3:0/3061378055 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38010470 con 0x7fcf401402d0 2022-07-21T01:43:06.175 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf2ffff700 1 -- 172.21.15.3:0/3061378055 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf380107d0 con 0x7fcf401402d0 2022-07-21T01:43:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 -- 172.21.15.3:0/3061378055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 msgr2=0x7fcf40140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 --2- 172.21.15.3:0/3061378055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf40140780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fcf380032f0 tx=0x7fcf3800b530).stop 2022-07-21T01:43:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 -- 172.21.15.3:0/3061378055 shutdown_connections 2022-07-21T01:43:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 --2- 172.21.15.3:0/3061378055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf40140780 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 --2- 172.21.15.3:0/3061378055 >> v2:172.21.15.143:3300/0 conn(0x7fcf400b31d0 0x7fcf40066020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 -- 172.21.15.3:0/3061378055 >> 172.21.15.3:0/3061378055 conn(0x7fcf40063870 msgr2=0x7fcf40063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:06.179 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 -- 172.21.15.3:0/3061378055 shutdown_connections 2022-07-21T01:43:06.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.155+0000 7fcf4602a700 1 -- 172.21.15.3:0/3061378055 wait complete. 2022-07-21T01:43:06.180 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf4602a700 1 Processor -- start 2022-07-21T01:43:06.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf4602a700 1 -- start start 2022-07-21T01:43:06.181 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf4602a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf400a9da0 0x7fcf400b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf4602a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf400b31d0 0x7fcf400acae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.182 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf4602a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf400acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.183 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf4602a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400ad4c0 con 0x7fcf400a9da0 2022-07-21T01:43:06.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf4602a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400ac6b0 con 0x7fcf400b31d0 2022-07-21T01:43:06.184 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf4602a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400ab3f0 con 0x7fcf401402d0 2022-07-21T01:43:06.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf45028700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf400a9da0 0x7fcf400b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.185 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf45028700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf400a9da0 0x7fcf400b4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52268/0 (socket says 172.21.15.3:52268) 2022-07-21T01:43:06.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf45028700 1 -- 172.21.15.3:0/638705893 learned_addr learned my addr 172.21.15.3:0/638705893 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:06.186 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf45829700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf400acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf44827700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf400b31d0 0x7fcf400acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.187 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf45028700 1 -- 172.21.15.3:0/638705893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 msgr2=0x7fcf400acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf45028700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf400acfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.188 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf45028700 1 -- 172.21.15.3:0/638705893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf400b31d0 msgr2=0x7fcf400acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf45028700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf400b31d0 0x7fcf400acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.189 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.159+0000 7fcf45028700 1 -- 172.21.15.3:0/638705893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3800b7d0 con 0x7fcf400a9da0 2022-07-21T01:43:06.190 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.163+0000 7fcf45028700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf400a9da0 0x7fcf400b4cb0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7fcf3c00c000 tx=0x7fcf3c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:06.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.163+0000 7fcf2e7fc700 1 -- 172.21.15.3:0/638705893 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf3c010780 con 0x7fcf400a9da0 2022-07-21T01:43:06.191 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.163+0000 7fcf2e7fc700 1 -- 172.21.15.3:0/638705893 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3c0108e0 con 0x7fcf400a9da0 2022-07-21T01:43:06.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.163+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf400ab650 con 0x7fcf400a9da0 2022-07-21T01:43:06.192 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.163+0000 7fcf2e7fc700 1 -- 172.21.15.3:0/638705893 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf3c010c40 con 0x7fcf400a9da0 2022-07-21T01:43:06.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.163+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf400b51a0 con 0x7fcf400a9da0 2022-07-21T01:43:06.193 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.163+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf40066af0 con 0x7fcf400a9da0 2022-07-21T01:43:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.163+0000 7fcf2e7fc700 1 -- 172.21.15.3:0/638705893 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcf3c01c460 con 0x7fcf400a9da0 2022-07-21T01:43:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.163+0000 7fcf2e7fc700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf20053980 0x7fcf20055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.167+0000 7fcf44827700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf20053980 0x7fcf20055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.167+0000 7fcf2e7fc700 1 -- 172.21.15.3:0/638705893 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fcf3c06c320 con 0x7fcf400a9da0 2022-07-21T01:43:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.167+0000 7fcf44827700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf20053980 0x7fcf20055e40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7fcf3000d000 tx=0x7fcf30007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:06.198 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.171+0000 7fcf2e7fc700 1 -- 172.21.15.3:0/638705893 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcf3c043080 con 0x7fcf400a9da0 2022-07-21T01:43:06.307 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.303+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcf400574e0 con 0x7fcf400a9da0 2022-07-21T01:43:06.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.303+0000 7fcf2e7fc700 1 -- 172.21.15.3:0/638705893 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v161) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcf3c0466a0 con 0x7fcf400a9da0 2022-07-21T01:43:06.309 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 8 2022-07-21T01:43:06.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf20053980 msgr2=0x7fcf20055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf20053980 0x7fcf20055e40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7fcf3000d000 tx=0x7fcf30007b90).stop 2022-07-21T01:43:06.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf400a9da0 msgr2=0x7fcf400b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf400a9da0 0x7fcf400b4cb0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7fcf3c00c000 tx=0x7fcf3c006df0).stop 2022-07-21T01:43:06.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 shutdown_connections 2022-07-21T01:43:06.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf401402d0 0x7fcf400acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fcf20053980 0x7fcf20055e40 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf400b31d0 0x7fcf400acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 --2- 172.21.15.3:0/638705893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf400a9da0 0x7fcf400b4cb0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 >> 172.21.15.3:0/638705893 conn(0x7fcf40063870 msgr2=0x7fcf4005f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:06.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 shutdown_connections 2022-07-21T01:43:06.315 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.307+0000 7fcf4602a700 1 -- 172.21.15.3:0/638705893 wait complete. 2022-07-21T01:43:06.321 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-21T01:43:06.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:43:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1b59e700 1 Processor -- start 2022-07-21T01:43:06.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1b59e700 1 -- start start 2022-07-21T01:43:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b2710 0x7f8b1c14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8b1c0b01a0 0x7f8b1c14b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1b59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8b1c0afd60 0x7f8b1c0a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1b59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1c0a7250 con 0x7f8b1c0afd60 2022-07-21T01:43:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1c0a7480 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c0a7720 con 0x7f8b1c0b2710 2022-07-21T01:43:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1ad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8b1c0afd60 0x7f8b1c0a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1ad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f8b1c0afd60 0x7f8b1c0a8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52280/0 (socket says 172.21.15.3:52280) 2022-07-21T01:43:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1ad9d700 1 -- 172.21.15.3:0/3014446473 learned_addr learned my addr 172.21.15.3:0/3014446473 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:06.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b19d9b700 1 --2- 172.21.15.3:0/3014446473 >> v2:172.21.15.143:3300/0 conn(0x7f8b1c0b01a0 0x7f8b1c14b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1a59c700 1 --2- 172.21.15.3:0/3014446473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b2710 0x7f8b1c14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1ad9d700 1 -- 172.21.15.3:0/3014446473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b2710 msgr2=0x7f8b1c14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:06.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1ad9d700 1 --2- 172.21.15.3:0/3014446473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b2710 0x7f8b1c14afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1ad9d700 1 -- 172.21.15.3:0/3014446473 >> v2:172.21.15.143:3300/0 conn(0x7f8b1c0b01a0 msgr2=0x7f8b1c14b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:06.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1ad9d700 1 --2- 172.21.15.3:0/3014446473 >> v2:172.21.15.143:3300/0 conn(0x7f8b1c0b01a0 0x7f8b1c14b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1ad9d700 1 -- 172.21.15.3:0/3014446473 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1c0a79c0 con 0x7f8b1c0afd60 2022-07-21T01:43:06.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b19d9b700 1 --2- 172.21.15.3:0/3014446473 >> v2:172.21.15.143:3300/0 conn(0x7f8b1c0b01a0 0x7f8b1c14b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:43:06.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1ad9d700 1 --2- 172.21.15.3:0/3014446473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1c0afd60 0x7f8b1c0a8540 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f8b14002f70 tx=0x7f8b14006eb0).ready entity=mon.0 client_cookie=5a85e17cad180bf server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:06.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1959a700 1 -- 172.21.15.3:0/3014446473 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b1400d570 con 0x7f8b1c0afd60 2022-07-21T01:43:06.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1959a700 1 -- 172.21.15.3:0/3014446473 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b14000e90 con 0x7f8b1c0afd60 2022-07-21T01:43:06.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.427+0000 7f8b1959a700 1 -- 172.21.15.3:0/3014446473 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b1400d900 con 0x7f8b1c0afd60 2022-07-21T01:43:06.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 -- 172.21.15.3:0/3014446473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1c0afd60 msgr2=0x7f8b1c0a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/3014446473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1c0afd60 0x7f8b1c0a8540 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f8b14002f70 tx=0x7f8b14006eb0).stop 2022-07-21T01:43:06.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 -- 172.21.15.3:0/3014446473 shutdown_connections 2022-07-21T01:43:06.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/3014446473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1c0afd60 0x7f8b1c0a8540 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/3014446473 >> v2:172.21.15.143:3300/0 conn(0x7f8b1c0b01a0 0x7f8b1c14b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/3014446473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b2710 0x7f8b1c14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 -- 172.21.15.3:0/3014446473 >> 172.21.15.3:0/3014446473 conn(0x7f8b1c060310 msgr2=0x7f8b1c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:06.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 -- 172.21.15.3:0/3014446473 shutdown_connections 2022-07-21T01:43:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 -- 172.21.15.3:0/3014446473 wait complete. 2022-07-21T01:43:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 Processor -- start 2022-07-21T01:43:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 -- start start 2022-07-21T01:43:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b1c0afd60 0x7f8b1c05c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b01a0 0x7f8b1c156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1c0b2710 0x7f8b1c156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c0acc10 con 0x7f8b1c0b2710 2022-07-21T01:43:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c0634a0 con 0x7f8b1c0afd60 2022-07-21T01:43:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c0ab430 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1c0b2710 0x7f8b1c156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b19d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b01a0 0x7f8b1c156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b1a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b1c0afd60 0x7f8b1c05c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b19d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b01a0 0x7f8b1c156a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:59200/0 (socket says 172.21.15.3:59200) 2022-07-21T01:43:06.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.431+0000 7f8b19d9b700 1 -- 172.21.15.3:0/390002949 learned_addr learned my addr 172.21.15.3:0/390002949 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:06.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b19d9b700 1 -- 172.21.15.3:0/390002949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b1c0afd60 msgr2=0x7f8b1c05c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b19d9b700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b1c0afd60 0x7f8b1c05c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b19d9b700 1 -- 172.21.15.3:0/390002949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1c0b2710 msgr2=0x7f8b1c156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b19d9b700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1c0b2710 0x7f8b1c156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b19d9b700 1 -- 172.21.15.3:0/390002949 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1400b1e0 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b19d9b700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b01a0 0x7f8b1c156a40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f8b0c006cc0 tx=0x7f8b0c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:06.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b137fe700 1 -- 172.21.15.3:0/390002949 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0c013620 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b137fe700 1 -- 172.21.15.3:0/390002949 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c013780 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1c0ab6d0 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b137fe700 1 -- 172.21.15.3:0/390002949 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0c013ae0 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1c0aa870 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1c0574e0 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b137fe700 1 -- 172.21.15.3:0/390002949 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8b0c013c40 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b137fe700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b00053950 0x7f8b00055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b137fe700 1 -- 172.21.15.3:0/390002949 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f8b0c06c220 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.435+0000 7f8b1a59c700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b00053950 0x7f8b00055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.439+0000 7f8b1a59c700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b00053950 0x7f8b00055e10 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f8b0400c000 tx=0x7f8b04008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.443+0000 7f8b137fe700 1 -- 172.21.15.3:0/390002949 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8b0c042fe0 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.611+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b1c0ab860 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.614 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.611+0000 7f8b137fe700 1 -- 172.21.15.3:0/390002949 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2489 (secure 0 0 0) 0x7f8b0c046600 con 0x7f8b1c0b01a0 2022-07-21T01:43:06.615 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:43:06.615 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":81,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/781340149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":781340149},{"type":"v1","addr":"172.21.15.143:6833","nonce":781340149}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79},{"gid":8712,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3829565046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3829565046},{"type":"v1","addr":"172.21.15.3:6835","nonce":3829565046}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:58.238508+0000","modified":"2022-07-21T01:42:59.241585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8727},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8727":{"gid":8727,"name":"c","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1787668325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1787668325},{"type":"v1","addr":"172.21.15.3:6837","nonce":1787668325}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-21T01:43:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b00053950 msgr2=0x7f8b00055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b00053950 0x7f8b00055e10 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f8b0400c000 tx=0x7f8b04008040).stop 2022-07-21T01:43:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b01a0 msgr2=0x7f8b1c156a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b01a0 0x7f8b1c156a40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f8b0c006cc0 tx=0x7f8b0c008040).stop 2022-07-21T01:43:06.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 shutdown_connections 2022-07-21T01:43:06.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1c0b2710 0x7f8b1c156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b1c0b01a0 0x7f8b1c156a40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f8b00053950 0x7f8b00055e10 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 --2- 172.21.15.3:0/390002949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b1c0afd60 0x7f8b1c05c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:06.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 >> 172.21.15.3:0/390002949 conn(0x7f8b1c060310 msgr2=0x7f8b1c148c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:06.619 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 shutdown_connections 2022-07-21T01:43:06.620 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:06.615+0000 7f8b1b59e700 1 -- 172.21.15.3:0/390002949 wait complete. 2022-07-21T01:43:06.620 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 81 2022-07-21T01:43:06.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-21T01:42:58.238508+0000', 'modified': '2022-07-21T01:42:59.241585+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8727}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8727': {'gid': 8727, 'name': 'c', 'rank': 0, 'incarnation': 80, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.3:6837/1787668325', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6836', 'nonce': 1787668325}, {'type': 'v1', 'addr': '172.21.15.3:6837', 'nonce': 1787668325}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-21T01:43:06.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-21T01:43:06.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json status 2022-07-21T01:43:07.048 INFO:teuthology.orchestra.run.smithi003.stdout:{"cluster_fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","whoami":0,"id":8727,"want_state":"up:active","state":"up:active","rank_uptime":8.7977241730000006,"mdsmap_epoch":81,"osdmap_epoch":161,"osdmap_epoch_barrier":160,"uptime":13.438983500000001} 2022-07-21T01:43:07.049 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", "id": 8727, "mdsmap_epoch": 81, "osdmap_epoch": 161, "osdmap_epoch_barrier": 160, "rank_uptime": 8.797724173, "state": "up:active", "uptime": 13.4389835, "want_state": "up:active", "whoami": 0 } 2022-07-21T01:43:07.049 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-21T01:43:07.050 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-21T01:43:07.050 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi174.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-21T01:43:07.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.051 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:43:07.060 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:43:07.075 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:43:07.075 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:43:07.131 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:43:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:43:07.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.134 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:43:07.140 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-21T01:43:07.140 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:43:07.143 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2022-07-21T01:43:07.158 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-21T01:43:07.159 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-21T01:43:07.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2022-07-21T01:43:07.215+0000 7f2889390080 -1 init, newargv = 0x556bad710cb0 newargc=9 2022-07-21T01:43:07.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[23145]: starting ceph client 2022-07-21T01:43:07.220 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-21T01:43:07.221 INFO:teuthology.orchestra.run.smithi174.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-21T01:43:07.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.222 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2022-07-21T01:43:07.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[23145]: starting fuse 2022-07-21T01:43:07.232 INFO:teuthology.orchestra.run.smithi174.stdout:52 2022-07-21T01:43:07.233 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-21T01:43:07.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.234 DEBUG:teuthology.orchestra.run.smithi174:> sudo python3 -c ' 2022-07-21T01:43:07.234 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2022-07-21T01:43:07.235 DEBUG:teuthology.orchestra.run.smithi174:> import re 2022-07-21T01:43:07.235 DEBUG:teuthology.orchestra.run.smithi174:> import os 2022-07-21T01:43:07.236 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2022-07-21T01:43:07.236 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:43:07.237 DEBUG:teuthology.orchestra.run.smithi174:> def find_socket(client_name): 2022-07-21T01:43:07.237 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-21T01:43:07.237 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2022-07-21T01:43:07.238 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:43:07.238 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2022-07-21T01:43:07.239 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2022-07-21T01:43:07.239 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2022-07-21T01:43:07.240 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2022-07-21T01:43:07.240 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:43:07.240 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2022-07-21T01:43:07.241 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-21T01:43:07.241 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-21T01:43:07.242 DEBUG:teuthology.orchestra.run.smithi174:> return f 2022-07-21T01:43:07.242 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-21T01:43:07.243 DEBUG:teuthology.orchestra.run.smithi174:> 2022-07-21T01:43:07.243 DEBUG:teuthology.orchestra.run.smithi174:> print(find_socket("client.1")) 2022-07-21T01:43:07.243 DEBUG:teuthology.orchestra.run.smithi174:> ' 2022-07-21T01:43:07.283 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.23145.asok 2022-07-21T01:43:07.288 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.23145.asok 2022-07-21T01:43:07.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.289 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.23145.asok status 2022-07-21T01:43:07.426 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2022-07-21T01:43:07.427 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2022-07-21T01:43:07.427 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-21T01:43:07.428 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-21T01:43:07.428 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2022-07-21T01:43:07.429 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2022-07-21T01:43:07.429 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-21T01:43:07.430 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "23145", 2022-07-21T01:43:07.430 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2022-07-21T01:43:07.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:43:07.431 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2022-07-21T01:43:07.432 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2022-07-21T01:43:07.432 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 7595, 2022-07-21T01:43:07.432 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2022-07-21T01:43:07.432 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2022-07-21T01:43:07.433 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2022-07-21T01:43:07.433 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 7595 2022-07-21T01:43:07.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:43:07.433 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:43:07.433 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:43:07.434 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:43:07.434 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 594693558 2022-07-21T01:43:07.434 INFO:teuthology.orchestra.run.smithi174.stdout: } 2022-07-21T01:43:07.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:43:07.435 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2022-07-21T01:43:07.435 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2022-07-21T01:43:07.435 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "172.21.15.174:0", 2022-07-21T01:43:07.435 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 594693558 2022-07-21T01:43:07.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2022-07-21T01:43:07.436 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.7595 172.21.15.174:0/594693558", 2022-07-21T01:43:07.436 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "172.21.15.174:0/594693558", 2022-07-21T01:43:07.436 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2022-07-21T01:43:07.437 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 81, 2022-07-21T01:43:07.437 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 161, 2022-07-21T01:43:07.437 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2022-07-21T01:43:07.437 INFO:teuthology.orchestra.run.smithi174.stdout: "blacklisted": false, 2022-07-21T01:43:07.437 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "cephfs" 2022-07-21T01:43:07.438 INFO:teuthology.orchestra.run.smithi174.stdout:} 2022-07-21T01:43:07.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.440 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:43:07.440 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:43:07.448 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:43:07.449 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:43:07.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.450 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:43:07.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.470 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:43:07.471 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:43:07.519 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:43:07.521 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:43:07.522 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.523 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:43:07.523 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:43:07.531 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2022-07-21T01:43:07.533 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-21T01:43:07.534 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2022-07-21T01:43:07.535 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-21T01:43:07.536 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:43:07.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[23145]: fuse finished with error 0 and tester_r 0 2022-07-21T01:43:07.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:07.555 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:43:07.556 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:43:07.604 INFO:teuthology.orchestra.run.smithi174.stdout:ext2/ext3 2022-07-21T01:43:07.606 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-21T01:43:07.607 INFO:teuthology.orchestra.run:waiting for 900 2022-07-21T01:43:07.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-21T01:43:07.613 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-21T01:43:07.620 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-21T01:43:07.621 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-07-21T01:43:07.642 INFO:teuthology.orchestra.run.smithi171.stdout: File: "." 2022-07-21T01:43:07.643 INFO:teuthology.orchestra.run.smithi171.stdout: ID: 0 Namelen: 255 Type: fuseblk 2022-07-21T01:43:07.643 INFO:teuthology.orchestra.run.smithi171.stdout:Block size: 4194304 Fundamental block size: 4194304 2022-07-21T01:43:07.643 INFO:teuthology.orchestra.run.smithi171.stdout:Blocks: Total: 86526 Free: 86526 Available: 86526 2022-07-21T01:43:07.644 INFO:teuthology.orchestra.run.smithi171.stdout:Inodes: Total: 1 Free: 0 2022-07-21T01:43:07.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-21T01:43:07.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88df59e700 1 Processor -- start 2022-07-21T01:43:07.765 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88df59e700 1 -- start start 2022-07-21T01:43:07.770 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88df59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88e00b2710 0x7f88e014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:07.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88df59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88e014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:07.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88df59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e00a7250 con 0x7f88e00b01a0 2022-07-21T01:43:07.771 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88df59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e00a7480 con 0x7f88e00b2710 2022-07-21T01:43:07.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88df59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00a7720 con 0x7f88e00a8c50 2022-07-21T01:43:07.772 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88ddd9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f88e00b01a0 0x7f88e00a8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60570/0 (socket says 172.21.15.3:60570) 2022-07-21T01:43:07.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88ddd9b700 1 -- 172.21.15.3:0/3557459693 learned_addr learned my addr 172.21.15.3:0/3557459693 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:07.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88ded9d700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88e014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:07.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88de59c700 1 --2- 172.21.15.3:0/3557459693 >> v2:172.21.15.143:3300/0 conn(0x7f88e00b2710 0x7f88e014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:07.773 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110085211 0 0) 0x7f88e00a7250 con 0x7f88e00b01a0 2022-07-21T01:43:07.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c0003260 con 0x7f88e00b01a0 2022-07-21T01:43:07.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066604662 0 0) 0x7f88c0003260 con 0x7f88e00b01a0 2022-07-21T01:43:07.774 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 msgr2=0x7f88e014b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:07.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88e014b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:07.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 >> v2:172.21.15.143:3300/0 conn(0x7f88e00b2710 msgr2=0x7f88e014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:07.775 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 --2- 172.21.15.3:0/3557459693 >> v2:172.21.15.143:3300/0 conn(0x7f88e00b2710 0x7f88e014afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:07.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e00a79c0 con 0x7f88e00b01a0 2022-07-21T01:43:07.776 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f88d80019e0 con 0x7f88e00b01a0 2022-07-21T01:43:07.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 >> v1:172.21.15.3:6789/0 conn(0x7f88e00b01a0 legacy=0x7f88e00a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:07.777 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88c0003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:07.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88c0003c80 0x7f88c0004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:07.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88c00049f0 0x7f88c0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:07.778 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00a7480 con 0x7f88c00049f0 2022-07-21T01:43:07.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00a7720 con 0x7f88c0003c80 2022-07-21T01:43:07.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c000b750 con 0x7f88e00a8c50 2022-07-21T01:43:07.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88de59c700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88c0003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:07.779 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88ded9d700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88c00049f0 0x7f88c0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:07.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88ddd9b700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88c0003c80 0x7f88c0004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:07.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88de59c700 1 -- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88c0003c80 msgr2=0x7f88c0004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:07.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88de59c700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88c0003c80 0x7f88c0004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:07.780 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88de59c700 1 -- 172.21.15.3:0/3557459693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88c00049f0 msgr2=0x7f88c0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:07.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88de59c700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88c00049f0 0x7f88c0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:07.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88de59c700 1 -- 172.21.15.3:0/3557459693 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0003260 con 0x7f88e00a8c50 2022-07-21T01:43:07.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.763+0000 7f88de59c700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88c0003750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f88c800c000 tx=0x7f88c8006df0).ready entity=mon.2 client_cookie=1139f617a3af7c9d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:07.781 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c800e830 con 0x7f88e00a8c50 2022-07-21T01:43:07.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c800e990 con 0x7f88e00a8c50 2022-07-21T01:43:07.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88dd59a700 1 -- 172.21.15.3:0/3557459693 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c800ecf0 con 0x7f88e00a8c50 2022-07-21T01:43:07.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 -- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 msgr2=0x7f88c0003750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:07.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 --2- 172.21.15.3:0/3557459693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88c0003750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f88c800c000 tx=0x7f88c8006df0).stop 2022-07-21T01:43:07.783 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ded9d700 1 -- 172.21.15.3:0/3557459693 reap_dead start 2022-07-21T01:43:07.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 -- 172.21.15.3:0/3557459693 shutdown_connections 2022-07-21T01:43:07.784 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 -- 172.21.15.3:0/3557459693 >> 172.21.15.3:0/3557459693 conn(0x7f88e0060310 msgr2=0x7f88e00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:07.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 -- 172.21.15.3:0/3557459693 shutdown_connections 2022-07-21T01:43:07.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 -- 172.21.15.3:0/3557459693 wait complete. 2022-07-21T01:43:07.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 Processor -- start 2022-07-21T01:43:07.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 -- start start 2022-07-21T01:43:07.793 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88e005c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:07.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e00b4290 0x7f88e0064cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:07.794 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e00651a0 0x7f88e0159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:07.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e0063510 con 0x7f88e00651a0 2022-07-21T01:43:07.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00b4ac0 con 0x7f88e00b4290 2022-07-21T01:43:07.795 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88df59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00b4d60 con 0x7f88e00a8c50 2022-07-21T01:43:07.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ded9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e00651a0 0x7f88e0159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:07.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ded9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e00651a0 0x7f88e0159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52306/0 (socket says 172.21.15.3:52306) 2022-07-21T01:43:07.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ded9d700 1 -- 172.21.15.3:0/2702163702 learned_addr learned my addr 172.21.15.3:0/2702163702 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:07.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88de59c700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88e005c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:07.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ddd9b700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e00b4290 0x7f88e0064cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:07.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ded9d700 1 -- 172.21.15.3:0/2702163702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 msgr2=0x7f88e005c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:07.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ded9d700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88e005c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:07.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ded9d700 1 -- 172.21.15.3:0/2702163702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e00b4290 msgr2=0x7f88e0064cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ded9d700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e00b4290 0x7f88e0064cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88ded9d700 1 -- 172.21.15.3:0/2702163702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e00b60e0 con 0x7f88e00651a0 2022-07-21T01:43:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.767+0000 7f88de59c700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88e005c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:43:07.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88ded9d700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e00651a0 0x7f88e0159ec0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f88d000b000 tx=0x7f88d0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88d77fe700 1 -- 172.21.15.3:0/2702163702 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d0019070 con 0x7f88e00651a0 2022-07-21T01:43:07.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88d77fe700 1 -- 172.21.15.3:0/2702163702 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d0000e90 con 0x7f88e00651a0 2022-07-21T01:43:07.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e005c900 con 0x7f88e00651a0 2022-07-21T01:43:07.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88d77fe700 1 -- 172.21.15.3:0/2702163702 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d000f8f0 con 0x7f88e00651a0 2022-07-21T01:43:07.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e015a3b0 con 0x7f88e00651a0 2022-07-21T01:43:07.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e00574e0 con 0x7f88e00651a0 2022-07-21T01:43:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88d77fe700 1 -- 172.21.15.3:0/2702163702 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f88d000fa50 con 0x7f88e00651a0 2022-07-21T01:43:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88d77fe700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c4053a70 0x7f88c4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:07.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88d77fe700 1 -- 172.21.15.3:0/2702163702 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7f88d006bd40 con 0x7f88e00651a0 2022-07-21T01:43:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88de59c700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c4053a70 0x7f88c4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.771+0000 7f88de59c700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c4053a70 0x7f88c4055f30 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f88c801c000 tx=0x7f88c8019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:07.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.775+0000 7f88d77fe700 1 -- 172.21.15.3:0/2702163702 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f88d0042b00 con 0x7f88e00651a0 2022-07-21T01:43:07.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.903+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f88e00b0720 con 0x7f88e00651a0 2022-07-21T01:43:07.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88d77fe700 1 -- 172.21.15.3:0/2702163702 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v161) v1 ==== 81+0+34051 (secure 0 0 0) 0x7f88d002c520 con 0x7f88e00651a0 2022-07-21T01:43:07.911 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:43:07.912 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T01:43:07.912 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 161, 2022-07-21T01:43:07.912 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "033d20cc-eff0-4d91-a7d1-56acf083b1d2", 2022-07-21T01:43:07.913 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2022-07-21T01:33:06.656628+0000", 2022-07-21T01:43:07.913 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2022-07-21T01:42:59.242307+0000", 2022-07-21T01:43:07.914 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:43:07.914 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2022-07-21T01:33:14.593772+0000", 2022-07-21T01:43:07.915 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-21T01:43:07.915 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2022-07-21T01:43:07.916 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2022-07-21T01:43:07.916 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2022-07-21T01:43:07.916 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2022-07-21T01:43:07.917 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2022-07-21T01:43:07.917 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2022-07-21T01:43:07.917 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:43:07.917 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2022-07-21T01:43:07.918 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2022-07-21T01:43:07.918 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-21T01:43:07.918 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-21T01:43:07.918 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2022-07-21T01:43:07.919 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 30, 2022-07-21T01:43:07.919 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2022-07-21T01:43:07.919 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "jewel", 2022-07-21T01:43:07.919 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2022-07-21T01:43:07.919 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "octopus", 2022-07-21T01:43:07.920 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2022-07-21T01:43:07.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:07.920 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2022-07-21T01:43:07.920 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "device_health_metrics", 2022-07-21T01:43:07.921 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:09.606566+0000", 2022-07-21T01:43:07.921 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:43:07.921 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:43:07.922 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:43:07.922 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:43:07.922 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:43:07.922 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:43:07.923 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:43:07.923 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:43:07.923 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2022-07-21T01:43:07.923 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2022-07-21T01:43:07.924 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2022-07-21T01:43:07.924 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2022-07-21T01:43:07.925 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2022-07-21T01:43:07.925 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:43:07.926 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:43:07.926 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:43:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:43:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:43:07.927 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:43:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:43:07.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "17", 2022-07-21T01:43:07.929 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:43:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:43:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:43:07.930 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:43:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:43:07.931 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:43:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:43:07.932 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:43:07.933 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:43:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:43:07.934 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:43:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:43:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:43:07.935 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:43:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:43:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:43:07.936 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:43:07.937 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:43:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:43:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:43:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:43:07.939 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:43:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:43:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:43:07.940 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:43:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:43:07.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:43:07.942 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:43:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:43:07.943 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:43:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:43:07.944 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:43:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:43:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:43:07.945 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:43:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:43:07.946 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:43:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:43:07.947 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2022-07-21T01:43:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.948 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:43:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_devicehealth": {} 2022-07-21T01:43:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:07.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:07.950 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2022-07-21T01:43:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2022-07-21T01:43:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:33:18.012897+0000", 2022-07-21T01:43:07.951 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:43:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:43:07.952 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:43:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:43:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:43:07.953 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:43:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:43:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:43:07.954 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:43:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:43:07.955 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:43:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:43:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:43:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:43:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:43:07.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:43:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:43:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:43:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:43:07.957 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:43:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.958 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "20", 2022-07-21T01:43:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:43:07.959 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:43:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:43:07.960 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:43:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:43:07.961 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:43:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:43:07.962 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:43:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:43:07.963 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:43:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:43:07.964 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:43:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:43:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:43:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:43:07.965 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:43:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:43:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:43:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:43:07.966 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:43:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:43:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:43:07.967 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:43:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:43:07.968 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:43:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:43:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:43:07.969 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:43:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:43:07.970 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:43:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:43:07.971 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:43:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:43:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:43:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:43:07.972 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:43:07.973 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:43:07.973 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:43:07.973 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:43:07.973 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2022-07-21T01:43:07.974 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:07.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:07.974 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 29, 2022-07-21T01:43:07.975 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_metadata", 2022-07-21T01:43:07.975 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:42:56.372231+0000", 2022-07-21T01:43:07.975 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:43:07.975 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:43:07.975 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:43:07.976 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:43:07.976 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:43:07.976 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:43:07.976 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:43:07.977 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:43:07.977 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:43:07.977 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:43:07.977 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:43:07.977 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:43:07.978 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:43:07.978 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:43:07.978 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:43:07.978 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:43:07.979 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:43:07.979 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:43:07.979 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:43:07.979 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:43:07.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.980 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "160", 2022-07-21T01:43:07.980 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:43:07.980 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:43:07.980 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:43:07.980 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:43:07.981 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:43:07.981 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:43:07.981 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:43:07.981 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:43:07.982 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:43:07.982 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:43:07.982 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:43:07.982 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:43:07.982 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:43:07.983 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:43:07.983 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:43:07.983 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:43:07.983 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:43:07.984 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:43:07.984 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:43:07.984 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:43:07.984 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:43:07.984 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:43:07.985 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:43:07.985 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:43:07.985 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:43:07.985 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:43:07.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.986 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:43:07.986 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:43:07.986 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:43:07.987 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:43:07.987 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:43:07.987 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:43:07.987 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:43:07.988 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:43:07.988 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:43:07.988 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:43:07.988 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:43:07.988 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2022-07-21T01:43:07.989 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4, 2022-07-21T01:43:07.989 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 16, 2022-07-21T01:43:07.989 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_priority": 5 2022-07-21T01:43:07.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.989 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:43:07.990 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:43:07.990 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": "cephfs" 2022-07-21T01:43:07.990 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:07.990 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:07.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:07.991 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 30, 2022-07-21T01:43:07.991 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "cephfs_data", 2022-07-21T01:43:07.992 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2022-07-21T01:42:57.252984+0000", 2022-07-21T01:43:07.992 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2022-07-21T01:43:07.992 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2022-07-21T01:43:07.992 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2022-07-21T01:43:07.992 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2022-07-21T01:43:07.993 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2022-07-21T01:43:07.993 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2022-07-21T01:43:07.993 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2022-07-21T01:43:07.993 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2022-07-21T01:43:07.994 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2022-07-21T01:43:07.994 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2022-07-21T01:43:07.994 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2022-07-21T01:43:07.994 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2022-07-21T01:43:07.994 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2022-07-21T01:43:07.995 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2022-07-21T01:43:07.995 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2022-07-21T01:43:07.995 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2022-07-21T01:43:07.995 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2022-07-21T01:43:07.996 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2022-07-21T01:43:07.996 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2022-07-21T01:43:07.996 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2022-07-21T01:43:07.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:07.996 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "161", 2022-07-21T01:43:07.997 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2022-07-21T01:43:07.997 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-21T01:43:07.997 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2022-07-21T01:43:07.997 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2022-07-21T01:43:07.998 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2022-07-21T01:43:07.998 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2022-07-21T01:43:07.998 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2022-07-21T01:43:07.998 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2022-07-21T01:43:07.999 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2022-07-21T01:43:07.999 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2022-07-21T01:43:07.999 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2022-07-21T01:43:07.999 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2022-07-21T01:43:07.999 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2022-07-21T01:43:08.000 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2022-07-21T01:43:08.000 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2022-07-21T01:43:08.000 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2022-07-21T01:43:08.000 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2022-07-21T01:43:08.001 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2022-07-21T01:43:08.001 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-21T01:43:08.001 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-21T01:43:08.001 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-21T01:43:08.002 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2022-07-21T01:43:08.002 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2022-07-21T01:43:08.002 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2022-07-21T01:43:08.002 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2022-07-21T01:43:08.002 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2022-07-21T01:43:08.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.003 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2022-07-21T01:43:08.003 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2022-07-21T01:43:08.003 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2022-07-21T01:43:08.004 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2022-07-21T01:43:08.004 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2022-07-21T01:43:08.004 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2022-07-21T01:43:08.004 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2022-07-21T01:43:08.004 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2022-07-21T01:43:08.005 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2022-07-21T01:43:08.005 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2022-07-21T01:43:08.005 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2022-07-21T01:43:08.005 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2022-07-21T01:43:08.005 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2022-07-21T01:43:08.006 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs": { 2022-07-21T01:43:08.006 INFO:teuthology.orchestra.run.smithi003.stdout: "data": "cephfs" 2022-07-21T01:43:08.006 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.006 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.007 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.007 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:43:08.007 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2022-07-21T01:43:08.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.007 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:43:08.008 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "098a3930-1b74-4dbc-980c-5137ac329c44", 2022-07-21T01:43:08.008 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:43:08.008 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:43:08.008 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:43:08.009 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:43:08.009 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:43:08.010 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:43:08.010 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:43:08.010 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 159, 2022-07-21T01:43:08.010 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:43:08.010 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:43:08.011 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:43:08.011 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.011 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.012 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2022-07-21T01:43:08.012 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:43:08.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.012 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.013 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2022-07-21T01:43:08.013 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:43:08.013 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.014 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.014 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:43:08.014 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.015 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.015 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2022-07-21T01:43:08.015 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:43:08.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.016 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.016 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2022-07-21T01:43:08.016 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:43:08.016 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.017 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.017 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:43:08.017 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.018 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.018 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6832", 2022-07-21T01:43:08.018 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:43:08.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.019 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.019 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6833", 2022-07-21T01:43:08.019 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:43:08.020 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.020 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.020 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:43:08.020 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.021 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.021 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2022-07-21T01:43:08.021 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:43:08.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.022 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.022 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2022-07-21T01:43:08.022 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22896 2022-07-21T01:43:08.023 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.023 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.023 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6816/22896", 2022-07-21T01:43:08.024 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6822/22896", 2022-07-21T01:43:08.024 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6833/22896", 2022-07-21T01:43:08.024 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6830/22896", 2022-07-21T01:43:08.024 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:43:08.024 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:43:08.025 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:43:08.025 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.026 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:43:08.026 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ec4680af-1feb-47f3-949c-315962389518", 2022-07-21T01:43:08.026 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:43:08.026 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:43:08.026 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:43:08.027 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:43:08.027 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:43:08.027 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:43:08.027 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:43:08.028 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 158, 2022-07-21T01:43:08.028 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:43:08.028 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:43:08.028 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:43:08.028 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.029 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2022-07-21T01:43:08.029 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:43:08.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.030 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.030 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2022-07-21T01:43:08.030 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:43:08.031 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.031 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.031 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:43:08.032 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.032 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.032 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2022-07-21T01:43:08.032 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:43:08.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.033 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.033 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2022-07-21T01:43:08.033 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:43:08.034 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.034 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.035 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:43:08.035 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.035 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.036 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2022-07-21T01:43:08.036 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:43:08.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.037 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.037 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2022-07-21T01:43:08.037 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:43:08.037 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.037 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.038 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:43:08.038 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.039 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.039 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2022-07-21T01:43:08.039 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:43:08.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.040 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.040 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2022-07-21T01:43:08.040 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22890 2022-07-21T01:43:08.040 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.041 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.041 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6811/22890", 2022-07-21T01:43:08.041 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6815/22890", 2022-07-21T01:43:08.041 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6826/22890", 2022-07-21T01:43:08.042 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/22890", 2022-07-21T01:43:08.042 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:43:08.042 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:43:08.042 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:43:08.043 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.043 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:43:08.043 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e321c85f-57d4-4b3d-92c6-18843b5f43aa", 2022-07-21T01:43:08.044 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:43:08.044 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:43:08.044 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:43:08.044 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:43:08.045 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:43:08.045 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:43:08.045 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:43:08.045 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 159, 2022-07-21T01:43:08.045 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:43:08.046 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:43:08.046 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:43:08.046 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.047 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.047 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2022-07-21T01:43:08.047 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:43:08.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.048 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.048 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2022-07-21T01:43:08.048 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:43:08.048 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.049 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.049 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:43:08.049 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.050 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.050 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2022-07-21T01:43:08.050 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:43:08.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.051 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.051 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2022-07-21T01:43:08.051 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:43:08.052 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.052 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.052 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:43:08.052 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.053 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.053 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2022-07-21T01:43:08.053 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:43:08.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.054 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.054 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2022-07-21T01:43:08.054 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:43:08.055 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.055 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.055 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:43:08.055 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.056 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.056 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2022-07-21T01:43:08.056 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:43:08.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.057 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.057 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2022-07-21T01:43:08.057 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22877 2022-07-21T01:43:08.058 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.058 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.058 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6803/22877", 2022-07-21T01:43:08.059 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6805/22877", 2022-07-21T01:43:08.059 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6809/22877", 2022-07-21T01:43:08.059 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/22877", 2022-07-21T01:43:08.059 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:43:08.059 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:43:08.060 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:43:08.060 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.061 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:43:08.061 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7b3e9c1c-2503-46ca-871b-61bd90000981", 2022-07-21T01:43:08.061 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:43:08.061 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:43:08.062 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:43:08.062 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:43:08.062 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:43:08.062 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:43:08.063 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 15, 2022-07-21T01:43:08.063 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 159, 2022-07-21T01:43:08.063 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:43:08.063 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:43:08.063 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:43:08.064 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.064 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.064 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2022-07-21T01:43:08.065 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:43:08.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.065 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.066 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2022-07-21T01:43:08.066 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:43:08.066 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.066 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.067 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:43:08.067 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.067 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.068 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2022-07-21T01:43:08.068 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:43:08.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.069 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.069 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2022-07-21T01:43:08.069 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:43:08.069 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.069 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.070 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:43:08.070 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.071 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.071 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2022-07-21T01:43:08.071 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:43:08.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.072 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.072 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2022-07-21T01:43:08.073 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:43:08.073 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.073 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.074 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:43:08.074 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.074 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.074 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.075 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2022-07-21T01:43:08.075 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:43:08.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.076 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.076 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2022-07-21T01:43:08.076 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 22884 2022-07-21T01:43:08.076 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.077 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.077 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/22884", 2022-07-21T01:43:08.077 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6823/22884", 2022-07-21T01:43:08.077 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/22884", 2022-07-21T01:43:08.078 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6828/22884", 2022-07-21T01:43:08.078 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:43:08.078 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:43:08.078 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:43:08.079 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.079 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:43:08.080 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "86ac80f1-2bd0-43c8-8335-a06fe0ac9606", 2022-07-21T01:43:08.080 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:43:08.080 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:43:08.080 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:43:08.081 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:43:08.081 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:43:08.081 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:43:08.081 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:43:08.081 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 158, 2022-07-21T01:43:08.082 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:43:08.082 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:43:08.082 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:43:08.082 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.083 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.083 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6808", 2022-07-21T01:43:08.083 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:43:08.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.084 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.084 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6809", 2022-07-21T01:43:08.084 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:43:08.085 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.085 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.085 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:43:08.086 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.086 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.086 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6810", 2022-07-21T01:43:08.087 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:43:08.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.087 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.087 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6811", 2022-07-21T01:43:08.088 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:43:08.088 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.088 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.089 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:43:08.089 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.089 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.089 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6814", 2022-07-21T01:43:08.090 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:43:08.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.090 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.091 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6815", 2022-07-21T01:43:08.091 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:43:08.091 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.091 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.092 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:43:08.092 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.093 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.093 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6812", 2022-07-21T01:43:08.093 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:43:08.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.094 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.094 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6813", 2022-07-21T01:43:08.095 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23339 2022-07-21T01:43:08.095 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.096 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.097 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6809/23339", 2022-07-21T01:43:08.097 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6811/23339", 2022-07-21T01:43:08.097 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6815/23339", 2022-07-21T01:43:08.098 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6813/23339", 2022-07-21T01:43:08.098 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:43:08.099 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:43:08.099 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:43:08.100 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.101 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:43:08.102 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "25f35213-a67d-47e3-ac21-0ec2654b7e56", 2022-07-21T01:43:08.102 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:43:08.102 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:43:08.103 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:43:08.103 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:43:08.104 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:43:08.104 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:43:08.105 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:43:08.105 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 159, 2022-07-21T01:43:08.106 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:43:08.106 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:43:08.107 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:43:08.107 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.108 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.108 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6800", 2022-07-21T01:43:08.109 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:43:08.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.110 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.111 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6801", 2022-07-21T01:43:08.111 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:43:08.112 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.112 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.113 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:43:08.114 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.115 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.115 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6802", 2022-07-21T01:43:08.115 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:43:08.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.117 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.117 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6803", 2022-07-21T01:43:08.118 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:43:08.118 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.119 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.119 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:43:08.120 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.121 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.121 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6806", 2022-07-21T01:43:08.122 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:43:08.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.123 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.124 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6807", 2022-07-21T01:43:08.124 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:43:08.124 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.125 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.125 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:43:08.126 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.126 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.126 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6804", 2022-07-21T01:43:08.127 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:43:08.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.127 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.127 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6805", 2022-07-21T01:43:08.128 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23340 2022-07-21T01:43:08.128 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.128 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.128 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6801/23340", 2022-07-21T01:43:08.129 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6803/23340", 2022-07-21T01:43:08.129 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6807/23340", 2022-07-21T01:43:08.129 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6805/23340", 2022-07-21T01:43:08.129 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:43:08.130 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:43:08.130 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:43:08.130 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.131 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:43:08.131 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2", 2022-07-21T01:43:08.131 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:43:08.131 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:43:08.132 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:43:08.132 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:43:08.132 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:43:08.132 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:43:08.133 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:43:08.133 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 159, 2022-07-21T01:43:08.133 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:43:08.133 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:43:08.133 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:43:08.134 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.134 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.134 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6824", 2022-07-21T01:43:08.135 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:43:08.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.135 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.135 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6825", 2022-07-21T01:43:08.136 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:43:08.136 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.136 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.136 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:43:08.137 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.137 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.137 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6826", 2022-07-21T01:43:08.138 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:43:08.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.138 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.138 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6827", 2022-07-21T01:43:08.139 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:43:08.139 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.139 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.140 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:43:08.140 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.140 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.140 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6830", 2022-07-21T01:43:08.141 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:43:08.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.141 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.141 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6831", 2022-07-21T01:43:08.142 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:43:08.142 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.142 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.143 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:43:08.143 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.143 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.143 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6828", 2022-07-21T01:43:08.144 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:43:08.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.144 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.145 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6829", 2022-07-21T01:43:08.145 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23342 2022-07-21T01:43:08.145 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.145 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.146 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6825/23342", 2022-07-21T01:43:08.146 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6827/23342", 2022-07-21T01:43:08.146 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6831/23342", 2022-07-21T01:43:08.146 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6829/23342", 2022-07-21T01:43:08.147 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:43:08.147 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:43:08.147 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:43:08.147 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.148 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:43:08.148 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f0260c49-f670-47ef-b037-b5d9ace42688", 2022-07-21T01:43:08.149 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2022-07-21T01:43:08.149 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2022-07-21T01:43:08.149 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2022-07-21T01:43:08.149 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2022-07-21T01:43:08.149 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2022-07-21T01:43:08.150 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2022-07-21T01:43:08.150 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 14, 2022-07-21T01:43:08.150 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 159, 2022-07-21T01:43:08.150 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2022-07-21T01:43:08.150 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2022-07-21T01:43:08.151 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2022-07-21T01:43:08.151 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.151 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.152 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6816", 2022-07-21T01:43:08.152 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:43:08.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.152 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.153 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6817", 2022-07-21T01:43:08.153 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:43:08.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.153 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.154 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2022-07-21T01:43:08.154 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.154 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.155 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6818", 2022-07-21T01:43:08.155 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:43:08.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.156 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.156 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6819", 2022-07-21T01:43:08.156 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:43:08.156 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.156 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.157 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2022-07-21T01:43:08.157 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.157 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.158 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6822", 2022-07-21T01:43:08.158 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:43:08.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.159 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.159 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6823", 2022-07-21T01:43:08.159 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:43:08.159 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.159 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.160 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2022-07-21T01:43:08.160 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2022-07-21T01:43:08.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2022-07-21T01:43:08.161 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6820", 2022-07-21T01:43:08.161 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:43:08.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.162 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2022-07-21T01:43:08.162 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.143:6821", 2022-07-21T01:43:08.162 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 23341 2022-07-21T01:43:08.162 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.163 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.163 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.143:6817/23341", 2022-07-21T01:43:08.163 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.143:6819/23341", 2022-07-21T01:43:08.164 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.143:6823/23341", 2022-07-21T01:43:08.164 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.143:6821/23341", 2022-07-21T01:43:08.164 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2022-07-21T01:43:08.164 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2022-07-21T01:43:08.165 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2022-07-21T01:43:08.165 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2022-07-21T01:43:08.165 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.165 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:43:08.165 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2022-07-21T01:43:08.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.166 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2022-07-21T01:43:08.166 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:43:08.166 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:43:08.167 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:43:08.167 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:43:08.167 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:43:08.167 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.616595+0000", 2022-07-21T01:43:08.167 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:43:08.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.168 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2022-07-21T01:43:08.168 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:43:08.169 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:43:08.169 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:43:08.169 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:43:08.169 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:43:08.169 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.374044+0000", 2022-07-21T01:43:08.170 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:43:08.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.170 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2022-07-21T01:43:08.171 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:43:08.171 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:43:08.171 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:43:08.171 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:43:08.171 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:43:08.172 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.657733+0000", 2022-07-21T01:43:08.172 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:43:08.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.173 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2022-07-21T01:43:08.173 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:43:08.173 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:43:08.173 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:43:08.173 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:43:08.174 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:43:08.174 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.654519+0000", 2022-07-21T01:43:08.174 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:43:08.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2022-07-21T01:43:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:43:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:43:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:43:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:43:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:43:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.426744+0000", 2022-07-21T01:43:08.176 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:43:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2022-07-21T01:43:08.177 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:43:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:43:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:43:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:43:08.178 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:43:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.113656+0000", 2022-07-21T01:43:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:43:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2022-07-21T01:43:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:43:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:43:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:43:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:43:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:43:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.413566+0000", 2022-07-21T01:43:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:43:08.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2022-07-21T01:43:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2022-07-21T01:43:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2022-07-21T01:43:08.182 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2022-07-21T01:43:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2022-07-21T01:43:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540138292840890367, 2022-07-21T01:43:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2022-07-21T01:43:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2022-07-21T01:33:12.398720+0000", 2022-07-21T01:43:08.183 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2022-07-21T01:43:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2022-07-21T01:43:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2022-07-21T01:43:08.184 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2022-07-21T01:43:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2022-07-21T01:43:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2022-07-21T01:43:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: "blacklist": {}, 2022-07-21T01:43:08.185 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2022-07-21T01:43:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_data": { 2022-07-21T01:43:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2022-07-21T01:43:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:43:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2022-07-21T01:43:08.186 INFO:teuthology.orchestra.run.smithi003.stdout: "jerasure-per-chunk-alignment": "false", 2022-07-21T01:43:08.187 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:43:08.187 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "2", 2022-07-21T01:43:08.187 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:43:08.187 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:43:08.188 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van", 2022-07-21T01:43:08.188 INFO:teuthology.orchestra.run.smithi003.stdout: "w": "8" 2022-07-21T01:43:08.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2022-07-21T01:43:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2022-07-21T01:43:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2022-07-21T01:43:08.189 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2022-07-21T01:43:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2022-07-21T01:43:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "ruleset-failure-domain": "osd", 2022-07-21T01:43:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2022-07-21T01:43:08.190 INFO:teuthology.orchestra.run.smithi003.stdout: } 2022-07-21T01:43:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2022-07-21T01:43:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2022-07-21T01:43:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2022-07-21T01:43:08.191 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2022-07-21T01:43:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2022-07-21T01:43:08.192 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {} 2022-07-21T01:43:08.192 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T01:43:08.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c4053a70 msgr2=0x7f88c4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c4053a70 0x7f88c4055f30 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f88c801c000 tx=0x7f88c8019000).stop 2022-07-21T01:43:08.194 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e00651a0 msgr2=0x7f88e0159ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e00651a0 0x7f88e0159ec0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f88d000b000 tx=0x7f88d0006df0).stop 2022-07-21T01:43:08.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 shutdown_connections 2022-07-21T01:43:08.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88e00651a0 0x7f88e0159ec0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.195 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f88c4053a70 0x7f88c4055f30 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88e00b4290 0x7f88e0064cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 --2- 172.21.15.3:0/2702163702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88e00a8c50 0x7f88e005c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 >> 172.21.15.3:0/2702163702 conn(0x7f88e0060310 msgr2=0x7f88e0149cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:08.196 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 shutdown_connections 2022-07-21T01:43:08.197 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:07.907+0000 7f88df59e700 1 -- 172.21.15.3:0/2702163702 wait complete. 2022-07-21T01:43:08.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:43:08.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.303+0000 7fedf359e700 1 Processor -- start 2022-07-21T01:43:08.308 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.303+0000 7fedf359e700 1 -- start start 2022-07-21T01:43:08.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf4064bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.310 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fedf41470d0 0x7fedf41403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf40b4f60 con 0x7fedf40647d0 2022-07-21T01:43:08.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf40b4a60 con 0x7fedf41470d0 2022-07-21T01:43:08.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40b4cc0 con 0x7fedf40b38a0 2022-07-21T01:43:08.311 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf1d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf4064bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf2d9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fedf41470d0 0x7fedf41403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf1d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf4064bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:59226/0 (socket says 172.21.15.3:59226) 2022-07-21T01:43:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf1d9b700 1 -- 172.21.15.3:0/905860003 learned_addr learned my addr 172.21.15.3:0/905860003 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf2d9d700 1 -- 172.21.15.3:0/905860003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 msgr2=0x7fedf4064bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf2d9d700 1 --2- 172.21.15.3:0/905860003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf4064bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf2d9d700 1 -- 172.21.15.3:0/905860003 >> v1:172.21.15.143:6789/0 conn(0x7fedf40647d0 legacy=0x7fedf41436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf2d9d700 1 -- 172.21.15.3:0/905860003 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf4143250 con 0x7fedf41470d0 2022-07-21T01:43:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf1d9b700 1 --2- 172.21.15.3:0/905860003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf4064bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T01:43:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf2d9d700 1 --2- 172.21.15.3:0/905860003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedf41470d0 0x7fedf41403f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fedec0032f0 tx=0x7fedec0091d0).ready entity=mon.1 client_cookie=e6f5e7369869784c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:08.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf159a700 1 -- 172.21.15.3:0/905860003 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedec00b6c0 con 0x7fedf41470d0 2022-07-21T01:43:08.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf159a700 1 -- 172.21.15.3:0/905860003 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec00b820 con 0x7fedf41470d0 2022-07-21T01:43:08.314 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf159a700 1 -- 172.21.15.3:0/905860003 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedec00b990 con 0x7fedf41470d0 2022-07-21T01:43:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 -- 172.21.15.3:0/905860003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedf41470d0 msgr2=0x7fedf41403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 --2- 172.21.15.3:0/905860003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedf41470d0 0x7fedf41403f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fedec0032f0 tx=0x7fedec0091d0).stop 2022-07-21T01:43:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 -- 172.21.15.3:0/905860003 shutdown_connections 2022-07-21T01:43:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 --2- 172.21.15.3:0/905860003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedf41470d0 0x7fedf41403f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 --2- 172.21.15.3:0/905860003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf4064bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 -- 172.21.15.3:0/905860003 >> 172.21.15.3:0/905860003 conn(0x7fedf405f840 msgr2=0x7fedf405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 -- 172.21.15.3:0/905860003 shutdown_connections 2022-07-21T01:43:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.307+0000 7fedf359e700 1 -- 172.21.15.3:0/905860003 wait complete. 2022-07-21T01:43:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 Processor -- start 2022-07-21T01:43:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 -- start start 2022-07-21T01:43:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedf40647d0 0x7fedf40b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf413e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedf41470d0 0x7fedf413ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40b6970 con 0x7fedf40647d0 2022-07-21T01:43:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf4153380 con 0x7fedf41470d0 2022-07-21T01:43:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf4153620 con 0x7fedf40b38a0 2022-07-21T01:43:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedf40647d0 0x7fedf40b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf1d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf413e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf2d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedf41470d0 0x7fedf413ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedf40647d0 0x7fedf40b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52318/0 (socket says 172.21.15.3:52318) 2022-07-21T01:43:08.323 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf1d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf413e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:59232/0 (socket says 172.21.15.3:59232) 2022-07-21T01:43:08.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf259c700 1 -- 172.21.15.3:0/351312866 learned_addr learned my addr 172.21.15.3:0/351312866 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:08.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf1d9b700 1 -- 172.21.15.3:0/351312866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedf41470d0 msgr2=0x7fedf413ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf1d9b700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedf41470d0 0x7fedf413ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf1d9b700 1 -- 172.21.15.3:0/351312866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedf40647d0 msgr2=0x7fedf40b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.324 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf1d9b700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedf40647d0 0x7fedf40b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf1d9b700 1 -- 172.21.15.3:0/351312866 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede400b1e0 con 0x7fedf40b38a0 2022-07-21T01:43:08.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf1d9b700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf413e890 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fedf40a4190 tx=0x7fede4006eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:08.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fede37fe700 1 -- 172.21.15.3:0/351312866 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fede4012600 con 0x7fedf40b38a0 2022-07-21T01:43:08.325 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fede37fe700 1 -- 172.21.15.3:0/351312866 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede4012760 con 0x7fedf40b38a0 2022-07-21T01:43:08.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedec009470 con 0x7fedf40b38a0 2022-07-21T01:43:08.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fede37fe700 1 -- 172.21.15.3:0/351312866 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fede4012ac0 con 0x7fedf40b38a0 2022-07-21T01:43:08.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.311+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf413f270 con 0x7fedf40b38a0 2022-07-21T01:43:08.326 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.315+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf4147560 con 0x7fedf40b38a0 2022-07-21T01:43:08.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.315+0000 7fede37fe700 1 -- 172.21.15.3:0/351312866 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fede4022070 con 0x7fedf40b38a0 2022-07-21T01:43:08.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.315+0000 7fede37fe700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedd00539e0 0x7fedd0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.315+0000 7fede37fe700 1 -- 172.21.15.3:0/351312866 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fede406c4f0 con 0x7fedf40b38a0 2022-07-21T01:43:08.327 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.315+0000 7fedf259c700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedd00539e0 0x7fedd0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.315+0000 7fedf259c700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedd00539e0 0x7fedd0055ea0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fede800d000 tx=0x7fede8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:08.328 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.323+0000 7fede37fe700 1 -- 172.21.15.3:0/351312866 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fede4043210 con 0x7fedf40b38a0 2022-07-21T01:43:08.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.487+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedf40574e0 con 0x7fedf40b38a0 2022-07-21T01:43:08.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.487+0000 7fede37fe700 1 -- 172.21.15.3:0/351312866 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2489 (secure 0 0 0) 0x7fede4046830 con 0x7fedf40b38a0 2022-07-21T01:43:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:43:08.490 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":81,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/781340149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":781340149},{"type":"v1","addr":"172.21.15.143:6833","nonce":781340149}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79},{"gid":8712,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3829565046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3829565046},{"type":"v1","addr":"172.21.15.3:6835","nonce":3829565046}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:58.238508+0000","modified":"2022-07-21T01:42:59.241585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8727},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8727":{"gid":8727,"name":"c","rank":0,"incarnation":80,"state":"up:active","state_seq":3,"addr":"172.21.15.3:6837/1787668325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1787668325},{"type":"v1","addr":"172.21.15.3:6837","nonce":1787668325}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-21T01:43:08.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedd00539e0 msgr2=0x7fedd0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.493 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedd00539e0 0x7fedd0055ea0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fede800d000 tx=0x7fede8009040).stop 2022-07-21T01:43:08.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 msgr2=0x7fedf413e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf413e890 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fedf40a4190 tx=0x7fede4006eb0).stop 2022-07-21T01:43:08.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 shutdown_connections 2022-07-21T01:43:08.494 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fedd00539e0 0x7fedd0055ea0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedf41470d0 0x7fedf413ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedf40b38a0 0x7fedf413e890 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 --2- 172.21.15.3:0/351312866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedf40647d0 0x7fedf40b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.495 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 >> 172.21.15.3:0/351312866 conn(0x7fedf405f840 msgr2=0x7fedf405cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 shutdown_connections 2022-07-21T01:43:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.491+0000 7fedf359e700 1 -- 172.21.15.3:0/351312866 wait complete. 2022-07-21T01:43:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 81 2022-07-21T01:43:08.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-21T01:43:08.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fe2f7700 1 Processor -- start 2022-07-21T01:43:08.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fe2f7700 1 -- start start 2022-07-21T01:43:08.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fe2f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 0x7fe4f80af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fe2f7700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f80b5a10 con 0x7fe4f8147730 2022-07-21T01:43:08.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fe2f7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f80ac990 con 0x7fe4f80b07a0 2022-07-21T01:43:08.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fe2f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f8145470 con 0x7fe4f80b5630 2022-07-21T01:43:08.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fdaf6700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe4f8147730 0x7fe4f80acdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60608/0 (socket says 172.21.15.3:60608) 2022-07-21T01:43:08.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fdaf6700 1 -- 172.21.15.3:0/715593226 learned_addr learned my addr 172.21.15.3:0/715593226 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:08.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fd2f5700 1 --2- 172.21.15.3:0/715593226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 0x7fe4f80af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4effff700 1 -- 172.21.15.3:0/715593226 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296833125 0 0) 0x7fe4f80b5a10 con 0x7fe4f8147730 2022-07-21T01:43:08.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4effff700 1 -- 172.21.15.3:0/715593226 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4dc003260 con 0x7fe4f8147730 2022-07-21T01:43:08.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4effff700 1 -- 172.21.15.3:0/715593226 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134643879 0 0) 0x7fe4f80ac990 con 0x7fe4f80b07a0 2022-07-21T01:43:08.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fd2f5700 1 -- 172.21.15.3:0/715593226 >> v1:172.21.15.143:6789/0 conn(0x7fe4f80b07a0 legacy=0x7fe4f8067730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fd2f5700 1 -- 172.21.15.3:0/715593226 >> v1:172.21.15.3:6789/0 conn(0x7fe4f8147730 legacy=0x7fe4f80acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.615+0000 7fe4fd2f5700 1 -- 172.21.15.3:0/715593226 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f80b1bc0 con 0x7fe4f80b5630 2022-07-21T01:43:08.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4fd2f5700 1 --2- 172.21.15.3:0/715593226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 0x7fe4f80af6e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fe4e8002f70 tx=0x7fe4e8006df0).ready entity=mon.2 client_cookie=74c4ee4c06df85fd server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:08.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4effff700 1 -- 172.21.15.3:0/715593226 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e800d570 con 0x7fe4f80b5630 2022-07-21T01:43:08.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4effff700 1 -- 172.21.15.3:0/715593226 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e8000e90 con 0x7fe4f80b5630 2022-07-21T01:43:08.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4effff700 1 -- 172.21.15.3:0/715593226 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e800d8c0 con 0x7fe4f80b5630 2022-07-21T01:43:08.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/715593226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 msgr2=0x7fe4f80af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4fe2f7700 1 --2- 172.21.15.3:0/715593226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 0x7fe4f80af6e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fe4e8002f70 tx=0x7fe4e8006df0).stop 2022-07-21T01:43:08.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/715593226 shutdown_connections 2022-07-21T01:43:08.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4fe2f7700 1 --2- 172.21.15.3:0/715593226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 0x7fe4f80af6e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/715593226 >> 172.21.15.3:0/715593226 conn(0x7fe4f80626f0 msgr2=0x7fe4f8062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:08.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/715593226 shutdown_connections 2022-07-21T01:43:08.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.619+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/715593226 wait complete. 2022-07-21T01:43:08.632 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 Processor -- start 2022-07-21T01:43:08.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 -- start start 2022-07-21T01:43:08.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80b07a0 0x7fe4f8144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 0x7fe4f8064b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4f8147730 0x7fe4f8156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f80ae180 con 0x7fe4f8147730 2022-07-21T01:43:08.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f8156b30 con 0x7fe4f80b07a0 2022-07-21T01:43:08.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f8064070 con 0x7fe4f80b5630 2022-07-21T01:43:08.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fdaf6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4f8147730 0x7fe4f8156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fd2f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80b07a0 0x7fe4f8144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fdaf6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4f8147730 0x7fe4f8156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52334/0 (socket says 172.21.15.3:52334) 2022-07-21T01:43:08.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fdaf6700 1 -- 172.21.15.3:0/2870558376 learned_addr learned my addr 172.21.15.3:0/2870558376 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:08.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fcaf4700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 0x7fe4f8064b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fdaf6700 1 -- 172.21.15.3:0/2870558376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 msgr2=0x7fe4f8064b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fdaf6700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 0x7fe4f8064b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fdaf6700 1 -- 172.21.15.3:0/2870558376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80b07a0 msgr2=0x7fe4f8144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:08.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fdaf6700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80b07a0 0x7fe4f8144af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:08.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fdaf6700 1 -- 172.21.15.3:0/2870558376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e800b180 con 0x7fe4f8147730 2022-07-21T01:43:08.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fdaf6700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4f8147730 0x7fe4f8156f50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7fe4f8063910 tx=0x7fe4f0007da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:08.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4ee7fc700 1 -- 172.21.15.3:0/2870558376 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4f0005ba0 con 0x7fe4f8147730 2022-07-21T01:43:08.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4ee7fc700 1 -- 172.21.15.3:0/2870558376 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f0005d00 con 0x7fe4f8147730 2022-07-21T01:43:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f80a86f0 con 0x7fe4f8147730 2022-07-21T01:43:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4ee7fc700 1 -- 172.21.15.3:0/2870558376 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4f00072c0 con 0x7fe4f8147730 2022-07-21T01:43:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.623+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f80642a0 con 0x7fe4f8147730 2022-07-21T01:43:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.627+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f8063690 con 0x7fe4f8147730 2022-07-21T01:43:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.627+0000 7fe4ee7fc700 1 -- 172.21.15.3:0/2870558376 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe4f0010070 con 0x7fe4f8147730 2022-07-21T01:43:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.627+0000 7fe4ee7fc700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4e0053a10 0x7fe4e0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.627+0000 7fe4ee7fc700 1 -- 172.21.15.3:0/2870558376 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6420+0+0 (secure 0 0 0) 0x7fe4f006b790 con 0x7fe4f8147730 2022-07-21T01:43:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.627+0000 7fe4fd2f5700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4e0053a10 0x7fe4e0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.627+0000 7fe4fd2f5700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4e0053a10 0x7fe4e0055ed0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fe4e8018000 tx=0x7fe4e8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.635+0000 7fe4ee7fc700 1 -- 172.21.15.3:0/2870558376 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4f00424f0 con 0x7fe4f8147730 2022-07-21T01:43:08.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:08.799+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe4f80b1680 con 0x7fe4f8147730 2022-07-21T01:43:09.280 INFO:tasks.ceph.mon.a.smithi003.stderr:2022-07-21T01:43:09.275+0000 7f6e58cb3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-21T01:43:09.293 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.291+0000 7fe4ee7fc700 1 -- 172.21.15.3:0/2870558376 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v82) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe4f0045b10 con 0x7fe4f8147730 2022-07-21T01:43:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4e0053a10 msgr2=0x7fe4e0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4e0053a10 0x7fe4e0055ed0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fe4e8018000 tx=0x7fe4e8014000).stop 2022-07-21T01:43:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4f8147730 msgr2=0x7fe4f8156f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4f8147730 0x7fe4f8156f50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7fe4f8063910 tx=0x7fe4f0007da0).stop 2022-07-21T01:43:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 shutdown_connections 2022-07-21T01:43:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4f8147730 0x7fe4f8156f50 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80b5630 0x7fe4f8064b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe4e0053a10 0x7fe4e0055ed0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 --2- 172.21.15.3:0/2870558376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80b07a0 0x7fe4f8144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 >> 172.21.15.3:0/2870558376 conn(0x7fe4f80626f0 msgr2=0x7fe4f8143b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 shutdown_connections 2022-07-21T01:43:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.295+0000 7fe4fe2f7700 1 -- 172.21.15.3:0/2870558376 wait complete. 2022-07-21T01:43:09.302 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-21T01:43:09.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-21T01:43:09.318 INFO:tasks.ceph.mds.c.smithi003.stdout:starting mds.c at 2022-07-21T01:43:09.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5dfff700 1 Processor -- start 2022-07-21T01:43:09.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5dfff700 1 -- start start 2022-07-21T01:43:09.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5dfff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fea580b3070 0x7fea58065720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5dfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fea5814b040 0x7fea5814b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5dfff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea580b2c20 con 0x7fea580b7840 2022-07-21T01:43:09.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5dfff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea58061830 con 0x7fea580b3070 2022-07-21T01:43:09.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5dfff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea58065c10 con 0x7fea5814b040 2022-07-21T01:43:09.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea57fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fea580b3070 0x7fea58065720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5cffd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fea580b7840 0x7fea58061120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60630/0 (socket says 172.21.15.3:60630) 2022-07-21T01:43:09.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5cffd700 1 -- 172.21.15.3:0/3958348322 learned_addr learned my addr 172.21.15.3:0/3958348322 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:09.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea5d7fe700 1 --2- 172.21.15.3:0/3958348322 >> v2:172.21.15.143:3300/0 conn(0x7fea5814b040 0x7fea5814b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea577fe700 1 -- 172.21.15.3:0/3958348322 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651545803 0 0) 0x7fea580b2c20 con 0x7fea580b7840 2022-07-21T01:43:09.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea577fe700 1 -- 172.21.15.3:0/3958348322 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea44003260 con 0x7fea580b7840 2022-07-21T01:43:09.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea57fff700 1 -- 172.21.15.3:0/3958348322 >> v2:172.21.15.143:3300/0 conn(0x7fea5814b040 msgr2=0x7fea5814b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:09.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea57fff700 1 --2- 172.21.15.3:0/3958348322 >> v2:172.21.15.143:3300/0 conn(0x7fea5814b040 0x7fea5814b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea57fff700 1 -- 172.21.15.3:0/3958348322 >> v1:172.21.15.3:6789/0 conn(0x7fea580b7840 legacy=0x7fea58061120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea57fff700 1 -- 172.21.15.3:0/3958348322 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580b7c20 con 0x7fea580b3070 2022-07-21T01:43:09.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea57fff700 1 --2- 172.21.15.3:0/3958348322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 0x7fea58065720 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7fea4c002f70 tx=0x7fea4c006df0).ready entity=mon.0 client_cookie=66681560dd0840ae server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:09.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea577fe700 1 -- 172.21.15.3:0/3958348322 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea4c00d570 con 0x7fea580b3070 2022-07-21T01:43:09.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea577fe700 1 -- 172.21.15.3:0/3958348322 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c000e90 con 0x7fea580b3070 2022-07-21T01:43:09.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.415+0000 7fea577fe700 1 -- 172.21.15.3:0/3958348322 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea4c00d8c0 con 0x7fea580b3070 2022-07-21T01:43:09.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 -- 172.21.15.3:0/3958348322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 msgr2=0x7fea58065720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 --2- 172.21.15.3:0/3958348322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 0x7fea58065720 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7fea4c002f70 tx=0x7fea4c006df0).stop 2022-07-21T01:43:09.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 -- 172.21.15.3:0/3958348322 shutdown_connections 2022-07-21T01:43:09.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 --2- 172.21.15.3:0/3958348322 >> v2:172.21.15.143:3300/0 conn(0x7fea5814b040 0x7fea5814b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 --2- 172.21.15.3:0/3958348322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 0x7fea58065720 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 -- 172.21.15.3:0/3958348322 >> 172.21.15.3:0/3958348322 conn(0x7fea5805f840 msgr2=0x7fea5805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:09.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 -- 172.21.15.3:0/3958348322 shutdown_connections 2022-07-21T01:43:09.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 -- 172.21.15.3:0/3958348322 wait complete. 2022-07-21T01:43:09.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 Processor -- start 2022-07-21T01:43:09.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 -- start start 2022-07-21T01:43:09.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 0x7fea580aae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea580b7840 0x7fea5814a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea5814b040 0x7fea58146300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea580ae070 con 0x7fea580b3070 2022-07-21T01:43:09.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea580aeb20 con 0x7fea5814b040 2022-07-21T01:43:09.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5dfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea580aedc0 con 0x7fea580b7840 2022-07-21T01:43:09.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5cffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 0x7fea580aae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5cffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 0x7fea580aae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52356/0 (socket says 172.21.15.3:52356) 2022-07-21T01:43:09.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.419+0000 7fea5cffd700 1 -- 172.21.15.3:0/4114024354 learned_addr learned my addr 172.21.15.3:0/4114024354 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:09.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea57fff700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea580b7840 0x7fea5814a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5d7fe700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea5814b040 0x7fea58146300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5cffd700 1 -- 172.21.15.3:0/4114024354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea580b7840 msgr2=0x7fea5814a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5cffd700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea580b7840 0x7fea5814a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5cffd700 1 -- 172.21.15.3:0/4114024354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea5814b040 msgr2=0x7fea58146300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5cffd700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea5814b040 0x7fea58146300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5cffd700 1 -- 172.21.15.3:0/4114024354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c00b180 con 0x7fea580b3070 2022-07-21T01:43:09.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5cffd700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 0x7fea580aae30 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7fea48001290 tx=0x7fea48002870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:09.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea55ffb700 1 -- 172.21.15.3:0/4114024354 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea480070d0 con 0x7fea580b3070 2022-07-21T01:43:09.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea55ffb700 1 -- 172.21.15.3:0/4114024354 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea48007230 con 0x7fea580b3070 2022-07-21T01:43:09.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea580ac0e0 con 0x7fea580b3070 2022-07-21T01:43:09.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea55ffb700 1 -- 172.21.15.3:0/4114024354 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea480073a0 con 0x7fea580b3070 2022-07-21T01:43:09.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea581467f0 con 0x7fea580b3070 2022-07-21T01:43:09.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea580b2980 con 0x7fea580b3070 2022-07-21T01:43:09.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.423+0000 7fea55ffb700 1 -- 172.21.15.3:0/4114024354 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fea4801a060 con 0x7fea580b3070 2022-07-21T01:43:09.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.427+0000 7fea55ffb700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea38053b30 0x7fea38055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.427+0000 7fea55ffb700 1 -- 172.21.15.3:0/4114024354 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fea4806b440 con 0x7fea580b3070 2022-07-21T01:43:09.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.427+0000 7fea57fff700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea38053b30 0x7fea38055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.427+0000 7fea57fff700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea38053b30 0x7fea38055ff0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fea4c01a000 tx=0x7fea4c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:09.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.431+0000 7fea55ffb700 1 -- 172.21.15.3:0/4114024354 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fea480420a0 con 0x7fea580b3070 2022-07-21T01:43:09.599 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.595+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea580574e0 con 0x7fea580b3070 2022-07-21T01:43:09.600 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.595+0000 7fea55ffb700 1 -- 172.21.15.3:0/4114024354 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2132 (secure 0 0 0) 0x7fea48018020 con 0x7fea580b3070 2022-07-21T01:43:09.601 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T01:43:09.601 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":82,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7433,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/781340149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":781340149},{"type":"v1","addr":"172.21.15.143:6833","nonce":781340149}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79},{"gid":8712,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/3829565046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":3829565046},{"type":"v1","addr":"172.21.15.3:6835","nonce":3829565046}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":82,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-21T01:42:58.238508+0000","modified":"2022-07-21T01:43:09.281075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":162,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-21T01:43:09.604 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.599+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea38053b30 msgr2=0x7fea38055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.599+0000 7fea5dfff700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea38053b30 0x7fea38055ff0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fea4c01a000 tx=0x7fea4c014000).stop 2022-07-21T01:43:09.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.599+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 msgr2=0x7fea580aae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.599+0000 7fea5dfff700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 0x7fea580aae30 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7fea48001290 tx=0x7fea48002870).stop 2022-07-21T01:43:09.605 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.603+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 shutdown_connections 2022-07-21T01:43:09.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.603+0000 7fea5dfff700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fea38053b30 0x7fea38055ff0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.603+0000 7fea5dfff700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea5814b040 0x7fea58146300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.603+0000 7fea5dfff700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea580b7840 0x7fea5814a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.606 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.603+0000 7fea5dfff700 1 --2- 172.21.15.3:0/4114024354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea580b3070 0x7fea580aae30 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.603+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 >> 172.21.15.3:0/4114024354 conn(0x7fea5805f840 msgr2=0x7fea5814b480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:09.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.603+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 shutdown_connections 2022-07-21T01:43:09.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.603+0000 7fea5dfff700 1 -- 172.21.15.3:0/4114024354 wait complete. 2022-07-21T01:43:09.608 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 82 2022-07-21T01:43:09.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-21T01:43:09.731 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.727+0000 7fc4fb390700 1 Processor -- start 2022-07-21T01:43:09.732 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.727+0000 7fc4fb390700 1 -- start start 2022-07-21T01:43:09.733 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc4f40614b0 0x7fc4f4061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc4f40ad750 0x7fc4f40b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f40aee80 con 0x7fc4f41405d0 2022-07-21T01:43:09.734 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f40b70b0 con 0x7fc4f40614b0 2022-07-21T01:43:09.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f40af0b0 con 0x7fc4f40ad750 2022-07-21T01:43:09.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fab8f700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc4f41405d0 0x7fc4f41466b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60646/0 (socket says 172.21.15.3:60646) 2022-07-21T01:43:09.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fa38e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc4f40614b0 0x7fc4f4061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fab8f700 1 -- 172.21.15.3:0/1861282582 learned_addr learned my addr 172.21.15.3:0/1861282582 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:09.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4f938c700 1 -- 172.21.15.3:0/1861282582 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206225113 0 0) 0x7fc4f40aee80 con 0x7fc4f41405d0 2022-07-21T01:43:09.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4f938c700 1 -- 172.21.15.3:0/1861282582 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e0003260 con 0x7fc4f41405d0 2022-07-21T01:43:09.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fa38e700 1 -- 172.21.15.3:0/1861282582 >> v2:172.21.15.143:3300/0 conn(0x7fc4f40ad750 msgr2=0x7fc4f40b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T01:43:09.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fa38e700 1 --2- 172.21.15.3:0/1861282582 >> v2:172.21.15.143:3300/0 conn(0x7fc4f40ad750 0x7fc4f40b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:43:09.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fa38e700 1 -- 172.21.15.3:0/1861282582 >> v1:172.21.15.3:6789/0 conn(0x7fc4f41405d0 legacy=0x7fc4f41466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.736 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fa38e700 1 -- 172.21.15.3:0/1861282582 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f40af350 con 0x7fc4f40614b0 2022-07-21T01:43:09.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fa38e700 1 --2- 172.21.15.3:0/1861282582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f40614b0 0x7fc4f4061890 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7fc4f40aed30 tx=0x7fc4e40091d0).ready entity=mon.0 client_cookie=a42acf8adc96931e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:09.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4f938c700 1 -- 172.21.15.3:0/1861282582 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4e400b680 con 0x7fc4f40614b0 2022-07-21T01:43:09.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4f938c700 1 -- 172.21.15.3:0/1861282582 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e400b7e0 con 0x7fc4f40614b0 2022-07-21T01:43:09.738 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4f938c700 1 -- 172.21.15.3:0/1861282582 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4e400b990 con 0x7fc4f40614b0 2022-07-21T01:43:09.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 -- 172.21.15.3:0/1861282582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f40614b0 msgr2=0x7fc4f4061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.744 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 --2- 172.21.15.3:0/1861282582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f40614b0 0x7fc4f4061890 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7fc4f40aed30 tx=0x7fc4e40091d0).stop 2022-07-21T01:43:09.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 -- 172.21.15.3:0/1861282582 shutdown_connections 2022-07-21T01:43:09.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 --2- 172.21.15.3:0/1861282582 >> v2:172.21.15.143:3300/0 conn(0x7fc4f40ad750 0x7fc4f40b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-21T01:43:09.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 --2- 172.21.15.3:0/1861282582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f40614b0 0x7fc4f4061890 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 -- 172.21.15.3:0/1861282582 >> 172.21.15.3:0/1861282582 conn(0x7fc4f4066f70 msgr2=0x7fc4f4067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:09.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 -- 172.21.15.3:0/1861282582 shutdown_connections 2022-07-21T01:43:09.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.731+0000 7fc4fb390700 1 -- 172.21.15.3:0/1861282582 wait complete. 2022-07-21T01:43:09.747 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fb390700 1 Processor -- start 2022-07-21T01:43:09.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fb390700 1 -- start start 2022-07-21T01:43:09.748 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fb390700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f40614b0 0x7fc4f4155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fb390700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f40ad750 0x7fc4f40b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.749 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fb390700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f41405d0 0x7fc4f413fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fb390700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f4143d90 con 0x7fc4f41405d0 2022-07-21T01:43:09.750 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fb390700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f4144130 con 0x7fc4f40ad750 2022-07-21T01:43:09.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fb390700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f4156520 con 0x7fc4f40614b0 2022-07-21T01:43:09.751 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fab8f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f41405d0 0x7fc4f413fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fab8f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f41405d0 0x7fc4f413fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52374/0 (socket says 172.21.15.3:52374) 2022-07-21T01:43:09.752 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fab8f700 1 -- 172.21.15.3:0/3771957990 learned_addr learned my addr 172.21.15.3:0/3771957990 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:09.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4f9b8d700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f40ad750 0x7fc4f40b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fa38e700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f40614b0 0x7fc4f4155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.753 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fab8f700 1 -- 172.21.15.3:0/3771957990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f40614b0 msgr2=0x7fc4f4155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fab8f700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f40614b0 0x7fc4f4155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.754 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fab8f700 1 -- 172.21.15.3:0/3771957990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f40ad750 msgr2=0x7fc4f40b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:09.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fab8f700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f40ad750 0x7fc4f40b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:09.755 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fab8f700 1 -- 172.21.15.3:0/3771957990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e4009470 con 0x7fc4f41405d0 2022-07-21T01:43:09.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fab8f700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f41405d0 0x7fc4f413fa80 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7fc4f40b38e0 tx=0x7fc4ec007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:09.756 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4f37fe700 1 -- 172.21.15.3:0/3771957990 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4ec00cd90 con 0x7fc4f41405d0 2022-07-21T01:43:09.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4f37fe700 1 -- 172.21.15.3:0/3771957990 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ec0070d0 con 0x7fc4f41405d0 2022-07-21T01:43:09.757 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f41567c0 con 0x7fc4f41405d0 2022-07-21T01:43:09.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.735+0000 7fc4f37fe700 1 -- 172.21.15.3:0/3771957990 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4ec007430 con 0x7fc4f41405d0 2022-07-21T01:43:09.758 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.739+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f4156240 con 0x7fc4f41405d0 2022-07-21T01:43:09.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.739+0000 7fc4f37fe700 1 -- 172.21.15.3:0/3771957990 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc4ec01a060 con 0x7fc4f41405d0 2022-07-21T01:43:09.759 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.739+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f40af5f0 con 0x7fc4f41405d0 2022-07-21T01:43:09.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.739+0000 7fc4f37fe700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc4d4053b00 0x7fc4d4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:09.760 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.739+0000 7fc4fa38e700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc4d4053b00 0x7fc4d4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:09.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.739+0000 7fc4f37fe700 1 -- 172.21.15.3:0/3771957990 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6506+0+0 (secure 0 0 0) 0x7fc4ec06b720 con 0x7fc4f41405d0 2022-07-21T01:43:09.761 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.739+0000 7fc4fa38e700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc4d4053b00 0x7fc4d4055fc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7fc4e401b000 tx=0x7fc4e4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:09.762 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.747+0000 7fc4f37fe700 1 -- 172.21.15.3:0/3771957990 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc4ec042410 con 0x7fc4f41405d0 2022-07-21T01:43:09.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:09.907+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc4f4062290 con 0x7fc4f41405d0 2022-07-21T01:43:10.298 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.295+0000 7fc4f37fe700 1 -- 172.21.15.3:0/3771957990 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v83) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc4ec018020 con 0x7fc4f41405d0 2022-07-21T01:43:10.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc4d4053b00 msgr2=0x7fc4d4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:10.302 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc4d4053b00 0x7fc4d4055fc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7fc4e401b000 tx=0x7fc4e4016000).stop 2022-07-21T01:43:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f41405d0 msgr2=0x7fc4f413fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f41405d0 0x7fc4f413fa80 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7fc4f40b38e0 tx=0x7fc4ec007a80).stop 2022-07-21T01:43:10.303 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 shutdown_connections 2022-07-21T01:43:10.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4f41405d0 0x7fc4f413fa80 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:10.304 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc4d4053b00 0x7fc4d4055fc0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f40ad750 0x7fc4f40b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 --2- 172.21.15.3:0/3771957990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f40614b0 0x7fc4f4155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:10.305 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 >> 172.21.15.3:0/3771957990 conn(0x7fc4f4066f70 msgr2=0x7fc4f4067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:10.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 shutdown_connections 2022-07-21T01:43:10.306 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.299+0000 7fc4fb390700 1 -- 172.21.15.3:0/3771957990 wait complete. 2022-07-21T01:43:10.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-21T01:43:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.427+0000 7f62e8031700 1 Processor -- start 2022-07-21T01:43:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.427+0000 7f62e8031700 1 -- start start 2022-07-21T01:43:10.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f62e00b5d80 0x7f62e00b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:10.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e0144240 con 0x7f62e00b6710 2022-07-21T01:43:10.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e00b6fe0 con 0x7f62e00b4550 2022-07-21T01:43:10.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e0143d00 con 0x7f62e00b5d80 2022-07-21T01:43:10.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e5dcd700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f62e00b6710 0x7f62e0061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60660/0 (socket says 172.21.15.3:60660) 2022-07-21T01:43:10.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e5dcd700 1 -- 172.21.15.3:0/581061688 learned_addr learned my addr 172.21.15.3:0/581061688 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:10.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e65ce700 1 --2- 172.21.15.3:0/581061688 >> v2:172.21.15.3:3300/0 conn(0x7f62e00b5d80 0x7f62e00b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:10.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e65ce700 1 -- 172.21.15.3:0/581061688 >> v1:172.21.15.143:6789/0 conn(0x7f62e00b4550 legacy=0x7f62e012bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:10.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e65ce700 1 -- 172.21.15.3:0/581061688 >> v1:172.21.15.3:6789/0 conn(0x7f62e00b6710 legacy=0x7f62e0061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:10.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e4dcb700 1 -- 172.21.15.3:0/581061688 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604933517 0 0) 0x7f62e0144240 con 0x7f62e00b6710 2022-07-21T01:43:10.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e65ce700 1 -- 172.21.15.3:0/581061688 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e0143fa0 con 0x7f62e00b5d80 2022-07-21T01:43:10.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e65ce700 1 --2- 172.21.15.3:0/581061688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b5d80 0x7f62e00b6af0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f62d80032f0 tx=0x7f62d80091d0).ready entity=mon.0 client_cookie=42fd7af5933276c7 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:10.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e4dcb700 1 -- 172.21.15.3:0/581061688 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62d8009ea0 con 0x7f62e00b5d80 2022-07-21T01:43:10.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e4dcb700 1 -- 172.21.15.3:0/581061688 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d800b8a0 con 0x7f62e00b5d80 2022-07-21T01:43:10.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e4dcb700 1 -- 172.21.15.3:0/581061688 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62d800ba10 con 0x7f62e00b5d80 2022-07-21T01:43:10.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 -- 172.21.15.3:0/581061688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b5d80 msgr2=0x7f62e00b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:10.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 --2- 172.21.15.3:0/581061688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b5d80 0x7f62e00b6af0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f62d80032f0 tx=0x7f62d80091d0).stop 2022-07-21T01:43:10.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 -- 172.21.15.3:0/581061688 shutdown_connections 2022-07-21T01:43:10.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 --2- 172.21.15.3:0/581061688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b5d80 0x7f62e00b6af0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:10.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 -- 172.21.15.3:0/581061688 >> 172.21.15.3:0/581061688 conn(0x7f62e005c620 msgr2=0x7f62e005ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:10.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 -- 172.21.15.3:0/581061688 shutdown_connections 2022-07-21T01:43:10.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 -- 172.21.15.3:0/581061688 wait complete. 2022-07-21T01:43:10.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.431+0000 7f62e8031700 1 Processor -- start 2022-07-21T01:43:10.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e8031700 1 -- start start 2022-07-21T01:43:10.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e8031700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b4550 0x7f62e0143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:10.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e8031700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62e00b5d80 0x7f62e0141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:10.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e8031700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62e00b6710 0x7f62e0141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:10.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e8031700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e00af140 con 0x7f62e00b4550 2022-07-21T01:43:10.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e8031700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e00adaf0 con 0x7f62e00b6710 2022-07-21T01:43:10.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e8031700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e00af3a0 con 0x7f62e00b5d80 2022-07-21T01:43:10.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e5dcd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b4550 0x7f62e0143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:10.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e5dcd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b4550 0x7f62e0143380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52386/0 (socket says 172.21.15.3:52386) 2022-07-21T01:43:10.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e5dcd700 1 -- 172.21.15.3:0/1371072357 learned_addr learned my addr 172.21.15.3:0/1371072357 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:10.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e65ce700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62e00b6710 0x7f62e0141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:10.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e55cc700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62e00b5d80 0x7f62e0141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:10.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e5dcd700 1 -- 172.21.15.3:0/1371072357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62e00b5d80 msgr2=0x7f62e0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:10.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e5dcd700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62e00b5d80 0x7f62e0141aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:10.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e5dcd700 1 -- 172.21.15.3:0/1371072357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62e00b6710 msgr2=0x7f62e0141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:10.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e5dcd700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62e00b6710 0x7f62e0141f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:10.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e5dcd700 1 -- 172.21.15.3:0/1371072357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d8009470 con 0x7f62e00b4550 2022-07-21T01:43:10.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e5dcd700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b4550 0x7f62e0143380 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f62d0000bc0 tx=0x7f62d0007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:10.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62ceffd700 1 -- 172.21.15.3:0/1371072357 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62d000ce00 con 0x7f62e00b4550 2022-07-21T01:43:10.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62ceffd700 1 -- 172.21.15.3:0/1371072357 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d00070d0 con 0x7f62e00b4550 2022-07-21T01:43:10.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62ceffd700 1 -- 172.21.15.3:0/1371072357 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62d0007430 con 0x7f62e00b4550 2022-07-21T01:43:10.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e00af640 con 0x7f62e00b4550 2022-07-21T01:43:10.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.435+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e0142480 con 0x7f62e00b4550 2022-07-21T01:43:10.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.439+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e0061780 con 0x7f62e00b4550 2022-07-21T01:43:10.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.439+0000 7f62ceffd700 1 -- 172.21.15.3:0/1371072357 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f62d0010070 con 0x7f62e00b4550 2022-07-21T01:43:10.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.439+0000 7f62ceffd700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f62c0053b00 0x7f62c0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:10.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.439+0000 7f62ceffd700 1 -- 172.21.15.3:0/1371072357 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6506+0+0 (secure 0 0 0) 0x7f62d006b790 con 0x7f62e00b4550 2022-07-21T01:43:10.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.439+0000 7f62e55cc700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f62c0053b00 0x7f62c0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:10.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.439+0000 7f62e55cc700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f62c0053b00 0x7f62c0055fc0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f62dc00d000 tx=0x7f62dc007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:10.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.447+0000 7f62ceffd700 1 -- 172.21.15.3:0/1371072357 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f62d0042370 con 0x7f62e00b4550 2022-07-21T01:43:10.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:10.575+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f62e00574e0 con 0x7f62e00b4550 2022-07-21T01:43:11.337 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.335+0000 7f62ceffd700 1 -- 172.21.15.3:0/1371072357 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v163) v1 ==== 185+0+0 (secure 0 0 0) 0x7f62d0045990 con 0x7f62e00b4550 2022-07-21T01:43:11.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f62c0053b00 msgr2=0x7f62c0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:11.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f62c0053b00 0x7f62c0055fc0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f62dc00d000 tx=0x7f62dc007b90).stop 2022-07-21T01:43:11.342 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b4550 msgr2=0x7f62e0143380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:11.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b4550 0x7f62e0143380 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f62d0000bc0 tx=0x7f62d0007c90).stop 2022-07-21T01:43:11.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 shutdown_connections 2022-07-21T01:43:11.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f62c0053b00 0x7f62c0055fc0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:11.343 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62e00b6710 0x7f62e0141f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:11.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62e00b5d80 0x7f62e0141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:11.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 --2- 172.21.15.3:0/1371072357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62e00b4550 0x7f62e0143380 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:11.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 >> 172.21.15.3:0/1371072357 conn(0x7f62e005c620 msgr2=0x7f62e0145ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:11.344 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 shutdown_connections 2022-07-21T01:43:11.345 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.339+0000 7f62e8031700 1 -- 172.21.15.3:0/1371072357 wait complete. 2022-07-21T01:43:11.345 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2022-07-21T01:43:11.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-21T01:43:11.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b1ba8a700 1 Processor -- start 2022-07-21T01:43:11.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b1ba8a700 1 -- start start 2022-07-21T01:43:11.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b1ba8a700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1b140b6710 0x7f1b140b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:11.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b1ba8a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b14144240 con 0x7f1b140b4550 2022-07-21T01:43:11.477 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b1ba8a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b14143d00 con 0x7f1b140b6af0 2022-07-21T01:43:11.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b1ba8a700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b14143fa0 con 0x7f1b140b6710 2022-07-21T01:43:11.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b19826700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f1b140b6710 0x7f1b140b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:11.478 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b19025700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1b140b4550 0x7f1b14061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60676/0 (socket says 172.21.15.3:60676) 2022-07-21T01:43:11.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b19025700 1 -- 172.21.15.3:0/3965324894 learned_addr learned my addr 172.21.15.3:0/3965324894 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T01:43:11.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b19826700 1 -- 172.21.15.3:0/3965324894 >> v1:172.21.15.143:6789/0 conn(0x7f1b140b6af0 legacy=0x7f1b1412bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:11.479 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b19826700 1 -- 172.21.15.3:0/3965324894 >> v1:172.21.15.3:6789/0 conn(0x7f1b140b4550 legacy=0x7f1b14061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b19826700 1 -- 172.21.15.3:0/3965324894 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b14061780 con 0x7f1b140b6710 2022-07-21T01:43:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b19826700 1 --2- 172.21.15.3:0/3965324894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6710 0x7f1b140b5d80 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f1b141432b0 tx=0x7f1b10006df0).ready entity=mon.0 client_cookie=7ac7bda733cb4f1b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b18824700 1 -- 172.21.15.3:0/3965324894 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b1000d610 con 0x7f1b140b6710 2022-07-21T01:43:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b18824700 1 -- 172.21.15.3:0/3965324894 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b10000e90 con 0x7f1b140b6710 2022-07-21T01:43:11.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.463+0000 7f1b18824700 1 -- 172.21.15.3:0/3965324894 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b1000d960 con 0x7f1b140b6710 2022-07-21T01:43:11.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/3965324894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6710 msgr2=0x7f1b140b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:11.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 --2- 172.21.15.3:0/3965324894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6710 0x7f1b140b5d80 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f1b141432b0 tx=0x7f1b10006df0).stop 2022-07-21T01:43:11.481 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/3965324894 shutdown_connections 2022-07-21T01:43:11.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 --2- 172.21.15.3:0/3965324894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6710 0x7f1b140b5d80 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:11.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/3965324894 >> 172.21.15.3:0/3965324894 conn(0x7f1b1405c620 msgr2=0x7f1b1405ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:11.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/3965324894 shutdown_connections 2022-07-21T01:43:11.482 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/3965324894 wait complete. 2022-07-21T01:43:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 Processor -- start 2022-07-21T01:43:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 -- start start 2022-07-21T01:43:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b140b4550 0x7f1b14126ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b140b6710 0x7f1b14144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6af0 0x7f1b14155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:11.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b14126420 con 0x7f1b140b6af0 2022-07-21T01:43:11.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b141266c0 con 0x7f1b140b6710 2022-07-21T01:43:11.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1ba8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1412b270 con 0x7f1b140b4550 2022-07-21T01:43:11.484 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1a027700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6af0 0x7f1b14155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1a027700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6af0 0x7f1b14155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52402/0 (socket says 172.21.15.3:52402) 2022-07-21T01:43:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b1a027700 1 -- 172.21.15.3:0/1824542324 learned_addr learned my addr 172.21.15.3:0/1824542324 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T01:43:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b19826700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b140b4550 0x7f1b14126ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:11.485 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.467+0000 7f1b19025700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b140b6710 0x7f1b14144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b1a027700 1 -- 172.21.15.3:0/1824542324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b140b4550 msgr2=0x7f1b14126ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b1a027700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b140b4550 0x7f1b14126ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b1a027700 1 -- 172.21.15.3:0/1824542324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b140b6710 msgr2=0x7f1b14144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b1a027700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b140b6710 0x7f1b14144b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b1a027700 1 -- 172.21.15.3:0/1824542324 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1000b180 con 0x7f1b140b6af0 2022-07-21T01:43:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b19025700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b140b6710 0x7f1b14144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T01:43:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b1a027700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6af0 0x7f1b14155ad0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f1b14060a40 tx=0x7f1b0c0013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b02ffd700 1 -- 172.21.15.3:0/1824542324 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b0c006aa0 con 0x7f1b140b6af0 2022-07-21T01:43:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b02ffd700 1 -- 172.21.15.3:0/1824542324 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0c006c00 con 0x7f1b140b6af0 2022-07-21T01:43:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b140b36d0 con 0x7f1b140b6af0 2022-07-21T01:43:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b02ffd700 1 -- 172.21.15.3:0/1824542324 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b0c007030 con 0x7f1b140b6af0 2022-07-21T01:43:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b1412b710 con 0x7f1b140b6af0 2022-07-21T01:43:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b14144ff0 con 0x7f1b140b6af0 2022-07-21T01:43:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b02ffd700 1 -- 172.21.15.3:0/1824542324 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1b0c010030 con 0x7f1b140b6af0 2022-07-21T01:43:11.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b02ffd700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1af4053a70 0x7f1af4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T01:43:11.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.471+0000 7f1b02ffd700 1 -- 172.21.15.3:0/1824542324 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f1b0c06bd60 con 0x7f1b140b6af0 2022-07-21T01:43:11.490 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.475+0000 7f1b19826700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1af4053a70 0x7f1af4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T01:43:11.491 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.475+0000 7f1b19826700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1af4053a70 0x7f1af4055f30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f1b1001c000 tx=0x7f1b10015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T01:43:11.492 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.479+0000 7f1b02ffd700 1 -- 172.21.15.3:0/1824542324 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1b0c042c80 con 0x7f1b140b6af0 2022-07-21T01:43:11.611 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:11.607+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1b14126880 con 0x7f1b140b6af0 2022-07-21T01:43:12.618 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.615+0000 7f1b02ffd700 1 -- 172.21.15.3:0/1824542324 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v164) v1 ==== 173+0+0 (secure 0 0 0) 0x7f1b0c0462a0 con 0x7f1b140b6af0 2022-07-21T01:43:12.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1af4053a70 msgr2=0x7f1af4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:12.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1af4053a70 0x7f1af4055f30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f1b1001c000 tx=0x7f1b10015000).stop 2022-07-21T01:43:12.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6af0 msgr2=0x7f1b14155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T01:43:12.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6af0 0x7f1b14155ad0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f1b14060a40 tx=0x7f1b0c0013a0).stop 2022-07-21T01:43:12.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 shutdown_connections 2022-07-21T01:43:12.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b140b6af0 0x7f1b14155ad0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:12.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1af4053a70 0x7f1af4055f30 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:12.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b140b6710 0x7f1b14144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 --2- 172.21.15.3:0/1824542324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b140b4550 0x7f1b14126ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T01:43:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 >> 172.21.15.3:0/1824542324 conn(0x7f1b1405c620 msgr2=0x7f1b14141100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T01:43:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 shutdown_connections 2022-07-21T01:43:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T01:43:12.619+0000 7f1b1ba8a700 1 -- 172.21.15.3:0/1824542324 wait complete. 2022-07-21T01:43:12.626 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2022-07-21T01:43:12.637 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-07-21T01:43:12.637 DEBUG:teuthology.orchestra.run.smithi171:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-07-21T01:43:32.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:43:32.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:43:32.716 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:43:32.718 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:43:42.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:43:42.743 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:43:42.743 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:44:02.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:44:02.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:44:02.790 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:44:02.793 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:44:12.645 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-21T01:44:12.653 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-21T01:44:32.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:44:32.852 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:44:32.856 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:44:32.860 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:45:02.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:45:02.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:45:02.931 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:45:02.934 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:45:33.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:45:33.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:45:33.032 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:45:33.035 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:46:03.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:46:03.115 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:46:03.118 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:46:03.121 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:46:33.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:46:33.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:46:33.223 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:46:33.226 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:47:03.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:47:03.290 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:47:03.295 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:47:03.299 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:47:33.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:47:33.354 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:47:33.357 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:47:33.360 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:48:03.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:48:03.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:48:03.434 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:48:03.436 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:48:33.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:48:33.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:48:33.505 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:48:33.508 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:49:03.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:49:03.580 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:49:03.583 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:49:03.585 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:49:33.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:49:33.680 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:49:33.684 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:49:33.688 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:50:03.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:50:03.772 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:50:03.776 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:50:03.780 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:50:33.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:50:33.863 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:50:33.865 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:50:33.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:51:03.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:51:03.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:51:03.936 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:51:03.939 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:51:34.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:51:34.034 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:51:34.037 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:51:34.040 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:52:04.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:52:04.099 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:52:04.102 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:52:04.104 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:52:34.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:52:34.162 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:52:34.165 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:52:34.167 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:53:04.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:53:04.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:53:04.237 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:53:04.239 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:53:34.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:53:34.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:53:34.330 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:53:34.334 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:54:04.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:54:04.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:54:04.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:54:04.411 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:54:34.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:54:34.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:54:34.484 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:54:34.489 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:55:04.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:55:04.576 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:55:04.579 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:55:04.582 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:55:34.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:55:34.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:55:34.659 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:55:34.662 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:56:04.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:56:04.726 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:56:04.728 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:56:04.731 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:56:34.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:56:34.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:56:34.836 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:56:34.839 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:57:04.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:57:04.896 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:57:04.900 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:57:04.903 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:57:35.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:57:35.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:57:35.072 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:57:35.076 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:58:05.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:58:05.138 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:58:05.142 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:58:05.145 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:58:35.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:58:35.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:58:35.205 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:58:35.208 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:58:42.748 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-21T01:58:42.749 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-21T01:58:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T01:58:42.754 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-21T01:58:42.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:58:42.775 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:58:42.796 DEBUG:teuthology.orchestra.run.smithi171:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:58:42.817 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-21T01:58:42.839 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-21T01:58:42.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T01:58:42.841 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T01:58:42.842 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T01:59:05.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:59:05.272 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:59:05.275 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:59:05.290 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:59:35.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:59:35.342 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:59:35.345 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T01:59:35.347 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:00:05.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:00:05.411 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:00:05.413 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:00:05.416 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:00:35.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:00:35.511 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:00:35.513 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:00:35.517 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:01:05.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:01:05.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:01:05.573 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:01:05.575 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:01:35.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:01:35.706 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:01:35.709 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:01:35.712 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:02:05.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:02:05.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:02:05.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:02:05.774 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:02:35.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:02:35.872 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:02:35.876 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:02:35.879 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:03:05.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:03:05.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:03:05.936 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:03:05.938 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:03:35.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:03:36.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:03:36.005 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:03:36.008 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:04:06.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:04:06.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:04:06.077 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:04:06.080 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:04:36.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:04:36.171 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:04:36.175 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:04:36.180 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:05:06.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:05:06.287 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:05:06.290 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:05:06.293 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:05:36.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:05:36.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:05:36.385 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:05:36.387 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:06:06.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:06:06.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:06:06.452 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:06:06.454 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:06:36.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:06:36.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:06:36.511 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:06:36.514 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:07:06.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:07:06.590 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:07:06.593 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:07:06.597 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:07:36.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:07:36.655 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:07:36.659 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:07:36.662 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:08:06.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:08:06.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:08:06.723 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:08:06.726 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:08:36.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:08:36.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:08:36.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:08:36.807 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:09:06.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:09:06.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:09:06.874 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:09:06.876 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:09:36.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:09:36.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:09:36.964 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:09:36.967 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:10:07.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:10:07.058 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:10:07.061 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:10:07.067 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:10:37.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:10:37.144 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:10:37.147 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:10:37.149 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:11:07.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:11:07.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:11:07.209 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:11:07.211 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:11:37.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:11:37.272 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:11:37.275 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:11:37.278 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:12:07.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:12:07.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:12:07.379 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:12:07.387 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:12:37.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:12:37.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:12:37.463 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:12:37.467 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:13:07.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:13:07.527 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:13:07.531 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:13:07.535 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:13:37.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:13:37.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:13:37.592 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:13:37.594 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:13:42.864 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-21T02:13:42.865 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-21T02:13:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T02:13:42.871 INFO:tasks.cephfs_test_runner:test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) ... ERROR 2022-07-21T02:13:42.872 INFO:tasks.cephfs_test_runner:ERROR 2022-07-21T02:13:42.874 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.874 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-21T02:13:42.875 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-21T02:13:42.876 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-21T02:13:42.876 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-21T02:13:42.877 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-21T02:13:42.878 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-21T02:13:42.878 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-21T02:13:42.879 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-21T02:13:42.880 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-21T02:13:42.880 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.881 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-21T02:13:42.881 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.882 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-21T02:13:42.882 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-07-21T02:13:42.883 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-07-21T02:13:42.884 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-07-21T02:13:42.884 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-07-21T02:13:42.885 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-21T02:13:42.886 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-21T02:13:42.887 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-21T02:13:42.887 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-21T02:13:42.888 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-21T02:13:42.889 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-21T02:13:42.889 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-21T02:13:42.890 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-21T02:13:42.891 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-21T02:13:42.891 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-21T02:13:42.892 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-21T02:13:42.892 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-21T02:13:42.893 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-21T02:13:42.893 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-21T02:13:42.894 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-21T02:13:42.895 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-21T02:13:42.895 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-21T02:13:42.896 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-21T02:13:42.897 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-21T02:13:42.897 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-21T02:13:42.897 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-21T02:13:42.898 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-21T02:13:42.899 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.899 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-21T02:13:42.900 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-21T02:13:42.900 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-21T02:13:42.901 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-21T02:13:42.901 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-21T02:13:42.902 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-21T02:13:42.902 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-21T02:13:42.903 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-21T02:13:42.903 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-21T02:13:42.904 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.905 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-21T02:13:42.905 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.906 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-21T02:13:42.906 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-07-21T02:13:42.907 INFO:tasks.cephfs_test_runner: m.teardown() 2022-07-21T02:13:42.907 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-07-21T02:13:42.908 INFO:tasks.cephfs_test_runner: self.umount() 2022-07-21T02:13:42.908 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-07-21T02:13:42.908 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-07-21T02:13:42.909 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-21T02:13:42.909 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-21T02:13:42.909 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-21T02:13:42.910 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-21T02:13:42.910 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-21T02:13:42.910 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-21T02:13:42.911 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-21T02:13:42.911 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-21T02:13:42.911 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-21T02:13:42.912 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-21T02:13:42.912 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-21T02:13:42.912 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-21T02:13:42.913 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-21T02:13:42.913 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-21T02:13:42.914 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-21T02:13:42.914 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-21T02:13:42.915 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-21T02:13:42.916 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-21T02:13:42.916 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-21T02:13:42.917 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-21T02:13:42.917 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-21T02:13:42.918 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-21T02:13:42.919 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.919 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-21T02:13:42.920 INFO:tasks.cephfs_test_runner:Ran 12 tests in 2393.186s 2022-07-21T02:13:42.921 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.921 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2022-07-21T02:13:42.922 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.922 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-21T02:13:42.923 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-21T02:13:42.924 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-21T02:13:42.924 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-21T02:13:42.925 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-21T02:13:42.926 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-21T02:13:42.926 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-21T02:13:42.927 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-21T02:13:42.927 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-21T02:13:42.928 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.929 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-21T02:13:42.929 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.930 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-21T02:13:42.930 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-07-21T02:13:42.930 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-07-21T02:13:42.931 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-07-21T02:13:42.931 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-07-21T02:13:42.931 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-21T02:13:42.932 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-21T02:13:42.932 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-21T02:13:42.932 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-21T02:13:42.932 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-21T02:13:42.933 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-21T02:13:42.933 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-21T02:13:42.933 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-21T02:13:42.934 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-21T02:13:42.934 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-21T02:13:42.934 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-21T02:13:42.935 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-21T02:13:42.935 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-21T02:13:42.935 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-21T02:13:42.936 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-21T02:13:42.936 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-21T02:13:42.936 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-21T02:13:42.936 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-21T02:13:42.937 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-21T02:13:42.937 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-21T02:13:42.937 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-21T02:13:42.938 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-21T02:13:42.938 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.938 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-21T02:13:42.939 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-21T02:13:42.939 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-21T02:13:42.939 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-21T02:13:42.940 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-21T02:13:42.940 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-21T02:13:42.940 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-21T02:13:42.941 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-21T02:13:42.941 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-21T02:13:42.941 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.941 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-21T02:13:42.942 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.942 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-21T02:13:42.942 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-07-21T02:13:42.943 INFO:tasks.cephfs_test_runner: m.teardown() 2022-07-21T02:13:42.943 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-07-21T02:13:42.943 INFO:tasks.cephfs_test_runner: self.umount() 2022-07-21T02:13:42.944 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-07-21T02:13:42.944 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-07-21T02:13:42.944 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-21T02:13:42.944 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-21T02:13:42.945 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-21T02:13:42.945 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-21T02:13:42.945 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-21T02:13:42.946 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-21T02:13:42.946 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-21T02:13:42.946 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-21T02:13:42.946 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-21T02:13:42.947 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-21T02:13:42.947 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-21T02:13:42.947 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-21T02:13:42.948 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-21T02:13:42.948 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-21T02:13:42.948 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-21T02:13:42.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-21T02:13:42.949 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-21T02:13:42.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-21T02:13:42.949 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-21T02:13:42.950 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-21T02:13:42.950 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-21T02:13:42.950 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-21T02:13:42.951 INFO:tasks.cephfs_test_runner: 2022-07-21T02:13:42.951 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-21T02:13:43.086 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6ca4ce78f14e4da08f53cbb958d9cea6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-21T02:13:43.089 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2022-07-21T02:13:43.097 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2022-07-21T02:13:43.112 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2022-07-21T02:13:43.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T02:13:43.114 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T02:13:43.114 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T02:14:07.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:14:07.662 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:14:07.665 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:14:07.668 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:14:37.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:14:37.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:14:37.745 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:14:37.748 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:15:07.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:15:07.814 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:15:07.817 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:15:07.819 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:15:37.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:15:37.911 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:15:37.913 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:15:37.917 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:16:07.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:16:07.978 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:16:07.981 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:16:07.984 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:16:38.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:16:38.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:16:38.048 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:16:38.052 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:17:08.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:17:08.121 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:17:08.123 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:17:08.136 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:17:38.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:17:38.238 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:17:38.241 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:17:38.244 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:18:08.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:18:08.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:18:08.303 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:18:08.308 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:18:38.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:18:38.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:18:38.367 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:18:38.370 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:19:08.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:19:08.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:19:08.434 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:19:08.438 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:19:38.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:19:38.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:19:38.522 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:19:38.526 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:20:08.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:20:08.593 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:20:08.596 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:20:08.600 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:20:38.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:20:38.681 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:20:38.684 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:20:38.687 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:21:08.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:21:08.748 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:21:08.751 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:21:08.755 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:21:38.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:21:38.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:21:38.929 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:21:38.937 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:22:09.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:22:09.254 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:22:09.257 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:22:09.260 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:22:39.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:22:39.313 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:22:39.316 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:22:39.319 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:23:09.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:23:09.376 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:23:09.380 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:23:09.383 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:23:39.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:23:39.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:23:39.495 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:23:39.499 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:24:09.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:24:09.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:24:09.635 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:24:09.639 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:24:39.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:24:39.783 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:24:39.786 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:24:39.789 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:25:09.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:25:09.868 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:25:09.871 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:25:09.873 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:25:39.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:25:39.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:25:39.955 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:25:39.963 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:26:10.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:26:10.303 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:26:10.307 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:26:10.310 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:26:40.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:26:40.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:26:40.414 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:26:40.419 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:27:10.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:27:10.531 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:27:10.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:27:10.537 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:27:40.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:27:40.609 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:27:40.614 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:27:40.617 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:28:10.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:28:10.691 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:28:10.697 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:28:10.703 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:28:40.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:28:40.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:28:40.891 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:28:40.904 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:28:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T02:28:43.145 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Connection timed out 2022-07-21T02:28:43.146 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-21T02:28:43.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-21T02:28:43.150 DEBUG:teuthology.orchestra.run.smithi171:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-21T02:28:43.151 DEBUG:teuthology.orchestra.run.smithi171:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-21T02:29:10.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:29:10.974 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:29:10.979 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:29:10.982 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:29:41.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:29:41.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:29:41.057 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:29:41.063 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:30:11.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:30:11.136 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:30:11.139 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:30:11.142 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:30:41.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:30:41.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:30:41.505 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:30:41.508 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:31:11.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:31:11.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:31:11.630 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:31:11.642 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:31:41.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:31:41.703 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:31:41.709 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:31:41.713 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:32:11.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:32:11.785 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:32:11.790 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:32:11.794 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:32:41.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:32:41.863 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:32:41.868 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:32:41.871 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:33:11.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:33:11.936 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:33:11.939 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:33:11.942 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:33:42.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:33:42.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:33:42.026 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:33:42.029 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:34:12.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:34:12.087 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:34:12.090 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:34:12.093 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:34:42.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:34:42.154 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:34:42.157 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:34:42.161 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:35:12.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:35:12.322 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:35:12.326 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:35:12.329 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:35:42.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:35:42.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:35:42.413 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:35:42.416 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:36:12.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:36:12.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:36:12.526 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:36:12.529 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:36:42.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:36:42.666 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:36:42.670 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:36:42.674 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:37:12.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:37:12.811 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:37:12.814 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:37:12.818 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:37:42.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:37:42.952 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:37:42.956 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:37:42.959 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:38:13.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:38:13.047 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:38:13.056 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:38:13.060 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:38:43.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:38:43.199 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:38:43.203 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:38:43.206 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:39:13.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:39:13.367 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:39:13.377 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:39:13.380 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:39:43.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:39:43.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:39:43.548 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:39:43.551 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:40:13.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:40:13.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:40:13.641 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:40:13.645 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:40:43.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:40:43.945 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:40:43.948 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:40:43.951 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:41:14.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:41:14.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:41:14.099 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:41:14.105 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:41:44.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:41:44.215 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:41:44.218 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:41:44.221 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:42:14.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:42:14.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:42:14.351 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:42:14.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:42:44.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:42:44.446 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:42:44.450 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:42:44.454 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:43:14.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:43:14.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:43:14.592 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:43:14.595 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:43:43.157 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-21T02:43:43.179 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-21T02:43:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T02:43:43.197 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-21T02:43:43.229 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2022-07-21T02:43:43.248 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2022-07-21T02:43:43.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T02:43:43.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.378+0000 7f2ebf5d9700 1 Processor -- start 2022-07-21T02:43:43.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.378+0000 7f2ebf5d9700 1 -- start start 2022-07-21T02:43:43.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2eb80b3630 0x7f2eb8143f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2eb80af910 0x7f2eb81402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb80b3a10 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb81443f0 con 0x7f2eb80af910 2022-07-21T02:43:43.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb8146b40 con 0x7f2eb80b3630 2022-07-21T02:43:43.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebcb74700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2eb80aa0d0 0x7f2eb80af200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60780/0 (socket says 172.21.15.3:60780) 2022-07-21T02:43:43.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebdb76700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f2eb80af910 0x7f2eb81402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebcb74700 1 -- 172.21.15.3:0/3164828158 learned_addr learned my addr 172.21.15.3:0/3164828158 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:43.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebd375700 1 --2- 172.21.15.3:0/3164828158 >> v2:172.21.15.143:3300/0 conn(0x7f2eb80b3630 0x7f2eb8143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2eaffff700 1 -- 172.21.15.3:0/3164828158 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347846725 0 0) 0x7f2eb80b3a10 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebd375700 1 -- 172.21.15.3:0/3164828158 >> v2:172.21.15.3:3300/0 conn(0x7f2eb80af910 msgr2=0x7f2eb81402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:43.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebd375700 1 --2- 172.21.15.3:0/3164828158 >> v2:172.21.15.3:3300/0 conn(0x7f2eb80af910 0x7f2eb81402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebd375700 1 -- 172.21.15.3:0/3164828158 >> v1:172.21.15.3:6789/0 conn(0x7f2eb80aa0d0 legacy=0x7f2eb80af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebd375700 1 -- 172.21.15.3:0/3164828158 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8146de0 con 0x7f2eb80b3630 2022-07-21T02:43:43.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebd375700 1 --2- 172.21.15.3:0/3164828158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80b3630 0x7f2eb8143f00 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f2eb4002f70 tx=0x7f2eb4006df0).ready entity=mon.1 client_cookie=c227053716f5fe8a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:43.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2eaffff700 1 -- 172.21.15.3:0/3164828158 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2eb400d570 con 0x7f2eb80b3630 2022-07-21T02:43:43.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2eaffff700 1 -- 172.21.15.3:0/3164828158 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb4000e90 con 0x7f2eb80b3630 2022-07-21T02:43:43.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2eaffff700 1 -- 172.21.15.3:0/3164828158 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2eb400d8c0 con 0x7f2eb80b3630 2022-07-21T02:43:43.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/3164828158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80b3630 msgr2=0x7f2eb8143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 --2- 172.21.15.3:0/3164828158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80b3630 0x7f2eb8143f00 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f2eb4002f70 tx=0x7f2eb4006df0).stop 2022-07-21T02:43:43.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/3164828158 shutdown_connections 2022-07-21T02:43:43.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 --2- 172.21.15.3:0/3164828158 >> v2:172.21.15.3:3300/0 conn(0x7f2eb80af910 0x7f2eb81402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 --2- 172.21.15.3:0/3164828158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80b3630 0x7f2eb8143f00 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/3164828158 >> 172.21.15.3:0/3164828158 conn(0x7f2eb8060950 msgr2=0x7f2eb8060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:43.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/3164828158 shutdown_connections 2022-07-21T02:43:43.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.382+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/3164828158 wait complete. 2022-07-21T02:43:43.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebf5d9700 1 Processor -- start 2022-07-21T02:43:43.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebf5d9700 1 -- start start 2022-07-21T02:43:43.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebf5d9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2eb80aa0d0 0x7f2eb80b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebf5d9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80af910 0x7f2eb8141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebf5d9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb80b3630 0x7f2eb8141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebf5d9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81527e0 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebf5d9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb8152a80 con 0x7f2eb80af910 2022-07-21T02:43:43.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebf5d9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb8152d20 con 0x7f2eb80b3630 2022-07-21T02:43:43.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebd375700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2eb80aa0d0 0x7f2eb80b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebd375700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2eb80aa0d0 0x7f2eb80b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52504/0 (socket says 172.21.15.3:52504) 2022-07-21T02:43:43.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebd375700 1 -- 172.21.15.3:0/2136922837 learned_addr learned my addr 172.21.15.3:0/2136922837 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:43.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebcb74700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80af910 0x7f2eb8141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebdb76700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb80b3630 0x7f2eb8141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebd375700 1 -- 172.21.15.3:0/2136922837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb80b3630 msgr2=0x7f2eb8141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebd375700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb80b3630 0x7f2eb8141810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.413 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebd375700 1 -- 172.21.15.3:0/2136922837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80af910 msgr2=0x7f2eb8141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebd375700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80af910 0x7f2eb8141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.386+0000 7f2ebd375700 1 -- 172.21.15.3:0/2136922837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb400b180 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.414 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2ebd375700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2eb80aa0d0 0x7f2eb80b2440 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7f2eb4002f70 tx=0x7f2eb4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:43.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2eae7fc700 1 -- 172.21.15.3:0/2136922837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2eb4016070 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2eae7fc700 1 -- 172.21.15.3:0/2136922837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb4013700 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.415 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb8152fc0 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2eae7fc700 1 -- 172.21.15.3:0/2136922837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2eb4013a60 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb8153460 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.416 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb8147080 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2eae7fc700 1 -- 172.21.15.3:0/2136922837 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2eb400f030 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2eae7fc700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ea0053b00 0x7f2ea0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2eae7fc700 1 -- 172.21.15.3:0/2136922837 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f2eb406c940 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.390+0000 7f2ebcb74700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ea0053b00 0x7f2ea0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.394+0000 7f2ebcb74700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ea0053b00 0x7f2ea0055fc0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f2ea800d000 tx=0x7f2ea8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:43.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.398+0000 7f2eae7fc700 1 -- 172.21.15.3:0/2136922837 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2eb4043940 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.526+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eb80574e0 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.526+0000 7f2eae7fc700 1 -- 172.21.15.3:0/2136922837 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+13551 (secure 0 0 0) 0x7f2eb4023090 con 0x7f2eb80aa0d0 2022-07-21T02:43:43.529 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T02:43:43.529 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":164,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:43:12.602152+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.3:6837/1787668325":"2022-07-22T01:43:08.803052+0000","172.21.15.3:6836/1787668325":"2022-07-22T01:43:08.803052+0000"},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T02:43:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ea0053b00 msgr2=0x7f2ea0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ea0053b00 0x7f2ea0055fc0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f2ea800d000 tx=0x7f2ea8009040).stop 2022-07-21T02:43:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2eb80aa0d0 msgr2=0x7f2eb80b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2eb80aa0d0 0x7f2eb80b2440 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7f2eb4002f70 tx=0x7f2eb4014000).stop 2022-07-21T02:43:43.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 shutdown_connections 2022-07-21T02:43:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb80b3630 0x7f2eb8141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f2ea0053b00 0x7f2ea0055fc0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80af910 0x7f2eb8141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 --2- 172.21.15.3:0/2136922837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2eb80aa0d0 0x7f2eb80b2440 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 >> 172.21.15.3:0/2136922837 conn(0x7f2eb8060950 msgr2=0x7f2eb8060d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 shutdown_connections 2022-07-21T02:43:43.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.530+0000 7f2ebf5d9700 1 -- 172.21.15.3:0/2136922837 wait complete. 2022-07-21T02:43:43.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-21T02:43:43.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.646+0000 7f83225e8700 1 Processor -- start 2022-07-21T02:43:43.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.646+0000 7f83225e8700 1 -- start start 2022-07-21T02:43:43.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f831c05d240 0x7f831c0b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c13f520 0x7f831c0aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c05cd40 con 0x7f831c05c960 2022-07-21T02:43:43.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f831c0ad6d0 con 0x7f831c05d240 2022-07-21T02:43:43.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c067f00 con 0x7f831c13f520 2022-07-21T02:43:43.661 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f8320de5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f831c05d240 0x7f831c0b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f8320de5700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f831c05d240 0x7f831c0b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52512/0 (socket says 172.21.15.3:52512) 2022-07-21T02:43:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f8320de5700 1 -- 172.21.15.3:0/2267887170 learned_addr learned my addr 172.21.15.3:0/2267887170 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f8321de7700 1 --2- 172.21.15.3:0/2267887170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c13f520 0x7f831c0aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f8320de5700 1 -- 172.21.15.3:0/2267887170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c13f520 msgr2=0x7f831c0aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f8320de5700 1 --2- 172.21.15.3:0/2267887170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c13f520 0x7f831c0aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f8320de5700 1 -- 172.21.15.3:0/2267887170 >> v1:172.21.15.143:6789/0 conn(0x7f831c05c960 legacy=0x7f831c0acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f8320de5700 1 -- 172.21.15.3:0/2267887170 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c0b1920 con 0x7f831c05d240 2022-07-21T02:43:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f8320de5700 1 --2- 172.21.15.3:0/2267887170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c05d240 0x7f831c0b1280 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7f830c002f70 tx=0x7f830c006eb0).ready entity=mon.0 client_cookie=3ac7deacc4b9a49a server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f830bfff700 1 -- 172.21.15.3:0/2267887170 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f830c00d570 con 0x7f831c05d240 2022-07-21T02:43:43.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f830bfff700 1 -- 172.21.15.3:0/2267887170 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830c000e90 con 0x7f831c05d240 2022-07-21T02:43:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f830bfff700 1 -- 172.21.15.3:0/2267887170 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f830c00d900 con 0x7f831c05d240 2022-07-21T02:43:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 -- 172.21.15.3:0/2267887170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c05d240 msgr2=0x7f831c0b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 --2- 172.21.15.3:0/2267887170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c05d240 0x7f831c0b1280 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7f830c002f70 tx=0x7f830c006eb0).stop 2022-07-21T02:43:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 -- 172.21.15.3:0/2267887170 shutdown_connections 2022-07-21T02:43:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 --2- 172.21.15.3:0/2267887170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c13f520 0x7f831c0aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 --2- 172.21.15.3:0/2267887170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c05d240 0x7f831c0b1280 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 -- 172.21.15.3:0/2267887170 >> 172.21.15.3:0/2267887170 conn(0x7f831c060310 msgr2=0x7f831c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 -- 172.21.15.3:0/2267887170 shutdown_connections 2022-07-21T02:43:43.668 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.650+0000 7f83225e8700 1 -- 172.21.15.3:0/2267887170 wait complete. 2022-07-21T02:43:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f83225e8700 1 Processor -- start 2022-07-21T02:43:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f83225e8700 1 -- start start 2022-07-21T02:43:43.669 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f83225e8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f831c05c960 0x7f831c0aa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f83225e8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c05d240 0x7f831c0aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f83225e8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c13f520 0x7f831c0ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.670 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f83225e8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c131ba0 con 0x7f831c13f520 2022-07-21T02:43:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f83225e8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c141360 con 0x7f831c05c960 2022-07-21T02:43:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f83225e8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c0ab9f0 con 0x7f831c05d240 2022-07-21T02:43:43.671 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f8321de7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c13f520 0x7f831c0ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f83215e6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f831c05c960 0x7f831c0aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f8321de7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c13f520 0x7f831c0ab2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52522/0 (socket says 172.21.15.3:52522) 2022-07-21T02:43:43.672 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f8321de7700 1 -- 172.21.15.3:0/1026869891 learned_addr learned my addr 172.21.15.3:0/1026869891 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.654+0000 7f8320de5700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c05d240 0x7f831c0aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f8321de7700 1 -- 172.21.15.3:0/1026869891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c05d240 msgr2=0x7f831c0aadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.673 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f8321de7700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c05d240 0x7f831c0aadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f8321de7700 1 -- 172.21.15.3:0/1026869891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f831c05c960 msgr2=0x7f831c0aa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f8321de7700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f831c05c960 0x7f831c0aa470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.674 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f8321de7700 1 -- 172.21.15.3:0/1026869891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c00b1e0 con 0x7f831c13f520 2022-07-21T02:43:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f83215e6700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f831c05c960 0x7f831c0aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T02:43:43.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f8321de7700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c13f520 0x7f831c0ab2c0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f831c1414a0 tx=0x7f83140091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f830a7fc700 1 -- 172.21.15.3:0/1026869891 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8314012600 con 0x7f831c13f520 2022-07-21T02:43:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f830a7fc700 1 -- 172.21.15.3:0/1026869891 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8314012760 con 0x7f831c13f520 2022-07-21T02:43:43.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f830a7fc700 1 -- 172.21.15.3:0/1026869891 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8314012ac0 con 0x7f831c13f520 2022-07-21T02:43:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831c0abc90 con 0x7f831c13f520 2022-07-21T02:43:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831c0ac1d0 con 0x7f831c13f520 2022-07-21T02:43:43.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831c1426a0 con 0x7f831c13f520 2022-07-21T02:43:43.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f830a7fc700 1 -- 172.21.15.3:0/1026869891 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8314022070 con 0x7f831c13f520 2022-07-21T02:43:43.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f830a7fc700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82fc0539e0 0x7f82fc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f830a7fc700 1 -- 172.21.15.3:0/1026869891 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f831406c380 con 0x7f831c13f520 2022-07-21T02:43:43.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.658+0000 7f83215e6700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82fc0539e0 0x7f82fc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.662+0000 7f83215e6700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82fc0539e0 0x7f82fc055ea0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f831800d000 tx=0x7f8318007b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:43.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.666+0000 7f830a7fc700 1 -- 172.21.15.3:0/1026869891 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8314043310 con 0x7f831c13f520 2022-07-21T02:43:43.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.790+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f831c0b1bc0 con 0x7f82fc0539e0 2022-07-21T02:43:43.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.794+0000 7f830a7fc700 1 -- 172.21.15.3:0/1026869891 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68226 (secure 0 0 0) 0x7f831c0b1bc0 con 0x7f82fc0539e0 2022-07-21T02:43:43.798 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T02:43:43.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82fc0539e0 msgr2=0x7f82fc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.801 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82fc0539e0 0x7f82fc055ea0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f831800d000 tx=0x7f8318007b50).stop 2022-07-21T02:43:43.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c13f520 msgr2=0x7f831c0ab2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.802 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c13f520 0x7f831c0ab2c0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f831c1414a0 tx=0x7f83140091d0).stop 2022-07-21T02:43:43.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 shutdown_connections 2022-07-21T02:43:43.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f831c13f520 0x7f831c0ab2c0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f831c05d240 0x7f831c0aadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f82fc0539e0 0x7f82fc055ea0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 --2- 172.21.15.3:0/1026869891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f831c05c960 0x7f831c0aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 >> 172.21.15.3:0/1026869891 conn(0x7f831c060310 msgr2=0x7f831c13e690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:43.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 shutdown_connections 2022-07-21T02:43:43.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.798+0000 7f83225e8700 1 -- 172.21.15.3:0/1026869891 wait complete. 2022-07-21T02:43:43.806 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2022-07-21T02:43:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":2272,"stamp":"2022-07-21T02:43:42.742455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8404928,"kb_used_data":16256,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746569792,"statfs":{"total":773094113280,"available":764487467008,"internally_reserved":8589934592,"allocated":16646144,"data_stored":8116488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003362"},"pg_stats":[{"pgid":"1.0","version":"16'2","reported_seq":147,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.423987+0000","last_change":"2022-07-21T01:33:15.640655+0000","last_active":"2022-07-21T01:43:11.423987+0000","last_peered":"2022-07-21T01:43:11.423987+0000","last_clean":"2022-07-21T01:43:11.423987+0000","last_became_active":"2022-07-21T01:33:15.639383+0000","last_became_peered":"2022-07-21T01:33:15.639383+0000","last_unstale":"2022-07-21T01:43:11.423987+0000","last_undegraded":"2022-07-21T01:43:11.423987+0000","last_fullsized":"2022-07-21T01:43:11.423987+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:10.553176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:10.553176+0000","last_clean_scrub_stamp":"2022-07-21T01:33:10.553176+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":150,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.335690+0000","last_change":"2022-07-21T01:33:19.894256+0000","last_active":"2022-07-21T01:43:11.335690+0000","last_peered":"2022-07-21T01:43:11.335690+0000","last_clean":"2022-07-21T01:43:11.335690+0000","last_became_active":"2022-07-21T01:33:19.893274+0000","last_became_peered":"2022-07-21T01:33:19.893274+0000","last_unstale":"2022-07-21T01:43:11.335690+0000","last_undegraded":"2022-07-21T01:43:11.335690+0000","last_fullsized":"2022-07-21T01:43:11.335690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.423951+0000","last_change":"2022-07-21T01:33:19.897103+0000","last_active":"2022-07-21T01:43:11.423951+0000","last_peered":"2022-07-21T01:43:11.423951+0000","last_clean":"2022-07-21T01:43:11.423951+0000","last_became_active":"2022-07-21T01:33:19.896369+0000","last_became_peered":"2022-07-21T01:33:19.896369+0000","last_unstale":"2022-07-21T01:43:11.423951+0000","last_undegraded":"2022-07-21T01:43:11.423951+0000","last_fullsized":"2022-07-21T01:43:11.423951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":140,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.337565+0000","last_change":"2022-07-21T01:33:19.896866+0000","last_active":"2022-07-21T01:43:11.337565+0000","last_peered":"2022-07-21T01:43:11.337565+0000","last_clean":"2022-07-21T01:43:11.337565+0000","last_became_active":"2022-07-21T01:33:19.896120+0000","last_became_peered":"2022-07-21T01:33:19.896120+0000","last_unstale":"2022-07-21T01:43:11.337565+0000","last_undegraded":"2022-07-21T01:43:11.337565+0000","last_fullsized":"2022-07-21T01:43:11.337565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":150,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.335687+0000","last_change":"2022-07-21T01:33:19.896530+0000","last_active":"2022-07-21T01:43:11.335687+0000","last_peered":"2022-07-21T01:43:11.335687+0000","last_clean":"2022-07-21T01:43:11.335687+0000","last_became_active":"2022-07-21T01:33:19.896110+0000","last_became_peered":"2022-07-21T01:33:19.896110+0000","last_unstale":"2022-07-21T01:43:11.335687+0000","last_undegraded":"2022-07-21T01:43:11.335687+0000","last_fullsized":"2022-07-21T01:43:11.335687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":144,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.337497+0000","last_change":"2022-07-21T01:33:20.650822+0000","last_active":"2022-07-21T01:43:11.337497+0000","last_peered":"2022-07-21T01:43:11.337497+0000","last_clean":"2022-07-21T01:43:11.337497+0000","last_became_active":"2022-07-21T01:33:20.649660+0000","last_became_peered":"2022-07-21T01:33:20.649660+0000","last_unstale":"2022-07-21T01:43:11.337497+0000","last_undegraded":"2022-07-21T01:43:11.337497+0000","last_fullsized":"2022-07-21T01:43:11.337497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":141,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.424202+0000","last_change":"2022-07-21T01:33:20.648160+0000","last_active":"2022-07-21T01:43:11.424202+0000","last_peered":"2022-07-21T01:43:11.424202+0000","last_clean":"2022-07-21T01:43:11.424202+0000","last_became_active":"2022-07-21T01:33:20.647021+0000","last_became_peered":"2022-07-21T01:33:20.647021+0000","last_unstale":"2022-07-21T01:43:11.424202+0000","last_undegraded":"2022-07-21T01:43:11.424202+0000","last_fullsized":"2022-07-21T01:43:11.424202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":144,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.338584+0000","last_change":"2022-07-21T01:33:19.900435+0000","last_active":"2022-07-21T01:43:11.338584+0000","last_peered":"2022-07-21T01:43:11.338584+0000","last_clean":"2022-07-21T01:43:11.338584+0000","last_became_active":"2022-07-21T01:33:19.900054+0000","last_became_peered":"2022-07-21T01:33:19.900054+0000","last_unstale":"2022-07-21T01:43:11.338584+0000","last_undegraded":"2022-07-21T01:43:11.338584+0000","last_fullsized":"2022-07-21T01:43:11.338584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":141,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.835880+0000","last_change":"2022-07-21T01:33:19.895590+0000","last_active":"2022-07-21T01:43:11.835880+0000","last_peered":"2022-07-21T01:43:11.835880+0000","last_clean":"2022-07-21T01:43:11.835880+0000","last_became_active":"2022-07-21T01:33:19.895046+0000","last_became_peered":"2022-07-21T01:33:19.895046+0000","last_unstale":"2022-07-21T01:43:11.835880+0000","last_undegraded":"2022-07-21T01:43:11.835880+0000","last_fullsized":"2022-07-21T01:43:11.835880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424510287,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:42:40 2022","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60399999999999998,"15min":0.61899999999999999},"min":{"1min":0.45000000000000001,"5min":0.42999999999999999,"15min":0.42199999999999999},"max":{"1min":0.81100000000000005,"5min":0.94899999999999995,"15min":1.2749999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68899999999999995,"15min":0.68500000000000005},"min":{"1min":0.52700000000000002,"5min":0.503,"15min":0.47899999999999998},"max":{"1min":1.026,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.81299999999999994}]},{"osd":1,"last update":"Thu Jul 21 02:42:40 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68600000000000005,"15min":0.69799999999999995},"min":{"1min":0.54300000000000004,"5min":0.496,"15min":0.39000000000000001},"max":{"1min":0.82799999999999996,"5min":1.0429999999999999,"15min":1.1919999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73499999999999999,"15min":0.73899999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.47999999999999998},"max":{"1min":0.89600000000000002,"5min":1.21,"15min":1.21},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jul 21 02:42:40 2022","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65800000000000003,"15min":0.67400000000000004},"min":{"1min":0.503,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.1419999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.79900000000000004,"15min":0.82399999999999995},"min":{"1min":0.65900000000000003,"5min":0.57199999999999995,"15min":0.55900000000000005},"max":{"1min":1.03,"5min":1.046,"15min":1.2569999999999999},"last":0.85099999999999998}]},{"osd":4,"last update":"Thu Jul 21 02:42:40 2022","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.92300000000000004,"15min":0.93500000000000005},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.44800000000000001},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.3959999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.94699999999999995,"15min":0.95099999999999996},"min":{"1min":0.747,"5min":0.72199999999999998,"15min":0.56599999999999995},"max":{"1min":1.2390000000000001,"5min":1.2470000000000001,"15min":1.343},"last":0.999}]},{"osd":5,"last update":"Thu Jul 21 02:42:40 2022","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.97099999999999997,"15min":0.96599999999999997},"min":{"1min":0.69599999999999995,"5min":0.66400000000000003,"15min":0.60699999999999998},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96299999999999997,"15min":0.96899999999999997},"min":{"1min":0.75,"5min":0.71199999999999997,"15min":0.52200000000000002},"max":{"1min":1.2929999999999999,"5min":1.365,"15min":1.365},"last":0.88}]},{"osd":6,"last update":"Thu Jul 21 02:42:40 2022","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.99199999999999999,"15min":0.999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.56200000000000006},"max":{"1min":1.3979999999999999,"5min":1.446,"15min":1.522},"last":1.036},{"interface":"front","average":{"1min":1.022,"5min":1.0580000000000001,"15min":1.054},"min":{"1min":0.84699999999999998,"5min":0.83099999999999996,"15min":0.70499999999999996},"max":{"1min":1.327,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.91500000000000004}]},{"osd":7,"last update":"Thu Jul 21 02:42:40 2022","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0680000000000001,"15min":1.0609999999999999},"min":{"1min":0.80600000000000005,"5min":0.79800000000000004,"15min":0.67700000000000005},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5900000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.071,"15min":1.0649999999999999},"min":{"1min":0.90600000000000003,"5min":0.75600000000000001,"15min":0.61799999999999999},"max":{"1min":1.296,"5min":1.4059999999999999,"15min":1.552},"last":0.94499999999999995}]}]},{"osd":0,"up_from":15,"seq":64424510289,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.498,"5min":0.46400000000000002,"15min":0.41999999999999998},"max":{"1min":0.79600000000000004,"5min":1.387,"15min":1.387},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76700000000000002,"15min":0.76200000000000001},"min":{"1min":0.66000000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0569999999999999,"5min":1.1970000000000001,"15min":1.3029999999999999},"last":0.75600000000000001}]},{"osd":2,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.626,"15min":0.628},"min":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.40000000000000002},"max":{"1min":0.96599999999999997,"5min":1.052,"15min":1.1830000000000001},"last":0.629},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.71499999999999997,"15min":0.71799999999999997},"min":{"1min":0.55000000000000004,"5min":0.52900000000000003,"15min":0.48899999999999999},"max":{"1min":0.97099999999999997,"5min":1.123,"15min":1.202},"last":0.56999999999999995}]},{"osd":3,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.56299999999999994,"5min":0.502,"15min":0.47499999999999998},"max":{"1min":0.98599999999999999,"5min":1.651,"15min":1.651},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.83899999999999997,"15min":0.83499999999999996},"min":{"1min":0.70799999999999996,"5min":0.64100000000000001,"15min":0.49199999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1699999999999999},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.69299999999999995,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.131,"5min":1.2,"15min":1.2},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91700000000000004,"15min":0.92700000000000005},"min":{"1min":0.78900000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1160000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.871}]},{"osd":5,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.95599999999999996,"15min":0.96899999999999997},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.56299999999999994},"max":{"1min":1.2210000000000001,"5min":1.399,"15min":1.399},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96599999999999997,"15min":0.98499999999999999},"min":{"1min":0.83599999999999997,"5min":0.69399999999999995,"15min":0.56899999999999995},"max":{"1min":1.151,"5min":1.698,"15min":1.698},"last":0.94599999999999995}]},{"osd":6,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0089999999999999,"15min":1.008},"min":{"1min":0.745,"5min":0.74199999999999999,"15min":0.70699999999999996},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":1.073},{"interface":"front","average":{"1min":1.054,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.85699999999999998,"5min":0.748,"15min":0.748},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.97399999999999998}]},{"osd":7,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.054,"15min":1.0629999999999999},"min":{"1min":0.91100000000000003,"5min":0.79300000000000004,"15min":0.65800000000000003},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.0820000000000001,"15min":1.0980000000000001},"min":{"1min":0.80500000000000005,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.0389999999999999}]}]},{"osd":3,"up_from":15,"seq":64424510287,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.49099999999999999,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":0.91400000000000003,"5min":1.607,"15min":1.607},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69699999999999995,"15min":0.69399999999999995},"min":{"1min":0.55000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.81599999999999995,"5min":1.758,"15min":1.758},"last":0.88400000000000001}]},{"osd":1,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65300000000000002,"15min":0.65800000000000003},"min":{"1min":0.499,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88300000000000001,"5min":1.1890000000000001,"15min":1.2410000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.77300000000000002,"15min":0.77600000000000002},"min":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.51700000000000002},"max":{"1min":1.145,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.92400000000000004}]},{"osd":2,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.77500000000000002,"15min":0.76900000000000002},"min":{"1min":0.58699999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"max":{"1min":0.96199999999999997,"5min":1.236,"15min":1.2609999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.84999999999999998,"15min":0.84399999999999997},"min":{"1min":0.65800000000000003,"5min":0.59999999999999998,"15min":0.56799999999999995},"max":{"1min":1.024,"5min":1.131,"15min":1.153},"last":1.056}]},{"osd":4,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.91400000000000003,"15min":0.89300000000000002},"min":{"1min":0.75800000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1479999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.96199999999999997,"15min":0.95099999999999996},"min":{"1min":0.73199999999999998,"5min":0.67700000000000005,"15min":0.66700000000000004},"max":{"1min":1.2070000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.97899999999999998,"15min":0.97699999999999998},"min":{"1min":0.78800000000000003,"5min":0.70699999999999996,"15min":0.70099999999999996},"max":{"1min":1.2869999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.873},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.028,"15min":1.0269999999999999},"min":{"1min":0.79800000000000004,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.268,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.1579999999999999}]},{"osd":6,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0089999999999999,"15min":1.0009999999999999},"min":{"1min":0.84599999999999997,"5min":0.73799999999999999,"15min":0.71599999999999997},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.5189999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.0629999999999999,"15min":1.0549999999999999},"min":{"1min":0.89600000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.355,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.0009999999999999}]},{"osd":7,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.101,"15min":1.0960000000000001},"min":{"1min":0.87,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.3220000000000001,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.21},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.097,"15min":1.079},"min":{"1min":0.89200000000000002,"5min":0.85399999999999998,"15min":0.77700000000000002},"max":{"1min":1.399,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.1160000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542992,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.90400000000000003,"15min":0.88700000000000001},"min":{"1min":0.71099999999999997,"5min":0.64600000000000002,"15min":0.46500000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.74199999999999999,"5min":0.66400000000000003,"15min":0.63200000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.196},"last":0.82899999999999996}]},{"osd":1,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82199999999999995,"15min":0.82099999999999995},"min":{"1min":0.622,"5min":0.58699999999999997,"15min":0.56399999999999995},"max":{"1min":1.0509999999999999,"5min":1.1020000000000001,"15min":1.214},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83299999999999996,"15min":0.82899999999999996},"min":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56999999999999995},"max":{"1min":1.0700000000000001,"5min":1.085,"15min":1.1579999999999999},"last":0.86499999999999999}]},{"osd":2,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.84199999999999997,"15min":0.83699999999999997},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.56399999999999995},"max":{"1min":1.107,"5min":1.117,"15min":1.1399999999999999},"last":0.63},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85899999999999999,"15min":0.84899999999999998},"min":{"1min":0.65900000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.145,"5min":1.145,"15min":1.1659999999999999},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93300000000000005,"15min":0.91200000000000003},"min":{"1min":0.74399999999999999,"5min":0.64100000000000001,"15min":0.58399999999999996},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.94499999999999995,"15min":0.94599999999999995},"min":{"1min":0.67700000000000005,"5min":0.64900000000000002,"15min":0.58399999999999996},"max":{"1min":1.135,"5min":1.155,"15min":1.2270000000000001},"last":0.97699999999999998}]},{"osd":4,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78500000000000003,"15min":0.78400000000000003},"min":{"1min":0.627,"5min":0.54700000000000004,"15min":0.53300000000000003},"max":{"1min":1.0680000000000001,"5min":1.242,"15min":1.242},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.86299999999999999,"15min":0.86199999999999999},"min":{"1min":0.60499999999999998,"5min":0.58999999999999997,"15min":0.56200000000000006},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.284},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.86599999999999999},"min":{"1min":0.67400000000000004,"5min":0.55800000000000005,"15min":0.55400000000000005},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.65600000000000003,"5min":0.59499999999999997,"15min":0.59199999999999997},"max":{"1min":1.149,"5min":1.194,"15min":1.2649999999999999},"last":1.0309999999999999}]},{"osd":6,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.82299999999999995,"5min":0.625,"15min":0.54900000000000004},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.1020000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":0.99399999999999999},"min":{"1min":0.74299999999999999,"5min":0.67500000000000004,"15min":0.66600000000000004},"max":{"1min":1.1870000000000001,"5min":1.379,"15min":1.379},"last":1.0680000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542991,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.82799999999999996,"15min":0.81899999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2090000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.83599999999999997,"15min":0.83799999999999997},"min":{"1min":0.55900000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.05,"5min":1.119,"15min":1.212},"last":0.89100000000000001}]},{"osd":1,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82099999999999995,"15min":0.84199999999999997},"min":{"1min":0.61299999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.075,"5min":1.1910000000000001,"15min":1.2030000000000001},"last":0.996},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.872,"15min":0.86499999999999999},"min":{"1min":0.65900000000000003,"5min":0.61799999999999999,"15min":0.60399999999999998},"max":{"1min":1.123,"5min":1.145,"15min":1.2609999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91900000000000004,"15min":0.90600000000000003},"min":{"1min":0.67500000000000004,"5min":0.625,"15min":0.61599999999999999},"max":{"1min":1.139,"5min":1.139,"15min":1.2509999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.80800000000000005,"15min":0.82099999999999995},"min":{"1min":0.65100000000000002,"5min":0.60099999999999998,"15min":0.58499999999999996},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1739999999999999},"last":0.64800000000000002}]},{"osd":3,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91100000000000003,"15min":0.91300000000000003},"min":{"1min":0.77500000000000002,"5min":0.71799999999999997,"15min":0.65000000000000002},"max":{"1min":1.099,"5min":1.1299999999999999,"15min":1.2010000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.71999999999999997,"5min":0.61899999999999999,"15min":0.61799999999999999},"max":{"1min":1.1499999999999999,"5min":1.165,"15min":1.3029999999999999},"last":0.83699999999999997}]},{"osd":4,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.60199999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.083,"5min":1.1679999999999999,"15min":1.169},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.85499999999999998,"15min":0.85899999999999999},"min":{"1min":0.62,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.202,"5min":1.202,"15min":1.2889999999999999},"last":0.80400000000000005}]},{"osd":5,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77800000000000002,"15min":0.80000000000000004},"min":{"1min":0.64500000000000002,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0149999999999999,"5min":1.097,"15min":1.2989999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.89600000000000002,"15min":0.92600000000000005},"min":{"1min":0.60499999999999998,"5min":0.57599999999999996,"15min":0.57099999999999995},"max":{"1min":1.24,"5min":1.24,"15min":1.2709999999999999},"last":0.91600000000000004}]},{"osd":7,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92000000000000004,"15min":0.94999999999999996},"min":{"1min":0.66700000000000004,"5min":0.622,"15min":0.57299999999999995},"max":{"1min":1.0640000000000001,"5min":1.214,"15min":1.5029999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.96399999999999997,"15min":0.97699999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.59799999999999998},"max":{"1min":1.0780000000000001,"5min":1.22,"15min":1.482},"last":0.94699999999999995}]}]},{"osd":5,"up_from":14,"seq":60129542991,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:42:45 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.55700000000000005,"5min":0.503,"15min":0.49299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.2070000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.86599999999999999,"15min":0.84699999999999998},"min":{"1min":0.68899999999999995,"5min":0.59899999999999998,"15min":0.51900000000000002},"max":{"1min":1.139,"5min":1.1719999999999999,"15min":1.226},"last":1.0349999999999999}]},{"osd":1,"last update":"Thu Jul 21 02:42:45 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.875,"15min":0.877},"min":{"1min":0.66100000000000003,"5min":0.58599999999999997,"15min":0.55500000000000005},"max":{"1min":1.198,"5min":1.198,"15min":1.282},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.85399999999999998,"15min":0.83699999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.56000000000000005},"max":{"1min":1.123,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.626}]},{"osd":2,"last update":"Thu Jul 21 02:42:45 2022","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86099999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.63},"max":{"1min":1.0589999999999999,"5min":1.204,"15min":1.23},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80300000000000005,"15min":0.79600000000000004},"min":{"1min":0.63400000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0680000000000001,"5min":1.081,"15min":1.153},"last":0.92800000000000005}]},{"osd":3,"last update":"Thu Jul 21 02:42:45 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.90800000000000003,"15min":0.89200000000000002},"min":{"1min":0.69299999999999995,"5min":0.624,"15min":0.59999999999999998},"max":{"1min":1.1020000000000001,"5min":1.171,"15min":1.1719999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.91000000000000003,"15min":0.90800000000000003},"min":{"1min":0.76100000000000001,"5min":0.65100000000000002,"15min":0.64100000000000001},"max":{"1min":1.1339999999999999,"5min":1.2749999999999999,"15min":1.276},"last":1.0009999999999999}]},{"osd":4,"last update":"Thu Jul 21 02:42:45 2022","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.83899999999999997,"15min":0.82999999999999996},"min":{"1min":0.60899999999999999,"5min":0.52300000000000002,"15min":0.45700000000000002},"max":{"1min":1.1319999999999999,"5min":1.1859999999999999,"15min":1.26},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.98299999999999998,"15min":0.97899999999999998},"min":{"1min":0.748,"5min":0.71599999999999997,"15min":0.54600000000000004},"max":{"1min":1.1679999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.88800000000000001}]},{"osd":6,"last update":"Thu Jul 21 02:42:45 2022","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.92600000000000005,"15min":0.91800000000000004},"min":{"1min":0.72499999999999998,"5min":0.61599999999999999,"15min":0.59899999999999998},"max":{"1min":1.1879999999999999,"5min":1.2130000000000001,"15min":1.3080000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.92500000000000004,"15min":0.92400000000000004},"min":{"1min":0.66500000000000004,"5min":0.66400000000000003,"15min":0.63900000000000001},"max":{"1min":1.1080000000000001,"5min":1.2090000000000001,"15min":1.3120000000000001},"last":0.75700000000000001}]},{"osd":7,"last update":"Thu Jul 21 02:42:45 2022","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.84199999999999997,"15min":0.84799999999999998},"min":{"1min":0.58399999999999996,"5min":0.56899999999999995,"15min":0.54900000000000004},"max":{"1min":1.1519999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":1.2350000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.99099999999999999,"15min":0.996},"min":{"1min":0.70999999999999996,"5min":0.70299999999999996,"15min":0.53300000000000003},"max":{"1min":1.161,"5min":1.2949999999999999,"15min":1.351},"last":0.71899999999999997}]}]},{"osd":4,"up_from":14,"seq":60129542991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.873,"15min":0.86099999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.46300000000000002},"max":{"1min":0.99199999999999999,"5min":1.117,"15min":1.234},"last":0.752},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.98699999999999999,"15min":0.99099999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.63400000000000001},"max":{"1min":1.097,"5min":1.226,"15min":1.258},"last":0.93700000000000006}]},{"osd":1,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.86299999999999999,"15min":0.85799999999999998},"min":{"1min":0.61599999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0700000000000001,"5min":1.163,"15min":1.2250000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.85299999999999998,"15min":0.83799999999999997},"min":{"1min":0.61199999999999999,"5min":0.61099999999999999,"15min":0.505},"max":{"1min":1.038,"5min":1.157,"15min":1.157},"last":0.89200000000000002}]},{"osd":2,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.90800000000000003,"15min":0.90000000000000002},"min":{"1min":0.74199999999999999,"5min":0.63800000000000001,"15min":0.60299999999999998},"max":{"1min":1.1319999999999999,"5min":1.1870000000000001,"15min":1.1910000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88100000000000001,"15min":0.85499999999999998},"min":{"1min":0.70399999999999996,"5min":0.61799999999999999,"15min":0.58999999999999997},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.1930000000000001},"last":0.97499999999999998}]},{"osd":3,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90000000000000002,"15min":0.89300000000000002},"min":{"1min":0.71399999999999997,"5min":0.63800000000000001,"15min":0.57099999999999995},"max":{"1min":1.0620000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.879},{"interface":"front","average":{"1min":0.871,"5min":0.89000000000000001,"15min":0.89400000000000002},"min":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.65200000000000002},"max":{"1min":1.028,"5min":1.115,"15min":1.163},"last":1.028}]},{"osd":5,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.751,"15min":0.74399999999999999},"min":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.46500000000000002},"max":{"1min":0.80600000000000005,"5min":1.1160000000000001,"15min":1.347},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.91400000000000003,"15min":0.90100000000000002},"min":{"1min":0.622,"5min":0.59799999999999998,"15min":0.54500000000000004},"max":{"1min":1.1100000000000001,"5min":1.1899999999999999,"15min":1.2609999999999999},"last":1.0009999999999999}]},{"osd":6,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.98799999999999999,"15min":0.96299999999999997},"min":{"1min":0.71099999999999997,"5min":0.70499999999999996,"15min":0.47799999999999998},"max":{"1min":1.1830000000000001,"5min":1.2190000000000001,"15min":1.2569999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86699999999999999,"15min":0.85999999999999999},"min":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.57999999999999996},"max":{"1min":1.0620000000000001,"5min":1.1910000000000001,"15min":1.288},"last":0.61899999999999999}]},{"osd":7,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.66300000000000003,"5min":0.59399999999999997,"15min":0.58899999999999997},"max":{"1min":1.1040000000000001,"5min":1.1930000000000001,"15min":1.478},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.93400000000000005,"15min":0.93100000000000005},"min":{"1min":0.68899999999999995,"5min":0.61799999999999999,"15min":0.51100000000000001},"max":{"1min":1.1240000000000001,"5min":1.3660000000000001,"15min":1.4299999999999999},"last":0.68899999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542992,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65900000000000003,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.441},"max":{"1min":0.85599999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.754},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.621,"5min":0.57299999999999995,"15min":0.56299999999999994},"max":{"1min":0.89100000000000001,"5min":1.1080000000000001,"15min":1.1319999999999999},"last":0.92000000000000004}]},{"osd":2,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.48399999999999999},"max":{"1min":0.87,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.70599999999999996},"min":{"1min":0.53600000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94899999999999995,"5min":1.1359999999999999,"15min":1.2370000000000001},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.51100000000000001,"5min":0.48399999999999999,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":1.137,"15min":1.147},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.89700000000000002,"15min":0.90000000000000002},"min":{"1min":0.72399999999999998,"5min":0.63100000000000001,"15min":0.54400000000000004},"max":{"1min":1.0820000000000001,"5min":1.2010000000000001,"15min":1.22},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.63500000000000001},"max":{"1min":1.0409999999999999,"5min":1.105,"15min":1.196},"last":1.056},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.218,"5min":1.218,"15min":1.26},"last":0.78500000000000003}]},{"osd":5,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.94199999999999995,"15min":0.93000000000000005},"min":{"1min":0.72199999999999998,"5min":0.71099999999999997,"15min":0.64700000000000002},"max":{"1min":1.3959999999999999,"5min":1.399,"15min":1.399},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.006,"15min":1.0149999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1839999999999999,"5min":1.232,"15min":1.3520000000000001},"last":0.97899999999999998}]},{"osd":6,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.92700000000000005,"15min":0.94899999999999995},"min":{"1min":0.72999999999999998,"5min":0.69999999999999996,"15min":0.69599999999999995},"max":{"1min":1.2010000000000001,"5min":1.3400000000000001,"15min":1.3740000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.028,"15min":1.016},"min":{"1min":0.77200000000000002,"5min":0.73699999999999999,"15min":0.66500000000000004},"max":{"1min":1.304,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.0049999999999999}]},{"osd":7,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.071,"15min":1.0669999999999999},"min":{"1min":0.85899999999999999,"5min":0.79300000000000004,"15min":0.69999999999999996},"max":{"1min":1.2829999999999999,"5min":1.5149999999999999,"15min":1.7310000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.0369999999999999,"15min":1.048},"min":{"1min":0.83299999999999996,"5min":0.78800000000000003,"15min":0.73699999999999999},"max":{"1min":1.337,"5min":1.337,"15min":1.498},"last":1.127}]}]}]}} 2022-07-21T02:43:43.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-21T02:43:43.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3973fff700 1 Processor -- start 2022-07-21T02:43:43.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3973fff700 1 -- start start 2022-07-21T02:43:43.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3973fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f39740b01a0 0x7f39740a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3973fff700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f39740afd60 0x7f39740a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3973fff700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39740a7250 con 0x7f39740b2710 2022-07-21T02:43:43.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3973fff700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39740a7480 con 0x7f39740afd60 2022-07-21T02:43:43.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3973fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39740a7720 con 0x7f39740b01a0 2022-07-21T02:43:43.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f39737fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f39740afd60 0x7f39740a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f39737fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f39740afd60 0x7f39740a8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52530/0 (socket says 172.21.15.3:52530) 2022-07-21T02:43:43.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f39737fe700 1 -- 172.21.15.3:0/1019117239 learned_addr learned my addr 172.21.15.3:0/1019117239 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:43.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f39727fc700 1 --2- 172.21.15.3:0/1019117239 >> v2:172.21.15.143:3300/0 conn(0x7f39740b01a0 0x7f39740a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f39737fe700 1 -- 172.21.15.3:0/1019117239 >> v2:172.21.15.143:3300/0 conn(0x7f39740b01a0 msgr2=0x7f39740a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:43.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f39737fe700 1 --2- 172.21.15.3:0/1019117239 >> v2:172.21.15.143:3300/0 conn(0x7f39740b01a0 0x7f39740a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f39737fe700 1 -- 172.21.15.3:0/1019117239 >> v1:172.21.15.3:6789/0 conn(0x7f39740b2710 legacy=0x7f397414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f39737fe700 1 -- 172.21.15.3:0/1019117239 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39740a79c0 con 0x7f39740afd60 2022-07-21T02:43:43.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f39737fe700 1 --2- 172.21.15.3:0/1019117239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740afd60 0x7f39740a8a30 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f396c002f70 tx=0x7f396c006eb0).ready entity=mon.0 client_cookie=95504a133f204c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:43.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3971ffb700 1 -- 172.21.15.3:0/1019117239 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f396c00d610 con 0x7f39740afd60 2022-07-21T02:43:43.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3971ffb700 1 -- 172.21.15.3:0/1019117239 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c000e90 con 0x7f39740afd60 2022-07-21T02:43:43.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.922+0000 7f3971ffb700 1 -- 172.21.15.3:0/1019117239 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f396c00d9a0 con 0x7f39740afd60 2022-07-21T02:43:43.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 -- 172.21.15.3:0/1019117239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740afd60 msgr2=0x7f39740a8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 --2- 172.21.15.3:0/1019117239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740afd60 0x7f39740a8a30 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f396c002f70 tx=0x7f396c006eb0).stop 2022-07-21T02:43:43.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 -- 172.21.15.3:0/1019117239 shutdown_connections 2022-07-21T02:43:43.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 --2- 172.21.15.3:0/1019117239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740afd60 0x7f39740a8a30 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 --2- 172.21.15.3:0/1019117239 >> v2:172.21.15.143:3300/0 conn(0x7f39740b01a0 0x7f39740a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 -- 172.21.15.3:0/1019117239 >> 172.21.15.3:0/1019117239 conn(0x7f3974060310 msgr2=0x7f39740b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:43.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 -- 172.21.15.3:0/1019117239 shutdown_connections 2022-07-21T02:43:43.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 -- 172.21.15.3:0/1019117239 wait complete. 2022-07-21T02:43:43.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 Processor -- start 2022-07-21T02:43:43.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 -- start start 2022-07-21T02:43:43.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39740afd60 0x7f397414cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740b01a0 0x7f39740ab910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39740b2710 0x7f3974066fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39740a8d30 con 0x7f39740b01a0 2022-07-21T02:43:43.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39740aa450 con 0x7f39740afd60 2022-07-21T02:43:43.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3973fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39740aa6f0 con 0x7f39740b2710 2022-07-21T02:43:43.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f39727fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740b01a0 0x7f39740ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f39727fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740b01a0 0x7f39740ab910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52534/0 (socket says 172.21.15.3:52534) 2022-07-21T02:43:43.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f39727fc700 1 -- 172.21.15.3:0/2090580405 learned_addr learned my addr 172.21.15.3:0/2090580405 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:43.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f39737fe700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39740b2710 0x7f3974066fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.926+0000 7f3972ffd700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39740afd60 0x7f397414cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f39727fc700 1 -- 172.21.15.3:0/2090580405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39740b2710 msgr2=0x7f3974066fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f39727fc700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39740b2710 0x7f3974066fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f39727fc700 1 -- 172.21.15.3:0/2090580405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39740afd60 msgr2=0x7f397414cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:43.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f39727fc700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39740afd60 0x7f397414cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:43.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f39727fc700 1 -- 172.21.15.3:0/2090580405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c00b1e0 con 0x7f39740b01a0 2022-07-21T02:43:43.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f39727fc700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740b01a0 0x7f39740ab910 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f396400a000 tx=0x7f3964008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f3963fff700 1 -- 172.21.15.3:0/2090580405 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3964013620 con 0x7f39740b01a0 2022-07-21T02:43:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f3963fff700 1 -- 172.21.15.3:0/2090580405 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3964013780 con 0x7f39740b01a0 2022-07-21T02:43:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f3963fff700 1 -- 172.21.15.3:0/2090580405 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3964013ae0 con 0x7f39740b01a0 2022-07-21T02:43:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39740aa990 con 0x7f39740b01a0 2022-07-21T02:43:43.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397414d430 con 0x7f39740b01a0 2022-07-21T02:43:43.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39740574e0 con 0x7f39740b01a0 2022-07-21T02:43:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f3963fff700 1 -- 172.21.15.3:0/2090580405 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3964013c40 con 0x7f39740b01a0 2022-07-21T02:43:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f3963fff700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3950053a70 0x7f3950055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.930+0000 7f3963fff700 1 -- 172.21.15.3:0/2090580405 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f396406bea0 con 0x7f39740b01a0 2022-07-21T02:43:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.934+0000 7f3972ffd700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3950053a70 0x7f3950055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.934+0000 7f3972ffd700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3950053a70 0x7f3950055f30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f396800d000 tx=0x7f3968007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:43.938+0000 7f3963fff700 1 -- 172.21.15.3:0/2090580405 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3964042ee0 con 0x7f39740b01a0 2022-07-21T02:43:44.066 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.066+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3974069a70 con 0x7f39740b01a0 2022-07-21T02:43:44.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.066+0000 7f3963fff700 1 -- 172.21.15.3:0/2090580405 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+13551 (secure 0 0 0) 0x7f3964046500 con 0x7f39740b01a0 2022-07-21T02:43:44.069 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T02:43:44.070 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":164,"fsid":"033d20cc-eff0-4d91-a7d1-56acf083b1d2","created":"2022-07-21T01:33:06.656628+0000","modified":"2022-07-21T01:43:12.602152+0000","last_up_change":"2022-07-21T01:33:14.593772+0000","last_in_change":"2022-07-21T01:33:14.593772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-21T01:33:09.606566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-21T01:33:18.012897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"098a3930-1b74-4dbc-980c-5137ac329c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":22896},{"type":"v1","addr":"172.21.15.3:6816","nonce":22896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6819","nonce":22896},{"type":"v1","addr":"172.21.15.3:6822","nonce":22896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":22896},{"type":"v1","addr":"172.21.15.3:6833","nonce":22896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6827","nonce":22896},{"type":"v1","addr":"172.21.15.3:6830","nonce":22896}]},"public_addr":"172.21.15.3:6816/22896","cluster_addr":"172.21.15.3:6822/22896","heartbeat_back_addr":"172.21.15.3:6833/22896","heartbeat_front_addr":"172.21.15.3:6830/22896","state":["exists","up"]},{"osd":1,"uuid":"ec4680af-1feb-47f3-949c-315962389518","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":22890},{"type":"v1","addr":"172.21.15.3:6811","nonce":22890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":22890},{"type":"v1","addr":"172.21.15.3:6815","nonce":22890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":22890},{"type":"v1","addr":"172.21.15.3:6826","nonce":22890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":22890},{"type":"v1","addr":"172.21.15.3:6821","nonce":22890}]},"public_addr":"172.21.15.3:6811/22890","cluster_addr":"172.21.15.3:6815/22890","heartbeat_back_addr":"172.21.15.3:6826/22890","heartbeat_front_addr":"172.21.15.3:6821/22890","state":["exists","up"]},{"osd":2,"uuid":"e321c85f-57d4-4b3d-92c6-18843b5f43aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":22877},{"type":"v1","addr":"172.21.15.3:6803","nonce":22877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":22877},{"type":"v1","addr":"172.21.15.3:6805","nonce":22877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":22877},{"type":"v1","addr":"172.21.15.3:6809","nonce":22877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":22877},{"type":"v1","addr":"172.21.15.3:6807","nonce":22877}]},"public_addr":"172.21.15.3:6803/22877","cluster_addr":"172.21.15.3:6805/22877","heartbeat_back_addr":"172.21.15.3:6809/22877","heartbeat_front_addr":"172.21.15.3:6807/22877","state":["exists","up"]},{"osd":3,"uuid":"7b3e9c1c-2503-46ca-871b-61bd90000981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":22884},{"type":"v1","addr":"172.21.15.3:6817","nonce":22884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":22884},{"type":"v1","addr":"172.21.15.3:6823","nonce":22884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6829","nonce":22884},{"type":"v1","addr":"172.21.15.3:6831","nonce":22884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6825","nonce":22884},{"type":"v1","addr":"172.21.15.3:6828","nonce":22884}]},"public_addr":"172.21.15.3:6817/22884","cluster_addr":"172.21.15.3:6823/22884","heartbeat_back_addr":"172.21.15.3:6831/22884","heartbeat_front_addr":"172.21.15.3:6828/22884","state":["exists","up"]},{"osd":4,"uuid":"86ac80f1-2bd0-43c8-8335-a06fe0ac9606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":23339},{"type":"v1","addr":"172.21.15.143:6809","nonce":23339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":23339},{"type":"v1","addr":"172.21.15.143:6811","nonce":23339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":23339},{"type":"v1","addr":"172.21.15.143:6815","nonce":23339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":23339},{"type":"v1","addr":"172.21.15.143:6813","nonce":23339}]},"public_addr":"172.21.15.143:6809/23339","cluster_addr":"172.21.15.143:6811/23339","heartbeat_back_addr":"172.21.15.143:6815/23339","heartbeat_front_addr":"172.21.15.143:6813/23339","state":["exists","up"]},{"osd":5,"uuid":"25f35213-a67d-47e3-ac21-0ec2654b7e56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":23340},{"type":"v1","addr":"172.21.15.143:6801","nonce":23340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":23340},{"type":"v1","addr":"172.21.15.143:6803","nonce":23340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":23340},{"type":"v1","addr":"172.21.15.143:6807","nonce":23340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":23340},{"type":"v1","addr":"172.21.15.143:6805","nonce":23340}]},"public_addr":"172.21.15.143:6801/23340","cluster_addr":"172.21.15.143:6803/23340","heartbeat_back_addr":"172.21.15.143:6807/23340","heartbeat_front_addr":"172.21.15.143:6805/23340","state":["exists","up"]},{"osd":6,"uuid":"e9a6d67d-c2b0-48bb-b7e3-7a3c803503e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":23342},{"type":"v1","addr":"172.21.15.143:6825","nonce":23342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":23342},{"type":"v1","addr":"172.21.15.143:6827","nonce":23342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":23342},{"type":"v1","addr":"172.21.15.143:6831","nonce":23342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":23342},{"type":"v1","addr":"172.21.15.143:6829","nonce":23342}]},"public_addr":"172.21.15.143:6825/23342","cluster_addr":"172.21.15.143:6827/23342","heartbeat_back_addr":"172.21.15.143:6831/23342","heartbeat_front_addr":"172.21.15.143:6829/23342","state":["exists","up"]},{"osd":7,"uuid":"f0260c49-f670-47ef-b037-b5d9ace42688","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":23341},{"type":"v1","addr":"172.21.15.143:6817","nonce":23341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":23341},{"type":"v1","addr":"172.21.15.143:6819","nonce":23341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":23341},{"type":"v1","addr":"172.21.15.143:6823","nonce":23341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":23341},{"type":"v1","addr":"172.21.15.143:6821","nonce":23341}]},"public_addr":"172.21.15.143:6817/23341","cluster_addr":"172.21.15.143:6819/23341","heartbeat_back_addr":"172.21.15.143:6823/23341","heartbeat_front_addr":"172.21.15.143:6821/23341","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.616595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.374044+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.657733+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.654519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.426744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.113656+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.413566+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-21T01:33:12.398720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.3:6837/1787668325":"2022-07-22T01:43:08.803052+0000","172.21.15.3:6836/1787668325":"2022-07-22T01:43:08.803052+0000"},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-21T02:43:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3950053a70 msgr2=0x7f3950055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3950053a70 0x7f3950055f30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f396800d000 tx=0x7f3968007b90).stop 2022-07-21T02:43:44.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740b01a0 msgr2=0x7f39740ab910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:44.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740b01a0 0x7f39740ab910 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f396400a000 tx=0x7f3964008040).stop 2022-07-21T02:43:44.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 shutdown_connections 2022-07-21T02:43:44.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39740b2710 0x7f3974066fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:44.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39740b01a0 0x7f39740ab910 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:44.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3950053a70 0x7f3950055f30 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:44.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 --2- 172.21.15.3:0/2090580405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39740afd60 0x7f397414cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:44.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 >> 172.21.15.3:0/2090580405 conn(0x7f3974060310 msgr2=0x7f3974149d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:44.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 shutdown_connections 2022-07-21T02:43:44.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:44.070+0000 7f3973fff700 1 -- 172.21.15.3:0/2090580405 wait complete. 2022-07-21T02:43:44.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:43:44.666 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:43:44.670 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:43:44.674 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:43:45.082 INFO:tasks.ceph:Scrubbing osd.0 2022-07-21T02:43:45.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2022-07-21T02:43:45.203 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fe57ed700 1 Processor -- start 2022-07-21T02:43:45.204 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fe57ed700 1 -- start start 2022-07-21T02:43:45.207 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fe57ed700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5fe00b2730 0x7f5fe00666b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fe57ed700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5fe0066ba0 0x7f5fe014dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fe57ed700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00654c0 con 0x7f5fe00b3190 2022-07-21T02:43:45.208 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fe57ed700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe0062190 con 0x7f5fe00b2730 2022-07-21T02:43:45.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fe57ed700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe00b3570 con 0x7f5fe0066ba0 2022-07-21T02:43:45.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fdf7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5fe00b2730 0x7f5fe00666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.209 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fdf7fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5fe00b2730 0x7f5fe00666b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52542/0 (socket says 172.21.15.3:52542) 2022-07-21T02:43:45.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fdf7fe700 1 -- 172.21.15.3:0/2669120933 learned_addr learned my addr 172.21.15.3:0/2669120933 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:45.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fe4fec700 1 --2- 172.21.15.3:0/2669120933 >> v2:172.21.15.143:3300/0 conn(0x7f5fe0066ba0 0x7f5fe014dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.210 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fdf7fe700 1 -- 172.21.15.3:0/2669120933 >> v2:172.21.15.143:3300/0 conn(0x7f5fe0066ba0 msgr2=0x7f5fe014dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:45.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fdf7fe700 1 --2- 172.21.15.3:0/2669120933 >> v2:172.21.15.143:3300/0 conn(0x7f5fe0066ba0 0x7f5fe014dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fdf7fe700 1 -- 172.21.15.3:0/2669120933 >> v1:172.21.15.143:6789/0 conn(0x7f5fe00b3190 legacy=0x7f5fe0060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.202+0000 7f5fdf7fe700 1 -- 172.21.15.3:0/2669120933 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe005fbf0 con 0x7f5fe00b2730 2022-07-21T02:43:45.211 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fdf7fe700 1 --2- 172.21.15.3:0/2669120933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 0x7f5fe00666b0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7f5fc8002f70 tx=0x7f5fc8006eb0).ready entity=mon.0 client_cookie=88066641982f6a81 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fdeffd700 1 -- 172.21.15.3:0/2669120933 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc800d610 con 0x7f5fe00b2730 2022-07-21T02:43:45.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fdeffd700 1 -- 172.21.15.3:0/2669120933 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc8000e90 con 0x7f5fe00b2730 2022-07-21T02:43:45.212 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fdeffd700 1 -- 172.21.15.3:0/2669120933 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc800d9a0 con 0x7f5fe00b2730 2022-07-21T02:43:45.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2669120933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 msgr2=0x7f5fe00666b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fe57ed700 1 --2- 172.21.15.3:0/2669120933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 0x7f5fe00666b0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7f5fc8002f70 tx=0x7f5fc8006eb0).stop 2022-07-21T02:43:45.213 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2669120933 shutdown_connections 2022-07-21T02:43:45.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fe57ed700 1 --2- 172.21.15.3:0/2669120933 >> v2:172.21.15.143:3300/0 conn(0x7f5fe0066ba0 0x7f5fe014dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fe57ed700 1 --2- 172.21.15.3:0/2669120933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 0x7f5fe00666b0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.214 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2669120933 >> 172.21.15.3:0/2669120933 conn(0x7f5fe0063870 msgr2=0x7f5fe0063c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:45.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2669120933 shutdown_connections 2022-07-21T02:43:45.215 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.206+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2669120933 wait complete. 2022-07-21T02:43:45.219 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fe57ed700 1 Processor -- start 2022-07-21T02:43:45.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fe57ed700 1 -- start start 2022-07-21T02:43:45.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fe57ed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe0066ba0 0x7f5fe0155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.220 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fe57ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 0x7f5fe0156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fe57ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe00b3190 0x7f5fe00af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fe57ed700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00ade00 con 0x7f5fe00b2730 2022-07-21T02:43:45.221 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fe57ed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00ad4f0 con 0x7f5fe0066ba0 2022-07-21T02:43:45.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fe57ed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00adbc0 con 0x7f5fe00b3190 2022-07-21T02:43:45.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 0x7f5fe0156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 0x7f5fe0156310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52548/0 (socket says 172.21.15.3:52548) 2022-07-21T02:43:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdf7fe700 1 -- 172.21.15.3:0/2003520658 learned_addr learned my addr 172.21.15.3:0/2003520658 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdffff700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe0066ba0 0x7f5fe0155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fe4fec700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe00b3190 0x7f5fe00af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdf7fe700 1 -- 172.21.15.3:0/2003520658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe00b3190 msgr2=0x7f5fe00af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdf7fe700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fe00b3190 0x7f5fe00af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.224 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdf7fe700 1 -- 172.21.15.3:0/2003520658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe0066ba0 msgr2=0x7f5fe0155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdf7fe700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fe0066ba0 0x7f5fe0155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdf7fe700 1 -- 172.21.15.3:0/2003520658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc800b1e0 con 0x7f5fe00b2730 2022-07-21T02:43:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdf7fe700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 0x7f5fe0156310 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7f5fc8002f70 tx=0x7f5fc8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.210+0000 7f5fdd7fa700 1 -- 172.21.15.3:0/2003520658 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc800d990 con 0x7f5fe00b2730 2022-07-21T02:43:45.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fdd7fa700 1 -- 172.21.15.3:0/2003520658 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc8011b40 con 0x7f5fe00b2730 2022-07-21T02:43:45.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe00ab2e0 con 0x7f5fe00b2730 2022-07-21T02:43:45.226 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fdd7fa700 1 -- 172.21.15.3:0/2003520658 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc8011ea0 con 0x7f5fe00b2730 2022-07-21T02:43:45.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe00ad720 con 0x7f5fe00b2730 2022-07-21T02:43:45.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5fe00ad920 con 0x7f5fe00b2730 2022-07-21T02:43:45.227 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fdd7fa700 1 -- 172.21.15.3:0/2003520658 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5fc8025030 con 0x7f5fe00b2730 2022-07-21T02:43:45.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fdd7fa700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5fc4053a70 0x7f5fc4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fdd7fa700 1 -- 172.21.15.3:0/2003520658 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f5fc806cef0 con 0x7f5fe00b2730 2022-07-21T02:43:45.228 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fdffff700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5fc4053a70 0x7f5fc4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fdd7fa700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f5fc405a700 0x7f5fc405cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fdd7fa700 1 -- 172.21.15.3:0/2003520658 --> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fc405d180 con 0x7f5fc405a700 2022-07-21T02:43:45.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fdd7fa700 1 -- 172.21.15.3:0/2003520658 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fc806f470 con 0x7f5fe00b2730 2022-07-21T02:43:45.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.214+0000 7f5fe4fec700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f5fc405a700 0x7f5fc405cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.218+0000 7f5fdffff700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5fc4053a70 0x7f5fc4055f30 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f5fd000d000 tx=0x7f5fd0007ef0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.218+0000 7f5fe4fec700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f5fc405a700 0x7f5fc405cba0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.218+0000 7f5fdd7fa700 1 -- 172.21.15.3:0/2003520658 <== osd.0 v2:172.21.15.3:6813/22896 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f5fc405d180 con 0x7f5fc405a700 2022-07-21T02:43:45.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.234+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 --> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5fe00afc20 con 0x7f5fc405a700 2022-07-21T02:43:45.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fdd7fa700 1 -- 172.21.15.3:0/2003520658 <== osd.0 v2:172.21.15.3:6813/22896 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5fe00afc20 con 0x7f5fc405a700 2022-07-21T02:43:45.239 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T02:43:45.239 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-21T02:43:45.240 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T02:43:45.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f5fc405a700 msgr2=0x7f5fc405cba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6813/22896,v1:172.21.15.3:6816/22896] conn(0x7f5fc405a700 0x7f5fc405cba0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5fc4053a70 msgr2=0x7f5fc4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5fc4053a70 0x7f5fc4055f30 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f5fd000d000 tx=0x7f5fd0007ef0).stop 2022-07-21T02:43:45.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 msgr2=0x7f5fe0156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 --2- 172.21.15.3:0/2003520658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fe00b2730 0x7f5fe0156310 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7f5fc8002f70 tx=0x7f5fc8015000).stop 2022-07-21T02:43:45.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe4fec700 1 -- 172.21.15.3:0/2003520658 reap_dead start 2022-07-21T02:43:45.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 shutdown_connections 2022-07-21T02:43:45.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 >> 172.21.15.3:0/2003520658 conn(0x7f5fe0063870 msgr2=0x7f5fe0143960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:45.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 shutdown_connections 2022-07-21T02:43:45.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.238+0000 7f5fe57ed700 1 -- 172.21.15.3:0/2003520658 wait complete. 2022-07-21T02:43:45.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2022-07-21T02:43:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b759e700 1 Processor -- start 2022-07-21T02:43:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b759e700 1 -- start start 2022-07-21T02:43:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f47b805c960 0x7f47b80b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b813f520 0x7f47b80aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b805cd40 con 0x7f47b805d240 2022-07-21T02:43:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b80ad6d0 con 0x7f47b805c960 2022-07-21T02:43:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b8067f00 con 0x7f47b813f520 2022-07-21T02:43:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b813f520 0x7f47b80aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b659c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f47b805c960 0x7f47b80b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b813f520 0x7f47b80aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.3:59476/0 (socket says 172.21.15.3:59476) 2022-07-21T02:43:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.366+0000 7f47b6d9d700 1 -- 172.21.15.3:0/1933367846 learned_addr learned my addr 172.21.15.3:0/1933367846 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:45.373 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b6d9d700 1 -- 172.21.15.3:0/1933367846 >> v2:172.21.15.143:3300/0 conn(0x7f47b805c960 msgr2=0x7f47b80b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b5d9b700 1 -- 172.21.15.3:0/1933367846 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587002618 0 0) 0x7f47b805cd40 con 0x7f47b805d240 2022-07-21T02:43:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b6d9d700 1 --2- 172.21.15.3:0/1933367846 >> v2:172.21.15.143:3300/0 conn(0x7f47b805c960 0x7f47b80b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b6d9d700 1 -- 172.21.15.3:0/1933367846 >> v1:172.21.15.143:6789/0 conn(0x7f47b805d240 legacy=0x7f47b80acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b6d9d700 1 -- 172.21.15.3:0/1933367846 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b80b1920 con 0x7f47b813f520 2022-07-21T02:43:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b6d9d700 1 --2- 172.21.15.3:0/1933367846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b813f520 0x7f47b80aee70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f47b00032f0 tx=0x7f47b0009290).ready entity=mon.2 client_cookie=2381d4afb35df0f3 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.375 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b5d9b700 1 -- 172.21.15.3:0/1933367846 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47b000ba90 con 0x7f47b813f520 2022-07-21T02:43:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b5d9b700 1 -- 172.21.15.3:0/1933367846 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b000bbf0 con 0x7f47b813f520 2022-07-21T02:43:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b5d9b700 1 -- 172.21.15.3:0/1933367846 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47b0010470 con 0x7f47b813f520 2022-07-21T02:43:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 -- 172.21.15.3:0/1933367846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b813f520 msgr2=0x7f47b80aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.380 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 --2- 172.21.15.3:0/1933367846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b813f520 0x7f47b80aee70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f47b00032f0 tx=0x7f47b0009290).stop 2022-07-21T02:43:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 -- 172.21.15.3:0/1933367846 shutdown_connections 2022-07-21T02:43:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 --2- 172.21.15.3:0/1933367846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b813f520 0x7f47b80aee70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 --2- 172.21.15.3:0/1933367846 >> v2:172.21.15.143:3300/0 conn(0x7f47b805c960 0x7f47b80b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 -- 172.21.15.3:0/1933367846 >> 172.21.15.3:0/1933367846 conn(0x7f47b8060310 msgr2=0x7f47b80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 -- 172.21.15.3:0/1933367846 shutdown_connections 2022-07-21T02:43:45.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 -- 172.21.15.3:0/1933367846 wait complete. 2022-07-21T02:43:45.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 Processor -- start 2022-07-21T02:43:45.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.370+0000 7f47b759e700 1 -- start start 2022-07-21T02:43:45.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47b805c960 0x7f47b81429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b805d240 0x7f47b80b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b813f520 0x7f47b8152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b80ab670 con 0x7f47b805c960 2022-07-21T02:43:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b80ab8a0 con 0x7f47b813f520 2022-07-21T02:43:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b8142260 con 0x7f47b805d240 2022-07-21T02:43:45.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47b805c960 0x7f47b81429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47b805c960 0x7f47b81429a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52562/0 (socket says 172.21.15.3:52562) 2022-07-21T02:43:45.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b659c700 1 -- 172.21.15.3:0/3771399837 learned_addr learned my addr 172.21.15.3:0/3771399837 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:45.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b6d9d700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b813f520 0x7f47b8152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47affff700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b805d240 0x7f47b80b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b659c700 1 -- 172.21.15.3:0/3771399837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b805d240 msgr2=0x7f47b80b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b659c700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b805d240 0x7f47b80b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b659c700 1 -- 172.21.15.3:0/3771399837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b813f520 msgr2=0x7f47b8152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b659c700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b813f520 0x7f47b8152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b659c700 1 -- 172.21.15.3:0/3771399837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b0009530 con 0x7f47b805c960 2022-07-21T02:43:45.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b659c700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47b805c960 0x7f47b81429a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7f47a800c000 tx=0x7f47a8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47af7fe700 1 -- 172.21.15.3:0/3771399837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47a8010780 con 0x7f47b805c960 2022-07-21T02:43:45.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47af7fe700 1 -- 172.21.15.3:0/3771399837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a80108e0 con 0x7f47b805c960 2022-07-21T02:43:45.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b8142500 con 0x7f47b805c960 2022-07-21T02:43:45.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47af7fe700 1 -- 172.21.15.3:0/3771399837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47a8010c40 con 0x7f47b805c960 2022-07-21T02:43:45.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.374+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b80ab140 con 0x7f47b805c960 2022-07-21T02:43:45.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.378+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b80b1bc0 con 0x7f47b805c960 2022-07-21T02:43:45.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.378+0000 7f47af7fe700 1 -- 172.21.15.3:0/3771399837 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f47a801c460 con 0x7f47b805c960 2022-07-21T02:43:45.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.378+0000 7f47af7fe700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f479c053a70 0x7f479c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.378+0000 7f47af7fe700 1 -- 172.21.15.3:0/3771399837 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f47a806c1e0 con 0x7f47b805c960 2022-07-21T02:43:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.378+0000 7f47affff700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f479c053a70 0x7f479c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.378+0000 7f47affff700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f479c053a70 0x7f479c055f30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f47a000d000 tx=0x7f47a0007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.382+0000 7f47af7fe700 1 -- 172.21.15.3:0/3771399837 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f47a8043110 con 0x7f47b805c960 2022-07-21T02:43:45.509 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f47b80574e0 con 0x7f479c053a70 2022-07-21T02:43:45.510 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47af7fe700 1 -- 172.21.15.3:0/3771399837 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f47b80574e0 con 0x7f479c053a70 2022-07-21T02:43:45.511 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f479c053a70 msgr2=0x7f479c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f479c053a70 0x7f479c055f30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f47a000d000 tx=0x7f47a0007a20).stop 2022-07-21T02:43:45.512 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47b805c960 msgr2=0x7f47b81429a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47b805c960 0x7f47b81429a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7f47a800c000 tx=0x7f47a8006df0).stop 2022-07-21T02:43:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 shutdown_connections 2022-07-21T02:43:45.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f479c053a70 0x7f479c055f30 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b813f520 0x7f47b8152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.514 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47b805d240 0x7f47b80b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 --2- 172.21.15.3:0/3771399837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47b805c960 0x7f47b81429a0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 >> 172.21.15.3:0/3771399837 conn(0x7f47b8060310 msgr2=0x7f47b80b3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 shutdown_connections 2022-07-21T02:43:45.516 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.510+0000 7f47b759e700 1 -- 172.21.15.3:0/3771399837 wait complete. 2022-07-21T02:43:45.516 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 0 to deep-scrub 2022-07-21T02:43:45.522 INFO:tasks.ceph:Scrubbing osd.1 2022-07-21T02:43:45.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2022-07-21T02:43:45.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 Processor -- start 2022-07-21T02:43:45.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 -- start start 2022-07-21T02:43:45.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86040614b0 0x7f8604061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86040ad750 0x7f86040b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.634 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040aee80 con 0x7f86041405d0 2022-07-21T02:43:45.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86040b70b0 con 0x7f86040ad750 2022-07-21T02:43:45.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86040af0b0 con 0x7f86040614b0 2022-07-21T02:43:45.635 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f86037fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86040ad750 0x7f86040b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f86037fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f86040ad750 0x7f86040b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52574/0 (socket says 172.21.15.3:52574) 2022-07-21T02:43:45.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f86037fe700 1 -- 172.21.15.3:0/1555808493 learned_addr learned my addr 172.21.15.3:0/1555808493 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:45.636 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8603fff700 1 --2- 172.21.15.3:0/1555808493 >> v2:172.21.15.143:3300/0 conn(0x7f86040614b0 0x7f8604061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f86037fe700 1 -- 172.21.15.3:0/1555808493 >> v2:172.21.15.143:3300/0 conn(0x7f86040614b0 msgr2=0x7f8604061890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:45.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f86037fe700 1 --2- 172.21.15.3:0/1555808493 >> v2:172.21.15.143:3300/0 conn(0x7f86040614b0 0x7f8604061890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.637 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f86037fe700 1 -- 172.21.15.3:0/1555808493 >> v1:172.21.15.143:6789/0 conn(0x7f86041405d0 legacy=0x7f86041466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f86037fe700 1 -- 172.21.15.3:0/1555808493 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040af350 con 0x7f86040ad750 2022-07-21T02:43:45.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8603fff700 1 --2- 172.21.15.3:0/1555808493 >> v2:172.21.15.143:3300/0 conn(0x7f86040614b0 0x7f8604061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T02:43:45.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f86037fe700 1 --2- 172.21.15.3:0/1555808493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 0x7f86040b9ff0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f86040aed30 tx=0x7f85f8006eb0).ready entity=mon.0 client_cookie=26cfea149ce2d289 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8602ffd700 1 -- 172.21.15.3:0/1555808493 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f800d610 con 0x7f86040ad750 2022-07-21T02:43:45.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8602ffd700 1 -- 172.21.15.3:0/1555808493 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f8000e90 con 0x7f86040ad750 2022-07-21T02:43:45.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8602ffd700 1 -- 172.21.15.3:0/1555808493 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f800d960 con 0x7f86040ad750 2022-07-21T02:43:45.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 -- 172.21.15.3:0/1555808493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 msgr2=0x7f86040b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 --2- 172.21.15.3:0/1555808493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 0x7f86040b9ff0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f86040aed30 tx=0x7f85f8006eb0).stop 2022-07-21T02:43:45.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 -- 172.21.15.3:0/1555808493 shutdown_connections 2022-07-21T02:43:45.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 --2- 172.21.15.3:0/1555808493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 0x7f86040b9ff0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 --2- 172.21.15.3:0/1555808493 >> v2:172.21.15.143:3300/0 conn(0x7f86040614b0 0x7f8604061890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 -- 172.21.15.3:0/1555808493 >> 172.21.15.3:0/1555808493 conn(0x7f8604066f70 msgr2=0x7f8604067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:45.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 -- 172.21.15.3:0/1555808493 shutdown_connections 2022-07-21T02:43:45.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.630+0000 7f8609500700 1 -- 172.21.15.3:0/1555808493 wait complete. 2022-07-21T02:43:45.646 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8609500700 1 Processor -- start 2022-07-21T02:43:45.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8609500700 1 -- start start 2022-07-21T02:43:45.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8609500700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86040614b0 0x7f86040b8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8609500700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 0x7f86040b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8609500700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86041405d0 0x7f860409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8609500700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86040a16b0 con 0x7f86040ad750 2022-07-21T02:43:45.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8609500700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86040b3f70 con 0x7f86041405d0 2022-07-21T02:43:45.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8609500700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86040b2ba0 con 0x7f86040614b0 2022-07-21T02:43:45.649 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f86037fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 0x7f86040b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f86037fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 0x7f86040b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52576/0 (socket says 172.21.15.3:52576) 2022-07-21T02:43:45.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f86037fe700 1 -- 172.21.15.3:0/483684145 learned_addr learned my addr 172.21.15.3:0/483684145 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:45.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8608cff700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86041405d0 0x7f860409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8603fff700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86040614b0 0x7f86040b8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f86037fe700 1 -- 172.21.15.3:0/483684145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86040614b0 msgr2=0x7f86040b8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.651 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f86037fe700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86040614b0 0x7f86040b8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f86037fe700 1 -- 172.21.15.3:0/483684145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86041405d0 msgr2=0x7f860409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f86037fe700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86041405d0 0x7f860409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.652 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f86037fe700 1 -- 172.21.15.3:0/483684145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f800b1e0 con 0x7f86040ad750 2022-07-21T02:43:45.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f8603fff700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86040614b0 0x7f86040b8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T02:43:45.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.634+0000 7f86037fe700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 0x7f86040b26b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7f86040aed30 tx=0x7f85f8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.653 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f86017fa700 1 -- 172.21.15.3:0/483684145 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f800d990 con 0x7f86040ad750 2022-07-21T02:43:45.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f86017fa700 1 -- 172.21.15.3:0/483684145 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f800bec0 con 0x7f86040ad750 2022-07-21T02:43:45.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86040a0cd0 con 0x7f86040ad750 2022-07-21T02:43:45.654 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f86017fa700 1 -- 172.21.15.3:0/483684145 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f8002f50 con 0x7f86040ad750 2022-07-21T02:43:45.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86040a1210 con 0x7f86040ad750 2022-07-21T02:43:45.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f86040af5f0 con 0x7f86040ad750 2022-07-21T02:43:45.655 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f86017fa700 1 -- 172.21.15.3:0/483684145 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f85f8025030 con 0x7f86040ad750 2022-07-21T02:43:45.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f86017fa700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f85e4053950 0x7f85e4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f86017fa700 1 -- 172.21.15.3:0/483684145 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f85f806cfa0 con 0x7f86040ad750 2022-07-21T02:43:45.656 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f8603fff700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f85e4053950 0x7f85e4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f86017fa700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f85e405a560 0x7f85e405ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f86017fa700 1 -- 172.21.15.3:0/483684145 --> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e405d080 con 0x7f85e405a560 2022-07-21T02:43:45.657 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.638+0000 7f86017fa700 1 -- 172.21.15.3:0/483684145 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85f806f630 con 0x7f86040ad750 2022-07-21T02:43:45.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.642+0000 7f8608cff700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f85e405a560 0x7f85e405ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.658 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.642+0000 7f8603fff700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f85e4053950 0x7f85e4055e10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f85f4008000 tx=0x7f85f4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.642+0000 7f8608cff700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f85e405a560 0x7f85e405ca00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.659 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.642+0000 7f86017fa700 1 -- 172.21.15.3:0/483684145 <== osd.1 v2:172.21.15.3:6810/22890 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f85e405d080 con 0x7f85e405a560 2022-07-21T02:43:45.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 --> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f860409f310 con 0x7f85e405a560 2022-07-21T02:43:45.660 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f86017fa700 1 -- 172.21.15.3:0/483684145 <== osd.1 v2:172.21.15.3:6810/22890 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f860409f310 con 0x7f85e405a560 2022-07-21T02:43:45.661 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T02:43:45.661 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-21T02:43:45.662 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T02:43:45.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f85e405a560 msgr2=0x7f85e405ca00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.663 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6810/22890,v1:172.21.15.3:6811/22890] conn(0x7f85e405a560 0x7f85e405ca00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f85e4053950 msgr2=0x7f85e4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.664 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f85e4053950 0x7f85e4055e10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f85f4008000 tx=0x7f85f4008040).stop 2022-07-21T02:43:45.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 msgr2=0x7f86040b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 --2- 172.21.15.3:0/483684145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86040ad750 0x7f86040b26b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7f86040aed30 tx=0x7f85f8015000).stop 2022-07-21T02:43:45.665 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8608cff700 1 -- 172.21.15.3:0/483684145 reap_dead start 2022-07-21T02:43:45.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 shutdown_connections 2022-07-21T02:43:45.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 >> 172.21.15.3:0/483684145 conn(0x7f8604066f70 msgr2=0x7f860413fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:45.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 shutdown_connections 2022-07-21T02:43:45.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.658+0000 7f8609500700 1 -- 172.21.15.3:0/483684145 wait complete. 2022-07-21T02:43:45.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2022-07-21T02:43:45.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 Processor -- start 2022-07-21T02:43:45.782 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 -- start start 2022-07-21T02:43:45.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 0x7f28e414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.785 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f28e40b01a0 0x7f28e414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e40a7250 con 0x7f28e40afd60 2022-07-21T02:43:45.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e40a7480 con 0x7f28e40b01a0 2022-07-21T02:43:45.786 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40a7720 con 0x7f28e40b2710 2022-07-21T02:43:45.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e2ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f28e40b01a0 0x7f28e414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e2ffd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f28e40b01a0 0x7f28e414b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52588/0 (socket says 172.21.15.3:52588) 2022-07-21T02:43:45.787 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e2ffd700 1 -- 172.21.15.3:0/3328722901 learned_addr learned my addr 172.21.15.3:0/3328722901 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:45.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e37fe700 1 --2- 172.21.15.3:0/3328722901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 0x7f28e414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.788 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e2ffd700 1 -- 172.21.15.3:0/3328722901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 msgr2=0x7f28e414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:45.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e2ffd700 1 --2- 172.21.15.3:0/3328722901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 0x7f28e414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e2ffd700 1 -- 172.21.15.3:0/3328722901 >> v1:172.21.15.3:6789/0 conn(0x7f28e40afd60 legacy=0x7f28e40a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.789 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e2ffd700 1 -- 172.21.15.3:0/3328722901 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e40a79c0 con 0x7f28e40b01a0 2022-07-21T02:43:45.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e2ffd700 1 --2- 172.21.15.3:0/3328722901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40b01a0 0x7f28e414b4c0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f28dc0032f0 tx=0x7f28dc009290).ready entity=mon.0 client_cookie=c9f1c2a5968053f9 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e27fc700 1 -- 172.21.15.3:0/3328722901 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28dc00b8a0 con 0x7f28e40b01a0 2022-07-21T02:43:45.790 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e27fc700 1 -- 172.21.15.3:0/3328722901 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28dc00ba00 con 0x7f28e40b01a0 2022-07-21T02:43:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e27fc700 1 -- 172.21.15.3:0/3328722901 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28dc00bb70 con 0x7f28e40b01a0 2022-07-21T02:43:45.796 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 -- 172.21.15.3:0/3328722901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40b01a0 msgr2=0x7f28e414b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 --2- 172.21.15.3:0/3328722901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40b01a0 0x7f28e414b4c0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f28dc0032f0 tx=0x7f28dc009290).stop 2022-07-21T02:43:45.797 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 -- 172.21.15.3:0/3328722901 shutdown_connections 2022-07-21T02:43:45.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 --2- 172.21.15.3:0/3328722901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40b01a0 0x7f28e414b4c0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 --2- 172.21.15.3:0/3328722901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 0x7f28e414afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.798 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 -- 172.21.15.3:0/3328722901 >> 172.21.15.3:0/3328722901 conn(0x7f28e4060310 msgr2=0x7f28e40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:45.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 -- 172.21.15.3:0/3328722901 shutdown_connections 2022-07-21T02:43:45.799 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.782+0000 7f28e8994700 1 -- 172.21.15.3:0/3328722901 wait complete. 2022-07-21T02:43:45.803 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 Processor -- start 2022-07-21T02:43:45.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 -- start start 2022-07-21T02:43:45.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40afd60 0x7f28e40b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.804 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e40b01a0 0x7f28e40b5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 0x7f28e40ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.805 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40b3920 con 0x7f28e40afd60 2022-07-21T02:43:45.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40afd60 0x7f28e40b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.806 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40afd60 0x7f28e40b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52592/0 (socket says 172.21.15.3:52592) 2022-07-21T02:43:45.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e37fe700 1 -- 172.21.15.3:0/1233640796 learned_addr learned my addr 172.21.15.3:0/1233640796 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:45.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40ac0a0 con 0x7f28e40b01a0 2022-07-21T02:43:45.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40ac340 con 0x7f28e40b2710 2022-07-21T02:43:45.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e3fff700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 0x7f28e40ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e2ffd700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e40b01a0 0x7f28e40b5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e37fe700 1 -- 172.21.15.3:0/1233640796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 msgr2=0x7f28e40ab2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e37fe700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 0x7f28e40ab2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e37fe700 1 -- 172.21.15.3:0/1233640796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e40b01a0 msgr2=0x7f28e40b5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e37fe700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e40b01a0 0x7f28e40b5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e37fe700 1 -- 172.21.15.3:0/1233640796 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc009530 con 0x7f28e40afd60 2022-07-21T02:43:45.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e2ffd700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e40b01a0 0x7f28e40b5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T02:43:45.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e37fe700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40afd60 0x7f28e40b32f0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7f28cc0030b0 tx=0x7f28cc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e0ff9700 1 -- 172.21.15.3:0/1233640796 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28cc011780 con 0x7f28e40afd60 2022-07-21T02:43:45.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e0ff9700 1 -- 172.21.15.3:0/1233640796 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc0118e0 con 0x7f28e40afd60 2022-07-21T02:43:45.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e40ac5e0 con 0x7f28e40afd60 2022-07-21T02:43:45.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e0ff9700 1 -- 172.21.15.3:0/1233640796 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28cc011c40 con 0x7f28e40afd60 2022-07-21T02:43:45.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.786+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e40b3c00 con 0x7f28e40afd60 2022-07-21T02:43:45.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.790+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e40ac770 con 0x7f28e40afd60 2022-07-21T02:43:45.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.790+0000 7f28e0ff9700 1 -- 172.21.15.3:0/1233640796 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f28cc01c460 con 0x7f28e40afd60 2022-07-21T02:43:45.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.790+0000 7f28e0ff9700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28c8053aa0 0x7f28c8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:45.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.790+0000 7f28e0ff9700 1 -- 172.21.15.3:0/1233640796 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f28cc06bec0 con 0x7f28e40afd60 2022-07-21T02:43:45.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.790+0000 7f28e2ffd700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28c8053aa0 0x7f28c8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:45.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.790+0000 7f28e2ffd700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28c8053aa0 0x7f28c8055f60 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f28dc019000 tx=0x7f28dc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:45.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.794+0000 7f28e0ff9700 1 -- 172.21.15.3:0/1233640796 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f28cc042e80 con 0x7f28e40afd60 2022-07-21T02:43:45.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f28e40574e0 con 0x7f28c8053aa0 2022-07-21T02:43:45.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e0ff9700 1 -- 172.21.15.3:0/1233640796 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f28e40574e0 con 0x7f28c8053aa0 2022-07-21T02:43:45.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28c8053aa0 msgr2=0x7f28c8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28c8053aa0 0x7f28c8055f60 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f28dc019000 tx=0x7f28dc016000).stop 2022-07-21T02:43:45.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40afd60 msgr2=0x7f28e40b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:45.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40afd60 0x7f28e40b32f0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7f28cc0030b0 tx=0x7f28cc008040).stop 2022-07-21T02:43:45.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 shutdown_connections 2022-07-21T02:43:45.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40b2710 0x7f28e40ab2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f28c8053aa0 0x7f28c8055f60 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e40b01a0 0x7f28e40b5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 --2- 172.21.15.3:0/1233640796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28e40afd60 0x7f28e40b32f0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:45.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 >> 172.21.15.3:0/1233640796 conn(0x7f28e4060310 msgr2=0x7f28e4149f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:45.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 shutdown_connections 2022-07-21T02:43:45.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:45.918+0000 7f28e8994700 1 -- 172.21.15.3:0/1233640796 wait complete. 2022-07-21T02:43:45.931 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 1 to deep-scrub 2022-07-21T02:43:45.933 INFO:tasks.ceph:Scrubbing osd.2 2022-07-21T02:43:45.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2022-07-21T02:43:46.052 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee35d61700 1 Processor -- start 2022-07-21T02:43:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee35d61700 1 -- start start 2022-07-21T02:43:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee35d61700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee300b4550 0x7fee300b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.054 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee35d61700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fee300b6af0 0x7fee3012bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee35d61700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee30144240 con 0x7fee300b6710 2022-07-21T02:43:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee35d61700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee30143d00 con 0x7fee300b4550 2022-07-21T02:43:46.055 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee35d61700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee30143fa0 con 0x7fee300b6af0 2022-07-21T02:43:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee300b4550 0x7fee300b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.057 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2f7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fee300b6710 0x7fee30061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60880/0 (socket says 172.21.15.3:60880) 2022-07-21T02:43:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2effd700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fee300b4550 0x7fee300b5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52600/0 (socket says 172.21.15.3:52600) 2022-07-21T02:43:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2f7fe700 1 -- 172.21.15.3:0/2283781188 learned_addr learned my addr 172.21.15.3:0/2283781188 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:46.058 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2ffff700 1 --2- 172.21.15.3:0/2283781188 >> v2:172.21.15.143:3300/0 conn(0x7fee300b6af0 0x7fee3012bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.059 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2effd700 1 -- 172.21.15.3:0/2283781188 >> v2:172.21.15.143:3300/0 conn(0x7fee300b6af0 msgr2=0x7fee3012bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:46.068 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2effd700 1 --2- 172.21.15.3:0/2283781188 >> v2:172.21.15.143:3300/0 conn(0x7fee300b6af0 0x7fee3012bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.069 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2effd700 1 -- 172.21.15.3:0/2283781188 >> v1:172.21.15.3:6789/0 conn(0x7fee300b6710 legacy=0x7fee30061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2effd700 1 -- 172.21.15.3:0/2283781188 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee30061780 con 0x7fee300b4550 2022-07-21T02:43:46.070 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.050+0000 7fee2effd700 1 --2- 172.21.15.3:0/2283781188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b4550 0x7fee300b5d80 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7fee301432b0 tx=0x7fee20006df0).ready entity=mon.0 client_cookie=847bc767a2d3eddb server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee2e7fc700 1 -- 172.21.15.3:0/2283781188 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee2000d5c0 con 0x7fee300b4550 2022-07-21T02:43:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee2e7fc700 1 -- 172.21.15.3:0/2283781188 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee20000e90 con 0x7fee300b4550 2022-07-21T02:43:46.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee2e7fc700 1 -- 172.21.15.3:0/2283781188 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee2000d910 con 0x7fee300b4550 2022-07-21T02:43:46.073 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee35d61700 1 -- 172.21.15.3:0/2283781188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b4550 msgr2=0x7fee300b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee35d61700 1 --2- 172.21.15.3:0/2283781188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b4550 0x7fee300b5d80 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7fee301432b0 tx=0x7fee20006df0).stop 2022-07-21T02:43:46.074 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee35d61700 1 -- 172.21.15.3:0/2283781188 shutdown_connections 2022-07-21T02:43:46.075 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee35d61700 1 --2- 172.21.15.3:0/2283781188 >> v2:172.21.15.143:3300/0 conn(0x7fee300b6af0 0x7fee3012bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee35d61700 1 --2- 172.21.15.3:0/2283781188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b4550 0x7fee300b5d80 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.076 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee35d61700 1 -- 172.21.15.3:0/2283781188 >> 172.21.15.3:0/2283781188 conn(0x7fee3005c620 msgr2=0x7fee3005ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee35d61700 1 -- 172.21.15.3:0/2283781188 shutdown_connections 2022-07-21T02:43:46.077 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.054+0000 7fee35d61700 1 -- 172.21.15.3:0/2283781188 wait complete. 2022-07-21T02:43:46.078 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 Processor -- start 2022-07-21T02:43:46.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 -- start start 2022-07-21T02:43:46.079 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee300b4550 0x7fee3012a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b6710 0x7fee3012b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.080 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee300b6af0 0x7fee30125530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.081 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3012a2d0 con 0x7fee300b6710 2022-07-21T02:43:46.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee30148c70 con 0x7fee300b6af0 2022-07-21T02:43:46.082 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee301497e0 con 0x7fee300b4550 2022-07-21T02:43:46.083 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b6710 0x7fee3012b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b6710 0x7fee3012b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52608/0 (socket says 172.21.15.3:52608) 2022-07-21T02:43:46.084 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2effd700 1 -- 172.21.15.3:0/79192811 learned_addr learned my addr 172.21.15.3:0/79192811 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:46.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2f7fe700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee300b4550 0x7fee3012a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.085 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2ffff700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee300b6af0 0x7fee30125530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.086 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2effd700 1 -- 172.21.15.3:0/79192811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee300b4550 msgr2=0x7fee3012a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2effd700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee300b4550 0x7fee3012a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.087 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2effd700 1 -- 172.21.15.3:0/79192811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee300b6af0 msgr2=0x7fee30125530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2effd700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee300b6af0 0x7fee30125530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2effd700 1 -- 172.21.15.3:0/79192811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2000b180 con 0x7fee300b6710 2022-07-21T02:43:46.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2ffff700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee300b6af0 0x7fee30125530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T02:43:46.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2f7fe700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee300b4550 0x7fee3012a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T02:43:46.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2effd700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b6710 0x7fee3012b230 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7fee301432b0 tx=0x7fee20012000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.096 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2cff9700 1 -- 172.21.15.3:0/79192811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee20016070 con 0x7fee300b6710 2022-07-21T02:43:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2cff9700 1 -- 172.21.15.3:0/79192811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2000d960 con 0x7fee300b6710 2022-07-21T02:43:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee30146c70 con 0x7fee300b6710 2022-07-21T02:43:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee2cff9700 1 -- 172.21.15.3:0/79192811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee2001e450 con 0x7fee300b6710 2022-07-21T02:43:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.058+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3012ae80 con 0x7fee300b6710 2022-07-21T02:43:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fee3012b8c0 con 0x7fee300b6710 2022-07-21T02:43:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2cff9700 1 -- 172.21.15.3:0/79192811 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fee2000f030 con 0x7fee300b6710 2022-07-21T02:43:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2cff9700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee10053b00 0x7fee10055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2cff9700 1 -- 172.21.15.3:0/79192811 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fee2006cc40 con 0x7fee300b6710 2022-07-21T02:43:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2f7fe700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee10053b00 0x7fee10055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2cff9700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7fee1005a790 0x7fee1005cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2cff9700 1 -- 172.21.15.3:0/79192811 --> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee1005d210 con 0x7fee1005a790 2022-07-21T02:43:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2cff9700 1 -- 172.21.15.3:0/79192811 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee200711e0 con 0x7fee300b6710 2022-07-21T02:43:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2ffff700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7fee1005a790 0x7fee1005cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2f7fe700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee10053b00 0x7fee10055fc0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7fee2400b000 tx=0x7fee24009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.062+0000 7fee2ffff700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7fee1005a790 0x7fee1005cc30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.066+0000 7fee2cff9700 1 -- 172.21.15.3:0/79192811 <== osd.2 v2:172.21.15.3:6802/22877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7fee1005d210 con 0x7fee1005a790 2022-07-21T02:43:46.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 --> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fee30061a20 con 0x7fee1005a790 2022-07-21T02:43:46.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee2cff9700 1 -- 172.21.15.3:0/79192811 <== osd.2 v2:172.21.15.3:6802/22877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fee30061a20 con 0x7fee1005a790 2022-07-21T02:43:46.107 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T02:43:46.107 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-21T02:43:46.108 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T02:43:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7fee1005a790 msgr2=0x7fee1005cc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6802/22877,v1:172.21.15.3:6803/22877] conn(0x7fee1005a790 0x7fee1005cc30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee10053b00 msgr2=0x7fee10055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fee10053b00 0x7fee10055fc0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7fee2400b000 tx=0x7fee24009040).stop 2022-07-21T02:43:46.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b6710 msgr2=0x7fee3012b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 --2- 172.21.15.3:0/79192811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee300b6710 0x7fee3012b230 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7fee301432b0 tx=0x7fee20012000).stop 2022-07-21T02:43:46.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee2ffff700 1 -- 172.21.15.3:0/79192811 reap_dead start 2022-07-21T02:43:46.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 shutdown_connections 2022-07-21T02:43:46.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 >> 172.21.15.3:0/79192811 conn(0x7fee3005c620 msgr2=0x7fee301432f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 shutdown_connections 2022-07-21T02:43:46.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.082+0000 7fee35d61700 1 -- 172.21.15.3:0/79192811 wait complete. 2022-07-21T02:43:46.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2022-07-21T02:43:46.229 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f405a4a8700 1 Processor -- start 2022-07-21T02:43:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f405a4a8700 1 -- start start 2022-07-21T02:43:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f405a4a8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40540b6a90 0x7f4054144950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f405a4a8700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f40540aa210 0x7f4054144e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f405a4a8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40540b6e70 con 0x7f405405ff60 2022-07-21T02:43:46.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f405a4a8700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40540ac6e0 con 0x7f40540aa210 2022-07-21T02:43:46.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f405a4a8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40540ac980 con 0x7f40540b6a90 2022-07-21T02:43:46.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f40537fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f40540aa210 0x7f4054144e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f40537fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f40540aa210 0x7f4054144e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52618/0 (socket says 172.21.15.3:52618) 2022-07-21T02:43:46.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f40537fe700 1 -- 172.21.15.3:0/3100495420 learned_addr learned my addr 172.21.15.3:0/3100495420 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:46.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f4053fff700 1 --2- 172.21.15.3:0/3100495420 >> v2:172.21.15.143:3300/0 conn(0x7f40540b6a90 0x7f4054144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f40537fe700 1 -- 172.21.15.3:0/3100495420 >> v2:172.21.15.143:3300/0 conn(0x7f40540b6a90 msgr2=0x7f4054144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:46.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f40537fe700 1 --2- 172.21.15.3:0/3100495420 >> v2:172.21.15.143:3300/0 conn(0x7f40540b6a90 0x7f4054144950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f40537fe700 1 -- 172.21.15.3:0/3100495420 >> v1:172.21.15.143:6789/0 conn(0x7f405405ff60 legacy=0x7f405414ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f40537fe700 1 -- 172.21.15.3:0/3100495420 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40540acc20 con 0x7f40540aa210 2022-07-21T02:43:46.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f40537fe700 1 --2- 172.21.15.3:0/3100495420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540aa210 0x7f4054144e40 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f4048002f70 tx=0x7f4048006eb0).ready entity=mon.0 client_cookie=6a3d0cdef33d8aae server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f4052ffd700 1 -- 172.21.15.3:0/3100495420 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f404800d5c0 con 0x7f40540aa210 2022-07-21T02:43:46.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f4052ffd700 1 -- 172.21.15.3:0/3100495420 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048000e90 con 0x7f40540aa210 2022-07-21T02:43:46.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f4052ffd700 1 -- 172.21.15.3:0/3100495420 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f404800d950 con 0x7f40540aa210 2022-07-21T02:43:46.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f405a4a8700 1 -- 172.21.15.3:0/3100495420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540aa210 msgr2=0x7f4054144e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.230+0000 7f405a4a8700 1 --2- 172.21.15.3:0/3100495420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540aa210 0x7f4054144e40 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f4048002f70 tx=0x7f4048006eb0).stop 2022-07-21T02:43:46.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.234+0000 7f405a4a8700 1 -- 172.21.15.3:0/3100495420 shutdown_connections 2022-07-21T02:43:46.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.234+0000 7f405a4a8700 1 --2- 172.21.15.3:0/3100495420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540aa210 0x7f4054144e40 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.234+0000 7f405a4a8700 1 --2- 172.21.15.3:0/3100495420 >> v2:172.21.15.143:3300/0 conn(0x7f40540b6a90 0x7f4054144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.234+0000 7f405a4a8700 1 -- 172.21.15.3:0/3100495420 >> 172.21.15.3:0/3100495420 conn(0x7f40540b3610 msgr2=0x7f40540b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.242 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.234+0000 7f405a4a8700 1 -- 172.21.15.3:0/3100495420 shutdown_connections 2022-07-21T02:43:46.243 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.234+0000 7f405a4a8700 1 -- 172.21.15.3:0/3100495420 wait complete. 2022-07-21T02:43:46.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.234+0000 7f405a4a8700 1 Processor -- start 2022-07-21T02:43:46.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.234+0000 7f405a4a8700 1 -- start start 2022-07-21T02:43:46.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f405a4a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f405405ff60 0x7f40540b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f405a4a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40540aa210 0x7f4054156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f405a4a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540b6a90 0x7f4054156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f405a4a8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4054141fc0 con 0x7f40540b6a90 2022-07-21T02:43:46.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f405a4a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4054149380 con 0x7f40540aa210 2022-07-21T02:43:46.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f405a4a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4054141d20 con 0x7f405405ff60 2022-07-21T02:43:46.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4058a45700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540b6a90 0x7f4054156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.249 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4053fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f405405ff60 0x7f40540b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4058a45700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540b6a90 0x7f4054156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52626/0 (socket says 172.21.15.3:52626) 2022-07-21T02:43:46.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4058a45700 1 -- 172.21.15.3:0/840555974 learned_addr learned my addr 172.21.15.3:0/840555974 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:46.250 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f40537fe700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40540aa210 0x7f4054156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4058a45700 1 -- 172.21.15.3:0/840555974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f405405ff60 msgr2=0x7f40540b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4058a45700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f405405ff60 0x7f40540b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4058a45700 1 -- 172.21.15.3:0/840555974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40540aa210 msgr2=0x7f4054156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4058a45700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40540aa210 0x7f4054156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4058a45700 1 -- 172.21.15.3:0/840555974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404800b1e0 con 0x7f40540b6a90 2022-07-21T02:43:46.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f4058a45700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540b6a90 0x7f4054156f10 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f40540b5f50 tx=0x7f404c007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f40517fa700 1 -- 172.21.15.3:0/840555974 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f404c002490 con 0x7f40540b6a90 2022-07-21T02:43:46.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f40517fa700 1 -- 172.21.15.3:0/840555974 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c0025f0 con 0x7f40540b6a90 2022-07-21T02:43:46.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4054148c40 con 0x7f40540b6a90 2022-07-21T02:43:46.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f40517fa700 1 -- 172.21.15.3:0/840555974 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f404c002950 con 0x7f40540b6a90 2022-07-21T02:43:46.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4054157400 con 0x7f40540b6a90 2022-07-21T02:43:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40541490e0 con 0x7f40540b6a90 2022-07-21T02:43:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.238+0000 7f40517fa700 1 -- 172.21.15.3:0/840555974 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f404c010030 con 0x7f40540b6a90 2022-07-21T02:43:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.242+0000 7f40517fa700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4034053980 0x7f4034055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.242+0000 7f40517fa700 1 -- 172.21.15.3:0/840555974 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f404c06be70 con 0x7f40540b6a90 2022-07-21T02:43:46.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.242+0000 7f4053fff700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4034053980 0x7f4034055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.242+0000 7f4053fff700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4034053980 0x7f4034055e40 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f404400b000 tx=0x7f4044006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.246+0000 7f40517fa700 1 -- 172.21.15.3:0/840555974 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f40541490e0 con 0x7f40540b6a90 2022-07-21T02:43:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f40541490e0 con 0x7f4034053980 2022-07-21T02:43:46.379 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f40517fa700 1 -- 172.21.15.3:0/840555974 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f40541490e0 con 0x7f4034053980 2022-07-21T02:43:46.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4034053980 msgr2=0x7f4034055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.382 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4034053980 0x7f4034055e40 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f404400b000 tx=0x7f4044006df0).stop 2022-07-21T02:43:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540b6a90 msgr2=0x7f4054156f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540b6a90 0x7f4054156f10 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f40540b5f50 tx=0x7f404c007f40).stop 2022-07-21T02:43:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 shutdown_connections 2022-07-21T02:43:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40540b6a90 0x7f4054156f10 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f4034053980 0x7f4034055e40 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40540aa210 0x7f4054156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 --2- 172.21.15.3:0/840555974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f405405ff60 0x7f40540b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 >> 172.21.15.3:0/840555974 conn(0x7f40540b3610 msgr2=0x7f40540b6020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 shutdown_connections 2022-07-21T02:43:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.378+0000 7f405a4a8700 1 -- 172.21.15.3:0/840555974 wait complete. 2022-07-21T02:43:46.388 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 2 to deep-scrub 2022-07-21T02:43:46.392 INFO:tasks.ceph:Scrubbing osd.3 2022-07-21T02:43:46.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2022-07-21T02:43:46.499 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1d027700 1 Processor -- start 2022-07-21T02:43:46.501 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1d027700 1 -- start start 2022-07-21T02:43:46.520 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1d027700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdc18060290 0x7fdc180617e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.521 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1d027700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065fb0 0x7fdc18066e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.522 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1d027700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc18060670 con 0x7fdc18065b00 2022-07-21T02:43:46.523 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1d027700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc180663f0 con 0x7fdc18060290 2022-07-21T02:43:46.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1d027700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc18060e00 con 0x7fdc18065fb0 2022-07-21T02:43:46.524 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1659c700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdc18065b00 0x7fdc18066760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60912/0 (socket says 172.21.15.3:60912) 2022-07-21T02:43:46.525 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1659c700 1 -- 172.21.15.3:0/298417401 learned_addr learned my addr 172.21.15.3:0/298417401 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:46.526 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc16d9d700 1 --2- 172.21.15.3:0/298417401 >> v2:172.21.15.143:3300/0 conn(0x7fdc18060290 0x7fdc180617e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.527 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.498+0000 7fdc1759e700 1 --2- 172.21.15.3:0/298417401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065fb0 0x7fdc18066e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.528 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc15d9b700 1 -- 172.21.15.3:0/298417401 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919379767 0 0) 0x7fdc18060670 con 0x7fdc18065b00 2022-07-21T02:43:46.529 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc15d9b700 1 -- 172.21.15.3:0/298417401 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf8003260 con 0x7fdc18065b00 2022-07-21T02:43:46.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1759e700 1 -- 172.21.15.3:0/298417401 >> v2:172.21.15.143:3300/0 conn(0x7fdc18060290 msgr2=0x7fdc180617e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:46.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1759e700 1 --2- 172.21.15.3:0/298417401 >> v2:172.21.15.143:3300/0 conn(0x7fdc18060290 0x7fdc180617e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.530 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1759e700 1 -- 172.21.15.3:0/298417401 >> v1:172.21.15.3:6789/0 conn(0x7fdc18065b00 legacy=0x7fdc18066760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1759e700 1 -- 172.21.15.3:0/298417401 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc180610a0 con 0x7fdc18065fb0 2022-07-21T02:43:46.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc16d9d700 1 --2- 172.21.15.3:0/298417401 >> v2:172.21.15.143:3300/0 conn(0x7fdc18060290 0x7fdc180617e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T02:43:46.531 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1759e700 1 --2- 172.21.15.3:0/298417401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065fb0 0x7fdc18066e70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fdc100032f0 tx=0x7fdc100091d0).ready entity=mon.2 client_cookie=60f5e96bdd892746 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc15d9b700 1 -- 172.21.15.3:0/298417401 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc1000b600 con 0x7fdc18065fb0 2022-07-21T02:43:46.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc15d9b700 1 -- 172.21.15.3:0/298417401 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1000b760 con 0x7fdc18065fb0 2022-07-21T02:43:46.532 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc15d9b700 1 -- 172.21.15.3:0/298417401 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc1000b8d0 con 0x7fdc18065fb0 2022-07-21T02:43:46.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1d027700 1 -- 172.21.15.3:0/298417401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065fb0 msgr2=0x7fdc18066e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.533 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1d027700 1 --2- 172.21.15.3:0/298417401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065fb0 0x7fdc18066e70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fdc100032f0 tx=0x7fdc100091d0).stop 2022-07-21T02:43:46.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1d027700 1 -- 172.21.15.3:0/298417401 shutdown_connections 2022-07-21T02:43:46.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1d027700 1 --2- 172.21.15.3:0/298417401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065fb0 0x7fdc18066e70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.534 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1d027700 1 --2- 172.21.15.3:0/298417401 >> v2:172.21.15.143:3300/0 conn(0x7fdc18060290 0x7fdc180617e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1d027700 1 -- 172.21.15.3:0/298417401 >> 172.21.15.3:0/298417401 conn(0x7fdc1805d0c0 msgr2=0x7fdc1805f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1d027700 1 -- 172.21.15.3:0/298417401 shutdown_connections 2022-07-21T02:43:46.535 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.502+0000 7fdc1d027700 1 -- 172.21.15.3:0/298417401 wait complete. 2022-07-21T02:43:46.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 Processor -- start 2022-07-21T02:43:46.536 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 -- start start 2022-07-21T02:43:46.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc18060290 0x7fdc181568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065b00 0x7fdc180a6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.537 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc18065fb0 0x7fdc181527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180aa140 con 0x7fdc18065fb0 2022-07-21T02:43:46.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180a84c0 con 0x7fdc18060290 2022-07-21T02:43:46.538 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc18155f20 con 0x7fdc18065b00 2022-07-21T02:43:46.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc18065fb0 0x7fdc181527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc18065fb0 0x7fdc181527d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52640/0 (socket says 172.21.15.3:52640) 2022-07-21T02:43:46.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1759e700 1 -- 172.21.15.3:0/3978717958 learned_addr learned my addr 172.21.15.3:0/3978717958 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:46.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc16d9d700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc18060290 0x7fdc181568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1659c700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065b00 0x7fdc180a6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1759e700 1 -- 172.21.15.3:0/3978717958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065b00 msgr2=0x7fdc180a6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1759e700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc18065b00 0x7fdc180a6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1759e700 1 -- 172.21.15.3:0/3978717958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc18060290 msgr2=0x7fdc181568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1759e700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc18060290 0x7fdc181568c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1759e700 1 -- 172.21.15.3:0/3978717958 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0000b180 con 0x7fdc18065fb0 2022-07-21T02:43:46.542 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1759e700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc18065fb0 0x7fdc181527d0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7fdc100032f0 tx=0x7fdc10016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc0ffff700 1 -- 172.21.15.3:0/3978717958 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc1000b8d0 con 0x7fdc18065fb0 2022-07-21T02:43:46.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc0ffff700 1 -- 172.21.15.3:0/3978717958 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1001f3f0 con 0x7fdc18065fb0 2022-07-21T02:43:46.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc10009470 con 0x7fdc18065fb0 2022-07-21T02:43:46.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc0ffff700 1 -- 172.21.15.3:0/3978717958 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc1000b600 con 0x7fdc18065fb0 2022-07-21T02:43:46.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.506+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc180a72e0 con 0x7fdc18065fb0 2022-07-21T02:43:46.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdc18153000 con 0x7fdc18065fb0 2022-07-21T02:43:46.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc0ffff700 1 -- 172.21.15.3:0/3978717958 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc10026030 con 0x7fdc18065fb0 2022-07-21T02:43:46.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc0ffff700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbfc053b30 0x7fdbfc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc0ffff700 1 -- 172.21.15.3:0/3978717958 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fdc1006c860 con 0x7fdc18065fb0 2022-07-21T02:43:46.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc16d9d700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbfc053b30 0x7fdbfc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc0ffff700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7fdbfc05a7c0 0x7fdbfc05cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc0ffff700 1 -- 172.21.15.3:0/3978717958 --> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdbfc05d240 con 0x7fdbfc05a7c0 2022-07-21T02:43:46.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc0ffff700 1 -- 172.21.15.3:0/3978717958 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc10071220 con 0x7fdc18065fb0 2022-07-21T02:43:46.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc1659c700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7fdbfc05a7c0 0x7fdbfc05cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc16d9d700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbfc053b30 0x7fdbfc055ff0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7fdc0000b000 tx=0x7fdc00006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.510+0000 7fdc1659c700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7fdbfc05a7c0 0x7fdbfc05cc60 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.514+0000 7fdc0ffff700 1 -- 172.21.15.3:0/3978717958 <== osd.3 v2:172.21.15.3:6814/22884 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7fdbfc05d240 con 0x7fdbfc05a7c0 2022-07-21T02:43:46.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.530+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 --> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdc18156620 con 0x7fdbfc05a7c0 2022-07-21T02:43:46.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.530+0000 7fdc0ffff700 1 -- 172.21.15.3:0/3978717958 <== osd.3 v2:172.21.15.3:6814/22884 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fdc080015e0 con 0x7fdbfc05a7c0 2022-07-21T02:43:46.551 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T02:43:46.551 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-21T02:43:46.552 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T02:43:46.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7fdbfc05a7c0 msgr2=0x7fdbfc05cc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6814/22884,v1:172.21.15.3:6817/22884] conn(0x7fdbfc05a7c0 0x7fdbfc05cc60 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbfc053b30 msgr2=0x7fdbfc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fdbfc053b30 0x7fdbfc055ff0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7fdc0000b000 tx=0x7fdc00006df0).stop 2022-07-21T02:43:46.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc18065fb0 msgr2=0x7fdc181527d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 --2- 172.21.15.3:0/3978717958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc18065fb0 0x7fdc181527d0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7fdc100032f0 tx=0x7fdc10016000).stop 2022-07-21T02:43:46.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1759e700 1 -- 172.21.15.3:0/3978717958 reap_dead start 2022-07-21T02:43:46.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 shutdown_connections 2022-07-21T02:43:46.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 >> 172.21.15.3:0/3978717958 conn(0x7fdc1805d0c0 msgr2=0x7fdc18142f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 shutdown_connections 2022-07-21T02:43:46.557 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.534+0000 7fdc1d027700 1 -- 172.21.15.3:0/3978717958 wait complete. 2022-07-21T02:43:46.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2022-07-21T02:43:46.666 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32b4a0c700 1 Processor -- start 2022-07-21T02:43:46.667 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32b4a0c700 1 -- start start 2022-07-21T02:43:46.675 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32b4a0c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f32b00ad000 0x7f32b01435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.676 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32b4a0c700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b0143ab0 con 0x7f32b00b08a0 2022-07-21T02:43:46.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32b4a0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b00b0c80 con 0x7f32b00b3260 2022-07-21T02:43:46.677 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32b4a0c700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b00ae810 con 0x7f32b00ad000 2022-07-21T02:43:46.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32aed9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f32b00b08a0 0x7f32b0142eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60926/0 (socket says 172.21.15.3:60926) 2022-07-21T02:43:46.678 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32aed9d700 1 -- 172.21.15.3:0/1019336055 learned_addr learned my addr 172.21.15.3:0/1019336055 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:46.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32af59e700 1 --2- 172.21.15.3:0/1019336055 >> v2:172.21.15.3:3300/0 conn(0x7f32b00ad000 0x7f32b01435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.679 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842845252 0 0) 0x7f32b0143ab0 con 0x7f32b00b08a0 2022-07-21T02:43:46.680 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3290004370 con 0x7f32b00b08a0 2022-07-21T02:43:46.681 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2271982022 0 0) 0x7f3290004370 con 0x7f32b00b08a0 2022-07-21T02:43:46.682 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 >> v2:172.21.15.3:3300/0 conn(0x7f32b00ad000 msgr2=0x7f32b01435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:46.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 --2- 172.21.15.3:0/1019336055 >> v2:172.21.15.3:3300/0 conn(0x7f32b00ad000 0x7f32b01435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 >> v1:172.21.15.143:6789/0 conn(0x7f32b00b3260 legacy=0x7f32b013fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b00aeab0 con 0x7f32b00b08a0 2022-07-21T02:43:46.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f32a00019e0 con 0x7f32b00b08a0 2022-07-21T02:43:46.685 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 >> v1:172.21.15.3:6789/0 conn(0x7f32b00b08a0 legacy=0x7f32b0142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3290004860 0x7f3290004c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.686 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3290005610 0x7f3290005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3290005f80 0x7f329000c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00ae810 con 0x7f3290004860 2022-07-21T02:43:46.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3290003180 con 0x7f3290005f80 2022-07-21T02:43:46.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3290003340 con 0x7f3290005610 2022-07-21T02:43:46.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32aed9d700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3290004860 0x7f3290004c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.666+0000 7f32af59e700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3290005f80 0x7f329000c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32ae59c700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3290005610 0x7f3290005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32aed9d700 1 -- 172.21.15.3:0/1019336055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3290005610 msgr2=0x7f3290005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:46.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32aed9d700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3290005610 0x7f3290005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32aed9d700 1 -- 172.21.15.3:0/1019336055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3290005f80 msgr2=0x7f329000c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:46.694 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32aed9d700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3290005f80 0x7f329000c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32aed9d700 1 -- 172.21.15.3:0/1019336055 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3290004370 con 0x7f3290004860 2022-07-21T02:43:46.695 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32af59e700 1 -- 172.21.15.3:0/1019336055 reap_dead start 2022-07-21T02:43:46.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32aed9d700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3290004860 0x7f3290004c60 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7f32a00016f0 tx=0x7f32b00618b0).ready entity=mon.0 client_cookie=f1855c460ec8481c server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.696 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32a000cc10 con 0x7f3290004860 2022-07-21T02:43:46.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a000cd70 con 0x7f3290004860 2022-07-21T02:43:46.697 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32add9b700 1 -- 172.21.15.3:0/1019336055 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32a0005b80 con 0x7f3290004860 2022-07-21T02:43:46.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32b4a0c700 1 -- 172.21.15.3:0/1019336055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3290004860 msgr2=0x7f3290004c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32b4a0c700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3290004860 0x7f3290004c60 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7f32a00016f0 tx=0x7f32b00618b0).stop 2022-07-21T02:43:46.698 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32b4a0c700 1 -- 172.21.15.3:0/1019336055 shutdown_connections 2022-07-21T02:43:46.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32b4a0c700 1 --2- 172.21.15.3:0/1019336055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3290004860 0x7f3290004c60 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32b4a0c700 1 -- 172.21.15.3:0/1019336055 >> 172.21.15.3:0/1019336055 conn(0x7f32b00612d0 msgr2=0x7f32b0062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.699 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32b4a0c700 1 -- 172.21.15.3:0/1019336055 shutdown_connections 2022-07-21T02:43:46.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32b4a0c700 1 -- 172.21.15.3:0/1019336055 wait complete. 2022-07-21T02:43:46.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.670+0000 7f32b4a0c700 1 Processor -- start 2022-07-21T02:43:46.700 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32b4a0c700 1 -- start start 2022-07-21T02:43:46.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32b4a0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3290004860 0x7f32b009f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32b4a0c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b009fba0 0x7f32b014e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.701 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32b4a0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b014e5a0 0x7f32b013dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32b4a0c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0064cf0 con 0x7f32b009fba0 2022-07-21T02:43:46.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32b4a0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0066dd0 con 0x7f3290004860 2022-07-21T02:43:46.702 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32b4a0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0065560 con 0x7f32b014e5a0 2022-07-21T02:43:46.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b009fba0 0x7f32b014e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b009fba0 0x7f32b014e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52660/0 (socket says 172.21.15.3:52660) 2022-07-21T02:43:46.703 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32ae59c700 1 -- 172.21.15.3:0/3762559861 learned_addr learned my addr 172.21.15.3:0/3762559861 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:46.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32aed9d700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3290004860 0x7f32b009f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32af59e700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b014e5a0 0x7f32b013dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.704 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32ae59c700 1 -- 172.21.15.3:0/3762559861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b014e5a0 msgr2=0x7f32b013dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32ae59c700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b014e5a0 0x7f32b013dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32ae59c700 1 -- 172.21.15.3:0/3762559861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3290004860 msgr2=0x7f32b009f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.705 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32ae59c700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3290004860 0x7f32b009f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32ae59c700 1 -- 172.21.15.3:0/3762559861 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b0064f20 con 0x7f32b009fba0 2022-07-21T02:43:46.706 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.674+0000 7f32ae59c700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b009fba0 0x7f32b014e0b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f329800b000 tx=0x7f3298009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.678+0000 7f32a7fff700 1 -- 172.21.15.3:0/3762559861 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3298014620 con 0x7f32b009fba0 2022-07-21T02:43:46.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.678+0000 7f32a7fff700 1 -- 172.21.15.3:0/3762559861 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3298014780 con 0x7f32b009fba0 2022-07-21T02:43:46.708 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.678+0000 7f32a7fff700 1 -- 172.21.15.3:0/3762559861 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3298014ae0 con 0x7f32b009fba0 2022-07-21T02:43:46.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.678+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b00652c0 con 0x7f32b009fba0 2022-07-21T02:43:46.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.678+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b0067230 con 0x7f32b009fba0 2022-07-21T02:43:46.709 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.678+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b0066f40 con 0x7f32b009fba0 2022-07-21T02:43:46.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.678+0000 7f32a7fff700 1 -- 172.21.15.3:0/3762559861 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3298014c40 con 0x7f32b009fba0 2022-07-21T02:43:46.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.678+0000 7f32a7fff700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3294053af0 0x7f3294055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.710 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.678+0000 7f32a7fff700 1 -- 172.21.15.3:0/3762559861 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f329806bb80 con 0x7f32b009fba0 2022-07-21T02:43:46.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.682+0000 7f32aed9d700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3294053af0 0x7f3294055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.682+0000 7f32aed9d700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3294053af0 0x7f3294055fb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f32a0019000 tx=0x7f32a0013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.711 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.682+0000 7f32a7fff700 1 -- 172.21.15.3:0/3762559861 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3298042c20 con 0x7f32b009fba0 2022-07-21T02:43:46.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.810+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f32b00574e0 con 0x7f3294053af0 2022-07-21T02:43:46.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.810+0000 7f32a7fff700 1 -- 172.21.15.3:0/3762559861 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f32b00574e0 con 0x7f3294053af0 2022-07-21T02:43:46.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3294053af0 msgr2=0x7f3294055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3294053af0 0x7f3294055fb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f32a0019000 tx=0x7f32a0013000).stop 2022-07-21T02:43:46.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b009fba0 msgr2=0x7f32b014e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b009fba0 0x7f32b014e0b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f329800b000 tx=0x7f3298009040).stop 2022-07-21T02:43:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 shutdown_connections 2022-07-21T02:43:46.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32b014e5a0 0x7f32b013dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b009fba0 0x7f32b014e0b0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3294053af0 0x7f3294055fb0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 --2- 172.21.15.3:0/3762559861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3290004860 0x7f32b009f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 >> 172.21.15.3:0/3762559861 conn(0x7f32b00612d0 msgr2=0x7f32b0142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 shutdown_connections 2022-07-21T02:43:46.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.814+0000 7f32b4a0c700 1 -- 172.21.15.3:0/3762559861 wait complete. 2022-07-21T02:43:46.822 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 3 to deep-scrub 2022-07-21T02:43:46.824 INFO:tasks.ceph:Scrubbing osd.4 2022-07-21T02:43:46.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2022-07-21T02:43:46.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.934+0000 7f1c5bbf5700 1 Processor -- start 2022-07-21T02:43:46.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.934+0000 7f1c5bbf5700 1 -- start start 2022-07-21T02:43:46.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.934+0000 7f1c5bbf5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1c540b4550 0x7f1c540b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.934+0000 7f1c5bbf5700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c54144240 con 0x7f1c540b6710 2022-07-21T02:43:46.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.934+0000 7f1c5bbf5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c54143d00 con 0x7f1c540b6af0 2022-07-21T02:43:46.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.934+0000 7f1c5bbf5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c54143fa0 con 0x7f1c540b4550 2022-07-21T02:43:46.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59991700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1c540b6710 0x7f1c54061ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60946/0 (socket says 172.21.15.3:60946) 2022-07-21T02:43:46.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59991700 1 -- 172.21.15.3:0/781228680 learned_addr learned my addr 172.21.15.3:0/781228680 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:46.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59190700 1 --2- 172.21.15.3:0/781228680 >> v2:172.21.15.143:3300/0 conn(0x7f1c540b4550 0x7f1c540b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477812620 0 0) 0x7f1c54144240 con 0x7f1c540b6710 2022-07-21T02:43:46.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3c003260 con 0x7f1c540b6710 2022-07-21T02:43:46.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710586152 0 0) 0x7f1c54143d00 con 0x7f1c540b6af0 2022-07-21T02:43:46.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c54144240 con 0x7f1c540b6af0 2022-07-21T02:43:46.947 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 747907209 0 0) 0x7f1c3c003260 con 0x7f1c540b6710 2022-07-21T02:43:46.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 >> v2:172.21.15.143:3300/0 conn(0x7f1c540b4550 msgr2=0x7f1c540b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:46.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 --2- 172.21.15.3:0/781228680 >> v2:172.21.15.143:3300/0 conn(0x7f1c540b4550 0x7f1c540b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 >> v1:172.21.15.143:6789/0 conn(0x7f1c540b6af0 legacy=0x7f1c5412bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c54061780 con 0x7f1c540b6710 2022-07-21T02:43:46.949 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7f1c50001660 con 0x7f1c540b6710 2022-07-21T02:43:46.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 >> v1:172.21.15.3:6789/0 conn(0x7f1c540b6710 legacy=0x7f1c54061ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.950 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c3c003910 0x7f1c3c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c3c004500 0x7f1c3c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c3c004e70 0x7f1c3c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c54143fa0 con 0x7f1c3c003910 2022-07-21T02:43:46.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c00bd20 con 0x7f1c3c004e70 2022-07-21T02:43:46.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c00bee0 con 0x7f1c3c004500 2022-07-21T02:43:46.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59991700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c3c003910 0x7f1c3c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5a192700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c3c004e70 0x7f1c3c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59190700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c3c004500 0x7f1c3c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59991700 1 -- 172.21.15.3:0/781228680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c3c004500 msgr2=0x7f1c3c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:46.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59991700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c3c004500 0x7f1c3c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59991700 1 -- 172.21.15.3:0/781228680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c3c004e70 msgr2=0x7f1c3c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:46.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59991700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c3c004e70 0x7f1c3c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5a192700 1 -- 172.21.15.3:0/781228680 reap_dead start 2022-07-21T02:43:46.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59991700 1 -- 172.21.15.3:0/781228680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c003260 con 0x7f1c3c003910 2022-07-21T02:43:46.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59190700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c3c004500 0x7f1c3c004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T02:43:46.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c59991700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c3c003910 0x7f1c3c003d10 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7f1c50001430 tx=0x7f1c54061910).ready entity=mon.0 client_cookie=d8ea92f7bd06d2ae server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c500071a0 con 0x7f1c3c003910 2022-07-21T02:43:46.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c50007300 con 0x7f1c3c003910 2022-07-21T02:43:46.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5898f700 1 -- 172.21.15.3:0/781228680 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c5000c8f0 con 0x7f1c3c003910 2022-07-21T02:43:46.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/781228680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c3c003910 msgr2=0x7f1c3c003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5bbf5700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c3c003910 0x7f1c3c003d10 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7f1c50001430 tx=0x7f1c54061910).stop 2022-07-21T02:43:46.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/781228680 shutdown_connections 2022-07-21T02:43:46.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5bbf5700 1 --2- 172.21.15.3:0/781228680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c3c003910 0x7f1c3c003d10 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.959 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/781228680 >> 172.21.15.3:0/781228680 conn(0x7f1c5405c620 msgr2=0x7f1c5405ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/781228680 shutdown_connections 2022-07-21T02:43:46.960 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.938+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/781228680 wait complete. 2022-07-21T02:43:46.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.942+0000 7f1c5bbf5700 1 Processor -- start 2022-07-21T02:43:46.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.942+0000 7f1c5bbf5700 1 -- start start 2022-07-21T02:43:46.961 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.942+0000 7f1c5bbf5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c3c003910 0x7f1c54141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.942+0000 7f1c5bbf5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c540b3350 0x7f1c540b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.942+0000 7f1c5bbf5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c540b2680 0x7f1c54125e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.962 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.942+0000 7f1c5bbf5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5412ad20 con 0x7f1c540b3350 2022-07-21T02:43:46.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.942+0000 7f1c5bbf5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5412afc0 con 0x7f1c540b2680 2022-07-21T02:43:46.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.942+0000 7f1c5bbf5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5412b260 con 0x7f1c3c003910 2022-07-21T02:43:46.963 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59190700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c540b3350 0x7f1c540b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59190700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c540b3350 0x7f1c540b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52680/0 (socket says 172.21.15.3:52680) 2022-07-21T02:43:46.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59190700 1 -- 172.21.15.3:0/4087407193 learned_addr learned my addr 172.21.15.3:0/4087407193 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:46.964 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59991700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c3c003910 0x7f1c54141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c5a192700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c540b2680 0x7f1c54125e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59190700 1 -- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c3c003910 msgr2=0x7f1c54141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.965 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59190700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c3c003910 0x7f1c54141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59190700 1 -- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c540b2680 msgr2=0x7f1c54125e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59190700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c540b2680 0x7f1c54125e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.966 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59190700 1 -- 172.21.15.3:0/4087407193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5412a9a0 con 0x7f1c540b3350 2022-07-21T02:43:46.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c59190700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c540b3350 0x7f1c540b2190 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f1c4400b000 tx=0x7f1c44008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c42ffd700 1 -- 172.21.15.3:0/4087407193 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c44017070 con 0x7f1c540b3350 2022-07-21T02:43:46.967 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c42ffd700 1 -- 172.21.15.3:0/4087407193 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c44000e90 con 0x7f1c540b3350 2022-07-21T02:43:46.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c540b3b30 con 0x7f1c540b3350 2022-07-21T02:43:46.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c42ffd700 1 -- 172.21.15.3:0/4087407193 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c440137e0 con 0x7f1c540b3350 2022-07-21T02:43:46.968 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c540b2ca0 con 0x7f1c540b3350 2022-07-21T02:43:46.971 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c42ffd700 1 -- 172.21.15.3:0/4087407193 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1c44013940 con 0x7f1c540b3350 2022-07-21T02:43:46.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.946+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c5412b460 con 0x7f1c540b3350 2022-07-21T02:43:46.972 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c42ffd700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c34053b30 0x7f1c34055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c42ffd700 1 -- 172.21.15.3:0/4087407193 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f1c4406bc30 con 0x7f1c540b3350 2022-07-21T02:43:46.973 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c59991700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c34053b30 0x7f1c34055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c42ffd700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7f1c3405a7c0 0x7f1c3405cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:46.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c42ffd700 1 -- 172.21.15.3:0/4087407193 --> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c3405d240 con 0x7f1c3405a7c0 2022-07-21T02:43:46.974 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c42ffd700 1 -- 172.21.15.3:0/4087407193 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c4406e240 con 0x7f1c540b3350 2022-07-21T02:43:46.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c5a192700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7f1c3405a7c0 0x7f1c3405cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:46.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c59991700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c34053b30 0x7f1c34055ff0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f1c50019000 tx=0x7f1c50013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.975 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c5a192700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7f1c3405a7c0 0x7f1c3405cc60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:46.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.950+0000 7f1c42ffd700 1 -- 172.21.15.3:0/4087407193 <== osd.4 v2:172.21.15.143:6808/23339 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f1c3405d240 con 0x7f1c3405a7c0 2022-07-21T02:43:46.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 --> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1c54061a20 con 0x7f1c3405a7c0 2022-07-21T02:43:46.976 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c42ffd700 1 -- 172.21.15.3:0/4087407193 <== osd.4 v2:172.21.15.143:6808/23339 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1c54061a20 con 0x7f1c3405a7c0 2022-07-21T02:43:46.977 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T02:43:46.977 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-21T02:43:46.978 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T02:43:46.979 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7f1c3405a7c0 msgr2=0x7f1c3405cc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.143:6808/23339,v1:172.21.15.143:6809/23339] conn(0x7f1c3405a7c0 0x7f1c3405cc60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:46.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c34053b30 msgr2=0x7f1c34055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.980 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f1c34053b30 0x7f1c34055ff0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f1c50019000 tx=0x7f1c50013000).stop 2022-07-21T02:43:46.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c540b3350 msgr2=0x7f1c540b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:46.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 --2- 172.21.15.3:0/4087407193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c540b3350 0x7f1c540b2190 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f1c4400b000 tx=0x7f1c44008040).stop 2022-07-21T02:43:46.981 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5a192700 1 -- 172.21.15.3:0/4087407193 reap_dead start 2022-07-21T02:43:46.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 shutdown_connections 2022-07-21T02:43:46.982 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 >> 172.21.15.3:0/4087407193 conn(0x7f1c5405c620 msgr2=0x7f1c54061de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:46.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 shutdown_connections 2022-07-21T02:43:46.983 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:46.970+0000 7f1c5bbf5700 1 -- 172.21.15.3:0/4087407193 wait complete. 2022-07-21T02:43:46.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2022-07-21T02:43:47.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcac090f700 1 Processor -- start 2022-07-21T02:43:47.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcac090f700 1 -- start start 2022-07-21T02:43:47.111 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcac090f700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcabc0b01a0 0x7fcabc0a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcac090f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcabc0afd60 0x7fcabc0a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcac090f700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcabc0a7250 con 0x7fcabc0b2710 2022-07-21T02:43:47.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcac090f700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcabc0a7480 con 0x7fcabc0b01a0 2022-07-21T02:43:47.113 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcac090f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcabc0a7720 con 0x7fcabc0afd60 2022-07-21T02:43:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcaba59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcabc0b01a0 0x7fcabc0a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcaba59c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fcabc0b01a0 0x7fcabc0a8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52688/0 (socket says 172.21.15.3:52688) 2022-07-21T02:43:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcaba59c700 1 -- 172.21.15.3:0/3463319627 learned_addr learned my addr 172.21.15.3:0/3463319627 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.094+0000 7fcabb59e700 1 --2- 172.21.15.3:0/3463319627 >> v2:172.21.15.143:3300/0 conn(0x7fcabc0afd60 0x7fcabc0a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcaba59c700 1 -- 172.21.15.3:0/3463319627 >> v2:172.21.15.143:3300/0 conn(0x7fcabc0afd60 msgr2=0x7fcabc0a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcaba59c700 1 --2- 172.21.15.3:0/3463319627 >> v2:172.21.15.143:3300/0 conn(0x7fcabc0afd60 0x7fcabc0a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcaba59c700 1 -- 172.21.15.3:0/3463319627 >> v1:172.21.15.3:6789/0 conn(0x7fcabc0b2710 legacy=0x7fcabc14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcaba59c700 1 -- 172.21.15.3:0/3463319627 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcabc0a79c0 con 0x7fcabc0b01a0 2022-07-21T02:43:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcaba59c700 1 --2- 172.21.15.3:0/3463319627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0b01a0 0x7fcabc0a8540 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7fcab0002f70 tx=0x7fcab0006eb0).ready entity=mon.0 client_cookie=95033f163c3b2466 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcab9d9b700 1 -- 172.21.15.3:0/3463319627 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcab000d610 con 0x7fcabc0b01a0 2022-07-21T02:43:47.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcab9d9b700 1 -- 172.21.15.3:0/3463319627 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab0000e90 con 0x7fcabc0b01a0 2022-07-21T02:43:47.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcab9d9b700 1 -- 172.21.15.3:0/3463319627 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcab000d9a0 con 0x7fcabc0b01a0 2022-07-21T02:43:47.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 -- 172.21.15.3:0/3463319627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0b01a0 msgr2=0x7fcabc0a8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 --2- 172.21.15.3:0/3463319627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0b01a0 0x7fcabc0a8540 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7fcab0002f70 tx=0x7fcab0006eb0).stop 2022-07-21T02:43:47.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 -- 172.21.15.3:0/3463319627 shutdown_connections 2022-07-21T02:43:47.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 --2- 172.21.15.3:0/3463319627 >> v2:172.21.15.143:3300/0 conn(0x7fcabc0afd60 0x7fcabc0a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 --2- 172.21.15.3:0/3463319627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0b01a0 0x7fcabc0a8540 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 -- 172.21.15.3:0/3463319627 >> 172.21.15.3:0/3463319627 conn(0x7fcabc060310 msgr2=0x7fcabc0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:47.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 -- 172.21.15.3:0/3463319627 shutdown_connections 2022-07-21T02:43:47.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 -- 172.21.15.3:0/3463319627 wait complete. 2022-07-21T02:43:47.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 Processor -- start 2022-07-21T02:43:47.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.098+0000 7fcac090f700 1 -- start start 2022-07-21T02:43:47.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcac090f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0afd60 0x7fcabc063fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcac090f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcabc0b01a0 0x7fcabc0644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcac090f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcabc0b2710 0x7fcabc066d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcac090f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcabc0b44a0 con 0x7fcabc0afd60 2022-07-21T02:43:47.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcac090f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcabc0b36d0 con 0x7fcabc0b2710 2022-07-21T02:43:47.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcac090f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcabc0b4150 con 0x7fcabc0b01a0 2022-07-21T02:43:47.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0afd60 0x7fcabc063fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0afd60 0x7fcabc063fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52692/0 (socket says 172.21.15.3:52692) 2022-07-21T02:43:47.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabad9d700 1 -- 172.21.15.3:0/1763224523 learned_addr learned my addr 172.21.15.3:0/1763224523 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:47.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabb59e700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcabc0b2710 0x7fcabc066d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.132 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcaba59c700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcabc0b01a0 0x7fcabc0644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabad9d700 1 -- 172.21.15.3:0/1763224523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcabc0b01a0 msgr2=0x7fcabc0644a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabad9d700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcabc0b01a0 0x7fcabc0644a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.133 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabad9d700 1 -- 172.21.15.3:0/1763224523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcabc0b2710 msgr2=0x7fcabc066d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabad9d700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcabc0b2710 0x7fcabc066d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.134 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabad9d700 1 -- 172.21.15.3:0/1763224523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab000b1e0 con 0x7fcabc0afd60 2022-07-21T02:43:47.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabb59e700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcabc0b2710 0x7fcabc066d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T02:43:47.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcabad9d700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0afd60 0x7fcabc063fb0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7fcabc14ae60 tx=0x7fcaac007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.135 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcaabfff700 1 -- 172.21.15.3:0/1763224523 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcaac012400 con 0x7fcabc0afd60 2022-07-21T02:43:47.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcaabfff700 1 -- 172.21.15.3:0/1763224523 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac012560 con 0x7fcabc0afd60 2022-07-21T02:43:47.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcabc0b3920 con 0x7fcabc0afd60 2022-07-21T02:43:47.136 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcaabfff700 1 -- 172.21.15.3:0/1763224523 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcaac0128c0 con 0x7fcabc0afd60 2022-07-21T02:43:47.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.102+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcabc157680 con 0x7fcabc0afd60 2022-07-21T02:43:47.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.106+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcabc0574e0 con 0x7fcabc0afd60 2022-07-21T02:43:47.137 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.106+0000 7fcaabfff700 1 -- 172.21.15.3:0/1763224523 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcaac010070 con 0x7fcabc0afd60 2022-07-21T02:43:47.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.106+0000 7fcaabfff700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca98053b30 0x7fca98055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.106+0000 7fcaabfff700 1 -- 172.21.15.3:0/1763224523 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fcaac06ada0 con 0x7fcabc0afd60 2022-07-21T02:43:47.138 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.106+0000 7fcaba59c700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca98053b30 0x7fca98055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.139 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.106+0000 7fcaba59c700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca98053b30 0x7fca98055ff0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fcab001c000 tx=0x7fcab0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.140 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.110+0000 7fcaabfff700 1 -- 172.21.15.3:0/1763224523 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcaac041d30 con 0x7fcabc0afd60 2022-07-21T02:43:47.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.234+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcabc14b280 con 0x7fca98053b30 2022-07-21T02:43:47.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.234+0000 7fcaabfff700 1 -- 172.21.15.3:0/1763224523 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcabc14b280 con 0x7fca98053b30 2022-07-21T02:43:47.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca98053b30 msgr2=0x7fca98055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.238 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca98053b30 0x7fca98055ff0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fcab001c000 tx=0x7fcab0015000).stop 2022-07-21T02:43:47.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0afd60 msgr2=0x7fcabc063fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0afd60 0x7fcabc063fb0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7fcabc14ae60 tx=0x7fcaac007c50).stop 2022-07-21T02:43:47.239 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 shutdown_connections 2022-07-21T02:43:47.240 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fca98053b30 0x7fca98055ff0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcabc0b2710 0x7fcabc066d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.241 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcabc0b01a0 0x7fcabc0644a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.246 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 --2- 172.21.15.3:0/1763224523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcabc0afd60 0x7fcabc063fb0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 >> 172.21.15.3:0/1763224523 conn(0x7fcabc060310 msgr2=0x7fcabc148cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:47.247 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 shutdown_connections 2022-07-21T02:43:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.238+0000 7fcac090f700 1 -- 172.21.15.3:0/1763224523 wait complete. 2022-07-21T02:43:47.250 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 4 to deep-scrub 2022-07-21T02:43:47.250 INFO:tasks.ceph:Scrubbing osd.5 2022-07-21T02:43:47.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2022-07-21T02:43:47.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.358+0000 7fac0128a700 1 Processor -- start 2022-07-21T02:43:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.358+0000 7fac0128a700 1 -- start start 2022-07-21T02:43:47.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac0128a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fabfc0ad000 0x7fabfc1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac0128a700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc143ab0 con 0x7fabfc0b3260 2022-07-21T02:43:47.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac0128a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc0b0c80 con 0x7fabfc0b08a0 2022-07-21T02:43:47.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac0128a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabfc0ae810 con 0x7fabfc0ad000 2022-07-21T02:43:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fabfb7fe700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fabfc0b3260 0x7fabfc13fa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60982/0 (socket says 172.21.15.3:60982) 2022-07-21T02:43:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fabfb7fe700 1 -- 172.21.15.3:0/776214268 learned_addr learned my addr 172.21.15.3:0/776214268 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:47.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac00a89700 1 --2- 172.21.15.3:0/776214268 >> v2:172.21.15.143:3300/0 conn(0x7fabfc0ad000 0x7fabfc1435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.424 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fabfaffd700 1 -- 172.21.15.3:0/776214268 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452915598 0 0) 0x7fabfc143ab0 con 0x7fabfc0b3260 2022-07-21T02:43:47.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fabfaffd700 1 -- 172.21.15.3:0/776214268 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe8003260 con 0x7fabfc0b3260 2022-07-21T02:43:47.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac00a89700 1 -- 172.21.15.3:0/776214268 >> v1:172.21.15.143:6789/0 conn(0x7fabfc0b08a0 legacy=0x7fabfc142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.425 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac00a89700 1 -- 172.21.15.3:0/776214268 >> v1:172.21.15.3:6789/0 conn(0x7fabfc0b3260 legacy=0x7fabfc13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac00a89700 1 -- 172.21.15.3:0/776214268 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc0aeab0 con 0x7fabfc0ad000 2022-07-21T02:43:47.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac00a89700 1 --2- 172.21.15.3:0/776214268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0ad000 0x7fabfc1435c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fabf40032f0 tx=0x7fabf40091d0).ready entity=mon.1 client_cookie=94b74cdec954b98d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.426 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fabfaffd700 1 -- 172.21.15.3:0/776214268 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf400b680 con 0x7fabfc0ad000 2022-07-21T02:43:47.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fabfaffd700 1 -- 172.21.15.3:0/776214268 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf400b7e0 con 0x7fabfc0ad000 2022-07-21T02:43:47.427 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fabfaffd700 1 -- 172.21.15.3:0/776214268 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf400b950 con 0x7fabfc0ad000 2022-07-21T02:43:47.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac0128a700 1 -- 172.21.15.3:0/776214268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0ad000 msgr2=0x7fabfc1435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.428 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac0128a700 1 --2- 172.21.15.3:0/776214268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0ad000 0x7fabfc1435c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fabf40032f0 tx=0x7fabf40091d0).stop 2022-07-21T02:43:47.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac0128a700 1 -- 172.21.15.3:0/776214268 shutdown_connections 2022-07-21T02:43:47.429 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac0128a700 1 --2- 172.21.15.3:0/776214268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0ad000 0x7fabfc1435c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.362+0000 7fac0128a700 1 -- 172.21.15.3:0/776214268 >> 172.21.15.3:0/776214268 conn(0x7fabfc0612d0 msgr2=0x7fabfc062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 -- 172.21.15.3:0/776214268 shutdown_connections 2022-07-21T02:43:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 -- 172.21.15.3:0/776214268 wait complete. 2022-07-21T02:43:47.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 Processor -- start 2022-07-21T02:43:47.431 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 -- start start 2022-07-21T02:43:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0ad000 0x7fabfc0b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b08a0 0x7fabfc066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b3260 0x7fabfc0a99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0ac490 con 0x7fabfc0b3260 2022-07-21T02:43:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc066600 con 0x7fabfc0b08a0 2022-07-21T02:43:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0ac6c0 con 0x7fabfc0ad000 2022-07-21T02:43:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac00a89700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b3260 0x7fabfc0a99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac00a89700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b3260 0x7fabfc0a99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52708/0 (socket says 172.21.15.3:52708) 2022-07-21T02:43:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac00a89700 1 -- 172.21.15.3:0/3335710458 learned_addr learned my addr 172.21.15.3:0/3335710458 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fabfbfff700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0ad000 0x7fabfc0b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fabfb7fe700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b08a0 0x7fabfc066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac00a89700 1 -- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0ad000 msgr2=0x7fabfc0b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac00a89700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabfc0ad000 0x7fabfc0b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac00a89700 1 -- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b08a0 msgr2=0x7fabfc066cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac00a89700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b08a0 0x7fabfc066cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac00a89700 1 -- 172.21.15.3:0/3335710458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf4009470 con 0x7fabfc0b3260 2022-07-21T02:43:47.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fabfb7fe700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabfc0b08a0 0x7fabfc066cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T02:43:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac00a89700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b3260 0x7fabfc0a99d0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7fabf40032f0 tx=0x7fabf4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fabf97fa700 1 -- 172.21.15.3:0/3335710458 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf400b680 con 0x7fabfc0b3260 2022-07-21T02:43:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fabf97fa700 1 -- 172.21.15.3:0/3335710458 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf4021630 con 0x7fabfc0b3260 2022-07-21T02:43:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc066830 con 0x7fabfc0b3260 2022-07-21T02:43:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fabf97fa700 1 -- 172.21.15.3:0/3335710458 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf4021990 con 0x7fabfc0b3260 2022-07-21T02:43:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.366+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc0a9ec0 con 0x7fabfc0b3260 2022-07-21T02:43:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fabfc0669c0 con 0x7fabfc0b3260 2022-07-21T02:43:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabf97fa700 1 -- 172.21.15.3:0/3335710458 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fabf400f070 con 0x7fabfc0b3260 2022-07-21T02:43:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabf97fa700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 0x7fabdc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabf97fa700 1 -- 172.21.15.3:0/3335710458 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fabf406d410 con 0x7fabfc0b3260 2022-07-21T02:43:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabf97fa700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7fabdc05a700 0x7fabdc05cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabf97fa700 1 -- 172.21.15.3:0/3335710458 --> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabdc05d180 con 0x7fabdc05a700 2022-07-21T02:43:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabfbfff700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 0x7fabdc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabf97fa700 1 -- 172.21.15.3:0/3335710458 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabf406faa0 con 0x7fabfc0b3260 2022-07-21T02:43:47.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabfb7fe700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7fabdc05a700 0x7fabdc05cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabfbfff700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 0x7fabdc055f30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7fabec00d000 tx=0x7fabec007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.370+0000 7fabfb7fe700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7fabdc05a700 0x7fabdc05cba0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.374+0000 7fabf97fa700 1 -- 172.21.15.3:0/3335710458 <== osd.5 v2:172.21.15.143:6800/23340 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7fabdc05d180 con 0x7fabdc05a700 2022-07-21T02:43:47.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.390+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 --> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fabfc0a90a0 con 0x7fabdc05a700 2022-07-21T02:43:47.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.390+0000 7fabf97fa700 1 -- 172.21.15.3:0/3335710458 <== osd.5 v2:172.21.15.143:6800/23340 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fabfc0a90a0 con 0x7fabdc05a700 2022-07-21T02:43:47.451 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T02:43:47.451 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-21T02:43:47.452 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T02:43:47.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.390+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7fabdc05a700 msgr2=0x7fabdc05cba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.390+0000 7fac0128a700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.143:6800/23340,v1:172.21.15.143:6801/23340] conn(0x7fabdc05a700 0x7fabdc05cba0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.390+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 msgr2=0x7fabdc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.390+0000 7fac0128a700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fabdc053a70 0x7fabdc055f30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7fabec00d000 tx=0x7fabec007b90).stop 2022-07-21T02:43:47.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.390+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b3260 msgr2=0x7fabfc0a99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.390+0000 7fac0128a700 1 --2- 172.21.15.3:0/3335710458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fabfc0b3260 0x7fabfc0a99d0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7fabf40032f0 tx=0x7fabf4016000).stop 2022-07-21T02:43:47.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.394+0000 7fac00a89700 1 -- 172.21.15.3:0/3335710458 reap_dead start 2022-07-21T02:43:47.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.394+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 shutdown_connections 2022-07-21T02:43:47.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.394+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 >> 172.21.15.3:0/3335710458 conn(0x7fabfc0612d0 msgr2=0x7fabfc1412d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:47.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.394+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 shutdown_connections 2022-07-21T02:43:47.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.394+0000 7fac0128a700 1 -- 172.21.15.3:0/3335710458 wait complete. 2022-07-21T02:43:47.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2022-07-21T02:43:47.589 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.586+0000 7fe3eecb6700 1 Processor -- start 2022-07-21T02:43:47.590 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.586+0000 7fe3eecb6700 1 -- start start 2022-07-21T02:43:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.586+0000 7fe3eecb6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e80b1160 0x7fe3e81448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.586+0000 7fe3eecb6700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e80b04e0 con 0x7fe3e8148530 2022-07-21T02:43:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.586+0000 7fe3eecb6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e80b0780 con 0x7fe3e80b1f40 2022-07-21T02:43:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.586+0000 7fe3eecb6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e80a9250 con 0x7fe3e80b1160 2022-07-21T02:43:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.586+0000 7fe3e7fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fe3e8148530 0x7fe3e80a7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60996/0 (socket says 172.21.15.3:60996) 2022-07-21T02:43:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.586+0000 7fe3e7fff700 1 -- 172.21.15.3:0/885124327 learned_addr learned my addr 172.21.15.3:0/885124327 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.586+0000 7fe3ed253700 1 --2- 172.21.15.3:0/885124327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e80b1160 0x7fe3e81448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3e77fe700 1 -- 172.21.15.3:0/885124327 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261943053 0 0) 0x7fe3e80b04e0 con 0x7fe3e8148530 2022-07-21T02:43:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3e77fe700 1 -- 172.21.15.3:0/885124327 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d4003260 con 0x7fe3e8148530 2022-07-21T02:43:47.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3ed253700 1 -- 172.21.15.3:0/885124327 >> v1:172.21.15.143:6789/0 conn(0x7fe3e80b1f40 legacy=0x7fe3e80b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3ed253700 1 -- 172.21.15.3:0/885124327 >> v1:172.21.15.3:6789/0 conn(0x7fe3e8148530 legacy=0x7fe3e80a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.595 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3ed253700 1 -- 172.21.15.3:0/885124327 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e80b0a20 con 0x7fe3e80b1160 2022-07-21T02:43:47.596 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3ed253700 1 --2- 172.21.15.3:0/885124327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e80b1160 0x7fe3e81448b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fe3e81478c0 tx=0x7fe3e0006df0).ready entity=mon.2 client_cookie=11876ec7ded6b172 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.607 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3e77fe700 1 -- 172.21.15.3:0/885124327 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3e000d5c0 con 0x7fe3e80b1160 2022-07-21T02:43:47.608 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3e77fe700 1 -- 172.21.15.3:0/885124327 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e0000e90 con 0x7fe3e80b1160 2022-07-21T02:43:47.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3e77fe700 1 -- 172.21.15.3:0/885124327 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3e000d910 con 0x7fe3e80b1160 2022-07-21T02:43:47.609 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3eecb6700 1 -- 172.21.15.3:0/885124327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e80b1160 msgr2=0x7fe3e81448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3eecb6700 1 --2- 172.21.15.3:0/885124327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e80b1160 0x7fe3e81448b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fe3e81478c0 tx=0x7fe3e0006df0).stop 2022-07-21T02:43:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3eecb6700 1 -- 172.21.15.3:0/885124327 shutdown_connections 2022-07-21T02:43:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3eecb6700 1 --2- 172.21.15.3:0/885124327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e80b1160 0x7fe3e81448b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3eecb6700 1 -- 172.21.15.3:0/885124327 >> 172.21.15.3:0/885124327 conn(0x7fe3e805f840 msgr2=0x7fe3e805fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:47.622 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3eecb6700 1 -- 172.21.15.3:0/885124327 shutdown_connections 2022-07-21T02:43:47.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.590+0000 7fe3eecb6700 1 -- 172.21.15.3:0/885124327 wait complete. 2022-07-21T02:43:47.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eecb6700 1 Processor -- start 2022-07-21T02:43:47.623 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eecb6700 1 -- start start 2022-07-21T02:43:47.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eecb6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e80b1160 0x7fe3e8141610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eecb6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e80b1f40 0x7fe3e80ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.624 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eecb6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e8148530 0x7fe3e80af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eecb6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e80aef60 con 0x7fe3e80b1160 2022-07-21T02:43:47.625 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eecb6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e8142260 con 0x7fe3e80b1f40 2022-07-21T02:43:47.626 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eecb6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e814b360 con 0x7fe3e8148530 2022-07-21T02:43:47.627 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eca52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e80b1160 0x7fe3e8141610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eca52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e80b1160 0x7fe3e8141610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52722/0 (socket says 172.21.15.3:52722) 2022-07-21T02:43:47.628 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eca52700 1 -- 172.21.15.3:0/1089784260 learned_addr learned my addr 172.21.15.3:0/1089784260 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:47.629 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3e7fff700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e80b1f40 0x7fe3e80ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3ed253700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e8148530 0x7fe3e80af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.630 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eca52700 1 -- 172.21.15.3:0/1089784260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e8148530 msgr2=0x7fe3e80af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.631 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eca52700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e8148530 0x7fe3e80af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.633 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eca52700 1 -- 172.21.15.3:0/1089784260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e80b1f40 msgr2=0x7fe3e80ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eca52700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e80b1f40 0x7fe3e80ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.638 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eca52700 1 -- 172.21.15.3:0/1089784260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e000b180 con 0x7fe3e80b1160 2022-07-21T02:43:47.639 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3e7fff700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e80b1f40 0x7fe3e80ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T02:43:47.640 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eca52700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e80b1160 0x7fe3e8141610 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7fe3e8061530 tx=0x7fe3dc007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.641 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3e5ffb700 1 -- 172.21.15.3:0/1089784260 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3dc00ce30 con 0x7fe3e80b1160 2022-07-21T02:43:47.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.594+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e80ae1c0 con 0x7fe3e80b1160 2022-07-21T02:43:47.642 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.598+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e80afa40 con 0x7fe3e80b1160 2022-07-21T02:43:47.643 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.598+0000 7fe3e5ffb700 1 -- 172.21.15.3:0/1089784260 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc007240 con 0x7fe3e80b1160 2022-07-21T02:43:47.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.598+0000 7fe3e5ffb700 1 -- 172.21.15.3:0/1089784260 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3dc006700 con 0x7fe3e80b1160 2022-07-21T02:43:47.644 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.598+0000 7fe3e5ffb700 1 -- 172.21.15.3:0/1089784260 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe3dc010030 con 0x7fe3e80b1160 2022-07-21T02:43:47.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.598+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c0003410 con 0x7fe3e80b1160 2022-07-21T02:43:47.645 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.598+0000 7fe3e5ffb700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3c80539e0 0x7fe3c8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.647 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.598+0000 7fe3e5ffb700 1 -- 172.21.15.3:0/1089784260 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fe3dc06bb50 con 0x7fe3e80b1160 2022-07-21T02:43:47.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.598+0000 7fe3e7fff700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3c80539e0 0x7fe3c8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.648 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.602+0000 7fe3e7fff700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3c80539e0 0x7fe3c8055ea0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7fe3d800b000 tx=0x7fe3d8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.650 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.606+0000 7fe3e5ffb700 1 -- 172.21.15.3:0/1089784260 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe3dc042a60 con 0x7fe3e80b1160 2022-07-21T02:43:47.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3c0002070 con 0x7fe3c80539e0 2022-07-21T02:43:47.735 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3e5ffb700 1 -- 172.21.15.3:0/1089784260 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe3c0002070 con 0x7fe3c80539e0 2022-07-21T02:43:47.737 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3c80539e0 msgr2=0x7fe3c8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3c80539e0 0x7fe3c8055ea0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7fe3d800b000 tx=0x7fe3d8009040).stop 2022-07-21T02:43:47.739 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e80b1160 msgr2=0x7fe3e8141610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.741 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e80b1160 0x7fe3e8141610 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7fe3e8061530 tx=0x7fe3dc007c10).stop 2022-07-21T02:43:47.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 shutdown_connections 2022-07-21T02:43:47.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3e8148530 0x7fe3e80af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.742 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe3c80539e0 0x7fe3c8055ea0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3e80b1f40 0x7fe3e80ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.743 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 --2- 172.21.15.3:0/1089784260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3e80b1160 0x7fe3e8141610 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.745 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 >> 172.21.15.3:0/1089784260 conn(0x7fe3e805f840 msgr2=0x7fe3e8147110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:47.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 shutdown_connections 2022-07-21T02:43:47.746 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.734+0000 7fe3eecb6700 1 -- 172.21.15.3:0/1089784260 wait complete. 2022-07-21T02:43:47.749 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 5 to deep-scrub 2022-07-21T02:43:47.750 INFO:tasks.ceph:Scrubbing osd.6 2022-07-21T02:43:47.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2022-07-21T02:43:47.872 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa348dc9700 1 Processor -- start 2022-07-21T02:43:47.873 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa348dc9700 1 -- start start 2022-07-21T02:43:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa348dc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440b2710 0x7fa34414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa348dc9700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3440a7250 con 0x7fa3440a8c50 2022-07-21T02:43:47.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa348dc9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3440a7480 con 0x7fa3440b01a0 2022-07-21T02:43:47.889 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa348dc9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440a7720 con 0x7fa3440b2710 2022-07-21T02:43:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa343fff700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa3440a8c50 0x7fa344151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32780/0 (socket says 172.21.15.3:32780) 2022-07-21T02:43:47.891 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa343fff700 1 -- 172.21.15.3:0/4140179891 learned_addr learned my addr 172.21.15.3:0/4140179891 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:47.892 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa3437fe700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440b2710 0x7fa34414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.870+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207833958 0 0) 0x7fa3440a7250 con 0x7fa3440a8c50 2022-07-21T02:43:47.893 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324003260 con 0x7fa3440a8c50 2022-07-21T02:43:47.894 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297633596 0 0) 0x7fa3440a7480 con 0x7fa3440b01a0 2022-07-21T02:43:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3440a7250 con 0x7fa3440b01a0 2022-07-21T02:43:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2361308500 0 0) 0x7fa324003260 con 0x7fa3440a8c50 2022-07-21T02:43:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440b2710 msgr2=0x7fa34414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440b2710 0x7fa34414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 >> v1:172.21.15.143:6789/0 conn(0x7fa3440b01a0 legacy=0x7fa3440a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.898 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440a79c0 con 0x7fa3440a8c50 2022-07-21T02:43:47.899 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 124874894 0 0) 0x7fa33c001660 con 0x7fa3440a8c50 2022-07-21T02:43:47.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 >> v1:172.21.15.3:6789/0 conn(0x7fa3440a8c50 legacy=0x7fa344151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.900 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa324003910 0x7fa324003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.901 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 0x7fa324004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.902 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa324004e70 0x7fa32400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440a7720 con 0x7fa324004500 2022-07-21T02:43:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32400bc40 con 0x7fa324004e70 2022-07-21T02:43:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32400be00 con 0x7fa324003910 2022-07-21T02:43:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa342ffd700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 0x7fa324004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3437fe700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440b2710 0x7fa34414afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T02:43:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa343fff700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa324004e70 0x7fa32400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa342ffd700 1 -- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa324003910 msgr2=0x7fa324003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3437fe700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa324003910 0x7fa324003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.908 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa342ffd700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa324003910 0x7fa324003d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa342ffd700 1 -- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa324004e70 msgr2=0x7fa32400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:47.910 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa342ffd700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa324004e70 0x7fa32400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa342ffd700 1 -- 172.21.15.3:0/4140179891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324003260 con 0x7fa324004500 2022-07-21T02:43:47.911 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa343fff700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa324004e70 0x7fa32400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-21T02:43:47.912 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa342ffd700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 0x7fa324004980 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7fa334001490 tx=0x7fa334007d60).ready entity=mon.0 client_cookie=cab1de8a1c4e524b server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa334006890 con 0x7fa324004500 2022-07-21T02:43:47.913 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3340069f0 con 0x7fa324004500 2022-07-21T02:43:47.914 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa3427fc700 1 -- 172.21.15.3:0/4140179891 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa334006d50 con 0x7fa324004500 2022-07-21T02:43:47.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa348dc9700 1 -- 172.21.15.3:0/4140179891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 msgr2=0x7fa324004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.915 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa348dc9700 1 --2- 172.21.15.3:0/4140179891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 0x7fa324004980 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7fa334001490 tx=0x7fa334007d60).stop 2022-07-21T02:43:47.916 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa343fff700 1 -- 172.21.15.3:0/4140179891 reap_dead start 2022-07-21T02:43:47.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa348dc9700 1 -- 172.21.15.3:0/4140179891 shutdown_connections 2022-07-21T02:43:47.917 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa348dc9700 1 -- 172.21.15.3:0/4140179891 >> 172.21.15.3:0/4140179891 conn(0x7fa344060310 msgr2=0x7fa3440b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:47.918 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa348dc9700 1 -- 172.21.15.3:0/4140179891 shutdown_connections 2022-07-21T02:43:47.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.874+0000 7fa348dc9700 1 -- 172.21.15.3:0/4140179891 wait complete. 2022-07-21T02:43:47.919 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 Processor -- start 2022-07-21T02:43:47.920 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 -- start start 2022-07-21T02:43:47.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 0x7fa34405c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.921 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440aae30 0x7fa344156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.922 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3440ab2b0 0x7fa344156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440aa910 con 0x7fa324004500 2022-07-21T02:43:47.923 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440ac0e0 con 0x7fa3440ab2b0 2022-07-21T02:43:47.924 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440ac380 con 0x7fa3440aae30 2022-07-21T02:43:47.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 0x7fa34405c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.925 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 0x7fa34405c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52742/0 (socket says 172.21.15.3:52742) 2022-07-21T02:43:47.926 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa3437fe700 1 -- 172.21.15.3:0/371072305 learned_addr learned my addr 172.21.15.3:0/371072305 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:47.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa342ffd700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440aae30 0x7fa344156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.927 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa3437fe700 1 -- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440aae30 msgr2=0x7fa344156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.928 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa3437fe700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440aae30 0x7fa344156a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa3437fe700 1 -- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3440ab2b0 msgr2=0x7fa344156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa3437fe700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3440ab2b0 0x7fa344156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-21T02:43:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa3437fe700 1 -- 172.21.15.3:0/371072305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440ac620 con 0x7fa324004500 2022-07-21T02:43:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa342ffd700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3440aae30 0x7fa344156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-21T02:43:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa3437fe700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 0x7fa34405c680 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 rx=0x7fa32c00b000 tx=0x7fa32c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa340ff9700 1 -- 172.21.15.3:0/371072305 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa32c017070 con 0x7fa324004500 2022-07-21T02:43:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa340ff9700 1 -- 172.21.15.3:0/371072305 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c000e90 con 0x7fa324004500 2022-07-21T02:43:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa340ff9700 1 -- 172.21.15.3:0/371072305 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa32c012a20 con 0x7fa324004500 2022-07-21T02:43:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3440ac8c0 con 0x7fa324004500 2022-07-21T02:43:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa344157420 con 0x7fa324004500 2022-07-21T02:43:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa3441576d0 con 0x7fa324004500 2022-07-21T02:43:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa340ff9700 1 -- 172.21.15.3:0/371072305 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa32c012b80 con 0x7fa324004500 2022-07-21T02:43:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.878+0000 7fa340ff9700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa328053b00 0x7fa328055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.937 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.882+0000 7fa340ff9700 1 -- 172.21.15.3:0/371072305 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fa32c06c930 con 0x7fa324004500 2022-07-21T02:43:47.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.882+0000 7fa340ff9700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fa32805a790 0x7fa32805cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:47.938 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.882+0000 7fa340ff9700 1 -- 172.21.15.3:0/371072305 --> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa32805d210 con 0x7fa32805a790 2022-07-21T02:43:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.882+0000 7fa340ff9700 1 -- 172.21.15.3:0/371072305 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa32c06ccf0 con 0x7fa324004500 2022-07-21T02:43:47.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.882+0000 7fa342ffd700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa328053b00 0x7fa328055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.940 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.882+0000 7fa343fff700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fa32805a790 0x7fa32805cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:47.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.882+0000 7fa342ffd700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa328053b00 0x7fa328055fc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fa334004000 tx=0x7fa334016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.941 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.882+0000 7fa343fff700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fa32805a790 0x7fa32805cc30 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:47.942 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.882+0000 7fa340ff9700 1 -- 172.21.15.3:0/371072305 <== osd.6 v2:172.21.15.143:6824/23342 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7fa32805d210 con 0x7fa32805a790 2022-07-21T02:43:47.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 --> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa3440ad680 con 0x7fa32805a790 2022-07-21T02:43:47.945 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa340ff9700 1 -- 172.21.15.3:0/371072305 <== osd.6 v2:172.21.15.143:6824/23342 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa3440ad680 con 0x7fa32805a790 2022-07-21T02:43:47.946 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T02:43:47.947 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-21T02:43:47.948 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T02:43:47.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fa32805a790 msgr2=0x7fa32805cc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.143:6824/23342,v1:172.21.15.143:6825/23342] conn(0x7fa32805a790 0x7fa32805cc30 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:47.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa328053b00 msgr2=0x7fa328055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fa328053b00 0x7fa328055fc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fa334004000 tx=0x7fa334016000).stop 2022-07-21T02:43:47.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 msgr2=0x7fa34405c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:47.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 --2- 172.21.15.3:0/371072305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa324004500 0x7fa34405c680 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 rx=0x7fa32c00b000 tx=0x7fa32c008040).stop 2022-07-21T02:43:47.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa343fff700 1 -- 172.21.15.3:0/371072305 reap_dead start 2022-07-21T02:43:47.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 shutdown_connections 2022-07-21T02:43:47.956 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 >> 172.21.15.3:0/371072305 conn(0x7fa344060310 msgr2=0x7fa34414b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:47.957 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 shutdown_connections 2022-07-21T02:43:47.958 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:47.902+0000 7fa348dc9700 1 -- 172.21.15.3:0/371072305 wait complete. 2022-07-21T02:43:47.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2022-07-21T02:43:48.088 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.070+0000 7fc61c22d700 1 Processor -- start 2022-07-21T02:43:48.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.070+0000 7fc61c22d700 1 -- start start 2022-07-21T02:43:48.089 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc6140b1de0 0x7fc6140ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc6140b2af0 0x7fc6140abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.090 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc614143650 con 0x7fc614139c70 2022-07-21T02:43:48.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6140b31a0 con 0x7fc6140b2af0 2022-07-21T02:43:48.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6140af4b0 con 0x7fc6140b1de0 2022-07-21T02:43:48.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61a7ca700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc6140b2af0 0x7fc6140abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61a7ca700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fc6140b2af0 0x7fc6140abbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52756/0 (socket says 172.21.15.3:52756) 2022-07-21T02:43:48.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61a7ca700 1 -- 172.21.15.3:0/826118915 learned_addr learned my addr 172.21.15.3:0/826118915 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:48.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc6197c8700 1 --2- 172.21.15.3:0/826118915 >> v2:172.21.15.143:3300/0 conn(0x7fc6140b1de0 0x7fc6140ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61a7ca700 1 -- 172.21.15.3:0/826118915 >> v2:172.21.15.143:3300/0 conn(0x7fc6140b1de0 msgr2=0x7fc6140ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:48.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61a7ca700 1 --2- 172.21.15.3:0/826118915 >> v2:172.21.15.143:3300/0 conn(0x7fc6140b1de0 0x7fc6140ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.105 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61a7ca700 1 -- 172.21.15.3:0/826118915 >> v1:172.21.15.143:6789/0 conn(0x7fc614139c70 legacy=0x7fc614142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.106 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61a7ca700 1 -- 172.21.15.3:0/826118915 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6140ac600 con 0x7fc6140b2af0 2022-07-21T02:43:48.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61a7ca700 1 --2- 172.21.15.3:0/826118915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b2af0 0x7fc6140abbb0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7fc6140af2b0 tx=0x7fc60c00b530).ready entity=mon.0 client_cookie=1b627c194ff9499d server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc618fc7700 1 -- 172.21.15.3:0/826118915 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc60c00ddb0 con 0x7fc6140b2af0 2022-07-21T02:43:48.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc618fc7700 1 -- 172.21.15.3:0/826118915 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c010470 con 0x7fc6140b2af0 2022-07-21T02:43:48.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc618fc7700 1 -- 172.21.15.3:0/826118915 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc60c0107d0 con 0x7fc6140b2af0 2022-07-21T02:43:48.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 -- 172.21.15.3:0/826118915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b2af0 msgr2=0x7fc6140abbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 --2- 172.21.15.3:0/826118915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b2af0 0x7fc6140abbb0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7fc6140af2b0 tx=0x7fc60c00b530).stop 2022-07-21T02:43:48.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 -- 172.21.15.3:0/826118915 shutdown_connections 2022-07-21T02:43:48.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 --2- 172.21.15.3:0/826118915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b2af0 0x7fc6140abbb0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 --2- 172.21.15.3:0/826118915 >> v2:172.21.15.143:3300/0 conn(0x7fc6140b1de0 0x7fc6140ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 -- 172.21.15.3:0/826118915 >> 172.21.15.3:0/826118915 conn(0x7fc61405f840 msgr2=0x7fc61405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:48.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 -- 172.21.15.3:0/826118915 shutdown_connections 2022-07-21T02:43:48.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.074+0000 7fc61c22d700 1 -- 172.21.15.3:0/826118915 wait complete. 2022-07-21T02:43:48.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc61c22d700 1 Processor -- start 2022-07-21T02:43:48.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc61c22d700 1 -- start start 2022-07-21T02:43:48.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc61c22d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b1de0 0x7fc61413ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc61c22d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6140b2af0 0x7fc6140aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc61c22d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc614142550 0x7fc614142db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc61c22d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc614152920 con 0x7fc6140b1de0 2022-07-21T02:43:48.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc61c22d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61413d920 con 0x7fc614142550 2022-07-21T02:43:48.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc61c22d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6141417d0 con 0x7fc6140b2af0 2022-07-21T02:43:48.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc619fc9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b1de0 0x7fc61413ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc619fc9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b1de0 0x7fc61413ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52758/0 (socket says 172.21.15.3:52758) 2022-07-21T02:43:48.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc619fc9700 1 -- 172.21.15.3:0/2701639736 learned_addr learned my addr 172.21.15.3:0/2701639736 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:48.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc61a7ca700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc614142550 0x7fc614142db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.078+0000 7fc6197c8700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6140b2af0 0x7fc6140aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc619fc9700 1 -- 172.21.15.3:0/2701639736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6140b2af0 msgr2=0x7fc6140aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc619fc9700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6140b2af0 0x7fc6140aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc619fc9700 1 -- 172.21.15.3:0/2701639736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc614142550 msgr2=0x7fc614142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc619fc9700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc614142550 0x7fc614142db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc619fc9700 1 -- 172.21.15.3:0/2701639736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c00b7d0 con 0x7fc6140b1de0 2022-07-21T02:43:48.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc619fc9700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b1de0 0x7fc61413ca90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7fc6140afc20 tx=0x7fc604007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc602ffd700 1 -- 172.21.15.3:0/2701639736 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc604007e70 con 0x7fc6140b1de0 2022-07-21T02:43:48.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc602ffd700 1 -- 172.21.15.3:0/2701639736 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc604007030 con 0x7fc6140b1de0 2022-07-21T02:43:48.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc602ffd700 1 -- 172.21.15.3:0/2701639736 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc604007390 con 0x7fc6140b1de0 2022-07-21T02:43:48.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61413db80 con 0x7fc6140b1de0 2022-07-21T02:43:48.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc614142a80 con 0x7fc6140b1de0 2022-07-21T02:43:48.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6141438c0 con 0x7fc6140b1de0 2022-07-21T02:43:48.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc602ffd700 1 -- 172.21.15.3:0/2701639736 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc604010030 con 0x7fc6140b1de0 2022-07-21T02:43:48.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.082+0000 7fc602ffd700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5f4053950 0x7fc5f4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.128 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.086+0000 7fc602ffd700 1 -- 172.21.15.3:0/2701639736 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fc60406b880 con 0x7fc6140b1de0 2022-07-21T02:43:48.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.086+0000 7fc6197c8700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5f4053950 0x7fc5f4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.129 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.086+0000 7fc6197c8700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5f4053950 0x7fc5f4055e10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fc61000b000 tx=0x7fc610006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.130 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.090+0000 7fc602ffd700 1 -- 172.21.15.3:0/2701639736 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc604042920 con 0x7fc6140b1de0 2022-07-21T02:43:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.222+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6140574e0 con 0x7fc5f4053950 2022-07-21T02:43:48.223 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.222+0000 7fc602ffd700 1 -- 172.21.15.3:0/2701639736 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc6140574e0 con 0x7fc5f4053950 2022-07-21T02:43:48.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.222+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5f4053950 msgr2=0x7fc5f4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.232 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.222+0000 7fc61c22d700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5f4053950 0x7fc5f4055e10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fc61000b000 tx=0x7fc610006df0).stop 2022-07-21T02:43:48.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.222+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b1de0 msgr2=0x7fc61413ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.233 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.222+0000 7fc61c22d700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b1de0 0x7fc61413ca90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7fc6140afc20 tx=0x7fc604007c00).stop 2022-07-21T02:43:48.234 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.226+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 shutdown_connections 2022-07-21T02:43:48.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.226+0000 7fc61c22d700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fc5f4053950 0x7fc5f4055e10 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.235 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.226+0000 7fc61c22d700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc614142550 0x7fc614142db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.226+0000 7fc61c22d700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6140b2af0 0x7fc6140aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.226+0000 7fc61c22d700 1 --2- 172.21.15.3:0/2701639736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6140b1de0 0x7fc61413ca90 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.236 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.226+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 >> 172.21.15.3:0/2701639736 conn(0x7fc61405f840 msgr2=0x7fc6140b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:48.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.226+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 shutdown_connections 2022-07-21T02:43:48.237 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.226+0000 7fc61c22d700 1 -- 172.21.15.3:0/2701639736 wait complete. 2022-07-21T02:43:48.238 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 6 to deep-scrub 2022-07-21T02:43:48.244 INFO:tasks.ceph:Scrubbing osd.7 2022-07-21T02:43:48.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2022-07-21T02:43:48.360 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.354+0000 7f576cfb2700 1 Processor -- start 2022-07-21T02:43:48.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.354+0000 7f576cfb2700 1 -- start start 2022-07-21T02:43:48.361 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f576805c960 0x7f57680b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f576805d240 0x7f576813f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.364 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57680acfc0 0x7f57680aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.365 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5768067f00 con 0x7f576805c960 2022-07-21T02:43:48.366 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576813fa10 con 0x7f576805d240 2022-07-21T02:43:48.367 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576805cd40 con 0x7f57680acfc0 2022-07-21T02:43:48.368 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57677fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f576805c960 0x7f57680b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57677fe700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f576805c960 0x7f57680b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52768/0 (socket says 172.21.15.3:52768) 2022-07-21T02:43:48.369 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57677fe700 1 -- 172.21.15.3:0/1164122306 learned_addr learned my addr 172.21.15.3:0/1164122306 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:48.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f5766ffd700 1 --2- 172.21.15.3:0/1164122306 >> v2:172.21.15.143:3300/0 conn(0x7f576805d240 0x7f576813f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.370 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f5767fff700 1 --2- 172.21.15.3:0/1164122306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57680acfc0 0x7f57680aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57677fe700 1 -- 172.21.15.3:0/1164122306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57680acfc0 msgr2=0x7f57680aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57677fe700 1 --2- 172.21.15.3:0/1164122306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57680acfc0 0x7f57680aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57677fe700 1 -- 172.21.15.3:0/1164122306 >> v2:172.21.15.143:3300/0 conn(0x7f576805d240 msgr2=0x7f576813f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:48.386 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57677fe700 1 --2- 172.21.15.3:0/1164122306 >> v2:172.21.15.143:3300/0 conn(0x7f576805d240 0x7f576813f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57677fe700 1 -- 172.21.15.3:0/1164122306 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57680b1920 con 0x7f576805c960 2022-07-21T02:43:48.388 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57677fe700 1 --2- 172.21.15.3:0/1164122306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 0x7f57680b1280 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7f57680ae9d0 tx=0x7f5758006eb0).ready entity=mon.0 client_cookie=3858550d7d10008 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.389 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57667fc700 1 -- 172.21.15.3:0/1164122306 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f575800e610 con 0x7f576805c960 2022-07-21T02:43:48.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57667fc700 1 -- 172.21.15.3:0/1164122306 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5758000e90 con 0x7f576805c960 2022-07-21T02:43:48.390 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f57667fc700 1 -- 172.21.15.3:0/1164122306 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f575800e9a0 con 0x7f576805c960 2022-07-21T02:43:48.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 -- 172.21.15.3:0/1164122306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 msgr2=0x7f57680b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.391 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 --2- 172.21.15.3:0/1164122306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 0x7f57680b1280 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7f57680ae9d0 tx=0x7f5758006eb0).stop 2022-07-21T02:43:48.392 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 -- 172.21.15.3:0/1164122306 shutdown_connections 2022-07-21T02:43:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 --2- 172.21.15.3:0/1164122306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57680acfc0 0x7f57680aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 --2- 172.21.15.3:0/1164122306 >> v2:172.21.15.143:3300/0 conn(0x7f576805d240 0x7f576813f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 --2- 172.21.15.3:0/1164122306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 0x7f57680b1280 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 -- 172.21.15.3:0/1164122306 >> 172.21.15.3:0/1164122306 conn(0x7f5768060310 msgr2=0x7f57680b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:48.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 -- 172.21.15.3:0/1164122306 shutdown_connections 2022-07-21T02:43:48.395 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.358+0000 7f576cfb2700 1 -- 172.21.15.3:0/1164122306 wait complete. 2022-07-21T02:43:48.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f576cfb2700 1 Processor -- start 2022-07-21T02:43:48.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f576cfb2700 1 -- start start 2022-07-21T02:43:48.396 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f576cfb2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 0x7f57680b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f576cfb2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576805d240 0x7f57680b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.397 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f576cfb2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57680acfc0 0x7f57680a9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f576cfb2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57680b3000 con 0x7f576805c960 2022-07-21T02:43:48.398 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f576cfb2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57680b32a0 con 0x7f57680acfc0 2022-07-21T02:43:48.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f576cfb2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57680b3540 con 0x7f576805d240 2022-07-21T02:43:48.399 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f57677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 0x7f57680b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f57677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 0x7f57680b3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52772/0 (socket says 172.21.15.3:52772) 2022-07-21T02:43:48.400 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f57677fe700 1 -- 172.21.15.3:0/3887627902 learned_addr learned my addr 172.21.15.3:0/3887627902 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:48.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f5767fff700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57680acfc0 0x7f57680a9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f5766ffd700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576805d240 0x7f57680b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.401 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f57677fe700 1 -- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576805d240 msgr2=0x7f57680b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f57677fe700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576805d240 0x7f57680b0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.402 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f57677fe700 1 -- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57680acfc0 msgr2=0x7f57680a9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f57677fe700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57680acfc0 0x7f57680a9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.362+0000 7f57677fe700 1 -- 172.21.15.3:0/3887627902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575800b1e0 con 0x7f576805c960 2022-07-21T02:43:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f57677fe700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 0x7f57680b3cf0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f57680ae9d0 tx=0x7f5758015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5764ff9700 1 -- 172.21.15.3:0/3887627902 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5758011760 con 0x7f576805c960 2022-07-21T02:43:48.404 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5764ff9700 1 -- 172.21.15.3:0/3887627902 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57580118c0 con 0x7f576805c960 2022-07-21T02:43:48.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57680b5ed0 con 0x7f576805c960 2022-07-21T02:43:48.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5764ff9700 1 -- 172.21.15.3:0/3887627902 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5758011c40 con 0x7f576805c960 2022-07-21T02:43:48.405 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57680b1490 con 0x7f576805c960 2022-07-21T02:43:48.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57680b1bc0 con 0x7f576805c960 2022-07-21T02:43:48.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5764ff9700 1 -- 172.21.15.3:0/3887627902 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f575800f070 con 0x7f576805c960 2022-07-21T02:43:48.406 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5764ff9700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57480539e0 0x7f5748055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5764ff9700 1 -- 172.21.15.3:0/3887627902 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f575806d6a0 con 0x7f576805c960 2022-07-21T02:43:48.407 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5766ffd700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57480539e0 0x7f5748055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.408 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5764ff9700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f574805a670 0x7f574805cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5764ff9700 1 -- 172.21.15.3:0/3887627902 --> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f574805d190 con 0x7f574805a670 2022-07-21T02:43:48.409 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5764ff9700 1 -- 172.21.15.3:0/3887627902 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=164) v2 ==== 24+0+0 (secure 0 0 0) 0x7f575806fd30 con 0x7f576805c960 2022-07-21T02:43:48.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.366+0000 7f5767fff700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f574805a670 0x7f574805cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.410 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.370+0000 7f5766ffd700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57480539e0 0x7f5748055ea0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f575c00b000 tx=0x7f575c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.370+0000 7f5767fff700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f574805a670 0x7f574805cb10 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.411 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.370+0000 7f5764ff9700 1 -- 172.21.15.3:0/3887627902 <== osd.7 v2:172.21.15.143:6816/23341 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23441 (crc 0 0 0) 0x7f574805d190 con 0x7f574805a670 2022-07-21T02:43:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.386+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 --> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f57680574e0 con 0x7f574805a670 2022-07-21T02:43:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.386+0000 7f5764ff9700 1 -- 172.21.15.3:0/3887627902 <== osd.7 v2:172.21.15.143:6816/23341 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f57680574e0 con 0x7f574805a670 2022-07-21T02:43:48.413 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2022-07-21T02:43:48.413 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-21T02:43:48.414 INFO:teuthology.orchestra.run.smithi003.stdout:} 2022-07-21T02:43:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.386+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f574805a670 msgr2=0x7f574805cb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.386+0000 7f576cfb2700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.143:6816/23341,v1:172.21.15.143:6817/23341] conn(0x7f574805a670 0x7f574805cb10 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.386+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57480539e0 msgr2=0x7f5748055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.418 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.386+0000 7f576cfb2700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f57480539e0 0x7f5748055ea0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f575c00b000 tx=0x7f575c006df0).stop 2022-07-21T02:43:48.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.386+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 msgr2=0x7f57680b3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.419 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.386+0000 7f576cfb2700 1 --2- 172.21.15.3:0/3887627902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f576805c960 0x7f57680b3cf0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f57680ae9d0 tx=0x7f5758015000).stop 2022-07-21T02:43:48.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.386+0000 7f5767fff700 1 -- 172.21.15.3:0/3887627902 reap_dead start 2022-07-21T02:43:48.420 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.390+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 shutdown_connections 2022-07-21T02:43:48.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.390+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 >> 172.21.15.3:0/3887627902 conn(0x7f5768060310 msgr2=0x7f576813d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:48.421 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.390+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 shutdown_connections 2022-07-21T02:43:48.422 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.390+0000 7f576cfb2700 1 -- 172.21.15.3:0/3887627902 wait complete. 2022-07-21T02:43:48.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2022-07-21T02:43:48.539 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e95b700 1 Processor -- start 2022-07-21T02:43:48.540 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e95b700 1 -- start start 2022-07-21T02:43:48.541 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e95b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b08a0 0x7fed48142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.543 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e95b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fed480ad000 0x7fed4813fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e95b700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed48143ab0 con 0x7fed480b3260 2022-07-21T02:43:48.544 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e95b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed480b0c80 con 0x7fed480ad000 2022-07-21T02:43:48.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e95b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed480ae810 con 0x7fed480b08a0 2022-07-21T02:43:48.545 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4d158700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fed480b3260 0x7fed481433a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32832/0 (socket says 172.21.15.3:32832) 2022-07-21T02:43:48.546 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4d158700 1 -- 172.21.15.3:0/1315724232 learned_addr learned my addr 172.21.15.3:0/1315724232 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:48.547 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e15a700 1 --2- 172.21.15.3:0/1315724232 >> v2:172.21.15.143:3300/0 conn(0x7fed480ad000 0x7fed4813fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.548 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4d959700 1 --2- 172.21.15.3:0/1315724232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b08a0 0x7fed48142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4c957700 1 -- 172.21.15.3:0/1315724232 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808750951 0 0) 0x7fed48143ab0 con 0x7fed480b3260 2022-07-21T02:43:48.549 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4c957700 1 -- 172.21.15.3:0/1315724232 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed2c003260 con 0x7fed480b3260 2022-07-21T02:43:48.550 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e15a700 1 -- 172.21.15.3:0/1315724232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b08a0 msgr2=0x7fed48142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:48.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e15a700 1 --2- 172.21.15.3:0/1315724232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b08a0 0x7fed48142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.551 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e15a700 1 -- 172.21.15.3:0/1315724232 >> v1:172.21.15.3:6789/0 conn(0x7fed480b3260 legacy=0x7fed481433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.552 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e15a700 1 -- 172.21.15.3:0/1315724232 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed480aeab0 con 0x7fed480ad000 2022-07-21T02:43:48.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4e15a700 1 --2- 172.21.15.3:0/1315724232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 0x7fed4813fa30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fed400032f0 tx=0x7fed400091d0).ready entity=mon.1 client_cookie=9148aba3aed8a854 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.553 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4c957700 1 -- 172.21.15.3:0/1315724232 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed4000f8a0 con 0x7fed480ad000 2022-07-21T02:43:48.554 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4c957700 1 -- 172.21.15.3:0/1315724232 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4000fa00 con 0x7fed480ad000 2022-07-21T02:43:48.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.534+0000 7fed4c957700 1 -- 172.21.15.3:0/1315724232 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed4000fd60 con 0x7fed480ad000 2022-07-21T02:43:48.555 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 -- 172.21.15.3:0/1315724232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 msgr2=0x7fed4813fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 --2- 172.21.15.3:0/1315724232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 0x7fed4813fa30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fed400032f0 tx=0x7fed400091d0).stop 2022-07-21T02:43:48.556 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 -- 172.21.15.3:0/1315724232 shutdown_connections 2022-07-21T02:43:48.558 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 --2- 172.21.15.3:0/1315724232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 0x7fed4813fa30 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 --2- 172.21.15.3:0/1315724232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b08a0 0x7fed48142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.559 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 -- 172.21.15.3:0/1315724232 >> 172.21.15.3:0/1315724232 conn(0x7fed480612d0 msgr2=0x7fed48062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:48.560 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 -- 172.21.15.3:0/1315724232 shutdown_connections 2022-07-21T02:43:48.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 -- 172.21.15.3:0/1315724232 wait complete. 2022-07-21T02:43:48.561 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 Processor -- start 2022-07-21T02:43:48.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 -- start start 2022-07-21T02:43:48.567 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 0x7fed48063f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed480b08a0 0x7fed4813ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b3260 0x7fed4813b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed48064f80 con 0x7fed480b08a0 2022-07-21T02:43:48.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4813b820 con 0x7fed480ad000 2022-07-21T02:43:48.570 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e95b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed480abc00 con 0x7fed480b3260 2022-07-21T02:43:48.571 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4d158700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed480b08a0 0x7fed4813ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4d959700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 0x7fed48063f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4d158700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed480b08a0 0x7fed4813ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52790/0 (socket says 172.21.15.3:52790) 2022-07-21T02:43:48.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4d158700 1 -- 172.21.15.3:0/1420985934 learned_addr learned my addr 172.21.15.3:0/1420985934 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:48.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.538+0000 7fed4e15a700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b3260 0x7fed4813b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4d959700 1 -- 172.21.15.3:0/1420985934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b3260 msgr2=0x7fed4813b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4d959700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b3260 0x7fed4813b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4d959700 1 -- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed480b08a0 msgr2=0x7fed4813ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4d959700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed480b08a0 0x7fed4813ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4d959700 1 -- 172.21.15.3:0/1420985934 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed40009470 con 0x7fed480ad000 2022-07-21T02:43:48.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4d158700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed480b08a0 0x7fed4813ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T02:43:48.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4d959700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 0x7fed48063f20 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fed3c00c000 tx=0x7fed3c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed3b7fe700 1 -- 172.21.15.3:0/1420985934 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed3c010780 con 0x7fed480ad000 2022-07-21T02:43:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed3b7fe700 1 -- 172.21.15.3:0/1420985934 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c0108e0 con 0x7fed480ad000 2022-07-21T02:43:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed480abea0 con 0x7fed480ad000 2022-07-21T02:43:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed3b7fe700 1 -- 172.21.15.3:0/1420985934 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed3c010c40 con 0x7fed480ad000 2022-07-21T02:43:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed480aae20 con 0x7fed480ad000 2022-07-21T02:43:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed480ac030 con 0x7fed480ad000 2022-07-21T02:43:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.542+0000 7fed3b7fe700 1 -- 172.21.15.3:0/1420985934 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fed3c01b400 con 0x7fed480ad000 2022-07-21T02:43:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.546+0000 7fed3b7fe700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed24053b00 0x7fed24055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.546+0000 7fed3b7fe700 1 -- 172.21.15.3:0/1420985934 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fed3c06bdb0 con 0x7fed480ad000 2022-07-21T02:43:48.583 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.546+0000 7fed4d158700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed24053b00 0x7fed24055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.546+0000 7fed4d158700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed24053b00 0x7fed24055fc0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fed4400d000 tx=0x7fed44009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.584 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.554+0000 7fed3b7fe700 1 -- 172.21.15.3:0/1420985934 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fed3c042df0 con 0x7fed480ad000 2022-07-21T02:43:48.683 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fed480aa800 con 0x7fed24053b00 2022-07-21T02:43:48.684 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed3b7fe700 1 -- 172.21.15.3:0/1420985934 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fed4400f730 con 0x7fed24053b00 2022-07-21T02:43:48.687 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed24053b00 msgr2=0x7fed24055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed24053b00 0x7fed24055fc0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fed4400d000 tx=0x7fed44009040).stop 2022-07-21T02:43:48.688 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 msgr2=0x7fed48063f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 0x7fed48063f20 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fed3c00c000 tx=0x7fed3c006df0).stop 2022-07-21T02:43:48.689 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 shutdown_connections 2022-07-21T02:43:48.690 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed480b3260 0x7fed4813b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed480b08a0 0x7fed4813ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fed24053b00 0x7fed24055fc0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.691 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 --2- 172.21.15.3:0/1420985934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480ad000 0x7fed48063f20 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 >> 172.21.15.3:0/1420985934 conn(0x7fed480612d0 msgr2=0x7fed48142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:48.692 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.682+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 shutdown_connections 2022-07-21T02:43:48.693 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.686+0000 7fed4e95b700 1 -- 172.21.15.3:0/1420985934 wait complete. 2022-07-21T02:43:48.693 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 7 to deep-scrub 2022-07-21T02:43:48.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-21T02:43:48.807 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326b59e700 1 Processor -- start 2022-07-21T02:43:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326b59e700 1 -- start start 2022-07-21T02:43:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f326c0614b0 0x7f326c061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326b59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f326c1405d0 0x7f326c140a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326b59e700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c0aee80 con 0x7f326c0ad750 2022-07-21T02:43:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326b59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c0b70b0 con 0x7f326c1405d0 2022-07-21T02:43:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c0af0b0 con 0x7f326c0614b0 2022-07-21T02:43:48.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326ad9d700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f326c1405d0 0x7f326c140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.811 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f3269d9b700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f326c0ad750 0x7f326c0b9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32846/0 (socket says 172.21.15.3:32846) 2022-07-21T02:43:48.812 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f3269d9b700 1 -- 172.21.15.3:0/1614428042 learned_addr learned my addr 172.21.15.3:0/1614428042 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:43:48.813 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326a59c700 1 --2- 172.21.15.3:0/1614428042 >> v2:172.21.15.143:3300/0 conn(0x7f326c0614b0 0x7f326c061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326a59c700 1 -- 172.21.15.3:0/1614428042 >> v2:172.21.15.3:3300/0 conn(0x7f326c1405d0 msgr2=0x7f326c140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:43:48.814 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326a59c700 1 --2- 172.21.15.3:0/1614428042 >> v2:172.21.15.3:3300/0 conn(0x7f326c1405d0 0x7f326c140a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.815 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326959a700 1 -- 172.21.15.3:0/1614428042 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897873205 0 0) 0x7f326c0aee80 con 0x7f326c0ad750 2022-07-21T02:43:48.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326a59c700 1 -- 172.21.15.3:0/1614428042 >> v1:172.21.15.3:6789/0 conn(0x7f326c0ad750 legacy=0x7f326c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.816 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326a59c700 1 -- 172.21.15.3:0/1614428042 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c0af350 con 0x7f326c0614b0 2022-07-21T02:43:48.817 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326ad9d700 1 --2- 172.21.15.3:0/1614428042 >> v2:172.21.15.3:3300/0 conn(0x7f326c1405d0 0x7f326c140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-21T02:43:48.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326a59c700 1 --2- 172.21.15.3:0/1614428042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f326c0614b0 0x7f326c061890 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f326c0aed30 tx=0x7f325c0091d0).ready entity=mon.1 client_cookie=63544c98d9946c4 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.818 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326959a700 1 -- 172.21.15.3:0/1614428042 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325c00b680 con 0x7f326c0614b0 2022-07-21T02:43:48.819 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326959a700 1 -- 172.21.15.3:0/1614428042 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c00b7e0 con 0x7f326c0614b0 2022-07-21T02:43:48.820 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326959a700 1 -- 172.21.15.3:0/1614428042 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325c00b950 con 0x7f326c0614b0 2022-07-21T02:43:48.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326b59e700 1 -- 172.21.15.3:0/1614428042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f326c0614b0 msgr2=0x7f326c061890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.821 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.802+0000 7f326b59e700 1 --2- 172.21.15.3:0/1614428042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f326c0614b0 0x7f326c061890 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f326c0aed30 tx=0x7f325c0091d0).stop 2022-07-21T02:43:48.822 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- 172.21.15.3:0/1614428042 shutdown_connections 2022-07-21T02:43:48.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 --2- 172.21.15.3:0/1614428042 >> v2:172.21.15.3:3300/0 conn(0x7f326c1405d0 0x7f326c140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.823 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 --2- 172.21.15.3:0/1614428042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f326c0614b0 0x7f326c061890 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.827 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- 172.21.15.3:0/1614428042 >> 172.21.15.3:0/1614428042 conn(0x7f326c066f70 msgr2=0x7f326c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:48.828 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- 172.21.15.3:0/1614428042 shutdown_connections 2022-07-21T02:43:48.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- 172.21.15.3:0/1614428042 wait complete. 2022-07-21T02:43:48.829 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 Processor -- start 2022-07-21T02:43:48.830 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- start start 2022-07-21T02:43:48.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f326c0614b0 0x7f326c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.831 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0ad750 0x7f326c0b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.832 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f326c1405d0 0x7f326c13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c09ea40 con 0x7f326c0ad750 2022-07-21T02:43:48.833 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c09ef10 con 0x7f326c0614b0 2022-07-21T02:43:48.834 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c0b3f40 con 0x7f326c1405d0 2022-07-21T02:43:48.839 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f3269d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0ad750 0x7f326c0b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f3269d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0ad750 0x7f326c0b09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52804/0 (socket says 172.21.15.3:52804) 2022-07-21T02:43:48.840 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f3269d9b700 1 -- 172.21.15.3:0/2700744423 learned_addr learned my addr 172.21.15.3:0/2700744423 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:43:48.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326a59c700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f326c0614b0 0x7f326c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.841 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326ad9d700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f326c1405d0 0x7f326c13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.842 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f3269d9b700 1 -- 172.21.15.3:0/2700744423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f326c1405d0 msgr2=0x7f326c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f3269d9b700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f326c1405d0 0x7f326c13f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.843 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f3269d9b700 1 -- 172.21.15.3:0/2700744423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f326c0614b0 msgr2=0x7f326c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f3269d9b700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f326c0614b0 0x7f326c155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.844 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f3269d9b700 1 -- 172.21.15.3:0/2700744423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326400b180 con 0x7f326c0ad750 2022-07-21T02:43:48.845 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f3269d9b700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0ad750 0x7f326c0b09b0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f3254000bc0 tx=0x7f3254007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f32637fe700 1 -- 172.21.15.3:0/2700744423 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32540070d0 con 0x7f326c0ad750 2022-07-21T02:43:48.846 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f32637fe700 1 -- 172.21.15.3:0/2700744423 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3254007230 con 0x7f326c0ad750 2022-07-21T02:43:48.847 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c009470 con 0x7f326c0ad750 2022-07-21T02:43:48.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f32637fe700 1 -- 172.21.15.3:0/2700744423 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3254005610 con 0x7f326c0ad750 2022-07-21T02:43:48.848 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.806+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c0b3bc0 con 0x7f326c0ad750 2022-07-21T02:43:48.849 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.810+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c0b1040 con 0x7f326c0ad750 2022-07-21T02:43:48.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.810+0000 7f32637fe700 1 -- 172.21.15.3:0/2700744423 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3254010070 con 0x7f326c0ad750 2022-07-21T02:43:48.850 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.810+0000 7f32637fe700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3244053a70 0x7f3244055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:43:48.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.810+0000 7f32637fe700 1 -- 172.21.15.3:0/2700744423 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f325406b390 con 0x7f326c0ad750 2022-07-21T02:43:48.851 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.810+0000 7f326a59c700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3244053a70 0x7f3244055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:43:48.852 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.810+0000 7f326a59c700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3244053a70 0x7f3244055f30 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f325c01b000 tx=0x7f325c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:43:48.853 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.814+0000 7f32637fe700 1 -- 172.21.15.3:0/2700744423 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3254042390 con 0x7f326c0ad750 2022-07-21T02:43:48.944 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.938+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f326c0af5f0 con 0x7f3244053a70 2022-07-21T02:43:48.948 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.946+0000 7f32637fe700 1 -- 172.21.15.3:0/2700744423 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68337 (secure 0 0 0) 0x7f326c0af5f0 con 0x7f3244053a70 2022-07-21T02:43:48.950 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T02:43:48.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.946+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3244053a70 msgr2=0x7f3244055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.951 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.946+0000 7f326b59e700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3244053a70 0x7f3244055f30 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f325c01b000 tx=0x7f325c016000).stop 2022-07-21T02:43:48.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.946+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0ad750 msgr2=0x7f326c0b09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:43:48.952 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.946+0000 7f326b59e700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0ad750 0x7f326c0b09b0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f3254000bc0 tx=0x7f3254007a00).stop 2022-07-21T02:43:48.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.950+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 shutdown_connections 2022-07-21T02:43:48.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.950+0000 7f326b59e700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f326c1405d0 0x7f326c13f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.953 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.950+0000 7f326b59e700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f326c0ad750 0x7f326c0b09b0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.950+0000 7f326b59e700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f3244053a70 0x7f3244055f30 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.954 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.950+0000 7f326b59e700 1 --2- 172.21.15.3:0/2700744423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f326c0614b0 0x7f326c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:43:48.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.950+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 >> 172.21.15.3:0/2700744423 conn(0x7f326c066f70 msgr2=0x7f326c05cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:43:48.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.950+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 shutdown_connections 2022-07-21T02:43:48.955 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:43:48.950+0000 7f326b59e700 1 -- 172.21.15.3:0/2700744423 wait complete. 2022-07-21T02:43:48.956 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2022-07-21T02:43:48.962 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":2275,"stamp":"2022-07-21T02:43:48.744064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8404928,"kb_used_data":16256,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746569792,"statfs":{"total":773094113280,"available":764487467008,"internally_reserved":8589934592,"allocated":16646144,"data_stored":8116488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003297"},"pg_stats":[{"pgid":"1.0","version":"16'2","reported_seq":147,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.423987+0000","last_change":"2022-07-21T01:33:15.640655+0000","last_active":"2022-07-21T01:43:11.423987+0000","last_peered":"2022-07-21T01:43:11.423987+0000","last_clean":"2022-07-21T01:43:11.423987+0000","last_became_active":"2022-07-21T01:33:15.639383+0000","last_became_peered":"2022-07-21T01:33:15.639383+0000","last_unstale":"2022-07-21T01:43:11.423987+0000","last_undegraded":"2022-07-21T01:43:11.423987+0000","last_fullsized":"2022-07-21T01:43:11.423987+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:10.553176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:10.553176+0000","last_clean_scrub_stamp":"2022-07-21T01:33:10.553176+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":150,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.335690+0000","last_change":"2022-07-21T01:33:19.894256+0000","last_active":"2022-07-21T01:43:11.335690+0000","last_peered":"2022-07-21T01:43:11.335690+0000","last_clean":"2022-07-21T01:43:11.335690+0000","last_became_active":"2022-07-21T01:33:19.893274+0000","last_became_peered":"2022-07-21T01:33:19.893274+0000","last_unstale":"2022-07-21T01:43:11.335690+0000","last_undegraded":"2022-07-21T01:43:11.335690+0000","last_fullsized":"2022-07-21T01:43:11.335690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.423951+0000","last_change":"2022-07-21T01:33:19.897103+0000","last_active":"2022-07-21T01:43:11.423951+0000","last_peered":"2022-07-21T01:43:11.423951+0000","last_clean":"2022-07-21T01:43:11.423951+0000","last_became_active":"2022-07-21T01:33:19.896369+0000","last_became_peered":"2022-07-21T01:33:19.896369+0000","last_unstale":"2022-07-21T01:43:11.423951+0000","last_undegraded":"2022-07-21T01:43:11.423951+0000","last_fullsized":"2022-07-21T01:43:11.423951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":140,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.337565+0000","last_change":"2022-07-21T01:33:19.896866+0000","last_active":"2022-07-21T01:43:11.337565+0000","last_peered":"2022-07-21T01:43:11.337565+0000","last_clean":"2022-07-21T01:43:11.337565+0000","last_became_active":"2022-07-21T01:33:19.896120+0000","last_became_peered":"2022-07-21T01:33:19.896120+0000","last_unstale":"2022-07-21T01:43:11.337565+0000","last_undegraded":"2022-07-21T01:43:11.337565+0000","last_fullsized":"2022-07-21T01:43:11.337565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":150,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.335687+0000","last_change":"2022-07-21T01:33:19.896530+0000","last_active":"2022-07-21T01:43:11.335687+0000","last_peered":"2022-07-21T01:43:11.335687+0000","last_clean":"2022-07-21T01:43:11.335687+0000","last_became_active":"2022-07-21T01:33:19.896110+0000","last_became_peered":"2022-07-21T01:33:19.896110+0000","last_unstale":"2022-07-21T01:43:11.335687+0000","last_undegraded":"2022-07-21T01:43:11.335687+0000","last_fullsized":"2022-07-21T01:43:11.335687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:47.358135+0000","last_change":"2022-07-21T02:43:47.358422+0000","last_active":"2022-07-21T02:43:47.358135+0000","last_peered":"2022-07-21T02:43:47.358135+0000","last_clean":"2022-07-21T02:43:47.358135+0000","last_became_active":"2022-07-21T01:33:20.649660+0000","last_became_peered":"2022-07-21T01:33:20.649660+0000","last_unstale":"2022-07-21T02:43:47.358135+0000","last_undegraded":"2022-07-21T02:43:47.358135+0000","last_fullsized":"2022-07-21T02:43:47.358135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T02:43:47.358134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T02:43:47.358134+0000","last_clean_scrub_stamp":"2022-07-21T02:43:47.358134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":141,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.424202+0000","last_change":"2022-07-21T01:33:20.648160+0000","last_active":"2022-07-21T01:43:11.424202+0000","last_peered":"2022-07-21T01:43:11.424202+0000","last_clean":"2022-07-21T01:43:11.424202+0000","last_became_active":"2022-07-21T01:33:20.647021+0000","last_became_peered":"2022-07-21T01:33:20.647021+0000","last_unstale":"2022-07-21T01:43:11.424202+0000","last_undegraded":"2022-07-21T01:43:11.424202+0000","last_fullsized":"2022-07-21T01:43:11.424202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":144,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.338584+0000","last_change":"2022-07-21T01:33:19.900435+0000","last_active":"2022-07-21T01:43:11.338584+0000","last_peered":"2022-07-21T01:43:11.338584+0000","last_clean":"2022-07-21T01:43:11.338584+0000","last_became_active":"2022-07-21T01:33:19.900054+0000","last_became_peered":"2022-07-21T01:33:19.900054+0000","last_unstale":"2022-07-21T01:43:11.338584+0000","last_undegraded":"2022-07-21T01:43:11.338584+0000","last_fullsized":"2022-07-21T01:43:11.338584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":141,"reported_epoch":163,"state":"active+clean","last_fresh":"2022-07-21T01:43:11.835880+0000","last_change":"2022-07-21T01:33:19.895590+0000","last_active":"2022-07-21T01:43:11.835880+0000","last_peered":"2022-07-21T01:43:11.835880+0000","last_clean":"2022-07-21T01:43:11.835880+0000","last_became_active":"2022-07-21T01:33:19.895046+0000","last_became_peered":"2022-07-21T01:33:19.895046+0000","last_unstale":"2022-07-21T01:43:11.835880+0000","last_undegraded":"2022-07-21T01:43:11.835880+0000","last_fullsized":"2022-07-21T01:43:11.835880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T01:33:18.866644+0000","last_clean_scrub_stamp":"2022-07-21T01:33:18.866644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424510288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.48599999999999999,"5min":0.42999999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":0.93600000000000005,"15min":1.2749999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68899999999999995,"15min":0.68799999999999994},"min":{"1min":0.495,"5min":0.495,"15min":0.47899999999999998},"max":{"1min":0.85999999999999999,"5min":1.0880000000000001,"15min":1.2410000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.67600000000000005,"15min":0.69599999999999995},"min":{"1min":0.53100000000000003,"5min":0.496,"15min":0.39000000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":1.1100000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71899999999999997,"15min":0.73899999999999999},"min":{"1min":0.60699999999999998,"5min":0.505,"15min":0.47999999999999998},"max":{"1min":0.85399999999999998,"5min":1.21,"15min":1.21},"last":0.68100000000000005}]},{"osd":3,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65800000000000003,"15min":0.67500000000000004},"min":{"1min":0.50600000000000001,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":0.89200000000000002,"5min":1.0820000000000001,"15min":1.1419999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.80100000000000005,"15min":0.82199999999999995},"min":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.55900000000000005},"max":{"1min":1.006,"5min":1.046,"15min":1.202},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.72299999999999998,"5min":0.68700000000000006,"15min":0.44800000000000001},"max":{"1min":1.0820000000000001,"5min":1.1950000000000001,"15min":1.3959999999999999},"last":0.753},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.94399999999999995,"15min":0.95199999999999996},"min":{"1min":0.79200000000000004,"5min":0.72199999999999998,"15min":0.56599999999999995},"max":{"1min":1.1990000000000001,"5min":1.2470000000000001,"15min":1.343},"last":0.998}]},{"osd":5,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.71299999999999997,"5min":0.66400000000000003,"15min":0.60699999999999998},"max":{"1min":1.2709999999999999,"5min":1.454,"15min":1.454},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.96199999999999997,"15min":0.96699999999999997},"min":{"1min":0.747,"5min":0.71199999999999997,"15min":0.52200000000000002},"max":{"1min":1.2010000000000001,"5min":1.365,"15min":1.365},"last":0.80100000000000005}]},{"osd":6,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.99299999999999999,"15min":0.998},"min":{"1min":0.82399999999999995,"5min":0.70799999999999996,"15min":0.56200000000000006},"max":{"1min":1.2110000000000001,"5min":1.446,"15min":1.522},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.048,"5min":1.0529999999999999,"15min":1.054},"min":{"1min":0.91500000000000004,"5min":0.83099999999999996,"15min":0.70499999999999996},"max":{"1min":1.21,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.93400000000000005}]},{"osd":7,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67700000000000005},"max":{"1min":1.3120000000000001,"5min":1.5349999999999999,"15min":1.5900000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0609999999999999,"15min":1.0629999999999999},"min":{"1min":0.90600000000000003,"5min":0.75600000000000001,"15min":0.61799999999999999},"max":{"1min":1.246,"5min":1.4059999999999999,"15min":1.552},"last":1.018}]}]},{"osd":0,"up_from":15,"seq":64424510290,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.498,"5min":0.46400000000000002,"15min":0.41999999999999998},"max":{"1min":0.79600000000000004,"5min":1.387,"15min":1.387},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76700000000000002,"15min":0.76200000000000001},"min":{"1min":0.66000000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0569999999999999,"5min":1.1970000000000001,"15min":1.3029999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.626,"15min":0.628},"min":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.40000000000000002},"max":{"1min":0.96599999999999997,"5min":1.052,"15min":1.1830000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.71499999999999997,"15min":0.71799999999999997},"min":{"1min":0.55000000000000004,"5min":0.52900000000000003,"15min":0.48899999999999999},"max":{"1min":0.97099999999999997,"5min":1.123,"15min":1.202},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.56299999999999994,"5min":0.502,"15min":0.47499999999999998},"max":{"1min":0.98599999999999999,"5min":1.651,"15min":1.651},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.83899999999999997,"15min":0.83499999999999996},"min":{"1min":0.70799999999999996,"5min":0.64100000000000001,"15min":0.49199999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1699999999999999},"last":0.66900000000000004}]},{"osd":4,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.69299999999999995,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.131,"5min":1.2,"15min":1.2},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91700000000000004,"15min":0.92700000000000005},"min":{"1min":0.78900000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1160000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.90800000000000003}]},{"osd":5,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.95599999999999996,"15min":0.96899999999999997},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.56299999999999994},"max":{"1min":1.2210000000000001,"5min":1.399,"15min":1.399},"last":1.026},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96599999999999997,"15min":0.98499999999999999},"min":{"1min":0.83599999999999997,"5min":0.69399999999999995,"15min":0.56899999999999995},"max":{"1min":1.151,"5min":1.698,"15min":1.698},"last":1.089}]},{"osd":6,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0089999999999999,"15min":1.008},"min":{"1min":0.745,"5min":0.74199999999999999,"15min":0.70699999999999996},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":1.042},{"interface":"front","average":{"1min":1.054,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.85699999999999998,"5min":0.748,"15min":0.748},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":1.127}]},{"osd":7,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.054,"15min":1.0629999999999999},"min":{"1min":0.91100000000000003,"5min":0.79300000000000004,"15min":0.65800000000000003},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.0820000000000001,"15min":1.0980000000000001},"min":{"1min":0.80500000000000005,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.077}]}]},{"osd":3,"up_from":15,"seq":64424510288,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.49099999999999999,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":0.91400000000000003,"5min":1.607,"15min":1.607},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69699999999999995,"15min":0.69399999999999995},"min":{"1min":0.55000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.81599999999999995,"5min":1.758,"15min":1.758},"last":0.69199999999999995}]},{"osd":1,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65300000000000002,"15min":0.65800000000000003},"min":{"1min":0.499,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88300000000000001,"5min":1.1890000000000001,"15min":1.2410000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.77300000000000002,"15min":0.77600000000000002},"min":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.51700000000000002},"max":{"1min":1.145,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.79100000000000004}]},{"osd":2,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.77500000000000002,"15min":0.76900000000000002},"min":{"1min":0.58699999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"max":{"1min":0.96199999999999997,"5min":1.236,"15min":1.2609999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.84999999999999998,"15min":0.84399999999999997},"min":{"1min":0.65800000000000003,"5min":0.59999999999999998,"15min":0.56799999999999995},"max":{"1min":1.024,"5min":1.131,"15min":1.153},"last":0.93100000000000005}]},{"osd":4,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.91400000000000003,"15min":0.89300000000000002},"min":{"1min":0.75800000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1479999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.96199999999999997,"15min":0.95099999999999996},"min":{"1min":0.73199999999999998,"5min":0.67700000000000005,"15min":0.66700000000000004},"max":{"1min":1.2070000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.872}]},{"osd":5,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.97899999999999998,"15min":0.97699999999999998},"min":{"1min":0.78800000000000003,"5min":0.70699999999999996,"15min":0.70099999999999996},"max":{"1min":1.2869999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.028,"15min":1.0269999999999999},"min":{"1min":0.79800000000000004,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.268,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.72599999999999998}]},{"osd":6,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0089999999999999,"15min":1.0009999999999999},"min":{"1min":0.84599999999999997,"5min":0.73799999999999999,"15min":0.71599999999999997},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.5189999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.077,"5min":1.0629999999999999,"15min":1.0549999999999999},"min":{"1min":0.89600000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.355,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.101,"15min":1.0960000000000001},"min":{"1min":0.87,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.3220000000000001,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.097,"15min":1.079},"min":{"1min":0.89200000000000002,"5min":0.85399999999999998,"15min":0.77700000000000002},"max":{"1min":1.399,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.0129999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.90400000000000003,"15min":0.88700000000000001},"min":{"1min":0.71099999999999997,"5min":0.64600000000000002,"15min":0.46500000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.74199999999999999,"5min":0.66400000000000003,"15min":0.63200000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.196},"last":0.63700000000000001}]},{"osd":1,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82199999999999995,"15min":0.82099999999999995},"min":{"1min":0.622,"5min":0.58699999999999997,"15min":0.56399999999999995},"max":{"1min":1.0509999999999999,"5min":1.1020000000000001,"15min":1.214},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83299999999999996,"15min":0.82899999999999996},"min":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56999999999999995},"max":{"1min":1.0700000000000001,"5min":1.085,"15min":1.1579999999999999},"last":0.496}]},{"osd":2,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.84199999999999997,"15min":0.83699999999999997},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.56399999999999995},"max":{"1min":1.107,"5min":1.117,"15min":1.1399999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85899999999999999,"15min":0.84899999999999998},"min":{"1min":0.65900000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.145,"5min":1.145,"15min":1.1659999999999999},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93300000000000005,"15min":0.91200000000000003},"min":{"1min":0.74399999999999999,"5min":0.64100000000000001,"15min":0.58399999999999996},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.94499999999999995,"15min":0.94599999999999995},"min":{"1min":0.67700000000000005,"5min":0.64900000000000002,"15min":0.58399999999999996},"max":{"1min":1.135,"5min":1.155,"15min":1.2270000000000001},"last":0.83199999999999996}]},{"osd":4,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78500000000000003,"15min":0.78400000000000003},"min":{"1min":0.627,"5min":0.54700000000000004,"15min":0.53300000000000003},"max":{"1min":1.0680000000000001,"5min":1.242,"15min":1.242},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.86299999999999999,"15min":0.86199999999999999},"min":{"1min":0.60499999999999998,"5min":0.58999999999999997,"15min":0.56200000000000006},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.284},"last":0.79900000000000004}]},{"osd":5,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.86599999999999999},"min":{"1min":0.67400000000000004,"5min":0.55800000000000005,"15min":0.55400000000000005},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.65600000000000003,"5min":0.59499999999999997,"15min":0.59199999999999997},"max":{"1min":1.149,"5min":1.194,"15min":1.2649999999999999},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.82299999999999995,"5min":0.625,"15min":0.54900000000000004},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":0.99399999999999999},"min":{"1min":0.74299999999999999,"5min":0.67500000000000004,"15min":0.66600000000000004},"max":{"1min":1.1870000000000001,"5min":1.379,"15min":1.379},"last":0.82199999999999995}]}]},{"osd":6,"up_from":14,"seq":60129542993,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.82799999999999996,"15min":0.81899999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2090000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.83599999999999997,"15min":0.83799999999999997},"min":{"1min":0.55900000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.05,"5min":1.119,"15min":1.212},"last":0.54500000000000004}]},{"osd":1,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82099999999999995,"15min":0.84199999999999997},"min":{"1min":0.61299999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.075,"5min":1.1910000000000001,"15min":1.2030000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.872,"15min":0.86499999999999999},"min":{"1min":0.65900000000000003,"5min":0.61799999999999999,"15min":0.60399999999999998},"max":{"1min":1.123,"5min":1.145,"15min":1.2609999999999999},"last":0.621}]},{"osd":2,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91900000000000004,"15min":0.90600000000000003},"min":{"1min":0.67500000000000004,"5min":0.625,"15min":0.61599999999999999},"max":{"1min":1.139,"5min":1.139,"15min":1.2509999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.80800000000000005,"15min":0.82099999999999995},"min":{"1min":0.65100000000000002,"5min":0.60099999999999998,"15min":0.58499999999999996},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1739999999999999},"last":1.1200000000000001}]},{"osd":3,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91100000000000003,"15min":0.91300000000000003},"min":{"1min":0.77500000000000002,"5min":0.71799999999999997,"15min":0.65000000000000002},"max":{"1min":1.099,"5min":1.1299999999999999,"15min":1.2010000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.71999999999999997,"5min":0.61899999999999999,"15min":0.61799999999999999},"max":{"1min":1.1499999999999999,"5min":1.165,"15min":1.3029999999999999},"last":1.0569999999999999}]},{"osd":4,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.60199999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.083,"5min":1.1679999999999999,"15min":1.169},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.85499999999999998,"15min":0.85899999999999999},"min":{"1min":0.62,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.202,"5min":1.202,"15min":1.2889999999999999},"last":0.84399999999999997}]},{"osd":5,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77800000000000002,"15min":0.80000000000000004},"min":{"1min":0.64500000000000002,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0149999999999999,"5min":1.097,"15min":1.2989999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.89600000000000002,"15min":0.92600000000000005},"min":{"1min":0.60499999999999998,"5min":0.57599999999999996,"15min":0.57099999999999995},"max":{"1min":1.24,"5min":1.24,"15min":1.2709999999999999},"last":1.1719999999999999}]},{"osd":7,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92000000000000004,"15min":0.94999999999999996},"min":{"1min":0.66700000000000004,"5min":0.622,"15min":0.57299999999999995},"max":{"1min":1.0640000000000001,"5min":1.214,"15min":1.5029999999999999},"last":1.083},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.96399999999999997,"15min":0.97699999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.59799999999999998},"max":{"1min":1.0780000000000001,"5min":1.22,"15min":1.482},"last":1.379}]}]},{"osd":5,"up_from":14,"seq":60129542993,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78300000000000003,"15min":0.77500000000000002},"min":{"1min":0.56499999999999995,"5min":0.503,"15min":0.49299999999999999},"max":{"1min":1.0069999999999999,"5min":1.093,"15min":1.2070000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.89000000000000001,"15min":0.85199999999999998},"min":{"1min":0.629,"5min":0.59899999999999998,"15min":0.51900000000000002},"max":{"1min":1.125,"5min":1.1719999999999999,"15min":1.226},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.871,"15min":0.877},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.55500000000000005},"max":{"1min":1.1890000000000001,"5min":1.198,"15min":1.282},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.83399999999999996,"15min":0.83199999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.98699999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.71599999999999997}]},{"osd":2,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.84899999999999998,"15min":0.85999999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":0.96099999999999997,"5min":1.204,"15min":1.23},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.98899999999999999,"5min":1.081,"15min":1.153},"last":0.98899999999999999}]},{"osd":3,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92100000000000004,"15min":0.89600000000000002},"min":{"1min":0.72599999999999998,"5min":0.625,"15min":0.624},"max":{"1min":1.1220000000000001,"5min":1.171,"15min":1.1719999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.90300000000000002,"15min":0.91000000000000003},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64100000000000001},"max":{"1min":1.018,"5min":1.2749999999999999,"15min":1.276},"last":0.90000000000000002}]},{"osd":4,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.82299999999999995,"15min":0.82499999999999996},"min":{"1min":0.59899999999999998,"5min":0.56200000000000006,"15min":0.45700000000000002},"max":{"1min":1.0569999999999999,"5min":1.1859999999999999,"15min":1.254},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.54600000000000004},"max":{"1min":1.0840000000000001,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":1.0409999999999999}]},{"osd":6,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.93400000000000005,"15min":0.91600000000000004},"min":{"1min":0.68200000000000005,"5min":0.61599999999999999,"15min":0.59899999999999998},"max":{"1min":1.0720000000000001,"5min":1.2130000000000001,"15min":1.248},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.92800000000000005,"15min":0.92300000000000004},"min":{"1min":0.70499999999999996,"5min":0.66400000000000003,"15min":0.63900000000000001},"max":{"1min":1.099,"5min":1.2090000000000001,"15min":1.3120000000000001},"last":0.96699999999999997}]},{"osd":7,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85099999999999998,"15min":0.84699999999999998},"min":{"1min":0.64400000000000002,"5min":0.56899999999999995,"15min":0.54900000000000004},"max":{"1min":1.2350000000000001,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.98799999999999999,"15min":0.99299999999999999},"min":{"1min":0.71899999999999997,"5min":0.70299999999999996,"15min":0.53300000000000003},"max":{"1min":1.198,"5min":1.242,"15min":1.3360000000000001},"last":1.0129999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.873,"15min":0.86099999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.46300000000000002},"max":{"1min":0.99199999999999999,"5min":1.117,"15min":1.234},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.98699999999999999,"15min":0.99099999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.63400000000000001},"max":{"1min":1.097,"5min":1.226,"15min":1.258},"last":1.038}]},{"osd":1,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.86299999999999999,"15min":0.85799999999999998},"min":{"1min":0.61599999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0700000000000001,"5min":1.163,"15min":1.2250000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.85299999999999998,"15min":0.83799999999999997},"min":{"1min":0.61199999999999999,"5min":0.61099999999999999,"15min":0.505},"max":{"1min":1.038,"5min":1.157,"15min":1.157},"last":0.65200000000000002}]},{"osd":2,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.90800000000000003,"15min":0.90000000000000002},"min":{"1min":0.74199999999999999,"5min":0.63800000000000001,"15min":0.60299999999999998},"max":{"1min":1.1319999999999999,"5min":1.1870000000000001,"15min":1.1910000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88100000000000001,"15min":0.85499999999999998},"min":{"1min":0.70399999999999996,"5min":0.61799999999999999,"15min":0.58999999999999997},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.1930000000000001},"last":0.88800000000000001}]},{"osd":3,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90000000000000002,"15min":0.89300000000000002},"min":{"1min":0.71399999999999997,"5min":0.63800000000000001,"15min":0.57099999999999995},"max":{"1min":1.0620000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.89000000000000001,"15min":0.89400000000000002},"min":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.65200000000000002},"max":{"1min":1.028,"5min":1.115,"15min":1.163},"last":0.753}]},{"osd":5,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.751,"15min":0.74399999999999999},"min":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.46500000000000002},"max":{"1min":0.80600000000000005,"5min":1.1160000000000001,"15min":1.347},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.91400000000000003,"15min":0.90100000000000002},"min":{"1min":0.622,"5min":0.59799999999999998,"15min":0.54500000000000004},"max":{"1min":1.1100000000000001,"5min":1.1899999999999999,"15min":1.2609999999999999},"last":0.96299999999999997}]},{"osd":6,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.98799999999999999,"15min":0.96299999999999997},"min":{"1min":0.71099999999999997,"5min":0.70499999999999996,"15min":0.47799999999999998},"max":{"1min":1.1830000000000001,"5min":1.2190000000000001,"15min":1.2569999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86699999999999999,"15min":0.85999999999999999},"min":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.57999999999999996},"max":{"1min":1.0620000000000001,"5min":1.1910000000000001,"15min":1.288},"last":0.85999999999999999}]},{"osd":7,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.66300000000000003,"5min":0.59399999999999997,"15min":0.58899999999999997},"max":{"1min":1.1040000000000001,"5min":1.1930000000000001,"15min":1.478},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.93400000000000005,"15min":0.93100000000000005},"min":{"1min":0.68899999999999995,"5min":0.61799999999999999,"15min":0.51100000000000001},"max":{"1min":1.1240000000000001,"5min":1.3660000000000001,"15min":1.4299999999999999},"last":0.77100000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65900000000000003,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.441},"max":{"1min":0.85599999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.747},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.621,"5min":0.57299999999999995,"15min":0.56299999999999994},"max":{"1min":0.89100000000000001,"5min":1.1080000000000001,"15min":1.1319999999999999},"last":0.61199999999999999}]},{"osd":2,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.48399999999999999},"max":{"1min":0.87,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.70599999999999996},"min":{"1min":0.53600000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94899999999999995,"5min":1.1359999999999999,"15min":1.2370000000000001},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.51100000000000001,"5min":0.48399999999999999,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":1.137,"15min":1.147},"last":0.498},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.89700000000000002,"15min":0.90000000000000002},"min":{"1min":0.72399999999999998,"5min":0.63100000000000001,"15min":0.54400000000000004},"max":{"1min":1.0820000000000001,"5min":1.2010000000000001,"15min":1.22},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.63500000000000001},"max":{"1min":1.0409999999999999,"5min":1.105,"15min":1.196},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.218,"5min":1.218,"15min":1.26},"last":0.71499999999999997}]},{"osd":5,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.94199999999999995,"15min":0.93000000000000005},"min":{"1min":0.72199999999999998,"5min":0.71099999999999997,"15min":0.64700000000000002},"max":{"1min":1.3959999999999999,"5min":1.399,"15min":1.399},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.006,"15min":1.0149999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1839999999999999,"5min":1.232,"15min":1.3520000000000001},"last":0.85399999999999998}]},{"osd":6,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.92700000000000005,"15min":0.94899999999999995},"min":{"1min":0.72999999999999998,"5min":0.69999999999999996,"15min":0.69599999999999995},"max":{"1min":1.2010000000000001,"5min":1.3400000000000001,"15min":1.3740000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.022,"5min":1.028,"15min":1.016},"min":{"1min":0.77200000000000002,"5min":0.73699999999999999,"15min":0.66500000000000004},"max":{"1min":1.304,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.91900000000000004}]},{"osd":7,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.071,"15min":1.0669999999999999},"min":{"1min":0.85899999999999999,"5min":0.79300000000000004,"15min":0.69999999999999996},"max":{"1min":1.2829999999999999,"5min":1.5149999999999999,"15min":1.7310000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.018,"5min":1.0369999999999999,"15min":1.048},"min":{"1min":0.83299999999999996,"5min":0.78800000000000003,"15min":0.73699999999999999},"max":{"1min":1.337,"5min":1.337,"15min":1.498},"last":0.81699999999999995}]}]}]}} 2022-07-21T02:43:48.972 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-07-21T01:33:10.553176+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=1, tm_min=33, tm_sec=10, tm_wday=3, tm_yday=202, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=2, tm_min=43, tm_sec=44, tm_wday=3, tm_yday=202, tm_isdst=0) 2022-07-21T02:43:48.973 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-07-21T01:33:18.866644+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=1, tm_min=33, tm_sec=18, tm_wday=3, tm_yday=202, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=2, tm_min=43, tm_sec=44, tm_wday=3, tm_yday=202, tm_isdst=0) 2022-07-21T02:43:48.974 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-07-21T01:33:18.866644+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=1, tm_min=33, tm_sec=18, tm_wday=3, tm_yday=202, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=2, tm_min=43, tm_sec=44, tm_wday=3, tm_yday=202, tm_isdst=0) 2022-07-21T02:43:48.975 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2022-07-21T01:33:18.866644+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=1, tm_min=33, tm_sec=18, tm_wday=3, tm_yday=202, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=2, tm_min=43, tm_sec=44, tm_wday=3, tm_yday=202, tm_isdst=0) 2022-07-21T02:43:48.976 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2022-07-21T01:33:18.866644+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=1, tm_min=33, tm_sec=18, tm_wday=3, tm_yday=202, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=2, tm_min=43, tm_sec=44, tm_wday=3, tm_yday=202, tm_isdst=0) 2022-07-21T02:43:48.977 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-07-21T01:33:18.866644+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=1, tm_min=33, tm_sec=18, tm_wday=3, tm_yday=202, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=2, tm_min=43, tm_sec=44, tm_wday=3, tm_yday=202, tm_isdst=0) 2022-07-21T02:43:48.978 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-07-21T01:33:18.866644+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=1, tm_min=33, tm_sec=18, tm_wday=3, tm_yday=202, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=2, tm_min=43, tm_sec=44, tm_wday=3, tm_yday=202, tm_isdst=0) 2022-07-21T02:43:48.979 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-07-21T01:33:18.866644+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=1, tm_min=33, tm_sec=18, tm_wday=3, tm_yday=202, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=21, tm_hour=2, tm_min=43, tm_sec=44, tm_wday=3, tm_yday=202, tm_isdst=0) 2022-07-21T02:43:48.979 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2022-07-21T02:44:08.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-21T02:44:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a58baf700 1 Processor -- start 2022-07-21T02:44:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a58baf700 1 -- start start 2022-07-21T02:44:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a58baf700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5a54067180 0x7f5a54143a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a58baf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5a540ad1c0 0x7f5a54143f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.099 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a58baf700 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a54067560 con 0x7f5a540b0870 2022-07-21T02:44:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a58baf700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a540b0160 con 0x7f5a54067180 2022-07-21T02:44:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a58baf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a540b0400 con 0x7f5a540ad1c0 2022-07-21T02:44:09.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a5259c700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7f5a54067180 0x7f5a54143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a52d9d700 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f5a540b0870 0x7f5a540b1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:32860/0 (socket says 172.21.15.3:32860) 2022-07-21T02:44:09.101 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a52d9d700 1 -- 172.21.15.3:0/205242333 learned_addr learned my addr 172.21.15.3:0/205242333 (peer_addr_for_me v1:172.21.15.3:0/0) 2022-07-21T02:44:09.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a51d9b700 1 --2- 172.21.15.3:0/205242333 >> v2:172.21.15.143:3300/0 conn(0x7f5a540ad1c0 0x7f5a54143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a5259c700 1 -- 172.21.15.3:0/205242333 >> v2:172.21.15.143:3300/0 conn(0x7f5a540ad1c0 msgr2=0x7f5a54143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:44:09.102 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a5259c700 1 --2- 172.21.15.3:0/205242333 >> v2:172.21.15.143:3300/0 conn(0x7f5a540ad1c0 0x7f5a54143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a5259c700 1 -- 172.21.15.3:0/205242333 >> v1:172.21.15.3:6789/0 conn(0x7f5a540b0870 legacy=0x7f5a540b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a5259c700 1 -- 172.21.15.3:0/205242333 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a540af530 con 0x7f5a54067180 2022-07-21T02:44:09.103 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a5259c700 1 --2- 172.21.15.3:0/205242333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a54067180 0x7f5a54143a60 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7f5a3c0032f0 tx=0x7f5a3c0091d0).ready entity=mon.0 client_cookie=4b7c2f455e178e9e server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:44:09.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a5159a700 1 -- 172.21.15.3:0/205242333 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a3c00b680 con 0x7f5a54067180 2022-07-21T02:44:09.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a5159a700 1 -- 172.21.15.3:0/205242333 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c00b7e0 con 0x7f5a54067180 2022-07-21T02:44:09.104 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.094+0000 7f5a5159a700 1 -- 172.21.15.3:0/205242333 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a3c00b950 con 0x7f5a54067180 2022-07-21T02:44:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 -- 172.21.15.3:0/205242333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a54067180 msgr2=0x7f5a54143a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.107 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 --2- 172.21.15.3:0/205242333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a54067180 0x7f5a54143a60 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7f5a3c0032f0 tx=0x7f5a3c0091d0).stop 2022-07-21T02:44:09.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 -- 172.21.15.3:0/205242333 shutdown_connections 2022-07-21T02:44:09.108 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 --2- 172.21.15.3:0/205242333 >> v2:172.21.15.143:3300/0 conn(0x7f5a540ad1c0 0x7f5a54143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 --2- 172.21.15.3:0/205242333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a54067180 0x7f5a54143a60 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.109 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 -- 172.21.15.3:0/205242333 >> 172.21.15.3:0/205242333 conn(0x7f5a5405f840 msgr2=0x7f5a5405fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:44:09.110 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 -- 172.21.15.3:0/205242333 shutdown_connections 2022-07-21T02:44:09.112 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 -- 172.21.15.3:0/205242333 wait complete. 2022-07-21T02:44:09.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 Processor -- start 2022-07-21T02:44:09.116 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.098+0000 7f5a58baf700 1 -- start start 2022-07-21T02:44:09.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a58baf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a54067180 0x7f5a540a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a58baf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a540b0870 0x7f5a540a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.117 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a58baf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a540a6630 0x7f5a540a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a58baf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a54153830 con 0x7f5a540b0870 2022-07-21T02:44:09.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a58baf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540a5ef0 con 0x7f5a540a6630 2022-07-21T02:44:09.118 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a58baf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540a6190 con 0x7f5a54067180 2022-07-21T02:44:09.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a51d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a540b0870 0x7f5a540a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.119 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a51d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a540b0870 0x7f5a540a3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52818/0 (socket says 172.21.15.3:52818) 2022-07-21T02:44:09.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a51d9b700 1 -- 172.21.15.3:0/808524792 learned_addr learned my addr 172.21.15.3:0/808524792 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:44:09.120 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a52d9d700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a540a6630 0x7f5a540a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a5259c700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a54067180 0x7f5a540a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a51d9b700 1 -- 172.21.15.3:0/808524792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a54067180 msgr2=0x7f5a540a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.121 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a51d9b700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a54067180 0x7f5a540a2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a51d9b700 1 -- 172.21.15.3:0/808524792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a540a6630 msgr2=0x7f5a540a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a51d9b700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a540a6630 0x7f5a540a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.122 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a51d9b700 1 -- 172.21.15.3:0/808524792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c009470 con 0x7f5a540b0870 2022-07-21T02:44:09.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a51d9b700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a540b0870 0x7f5a540a3960 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7f5a4400c000 tx=0x7f5a44006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:44:09.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a4b7fe700 1 -- 172.21.15.3:0/808524792 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a44010780 con 0x7f5a540b0870 2022-07-21T02:44:09.123 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a4b7fe700 1 -- 172.21.15.3:0/808524792 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4400b3b0 con 0x7f5a540b0870 2022-07-21T02:44:09.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a541534b0 con 0x7f5a540b0870 2022-07-21T02:44:09.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a4b7fe700 1 -- 172.21.15.3:0/808524792 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a44010ad0 con 0x7f5a540b0870 2022-07-21T02:44:09.124 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.102+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a540a3730 con 0x7f5a540b0870 2022-07-21T02:44:09.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.106+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a540a2770 con 0x7f5a540b0870 2022-07-21T02:44:09.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.106+0000 7f5a4b7fe700 1 -- 172.21.15.3:0/808524792 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5a44010c30 con 0x7f5a540b0870 2022-07-21T02:44:09.125 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.106+0000 7f5a4b7fe700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5a2c053a70 0x7f5a2c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.106+0000 7f5a4b7fe700 1 -- 172.21.15.3:0/808524792 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7f5a4406c170 con 0x7f5a540b0870 2022-07-21T02:44:09.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.106+0000 7f5a5259c700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5a2c053a70 0x7f5a2c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.126 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.106+0000 7f5a5259c700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5a2c053a70 0x7f5a2c055f30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f5a3c01b000 tx=0x7f5a3c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:44:09.127 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.110+0000 7f5a4b7fe700 1 -- 172.21.15.3:0/808524792 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5a44043110 con 0x7f5a540b0870 2022-07-21T02:44:09.244 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.238+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 --> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a540a5d70 con 0x7f5a2c053a70 2022-07-21T02:44:09.245 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a4b7fe700 1 -- 172.21.15.3:0/808524792 <== mgr.4103 v2:172.21.15.3:6800/22700 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68326 (secure 0 0 0) 0x7f5a3c006cc0 con 0x7f5a2c053a70 2022-07-21T02:44:09.247 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-07-21T02:44:09.251 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5a2c053a70 msgr2=0x7f5a2c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.252 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5a2c053a70 0x7f5a2c055f30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f5a3c01b000 tx=0x7f5a3c016000).stop 2022-07-21T02:44:09.253 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a540b0870 msgr2=0x7f5a540a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a540b0870 0x7f5a540a3960 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7f5a4400c000 tx=0x7f5a44006df0).stop 2022-07-21T02:44:09.254 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 shutdown_connections 2022-07-21T02:44:09.255 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7f5a2c053a70 0x7f5a2c055f30 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a540a6630 0x7f5a540a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.256 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a540b0870 0x7f5a540a3960 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 --2- 172.21.15.3:0/808524792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a54067180 0x7f5a540a2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.257 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 >> 172.21.15.3:0/808524792 conn(0x7f5a5405f840 msgr2=0x7f5a540b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:44:09.258 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 shutdown_connections 2022-07-21T02:44:09.259 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.242+0000 7f5a58baf700 1 -- 172.21.15.3:0/808524792 wait complete. 2022-07-21T02:44:09.259 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2022-07-21T02:44:09.262 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":2285,"stamp":"2022-07-21T02:44:08.749821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9695,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":8404928,"kb_used_data":16256,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746569792,"statfs":{"total":773094113280,"available":764487467008,"internally_reserved":8589934592,"allocated":16646144,"data_stored":8116488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003348"},"pg_stats":[{"pgid":"1.0","version":"164'4","reported_seq":152,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:50.320230+0000","last_change":"2022-07-21T02:43:50.320436+0000","last_active":"2022-07-21T02:43:50.320230+0000","last_peered":"2022-07-21T02:43:50.320230+0000","last_clean":"2022-07-21T02:43:50.320230+0000","last_became_active":"2022-07-21T01:33:15.639383+0000","last_became_peered":"2022-07-21T01:33:15.639383+0000","last_unstale":"2022-07-21T02:43:50.320230+0000","last_undegraded":"2022-07-21T02:43:50.320230+0000","last_fullsized":"2022-07-21T02:43:50.320230+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'4","last_scrub_stamp":"2022-07-21T02:43:50.320230+0000","last_deep_scrub":"164'4","last_deep_scrub_stamp":"2022-07-21T02:43:50.320230+0000","last_clean_scrub_stamp":"2022-07-21T02:43:50.320230+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9695,"num_omap_keys":2,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":153,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:56.742342+0000","last_change":"2022-07-21T02:43:56.742561+0000","last_active":"2022-07-21T02:43:56.742342+0000","last_peered":"2022-07-21T02:43:56.742342+0000","last_clean":"2022-07-21T02:43:56.742342+0000","last_became_active":"2022-07-21T01:33:19.893274+0000","last_became_peered":"2022-07-21T01:33:19.893274+0000","last_unstale":"2022-07-21T02:43:56.742342+0000","last_undegraded":"2022-07-21T02:43:56.742342+0000","last_fullsized":"2022-07-21T02:43:56.742342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T02:43:56.742341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T02:43:56.742341+0000","last_clean_scrub_stamp":"2022-07-21T02:43:56.742341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":144,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:53.275700+0000","last_change":"2022-07-21T02:43:53.275893+0000","last_active":"2022-07-21T02:43:53.275700+0000","last_peered":"2022-07-21T02:43:53.275700+0000","last_clean":"2022-07-21T02:43:53.275700+0000","last_became_active":"2022-07-21T01:33:19.896369+0000","last_became_peered":"2022-07-21T01:33:19.896369+0000","last_unstale":"2022-07-21T02:43:53.275700+0000","last_undegraded":"2022-07-21T02:43:53.275700+0000","last_fullsized":"2022-07-21T02:43:53.275700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T02:43:53.275699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T02:43:53.275699+0000","last_clean_scrub_stamp":"2022-07-21T02:43:53.275699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:46.682514+0000","last_change":"2022-07-21T02:43:46.682708+0000","last_active":"2022-07-21T02:43:46.682514+0000","last_peered":"2022-07-21T02:43:46.682514+0000","last_clean":"2022-07-21T02:43:46.682514+0000","last_became_active":"2022-07-21T01:33:19.896120+0000","last_became_peered":"2022-07-21T01:33:19.896120+0000","last_unstale":"2022-07-21T02:43:46.682514+0000","last_undegraded":"2022-07-21T02:43:46.682514+0000","last_fullsized":"2022-07-21T02:43:46.682514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T02:43:46.682513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T02:43:46.682513+0000","last_clean_scrub_stamp":"2022-07-21T02:43:46.682513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":153,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:54.698528+0000","last_change":"2022-07-21T02:43:54.698613+0000","last_active":"2022-07-21T02:43:54.698528+0000","last_peered":"2022-07-21T02:43:54.698528+0000","last_clean":"2022-07-21T02:43:54.698528+0000","last_became_active":"2022-07-21T01:33:19.896110+0000","last_became_peered":"2022-07-21T01:33:19.896110+0000","last_unstale":"2022-07-21T02:43:54.698528+0000","last_undegraded":"2022-07-21T02:43:54.698528+0000","last_fullsized":"2022-07-21T02:43:54.698528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T02:43:54.698528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T02:43:54.698528+0000","last_clean_scrub_stamp":"2022-07-21T02:43:54.698528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:47.358135+0000","last_change":"2022-07-21T02:43:47.358422+0000","last_active":"2022-07-21T02:43:47.358135+0000","last_peered":"2022-07-21T02:43:47.358135+0000","last_clean":"2022-07-21T02:43:47.358135+0000","last_became_active":"2022-07-21T01:33:20.649660+0000","last_became_peered":"2022-07-21T01:33:20.649660+0000","last_unstale":"2022-07-21T02:43:47.358135+0000","last_undegraded":"2022-07-21T02:43:47.358135+0000","last_fullsized":"2022-07-21T02:43:47.358135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T02:43:47.358134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T02:43:47.358134+0000","last_clean_scrub_stamp":"2022-07-21T02:43:47.358134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":144,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:55.222042+0000","last_change":"2022-07-21T02:43:55.222273+0000","last_active":"2022-07-21T02:43:55.222042+0000","last_peered":"2022-07-21T02:43:55.222042+0000","last_clean":"2022-07-21T02:43:55.222042+0000","last_became_active":"2022-07-21T01:33:20.647021+0000","last_became_peered":"2022-07-21T01:33:20.647021+0000","last_unstale":"2022-07-21T02:43:55.222042+0000","last_undegraded":"2022-07-21T02:43:55.222042+0000","last_fullsized":"2022-07-21T02:43:55.222042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T02:43:55.222041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T02:43:55.222041+0000","last_clean_scrub_stamp":"2022-07-21T02:43:55.222041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:49.383252+0000","last_change":"2022-07-21T02:43:49.383374+0000","last_active":"2022-07-21T02:43:49.383252+0000","last_peered":"2022-07-21T02:43:49.383252+0000","last_clean":"2022-07-21T02:43:49.383252+0000","last_became_active":"2022-07-21T01:33:19.900054+0000","last_became_peered":"2022-07-21T01:33:19.900054+0000","last_unstale":"2022-07-21T02:43:49.383252+0000","last_undegraded":"2022-07-21T02:43:49.383252+0000","last_fullsized":"2022-07-21T02:43:49.383252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T02:43:49.383252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T02:43:49.383252+0000","last_clean_scrub_stamp":"2022-07-21T02:43:49.383252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":144,"reported_epoch":164,"state":"active+clean","last_fresh":"2022-07-21T02:43:48.825348+0000","last_change":"2022-07-21T02:43:48.825433+0000","last_active":"2022-07-21T02:43:48.825348+0000","last_peered":"2022-07-21T02:43:48.825348+0000","last_clean":"2022-07-21T02:43:48.825348+0000","last_became_active":"2022-07-21T01:33:19.895046+0000","last_became_peered":"2022-07-21T01:33:19.895046+0000","last_unstale":"2022-07-21T02:43:48.825348+0000","last_undegraded":"2022-07-21T02:43:48.825348+0000","last_fullsized":"2022-07-21T02:43:48.825348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-21T02:43:48.825348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-21T02:43:48.825348+0000","last_clean_scrub_stamp":"2022-07-21T02:43:48.825348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9695,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424510292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.48599999999999999,"5min":0.42999999999999999,"15min":0.42199999999999999},"max":{"1min":0.92400000000000004,"5min":0.93600000000000005,"15min":1.2749999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68899999999999995,"15min":0.68799999999999994},"min":{"1min":0.495,"5min":0.495,"15min":0.47899999999999998},"max":{"1min":0.85999999999999999,"5min":1.0880000000000001,"15min":1.2410000000000001},"last":0.58599999999999997}]},{"osd":1,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.67600000000000005,"15min":0.69599999999999995},"min":{"1min":0.53100000000000003,"5min":0.496,"15min":0.39000000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":1.1100000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71899999999999997,"15min":0.73899999999999999},"min":{"1min":0.60699999999999998,"5min":0.505,"15min":0.47999999999999998},"max":{"1min":0.85399999999999998,"5min":1.21,"15min":1.21},"last":0.69599999999999995}]},{"osd":3,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.65800000000000003,"15min":0.67500000000000004},"min":{"1min":0.50600000000000001,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":0.89200000000000002,"5min":1.0820000000000001,"15min":1.1419999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.80100000000000005,"15min":0.82199999999999995},"min":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.55900000000000005},"max":{"1min":1.006,"5min":1.046,"15min":1.202},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.72299999999999998,"5min":0.68700000000000006,"15min":0.44800000000000001},"max":{"1min":1.0820000000000001,"5min":1.1950000000000001,"15min":1.3959999999999999},"last":0.995},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.94399999999999995,"15min":0.95199999999999996},"min":{"1min":0.79200000000000004,"5min":0.72199999999999998,"15min":0.56599999999999995},"max":{"1min":1.1990000000000001,"5min":1.2470000000000001,"15min":1.343},"last":1.2370000000000001}]},{"osd":5,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.71299999999999997,"5min":0.66400000000000003,"15min":0.60699999999999998},"max":{"1min":1.2709999999999999,"5min":1.454,"15min":1.454},"last":1.1970000000000001},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.96199999999999997,"15min":0.96699999999999997},"min":{"1min":0.747,"5min":0.71199999999999997,"15min":0.52200000000000002},"max":{"1min":1.2010000000000001,"5min":1.365,"15min":1.365},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.99299999999999999,"15min":0.998},"min":{"1min":0.82399999999999995,"5min":0.70799999999999996,"15min":0.56200000000000006},"max":{"1min":1.2110000000000001,"5min":1.446,"15min":1.522},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.048,"5min":1.0529999999999999,"15min":1.054},"min":{"1min":0.91500000000000004,"5min":0.83099999999999996,"15min":0.70499999999999996},"max":{"1min":1.21,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":1.0600000000000001}]},{"osd":7,"last update":"Thu Jul 21 02:43:41 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.67700000000000005},"max":{"1min":1.3120000000000001,"5min":1.5349999999999999,"15min":1.5900000000000001},"last":1.119},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0609999999999999,"15min":1.0629999999999999},"min":{"1min":0.90600000000000003,"5min":0.75600000000000001,"15min":0.61799999999999999},"max":{"1min":1.246,"5min":1.4059999999999999,"15min":1.552},"last":0.98199999999999998}]}]},{"osd":0,"up_from":15,"seq":64424510294,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.498,"5min":0.46400000000000002,"15min":0.41999999999999998},"max":{"1min":0.79600000000000004,"5min":1.387,"15min":1.387},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76700000000000002,"15min":0.76200000000000001},"min":{"1min":0.66000000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0569999999999999,"5min":1.1970000000000001,"15min":1.3029999999999999},"last":0.87}]},{"osd":2,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.626,"15min":0.628},"min":{"1min":0.49099999999999999,"5min":0.48799999999999999,"15min":0.40000000000000002},"max":{"1min":0.96599999999999997,"5min":1.052,"15min":1.1830000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.71499999999999997,"15min":0.71799999999999997},"min":{"1min":0.55000000000000004,"5min":0.52900000000000003,"15min":0.48899999999999999},"max":{"1min":0.97099999999999997,"5min":1.123,"15min":1.202},"last":0.65500000000000003}]},{"osd":3,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.56299999999999994,"5min":0.502,"15min":0.47499999999999998},"max":{"1min":0.98599999999999999,"5min":1.651,"15min":1.651},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.83899999999999997,"15min":0.83499999999999996},"min":{"1min":0.70799999999999996,"5min":0.64100000000000001,"15min":0.49199999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1699999999999999},"last":0.78700000000000003}]},{"osd":4,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.69299999999999995,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.131,"5min":1.2,"15min":1.2},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91700000000000004,"15min":0.92700000000000005},"min":{"1min":0.78900000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1160000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":1.3}]},{"osd":5,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.95599999999999996,"15min":0.96899999999999997},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.56299999999999994},"max":{"1min":1.2210000000000001,"5min":1.399,"15min":1.399},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96599999999999997,"15min":0.98499999999999999},"min":{"1min":0.83599999999999997,"5min":0.69399999999999995,"15min":0.56899999999999995},"max":{"1min":1.151,"5min":1.698,"15min":1.698},"last":1.2490000000000001}]},{"osd":6,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0089999999999999,"15min":1.008},"min":{"1min":0.745,"5min":0.74199999999999999,"15min":0.70699999999999996},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.054,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.85699999999999998,"5min":0.748,"15min":0.748},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jul 21 02:43:26 2022","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.054,"15min":1.0629999999999999},"min":{"1min":0.91100000000000003,"5min":0.79300000000000004,"15min":0.65800000000000003},"max":{"1min":1.5860000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.0820000000000001,"15min":1.0980000000000001},"min":{"1min":0.80500000000000005,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.137}]}]},{"osd":3,"up_from":15,"seq":64424510292,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.63900000000000001,"15min":0.628},"min":{"1min":0.49099999999999999,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":0.91400000000000003,"5min":1.607,"15min":1.607},"last":0.88},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69699999999999995,"15min":0.69399999999999995},"min":{"1min":0.55000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.81599999999999995,"5min":1.758,"15min":1.758},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65300000000000002,"15min":0.65800000000000003},"min":{"1min":0.499,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88300000000000001,"5min":1.1890000000000001,"15min":1.2410000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.77300000000000002,"15min":0.77600000000000002},"min":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.51700000000000002},"max":{"1min":1.145,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.97099999999999997}]},{"osd":2,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.77500000000000002,"15min":0.76900000000000002},"min":{"1min":0.58699999999999997,"5min":0.54100000000000004,"15min":0.53300000000000003},"max":{"1min":0.96199999999999997,"5min":1.236,"15min":1.2609999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.84999999999999998,"15min":0.84399999999999997},"min":{"1min":0.65800000000000003,"5min":0.59999999999999998,"15min":0.56799999999999995},"max":{"1min":1.024,"5min":1.131,"15min":1.153},"last":1.2490000000000001}]},{"osd":4,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.91400000000000003,"15min":0.89300000000000002},"min":{"1min":0.75800000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1479999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.96199999999999997,"15min":0.95099999999999996},"min":{"1min":0.73199999999999998,"5min":0.67700000000000005,"15min":0.66700000000000004},"max":{"1min":1.2070000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.79600000000000004}]},{"osd":5,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.97899999999999998,"15min":0.97699999999999998},"min":{"1min":0.78800000000000003,"5min":0.70699999999999996,"15min":0.70099999999999996},"max":{"1min":1.2869999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.028,"15min":1.0269999999999999},"min":{"1min":0.79800000000000004,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.268,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.486}]},{"osd":6,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0089999999999999,"15min":1.0009999999999999},"min":{"1min":0.84599999999999997,"5min":0.73799999999999999,"15min":0.71599999999999997},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.5189999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.077,"5min":1.0629999999999999,"15min":1.0549999999999999},"min":{"1min":0.89600000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.355,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.1080000000000001}]},{"osd":7,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.101,"15min":1.0960000000000001},"min":{"1min":0.87,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.3220000000000001,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.097,"15min":1.079},"min":{"1min":0.89200000000000002,"5min":0.85399999999999998,"15min":0.77700000000000002},"max":{"1min":1.399,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.99399999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542998,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.90400000000000003,"15min":0.88700000000000001},"min":{"1min":0.71099999999999997,"5min":0.64600000000000002,"15min":0.46500000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.74199999999999999,"5min":0.66400000000000003,"15min":0.63200000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.196},"last":0.999}]},{"osd":1,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82199999999999995,"15min":0.82099999999999995},"min":{"1min":0.622,"5min":0.58699999999999997,"15min":0.56399999999999995},"max":{"1min":1.0509999999999999,"5min":1.1020000000000001,"15min":1.214},"last":1.069},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83299999999999996,"15min":0.82899999999999996},"min":{"1min":0.60199999999999998,"5min":0.57299999999999995,"15min":0.56999999999999995},"max":{"1min":1.0700000000000001,"5min":1.085,"15min":1.1579999999999999},"last":0.82499999999999996}]},{"osd":2,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.84199999999999997,"15min":0.83699999999999997},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.56399999999999995},"max":{"1min":1.107,"5min":1.117,"15min":1.1399999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85899999999999999,"15min":0.84899999999999998},"min":{"1min":0.65900000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.145,"5min":1.145,"15min":1.1659999999999999},"last":0.84899999999999998}]},{"osd":3,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93300000000000005,"15min":0.91200000000000003},"min":{"1min":0.74399999999999999,"5min":0.64100000000000001,"15min":0.58399999999999996},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.94499999999999995,"15min":0.94599999999999995},"min":{"1min":0.67700000000000005,"5min":0.64900000000000002,"15min":0.58399999999999996},"max":{"1min":1.135,"5min":1.155,"15min":1.2270000000000001},"last":1.042}]},{"osd":4,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78500000000000003,"15min":0.78400000000000003},"min":{"1min":0.627,"5min":0.54700000000000004,"15min":0.53300000000000003},"max":{"1min":1.0680000000000001,"5min":1.242,"15min":1.242},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.86299999999999999,"15min":0.86199999999999999},"min":{"1min":0.60499999999999998,"5min":0.58999999999999997,"15min":0.56200000000000006},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.284},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.86599999999999999},"min":{"1min":0.67400000000000004,"5min":0.55800000000000005,"15min":0.55400000000000005},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.65600000000000003,"5min":0.59499999999999997,"15min":0.59199999999999997},"max":{"1min":1.149,"5min":1.194,"15min":1.2649999999999999},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jul 21 02:43:11 2022","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.82299999999999995,"5min":0.625,"15min":0.54900000000000004},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":0.99399999999999999},"min":{"1min":0.74299999999999999,"5min":0.67500000000000004,"15min":0.66600000000000004},"max":{"1min":1.1870000000000001,"5min":1.379,"15min":1.379},"last":0.71199999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542997,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.82799999999999996,"15min":0.81899999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2090000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.83599999999999997,"15min":0.83799999999999997},"min":{"1min":0.55900000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.05,"5min":1.119,"15min":1.212},"last":0.79700000000000004}]},{"osd":1,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82099999999999995,"15min":0.84199999999999997},"min":{"1min":0.61299999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.075,"5min":1.1910000000000001,"15min":1.2030000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.872,"15min":0.86499999999999999},"min":{"1min":0.65900000000000003,"5min":0.61799999999999999,"15min":0.60399999999999998},"max":{"1min":1.123,"5min":1.145,"15min":1.2609999999999999},"last":1.0640000000000001}]},{"osd":2,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91900000000000004,"15min":0.90600000000000003},"min":{"1min":0.67500000000000004,"5min":0.625,"15min":0.61599999999999999},"max":{"1min":1.139,"5min":1.139,"15min":1.2509999999999999},"last":0.875},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.80800000000000005,"15min":0.82099999999999995},"min":{"1min":0.65100000000000002,"5min":0.60099999999999998,"15min":0.58499999999999996},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1739999999999999},"last":0.84199999999999997}]},{"osd":3,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91100000000000003,"15min":0.91300000000000003},"min":{"1min":0.77500000000000002,"5min":0.71799999999999997,"15min":0.65000000000000002},"max":{"1min":1.099,"5min":1.1299999999999999,"15min":1.2010000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.71999999999999997,"5min":0.61899999999999999,"15min":0.61799999999999999},"max":{"1min":1.1499999999999999,"5min":1.165,"15min":1.3029999999999999},"last":0.66100000000000003}]},{"osd":4,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.60199999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.083,"5min":1.1679999999999999,"15min":1.169},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.85499999999999998,"15min":0.85899999999999999},"min":{"1min":0.62,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.202,"5min":1.202,"15min":1.2889999999999999},"last":0.98399999999999999}]},{"osd":5,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77800000000000002,"15min":0.80000000000000004},"min":{"1min":0.64500000000000002,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0149999999999999,"5min":1.097,"15min":1.2989999999999999},"last":0.748},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.89600000000000002,"15min":0.92600000000000005},"min":{"1min":0.60499999999999998,"5min":0.57599999999999996,"15min":0.57099999999999995},"max":{"1min":1.24,"5min":1.24,"15min":1.2709999999999999},"last":1.008}]},{"osd":7,"last update":"Thu Jul 21 02:43:27 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92000000000000004,"15min":0.94999999999999996},"min":{"1min":0.66700000000000004,"5min":0.622,"15min":0.57299999999999995},"max":{"1min":1.0640000000000001,"5min":1.214,"15min":1.5029999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.96399999999999997,"15min":0.97699999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.59799999999999998},"max":{"1min":1.0780000000000001,"5min":1.22,"15min":1.482},"last":1.0429999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542997,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78300000000000003,"15min":0.77500000000000002},"min":{"1min":0.56499999999999995,"5min":0.503,"15min":0.49299999999999999},"max":{"1min":1.0069999999999999,"5min":1.093,"15min":1.2070000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.89000000000000001,"15min":0.85199999999999998},"min":{"1min":0.629,"5min":0.59899999999999998,"15min":0.51900000000000002},"max":{"1min":1.125,"5min":1.1719999999999999,"15min":1.226},"last":0.98799999999999999}]},{"osd":1,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.871,"15min":0.877},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.55500000000000005},"max":{"1min":1.1890000000000001,"5min":1.198,"15min":1.282},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.83399999999999996,"15min":0.83199999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.98699999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.84899999999999998,"15min":0.85999999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":0.96099999999999997,"5min":1.204,"15min":1.23},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.98899999999999999,"5min":1.081,"15min":1.153},"last":0.74399999999999999}]},{"osd":3,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92100000000000004,"15min":0.89600000000000002},"min":{"1min":0.72599999999999998,"5min":0.625,"15min":0.624},"max":{"1min":1.1220000000000001,"5min":1.171,"15min":1.1719999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.90300000000000002,"15min":0.91000000000000003},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64100000000000001},"max":{"1min":1.018,"5min":1.2749999999999999,"15min":1.276},"last":0.80900000000000005}]},{"osd":4,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.82299999999999995,"15min":0.82499999999999996},"min":{"1min":0.59899999999999998,"5min":0.56200000000000006,"15min":0.45700000000000002},"max":{"1min":1.0569999999999999,"5min":1.1859999999999999,"15min":1.254},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.97199999999999998,"15min":0.97699999999999998},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.54600000000000004},"max":{"1min":1.0840000000000001,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.85199999999999998}]},{"osd":6,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.93400000000000005,"15min":0.91600000000000004},"min":{"1min":0.68200000000000005,"5min":0.61599999999999999,"15min":0.59899999999999998},"max":{"1min":1.0720000000000001,"5min":1.2130000000000001,"15min":1.248},"last":1.042},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.92800000000000005,"15min":0.92300000000000004},"min":{"1min":0.70499999999999996,"5min":0.66400000000000003,"15min":0.63900000000000001},"max":{"1min":1.099,"5min":1.2090000000000001,"15min":1.3120000000000001},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jul 21 02:43:46 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85099999999999998,"15min":0.84699999999999998},"min":{"1min":0.64400000000000002,"5min":0.56899999999999995,"15min":0.54900000000000004},"max":{"1min":1.2350000000000001,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.98799999999999999,"15min":0.99299999999999999},"min":{"1min":0.71899999999999997,"5min":0.70299999999999996,"15min":0.53300000000000003},"max":{"1min":1.198,"5min":1.242,"15min":1.3360000000000001},"last":0.628}]}]},{"osd":4,"up_from":14,"seq":60129542997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.873,"15min":0.86099999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.46300000000000002},"max":{"1min":0.99199999999999999,"5min":1.117,"15min":1.234},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.98699999999999999,"15min":0.99099999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.63400000000000001},"max":{"1min":1.097,"5min":1.226,"15min":1.258},"last":1}]},{"osd":1,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.86299999999999999,"15min":0.85799999999999998},"min":{"1min":0.61599999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0700000000000001,"5min":1.163,"15min":1.2250000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.85299999999999998,"15min":0.83799999999999997},"min":{"1min":0.61199999999999999,"5min":0.61099999999999999,"15min":0.505},"max":{"1min":1.038,"5min":1.157,"15min":1.157},"last":0.84399999999999997}]},{"osd":2,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.90800000000000003,"15min":0.90000000000000002},"min":{"1min":0.74199999999999999,"5min":0.63800000000000001,"15min":0.60299999999999998},"max":{"1min":1.1319999999999999,"5min":1.1870000000000001,"15min":1.1910000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.88100000000000001,"15min":0.85499999999999998},"min":{"1min":0.70399999999999996,"5min":0.61799999999999999,"15min":0.58999999999999997},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.1930000000000001},"last":0.89500000000000002}]},{"osd":3,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90000000000000002,"15min":0.89300000000000002},"min":{"1min":0.71399999999999997,"5min":0.63800000000000001,"15min":0.57099999999999995},"max":{"1min":1.0620000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.89000000000000001,"15min":0.89400000000000002},"min":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.65200000000000002},"max":{"1min":1.028,"5min":1.115,"15min":1.163},"last":0.92900000000000005}]},{"osd":5,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.751,"15min":0.74399999999999999},"min":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.46500000000000002},"max":{"1min":0.80600000000000005,"5min":1.1160000000000001,"15min":1.347},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.91400000000000003,"15min":0.90100000000000002},"min":{"1min":0.622,"5min":0.59799999999999998,"15min":0.54500000000000004},"max":{"1min":1.1100000000000001,"5min":1.1899999999999999,"15min":1.2609999999999999},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.98799999999999999,"15min":0.96299999999999997},"min":{"1min":0.71099999999999997,"5min":0.70499999999999996,"15min":0.47799999999999998},"max":{"1min":1.1830000000000001,"5min":1.2190000000000001,"15min":1.2569999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86699999999999999,"15min":0.85999999999999999},"min":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.57999999999999996},"max":{"1min":1.0620000000000001,"5min":1.1910000000000001,"15min":1.288},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Jul 21 02:43:38 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.66300000000000003,"5min":0.59399999999999997,"15min":0.58899999999999997},"max":{"1min":1.1040000000000001,"5min":1.1930000000000001,"15min":1.478},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.93400000000000005,"15min":0.93100000000000005},"min":{"1min":0.68899999999999995,"5min":0.61799999999999999,"15min":0.51100000000000001},"max":{"1min":1.1240000000000001,"5min":1.3660000000000001,"15min":1.4299999999999999},"last":0.86899999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542998,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050616,"kb_used_data":2032,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321224,"statfs":{"total":96636764160,"available":95560933376,"internally_reserved":1073741824,"allocated":2080768,"data_stored":1014561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65900000000000003,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.441},"max":{"1min":0.85599999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.621,"5min":0.57299999999999995,"15min":0.56299999999999994},"max":{"1min":0.89100000000000001,"5min":1.1080000000000001,"15min":1.1319999999999999},"last":0.76000000000000001}]},{"osd":2,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.48399999999999999},"max":{"1min":0.87,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.70599999999999996},"min":{"1min":0.53600000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.94899999999999995,"5min":1.1359999999999999,"15min":1.2370000000000001},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.51100000000000001,"5min":0.48399999999999999,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":1.137,"15min":1.147},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.89700000000000002,"15min":0.90000000000000002},"min":{"1min":0.72399999999999998,"5min":0.63100000000000001,"15min":0.54400000000000004},"max":{"1min":1.0820000000000001,"5min":1.2010000000000001,"15min":1.22},"last":0.83899999999999997}]},{"osd":4,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.63500000000000001},"max":{"1min":1.0409999999999999,"5min":1.105,"15min":1.196},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.218,"5min":1.218,"15min":1.26},"last":0.90500000000000003}]},{"osd":5,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.94199999999999995,"15min":0.93000000000000005},"min":{"1min":0.72199999999999998,"5min":0.71099999999999997,"15min":0.64700000000000002},"max":{"1min":1.3959999999999999,"5min":1.399,"15min":1.399},"last":0.751},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.006,"15min":1.0149999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1839999999999999,"5min":1.232,"15min":1.3520000000000001},"last":0.97199999999999998}]},{"osd":6,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.92700000000000005,"15min":0.94899999999999995},"min":{"1min":0.72999999999999998,"5min":0.69999999999999996,"15min":0.69599999999999995},"max":{"1min":1.2010000000000001,"5min":1.3400000000000001,"15min":1.3740000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.022,"5min":1.028,"15min":1.016},"min":{"1min":0.77200000000000002,"5min":0.73699999999999999,"15min":0.66500000000000004},"max":{"1min":1.304,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.88400000000000001}]},{"osd":7,"last update":"Thu Jul 21 02:43:25 2022","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.071,"15min":1.0669999999999999},"min":{"1min":0.85899999999999999,"5min":0.79300000000000004,"15min":0.69999999999999996},"max":{"1min":1.2829999999999999,"5min":1.5149999999999999,"15min":1.7310000000000001},"last":1.04},{"interface":"front","average":{"1min":1.018,"5min":1.0369999999999999,"15min":1.048},"min":{"1min":0.83299999999999996,"5min":0.78800000000000003,"15min":0.73699999999999999},"max":{"1min":1.337,"5min":1.337,"15min":1.498},"last":0.86299999999999999}]}]}]}} 2022-07-21T02:44:09.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2022-07-21T02:44:09.432 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 Processor -- start 2022-07-21T02:44:09.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 -- start start 2022-07-21T02:44:09.433 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b00b13c0 0x7fe4b00b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe4b00afd80 0x7fe4b0142910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.434 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b00ac520 con 0x7fe4b0142e00 2022-07-21T02:44:09.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 -- --> v2:172.21.15.3:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b00b17a0 con 0x7fe4b00afd80 2022-07-21T02:44:09.435 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b00b0450 con 0x7fe4b00b13c0 2022-07-21T02:44:09.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4add9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe4b00afd80 0x7fe4b0142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.436 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4add9b700 1 --2- >> v2:172.21.15.3:3300/0 conn(0x7fe4b00afd80 0x7fe4b0142910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52824/0 (socket says 172.21.15.3:52824) 2022-07-21T02:44:09.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4add9b700 1 -- 172.21.15.3:0/3657196023 learned_addr learned my addr 172.21.15.3:0/3657196023 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:44:09.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4ae59c700 1 --2- 172.21.15.3:0/3657196023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b00b13c0 0x7fe4b00b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.437 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4add9b700 1 -- 172.21.15.3:0/3657196023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b00b13c0 msgr2=0x7fe4b00b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-21T02:44:09.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4add9b700 1 --2- 172.21.15.3:0/3657196023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b00b13c0 0x7fe4b00b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4add9b700 1 -- 172.21.15.3:0/3657196023 >> v1:172.21.15.143:6789/0 conn(0x7fe4b0142e00 legacy=0x7fe4b00ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.438 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4add9b700 1 -- 172.21.15.3:0/3657196023 --> v2:172.21.15.3:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b00b06f0 con 0x7fe4b00afd80 2022-07-21T02:44:09.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4add9b700 1 --2- 172.21.15.3:0/3657196023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 0x7fe4b0142910 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7fe4a4002f70 tx=0x7fe4a4006eb0).ready entity=mon.0 client_cookie=fdc3fa3fa3b8ecf0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:44:09.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4ad59a700 1 -- 172.21.15.3:0/3657196023 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4a400d570 con 0x7fe4b00afd80 2022-07-21T02:44:09.439 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4ad59a700 1 -- 172.21.15.3:0/3657196023 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a4000e90 con 0x7fe4b00afd80 2022-07-21T02:44:09.440 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4ad59a700 1 -- 172.21.15.3:0/3657196023 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4a400d900 con 0x7fe4b00afd80 2022-07-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 -- 172.21.15.3:0/3657196023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 msgr2=0x7fe4b0142910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 --2- 172.21.15.3:0/3657196023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 0x7fe4b0142910 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7fe4a4002f70 tx=0x7fe4a4006eb0).stop 2022-07-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 -- 172.21.15.3:0/3657196023 shutdown_connections 2022-07-21T02:44:09.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 --2- 172.21.15.3:0/3657196023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 0x7fe4b0142910 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.443 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 --2- 172.21.15.3:0/3657196023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b00b13c0 0x7fe4b00b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.444 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 -- 172.21.15.3:0/3657196023 >> 172.21.15.3:0/3657196023 conn(0x7fe4b00626f0 msgr2=0x7fe4b0062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:44:09.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 -- 172.21.15.3:0/3657196023 shutdown_connections 2022-07-21T02:44:09.445 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.418+0000 7fe4af59e700 1 -- 172.21.15.3:0/3657196023 wait complete. 2022-07-21T02:44:09.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 Processor -- start 2022-07-21T02:44:09.446 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 -- start start 2022-07-21T02:44:09.447 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 0x7fe4b00a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b00b13c0 0x7fe4b0144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.448 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b0142e00 0x7fe4b0155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.449 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b009fc80 con 0x7fe4b00afd80 2022-07-21T02:44:09.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b0147280 con 0x7fe4b00b13c0 2022-07-21T02:44:09.450 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b0146d40 con 0x7fe4b0142e00 2022-07-21T02:44:09.451 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 0x7fe4b00a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 0x7fe4b00a00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52830/0 (socket says 172.21.15.3:52830) 2022-07-21T02:44:09.452 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4ae59c700 1 -- 172.21.15.3:0/3770288122 learned_addr learned my addr 172.21.15.3:0/3770288122 (peer_addr_for_me v2:172.21.15.3:0/0) 2022-07-21T02:44:09.453 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4add9b700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b00b13c0 0x7fe4b0144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.454 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4aed9d700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b0142e00 0x7fe4b0155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4ae59c700 1 -- 172.21.15.3:0/3770288122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b0142e00 msgr2=0x7fe4b0155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.455 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4ae59c700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b0142e00 0x7fe4b0155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.456 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4ae59c700 1 -- 172.21.15.3:0/3770288122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b00b13c0 msgr2=0x7fe4b0144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.457 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4ae59c700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b00b13c0 0x7fe4b0144680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4ae59c700 1 -- 172.21.15.3:0/3770288122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a400b1e0 con 0x7fe4b00afd80 2022-07-21T02:44:09.458 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4ae59c700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 0x7fe4b00a00e0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7fe4a000c000 tx=0x7fe4a0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:44:09.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe49f7fe700 1 -- 172.21.15.3:0/3770288122 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4a0010780 con 0x7fe4b00afd80 2022-07-21T02:44:09.459 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe49f7fe700 1 -- 172.21.15.3:0/3770288122 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a00108e0 con 0x7fe4b00afd80 2022-07-21T02:44:09.460 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b0146fe0 con 0x7fe4b00afd80 2022-07-21T02:44:09.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe49f7fe700 1 -- 172.21.15.3:0/3770288122 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4a0010c40 con 0x7fe4b00afd80 2022-07-21T02:44:09.461 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.422+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b0144b70 con 0x7fe4b00afd80 2022-07-21T02:44:09.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.426+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b0147670 con 0x7fe4b00afd80 2022-07-21T02:44:09.462 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.426+0000 7fe49f7fe700 1 -- 172.21.15.3:0/3770288122 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe4a001c460 con 0x7fe4b00afd80 2022-07-21T02:44:09.463 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.426+0000 7fe49f7fe700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe48c053b00 0x7fe48c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-21T02:44:09.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.426+0000 7fe4add9b700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe48c053b00 0x7fe48c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-21T02:44:09.464 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.426+0000 7fe49f7fe700 1 -- 172.21.15.3:0/3770288122 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5686+0+0 (secure 0 0 0) 0x7fe4a006c280 con 0x7fe4b00afd80 2022-07-21T02:44:09.465 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.426+0000 7fe4add9b700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe48c053b00 0x7fe48c055fc0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7fe4a401c000 tx=0x7fe4a4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-21T02:44:09.466 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.430+0000 7fe49f7fe700 1 -- 172.21.15.3:0/3770288122 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4a00431b0 con 0x7fe4b00afd80 2022-07-21T02:44:09.562 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.562+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fe4b00574e0 con 0x7fe4b00afd80 2022-07-21T02:44:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.566+0000 7fe49f7fe700 1 -- 172.21.15.3:0/3770288122 <== mon.0 v2:172.21.15.3:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe4a00431b0 con 0x7fe4b00afd80 2022-07-21T02:44:09.569 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.566+0000 7fe49f7fe700 1 -- 172.21.15.3:0/3770288122 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v25)=0 v25) v1 ==== 123+0+0 (secure 0 0 0) 0x7fe4a00467d0 con 0x7fe4b00afd80 2022-07-21T02:44:09.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe48c053b00 msgr2=0x7fe48c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.572 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe48c053b00 0x7fe48c055fc0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7fe4a401c000 tx=0x7fe4a4015000).stop 2022-07-21T02:44:09.573 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 msgr2=0x7fe4b00a00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-21T02:44:09.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 0x7fe4b00a00e0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7fe4a000c000 tx=0x7fe4a0006df0).stop 2022-07-21T02:44:09.574 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 shutdown_connections 2022-07-21T02:44:09.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4b0142e00 0x7fe4b0155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.575 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:6800/22700,v1:172.21.15.3:6801/22700] conn(0x7fe48c053b00 0x7fe48c055fc0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.576 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b00b13c0 0x7fe4b0144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 --2- 172.21.15.3:0/3770288122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4b00afd80 0x7fe4b00a00e0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-21T02:44:09.577 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 >> 172.21.15.3:0/3770288122 conn(0x7fe4b00626f0 msgr2=0x7fe4b013d650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-21T02:44:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 shutdown_connections 2022-07-21T02:44:09.579 INFO:teuthology.orchestra.run.smithi003.stderr:2022-07-21T02:44:09.570+0000 7fe4af59e700 1 -- 172.21.15.3:0/3770288122 wait complete. 2022-07-21T02:44:09.584 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph.py", line 1858, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-21T02:44:09.595 INFO:teuthology.misc:Shutting down mds daemons... 2022-07-21T02:44:09.597 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-21T02:44:09.598 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:09.613 INFO:tasks.ceph.mds.a:Stopped 2022-07-21T02:44:09.615 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2022-07-21T02:44:09.615 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:09.631 INFO:tasks.ceph.mds.c:Stopped 2022-07-21T02:44:09.633 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-21T02:44:09.634 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:09.651 INFO:tasks.ceph.mds.b:Stopped 2022-07-21T02:44:09.653 INFO:teuthology.misc:Shutting down osd daemons... 2022-07-21T02:44:09.654 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2022-07-21T02:44:09.655 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:09.741 INFO:tasks.ceph.osd.0:Stopped 2022-07-21T02:44:09.743 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2022-07-21T02:44:09.744 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:09.809 INFO:tasks.ceph.osd.1:Stopped 2022-07-21T02:44:09.810 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2022-07-21T02:44:09.811 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:09.889 INFO:tasks.ceph.osd.2:Stopped 2022-07-21T02:44:09.891 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2022-07-21T02:44:09.892 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:09.954 INFO:tasks.ceph.osd.3:Stopped 2022-07-21T02:44:09.957 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2022-07-21T02:44:09.958 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:10.054 INFO:tasks.ceph.osd.4:Stopped 2022-07-21T02:44:10.055 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2022-07-21T02:44:10.056 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:10.131 INFO:tasks.ceph.osd.5:Stopped 2022-07-21T02:44:10.133 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2022-07-21T02:44:10.134 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:10.209 INFO:tasks.ceph.osd.6:Stopped 2022-07-21T02:44:10.210 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2022-07-21T02:44:10.211 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:10.287 INFO:tasks.ceph.osd.7:Stopped 2022-07-21T02:44:10.288 INFO:teuthology.misc:Shutting down mgr daemons... 2022-07-21T02:44:10.289 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2022-07-21T02:44:10.290 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:10.317 INFO:tasks.ceph.mgr.y:Stopped 2022-07-21T02:44:10.318 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2022-07-21T02:44:10.319 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:10.351 INFO:tasks.ceph.mgr.x:Stopped 2022-07-21T02:44:10.352 INFO:teuthology.misc:Shutting down mon daemons... 2022-07-21T02:44:10.354 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2022-07-21T02:44:10.355 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:10.384 INFO:tasks.ceph.mon.a:Stopped 2022-07-21T02:44:10.385 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2022-07-21T02:44:10.386 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:10.408 INFO:tasks.ceph.mon.b:Stopped 2022-07-21T02:44:10.409 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2022-07-21T02:44:10.410 INFO:teuthology.orchestra.run:waiting for 300 2022-07-21T02:44:10.475 INFO:tasks.ceph.mon.c:Stopped 2022-07-21T02:44:10.476 INFO:tasks.ceph:Checking cluster log for badness... 2022-07-21T02:44:10.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'evicting unresponsive client' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'has not responded to cap revoke by MDS for over' | egrep -v MDS_CLIENT_LATE_RELEASE | egrep -v 'responding to mclientcaps' | head -n 1 2022-07-21T02:44:10.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2022-07-21T02:44:10.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2022-07-21T02:44:10.515 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T02:44:10.517 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2022-07-21T02:44:10.812 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T02:44:10.813 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2022-07-21T02:44:11.031 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T02:44:11.032 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2022-07-21T02:44:11.222 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T02:44:11.223 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2022-07-21T02:44:11.389 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T02:44:11.390 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2022-07-21T02:44:11.688 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T02:44:11.689 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2022-07-21T02:44:11.927 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T02:44:11.929 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2022-07-21T02:44:12.115 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T02:44:12.118 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2022-07-21T02:44:12.282 INFO:tasks.ceph:Archiving mon data... 2022-07-21T02:44:12.288 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/data/mon.a.tgz 2022-07-21T02:44:12.291 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2022-07-21T02:44:12.300 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.s556ygNFty 2022-07-21T02:44:12.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.s556ygNFty 2022-07-21T02:44:12.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.s556ygNFty 2022-07-21T02:44:12.640 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.s556ygNFty is 537KB 2022-07-21T02:44:12.707 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.s556ygNFty 2022-07-21T02:44:12.722 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/data/mon.b.tgz 2022-07-21T02:44:12.725 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T02:44:12.736 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.exTpbO9ZRc 2022-07-21T02:44:12.737 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.exTpbO9ZRc 2022-07-21T02:44:13.003 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.exTpbO9ZRc 2022-07-21T02:44:13.076 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.exTpbO9ZRc is 646KB 2022-07-21T02:44:13.149 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.exTpbO9ZRc 2022-07-21T02:44:13.158 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/data/mon.c.tgz 2022-07-21T02:44:13.160 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2022-07-21T02:44:13.168 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.EMxGmSwxkP 2022-07-21T02:44:13.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.EMxGmSwxkP 2022-07-21T02:44:13.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.EMxGmSwxkP 2022-07-21T02:44:13.454 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.EMxGmSwxkP is 646KB 2022-07-21T02:44:13.532 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.EMxGmSwxkP 2022-07-21T02:44:13.540 INFO:tasks.ceph:Cleaning ceph cluster... 2022-07-21T02:44:13.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-21T02:44:13.546 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-21T02:44:13.550 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-21T02:44:13.555 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-21T02:44:13.573 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2022-07-21T02:44:13.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-21T02:44:13.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-21T02:44:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-21T02:44:13.608 DEBUG:teuthology.orchestra.run.smithi171:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-21T02:44:13.611 DEBUG:teuthology.orchestra.run.smithi174:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-21T02:44:13.632 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-21T02:44:13.639 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-21T02:44:13.642 INFO:teuthology.orchestra.run.smithi174.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-21T02:44:13.649 INFO:tasks.ceph:Archiving crash dumps... 2022-07-21T02:44:13.655 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi003/crash 2022-07-21T02:44:13.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-21T02:44:13.680 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi143/crash 2022-07-21T02:44:13.683 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-21T02:44:13.707 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi171/crash 2022-07-21T02:44:13.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-21T02:44:13.730 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi174/crash 2022-07-21T02:44:13.732 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-21T02:44:13.752 INFO:tasks.ceph:Shutting down logrotate 2022-07-21T02:44:13.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:44:13.756 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:44:13.760 DEBUG:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:44:13.763 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-21T02:44:13.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-21T02:44:13.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-21T02:44:13.863 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-21T02:44:13.881 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-21T02:44:13.898 INFO:tasks.ceph:Compressing logs... 2022-07-21T02:44:13.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-21T02:44:13.902 DEBUG:teuthology.orchestra.run.smithi143:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-21T02:44:13.906 DEBUG:teuthology.orchestra.run.smithi171:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-21T02:44:13.909 DEBUG:teuthology.orchestra.run.smithi174:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-21T02:44:15.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2022-07-21T02:44:15.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2022-07-21T02:44:28.523 INFO:tasks.ceph:Archiving logs... 2022-07-21T02:44:28.525 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi003/log 2022-07-21T02:44:28.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-21T02:44:32.407 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi143/log 2022-07-21T02:44:32.409 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-21T02:44:34.915 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi171/log 2022-07-21T02:44:34.917 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-21T02:44:35.349 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi174/log 2022-07-21T02:44:35.351 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-21T02:44:35.387 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-07-21T02:44:35.402 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/task/install/__init__.py", line 619, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-21T02:44:35.406 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-07-21T02:44:35.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-21T02:44:35.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-21T02:44:35.418 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-21T02:44:35.421 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-21T02:44:35.548 DEBUG:teuthology.run_tasks:Unwinding manager clock 2022-07-21T02:44:35.571 INFO:teuthology.task.clock:Checking final clock skew... 2022-07-21T02:44:35.572 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-21T02:44:35.575 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-21T02:44:35.578 DEBUG:teuthology.orchestra.run.smithi171:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-21T02:44:35.581 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-21T02:44:35.600 INFO:teuthology.orchestra.run.smithi003.stdout: remote refid st t when poll reach delay offset jitter 2022-07-21T02:44:35.601 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================== 2022-07-21T02:44:35.601 INFO:teuthology.orchestra.run.smithi003.stdout:+hv01.front.sepi 162.248.241.94 3 u 42 128 377 0.179 2.210 0.273 2022-07-21T02:44:35.602 INFO:teuthology.orchestra.run.smithi003.stdout:*hv02.front.sepi 163.237.218.18 2 u 37 128 377 0.205 -4.571 0.417 2022-07-21T02:44:35.603 INFO:teuthology.orchestra.run.smithi003.stdout:+hv03.front.sepi 216.229.4.69 3 u 39 128 377 0.182 -2.258 0.194 2022-07-21T02:44:35.604 INFO:teuthology.orchestra.run.smithi003.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-21T02:44:35.606 INFO:teuthology.orchestra.run.smithi171.stdout: remote refid st t when poll reach delay offset jitter 2022-07-21T02:44:35.607 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================== 2022-07-21T02:44:35.608 INFO:teuthology.orchestra.run.smithi171.stdout:+hv01.front.sepi 162.248.241.94 3 u 45 128 377 0.182 3.257 0.219 2022-07-21T02:44:35.608 INFO:teuthology.orchestra.run.smithi171.stdout:*hv02.front.sepi 163.237.218.18 2 u 66 128 377 0.170 -3.271 0.381 2022-07-21T02:44:35.609 INFO:teuthology.orchestra.run.smithi171.stdout:+hv03.front.sepi 216.229.4.69 3 u 114 128 377 0.191 -1.429 0.091 2022-07-21T02:44:35.610 INFO:teuthology.orchestra.run.smithi171.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-21T02:44:35.610 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2022-07-21T02:44:35.611 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2022-07-21T02:44:35.612 INFO:teuthology.orchestra.run.smithi143.stdout:+hv01.front.sepi 162.248.241.94 3 u 44 128 377 0.193 2.304 0.312 2022-07-21T02:44:35.612 INFO:teuthology.orchestra.run.smithi143.stdout:*hv02.front.sepi 163.237.218.18 2 u 97 128 377 0.203 -3.895 0.350 2022-07-21T02:44:35.612 INFO:teuthology.orchestra.run.smithi143.stdout:+hv03.front.sepi 216.229.4.69 3 u 46 128 377 0.217 -2.062 0.114 2022-07-21T02:44:35.613 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-21T02:44:35.614 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2022-07-21T02:44:35.615 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2022-07-21T02:44:35.616 INFO:teuthology.orchestra.run.smithi174.stdout:+hv01.front.sepi 216.240.36.24 3 u 112 256 377 0.124 2.726 0.264 2022-07-21T02:44:35.616 INFO:teuthology.orchestra.run.smithi174.stdout:*hv02.front.sepi 163.237.218.18 2 u 181 256 377 0.200 -4.090 0.318 2022-07-21T02:44:35.617 INFO:teuthology.orchestra.run.smithi174.stdout:+hv03.front.sepi 216.229.4.69 3 u 113 256 377 0.185 -1.846 0.297 2022-07-21T02:44:35.618 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-21T02:44:35.618 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-07-21T02:44:35.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-07-21T02:44:35.636 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-07-21T02:44:35.658 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-07-21T02:44:35.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-07-21T02:44:35.689 INFO:teuthology.task.internal:Duration was 4733.294363 seconds 2022-07-21T02:44:35.692 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-07-21T02:44:35.711 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-07-21T02:44:35.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-21T02:44:35.716 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-21T02:44:35.719 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-21T02:44:35.723 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-21T02:44:35.777 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-07-21T02:44:35.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2022-07-21T02:44:35.778 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-21T02:44:35.795 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2022-07-21T02:44:35.796 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-21T02:44:35.818 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi171.front.sepia.ceph.com 2022-07-21T02:44:35.819 DEBUG:teuthology.orchestra.run.smithi171:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-21T02:44:35.840 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2022-07-21T02:44:35.840 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-21T02:44:35.859 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-07-21T02:44:35.861 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-21T02:44:35.865 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-21T02:44:35.869 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-21T02:44:35.884 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-21T02:44:35.917 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-07-21T02:44:35.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-21T02:44:35.943 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-21T02:44:35.946 DEBUG:teuthology.orchestra.run.smithi171:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-21T02:44:35.949 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-21T02:44:36.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-07-21T02:44:36.114 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-07-21T02:44:36.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-21T02:44:36.136 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-21T02:44:36.158 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-21T02:44:36.175 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-21T02:44:36.193 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-07-21T02:44:36.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-21T02:44:36.210 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-21T02:44:36.214 DEBUG:teuthology.orchestra.run.smithi171:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-21T02:44:36.219 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-21T02:44:36.223 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2022-07-21T02:44:36.225 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2022-07-21T02:44:36.230 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = core 2022-07-21T02:44:36.250 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2022-07-21T02:44:36.264 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-21T02:44:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T02:44:36.273 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-21T02:44:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T02:44:36.311 DEBUG:teuthology.orchestra.run.smithi171:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-21T02:44:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T02:44:36.319 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-21T02:44:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T02:44:36.327 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-07-21T02:44:36.344 INFO:teuthology.task.internal:Transferring archived files... 2022-07-21T02:44:36.346 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi003 2022-07-21T02:44:36.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-21T02:44:36.393 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi143 2022-07-21T02:44:36.395 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-21T02:44:36.438 DEBUG:teuthology.misc:Transferring archived files from smithi171:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi171 2022-07-21T02:44:36.439 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-21T02:44:36.481 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-20_20:56:09-fs-octopus-release-distro-default-smithi/6941359/remote/smithi174 2022-07-21T02:44:36.482 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-21T02:44:36.530 INFO:teuthology.task.internal:Removing archive directory... 2022-07-21T02:44:36.532 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-21T02:44:36.536 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-21T02:44:36.539 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-21T02:44:36.545 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-21T02:44:36.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-07-21T02:44:36.571 INFO:teuthology.task.internal:Not uploading archives. 2022-07-21T02:44:36.572 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-07-21T02:44:36.588 INFO:teuthology.task.internal:Tidying up after the test... 2022-07-21T02:44:36.589 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-21T02:44:36.592 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-21T02:44:36.595 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-21T02:44:36.597 INFO:teuthology.orchestra.run.smithi003.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 21 02:44 /home/ubuntu/cephtest 2022-07-21T02:44:36.598 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-21T02:44:36.601 INFO:teuthology.orchestra.run.smithi143.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 21 02:44 /home/ubuntu/cephtest 2022-07-21T02:44:36.610 INFO:teuthology.orchestra.run.smithi174.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 21 02:44 /home/ubuntu/cephtest 2022-07-21T13:25:59.418 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-07-21T13:25:59.454 DEBUG:teuthology.task.console_log:Killing console logger for smithi003 2022-07-21T13:25:59.463 DEBUG:teuthology.task.console_log:Killing console logger for smithi143 2022-07-21T13:25:59.464 DEBUG:teuthology.task.console_log:Killing console logger for smithi171 2022-07-21T13:25:59.465 DEBUG:teuthology.task.console_log:Killing console logger for smithi174 2022-07-21T13:25:59.465 DEBUG:teuthology.exit:Finished running handlers